Commit Graph

1796 Commits

Author SHA1 Message Date
Alisdair McDiarmid c798dc98db command: Show diffs when only sensitivity changes
When an attribute's sensitivity changes, but its value remains the same,
we consider this an update operation for the plan. This commit updates
the diff renderer to match this, detecting and displaying the change in
sensitivity.

Previously, the renderer would detect no changes to the value of the
attribute, and consider it a no-op action. This resulted in suppression
of the attribute when the plan is in concise mode.

This is achieved with a new helper function, ctyEqualValueAndMarks. We
call this function whenever we want to check that two values are equal
in order to determine whether the action is update or no-op.
2020-10-13 13:55:16 -04:00
James Bardin 5eca0788c6 rely solely on the plan changes for outputs
Now that outputs changes are tracked in full, we can remove the
comparisons with the prior state and use the planned changes directly.
2020-10-12 18:59:14 -04:00
James Bardin 03640057be
Merge pull request #26533 from hashicorp/jbardin/plan-output-changes
Use recorded changes for outputs and plan root output removals
2020-10-12 17:35:36 -04:00
James Bardin d2514a9abd update new outputs plan json 2020-10-12 17:29:45 -04:00
Martin Atkins e1cf0ac801 internal/depsfile: Control how the "hashes" value is formatted
Previously we were just letting hclwrite do its default formatting
behavior here. The current behavior there isn't ideal anyway -- it puts
big data structures all on one line -- but even ignoring that our goal
for this file format is to keep things in a highly-normalized shape so
that diffs against the file are clear and easy to read.

With that in mind, here we directly control how we write that value into
the file, which means that later changes to hclwrite's list/set
presentation won't affect it, regardless of what form they take.
2020-10-09 09:26:23 -07:00
Martin Atkins e270291f19 command: terraform providers lock
This command is intended to help support situations where Terraform is
configured to use only local mirrors for provider installation and so the
normal "terraform init" flow would not have direct access to the official
package checksums published in the origin registry.

The intended workflow here is to use this command only when adding a new
provider or changing an existing provider's version in the configuration,
to augment the lock file with all of the checksums required to verify
the provider across a variety of different platforms. Once this command
has recorded all of the official checksums, future runs of
"terraform init" will verify that provider packages obtained from a local
mirror match with those upstream checksums.
2020-10-09 09:26:23 -07:00
Martin Atkins 2611e08430 command/init: Mention using the lock file for provider selection
This probably isn't the best UI we could do here, but it's a placeholder
for now just to avoid making it seem like we're ignoring the lock file
and checking for new versions anyway.
2020-10-09 09:26:23 -07:00
Martin Atkins b3f5c7f1e6 command/init: Read, respect, and update provider dependency locks
This changes the approach used by the provider installer to remember
between runs which selections it has previously made, using the lock file
format implemented in internal/depsfile.

This means that version constraints in the configuration are considered
only for providers we've not seen before or when -upgrade mode is active.
2020-10-09 09:26:23 -07:00
Pam Selle eb2a027684
Merge pull request #26518 from hashicorp/pselle/sensitive-experiment
Remove sensitive_variables experiment
2020-10-08 16:14:37 -04:00
James Bardin 57a47d0b82
Merge pull request #26519 from hashicorp/jbardin/FIXME
Cleanup and remove legacy types from test mocks
2020-10-08 14:54:40 -04:00
James Bardin 027a6c2cf6 update command tests to use new mock provider 2020-10-08 13:49:41 -04:00
Pam Selle bc57c20d10 Remove sensitive_variables experiment
Ahead of the beta, remove the sensitive_variable experiment
and update tests accordingly
2020-10-08 11:22:20 -04:00
Kristin Laemmert 04be220f5f deprecate helper/copy
helper/copy CopyDir was used heavily in tests. It differes from
internal/copydir in a few ways, the main one being that it creates the
dst directory while the internal version expected the dst to exist
(there are other differences, which is why I did not just switch tests
to using internal's CopyDir).

I moved the CopyDir func from helper/copy into command_test.go; I could
also have moved it into internal/copy and named it something like
CreateDirAndCopy so if that seems like a better option please let me
know.

helper/copy/CopyFile was used in a couple of spots so I moved it into
internal, at which point I thought it made more sense to rename the
package copy (instead of copydir).

There's also a `go mod tidy` included.
2020-10-08 08:42:16 -04:00
Alisdair McDiarmid d05e3b40bf
Merge pull request #26492 from hashicorp/alisdair/sensitive-value-force-replacement
command: Fix missing force new for sensitive vars and blocks
2020-10-07 11:10:36 -04:00
Kristin Laemmert d2e999ba1f
remove unused code (#26503)
* remove unused code

I've removed the provider-specific code under registry, and unused nil
backend, and replaced a call to helper from backend/oss (the other
callers of that func are provisioners scheduled to be deprecated).

I also removed the Dockerfile, as our build process uses a different
file.

Finally I removed the examples directory, which had outdated examples
and links. There are better, actively maintained examples available.

* command: remove various unused bits

* test wasn't running

* backend: remove unused err
2020-10-07 11:00:06 -04:00
Alisdair McDiarmid 62e6f56a50 command: Fix missing force new for sensitive blocks
If an entire block is marked sensitive (possibly because it is of type
NestedSet) and results in replacement of the resource, we should render
the standard "forces replacement" text after the opening line of the
block.
2020-10-07 10:50:54 -04:00
Alisdair McDiarmid 79a3e33c4d command: Fix missing force new for sensitive vars
If a value rendered for the diff is sensitive and results in replacement
of the resource, we should render the standard "forces replacement" text
after the "(sensitive)" value display.
2020-10-06 13:05:30 -04:00
Alisdair McDiarmid eb873f5021
Merge pull request #25725 from FGtatsuro/diff_heredoc_hyphen
Use valid heredoc begin symbol in diff output.
2020-10-06 12:52:23 -04:00
Pam Selle c57ca152e6 Obfuscate sensitive vals in console
Updates terraform console to show "(sensitive)"
when a value is marked as sensitive.
2020-10-05 13:16:34 -04:00
Kristin Laemmert 3933cbd491
remove LegacyProvider (#26433) 2020-10-05 08:33:49 -04:00
Pam Selle 111aadd0f0 Extend tests further 2020-10-02 15:01:17 -04:00
Pam Selle 55c96da27e Move nested block printing to own method for readability 2020-10-02 14:56:50 -04:00
Pam Selle 2ec95f1abc Make an after val unknown to exercise the known check 2020-10-02 13:11:55 -04:00
Pam Selle 73b1d8b0d1 Add special diff for nested blocks
When a value in a nested block is marked as sensitive,
it will result in the behavior of every member of
that block being sensitive. As such, show a
specialized diff that reduces noise of (sensitive)
for many attributes that we won't show. Also update
the warning to differentiate between showing a warning
for an attribute or warning for blocks.
2020-10-02 13:11:55 -04:00
Pam Selle 6617c2729c Test additions for nested blocks
Add some coverage for in-place on nested block

Add nested block to deletion test
2020-10-02 13:04:54 -04:00
Pam Selle 3e7be13dff Update ordering for marking/unmarking and asserting plan valid
Update when we unmark objects so we can assert the plan is valid,
and process UnknownAsNull on the unmarked value
2020-10-02 13:03:11 -04:00
Pam Selle 26f77564d7
Merge pull request #26459 from hashicorp/pselle/sensitive-vals-map
Add format/diff coverage for maps with sensitivity
2020-10-02 13:00:00 -04:00
Pam Selle 1817c8ac3c
Merge pull request #26412 from OwenTuz/issue-26411-fix-failing-e2e-test-provider-not-found
Fix bad string match that was causing TestInitProviderNotFound to fail
2020-10-02 10:43:18 -04:00
Pam Selle 52b6f7f53e Move common IsMarked checks above switch statement 2020-10-01 14:34:44 -04:00
Pam Selle 1780351636 If the whole map is marked, have the same sensitivity behavior as a single value 2020-10-01 14:30:33 -04:00
Pam Selle 0520f143a2 Add diff coverage for maps
Considers wholly marked maps as well
as map keys
2020-10-01 14:18:40 -04:00
Kristin Laemmert d76cfc8c0c
Merge pull request #26440 from hashicorp/mildwonkey/remove-older-state-vs
refactor tests to use modern states.State in favor of terraform.State where possible
2020-10-01 08:11:27 -04:00
Kristin Laemmert 479655ad47 refactor tests to use modern states.State in favor of terraform.State where possible 2020-09-30 16:07:54 -04:00
James Bardin c51104fb7c
Merge pull request #26435 from hashicorp/jbardin/races
Fix race conditions
2020-09-30 15:19:58 -04:00
James Bardin 59110a2ca5 e2etest server was unsynchronized 2020-09-30 14:28:02 -04:00
Martin Atkins 59b116f7bf command/init: Remove support for legacy provider addresses
We no longer need to support 0.12-and-earlier-style provider addresses
because users should've upgraded their existing configurations and states
on Terraform 0.13 already.

For now this is only checked in the "init" command, because various test
shims are still relying on the idea of legacy providers the core layer.
However, rejecting these during init is sufficient grounds to avoid
supporting legacy provider addresses in the new dependency lock file
format, and thus sets the stage for a more severe removal of legacy
provider support in a later commit.
2020-09-30 08:54:57 -07:00
Owen Tuz 5453147381 Fix bad string match that was causing TestInitProviderNotFound to fail on Macs
The tmp directory path is longer on Macs than other systems and was wrapping
across lines when printed, breaking the string match in the test.

Fix suggested by @apparentlymart is to add two spaces before the leading
'-' when printing to prevent the diagnostic renderer wrapping the line.
2020-09-30 09:30:02 +01:00
Martin Atkins f53264d378 command/init: Better error message for provider unsupported platform
As we add and remove support for different target platforms over time,
there will be transition periods where the available platforms for
each provider will be different than the available platforms for Terraform
CLI itself.

In recognition of that possibility, here we add a more specialized error
message for that situation which tries to explain the problem a little
more clearly than the generic error message that came before it.

In an ideal world we'd be able to detect situations where a newer or older
version has support in a similar vein to what we do with provider protocol
incompatibilities, but we don't currently have logic to fetch the data
necessary to implement that, so this is instead a presentation-only change
aimed at allowing some possible near-future changes to the supported
target platforms.
2020-09-29 10:28:26 -07:00
Martin Atkins 0b734a2803 command: Make provider installation interruptible
In earlier commits we started to make the installation codepath
context-aware so that it could be canceled in the event of a SIGINT, but
we didn't complete wiring that through the API of the getproviders
package.

Here we make the getproviders.Source interface methods, along with some
other functions that can make network requests, take a context.Context
argument and act appropriately if that context is cancelled.

The main providercache.Installer.EnsureProviderVersions method now also
has some context-awareness so that it can abort its work early if its
context reports any sort of error. That avoids waiting for the process
to wind through all of the remaining iterations of the various loops,
logging each request failure separately, and instead returns just
a single aggregate "canceled" error.

We can then set things up in the "terraform init" and
"terraform providers mirror" commands so that the context will be
cancelled if we get an interrupt signal, allowing provider installation
to abort early while still atomically completing any local-side effects
that may have started.
2020-09-29 10:00:35 -07:00
Martin Atkins f0ccee854c command/0.13upgrade: Remove this subcommand
We only preserve these major upgrade versions for one major version after
they are added, because our upgrade path assumes moving forward only one
major version at a time. Now that our main branch is tracking towards
Terraform 0.14, we no longer need the 0.13upgrade subcommand.

This also includes some minor adjustments to the 0.12upgrade command to
align the terminology used in the output of both commands. We usually
use the word "deprecated" to mean that something is still available but
no longer recommended, but neither of these commands is actually available
so "removed" is clearer.

We could in principle have removed even the removal notice for 0.12upgrade
here, but it's relatively little code and not a big deal to keep around
to help prompt those who might try to upgrade directly from 0.11 to 0.14.
We may still remove the historical configuration upgrade commands prior to
releasing Terraform 1.0, though.
2020-09-29 10:00:35 -07:00
Kristin Laemmert fc94c819e5
command: remove unused method on pluginSHA256LockFile (#26402)
* tmp is haunted
* remove unused code
2020-09-29 08:46:51 -04:00
Nick McClendon 529ee04269
Fix taint and untaint commands when in a workspace (#22467)
* Fix taint and untaint commands when in a workspace

Fixes #22157. Removes DefaultStateFilepath as the default for the
-state flag, allowing workspaces to be used properly.

* update test with modern state types

Co-authored-by: Kristin Laemmert <mildwonkey@users.noreply.github.com>
2020-09-28 13:09:37 -04:00
Martin Atkins ff0dbd6215 command/fmt: Restore some opinionated behaviors
In Terraform 0.11 and earlier, the "terraform fmt" command was very
opinionated in the interests of consistency. While that remains its goal,
for pragmatic reasons Terraform 0.12 significantly reduced the number
of formatting behaviors in the fmt command. We've held off on introducing
0.12-and-later-flavored cleanups out of concern it would make it harder
to maintain modules that are cross-compatible with both Terraform 0.11
and 0.12, but with this aimed to land in 0.14 -- two major releases
later -- our new goal is to help those who find older Terraform language
examples learn about the more modern idiom.

More rules may follow later, now that the implementation is set up to
allow modifications to tokens as well as modifications to whitespace, but
for this initial pass the command will now apply the following formatting
conventions:

 - 0.11-style quoted variable type constraints will be replaced with their
   0.12 syntax equivalents. For example, "string" becomes just string.
   (This change quiets a deprecation warning.)
 - Collection type constraints that don't specify an element type will
   be rewritten to specify the "any" element type explicitly, so
   list becomes list(any).
 - Arguments whose expressions consist of a quoted string template with
   only a single interpolation sequence inside will be "unwrapped" to be
   the naked expression instead, which is functionally equivalent.
   (This change quiets a deprecation warning.)
 - Block labels are given in quotes.

Two of the rules above are coming from a secondary motivation of
continuing down the deprecation path for two existing warnings, so authors
can have two active deprecation warnings quieted automatically by
"terraform fmt", without the need to run any third-party tools.

All of these rules match with current documented idiom as shown in the
Terraform documentation, so anyone who follows the documented style should
see no changes as a result of this. Those who have adopted other local
style will see their configuration files rewritten to the standard
Terraform style, but it should not make any changes that affect the
functionality of the configuration.

There are some further similar rewriting rules that could be added in
future, such as removing 0.11-style quotes around various keyword or
static reference arguments, but this initial pass focused only on some
rules that have been proven out in the third-party tool
terraform-clean-syntax, from which much of this commit is a direct port.

For now this doesn't attempt to re-introduce any rules about vertical
whitespace, even though the 0.11 "terraform fmt" would previously apply
such changes. We'll be more cautious about those because the results of
those rules in Terraform 0.11 were often sub-optimal and so we'd prefer
to re-introduce those with some care to the implications for those who
may be using vertical formatting differences for some semantic purpose,
like grouping together related arguments.
2020-09-28 09:04:03 -07:00
Martin Atkins 7951a6db0d command/fmt: Format using the full hclwrite syntax tree
Previously we were just using hclwrite.Format, a token-only formatting
pass. Now we'll do that via the full hclwrite parser, getting the
formatting as a side-effect of the parsing and re-serialization.

This should have no change in observable behavior as-is, but in a future
commit we'll add some additional processing rules that modify the syntax
tree before re-serializing it.
2020-09-28 09:04:03 -07:00
Martin Atkins 05f6a62399 command/fmt: Factor out the actual formatting
Previously formatting was just a simple wrapper around hclwrite.Format.
That remains true here, but the call is factored out into a separate
method in preparation for making it also do some Terraform-specific
cleanups in a future commit.
2020-09-28 09:04:03 -07:00
Pam Selle 40ea3f4cb8
Merge pull request #26373 from hashicorp/pselle/sensitive-vals-list
Support list diffs with sensitivity
2020-09-25 13:46:37 -04:00
Pam Selle 634e83ab63 Change sensitivity warning to be yellow only on 'Warning' 2020-09-25 10:22:56 -04:00
Pam Selle 3dde9efc75 Support list diffs with sensitivity
Adds support for specialized diffs with lists
2020-09-25 10:18:33 -04:00
Pam Selle f2f84003ee
Merge pull request #26367 from hashicorp/pselle/sensitive-diff-format
Warnings and specialized diffs when switching between sensitive values
2020-09-24 17:45:50 -04:00
Martin Atkins 6694cfaa0e getproviders: Add a real type Hash for package hashes
The logic for what constitutes a valid hash and how different hash schemes
are represented was starting to get sprawled over many different files and
packages.

Consistently with other cases where we've used named types to gather the
definition of a particular string into a single place and have the Go
compiler help us use it properly, this introduces both getproviders.Hash
representing a hash value and getproviders.HashScheme representing the
idea of a particular hash scheme.

Most of this changeset is updating existing uses of primitive strings to
uses of getproviders.Hash. The new type definitions are in
internal/getproviders/hash.go.
2020-09-24 14:01:54 -07:00
Pam Selle 5b549224ae Refactor to call ContainsMarked less and use len() instead 2020-09-24 16:42:03 -04:00
Alisdair McDiarmid 60c469b4a5
Merge pull request #26345 from hashicorp/alisdair/taint-should-respect-required-version
command: Taint should respect required_version
2020-09-24 15:52:23 -04:00
Pam Selle 3c9fad0b0e Move plan action check into the sensitivity warning method 2020-09-24 13:49:34 -04:00
Pam Selle 531728f6e9 Sensitive diffs for primitive types
When showing primitive type diffs, hide possibly
sensitive values
2020-09-24 13:27:15 -04:00
Pam Selle 20921dbfb8 Add warning about sensitivity change
This commit adds a warning before displaying
a sensitive diff, and always obfuscates the old value (even
if it was not previously marked as sensitive)
2020-09-24 12:57:40 -04:00
Pam Selle 0a02e7040f
Store sensitive attribute paths in state (#26338)
* Add creation test and simplify in-place test

* Add deletion test

* Start adding marking from state

Start storing paths that should be marked
when pulled out of state. Implements deep
copy for attr paths. This commit also includes some
comment noise from investigations, and fixing the diff test

* Fix apply stripping marks

* Expand diff tests

* Basic apply test

* Update comments on equality checks to clarify current understanding

* Add JSON serialization for sensitive paths

We need to serialize a slice of cty.Path values to be used to re-mark
the sensitive values of a resource instance when loading the state file.
Paths consist of a list of steps, each of which may be either getting an
attribute value by name, or indexing into a collection by string or
number.

To serialize these without building a complex parser for a compact
string form, we render a nested array of small objects, like so:

[
  [
    { type: "get_attr", value: "foo" },
    { type: "index", value: { "type": "number", "value": 2 } }
  ]
]

The above example is equivalent to a path `foo[2]`.

* Format diffs with map types

Comparisons need unmarked values to operate on,
so create unmarked values for those operations. Additionally,
change diff to cover map types

* Remove debugging printing

* Fix bug with marking non-sensitive values

When pulling a sensitive value from state,
we were previously using those marks to remark
the planned new value, but that new value
might *not* be sensitive, so let's not do that

* Fix apply test

Apply was not passing the second state
through to the third pass at apply

* Consistency in checking for length of paths vs inspecting into value

* In apply, don't mark with before paths

* AttrPaths test coverage for DeepCopy

* Revert format changes

Reverts format changes in format/diff for this
branch so those changes can be discussed on a separate PR

* Refactor name of AttrPaths to AttrSensitivePaths

* Rename AttributePaths/attributePaths for naming consistency

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2020-09-24 12:40:17 -04:00
James Bardin 2e5366ac69 remove -mod=vendor from command tests 2020-09-23 17:56:19 -04:00
Alisdair McDiarmid 14a233b019 command: Taint should respect required_version
Despite not requiring the configuration for any other reason, the taint
subcommand should not execute if the required_version constraints cannot
be met. Doing so can result in an undesirable state file upgrade.
2020-09-22 17:33:09 -04:00
Alisdair McDiarmid 18f9ea53b9 command: Providers schema shows required_providers
The providers schema command is using the Config.ProviderTypes method,
which had not been kept up to date with the changes to provider
requirements detection made in Config.ProviderRequirements. This
resulted in any currently-unused providers being omitted from the
output.

This commit changes the ProviderTypes method to use the same underlying
logic as ProviderRequirements, which ensures that `required_providers`
blocks are taken into account.

Includes an integration test case to verify that this fixes the provider
schemas command bug.
2020-09-22 10:28:32 -04:00
James Bardin ab06f0c9f8 we can roll back the e2e tests
the data sources no longer show up in the tests
2020-09-22 09:55:19 -04:00
James Bardin c3182bd589 mock provider needs to return a valid response 2020-09-21 16:17:45 -04:00
James Bardin 3b3ff98356 Revert "fix show -json tests"
This reverts commit e54949f2e1.

Changes incorrectly applied to the planned state tests
2020-09-21 16:17:45 -04:00
James Bardin 86dd8938c9 data sources now show up in the initial plan 2020-09-17 09:55:00 -04:00
James Bardin e54949f2e1 fix show -json tests
The prior state recorded in the plans did not match the actual prior
state. Make the plans and state match depending on whether there was
existing state or not.
2020-09-17 09:55:00 -04:00
James Bardin ad5899d8bb ReadResource is called during plan but not destroy 2020-09-17 09:55:00 -04:00
Joe Ceresini 04b39372b0 Add scopes support to password grant type 2020-09-15 19:32:41 -04:00
Joe Ceresini 2ffa84a4de Add command/login test for scopes 2020-09-15 19:30:48 -04:00
Joe Ceresini 2acb7fac9f Include scopes in login URL 2020-09-14 12:21:20 -04:00
Alisdair McDiarmid 8b2b569d6e repl: Improved value renderer for console outputs
Use a slightly modified value renderer from terraform-provider-testing
to display values in the console REPL, as well as outputs from the apply
and outputs subcommands.

Derived from code in this repository, MIT licensed:

https://github.com/apparentlymart/terraform-provider-testing

Note that this is technically a breaking change for the console
subcommand, which would previously error if the user attempted to render
an unknown value (such as an unset variable). This was marked as an
unintentional side effect, with the goal being the new behaviour of
rendering "(unknown)", which is why I changed the behaviour in this
commit.
2020-09-14 09:47:12 -04:00
Pam Selle 6a126df0c6
Merge pull request #26183 from hashicorp/pselle/sensitive-values
Add sensitive attribute to variables
2020-09-11 11:24:18 -04:00
Pam Selle 8d8389da74 Add diff test with a sensitive change
Adds a diff test for a changed value,
and modifies the diff file to cover variable
diffs on sensitive values
2020-09-10 16:45:31 -04:00
Alisdair McDiarmid 59662c01af vendor: Upgrade mitchellh/cli
Update tests to match the fix in mitchellh/cli#71, which aligns MockUi
with BasicUi and allows newlines in user input.

We are not using the new ErrorWriter, added in mitchellh/cli#81, as it
does not appear to interact correctly with panicwrap. All error output
from CLI parsing will continue to appear on stdout, not stderr.
2020-09-10 15:23:02 -04:00
Pam Selle e4e16ccbd3 Rebase fix 2020-09-10 11:06:40 -04:00
Pam Selle 61c78fd3b9 Add case to compactValueStr not to expose sensitive vals in diagnostics 2020-09-10 11:04:17 -04:00
Pam Selle bc55b6a28b Use UnmarkDeepWithPaths and MarkWithPaths
Updates existing code to use the new Value
methods for unmarking/marking and removes
panics/workarounds in cty marshall methods
2020-09-10 11:04:17 -04:00
Pam Selle 896d277a69 If the path is empty, we should not be marking the path 2020-09-10 11:04:17 -04:00
Pam Selle 84d118e18f Track sensitivity through evaluation
Mark sensitivity on a value. However, when the value is encoded to send to the
provider to produce a changeset we must remove the marks, so unmark the value
and remark it with the saved path afterwards
2020-09-10 11:04:17 -04:00
Alisdair McDiarmid 09d8355f43 command: Add experimental concise diff renderer
When rendering a diff between current state and projected state, we only
show resources and outputs which have changes. However, we show a full
structural diff for these values, which includes all attributes and
blocks for a changed resource or output. The result can be a very long
diff, which makes it difficult to verify what the changed fields are.

This commit adds an experimental concise diff renderer, which suppresses
most unchanged fields, only displaying the most relevant changes and
some identifying context. This means:

- Always show all identifying attributes, initially defined as `id`,
  `name`, and `tags`, even if unchanged;
- Only show changed, added, or removed primitive values: `string`,
  `number`, or `bool`;
- Only show added or removed elements in unordered collections and
  structural types: `map`, `set`, and `object`;
- Show added or removed elements with any surrounding unchanged elements
  for sequence types: `list` and `tuple`;
- Only show added or removed nested blocks, or blocks with changed
  attributes.

If any attributes, collection elements, or blocks are hidden, a count
is kept and displayed at the end of the parent scope. This ensures that
it is clear that the diff is only displaying a subset of the resource.

The experiment is currently enabled by default, but can be disabled by
setting the TF_X_CONCISE_DIFF environment variable to 0.
2020-09-10 10:35:55 -04:00
Paul Tyng f3ff843ffd Remove unused env var TF_SKIP_PROVIDER_VERIFY 2020-09-10 09:03:56 -04:00
Alisdair McDiarmid 1c8a7f65d0
Merge pull request #26137 from hashicorp/alisdair/output-changes-tests
command: Add tests for format.OutputChanges
2020-09-08 09:45:05 -04:00
Martin Atkins efe78b2910 main: new global option -chdir
This new option is intended to address the previous inconsistencies where
some older subcommands supported partially changing the target directory
(where Terraform would use the new directory inconsistently) where newer
commands did not support that override at all.

Instead, now Terraform will accept a -chdir command at the start of the
command line (before the subcommand) and will interpret it as a request
to direct all actions that would normally be taken in the current working
directory into the target directory instead. This is similar to options
offered by some other similar tools, such as the -C option in "make".

The new option is only accepted at the start of the command line (before
the subcommand) as a way to reflect that it is a global command (not
specific to a particular subcommand) and that it takes effect _before_
executing the subcommand. This also means it'll be forced to appear before
any other command-specific arguments that take file paths, which hopefully
communicates that those other arguments are interpreted relative to the
overridden path.

As a measure of pragmatism for existing uses, the path.cwd object in
the Terraform language will continue to return the _original_ working
directory (ignoring -chdir), in case that is important in some exceptional
workflows. The path.root object gives the root module directory, which
will always match the overriden working directory unless the user
simultaneously uses one of the legacy directory override arguments, which
is not a pattern we intend to support in the long run.

As a first step down the deprecation path, this commit adjusts the
documentation to de-emphasize the inconsistent old command line arguments,
including specific guidance on what to use instead for the main three
workflow commands, but all of those options remain supported in the same
way as they were before. In a later commit we'll make those arguments
produce a visible deprecation warning in Terraform's output, and then
in an even later commit we'll remove them entirely so that -chdir is the
single supported way to run Terraform from a directory other than the
one containing the root module configuration.
2020-09-04 15:31:08 -07:00
Alisdair McDiarmid ba9baac36d command: Add tests for format.OutputChanges
Most of the functionality for rendering output changes is covered by the
tests for ResourceChanges, as they both share the same diff renderer.
This commit adds a few tests to cover some of the output specific code.
2020-09-04 16:22:23 -04:00
Graham Hargreaves e9394dfb38
command/clistate: Return an error on unlock failure (#25729)
* Return an error on unlock failure

When the lock can't be released return the err even if there is no previous error with the current action. This allows faster failure in CI/CD systems. Without this failure to remove the lock would result in the failure happening on a subsequent plan or apply which slows down the feedback loop in automated systems.

* Update command/clistate/state.go

Accept review suggestion

Co-authored-by: ZymoticB <ZymoticB@users.noreply.github.com>

* add test

Co-authored-by: ZymoticB <ZymoticB@users.noreply.github.com>
Co-authored-by: Kristin Laemmert <mildwonkey@users.noreply.github.com>
2020-09-03 16:05:16 -04:00
Alisdair McDiarmid 9f824c53a5 command: Better in-house provider install errors
When init attempts to install a legacy provider required by state and
fails, but another provider with the same type is successfully
installed, this almost definitely means that the user is migrating an
in-house provider. The solution here is to use the `terraform state
replace-provider` subcommand.

This commit makes that next step clearer, by detecting this specific
case, and displaying a list of commands to fix the existing state
provider references.
2020-09-01 14:02:19 -04:00
Alisdair McDiarmid 3547f9e368 format: Don't wrap space-prefixed diag details
Diagnostic detail lines sometimes contain lines which include commands
suggested for the user to execute. By convention, these start with
leading whitespace to indicate that they are not prose.

This commit changes the diagnostic formatter to wrap each line of the
detail separately, and skips word wrapping for lines prefixed with
space. This prevents ugly and confusing wrapping of long command lines.
2020-09-01 14:02:19 -04:00
Alisdair McDiarmid 6d228cc560
Merge pull request #26036 from hashicorp/alisdair/output-empty-should-be-warning
command: Warn instead of error for empty output
2020-09-01 10:23:51 -04:00
Alisdair McDiarmid 89e8d0868a
Merge pull request #26061 from hashicorp/alisdair/013upgrade-registry-provider-redirect
command: Add redirect support to 0.13upgrade
2020-09-01 08:59:35 -04:00
Kristin Laemmert 196c183dda
terraform: remove state from `validate` graph walk (#26063)
This pull reverts a recent change to backend/local which created two context, one with and one without state. Instead I have removed the state entirely from the validate graph (by explicitly passing a states.NewState() to the validate graph builder).

This changed caused a test failure, which (ty so much for the help) @jbardin discovered was inaccurate all along: the test's call to `Validate()` was actually what was removing the output from state. The new expected test output matches terraform's actual behavior on the command line: if you use -target to destroy a resource, an output that references only that resource is *not* removed from state even though that test would lead you to believe it did.

This includes two tests to cover the expected behavior:

TestPlan_varsUnset has been updated so it will panic if it gets more than one request to input a variable
TestPlan_providerArgumentUnset covers #26035

Fixes #26035, #26027
2020-08-31 15:45:39 -04:00
Alisdair McDiarmid fc7e467d19 command: Add redirect support to 0.13upgrade
If a provider changes namespace in the registry, we can detect this when
running the 0.13upgrade command. As long as there is a version matching
the user's constraints, we now use the provider's new source address.
Otherwise, warn the user that the provider has moved and a version
upgrade is necessary to move to it.
2020-08-31 14:53:35 -04:00
Pam Selle f2d213c461
Merge pull request #25657 from pdecat/typo_hierarchical
Typo: heirarchical => hierarchical
2020-08-28 12:37:49 -04:00
Alisdair McDiarmid 2323e9370a command: Warn instead of error for empty output
When the output subcommand is called with no arguments, and there are no
outputs to show, we previously rendered an error message but returned a
non-error status code. This is confusing.

This commit changes the text UI to use a warning diagnostic, which makes
it clearer that this is a non-error situation. We do not change the exit
code or the text of the warning, so hopefully this is not considered a
breaking change.
2020-08-28 11:05:46 -04:00
Alisdair McDiarmid 45437721c9 command: Add test for backend config validation 2020-08-26 11:37:11 -04:00
Alisdair McDiarmid f028b0a2bf command: Fix backend config schema validation
When applying a backend config override file, we must not check for the
presence of all required fields, as the override can be a partial
configuration. It is only valid to check for required fields after all
overrides have been merged, which init already does.
2020-08-26 10:50:47 -04:00
Alisdair McDiarmid c5d9935c0e
Merge pull request #25960 from hashicorp/alisdair/backend-config-override-fix
command: Fix backend config override validation
2020-08-24 10:14:16 -04:00
Alisdair McDiarmid 35125717b2
Merge pull request #25847 from zachwhaley/fix-tf-cli-args-no-color
Fix error when multiple -no-color arguments are used
2020-08-24 10:13:59 -04:00
Alisdair McDiarmid 19efd7bc3b
Merge pull request #25262 from hashicorp/alisdair/always-validate-workspace-name
command: Always validate workspace name
2020-08-24 10:11:13 -04:00
Alisdair McDiarmid 677aabc767 command: Fix backend config override validation
When loading a backend config override file, init was doing two things
wrong:

- First, if the file failed to parse, we accidentally didn't return,
  which caused a panic due to the parsed body being nil;
- Secondly, we were overzealous with the validation of the file,
  allowing only attributes. While most backend configs are attributes
  only, the enhanced remote backend body also contains a `workspaces`
  block, which we need to support here.

This commit fixes the first bug with an early return and adds test cases
for missing file and intentionally-blank filename (to clear the config).

We also add a schema validation for the backend block, based on the
backend schema itself. This requires constructing an HCL body schema so
that we can call `Content` and check for diagnostic errors.

The result is more useful errors when an invalid backend config override
file is used, while also supporting the enhanced remote backend config
fully.

Does not include tests specific to the remote backend, because the
mocking involved to allow the backend to fully initialize is too
involved to be worth it.
2020-08-21 16:21:13 -04:00
Cameron Stitt 54e32652f7
Ensure depends_on is in module calls for config 2020-08-20 07:49:03 +10:00
Alisdair McDiarmid 30c7dfca62
Merge pull request #25898 from hashicorp/alisdair/fix-required-version-diags
terraform: Fix required version constraint diags
2020-08-19 11:26:03 -04:00
Alisdair McDiarmid 3114e2ad7c
Merge pull request #25890 from hashicorp/import-our-nemesis
terraform: Eval module call arguments for import
2020-08-19 11:25:38 -04:00
kmoe 6486b857d6
Merge pull request #25811 from hashicorp/terraform-version-order
terraform version: fix output order
2020-08-19 13:54:18 +01:00
Alisdair McDiarmid c98f352dc8 terraform: Fix required version constraint diags
If a module has multiple terraform.required_version constraints, any
failures would point at the last constraint in the error diagnostics. If
an earlier constraint was the actual problem, this leads to confusing
errors like this:

    Error: Unsupported Terraform Core version

      on main.tf line 6, in terraform:
       6:   required_version = ">= 0.13.0"

    This configuration does not support Terraform version 0.13.0.

The error was due to storing the declaration range of the constraint as
a pointer to the contents of a loop variable, which was later
overwritten in later iterations of the loop.  Instead we now use HCL's
handy Ptr() method to create a direct pointer to the range struct.
2020-08-18 09:35:32 -04:00
Alisdair McDiarmid d8e9964363 terraform: Eval module call arguments for import
Include the import walk in the list of operations for which we create an
EvalModuleCallArgument node. This causes module call arguments to be
evaluated even if the module variables have defaults, ensuring that
invalid default values (such as the common "{}" for variables thought of
as maps) do not cause failures specific to import.

This fixes a bug where a child module evaluates an input variable in its
locals block, assuming that it is a nested object structure. The bug
report includes a default value of "{}", which is overridden by a root
variable value. Without the eval node added in this commit, the default
value is used and the local evaluation errors.
2020-08-17 17:14:12 -04:00
Zachary Whaley af8d5a69d8 Fix error when TF_CLI_ARGS adds -no-color argument
Fixes #25845
2020-08-13 13:41:34 -05:00
Katy Moe e59553ef02
test outdated version output 2020-08-12 21:22:39 +01:00
James Bardin 1c09df1a66
Merge pull request #25779 from hashicorp/jbardin/remove-state-attrs
Remove resource state attributes that are no longer in the schema
2020-08-12 10:49:44 -04:00
Katy Moe 25337b7217
fix outdated message output order 2020-08-11 23:27:29 +01:00
Alisdair McDiarmid 96d2265ddb command: Add test for invalid selected workspace
If somehow an invalid workspace has been selected, the Meta.Workspace
method should not return an error, to ensure that we don't break any
existing workflows with invalid workspace names.
2020-08-11 12:33:12 -04:00
Alisdair McDiarmid 8252920e9a command: Allow workspace delete with invalid name
We are validating the workspace name for all workspace commands. Due to
a bug with the TF_WORKSPACE environment variable, it has been possible
to accidentally create a workspace with an invalid name.

This commit removes the valid workspace name check for workspace delete
to allow users to clean up any invalid workspaces.
2020-08-11 12:33:12 -04:00
Alisdair McDiarmid b239570abb command: Always validate workspace name
The workspace name can be overridden by setting a TF_WORKSPACE
environment variable. If this is done, we should still validate the
resulting workspace name; otherwise, we could end up with an invalid and
unselectable workspace.

This change updates the Meta.Workspace function to return an error, and
handles that error wherever necessary.
2020-08-11 12:33:12 -04:00
Alisdair McDiarmid ef65f552c8
Merge pull request #25647 from hashicorp/alisdair/remove-stubbed-out-apply-diff-code
command: Remove stubbed-out apply diff output code
2020-08-11 12:18:49 -04:00
Alisdair McDiarmid 17e1c9dd05 command: Fix state mv for only resource in module
When moving a resource block with multiple instances to a new address
within the same module, we need to ensure that the target module is
present as late as possible. Otherwise, deleting the resource from the
original address triggers pruning, and the module is removed just before
we try to add the resource to it, which causes a crash.

Includes regression test which panics without this code change.
2020-08-11 11:54:35 -04:00
Kristin Laemmert 6621501ae3
state: remove deprecated state package (#25490)
Most of the state package has been deprecated by the states package.
This PR replaces all the references to the old state package that
can be done simply - the low-hanging fruit.

* states: move state.Locker to statemgr

The state.Locker interface was a wrapper around a statemgr.Full, so
moving this was relatively straightforward.

* command: remove unnecessary use of state package for writing local terraform state files

* move state.LocalState into terraform package

state.LocalState is responsible for managing terraform.States, so it
made sense (to me) to move it into the terraform package.

* slight change of heart: move state.LocalState into clistate instead of
terraform
2020-08-11 11:43:01 -04:00
Kristin Laemmert 86e9ba3d65
* backend/local: push responsibility for unlocking state into individual operations
* unlock the state if Context() has an error, exactly as backend/remote does today
* terraform console and terraform import will exit before unlocking state in case of error in Context()
* responsibility for unlocking state in the local backend is pushed down the stack, out of backend.go and into each individual state operation
* add tests confirming that state is not locked after apply and plan

* backend/local: add checks that the state is unlocked after operations

This adds tests to plan, apply and refresh which validate that the state
is unlocked after all operations, regardless of exit status. I've also
added specific tests that force Context() to fail during each operation
to verify that locking behavior specifically.
2020-08-11 11:23:42 -04:00
James Bardin 99cd3ab223 fix command tests
A number of tests had invalid configs or providers, but were never
properly validated
2020-08-07 14:13:57 -04:00
FGtatsuro b6cf1c3673 Use valid heredoc begin symbol in diff output. 2020-08-03 00:33:35 +09:00
Patrick Decat 062865735f Typo: heirarchical => hierarchical 2020-07-23 15:09:22 +02:00
Alisdair McDiarmid ee19b0d6c5 command: Remove stubbed-out apply diff output code
This code was made to do nothing pre-0.12, and we have no plans to
reintroduce a diff in the apply output, so it seems reasonable to now
remove it altogether.
2020-07-22 14:19:00 -04:00
Kristin Laemmert f8e3456867
command/show: fix bug displaying provider config in json output of tf plan (#25577)
A lingering FIXME caused missing configuration from provider config
blocks in the json output of terraform plan. This fixes the regression
and adds a test. For the sake of testing, I added an optional attribute
to the show test provider, which resulted in the providers schema test
getting an update - not a bad addition, but we can always add a
test-specific provider schema as needed.
2020-07-14 15:28:31 -04:00
Alisdair McDiarmid 53e587e1a6
Merge pull request #25504 from hashicorp/alisdair/post-install-cache-validate
Add post-install provider cache validation and error reporting
2020-07-09 14:55:17 -04:00
Martin Atkins 7909dd318d command/init: Specialized error message for host that isn't registry
If the user specifies a host that isn't a provider registry in a provider
source address then we'll print out some specialized error messages for
different variants of that situation.

In particular, this includes a special case for when the error is on the
hostname "github.com", in anticipation of folks incorrectly attempting to
use GitHub repository URLs (or Go-style module paths that happen to be
on GitHub) to specify providers, so we can give a more specific hint about
that.

This is just a different presentation of an existing error case that we
are already covering in the installer tests, so there are no new tests
here. We could in principle have a test covering the exact text of these
error messages, but we don't have much precedent for command package tests
covering that level of cosmetic detail.
2020-07-08 10:18:55 -07:00
Martin Atkins 80ab867e57 command/init: Remove special 0.12upgrade heuristic
For Terraform v0.12 we introduced a special loading mode where we would
use the 0.11-syntax-compatible "earlyconfig" package as a heuristic to
identify situations where it was likely that the user was trying to use
0.11-only syntax that the upgrade tool might help with.

However, as the language has moved on that is no longer a suitable
heuristic in Terraform 0.13 and later: other new additions to the
language can cause the main loader to disagree with earlyconfig, which
would lead us to give poor advice about how to respond.

Instead, we'll now return the same generic "there are errors" message in
all syntax error cases. We have an extra message for errors in this
case (as compared to other commands) because "terraform init" is usually
the first command a new user interacts with and so this message gives some
extra explanation about what "terraform init" will do with the
configuration once it's valid.

This also includes a reset control character in the output of the message
as part of our ongoing mission to stop Terraform printing out whole
paragraphs of colored text, which can often be hard to read for various
reasons.
2020-07-08 10:18:55 -07:00
Alisdair McDiarmid 87d1fb4006 command/init: Display provider validation errors
After installing providers, we validate the presence of an executable
file, and generate a selected versions lockfile. If this process fails,
notify the user. One possible cause for this is an invalid provider
package with a missing or misnamed executable file.
2020-07-07 15:20:20 -04:00
Alisdair McDiarmid a18b531b14 getproviders: FakeInstallablePackageMeta filename
Add an optional execFilename argument to the test helper function
FakeInstallablePackageMeta, which allows the creation of invalid
packages.
2020-07-07 15:18:30 -04:00
Alisdair McDiarmid 8e87ccb689 providercache: Lazily detect executable file
Instead of searching the installed provider package directory for a
binary as we install it, we can lazily detect the executable as it is
required. Doing so allows us to separately report an invalid unpacked
package, giving the user more actionable error messages.
2020-07-07 15:18:27 -04:00
Kristin Laemmert 604e65bb62 Revert "backend/local: release lock if there is an error in Context() (#25427)"
This reverts commit 1ba0d615e7.
2020-06-30 14:12:32 -04:00
Kristin Laemmert 1ba0d615e7
backend/local: release lock if there is an error in Context() (#25427)
* command/console: return in case of errors before trying to unlock remote
state

The remote backend `Context` would exit without an active lock if there
was an error, while the local backend `Context` exited *with* a lock. This
caused a problem in `terraform console`, which would call unlock
regardless of error status.

This commit makes the local and remote backend consistently unlock the
state incase of error, and updates terraform console to check for errors
before trying to unlock the state.

* adding tests for remote and local backends
2020-06-29 14:57:42 -04:00
Kristin Laemmert df244b87c2
command/init: return an error with invalid -backend-config files (#25411)
* command/init: return an error with invalid -backend-config files

The -backend-config flag expects a set of key-value pairs or a file
containing key-value pairs. If the file instead contains a full backend
configuration block, it was silently ignored. This commit adds a check
for blocks in the file and returns an error if they are encountered.

Fixes #24845

* emphasize backend configuration file in docs
2020-06-26 12:49:31 -04:00
Alisdair McDiarmid b4cc77b6cd
Merge pull request #25379 from hashicorp/alisdair/yes
command/login: Require "yes" to confirm
2020-06-25 13:00:02 -04:00
Alisdair McDiarmid f6f707c669
Merge pull request #25381 from hashicorp/alisdair/013upgrade-preserves-more-comments
command: Fix 0.13upgrade to preserve more comments
2020-06-25 12:56:34 -04:00
Alisdair McDiarmid 779fe37a1c command/login: Require "yes" to confirm
This is for consistency with other commands which use prompts, all of
which require "yes" rather than "y" to confirm.

We also migrate the login command to use UIInput, which now supports
securely asking for passwords or secrets via the speakeasy library.
2020-06-25 11:46:51 -04:00
Kristin Laemmert 47e657c611
internal/getproviders: decode and return any registry warnings (#25337)
* internal/getproviders: decode and return any registry warnings

The public registry may include a list of warnings in the "versions"
response for any given provider. This PR adds support for warnings from
the registry and an installer event to return those warnings to the
user.
2020-06-25 10:49:48 -04:00
James Bardin 98ff2065bc
Merge pull request #25377 from hashicorp/jbardin/remote-state
Correct remote state return value
2020-06-25 10:43:22 -04:00
Alisdair McDiarmid b6739829e7 command: Fix 0.13upgrade to preserve more comments
Previously, any comments inside the required provider configuration for
a given provider would be wiped out upon rerunning the 0.13upgrade
command. This commit attempts to preserve those comments if the existing
entry is semantically equivalent to the entry we are about to write.
2020-06-24 15:54:46 -04:00
James Bardin 731b19ab46 e2e test for remote state read 2020-06-24 14:09:59 -04:00
João G. Packer fd47260fdc
command: adjust exit code of state rm (#22300)
* command: adjust exit code of state rm and state mv

Commands `state rm` and `state mv` will now exit with code 1 when the
target resource is not found in the current state.
This is consistent with `terraform state show non_existent_resource`.

Fixes #17800
2020-06-24 14:03:23 -04:00
Alisdair McDiarmid f714a9da79
Merge pull request #25352 from hashicorp/alisdair/fix-import-allow-missing-config
command/import: Fix allow-missing-config option
2020-06-24 13:37:13 -04:00
Alisdair McDiarmid 550d75f5ce
Merge pull request #25300 from hashicorp/alisdair/fix-init-flags-silent-exit
command: Fix init flags silent exit bug
2020-06-24 10:33:09 -04:00
Alisdair McDiarmid 9ab9ef6291 command/import: Fix allow-missing-config option
We previously intentionally removed support for the allow-missing-config
option to terraform import, requiring that all imported resources have
matching config. See #24412.

However, the option was not removed from the import command, and it is
widely used. This commit reintroduces support for importing with a
missing configuration by falling back to implying the provider FQN based
on the resource type.
2020-06-23 14:20:50 -04:00
Alisdair McDiarmid 820ed48813 command: Fix init flags silent exit bug
When using `-flag=value` with Powershell, unquoted values are broken
into separate arguments. This means that the following command:

  terraform init -backend-config=./backend.conf

is interpreted by Terraform as:

  terraform init -backend-config= ./backend.conf

This results in an empty backend-config setting (which is semantically
valid!) followed by a custom configuration path (pointing at a file).

Due to a bug where we could exit without printing diagnostics, this
would result in a silent failure that was very difficult to diagnose.
2020-06-18 17:58:45 -04:00
Alisdair McDiarmid 0ed04d05aa command: Fix command fixture modify-in-place bugs
Some of the tests in the command package were running directly on the
fixture directories, and modifying or locking files within them. This
could cause state to leak between tests.

This commit cleans up all such cases that I could find.
2020-06-18 17:09:20 -04:00
Alisdair McDiarmid 163c943e9b command: Fix bug with -v/-version/--version flags
We globally support a -v/-version/--version flag, which triggers the
version subcommand. The recent introduction of JSON output support meant
we started parsing the flags for the first time, but we didn't add flags
for these global version arguments.

This commit adds those flags (but doesn't check them, since they have no
effect on the version command itself). Also adds usage information for
terraform version.
2020-06-17 15:24:15 -04:00
Katy Moe 6b3d0ee64b add test for terraform version 2020-06-16 14:47:57 -04:00
Katy Moe dbe139e616 add test for terraform version -json 2020-06-16 14:47:57 -04:00
Katy Moe d8e22f1804 add terraform version -json 2020-06-16 14:47:57 -04:00
Alisdair McDiarmid d2b215f4d5 command: Fix test cleanup
This test runs `init`, which causes some dangling `.terraform` state
after running tests.
2020-06-16 14:11:08 -04:00
James Bardin b35a56cf6e command tests looked for providers in the graphs
The quotes in provider names needs to be escaped, so these tests were
looking for the wrong output to begin with.
2020-06-15 14:40:59 -04:00
Alisdair McDiarmid 7d4cab126f
Merge pull request #25233 from hashicorp/alisdair/remove-implicit-init-then-apply
command/apply: Remove implicit init call
2020-06-12 15:52:06 -04:00
Kristin Laemmert b611bd7209 reproduction test 2020-06-12 15:39:55 -04:00
Alisdair McDiarmid 0d614a8d01 command/apply: Remove implicit init call
The positional argument passed to apply was once used to specify a
source for a Terraform module to fetch and initialize (#337). This
functionality was removed from the init command later (#15032) but not
completely removed from apply.

This code was non-functional but largely not harmful, except for a very
specific case: when passing an absolute path to a plan file as the
positional argument on Windows, the getter.Detect code would incorrectly
interpret the path as a URL. This caused init to fail and the apply
command would exit with code 1 but without diagnostics.

This commit removes this codepath, which fixes this bug, and should
otherwise have no effect on the supported behaviour of apply.
2020-06-12 12:39:15 -04:00
Alisdair McDiarmid 08b735984a
Merge pull request #25191 from hashicorp/alisdair/better-provider-upgrade-hints-on-init
command/init: Improve diags for legacy providers
2020-06-12 12:31:33 -04:00
Martin Atkins 17feb2abfc vendor: go get github.com/apparentlymart/go-versions@v1.0.0
This new version permits omitting the space between the operator and the
boundary in a ruby-style version constraint, like ">1.0.0" instead of
"> 1.0.0".
2020-06-12 08:45:14 -07:00
Alisdair McDiarmid 9263b28e99 command/init: Improve diags for legacy providers
When initializing a configuration which refers to re-namespaced legacy
providers, we attempt to detect this and display a diagnostic message.
Previously this message would direct the user to run the 0.13upgrade
command, but without specifying in which directories.

This commit detects which modules are using the providers in question,
and for local modules displays a list of upgrade commands which specify
the source directories of these modules.

For remote modules, we display a separate list noting that they need to
be upgraded elsewhere, providing both the local module call name and the
module source address.
2020-06-12 09:57:01 -04:00
Kristin Laemmert 5450e8515d
command/013upgrade: detect builtin terraform provider (#25215)
* command/013upgrade: detect builtin terraform provider
2020-06-11 14:10:47 -04:00
Alisdair McDiarmid 1c1e4a4de0 command/providers: Show provider requirements tree
Providers can be required from multiple sources. The previous
implementation of the providers sub-command displayed only a flat list
of provider requirements, which made it difficult to see which modules
required each provider.

This commit reintroduces the tree display of provider requirements, and
adds a separate output block for providers required by existing state.
2020-06-09 14:21:53 -04:00
Masayuki Morita ee913a1c87
command/0.13upgrade: make confirmation more user-friendly (#25142)
I feel the current confirmation prompt for 0.13upgrade command is
ambiguous what is expected. Actually, when I used it for the first time,
I cancelled it by typing `y` instead of `yes`.

I believe it would be great if the 0.13upgrade command tell us the
expected value for confirmation like 0.12upgrade.
2020-06-08 13:36:19 -04:00
Alisdair McDiarmid c87c0a9d2a command/format: Fix empty overlap diagnostics
Diagnostics where the highlight range has an empty overlap with a line
would skip lines of the output. This is because if two ranges abut each
other, they can be considered to overlap, but that overlap is empty.
This results in an edge case in the diagnostic printer which causes the
line not to be printed.
2020-06-05 16:17:27 -04:00
Shunsuke Miyoshi dc3ce1bbf7
command/refresh: fix state file path (#22885)
* fix default state file path for refresh command
2020-06-04 10:24:53 -04:00
Alisdair McDiarmid fbb966b97b command: Fix 0.13upgrade usage 2020-06-04 08:51:24 -04:00
Masayuki Morita f266956a5d Add `-yes` flag to 0.13upgrade help message
It seems to be implemented but not shown in help message.

FYI: A help message for 0.12upgrade command on the 0.12 branch.
https://github.com/hashicorp/terraform/blob/v0.12.26/command/012_config_upgrade.go#L235-L243
2020-06-04 11:11:46 +09:00
Martin Atkins 49e2e00231 command: terraform providers mirror
This new command is intended to make it easy to create or update a mirror
directory containing suitable providers for the current configuration,
producing a layout that is appropriate both for a filesystem mirror or,
if copied into the document root of an HTTP server, a network mirror.

This initial version is not customizable aside from being able to select
multiple platforms to install packages for.

Future iterations of this could include commands to turn the JSON index
generation on and off, or to instruct it to produce the unpacked directory
layout instead of the packed directory layout as it currently does. Both
of those options would make the generated directory unsuitable to be
a network mirror, but it would still work as a filesystem mirror.

In the long run this will hopefully form part of a replacement workflow to
terraform-bundle as a way to put copies of providers somewhere so we don't
need to re-download them every time, but some other changes will be needed
outside of just this command before that'd be true, such as adding support
for network and/or filesystem mirrors in Terraform Enterprise.
2020-06-01 14:49:43 -07:00
Alisdair McDiarmid 5e2b11657e command: Fix 0.12upgrade stub 2020-06-01 16:12:30 -04:00
Kristin Laemmert 020084f6d0 update e2etests for windows compatibility 2020-05-29 11:57:50 -04:00
Alisdair McDiarmid 83482770c4
Merge pull request #25055 from hashicorp/alisdair/detected-legacy-provider-on-init
command/init: Better diagnostics for provider 404s
2020-05-29 11:57:18 -04:00
Martin Atkins 31a4b44d2e backend/local: treat output changes as side-effects to be applied
This is a baby-step towards an intended future where all Terraform actions
which have side-effects in either remote objects or the Terraform state
can go through the plan+apply workflow.

This initial change is focused only on allowing plan+apply for changes to
root module output values, so that these can be written into a new state
snapshot (for consumption by terraform_remote_state elsewhere) without
having to go outside of the primary workflow by running
"terraform refresh".

This is also better than "terraform refresh" because it gives an
opportunity to review the proposed changes before applying them, as we're
accustomed to with resource changes.

The downside here is that Terraform Core was not designed to produce
accurate changesets for root module outputs. Although we added a place for
it in the plan model in Terraform 0.12, Terraform Core currently produces
inaccurate changesets there which don't properly track the prior values.

We're planning to rework Terraform Core's evaluation approach in a
forthcoming release so it would itself be able to distinguish between the
prior state and the planned new state to produce an accurate changeset,
but this commit introduces a temporary stop-gap solution of implementing
the logic up in the local backend code, where we can freeze a snapshot of
the prior state before we take any other actions and then use that to
produce an accurate output changeset to decide whether the plan has
externally-visible side-effects and render any changes to output values.

This temporary approach should be replaced by a more appropriately-placed
solution in Terraform Core in a release, which should then allow further
behaviors in similar vein, such as user-visible drift detection for
resource instances.
2020-05-29 07:36:40 -07:00
Alisdair McDiarmid ca40107066 command/init: Better diagnostics for provider 404s
Fetching a default namespace provider from the public registry can
result in 404 Not Found error. This might be caused by a previously-
default provider moving to a new namespace, which means that the
configuration needs to be upgraded to use an explicit provider source.

This commit adds a more detailed diagnostic for this situation,
suggesting that the intended provider might be in a new namespace. The
recommended course of action is to run the 0.13upgrade command to
generate the correct required_providers configuration.
2020-05-28 09:24:32 -04:00
Alisdair McDiarmid ef28671b34
Merge pull request #24932 from hashicorp/signing-language
Modify language for reporting signing state
2020-05-28 09:09:34 -04:00
Paddy 5127f1ef8b
command: Unmanaged providers
This adds supports for "unmanaged" providers, or providers with process
lifecycles not controlled by Terraform. These providers are assumed to
be started before Terraform is launched, and are assumed to shut
themselves down after Terraform has finished running.

To do this, we must update the go-plugin dependency to v1.3.0, which
added support for the "test mode" plugin serving that powers all this.

As a side-effect of not needing to manage the process lifecycle anymore,
Terraform also no longer needs to worry about the provider's binary, as
it won't be used for anything anymore. Because of this, we can disable
the init behavior that concerns itself with downloading that provider's
binary, checking its version, and otherwise managing the binary.

This is all managed on a per-provider basis, so managed providers that
Terraform downloads, starts, and stops can be used in the same commands
as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable
is added, and is a JSON encoding of the provider's address to the
information we need to connect to it.

This change enables two benefits: first, delve and other debuggers can
now be attached to provider server processes, and Terraform can connect.
This allows for attaching debuggers to provider processes, which before
was difficult to impossible. Second, it allows the SDK test framework to
host the provider in the same process as the test driver, while running
a production Terraform binary against the provider. This allows for Go's
built-in race detector and test coverage tooling to work as expected in
provider tests.

Unmanaged providers are expected to work in the exact same way as
managed providers, with one caveat: Terraform kills provider processes
and restarts them once per graph walk, meaning multiple times during
most Terraform CLI commands. As unmanaged providers can't be killed by
Terraform, and have no visibility into graph walks, unmanaged providers
are likely to have differences in how their global mutable state behaves
when compared to managed providers. Namely, unmanaged providers are
likely to retain global state when managed providers would have reset
it. Developers relying on global state should be aware of this.
2020-05-26 17:48:57 -07:00
Alisdair McDiarmid 5d0b75df7a
Merge pull request #25013 from hashicorp/alisdair/ui-data-source-read
command: Add UI hooks for read actions
2020-05-26 15:59:30 -04:00
Paul Tyng 22ef5cc99c Modify language for reporting signing state
Be more explicit about the signing status of fetched plugins and provide documentation about the different signing options.
2020-05-26 13:14:05 -04:00
Alisdair McDiarmid 62d826e066 command/init: Use full config for provider reqs
Relying on the early config for provider requirements was necessary in
Terraform 0.12, to allow the 0.12upgrade command to run after init
installs providers.

However in 0.13, the same restrictions do not apply, and the detection
of provider requirements has changed. As a result, the early config
loader gives incorrect provider requirements in some circumstances,
such as those in the new test in this commit.

Therefore we are changing the init command to use the requirements found
by the full configuration loader. This also means that we can remove the
internal initwd CheckCoreVersionRequirements function.
2020-05-25 16:50:12 -04:00
Alisdair McDiarmid 841ff590ea command: Add UI hooks for read actions 2020-05-21 16:02:18 -04:00
Kristin Laemmert eead4c49fe command/init: add e2e tests for provider not found messages 2020-05-20 11:04:11 -04:00
Kristin Laemmert 8d28d73de3 getproviders: add a registry-specific error and modify output when a
provider is not found.

Previously a user would see the following error even if terraform was
only searching the local filesystem:

"provider registry registry.terraform.io does not have a provider named
...."

This PR adds a registry-specific error type and modifies the MultiSource
installer to check for registry errors. It will return the
registry-specific error message if there is one, but if not the error
message will list all locations searched.
2020-05-20 11:04:11 -04:00
Kristin Laemmert a33a613703 command/init: add debug messages when -plugin-dir is set 2020-05-20 11:04:11 -04:00
Kristin Laemmert 0d620018fe
provider cache: log errors and validate dir exists (#24993)
* providercache: add logging for errors from getproviders.SearchLocalDirectory

providercache.fillMetaCache() was silently swallowing errors when
searching the cache directory. This commit logs the error without
changing the behavior otherwise.

* command/cliconfig: validate plugin cache dir exists

The plugin cache directory must exist for terraform to use it, so we
will add a check at the begining.
2020-05-19 15:32:36 -04:00
James Bardin 756e889697 remove unused DependsOn usage in state 2020-05-14 15:46:08 -04:00
Kristin Laemmert 862dc36884
command/init: include config loader diagnostics in output (#24921)
Previously the diagnostics from the config loaders (earlyconfig and
regular) were only appended to the overall diags if an error was found.
This adds all diagnostics from the regular config loader so that any
generated warnings will be displayed, even if there are no errors.

I did not add the `earlyconfig` warnings since they will be displayed if
there is an error and are likely to be duplicated by the config loader.
2020-05-12 08:39:12 -04:00
Kristin Laemmert 60321b41e8
getproviders: move protocol compatibility functions into registry client (#24846)
* internal/registry source: return error if requested provider version protocols are not supported

* getproviders: move responsibility for protocol compatibility checks into the registry client

The original implementation had the providercache checking the provider
metadata for protocol compatibility, but this is only relevant for the
registry source so it made more sense to move the logic into
getproviders.

This also addresses an issue where we were pulling the metadata for
every provider version until we found one that was supported. I've
extended the registry client to unmarshal the protocols in
`ProviderVersions` so we can filter through that list, instead of
pulling each version's metadata.
2020-05-11 13:49:12 -04:00
Alisdair McDiarmid 82ebbf9a23 command: Add prompt & confirmation to 0.13upgrade 2020-05-11 09:32:11 -04:00
Alisdair McDiarmid 62b0cbed12 internal: Fix LookupLegacyProvider
When looking up the namespace for a legacy provider source, we need to
use the /v1/providers/-/{name}/versions endpoint. For non-HashiCorp
providers, the /v1/providers/-/{name} endpoint returns a 404.

This commit updates the LegacyProviderDefaultNamespace method and the
mock registry servers accordingly.
2020-05-08 12:29:25 -04:00
Alisdair McDiarmid 1fdcbc4825 command: Fix 0.13upgrade bug with multiple blocks
If a configuration had multiple blocks in the versions.tf file, it would
be added to the `rewritePaths` list multiple times. We would then remove
it from this slice, but only once, and so the output file would later be
rewritten to remove the required providers block.

This commit uses a set instead of a list to prevent this case, and adds
a regression test.
2020-05-07 20:11:44 -04:00
Alisdair McDiarmid e2be704d81 command: Extract reused hclwrite helper code 2020-05-07 20:11:44 -04:00
Alisdair McDiarmid a740b739e0 command: Change 0.13upgrade default to versions.tf
Instead of using providers.tf as the default output file for the
upgrader, we now default to versions.tf. This means that if the
configuration has no `required_providers` blocks at all, or has
multiple, the provider version requirements will be stored in the
versions.tf file.

We now also update the versions.tf file to set a `required_version`
attribute in the first `terraform` block, with value ">= 0.13". This
is similar to the behaviour of the 0.12upgrade command, and signals that
the configuration should not be used with older versions of Terraform.
2020-05-07 15:45:48 -04:00
Alisdair McDiarmid 01a3376ead command: Check required_version before upgrading
If a configuration has a version constraint which prevents use with
Terraform 0.13, the upgrade command should exit before making any
changes.
2020-05-07 15:45:48 -04:00
Alisdair McDiarmid ae98bd12a7 command: Rework 0.13upgrade sub-command
This commit implements most of the intended functionality of the upgrade
command for rewriting configurations.

For a given module, it makes a list of all providers in use. Then it
attempts to detect the source address for providers without an explicit
source.

Once this step is complete, the tool rewrites the relevant configuration
files. This results in a single "required_providers" block for the
module, with a source for each provider.

Any providers for which the source cannot be detected (for example,
unofficial providers) will need a source to be defined by the user. The
tool writes an explanatory comment to the configuration to help with
this.
2020-05-07 11:38:55 -04:00
Mike Morris 9568de6b90
command: Fix OAuth2 PKCE arguments
Providers like Okta and AWS Cognito expect that the PKCE challenge
uses base64 URL Encoding without any padding (base64.RawURLEncoding)

Additionally, Okta strictly adheres to section 4.2 of RFC 7636 and
requires that the unencoded key for the PKCE data is at least 43
characters in length.
2020-05-05 09:58:48 -07:00
Kristin Laemmert 0b76100da0
init: return proper config errors (#24865)
Fixed a bug where we were returning earlyConfDiags instead of confDiags.
2020-05-05 10:08:05 -04:00
James Bardin 9debd341bc
Merge pull request #24807 from hashicorp/jbardin/remove-each-mode
Remove EachMode from state
2020-05-03 10:54:07 -04:00
Kristin Laemmert ce03f1255f
internal/providercache: fix error message for protocol mismatch (#24818)
There was a bug in the installer trying to pass a nil error.
2020-04-30 11:12:04 -04:00
James Bardin 2bfaddcf57 fix state mv to work without EachMode
The only situation where `state mv` needs to understand the each mode is
when with resource addresses that may reference a single instance, or a
group of for_each or count instances. In this case we can differentiate
the two by checking the existence of the NoKey instance key.
2020-04-30 09:22:15 -04:00
James Bardin 15a95031e5 remove a few traces of states.EachMode 2020-04-30 09:22:15 -04:00
Pam Selle 87bce5f9dd
Support reading module outputs in terraform console (#24808)
* Include eval in output walk

This allows outputs to be evaluated in the evalwalk,
impacting terraform console. Outputs are still not evaluated
for terraform console in the root module, so this has
no impact on writing to state (as child module outputs are not
written to state). Also adds test coverage to the console command,
including for evaluating locals (another use of the evalwalk)
2020-04-30 09:21:42 -04:00
Alisdair McDiarmid 9266e944fa
Merge pull request #24752 from hashicorp/alisdair/terraform-state-replace-provider-docs
website: Add docs for state replace-provider
2020-04-24 16:33:08 -04:00
Martin Atkins 1ce3c60693
command/cliconfig: Explicit provider installation method configuration
This set of commits allows explicit configuration of provider installation methods
in the CLI config, overriding the implicit method selections.
2020-04-23 10:58:00 -07:00
Martin Atkins 622abf707d command/cliconfig: Remove redundant struct types
These were being used in an earlier iteration of the provider installation
configuration but it was all collapsed down into a single
ProviderInstallationMethod type later, making these redundant.
2020-04-23 10:52:01 -07:00
Martin Atkins 6b2050f42a main: Properly handle provider installation method exclusions
Previously we were incorrectly using the Include configuration for both
the include and exclude list, making the include portion totally
ineffective.
2020-04-23 10:52:01 -07:00
Martin Atkins 3167067029 command/e2etest: provider installation with explicit install methods
This exercises the ability to customize the installation methods used by
the provider plugin installer, in this case forcing the use of a custom
local directory with a result essentially the same as what happens when
you pass -plugin-dir to "terraform init".
2020-04-23 10:52:01 -07:00
Martin Atkins c7fe6b9160 command/cliconfig: handle provider_installation block in JSON syntax
The CLI config can be written in both native HCL and HCL JSON syntaxes, so
the provider_installation block must be expressible using JSON too. Our
previous checks to approximate HCL 2-level strictness were too strict for
HCL JSON where things are more ambiguous even in HCL 2, so this includes
some additional relaxations if we detect that we're decoding an AST
produced from a JSON file.

This is still subject to the quirky ways HCL 1 handles JSON though, so
the JSON value must be structured in a way that doesn't trigger HCL's
heuristics that try to guess what is a block and what is an attribute.
(This is the issue that HCL 2 fixes by always decoding using a schema;
there's more context on this in:
  https://log.martinatkins.me/2019/04/25/hcl-json/ )
2020-04-23 10:52:01 -07:00
Martin Atkins f5012c12da command/cliconfig: Installation methods, not installation sources
Unfortunately in the user model the noun "source" is already used for the
argument in the required_providers block to specify which provider to use,
so it's confusing to use the same noun to also refer to the method used to
obtain that provider.

In the hope of mitigating that confusion, here we use the noun "method",
as in "installation method", to talk about the decision between getting
a provider directly from its origin registry or getting it from some
mirror. This is distinct from the provider's "source", which is the
location where a provider _originates_ (prior to mirroring).

This noun is also not super awesome, but better than overloading an
existing term in the same feature.
2020-04-23 10:52:01 -07:00