Commit Graph

100 Commits

Author SHA1 Message Date
James Bardin ced676d50d fixed configloader doesn't add unnecessary diags 2021-05-07 12:28:09 -04:00
Alisdair McDiarmid a5b7394f9a command/jsonplan: Add replace_paths
The set of paths which caused a resource update to require replacement
has been stored in the plan since 0.15.0 (#28201). This commit adds a
simple JSON representation of these paths, allowing consumers of this
format to determine exactly which paths caused the resource to be
replaced.

This representation is intentionally more loosely encoded than the JSON
state serialization of paths used for sensitive attributes. Instead of a
path step being represented by an object with type and value, we use a
more-JavaScripty heterogenous array of numbers and strings. Any
practical consumer of this format will likely traverse an object tree
using the index operator, which should work more easily with this
format. It also allows easy prefix comparison for consumers which are
tracking paths.

While updating the documentation to include this new field, I noticed
that some others were missing, so added them too.
2021-05-04 16:51:51 -04:00
Martin Atkins 1d3e34e35e command: New -replace=... planning option
This allows a similar effect to pre-tainting an object but does the action
within the context of a normal plan and apply, avoiding the need for an
intermediate state where the old object still exists but is marked as
tainted.

The core functionality for this was already present, so this commit is
just the UI-level changes to make that option available for use and to
explain how it contributed to the resulting plan in Terraform's output.
2021-05-03 15:43:23 -07:00
Alisdair McDiarmid f72730a02b cli: Add JSON logs for operations commands 2021-04-30 11:37:36 -04:00
Martin Atkins b802237e03 plans: Track an optional extra "reason" for some planned actions
Previously we were repeating some logic in the UI layer in order to
recover relevant additional context about a change to report to a user.
In order to help keep things consistent, and to have a clearer path for
adding more such things in the future, here we capture this user-facing
idea of an "action reason" within the plan model, and then use that
directly in order to decide how to describe the change to the user.

For the moment the "tainted" situation is the only one that gets a special
message, matching what we had before, but we can expand on this in future
in order to give better feedback about the other replace situations too.

This also preemptively includes the "replacing by request" reason, which
is currently not reachable but will be used in the near future as part of
implementing the -replace=... plan command line option to allow forcing
a particular object to be replaced.

So far we don't have any special reasons for anything other than replacing,
which makes sense because replacing is the only one that is in a sense
a special case of another action (Update), but this could expand to
other kinds of reasons in the future, such as explaining which of the
few different reasons a data source read might be deferred until the
apply step.
2021-04-29 17:50:46 -07:00
Alisdair McDiarmid c89004d223 core: Add sensitive provider attrs to JSON plan
When rendering a stored plan file as JSON, we include a data structure
representing the sensitivity of the changed resource values. Prior to
this commit, this was a direct representation of the sensitivity marks
applied to values via mechanisms such as sensitive variables, sensitive
outputs, and the `sensitive` function.

This commit extends this to include sensitivity based on the provider
schema. This is in line with the UI rendering of the plan, which
considers these two different types of sensitivity to be equivalent.

Co-authored-by: Kristin Laemmert <mildwonkey@users.noreply.github.com>
2021-04-27 10:29:34 -04:00
Chris Arcand 58cba1dea3 Add customized login success output for TFC/E
When logging in to Terraform Cloud or Terraform Enterprise, change the
success output to be a bit more customized for the platform. For
Terraform Cloud, fetch a dynamic welcome banner that intentionally fails
open and defaults to a hardcoded message if its not available for any
reason.
2021-04-21 21:23:42 -05:00
Alisdair McDiarmid 340543d122
Merge pull request #28201 from hashicorp/alisdair/planfile-sensitive-required-replace
Add sensitivity and required-replace fields to plan file, and expose sensitivity to JSON plan
2021-03-29 15:08:36 -04:00
Alisdair McDiarmid 5e30d58dc2 command/jsonplan: Add output change sensitivity
When an output value changes, we have a small amount of information we
can convey about its sensitivity. If either the output was previously
marked sensitive, or is currently marked sensitive in the config, this
is tracked in the output change data.

This commit encodes this boolean in the change struct's
`before_sensitive` and `after_sensitive` fields, in the a way which
matches resource value sensitivity. Since we have so little information
to work with, these two values will always be booleans, and always equal
each.

This is logically consistent with how else we want to obscure sensitive
data: a changing output which was or is marked sensitive should not have
the value shown in human-readable output.
2021-03-26 19:26:11 -04:00
Alisdair McDiarmid 63613ca1b0 command/jsonconfig: Add variable sensitive flag 2021-03-26 19:26:11 -04:00
Alisdair McDiarmid e27aacebf9 command/jsonplan: Add sensitive value mapping data
Similar to `after_unknown`, `before_sensitive` and `after_sensitive` are
values with similar structure to `before` and `after` which encode the
presence of sensitive values in a planned change. These should be used
to obscure sensitive values from human-readable output.

These values follow the same structure as the `before` and `after`
values, replacing sensitive values with `true`, and non-sensitive values
with `false`. Following the `after_unknown` precedent, we omit
non-sensitive `false` values for object attributes/map values, to make
serialization more compact.

One difference from `after_unknown` is that a sensitive complex value
(collection or structural type) is replaced with `true`. If the complex
value itself is sensitive, all of its contents should be obscured.
2021-03-26 19:26:10 -04:00
Alisdair McDiarmid d71f0c6149 cli: Fix fmt output for multi-line value exprs
The formatter for value expressions which use legacy interpolation
syntax was previously behaving incorrectly with some multi-line
expressions. Any HCL expression which requires parenthesis to be allowed
to span multiple lines could be skip those parens if already inside
string interpolation (`"${}"`).

When removing string interpolation, we now check for a resulting
multi-line expression, and conservatively ensure that it starts and ends
with parenthesis. These may be redundant, as not all expressions require
parens to permit spanning multiple lines, but at least it will be valid
output.
2021-03-25 15:40:54 -04:00
Kristin Laemmert 77562d9b57
command/jsonprovider: bump format version (#28115)
Support for attributes with NestedTypes was added in https://github.com/hashicorp/terraform/pull/28055, and should have included a format version bump: this is a backwards-compatible change, but consumers will need to be updated in order to properly decode attributes (with NestedTypes) going forward.
2021-03-22 11:45:36 -04:00
Alisdair McDiarmid 46a29b13ed cli: Add format version to validate -json output
In line with the other complex JSON output formats for plan and provider
schema, here we add an explicit `format_version` field to the JSON
output of terraform validate.
2021-03-16 09:46:36 -04:00
Alisdair McDiarmid c92826c14d cli: Use standard JSON diagnostics for validate
Now that we have a comprehensive JSON diagnostic structure, we can use
it in the `validate -json` output instead of the inline version. Note
that this changes the output of `validate -json` in two ways:

1. We fix some off-by-one errors caused by zero-width highlight ranges.
   This aligns the JSON diagnostic output with the text output seen by
   most Terraform users, so I consider this a bug fix.
2. We add the `snippet` field to the JSON diagnostics where available.
   This is purely additive and is permitted under our JSON format
   stability guarantees.
2021-03-12 13:12:37 -05:00
Kristin Laemmert b26ae9cf48
add support for attributes with nested types in providers schema (#28055)
This PR extends jsonprovider to support attributes with NestedTypes and extends test coverage in jsonprovider and the providers schemas tests. I've also cleaned up some comments and extracted the logic to parse the nesting mode so it can be used in both marshalling blocks and attributes.
2021-03-12 08:28:22 -05:00
Masayuki Morita 31a5aa1878
command/init: Add a new flag `-lockfile=readonly` (#27630)
Fixes #27506

Add a new flag `-lockfile=readonly` to `terraform init`.
It would be useful to allow us to suppress dependency lockfile changes
explicitly.

The type of the `-lockfile` flag is string rather than bool, leaving
room for future extensions to other behavior variants.

The readonly mode suppresses lockfile changes, but should verify
checksums against the information already recorded. It should conflict
with the `-upgrade` flag.

Note: In the original use-case described in #27506, I would like to
suppress adding zh hashes, but a test code here suppresses adding h1
hashes because it's easy for testing.

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2021-03-09 11:12:00 -05:00
Martin Atkins 7f78342953 command: Experimental "terraform test" command
This is just a prototype to gather some feedback in our ongoing research
on integration testing of Terraform modules. The hope is that by having a
command integrated into Terraform itself it'll be easier for interested
module authors to give it a try, and also easier for us to iterate quickly
based on feedback without having to coordinate across multiple codebases.

Everything about this is subject to change even in future patch releases.
Since it's a CLI command rather than a configuration language feature it's
not using the language experiments mechanism, but generates a warning
similar to the one language experiments generate in order to be clear that
backward compatibility is not guaranteed.
2021-02-22 14:21:45 -08:00
Pam Selle 8f7807684a Upgrade to quoted keywords to error
The warning about deprecation is upgraded to an error
2021-02-21 20:27:20 -05:00
Pam Selle fa7c3d7e10 Remove interpolation-only warning
These interpolations are removed when upgrading using 0.12upgrade,
and are removed in terraform fmt in many cases
2021-02-19 10:59:09 -05:00
Pam Selle 156b1615a2 Remove 0.13upgrade test fixtures 2021-02-18 12:57:58 -05:00
Alisdair McDiarmid bb3f59b016
Merge pull request #27697 from hashicorp/alisdair/json-plan-provider-required-versions
cli: Fix for missing provider requirements in JSON plan
2021-02-16 09:05:14 -05:00
Alisdair McDiarmid 4991cc4835 cli: Improve error for invalid -target flags
Errors encountered when parsing flags for apply, plan, and refresh were
being suppressed. This resulted in a generic usage error when using an
invalid `-target` flag.

This commit makes several changes to address this. First, these commands
now output the flag parse error before exiting, leaving at least some
hint about the error. You can verify this manually with something like:

    terraform apply -invalid-flag

We also change how target attributes are parsed, moving the
responsibility from the flags instance to the command. This allows us to
customize the diagnostic output to be more user friendly. The
diagnostics now look like:

```shellsession
$ terraform apply -no-color -target=foo

Error: Invalid target "foo"

Resource specification must include a resource type and name.
```

Finally, we add test coverage for both parsing of target flags, and at
the command level for successful use of resource targeting. These tests
focus on the UI output (via the change summary and refresh logs), as the
functionality of targeting is covered by the context tests in the
terraform package.
2021-02-08 13:48:04 -05:00
Alisdair McDiarmid 7cae76383a cli: Fix for provider requirements in JSON plan
The JSON plan output format includes a serialized, simplified version of
the configuration. One component of this config is a map of provider
configurations, which includes version constraints.

Until now, only version constraints specified in the provider config
blocks were exposed in the JSON plan output. This is a deprecated method
of specifying provider versions, and the recommended use of a
required_providers block resulted in the version constraints being
omitted.

This commit fixes this with two changes:

- When processing the provider configurations from a module, output the
  fully-merged version constraints for the entire module, instead of any
  constraints set in the provider configuration block itself;
- After all provider configurations are processed, iterate over the
  required_providers entries to ensure that any configuration-less
  providers are output to the JSON plan too.

No changes are necessary to the structure of the JSON plan output, so
this is effectively a semantic level bug fix.
2021-02-05 14:01:58 -05:00
Nick Fagerlund ed7f97c34f Update language docs URLs in code and tests 2021-01-22 12:22:21 -08:00
Kristin Laemmert 8bab3dd374
command/state list: list resources in nested and expanded modules (#27268)
* command/state list: list resources in nested and expaneded modules

A few distinct bugs fixed in here:

There was a bug in the logic checking if a given module was the child of
the targetAddr, now fixed. That resolved the basic issue where resources
in nested submodules were not listed.

The logic around allowMissing needed some tweaking to allow for empty
modules, as long as those modules had submodules with resources. state
list is the only command using allowMissing with false so this felt safe
to do.

Finally I extended the logic so list would included expanded modules,
which is to say giving module.foo would result in resources from
module.foo[1], module.foo[0], etc.

* update state list docs to show that module filtering includes any nested
modules
2020-12-14 11:07:15 -05:00
Alisdair McDiarmid f1b95788b9 command: Add tests for terraform validate -json
Also uncomment and fix some tests which had been skipped for a couple of
years. Those validate cases work now!

Note that these test cases and the JSON output are not especially
minimized, making them snapshot/golden tests. The output looks correct
at time of writing, and we don't expect to change validate significantly
any time soon, but if we do there will be some churn here.
2020-12-11 13:09:25 -05:00
Alisdair McDiarmid 21d80a26ea command: Fix fmt to preserve blank block labels 2020-11-18 11:59:10 -05:00
Pam Selle 66091ae36c Unmark values before showing in JSON
This prevents "sensitive" values from unintentionally
showing as nil when running terraform show -json
2020-10-28 15:30:04 -04:00
Martin Atkins e70ab09bf1 command: new cache directory .terraform/providers for providers
Terraform v0.10 introduced .terraform/plugins as a cache directory for
automatically-installed plugins, Terraform v0.13 later reorganized the
directory structure inside but retained its purpose as a cache.

The local cache used to also serve as a record of specifically which
packages were selected in a particular working directory, with the intent
that a second run of "terraform init" would always select the same
packages again. That meant that in some sense it behaved a bit like a
local filesystem mirror directory, even though that wasn't its intended
purpose.

Due to some unfortunate miscommunications, somewhere a long the line we
published some documentation that _recommended_ using the cache directory
as if it were a filesystem mirror directory when working with Terraform
Cloud. That was really only working as an accident of implementation
details, and Terraform v0.14 is now going to break that because the source
of record for the currently-selected provider versions is now the
public-facing dependency lock file rather than the contents of an existing
local cache directory on disk.

After some consideration of how to move forward here, this commit
implements a compromise that tries to avoid silently doing anything
surprising while still giving useful guidance to folks who were previously
using the unsupported strategy. Specifically:

- The local cache directory will now be .terraform/providers rather than
  .terraform/plugins, because .terraform/plugins is effectively "poisoned"
  by the incorrect usage that we can't reliably distinguish from prior
  version correct usage.

- The .terraform/plugins directory is now the "legacy cache directory". It
  is intentionally _not_ now a filesystem mirror directory, because that
  would risk incorrectly interpreting providers automatically installed
  by Terraform v0.13 as if they were a local mirror, and thus upgrades
  and checksum fetches from the origin registry would be blocked.

- Because of the previous two points, someone who _was_ trying to use the
  legacy cache directory as a filesystem mirror would see installation
  fail for any providers they manually added to the legacy directory.

  To avoid leaving that user stumped as to what went wrong, there's a
  heuristic for the case where a non-official provider fails installation
  and yet we can see it in the legacy cache directory. If that heuristic
  matches then we'll produce a warning message hinting to move the
  provider under the terraform.d/plugins directory, which is a _correct_
  location for "bundled" provider plugins that belong only to a single
  configuration (as opposed to being installed globally on a system).

This does unfortunately mean that anyone who was following the
incorrectly-documented pattern will now encounter an error (and the
aforementioned warning hint) after upgrading to Terraform v0.14. This
seems like the safest compromise because Terraform can't automatically
infer the intent of files it finds in .terraform/plugins in order to
decide automatically how best to handle them.

The internals of the .terraform directory are always considered
implementation detail for a particular Terraform version and so switching
to a new directory for the _actual_ cache directory fits within our usual
set of guarantees, though it's definitely non-ideal in isolation but okay
when taken in the broader context of this problem, where the alternative
would be silent misbehavior when upgrading.
2020-10-14 07:53:41 -07:00
James Bardin d2514a9abd update new outputs plan json 2020-10-12 17:29:45 -04:00
Pam Selle bc57c20d10 Remove sensitive_variables experiment
Ahead of the beta, remove the sensitive_variable experiment
and update tests accordingly
2020-10-08 11:22:20 -04:00
Pam Selle c57ca152e6 Obfuscate sensitive vals in console
Updates terraform console to show "(sensitive)"
when a value is marked as sensitive.
2020-10-05 13:16:34 -04:00
Kristin Laemmert 479655ad47 refactor tests to use modern states.State in favor of terraform.State where possible 2020-09-30 16:07:54 -04:00
Martin Atkins ff0dbd6215 command/fmt: Restore some opinionated behaviors
In Terraform 0.11 and earlier, the "terraform fmt" command was very
opinionated in the interests of consistency. While that remains its goal,
for pragmatic reasons Terraform 0.12 significantly reduced the number
of formatting behaviors in the fmt command. We've held off on introducing
0.12-and-later-flavored cleanups out of concern it would make it harder
to maintain modules that are cross-compatible with both Terraform 0.11
and 0.12, but with this aimed to land in 0.14 -- two major releases
later -- our new goal is to help those who find older Terraform language
examples learn about the more modern idiom.

More rules may follow later, now that the implementation is set up to
allow modifications to tokens as well as modifications to whitespace, but
for this initial pass the command will now apply the following formatting
conventions:

 - 0.11-style quoted variable type constraints will be replaced with their
   0.12 syntax equivalents. For example, "string" becomes just string.
   (This change quiets a deprecation warning.)
 - Collection type constraints that don't specify an element type will
   be rewritten to specify the "any" element type explicitly, so
   list becomes list(any).
 - Arguments whose expressions consist of a quoted string template with
   only a single interpolation sequence inside will be "unwrapped" to be
   the naked expression instead, which is functionally equivalent.
   (This change quiets a deprecation warning.)
 - Block labels are given in quotes.

Two of the rules above are coming from a secondary motivation of
continuing down the deprecation path for two existing warnings, so authors
can have two active deprecation warnings quieted automatically by
"terraform fmt", without the need to run any third-party tools.

All of these rules match with current documented idiom as shown in the
Terraform documentation, so anyone who follows the documented style should
see no changes as a result of this. Those who have adopted other local
style will see their configuration files rewritten to the standard
Terraform style, but it should not make any changes that affect the
functionality of the configuration.

There are some further similar rewriting rules that could be added in
future, such as removing 0.11-style quotes around various keyword or
static reference arguments, but this initial pass focused only on some
rules that have been proven out in the third-party tool
terraform-clean-syntax, from which much of this commit is a direct port.

For now this doesn't attempt to re-introduce any rules about vertical
whitespace, even though the 0.11 "terraform fmt" would previously apply
such changes. We'll be more cautious about those because the results of
those rules in Terraform 0.11 were often sub-optimal and so we'd prefer
to re-introduce those with some care to the implications for those who
may be using vertical formatting differences for some semantic purpose,
like grouping together related arguments.
2020-09-28 09:04:03 -07:00
Alisdair McDiarmid 14a233b019 command: Taint should respect required_version
Despite not requiring the configuration for any other reason, the taint
subcommand should not execute if the required_version constraints cannot
be met. Doing so can result in an undesirable state file upgrade.
2020-09-22 17:33:09 -04:00
Alisdair McDiarmid 18f9ea53b9 command: Providers schema shows required_providers
The providers schema command is using the Config.ProviderTypes method,
which had not been kept up to date with the changes to provider
requirements detection made in Config.ProviderRequirements. This
resulted in any currently-unused providers being omitted from the
output.

This commit changes the ProviderTypes method to use the same underlying
logic as ProviderRequirements, which ensures that `required_providers`
blocks are taken into account.

Includes an integration test case to verify that this fixes the provider
schemas command bug.
2020-09-22 10:28:32 -04:00
James Bardin 3b3ff98356 Revert "fix show -json tests"
This reverts commit e54949f2e1.

Changes incorrectly applied to the planned state tests
2020-09-21 16:17:45 -04:00
James Bardin e54949f2e1 fix show -json tests
The prior state recorded in the plans did not match the actual prior
state. Make the plans and state match depending on whether there was
existing state or not.
2020-09-17 09:55:00 -04:00
Alisdair McDiarmid 9f824c53a5 command: Better in-house provider install errors
When init attempts to install a legacy provider required by state and
fails, but another provider with the same type is successfully
installed, this almost definitely means that the user is migrating an
in-house provider. The solution here is to use the `terraform state
replace-provider` subcommand.

This commit makes that next step clearer, by detecting this specific
case, and displaying a list of commands to fix the existing state
provider references.
2020-09-01 14:02:19 -04:00
Alisdair McDiarmid fc7e467d19 command: Add redirect support to 0.13upgrade
If a provider changes namespace in the registry, we can detect this when
running the 0.13upgrade command. As long as there is a version matching
the user's constraints, we now use the provider's new source address.
Otherwise, warn the user that the provider has moved and a version
upgrade is necessary to move to it.
2020-08-31 14:53:35 -04:00
Alisdair McDiarmid 677aabc767 command: Fix backend config override validation
When loading a backend config override file, init was doing two things
wrong:

- First, if the file failed to parse, we accidentally didn't return,
  which caused a panic due to the parsed body being nil;
- Secondly, we were overzealous with the validation of the file,
  allowing only attributes. While most backend configs are attributes
  only, the enhanced remote backend body also contains a `workspaces`
  block, which we need to support here.

This commit fixes the first bug with an early return and adds test cases
for missing file and intentionally-blank filename (to clear the config).

We also add a schema validation for the backend block, based on the
backend schema itself. This requires constructing an HCL body schema so
that we can call `Content` and check for diagnostic errors.

The result is more useful errors when an invalid backend config override
file is used, while also supporting the enhanced remote backend config
fully.

Does not include tests specific to the remote backend, because the
mocking involved to allow the backend to fully initialize is too
involved to be worth it.
2020-08-21 16:21:13 -04:00
Cameron Stitt 54e32652f7
Ensure depends_on is in module calls for config 2020-08-20 07:49:03 +10:00
Alisdair McDiarmid 30c7dfca62
Merge pull request #25898 from hashicorp/alisdair/fix-required-version-diags
terraform: Fix required version constraint diags
2020-08-19 11:26:03 -04:00
Alisdair McDiarmid c98f352dc8 terraform: Fix required version constraint diags
If a module has multiple terraform.required_version constraints, any
failures would point at the last constraint in the error diagnostics. If
an earlier constraint was the actual problem, this leads to confusing
errors like this:

    Error: Unsupported Terraform Core version

      on main.tf line 6, in terraform:
       6:   required_version = ">= 0.13.0"

    This configuration does not support Terraform version 0.13.0.

The error was due to storing the declaration range of the constraint as
a pointer to the contents of a loop variable, which was later
overwritten in later iterations of the loop.  Instead we now use HCL's
handy Ptr() method to create a direct pointer to the range struct.
2020-08-18 09:35:32 -04:00
Alisdair McDiarmid d8e9964363 terraform: Eval module call arguments for import
Include the import walk in the list of operations for which we create an
EvalModuleCallArgument node. This causes module call arguments to be
evaluated even if the module variables have defaults, ensuring that
invalid default values (such as the common "{}" for variables thought of
as maps) do not cause failures specific to import.

This fixes a bug where a child module evaluates an input variable in its
locals block, assuming that it is a nested object structure. The bug
report includes a default value of "{}", which is overridden by a root
variable value. Without the eval node added in this commit, the default
value is used and the local evaluation errors.
2020-08-17 17:14:12 -04:00
James Bardin 1c09df1a66
Merge pull request #25779 from hashicorp/jbardin/remove-state-attrs
Remove resource state attributes that are no longer in the schema
2020-08-12 10:49:44 -04:00
Kristin Laemmert 6621501ae3
state: remove deprecated state package (#25490)
Most of the state package has been deprecated by the states package.
This PR replaces all the references to the old state package that
can be done simply - the low-hanging fruit.

* states: move state.Locker to statemgr

The state.Locker interface was a wrapper around a statemgr.Full, so
moving this was relatively straightforward.

* command: remove unnecessary use of state package for writing local terraform state files

* move state.LocalState into terraform package

state.LocalState is responsible for managing terraform.States, so it
made sense (to me) to move it into the terraform package.

* slight change of heart: move state.LocalState into clistate instead of
terraform
2020-08-11 11:43:01 -04:00
James Bardin 99cd3ab223 fix command tests
A number of tests had invalid configs or providers, but were never
properly validated
2020-08-07 14:13:57 -04:00
Kristin Laemmert f8e3456867
command/show: fix bug displaying provider config in json output of tf plan (#25577)
A lingering FIXME caused missing configuration from provider config
blocks in the json output of terraform plan. This fixes the regression
and adds a test. For the sake of testing, I added an optional attribute
to the show test provider, which resulted in the providers schema test
getting an update - not a bad addition, but we can always add a
test-specific provider schema as needed.
2020-07-14 15:28:31 -04:00