Commit Graph

1829 Commits

Author SHA1 Message Date
Martin Atkins 8f233cde4c cli: Diagnostics can include collections with sensitive elements
We previously had a shallow IsMarked call in compactValueStr's caller but
then a more-conservative deep ContainsMarked call inside compactValueStr
with a different resulting message. As well as causing an inconsistency
in messages, this was also a bit confusing because it made it seem like
a non-sensitive collection containing a sensitive element value was wholly
sensitive, making the debug information in the diagnostic messages not
trustworthy for debugging certain varieties of problem.

I originally considered just removing the redundant check in
compactValueStr here, but ultimately I decided to keep it as a sort of
defense in depth in case a future refactoring disconnects these two
checks. This should also serve as a prompt to someone making later changes
to compactValueStr to think about the implications of sensitive values
in there, which otherwise wouldn't be mentioned at all.

Disclosing information about a collection containing sensitive values is
safe here because compactValueStr only discloses information about the
value's type and element keys, and neither of those can be sensitive in
isolation. (Constructing a map with sensitive keys reduces to a sensitive
overall map.)
2021-04-19 11:58:49 -07:00
Alisdair McDiarmid 8dcf768f4e backend/remote: Add IsLocalOperations
To ensure that the apply command can determine whether an operation is
executed locally or remotely, we add an IsLocalOperations method on the
remote backend. This returns the internal forceLocal boolean.

We also update this flag after checking if the corresponding remote
workspace is in local operations mode or not. This ensures that we know
if an operation is running locally (entirely on the practitioner's
machine), pseudo-locally (on a Terraform Cloud worker), or remotely
(executing on a worker, rendering locally).
2021-04-16 11:43:57 -04:00
Alisdair McDiarmid 69e7922a33 cli: Fix missing apply summary for remote runs
Disabling the resource count and outputs rendering when the remote
backend is in use causes them to be omitted from Terraform Cloud runs.
This commit changes the condition to render these values if either the
remote backend is not in use, or the command is running in automation
via the TF_IN_AUTOMATION flag. As this is intended to be set by
Terraform Cloud and other remote backend implementations, this addresses
the problem.
2021-04-16 10:03:22 -04:00
Alisdair McDiarmid fad305f884 cli: Fix remote backend UI issues
Fix two bugs which surface when using the remote backend:

- When migrating to views, we removed the call to `(*Meta).process`
  which initialized the color boolean. This resulted in the legacy UI
  calls in the remote backend stripping color codes. To fix this, we
  populate this boolean from the common arguments.
- Remote apply will output the resource summary and output changes, and
  these are rendered via the remote backend streaming. We need to
  special case this in the apply command and prevent displaying a
  zero-change summary line.

Neither of these are coverable by automated tests, as we don't have any
command-package level testing for the remote backend. Manually verified.
2021-04-16 08:28:33 -04:00
James Bardin d04999863c "with" formatting 2021-04-06 15:50:30 -04:00
James Bardin 265b5106ca call the InConfigBody with addresses 2021-04-06 15:15:52 -04:00
James Bardin 406ac97965 add the address field to the view diagnostics 2021-04-06 15:15:52 -04:00
Alisdair McDiarmid ff32fab41a command/jsonplan: Fix sensitive/unknown crash
When rendering the JSON plan sensitivity output, if the plan contained
unknown collection or structural types, Terraform would crash. We need
to detect unknown values before attempting to iterate them.

Unknown collection or structural values cannot have sensitive contents
accidentally displayed, as those values are not known until after apply.
As a result we return an empty value of the appropriate type for the
sensitivity mapping.
2021-03-31 14:29:15 -04:00
Alisdair McDiarmid 786d99207e cli: Only rewrite provider locks file if changed
If the provider locks have not changed, there is no need to rewrite the
locks file. Preventing this needless rewrite should allow Terraform to
operate in a read-only directory, so long as the provider requirements
don't change.
2021-03-29 16:09:07 -04:00
Alisdair McDiarmid 340543d122
Merge pull request #28201 from hashicorp/alisdair/planfile-sensitive-required-replace
Add sensitivity and required-replace fields to plan file, and expose sensitivity to JSON plan
2021-03-29 15:08:36 -04:00
Alisdair McDiarmid 5e30d58dc2 command/jsonplan: Add output change sensitivity
When an output value changes, we have a small amount of information we
can convey about its sensitivity. If either the output was previously
marked sensitive, or is currently marked sensitive in the config, this
is tracked in the output change data.

This commit encodes this boolean in the change struct's
`before_sensitive` and `after_sensitive` fields, in the a way which
matches resource value sensitivity. Since we have so little information
to work with, these two values will always be booleans, and always equal
each.

This is logically consistent with how else we want to obscure sensitive
data: a changing output which was or is marked sensitive should not have
the value shown in human-readable output.
2021-03-26 19:26:11 -04:00
Alisdair McDiarmid 63613ca1b0 command/jsonconfig: Add variable sensitive flag 2021-03-26 19:26:11 -04:00
Alisdair McDiarmid e27aacebf9 command/jsonplan: Add sensitive value mapping data
Similar to `after_unknown`, `before_sensitive` and `after_sensitive` are
values with similar structure to `before` and `after` which encode the
presence of sensitive values in a planned change. These should be used
to obscure sensitive values from human-readable output.

These values follow the same structure as the `before` and `after`
values, replacing sensitive values with `true`, and non-sensitive values
with `false`. Following the `after_unknown` precedent, we omit
non-sensitive `false` values for object attributes/map values, to make
serialization more compact.

One difference from `after_unknown` is that a sensitive complex value
(collection or structural type) is replaced with `true`. If the complex
value itself is sensitive, all of its contents should be obscured.
2021-03-26 19:26:10 -04:00
Alisdair McDiarmid fd7c4105f2
Merge pull request #28202 from hashicorp/alisdair/fmt-multiline-value-expr-unwrap
cli: Fix fmt output for multi-line value exprs
2021-03-26 11:33:14 -04:00
Martin Atkins 6f35c2847b command: Reorganize docs of the local backend's legacy CLI options
We have these funny extra options that date back to before Terraform even
had remote state, which we've preserved along the way by most recently
incorporating them as special-case overrides for the local backend.

The documentation we had for these has grown less accurate over time as
the details have shifted, and was in many cases missing the requisite
caveats that they are only for the local backend and that backend
configuration is the modern, preferred way to deal with the use-cases they
were intended for.

We always have a bit of a tension with this sort of legacy option because
we want to keep them documented just enough to be useful to someone who
finds an existing script/etc using them and wants to know what they do,
but not to take up so much space that they might distract users from
finding the modern alternative they should consider instead.

As a compromise in that vein here I've created a new section about these
options under the local backend documentation, which then gives us the
space to go into some detail about the various behaviors and interactions
and also to discuss their history and our recommended alternatives. I then
simplified all of the other mentions of these in command documentation
to just link to or refer to the local backend documentation. My hope then
is that folks who need to know what these do can still find the docs, but
that information can be kept out of the direct path of new users so they
can focus on learning about remote backends instead.

This is certainly not the most ideal thing ever, but it seemed like the
best compromise between the competing priorities I described above.
2021-03-25 13:56:48 -07:00
Alisdair McDiarmid d71f0c6149 cli: Fix fmt output for multi-line value exprs
The formatter for value expressions which use legacy interpolation
syntax was previously behaving incorrectly with some multi-line
expressions. Any HCL expression which requires parenthesis to be allowed
to span multiple lines could be skip those parens if already inside
string interpolation (`"${}"`).

When removing string interpolation, we now check for a resulting
multi-line expression, and conservatively ensure that it starts and ends
with parenthesis. These may be redundant, as not all expressions require
parens to permit spanning multiple lines, but at least it will be valid
output.
2021-03-25 15:40:54 -04:00
Alisdair McDiarmid 841cced6c9
Merge pull request #28138 from hashicorp/alisdair/command-views-validate
cli: Migrate validate command to views
2021-03-23 13:13:06 -04:00
Alisdair McDiarmid 7242d9af2b Fix typo in plan no changes output message 2021-03-22 16:39:53 -04:00
Alisdair McDiarmid 46c2dce0fb Fix broken test 2021-03-22 16:38:56 -04:00
Kristin Laemmert b9138f4465
terraform: validate providers' schemas during NewContext (#28124)
* checkpoint save: update InternalValidate tests to compare exact error

* configschema: extract and extend attribute validation

This commit adds an attribute-specific InternalValidate which was extracted directly from the block.InternalValidate logic and extended to verify any NestedTypes inside an Attribute. Only one error message changed, since it is now valid to have a cty.NilType for Attribute.Type as long as NestedType is set.

* terraform: validate provider schema's during NewContext

We haven't been able to guarantee that providers are validating their own schemas using (some version of) InternalValidate since providers were split out of the main codebase. This PR adds a call to InternalValidate when provider schemas are initially loaded by NewContext, which required a few other changes:

InternalValidate's handling of errors vs multierrors was a little weird - before this PR, it was occasionally returning a non-nil error which only stated "0 errors occurred" - so I addressed that in InternalValidate. I then tested this with a configuration that was using all of our most popular providers, and found that at least on provider had some invalid attribute names, so I commented that particular validation out. Adding that in would be a breaking change which we would have to coordinate with enablement and providers and (especially in this case) make sure it's well communicated to external provider developers.

I ran a few very unscientific tests comparing the timing with and without this validation, and it appeared to only cause a sub-second increase.

* refactor validate error message to closer match the sdk's message

* better error message

* tweak error message: move the instruction to run init to the end of the message, after the specific error.
2021-03-22 13:17:50 -04:00
Kristin Laemmert 77562d9b57
command/jsonprovider: bump format version (#28115)
Support for attributes with NestedTypes was added in https://github.com/hashicorp/terraform/pull/28055, and should have included a format version bump: this is a backwards-compatible change, but consumers will need to be updated in order to properly decode attributes (with NestedTypes) going forward.
2021-03-22 11:45:36 -04:00
Alisdair McDiarmid dd380d0b58 cli: Migrate validate command to views 2021-03-18 15:56:28 -04:00
Alisdair McDiarmid 46a29b13ed cli: Add format version to validate -json output
In line with the other complex JSON output formats for plan and provider
schema, here we add an explicit `format_version` field to the JSON
output of terraform validate.
2021-03-16 09:46:36 -04:00
Kristin Laemmert fac60ab124
Add formatting for attributes with nested types (#28069)
* format/diff: extract attributes-writing logic to a function

This is a stepping-stone commit (for easier reviewability, and to prove that tests did not change) as part of writing a NestedType-specific diff printer.

* command/format: add support for formatting attributes with NestedTypes

This commit adds custom formatting for NestedType attributes. THe logic was mostly copied from the block diff printer, with minor tweaks here and there. I used the (excellent) existing test coverage and added a NestedType attribute to every test.

Since the (nested-block specific) test schemas were nearly identical, I added a function that returns the schema with the requested NestingMode.
2021-03-15 09:31:23 -04:00
Alisdair McDiarmid c92826c14d cli: Use standard JSON diagnostics for validate
Now that we have a comprehensive JSON diagnostic structure, we can use
it in the `validate -json` output instead of the inline version. Note
that this changes the output of `validate -json` in two ways:

1. We fix some off-by-one errors caused by zero-width highlight ranges.
   This aligns the JSON diagnostic output with the text output seen by
   most Terraform users, so I consider this a bug fix.
2. We add the `snippet` field to the JSON diagnostics where available.
   This is purely additive and is permitted under our JSON format
   stability guarantees.
2021-03-12 13:12:37 -05:00
Alisdair McDiarmid d9d88b9243 cli: Add comprehensive JSON diagnostic structure
This commit adds a comprehensive JSON format for diagnostics, which
ensures that all current diagnostic output can be semantically
represented in a machine-readable format. The diagnostic formatter
interface remains unchanged, but it first transforms its input via the
JSON format to ensure that there is only one code path for creating the
diagnostic data.

The JSON diagnostic renderer extracts the non-presentational logic from
the format package, and returns a structure which can either be
marshaled into JSON or rendered as text. The resulting text diagnostic
output is unchanged for all cases covered by unit tests and my own
manual testing.

Included in this commit are a number of golden reference files for the
marshaled JSON output of a diagnostic. This format should change rarely
if at all, and these are in place to ensure that any changes to the
format are intentional and considered.
2021-03-12 13:12:37 -05:00
Kristin Laemmert b26ae9cf48
add support for attributes with nested types in providers schema (#28055)
This PR extends jsonprovider to support attributes with NestedTypes and extends test coverage in jsonprovider and the providers schemas tests. I've also cleaned up some comments and extracted the logic to parse the nesting mode so it can be used in both marshalling blocks and attributes.
2021-03-12 08:28:22 -05:00
Pam Selle 242f319638
Give suggestions & remind users to use required_providers when provider not in registry (#28014)
* Add helper suggestion when failed registry err

When someone has a failed registry error on init, remind them that
they should have required_providers in every module

* Give suggestion for a provider based on reqs

Suggest another provider on a registry error, from the list of
requirements we have on init. This skips the legacy lookup
process if there is a similar provider existing in requirements.
2021-03-11 08:54:18 -05:00
Masayuki Morita 31a5aa1878
command/init: Add a new flag `-lockfile=readonly` (#27630)
Fixes #27506

Add a new flag `-lockfile=readonly` to `terraform init`.
It would be useful to allow us to suppress dependency lockfile changes
explicitly.

The type of the `-lockfile` flag is string rather than bool, leaving
room for future extensions to other behavior variants.

The readonly mode suppresses lockfile changes, but should verify
checksums against the information already recorded. It should conflict
with the `-upgrade` flag.

Note: In the original use-case described in #27506, I would like to
suppress adding zh hashes, but a test code here suppresses adding h1
hashes because it's easy for testing.

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2021-03-09 11:12:00 -05:00
Alisdair McDiarmid 86495f93cb views: Fix missing source in diagnostic output
The previous implementation of views was copying and embedding the base
View struct in each individual view. While this allowed for easy access
to the interface of that struct (both in the view and externally), it more
importantly completely broke the ability of the diagnostic printer to
output source code snippets.

This is because the `configSources` field on the base view is lazily set
after the config loader is initialized. In the commands ported to use
views, this happens after the base View struct is copied, so we are
updating the wrong copy of the struct.

This commit fixes this with a simple mechanical refactor: keep a pointer
to the base View struct instead, and update all of the individual views
to explicitly refer to that struct to access its fields and methods.

This is not a particularly satisfying solution, but I can't find
anything clearly better. It might be worth exploring the alternative
approach in the view for the new test command, which explicitly pulls
its dependencies out of the base view, rather than retaining a full
reference. Maybe there's a third way which is better still.
2021-02-26 16:43:03 -05:00
Alisdair McDiarmid 7c0ec0107f backend: Replace ShowDiagnostics with view.Diagnostics 2021-02-25 11:26:05 -05:00
Alisdair McDiarmid 3737ee5081 views: Remove outdated comment 2021-02-25 09:34:32 -05:00
Alisdair McDiarmid 4271b0e11d cli: Remove -var/-var-file flags from validate
These flags are not used and have been deprecated since 0.12.11.
2021-02-24 12:29:56 -05:00
Alisdair McDiarmid 7a03c3153e
Merge pull request #27861 from hashicorp/alisdair/help-global-options
cli: Add reference to global options to help text
2021-02-23 14:19:08 -05:00
James Bardin c103242bef
Merge pull request #27885 from hashicorp/jbardin/show-json
jsonstate: indicate schema version mismatch during encoding
2021-02-23 12:57:06 -05:00
James Bardin 3449a8aa35 don't marshal state with the wrong schema
Instead of returning an error with no context about unexpected
attributes or incorrect types, notify users that the schema stored in
the state does not match the current provider.

User can only encounter this error if the providers have updated their
schemas since the state was stored. This would appears when running
`terraform show -json` to display the current state, or
`terraform show -json planfile` if that plan was created with
`-refresh=false`. In either case, the state must be refreshed in order
to properly json encoded.
2021-02-23 10:19:24 -05:00
Alisdair McDiarmid b7f54b30d5 cli: Fix overly broad auto-approve argument
The auto-approve argument was part of the arguments.Operation type,
which resulted in adding a silent -auto-approve flag to plan and
refresh. This was unintended, and is fixed in this commit by moving the
flag to the arguments.Apply type and updating the downstream callers.
2021-02-23 10:09:30 -05:00
Alisdair McDiarmid 8d9a08e3a1 cli: Migrate plan to command views 2021-02-23 10:09:25 -05:00
Alisdair McDiarmid eeb5dfbf76
Merge pull request #27865 from hashicorp/alisdair/command-views-refresh
cli: Migrate refresh to command views
2021-02-23 09:27:33 -05:00
Martin Atkins 7f78342953 command: Experimental "terraform test" command
This is just a prototype to gather some feedback in our ongoing research
on integration testing of Terraform modules. The hope is that by having a
command integrated into Terraform itself it'll be easier for interested
module authors to give it a try, and also easier for us to iterate quickly
based on feedback without having to coordinate across multiple codebases.

Everything about this is subject to change even in future patch releases.
Since it's a CLI command rather than a configuration language feature it's
not using the language experiments mechanism, but generates a warning
similar to the one language experiments generate in order to be clear that
backward compatibility is not guaranteed.
2021-02-22 14:21:45 -08:00
Alisdair McDiarmid 50d58b592a cli: Migrate refresh to command views 2021-02-22 11:58:52 -05:00
Alisdair McDiarmid 56b756cfd9
Merge pull request #27841 from hashicorp/alisdair/command-views-apply
cli: Migrate apply to command views
2021-02-22 11:53:57 -05:00
Alisdair McDiarmid 6bef157f8e
Merge pull request #27860 from hashicorp/alisdair/remove-noop-plan-backend
cli: Remove no-op plan backend setup
2021-02-22 11:48:58 -05:00
Alisdair McDiarmid 43028c4912 cli: Migrate apply to command views 2021-02-22 11:47:40 -05:00
Kristin Laemmert 7b7273e3aa
Add support for plugin protocol v6 (#27826)
* Add support for plugin protocol v6

This PR turns on support for plugin protocol v6. A provider can
advertise itself as supporting protocol version 6 and terraform will
use the correct client.

Todo:

The "unmanaged" providers functionality does not support protocol
version, so at the moment terraform will continue to assume that
"unmanaged" providers are on protocol v5. This will require some
upstream work on go-plugin (I believe).

I would like to convert the builtin providers to use protocol v6 in a
future PR; however it is not necessary until we remove protocol v6.

* add e2e test for using both plugin protocol versions

- copied grpcwrap and made a version that returns protocol v6 provider
- copied the test provider, provider-simple, and made a version that's
  using protocol v6 with the above fun
- added an e2etest
2021-02-22 10:22:45 -05:00
Alisdair McDiarmid 94340b5940 cli: Add reference to global options to help text 2021-02-22 09:25:56 -05:00
Alisdair McDiarmid b976178018 cli: Remove no-op plan backend setup
This code does not appear to have any effect. The operation request has
its PlanOutBackend field populated directly in the Meta.Operation
method, by calling m.backendForState.ForPlan().

I tested a trivial null-resource config with a Consul backend, and the
saved plans with and without this code present were identical.
2021-02-22 07:43:18 -05:00
Pam Selle 8f7807684a Upgrade to quoted keywords to error
The warning about deprecation is upgraded to an error
2021-02-21 20:27:20 -05:00
Pam Selle fa7c3d7e10 Remove interpolation-only warning
These interpolations are removed when upgrading using 0.12upgrade,
and are removed in terraform fmt in many cases
2021-02-19 10:59:09 -05:00
James Bardin 3da5d2bdf6
Merge pull request #27692 from alexwlchan/abolish-the-heirarchy
Correct the spelling of heirarchy/hierarchy throughout
2021-02-18 16:34:28 -05:00