Commit Graph

125 Commits

Author SHA1 Message Date
James Bardin ef88c54604
Merge pull request #28687 from hashicorp/jbardin/sensitive-changes
Decode change values with marks
2021-05-13 12:44:03 -04:00
Martin Atkins 3c8a4e6e05 command+backend/local: -refresh-only and drift detection
This is a light revamp of our plan output to make use of Terraform core's
new ability to report both the previous run state and the refreshed state,
allowing us to explicitly report changes made outside of Terraform.

Because whether a plan has "changes" or not is no longer such a
straightforward matter, this now merges views.Operation.Plan with
views.Operation.PlanNoChanges to produce a single function that knows how
to report all of the various permutations. This was also an opportunity
to fill some holes in our previous logic which caused it to produce some
confusing messages, including a new tailored message for when
"terraform destroy" detects that nothing needs to be destroyed.

This also allows users to request the refresh-only planning mode using a
new -refresh-only command line option. In that case, Terraform _only_
performs drift detection, and so applying a refresh-only plan only
involves writing a new state snapshot, without changing any real
infrastructure objects.
2021-05-13 09:05:06 -07:00
James Bardin 213f3391c3 NormalizeObjectFromLegacySDK should handle marks
Make sure that this function can handle any unexpectedly marked values.
The only remaining caller of this function is in the diff formatter,
which uses it to suppress meaningless diffs created by legacy providers.
2021-05-12 09:36:25 -04:00
James Bardin e29eb78cfd decode change values with marks
Marks stored in a plans.ChangeSrc were not decoded along with the
stored values. This was working in many cases by evaluation correctly
re-evaluating the marks, but this cannot happen in all cases.
2021-05-11 17:37:07 -04:00
Martin Atkins 0f936b9d80 plans/planfile: Read state snapshots as part of reading a plan
Our model for plans/planfile has unfortunately grown inconsistent with
changes to our modeling of plans.Plan.

Originally we considered the plan "header" and the planned changes as an
entirely separate artifact from the prior state, but we later realized
that carrying the prior state around with the plan is important to
ensuring we always have enough context to faithfully render a plan to the
user, and so we added the prior state as a field of plans.Plan.
More recently we've also added the "previous run state" to plans.Plan for
similar reasons.

Unfortunately as a result of that modeling drift our ReadPlan method was
silently producing an incomplete plans.Plan object, causing use-cases like
"terraform show" to produce slightly different results due to the
plan object not round-tripping completely.

As a short-term tactical fix, here we add state snapshot reading into the
ReadPlan function. This is not an ideal solution because it means that
in the case of applying a plan, where we really do need access to the
state _file_, we'll end up reading the prior state file twice. However,
the goal here is only to heal the modelling quirk with as little change
as possible, because we're not currently at a point where we'd be willing
to risk regressions from a larger refactoring.
2021-05-10 09:22:47 -07:00
Martin Atkins 7c6e78bcb0 plans: Track both the previous run and prior states in the plan
Until now we've not really cared much about the state snapshot produced
by the previous Terraform operation, except to use it as a jumping-off
point for our refresh step.

However, we'd like to be able to report to an end-user whenever Terraform
detects a change that occurred outside of Terraform, because that's often
helpful context for understanding why a plan contains changes that don't
seem to have corresponding changes in the configuration.

As part of reporting that we'll need to keep track of the state as it
was before we did any refreshing work, so we can then compare that against
the state after refreshing. To retain enough data to achieve that, the
existing Plan field State is now two fields: PrevRunState and PriorState.

This also includes a very shallow change in the core package to make it
populate something somewhat-reasonable into this field so that integration
tests can function reasonably. However, this shallow implementation isn't
really sufficient for real-world use of PrevRunState because we'll
actually need to update PrevRunState as part of planning in order to
incorporate the results of any provider-specific state upgrades to make
the PrevRunState objects compatible with the current provider schema, or
else our diffs won't be valid. This deeper awareness of PrevRunState in
Terraform Core will follow in a subsequent commit, prior to anything else
making use of Plan.PrevRunState.
2021-05-05 15:11:05 -07:00
Martin Atkins 06adc69e2c plans: Plan.Mode is now Plan.UIMode
This is to make it more obvious at all uses of this field that it's not
something to be used for anything other than UI decisions, hopefully
prompting a reader of code elsewhere to refer to the comments to
understand why it has this unusual prefix and thus see what its intended
purpose is.
2021-04-30 10:30:56 -07:00
Martin Atkins b37b1beddd core: Minimal initial implementation of -replace=... option
This only includes the internal mechanisms to make it work, and not any
of the necessary UI changes to "terraform plan" and "terraform apply" to
activate it yet.

The force-replace options are ultimately handled inside the
NodeAbstractResourceInstance.plan method, at the same place we handle the
similar situation of the provider indicating that replacement is needed,
and so the rest of the changes here are just to propagate the settings
through all of the layers in order to reach that point.
2021-04-30 10:30:56 -07:00
Martin Atkins 1b464e1e9a core: Minimal initial implementation of "refresh only" planning mode
This only includes the core mechanisms to make it work. There's not yet
any way to turn this mode on as an end-user, because we have to do some
more work at the UI layer to present this well before we could include it
as an end-user-visible feature in a release.

At the lowest level of abstraction inside the graph nodes themselves, this
effectively mirrors the existing option to disable refreshing with a new
option to disable change-planning, so that either "half" of the process
can be disabled. As far as the nodes are concerned it would be possible
in principle to disable _both_, but the higher-level representation of
these modes prevents that combination from reaching Terraform Core in
practice, because we block using -refresh-only and -refresh=false at the
same time.
2021-04-30 10:30:56 -07:00
Martin Atkins b802237e03 plans: Track an optional extra "reason" for some planned actions
Previously we were repeating some logic in the UI layer in order to
recover relevant additional context about a change to report to a user.
In order to help keep things consistent, and to have a clearer path for
adding more such things in the future, here we capture this user-facing
idea of an "action reason" within the plan model, and then use that
directly in order to decide how to describe the change to the user.

For the moment the "tainted" situation is the only one that gets a special
message, matching what we had before, but we can expand on this in future
in order to give better feedback about the other replace situations too.

This also preemptively includes the "replacing by request" reason, which
is currently not reachable but will be used in the near future as part of
implementing the -replace=... plan command line option to allow forcing
a particular object to be replaced.

So far we don't have any special reasons for anything other than replacing,
which makes sense because replacing is the only one that is in a sense
a special case of another action (Update), but this could expand to
other kinds of reasons in the future, such as explaining which of the
few different reasons a data source read might be deferred until the
apply step.
2021-04-29 17:50:46 -07:00
Martin Atkins c6a7d080d9 core: Generalize the idea of a "plan mode", vs just destroy flag
Previously there were only two planning modes: normal mode and destroy
mode. In that context it made sense for these to be distinguished only by
a boolean flag.

We're now getting ready to add our third mode, "refresh only". This
establishes the idea that planning can be done in one of a number of
mutually-exclusive "modes", which are related to but separate from the
various other options that serve as modifiers for the plan operation.

This commit only introduces the new plans.Mode type and replaces the
existing "destroy" flag with a variable of that type. This doesn't cause
any change in effective behavior because Terraform Core still supports
only NormalMode and DestroyMode, with NewContext rejecting an attempt to
create a RefreshMode context for now.

It is in retrospect a little odd that the "destroy" flag was part of
ContextOpts rather than just an argument to the Plan method, but
refactoring that would be too invasive a change for right now so we'll
leave this as a field of the context for now and save revisiting that for
another day.
2021-04-27 08:23:54 -07:00
James Bardin 59c8281378 use new dynamic decoding behavior
Dynamic blocks with unknown for_each expressions are now decoded into an
unknown value rather than using a sentinel object with unknown
and null attributes. This will allow providers to precisely plan the
block values, rather than trying to heuristically paper over the
incorrect plans when dynamic is in use.
2021-04-14 12:13:22 -04:00
Alisdair McDiarmid a12c413b84 plans/planfile: Add required-replace and sensitive
The stored planfile now serializes the required-replace path set and the
collection of before/after sensitivity marks. This ensures that storing
a plan and displaying it with `terraform show` renders the same output
for plans with required-replace resources, and those with sensitive
values in the diff.
2021-03-25 14:42:34 -04:00
Alisdair McDiarmid a43b035a51 core: Reduce string allocations for addrs Equal
Generating strings and comparing them to implement Equal is a quick and
easy solution. Unfortunately when this code is in the hot path, it
becomes very expensive, so this commit changes some of those instances
to compare the values directly.

Combined with using addr.Equal instead of checking for string equality,
this makes Terraform dramatically faster for some operations, such as
generating large JSON plans.
2021-03-05 13:23:32 -05:00
James Bardin cd7fb9bd5a catch invalidly planned attributes earlier
Catch attributes which are planed but not computed separately to provide
a clearer error to provider developers.

The error conditions were previously caught, however it was unclear from
the error text as to _why_ the change was an error. The statements about
value inequality would be incorrect when planning no changes for a value
which should not have been set in the first place.
2021-02-24 12:13:12 -05:00
James Bardin 0d63b3ec24
Merge pull request #27791 from hashicorp/jbardin/test-conformance-dynamic
reverse call to TestConformance in objchange
2021-02-16 15:42:12 -05:00
James Bardin 22f21db229 reverse call to TestConformance in objchange
The call to TestConformance needs to be reversed, since we want to
verify that the actual value returned conforms to the planned type.
While the inverse (checking that the planned value conforms to the
applied type) works for everything terraform has been exposed to up
until now, this fails when the planned type has dynamic attributes which
are allowed to become concrete types.
2021-02-16 12:55:02 -05:00
Kristin Laemmert 8c2abbc0f0 return the properly-typed nulls, instead of empty containers, in proposedNewNestedType 2021-02-12 13:37:45 -05:00
Kristin Laemmert 77af601543 plans/objchange: extended ProposedNewObject to descend into attributes
with NestedType objects.

There are a handful of mostly cosmetic changes in this PR which likely
make the diff awkward to read; I renamed several functions to
(hopefully) clarifiy which funcs worked with Blocks vs other types. I
also extracted some small code snippets into their own functions for
reusability.

The code that descends into attributes with NestedTypes is similar to
the block-handling code, and differs in all the ways blocks and
attributes differ: null is valid for attributes, unlike blocks which can
only be present or empty.
2021-02-10 09:58:56 -05:00
Kristin Laemmert da6ac9d6cd plans/objchange: add handling of NestedTypes inside attributes
- rename ProposedNewObject to ProposedNew:
Now that there is an actual configschema.Object it will be clearer if
the function names match the type the act upon.

- extract attribute-handling logic from assertPlanValid and extend
A new function, assertPlannedAttrsValid, takes the existing
functionality and extends it to validate attributes with NestedTypes.
The NestedType-specific handling is in assertPlannedObjectValid, which
is very similar to the block-handling logic, except that nulls are a
valid plan (an attribute can be null, but not a block).
2021-02-05 13:41:06 -05:00
Pam Selle e6daf3dbf1 Unmark before ElementIterator in couldHaveUnknownBlockPlaceholder
This is needed for cases where a variable may be fetched and become
a member of a set, and thus the whole set is marked, which means
ElementIterator will panic on unmarked values
2021-01-29 17:06:12 -05:00
James Bardin a5f013830f plans: staticcheck 2020-12-02 13:59:19 -05:00
James Bardin fa3e534142 plans/planfile: staticcheck 2020-12-02 13:59:19 -05:00
James Bardin 913f6c2091 regenerate the plans proto with current tooling 2020-12-01 14:07:15 -05:00
James Bardin 5f4ff0e8be don't render plan for module outputs
Module outputs should not trigger plan rendering.
2020-11-17 09:31:18 -05:00
James Bardin ef086399f9 compare empty strings as null in sets
The Legacy SDK cannot handle missing strings from objects in sets, and
will insert an empty string when planning the missing value. This
subverts the `couldHaveUnknownBlockPlaceholder` check, and causes
errors when `dynamic` is used with NestingSet blocks.

We don't have a separate codepath to handle the internals of
AssertObjectCompatible differently for the legacy SDK, but we can treat
empty strings as null strings within set objects to avoid the failed
assertions.
2020-10-19 18:07:45 -04:00
James Bardin 77af322c1c handle non-null, but empty NestingMap in a set 2020-10-15 21:21:14 -04:00
James Bardin b59c64245b refactor ifs to reduce indentation 2020-10-15 20:55:56 -04:00
James Bardin f128b8c4fa take dynamic types into account when comparing set
If a NestingList or NestingMap contains a dynamic type, they must be
handled as a cty.Tuple and cty.Object respectively, because the elements
may not have precisely matching types.
2020-10-15 20:07:00 -04:00
Alexander Ovechkin d7db008df2 added empty list test case 2020-10-15 19:21:41 -04:00
Alexander Ovechkin 8fbb4d0163 Converting ListVal to ListVal instead of TupleVal in setElementCompareValue 2020-10-15 19:21:41 -04:00
Pam Selle fcae49611c
Merge pull request #26555 from hashicorp/pselle/sensitive-var-value-compat
Avoid disclosing values in errors on marked vals
2020-10-13 10:51:25 -04:00
James Bardin 5eca0788c6 rely solely on the plan changes for outputs
Now that outputs changes are tracked in full, we can remove the
comparisons with the prior state and use the planned changes directly.
2020-10-12 18:59:14 -04:00
Pam Selle da4ddd0160 Avoid disclosing values in errors on marked vals
AssertObjectCompatible is a special case that will
expose Go string values of values unless otherwise
stopped. This adds that check.
2020-10-12 15:53:34 -04:00
James Bardin c48af3f18b
Merge pull request #26470 from hashicorp/jbardin/inverse-destroy-references
Allow special-case evaluation of instances pending deletion.
2020-10-05 16:20:22 -04:00
James Bardin 0c72c6f144 s/FullDestroy/IsFullDdestroy/ 2020-10-05 10:50:25 -04:00
Kristin Laemmert 3933cbd491
remove LegacyProvider (#26433) 2020-10-05 08:33:49 -04:00
Pam Selle f35b530837 Update compatibility checks for blocks to not use marks
Remove marks for object compatibility tests to allow apply
to continue. Adds a block to the test provider to use
in testing, and extends the sensitivity apply test to include a block
2020-10-02 13:11:55 -04:00
James Bardin fa8f8df7b6 add ChangesSync.FullDestroy
In order to handle various edge cases during a full destroy, add
FullDestroy to the synchronized changes so we can attempt to deduce if
the plan was created from `terraform destroy`.

It's possible that the plan was created by removing all resourced from
the configuration, but in that case the end result is the same. Any of
the edge cases with provider or destroy provisioner configurations would
not apply, since there would not be any configuration references to
resolve.
2020-10-01 17:08:25 -04:00
Pam Selle 0b3c21a3eb Support lists of deeply marked values 2020-09-25 13:33:44 -04:00
Pam Selle 3dde9efc75 Support list diffs with sensitivity
Adds support for specialized diffs with lists
2020-09-25 10:18:33 -04:00
James Bardin 8cef62e455 add state to plans.Plan
Since the refreshed state is now an artifact of the plan process, it
makes sense to add it to the Plan type, rather than adding an additional
return value to the Context.Plan method.
2020-09-17 09:54:59 -04:00
Pam Selle 20ee878d0e Updates and improvements to comments 2020-09-11 11:15:44 -04:00
Pam Selle 5b0b1a13a5 Update object compatible check to unmark
The hack approach appears consistent,
as we can remove marks before calling the
value validation
2020-09-10 11:04:17 -04:00
Pam Selle bc55b6a28b Use UnmarkDeepWithPaths and MarkWithPaths
Updates existing code to use the new Value
methods for unmarking/marking and removes
panics/workarounds in cty marshall methods
2020-09-10 11:04:17 -04:00
Pam Selle 6c129a921b Unmark/remark in apply process to allow apply 2020-09-10 11:04:17 -04:00
Pam Selle 84d118e18f Track sensitivity through evaluation
Mark sensitivity on a value. However, when the value is encoded to send to the
provider to produce a changeset we must remove the marks, so unmark the value
and remark it with the saved path afterwards
2020-09-10 11:04:17 -04:00
James Bardin 2b4101fdff Unknown set blocks with dynamic may have 0 elems
The couldHaveUnknownBlockPlaceholder helper was added to detect when a
set block has a placeholder for an unknown number of values. This worked
fine when the number increased from 1, but we were still attempting to
validate the unknown placeholder against the empty set when the final
count turned out to be 0.

Since we can't differentiate the unknown dynamic placeholder value from
an actual set value, we have to skip that object's validation
altogether.
2020-07-23 15:47:34 -04:00
Chris Stephens 2dd64a7816
plans: Update error message for apply validation (#21312)
* Update error message for apply validation

Add a hint that the validation failure has occurred at the root of the resource
schema to the error message. This is because the root resource has an empty
path when being validated and the path is being relied upon to provide context
into the error message.
2020-06-05 15:08:10 -04:00
Martin Atkins 31a4b44d2e backend/local: treat output changes as side-effects to be applied
This is a baby-step towards an intended future where all Terraform actions
which have side-effects in either remote objects or the Terraform state
can go through the plan+apply workflow.

This initial change is focused only on allowing plan+apply for changes to
root module output values, so that these can be written into a new state
snapshot (for consumption by terraform_remote_state elsewhere) without
having to go outside of the primary workflow by running
"terraform refresh".

This is also better than "terraform refresh" because it gives an
opportunity to review the proposed changes before applying them, as we're
accustomed to with resource changes.

The downside here is that Terraform Core was not designed to produce
accurate changesets for root module outputs. Although we added a place for
it in the plan model in Terraform 0.12, Terraform Core currently produces
inaccurate changesets there which don't properly track the prior values.

We're planning to rework Terraform Core's evaluation approach in a
forthcoming release so it would itself be able to distinguish between the
prior state and the planned new state to produce an accurate changeset,
but this commit introduces a temporary stop-gap solution of implementing
the logic up in the local backend code, where we can freeze a snapshot of
the prior state before we take any other actions and then use that to
produce an accurate output changeset to decide whether the plan has
externally-visible side-effects and render any changes to output values.

This temporary approach should be replaced by a more appropriately-placed
solution in Terraform Core in a release, which should then allow further
behaviors in similar vein, such as user-visible drift detection for
resource instances.
2020-05-29 07:36:40 -07:00