Commit Graph

28887 Commits

Author SHA1 Message Date
hc-github-team-tf-core d14bbbb6f2
Release v1.1.0-alpha20210922 2021-09-22 17:49:31 +00:00
Martin Atkins 3f1c15c792
Upgrade to Go 1.17.1 2021-09-22 10:31:31 -07:00
Martin Atkins 83f0376673 refactoring: ApplyMoves new return type
When we originally stubbed ApplyMoves we didn't know yet how exactly we'd
be using the result, so we made it a double-indexed map allowing looking
up moves in both directions.

However, in practice we only actually need to look up old addresses by new
addresses, and so this commit first removes the double indexing so that
each move is only represented by one element in the map.

We also need to describe situations where a move was blocked, because in
a future commit we'll generate some warnings in those cases. Therefore
ApplyMoves now returns a MoveResults object which contains both a map of
changes and a map of blocks. The map of blocks isn't used yet as of this
commit, but we'll use it in a later commit to produce warnings within
the "terraform" package.
2021-09-22 09:01:10 -07:00
Martin Atkins d054102d38 addrs: AbsResource.UniqueKey distinct from AbsResourceInstance.UniqueKey
The whole point of UniqueKey is to deal with the fact that we have some
distinct address types which have an identical string representation, but
unfortunately that fact caused us to not notice that we'd incorrectly
made AbsResource.UniqueKey return a no-key instance UniqueKey instead of
its own distinct unique key type.
2021-09-22 09:01:10 -07:00
Chris Arcand 08a86b6c13
Merge pull request #29621 from hashicorp/error-on-unused-test-answers
command: Ensure all answers were used in command.testInputResponseMap
2021-09-22 09:42:21 -05:00
Chris Arcand 8684a85e26 command: Ensure all answers were used in command.testInputResponseMap
Remove answers from testInputResponse as they are given, and raise an
error during cleanup if any answers remain unused.

This enables tests to ensure that the expected mock answers are actually
used in a test; previously, an entire branch of code including an input
sequence could be omitted and the test(s) would not fail.

The only test that had unused answers in this map is one leftover from
legacy state migrations, a prompt that was removed in
7c93b2e5e6
2021-09-21 22:26:16 -05:00
Paddy 4fe75bead3
Remove panic debugging information (#29512)
It was out of date.

Fixes #1113.
2021-09-21 05:45:04 -07:00
Laura Pacilio 8f09e27597
Merge pull request #29567 from drasko95/patch-1
Fix a documentation typo
2021-09-20 17:24:31 -04:00
Alisdair McDiarmid 5386d6c5b8
Merge pull request #29603 from hashicorp/alisdair/moved-json-output
json-output: Add support for config-driven move to plan JSON and streaming JSON UI
2021-09-20 16:49:11 -04:00
Alisdair McDiarmid b59b057591 json-output: Config-driven move support in JSON UI
Add previous address information to the `planned_change` and
`resource_drift` messages for the streaming JSON UI output of plan and
apply operations.

Here we also add a "move" action value to the `change` object of these
messages, to represent a move-only operation.

As part of this work we also simplify this code to use the plan's
DriftedResources values instead of recomputing the drift from state.
2021-09-20 15:25:23 -04:00
Alisdair McDiarmid 78c4a8c461 json-output: Previous address for resource changes
Configuration-driven moves are represented in the plan file by setting
the resource's `PrevRunAddr` to a different value than its `Addr`. For
JSON plan output, we here add a new field to resource changes,
`previous_address`, which is present and non-empty only if the resource
is planned to be moved.

Like the CLI UI, refresh-only plans will include move-only changes in
the resource drift JSON output. In normal plan mode, these are elided to
avoid redundancy with planned changes.
2021-09-20 15:25:23 -04:00
Martin Atkins 78705f4f10
Update CHANGELOG.md 2021-09-20 11:29:24 -07:00
Martin Atkins f0034beb33 core: refactoring.ImpliedMoveStatements replaces NodeCountBoundary
Going back a long time we've had a special magic behavior which tries to
recognize a situation where a module author either added or removed the
"count" argument from a resource that already has instances, and to
silently rename the zeroth or no-key instance so that we don't plan to
destroy and recreate the associated object.

Now we have a more general idea of "move statements", and specifically
the idea of "implied" move statements which replicates the same heuristic
we used to use for this behavior, we can treat this magic renaming rule as
just another "move statement", special only in that Terraform generates it
automatically rather than it being written out explicitly in the
configuration.

In return for wiring that in, we can now remove altogether the
NodeCountBoundary graph node type and its associated graph transformer,
CountBoundaryTransformer. We handle moves as a preprocessing step before
building the plan graph, so we no longer need to include any special nodes
in the graph to deal with that situation.

The test updates here are mainly for the graph builders themselves, to
acknowledge that indeed we're no longer inserting the NodeCountBoundary
vertices. The vertices that NodeCountBoundary previously depended on now
become dependencies of the special "root" vertex, although in many cases
here we don't see that explicitly because of the transitive reduction
algorithm, which notices when there's already an equivalent indirect
dependency chain and removes the redundant edge.

We already have plenty of test coverage for these "count boundary" cases
in the context tests whose names start with TestContext2Plan_count and
TestContext2Apply_resourceCount, all of which continued to pass here
without any modification and so are not visible in the diff. The test
functions particularly relevant to this situation are:
 - TestContext2Plan_countIncreaseFromNotSet
 - TestContext2Plan_countDecreaseToOne
 - TestContext2Plan_countOneIndex
 - TestContext2Apply_countDecreaseToOneCorrupted

The last of those in particular deals with the situation where we have
both a no-key instance _and_ a zero-key instance in the prior state, which
is interesting here because to exercises an intentional interaction
between refactoring.ImpliedMoveStatements and refactoring.ApplyMoves,
where we intentionally generate an implied move statement that produces
a collision and then expect ApplyMoves to deal with it in the same way as
it would deal with all other collisions, and thus ensure we handle both
the explicit and implied collisions in the same way.

This does affect some UI-level tests, because a nice side-effect of this
new treatment of this old feature is that we can now report explicitly
in the UI that we're assigning new addresses to these objects, whereas
before we just said nothing and hoped the user would just guess what had
happened and why they therefore weren't seeing a diff.

The backend/local plan tests actually had a pre-existing bug where they
were using a state with a different instance key than the config called
for but getting away with it because we'd previously silently fix it up.
That's still fixed up, but now done with an explicit mention in the UI
and so I made the state consistent with the configuration here so that the
tests would be able to recognize _real_ differences where present, as
opposed to the errant difference caused by that inconsistency.
2021-09-20 09:06:22 -07:00
Martin Atkins ee9e346039 refactoring: ApplyMoves skips moving when destination address occupied
Per our rule that the content of the state can never make a move statement
invalid, our behavior for two objects trying to occupy the same address
will be to just ignore that and let the object already at the address
take priority.

For the moment this is silent from an end-user perspective and appears
only in our internal logs. However, I'm hoping that our future planned
adjustment to the interface of this function will include some way to
allow reporting these collisions in some end-user-visible way, either as
a separate warning per collision or as a single warning that collects
together all of the collisions into a single message somehow.

This situation can arise both because the previous run state already
contained an object at the target address of a move and because more than
one move ends up trying to target the same location. In the latter case,
which one "wins" is decided by our depth-first traversal order, which is
in turn derived from our chaining and nesting rules and is therefore
arbitrary but deterministic.
2021-09-20 09:06:22 -07:00
Martin Atkins ef5a1c9cfe refactoring: ImpliedMoveStatements function
This new function complements the existing function FindMoveStatements
by potentially generating additional "implied" move statements that aren't
written explicit in the configuration but that we'll infer by comparing
the configuration and te previous run state.

The goal here is to infer only enough to replicate the effect of the
"count boundary fixup" graph node (terraform.NodeCountBoundary) that we
currently use to deal with this concern of preserving the zero-instance
when switching between "count" and not "count".

This is just dead code for now. A subsequent commit will introduce this
into the "terraform" package while also removing
terraform.NodeCountBoundary, thus achieving the same effect as before but
in a way that'll get reported in the UI as a move, using the same language
that we'd use for an explicit move statement.
2021-09-20 09:06:22 -07:00
Martin Atkins 7f99a8802e addrs: MoveEndpointInModule.SelectsResource
This is similar to the existing SelectsModule method, returning true if
the reciever selects either a particular resource as a whole or any of the
instances of that resource.

We don't need this test in the normal case, but we will need it in a
subsequent commit when we'll be possibly generating _implied_ move
statements between instances of resources, but only if there aren't
explicit move statements mentioning those resources already.
2021-09-20 09:06:22 -07:00
Alisdair McDiarmid 2b6a1be18f
Merge pull request #29597 from hashicorp/alisdair/move-refresh-ui
cli: Improved plan UI for move-only changes
2021-09-17 14:54:39 -04:00
Alisdair McDiarmid 638784b195 cli: Omit move-only drift, except for refresh-only
The set of drifted resources now includes move-only changes, where the
object value is identical but a move has been executed. In normal
operation, we previousl displayed these moves twice: once as part of
drift output, and once as part of planned changes.

As of this commit we omit move-only changes from drift display, except
for refresh-only plans. This fixes the redundant output.
2021-09-17 14:47:00 -04:00
Alisdair McDiarmid 61b2d8e3fe core: Plan drift includes move-only changes
Previously, drifted resources included only updates and deletes. To
correctly display the full changes which would result as part of a
refresh-only apply, the drifted resources must also include move-only
changes.
2021-09-17 14:47:00 -04:00
Alisdair McDiarmid 9a7bbdab6f Fix terraform add test failure due to bad merge 2021-09-17 14:46:44 -04:00
Alisdair McDiarmid d425c26d77
Merge pull request #29589 from hashicorp/alisdair/planfile-drifted-resources
core: Compute resource drift during plan phase, store in plan file
2021-09-17 14:23:04 -04:00
Alisdair McDiarmid 1f57b7a8bd
Merge pull request #29235 from magodo/terraform_add_output_append
`terraform add`: `-out` option append to existing config & optionally check resource existance
2021-09-17 11:19:55 -04:00
Laura Pacilio dfbef12a6c
Merge pull request #29598 from hashicorp/laura-add-mrui-to-sidebar
Add Machine-Readable UI to sidebar
2021-09-16 17:30:37 -04:00
Laura Pacilio a8e5b6a4ad Fix alphabetical order of sidebar 2021-09-16 17:11:31 -04:00
Laura Pacilio 4d1baaceab Add Machine-Readable UI to sidebar and add hyphen :-) 2021-09-16 17:06:52 -04:00
James Bardin 1bd5987a8c
Merge pull request #29573 from hashicorp/renamed-typo
Correct terraform.env deprecation message typo
2021-09-16 17:02:20 -04:00
Alisdair McDiarmid d5b5407ccc format: Fix incorrect nesting of Color/Sprintf
Colorizing the result of an interpolated string can result in
incorrect output, if the values used to generate the string happen to
include color codes such as `[red]` or `[bold]`. Instead we should
always colorize the format string before calling functions like
`Sprintf`. This commit fixes all instances in this file.
2021-09-16 15:22:37 -04:00
Alisdair McDiarmid f0cf4235f9 cli: Refactor resource drift rendering 2021-09-16 15:22:37 -04:00
Alisdair McDiarmid c4688345a1 plans: Add resource drift to the plan file format 2021-09-16 15:22:37 -04:00
Alisdair McDiarmid bebf1ad23a core: Compute resource drift after plan walk
Rather than delaying resource drift detection until it is ready to be
presented, here we perform that computation after the plan walk has
completed. The resulting drift is represented like planned resource
changes, using a slice of ResourceInstanceChangeSrc values.
2021-09-16 15:22:37 -04:00
Martin Atkins e6a76d8ba0 core: Fail if a moved resource instance is excluded by -target
Because "moved" blocks produce changes that span across more than one
resource instance address at the same time, we need to take extra care
with them during planning.

The -target option allows for restricting Terraform's attention only to
a subset of resources when planning, as an escape hatch to recover from
bugs and mistakes.

However, we need to avoid any situation where only one "side" of a move
would be considered in a particular plan, because that'd create a new
situation that would be otherwise unreachable and would be difficult to
recover from.

As a compromise then, we'll reject an attempt to create a targeted plan if
the plan involves resolving a pending move and if the source address of
that move is not included in the targets.

Our error message offers the user two possible resolutions: to create an
untargeted plan, thus allowing everything to resolve, or to add additional
-target options to include just the existing resource instances that have
pending moves to resolve.

This compromise recognizes that it is possible -- though hopefully rare --
that a user could potentially both be recovering from a bug or mistake at
the same time as processing a move, if e.g. the bug was fixed by upgrading
a module and the new version includes a new "moved" block. In that edge
case, it might be necessary to just add the one additional address to
the targets rather than removing the targets altogether, if creating a
normal untargeted plan is impossible due to whatever bug they're trying to
recover from.
2021-09-16 08:57:59 -07:00
Martin Atkins b4594551f7 refactoring: TestValidateMoves/cyclic_chain can now pass
When originally filling out these test cases we didn't yet have the logic
in place to detect chained moves and so this test couldn't succeed in
spite of being correct.

We now have chain-detection implemented and so consequently we can also
detect cyclic chains. This commit largely just enables the original test
unchanged, although it does include the text of the final error message
for reporting cyclic move chains which wasn't yet finalized when we were
stubbing out this test case originally.
2021-09-15 08:46:12 -07:00
James Bardin 2afa0a5e75
Merge pull request #29580 from hashicorp/jbardin/proposed-new-empty-containers
handle empty containers in ProposedNew NestedTypes
2021-09-15 09:06:29 -04:00
James Bardin 331dc8b14c handle empty containers in ProposedNew NestedTypes
Empty containers of NestedTypes were not handled in ProposedNew, causing
plans to be submitted with null values where there was configuration
present.
2021-09-15 08:33:25 -04:00
James Bardin 9029870613 handle NestedTypes in Block.CoerceValue
The CoerceValue code was not updated to handle NestedTypes, and while
none of the new codepaths make use of this method, there are still some
internal uses.
2021-09-14 16:02:50 -04:00
Martin Atkins 332ea1f233 backend/local: TestLocal_plan_context_error to fail terraform.NewContext
The original intent of this test was to verify that we properly release
the state lock if terraform.NewContext fails. This was in response to a
bug in an earlier version of Terraform where that wasn't true.

In the recent refactoring that made terraform.NewContext no longer
responsible for provider constraint/checksum verification, this test began
testing a failed plan operation instead, which left the error return path
from terraform.NewContext untested.

An invalid parallelism value is the one remaining case where
terraform.NewContext can return an error, so as a localized fix for this
test I've switched it to just intentionally set an invalid parallelism
value. This is still not ideal because it's still testing an
implementation detail, but I've at least left a comment inline to try to
be clearer about what the goal is here so that we can respond in a more
appropriate way if future changes cause this test to fail again.

In the long run I'd like to move this last remaining check out to be the
responsibility of the CLI layer, with terraform.NewContext either just
assuming the value correct or panicking when it isn't, but the handling
of this CLI option is currently rather awkwardly spread across the
command and backend packages so we'll save that refactoring for a later
date.
2021-09-14 10:35:08 -07:00
Martin Atkins 718fa3895f backend: Remove Operation.Parallelism field
The presence of this field was confusing because in practice the local
backend doesn't use it for anything and the remote backend was using it
only to return an error if it's set to anything other than the default,
under the assumption that it would always match ContextOpts.Parallelism.

The "command" package is the one actually responsible for handling this
option, and it does so by placing it into the partial ContextOpts which it
passes into the backend when preparing for a local operation. To make that
clearer, here we remove Operation.Parallelism and change the few uses of
it to refer to ContextOpts.Parallelism instead, so that everyone is
reading and writing this value from the same place.
2021-09-14 10:35:08 -07:00
Chris Arcand 172200808b Correct terraform.env deprecation message typo 2021-09-13 14:21:26 -05:00
James Bardin d33a423722
Merge pull request #29571 from hashicorp/jbardin/tmp-dir-fixes
fix temp directory handling in some tests
2021-09-13 13:55:47 -04:00
James Bardin d0993b0e80 fix temp directory handling in some tests
Cleanup some more test fixtures to use t.TempDir

Use EvalSymlinks with temp dir paths to help with MacOS errors from
various terraform components.
2021-09-13 13:45:04 -04:00
Laura Pacilio dcf2d3c1ef
Merge pull request #29494 from magnetikonline/docs-s3-backend-dynamodb-partition-key
S3 backend documentation update - DynamoDB uses Partition keys, not primary keys - redux
2021-09-13 10:39:02 -04:00
James Bardin 8ed9a270e5
Merge pull request #29559 from hashicorp/jbardin/optional-attrs
Prevent `ObjectWithOptionalAttrs` from escaping
2021-09-13 08:58:11 -04:00
James Bardin 7f26531d4f variable types should always be populated 2021-09-13 08:51:32 -04:00
James Bardin 53a73a8ab6 configs: add ConstraintType to config.Variable
In order to handle optional attributes, the Variable type needs to keep
track of the type constraint for decoding and conversion, as well as the
concrete type for creating values and type comparison.

Since the Type field is referenced throughout the codebase, and for
future refactoring if the handling of optional attributes changes
significantly, the constraint is now loaded into an entirely new field
called ConstraintType. This prevents types containing
ObjectWithOptionalAttrs from escaping the decode/conversion codepaths
into the rest of the codebase.
2021-09-13 08:51:32 -04:00
James Bardin 43123d284e optional attrs should not be in ProposedNew 2021-09-13 08:51:32 -04:00
James Bardin af4f4540a9 configschema: do not expose optional attributes
Objects with optional attributes are only used for the decoding of
HCL, and those types should never be exposed elsewhere within
terraform.

Separate the external ImpliedType method from the cty.Type generated
internally for the decoder spec. This unfortunately causes our
ImpliedType method to return a different type than the
hcldec.ImpliedType function, but the former is only used within
terraform for concrete values, while the latter is used to decode
HCL. Renaming the ImpliedType methods could be done to further
differentiate them, but that does cause fairly large diff in the
codebase that does not seem worth the effort at this time.
2021-09-13 08:51:32 -04:00
Draško Radovanović c8e2be76d2
Fix a documentation typo 2021-09-11 15:33:14 +02:00
Martin Atkins 65e0c448a0 workdir: Start of a new package for working directory state management
Thus far our various interactions with the bits of state we keep
associated with a working directory have all been implemented directly
inside the "command" package -- often in the huge command.Meta type -- and
not managed collectively via a single component.

There's too many little codepaths reading and writing from the working
directory and data directory to refactor it all in one step, but this is
an attempt at a first step towards a future where everything that reads
and writes from the current working directory would do so via an object
that encapsulates the implementation details and offers a high-level API
to read and write all of these session-persistent settings.

The design here continues our gradual path towards using a dependency
injection style where "package main" is solely responsible for directly
interacting with the OS command line, the OS environment, the OS working
directory, the stdio streams, and the CLI configuration, and then
communicating the resulting information to the rest of Terraform by wiring
together objects. It seems likely that eventually we'll have enough wiring
code in package main to justify a more explicit organization of that code,
but for this commit the new "workdir.Dir" object is just wired directly in
place of its predecessors, without any significant change of code
organization at that top layer.

This first commit focuses on the main files and directories we use to
find provider plugins, because a subsequent commit will lightly reorganize
the separation of concerns for plugin launching with a similar goal of
collecting all of the relevant logic together into one spot.
2021-09-10 14:56:49 -07:00
Martin Atkins 343279110a core: Graph walk loads plugin schemas opportunistically
Previously our graph walker expected to recieve a data structure
containing schemas for all of the provider and provisioner plugins used in
the configuration and state. That made sense back when
terraform.NewContext was responsible for loading all of the schemas before
taking any other action, but it no longer has that responsiblity.

Instead, we'll now make sure that the "contextPlugins" object reaches all
of the locations where we need schema -- many of which already had access
to that object anyway -- and then load the needed schemas just in time.

The contextPlugins object memoizes schema lookups, so we can safely call
it many times with the same provider address or provisioner type name and
know that it'll still only load each distinct plugin once per Context
object.

As of this commit, the Context.Schemas method is now a public interface
only and not used by logic in the "terraform" package at all. However,
that does leave us in a rather tenuous situation of relying on the fact
that all practical users of terraform.Context end up calling "Schemas" at
some point in order to verify that we have all of the expected versions
of plugins. That's a non-obvious implicit dependency, and so in subsequent
commits we'll gradually move all responsibility for verifying plugin
versions into the caller of terraform.NewContext, which'll heal a
long-standing architectural wart whereby the caller is responsible for
installing and locating the plugin executables but not for verifying that
what's installed is conforming to the current configuration and dependency
lock file.
2021-09-10 14:56:49 -07:00
Martin Atkins 38ec730b0e core: Opportunistic schema loading during graph construction
Previously the graph builders all expected to be given a full manifest
of all of the plugin component schemas that they could need during their
analysis work. That made sense when terraform.NewContext would always
proactively load all of the schemas before doing any other work, but we
now have a load-as-needed strategy for schemas.

We'll now have the graph builders use the contextPlugins object they each
already hold to retrieve individual schemas when needed. This avoids the
need to prepare a redundant data structure to pass alongside the
contextPlugins object, and leans on the memoization behavior inside
contextPlugins to preserve the old behavior of loading each provider's
schema only once.
2021-09-10 14:56:49 -07:00