Commit Graph

36 Commits

Author SHA1 Message Date
Alisdair McDiarmid dc2e7520e5 states: Disable Terraform version check
For this version of Terraform and forward, we no longer refuse to read
compatible state files written by future versions of Terraform. This is
a commitment that any changes to the semantics or format of the state
file after this commit will require a new state file version 5.

The result of this is that users of this Terraform version will be able
to share remote state with users of future versions, and all users will
be able to read and write state. This will be true until the next major
state file version is required.

This does not affect users of previous versions of Terraform, which will
continue to refuse to read state written by later versions.
2020-10-28 16:52:35 -04:00
Kristin Laemmert 3933cbd491
remove LegacyProvider (#26433) 2020-10-05 08:33:49 -04:00
Pam Selle 0a02e7040f
Store sensitive attribute paths in state (#26338)
* Add creation test and simplify in-place test

* Add deletion test

* Start adding marking from state

Start storing paths that should be marked
when pulled out of state. Implements deep
copy for attr paths. This commit also includes some
comment noise from investigations, and fixing the diff test

* Fix apply stripping marks

* Expand diff tests

* Basic apply test

* Update comments on equality checks to clarify current understanding

* Add JSON serialization for sensitive paths

We need to serialize a slice of cty.Path values to be used to re-mark
the sensitive values of a resource instance when loading the state file.
Paths consist of a list of steps, each of which may be either getting an
attribute value by name, or indexing into a collection by string or
number.

To serialize these without building a complex parser for a compact
string form, we render a nested array of small objects, like so:

[
  [
    { type: "get_attr", value: "foo" },
    { type: "index", value: { "type": "number", "value": 2 } }
  ]
]

The above example is equivalent to a path `foo[2]`.

* Format diffs with map types

Comparisons need unmarked values to operate on,
so create unmarked values for those operations. Additionally,
change diff to cover map types

* Remove debugging printing

* Fix bug with marking non-sensitive values

When pulling a sensitive value from state,
we were previously using those marks to remark
the planned new value, but that new value
might *not* be sensitive, so let's not do that

* Fix apply test

Apply was not passing the second state
through to the third pass at apply

* Consistency in checking for length of paths vs inspecting into value

* In apply, don't mark with before paths

* AttrPaths test coverage for DeepCopy

* Revert format changes

Reverts format changes in format/diff for this
branch so those changes can be discussed on a separate PR

* Refactor name of AttrPaths to AttrSensitivePaths

* Rename AttributePaths/attributePaths for naming consistency

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2020-09-24 12:40:17 -04:00
Alisdair McDiarmid e02e53ed4d states: Add roundtrip v3->v4 with builtin provider 2020-08-14 15:40:48 -04:00
Kristin Laemmert 5964e928dc
states/statefile: consistently sort resources across modules (#25498)
If a statefile had resources with the same name in different modules,
the sort order could be inconsistent between state refreshes. This adds
the module to the Less() function used in sorting and a minimal test to
verify consistent ordering.
2020-08-11 11:50:07 -04:00
Kristin Laemmert 58bcc2e9bb
addrs: detect builtin provider when parsing legacy provider string (#25154)
* addrs: detect builtin provider when parsing legacy provider string

The ParseLegacyAbsProviderConfig was not detecting builtin providers
("terraform"), which caused issues for all users with 0.12 state and
the "terraform_remote_state" data source. Since "terraform" is the only
built-in provider this adds a very simple check to the parser so it
properly returns the builtin FQN.

* add tests to the addrs package
2020-06-08 08:27:36 -04:00
James Bardin 08128fe775 add omitempty to create_before_destroy
no need to litter the state file with "create_before_destroy":false
2020-05-18 11:31:36 -04:00
James Bardin 9fe87fe520 remove unused depends_on field and add test
The depends_on fields was deprecated during 0.12, and is no longer used.

Add a roundtrip test for the create_before_destroy status.
2020-05-14 15:46:08 -04:00
James Bardin 23f5fc3e5a thread create_before_destroy to the state file 2020-05-14 15:46:08 -04:00
James Bardin f915c5d957 remove EachMode from resource state
Due to the fact that resources can transition between each modes, trying
to track the mode for a resource as a whole in state doesn't work,
because there may be instances with a mode different from the resource
as a whole. This is difficult for core to track, as this metadata being
changed as a side effect from multiple places often causes core to see
the incorrect mode when evaluating instances.

Since core can always determine the correct mode to evaluate from the
configuration, we don't need to interrogate the state to know the mode.
Once core no longer needs to reference EachMode from states, the
resource state can simply be a container for instances, and doesn't need
to try and track the "current" mode.
2020-04-30 09:22:14 -04:00
James Bardin e9eb8e04cc add AbsOutputAddrs to state outputs
We need all module instance outputs to build the objects for evaluation,
but there is no need to copy all the resource instances along with that.
This allows us to only return the output states, with enough information
to connect them with their module instances.
2020-04-13 16:37:59 -04:00
Kristin Laemmert c4873778c8 Mildwonkey/tests (#24548)
* helper/resource: remove provider resolver test
* repl tests passing
* helper/resource: add some extra shimming to ShimLegacyState

Some of the tests in helper/resource have to shim between legacy and
modern states. Terraform expects modern states to have the Provider set
for each resource (and not be a legacy provider). This PR adds a wrapper
around terraform.ShimLegacyState which iterates through the resources
in the state (after the shim) and adds the provider FQN.
2020-04-06 09:24:23 -07:00
Kristin Laemmert d95667f264 states/version3 upgrade: revert provider FQN changes (#24504)
This commit reverts an earlier change which automatically converted
provider strings to legacy provider FQNs. It has become apparent that a
state upgrade step will be required before upgrading to v0.13.
2020-04-06 09:24:23 -07:00
James Bardin b3fc0dab94 use addrs.ConfigResource for dependency tracking
We can't get module instances during transformation, so we need to
reduce the Dependencies to using `addrs.ConfigResource` for now.
2020-03-25 17:03:06 -04:00
Kristin Laemmert ed1aebbeda
terraform: large refactor to use Provider from configs.Resource (#24396)
* terraform: large refactor to use Provider from configs.Resource

configs.Resource.ImpliedProvider() now returns a string; it is the
callers' responsibility to turn that into an addrs.Provider if needed.

GraphNodeProviderConsumer ProvidedBy() no longer returns nil (reverting
to earlier, pre-provider-fqn behavior): it will return either the
provider set in config, provider set in state, or the default provider.
2020-03-18 08:58:20 -04:00
James Bardin ea51b790bc states.Resource needs to record its module too
The ModuleInstance is known while building the state resource, but it's
not recorded. Since a resource may be retrieved via a ConfigResource
address, we need to know from which module instance it was loaded.
2020-03-16 11:16:23 -04:00
James Bardin 8497adcb6e AbsProviderConfig to use addrs.Module
Change ModuleInstance to Module in AbsProviderConfig, because providers
need to be handled before module expansion, and should not be used
defined inside an expanded module at all.

Renaming of the addrs type can happen later, when there's less work
in-flight around provider configuration.
2020-03-10 20:25:44 -04:00
Kristin Laemmert 47a16b0937
addrs: embed Provider in AbsProviderConfig instead of Type
a large refactor to addrs.AbsProviderConfig, embedding the addrs.Provider instead of a Type string. I've added and updated tests, added some Legacy functions to support older state formats and shims, and added a normalization step when reading v4 (current) state files (not the added tests under states/statefile/roundtrip which work with both current and legacy-style AbsProviderConfig strings).

The remaining 'fixme' and 'todo' comments are mostly going to be addressed in a subsequent PR and involve looking up a given local provider config's FQN. This is fine for now as we are only working with default assumption.
2020-02-13 15:32:58 -05:00
Martin Atkins 8b511524d6
Initial steps towards AbsProviderConfig/LocalProviderConfig separation (#23978)
* Introduce "Local" terminology for non-absolute provider config addresses

In a future change AbsProviderConfig and LocalProviderConfig are going to
become two entirely distinct types, rather than Abs embedding Local as
written here. This naming change is in preparation for that subsequent
work, which will also include introducing a new "ProviderConfig" type
that is an interface that AbsProviderConfig and LocalProviderConfig both
implement.

This is intended to be largely just a naming change to get started, so
we can deal with all of the messy renaming. However, this did also require
a slight change in modeling where the Resource.DefaultProviderConfig
method has become Resource.DefaultProvider returning a Provider address
directly, because this method doesn't have enough information to construct
a true and accurate LocalProviderConfig -- it would need to refer to the
configuration to know what this module is calling the provider it has
selected.

In order to leave a trail to follow for subsequent work, all of the
changes here are intended to ensure that remaining work will become
obvious via compile-time errors when all of the following changes happen:
- The concept of "legacy" provider addresses is removed from the addrs
  package, including removing addrs.NewLegacyProvider and
  addrs.Provider.LegacyString.
- addrs.AbsProviderConfig stops having addrs.LocalProviderConfig embedded
  in it and has an addrs.Provider and a string alias directly instead.
- The provider-schema-handling parts of Terraform core are updated to
  work with addrs.Provider to identify providers, rather than legacy
  strings.

In particular, there are still several codepaths here making legacy
provider address assumptions (in order to limit the scope of this change)
but I've made sure each one is doing something that relies on at least
one of the above changes not having been made yet.

* addrs: ProviderConfig interface

In a (very) few special situations in the main "terraform" package we need
to make runtime decisions about whether a provider config is absolute
or local.

We currently do that by exploiting the fact that AbsProviderConfig has
LocalProviderConfig nested inside of it and so in the local case we can
just ignore the wrapping AbsProviderConfig and use the embedded value.

In a future change we'll be moving away from that embedding and making
these two types distinct in order to represent that mapping between them
requires consulting a lookup table in the configuration, and so here we
introduce a new interface type ProviderConfig that can represent either
AbsProviderConfig or LocalProviderConfig decided dynamically at runtime.

This also includes the Config.ResolveAbsProviderAddr method that will
eventually be responsible for that local-to-absolute translation, so
that callers with access to the configuration can normalize to an
addrs.AbsProviderConfig given a non-nil addrs.ProviderConfig. That's
currently unused because existing callers are still relying on the
simplistic structural transform, but we'll switch them over in a later
commit.

* rename LocalType to LocalName

Co-authored-by: Kristin Laemmert <mildwonkey@users.noreply.github.com>
2020-01-31 08:23:07 -05:00
Kristin Laemmert 67fc4dd5a1 configs: move ProviderConfigCompact[Str] from addrs to configs
The configs package is aware of provider name and type (which are the
same thing today, but expected to be two different things in a future
release), and should be the source of truth for a provider config
address.
2019-12-09 08:30:08 -05:00
Martin Atkins 7cf782c1bf states/statefile: Better error messages for some unsupported v3 cases
There are a few situations that we've seen arise quite commonly for folks
upgrading from Terraform 0.11 to 0.12. These particular problems are not
things that Terraform 0.12 can fix automatically during upgrading, but
we can at least give some better feedback to users that they ought to be
addressed _before_ upgrading.

The provider address problem is already detected and flagged by the
"terraform 0.11checklist" command that folks should run as part of their
upgrade process, but the module address problem is not something we
noticed was lacking validation in 0.11 and so the checklist tool doesn't
cover it. Due to the lack of coverage in the checklist tool, this commit
also includes an additional section in the upgrade guide that mentions
the problem and gives instructions on how to address it.
2019-11-21 09:18:49 -08:00
Martin Atkins 1ae47ae314 states/statefile: tolerate and ignore invalid depends_on in version 3
The state refactoring command "terraform state mv" in Terraform 0.11 does
not update existing dependency addresses recorded in the state when it
moves objects around, and Terraform only updates the dependency addresses
in the state when it performs a full update on a resource instance, and
so it's a common problem for folks updating from Terraform 0.11 with
resource names that are not valid identifiers to run into state upgrade
errors even though they have followed the instructions produced by
"terraform 0.12checklist".

Dependencies are synced from config during every refresh walk anyway, so
in practice we can get away with just discarding invalid dependency
addresses and letting the refresh walk update them. In practice these
addresses are unlikely to be pointing at a resource that actually exists
anyway, because if so Terraform 0.12's configuration parser wouldn't be
able to interpret it.

Discarding invalid dependency addresses allows the state upgrade to
complete successfully in such cases and thus gives the refresh step an
opportunity to repair the problem.
2019-11-21 09:18:49 -08:00
James Bardin 5a0a0020a0 read+write the new dependencies in the statefile
The test fixture did not like having modules when using the generic json
map, so read and compare the states in the final *File datastructure.
2019-11-07 17:49:03 -05:00
Radek Simko 95cbf1c282
Fix copy-paste errors in test data 2019-08-21 11:45:06 +01:00
Martin Atkins d41fc8d517 states/statefile: additional context for some v3 upgrade error messages
Some of our errors returned here were lacking context about what part of
the file was problematic, which led to some useless error reporting for
some real-world situations that this upgrade process doesn't seem to be
catching.

Here we add additional context to those error cases, as a step towards
tracking down exactly which upgrade cases are missing here so that we can
potentially fix them in a subsequent release.
2019-07-18 18:20:53 -07:00
Masayuki Morita ac83827c90 states/statefile: decode backend hash as uint64
Fixes #21478

In #19688, `terraform.BackendState.Hash` was fixed.
We also need to fix `states/statefile.backendStateV2.Hash`
2019-05-28 23:59:36 +09:00
James Bardin 1e1dcfd95a always write the current state terraform_version
Any time a new state is written, the current terraform version should be
recorded.
2019-05-21 18:55:53 -04:00
Kristin Laemmert d7dda4e436
states/statefile: upgrade legacy dependency syntax (#21159)
We used to allow "foo.1" etc as a reference, but now it's "foo[1]".
2019-04-30 14:34:01 -04:00
Sander van Harmelen 634430ebb2 Fix wildcard dependencies when upgrading states
Fixes #19347
2018-11-15 13:17:15 +01:00
Martin Atkins 9ba399bca8 command: Fix TestInit_getProvider
After all of the refactoring we were no longer checking the Terraform
version field in a state file, causing this test to fail.

This restores that check, though with a slightly different error message.
2018-11-12 15:19:55 -08:00
Martin Atkins 331cb07a05 states/statefile: Tolerate nil state in statefile.New
For historical reasons sometimes we have nil state in situations where
we'd still like to persist state snapshots to a store. To make life easier
for those callers, we'll substitute an empty state if we are given a nil
one, thus allowing us to still generate a valid serialization that will
load back in as an empty state.
2018-10-16 19:14:11 -07:00
Martin Atkins b0016e9cf6 command: Allow tests to run to completion without panics or hangs
There are still 160 test failures as of this commit, but at least the test
program can run to completion and list out all the failures.
2018-10-16 19:14:11 -07:00
Martin Atkins fb57801dfe states: object Private data is []byte, not cty.Value
We're going to allow the provider to encode whatever it wants in here, so
a provider can use whatever is most convenient for its implementation
language and to avoid some of the bugs we saw with the prior model where
the forced round-trip through JSON and back into interface{} would cause
some loss of fidelity, leading to bugs.
2018-10-16 19:11:09 -07:00
Martin Atkins a3403f2766 terraform: Ugly huge change to weave in new State and Plan types
Due to how often the state and plan types are referenced throughout
Terraform, there isn't a great way to switch them out gradually. As a
consequence, this huge commit gets us from the old world to a _compilable_
new world, but still has a large number of known test failures due to
key functionality being stubbed out.

The stubs here are for anything that interacts with providers, since we
now need to do the follow-up work to similarly replace the old
terraform.ResourceProvider interface with its replacement in the new
"providers" package. That work, along with work to fix the remaining
failing tests, will follow in subsequent commits.

The aim here was to replace all references to terraform.State and its
downstream types with states.State, terraform.Plan with plans.Plan,
state.State with statemgr.State, and switch to the new implementations of
the state and plan file formats. However, due to the number of times those
types are used, this also ended up affecting numerous other parts of core
such as terraform.Hook, the backend.Backend interface, and most of the CLI
commands.

Just as with 5861dbf3fc49b19587a31816eb06f511ab861bb4 before, I apologize
in advance to the person who inevitably just found this huge commit while
spelunking through the commit history.
2018-10-16 19:11:09 -07:00
Martin Atkins 424afe0ace states: separate types for encoded and decoded state objects
The types here were originally written to allow us to defer decoding of
object values until schemas are available, but it turns out that this was
forcing us to defer decoding longer than necessary and potentially decode
the same value multiple times.

To avoid this, we create pairs of types to represent the encoded and
decoded versions and methods for moving between them. These types are
identical to one another apart from how the dynamic values are
represented.
2018-10-16 18:58:49 -07:00
Martin Atkins 5c1c6e9d9c statefile: New package for loading and saving state files
Whereas the parent directory "states" contains the models that represent
state in memory, this package's responsibility is in serializing a subset
of that data to a JSON-based file format and then reloading that data
back into memory later.

For reading, this package supports state file formats going back to
version 1, using lightly-adapted versions of the migration code previously
used in the "terraform" package. State data is upgraded to the latest
version step by step and then transformed into the in-memory state
representation, which is distinct from any of the file format structs in
this package to enable these to evolve separately.

For writing, only the latest version (4) is supported, which is a new
format that is a slightly-flattened version of the new in-memory state
models introduced in the prior commit. This format retains the outputs
from only the root module and it flattens out the module and instance
parts of the hierarchy by including the identifiers for these inside
the child object. The loader then reconstructs the multi-layer structure
we use for more convenient access in memory.

For now, the only testing in this package is of round-tripping different
versions of state through a read and a write, ensuring the output is
as desired. This exercises all of the reading, upgrading, and writing
functions but should be augmented in later commits to improve coverage
and introduce more focused tests for specific parts of the functionality.
2018-10-16 18:49:20 -07:00