Commit Graph

129 Commits

Author SHA1 Message Date
Alisdair McDiarmid dc2e7520e5 states: Disable Terraform version check
For this version of Terraform and forward, we no longer refuse to read
compatible state files written by future versions of Terraform. This is
a commitment that any changes to the semantics or format of the state
file after this commit will require a new state file version 5.

The result of this is that users of this Terraform version will be able
to share remote state with users of future versions, and all users will
be able to read and write state. This will be true until the next major
state file version is required.

This does not affect users of previous versions of Terraform, which will
continue to refuse to read state written by later versions.
2020-10-28 16:52:35 -04:00
James Bardin 0b31ffa587 use a single log writer
Use a single log writer instance for all std library logging.

Setup the std log writer in the logging package, and remove boilerplate
from test packages.
2020-10-19 14:29:54 -04:00
James Bardin 6ca477f042 move helper/logging to internal
remove a dead code file too
2020-10-19 14:27:53 -04:00
Pam Selle 923f9f226d Update some comments in states package 2020-10-07 10:57:25 -04:00
Kristin Laemmert 3933cbd491
remove LegacyProvider (#26433) 2020-10-05 08:33:49 -04:00
Pam Selle 3e7be13dff Update ordering for marking/unmarking and asserting plan valid
Update when we unmark objects so we can assert the plan is valid,
and process UnknownAsNull on the unmarked value
2020-10-02 13:03:11 -04:00
Pam Selle 0a02e7040f
Store sensitive attribute paths in state (#26338)
* Add creation test and simplify in-place test

* Add deletion test

* Start adding marking from state

Start storing paths that should be marked
when pulled out of state. Implements deep
copy for attr paths. This commit also includes some
comment noise from investigations, and fixing the diff test

* Fix apply stripping marks

* Expand diff tests

* Basic apply test

* Update comments on equality checks to clarify current understanding

* Add JSON serialization for sensitive paths

We need to serialize a slice of cty.Path values to be used to re-mark
the sensitive values of a resource instance when loading the state file.
Paths consist of a list of steps, each of which may be either getting an
attribute value by name, or indexing into a collection by string or
number.

To serialize these without building a complex parser for a compact
string form, we render a nested array of small objects, like so:

[
  [
    { type: "get_attr", value: "foo" },
    { type: "index", value: { "type": "number", "value": 2 } }
  ]
]

The above example is equivalent to a path `foo[2]`.

* Format diffs with map types

Comparisons need unmarked values to operate on,
so create unmarked values for those operations. Additionally,
change diff to cover map types

* Remove debugging printing

* Fix bug with marking non-sensitive values

When pulling a sensitive value from state,
we were previously using those marks to remark
the planned new value, but that new value
might *not* be sensitive, so let's not do that

* Fix apply test

Apply was not passing the second state
through to the third pass at apply

* Consistency in checking for length of paths vs inspecting into value

* In apply, don't mark with before paths

* AttrPaths test coverage for DeepCopy

* Revert format changes

Reverts format changes in format/diff for this
branch so those changes can be discussed on a separate PR

* Refactor name of AttrPaths to AttrSensitivePaths

* Rename AttributePaths/attributePaths for naming consistency

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2020-09-24 12:40:17 -04:00
James Bardin 53a8e321eb remove mod=vendor in statemgr test 2020-09-24 08:44:49 -04:00
Pam Selle 843ed8911b Don't save PathValueMarks on instance_object 2020-09-10 11:04:17 -04:00
Pam Selle bc55b6a28b Use UnmarkDeepWithPaths and MarkWithPaths
Updates existing code to use the new Value
methods for unmarking/marking and removes
panics/workarounds in cty marshall methods
2020-09-10 11:04:17 -04:00
Pam Selle 2c35869a32 Fix bug for force push for backends besides the remote backend
In refactoring the force push code when implementing force push
for the Terraform remote backend, a bug was introduced that
meant that backends that don't implement the EnableForcePush
method would still have their state validated. This commit
fixes that, and adds test coverage such that there is a separate
mockRemoteClient that has this method implemented.
2020-09-10 09:13:57 -04:00
Alisdair McDiarmid e02e53ed4d states: Add roundtrip v3->v4 with builtin provider 2020-08-14 15:40:48 -04:00
Kristin Laemmert 95eca06782
statemgr: add a NewUnlockErrorFull state manager for tests (#25823)
* statemgr: add a NewUnlockErrorFull state manager for tests

I've frequently needed to coerce Unlock() errors for tests and it's been
awkward and fraught every time, so I decided to add a full state manger
that returns *mostly* errors. I intend to use this in conjunction with
the clistate.Locker interface, which first calls Lock() (to block if the
mutex is in use) at the start of Unlock(), so Lock() rather awkwardly needed to succeed.
2020-08-14 14:14:51 -04:00
Kristin Laemmert 5964e928dc
states/statefile: consistently sort resources across modules (#25498)
If a statefile had resources with the same name in different modules,
the sort order could be inconsistent between state refreshes. This adds
the module to the Less() function used in sorting and a minimal test to
verify consistent ordering.
2020-08-11 11:50:07 -04:00
Kristin Laemmert 6621501ae3
state: remove deprecated state package (#25490)
Most of the state package has been deprecated by the states package.
This PR replaces all the references to the old state package that
can be done simply - the low-hanging fruit.

* states: move state.Locker to statemgr

The state.Locker interface was a wrapper around a statemgr.Full, so
moving this was relatively straightforward.

* command: remove unnecessary use of state package for writing local terraform state files

* move state.LocalState into terraform package

state.LocalState is responsible for managing terraform.States, so it
made sense (to me) to move it into the terraform package.

* slight change of heart: move state.LocalState into clistate instead of
terraform
2020-08-11 11:43:01 -04:00
Kristin Laemmert 58bcc2e9bb
addrs: detect builtin provider when parsing legacy provider string (#25154)
* addrs: detect builtin provider when parsing legacy provider string

The ParseLegacyAbsProviderConfig was not detecting builtin providers
("terraform"), which caused issues for all users with 0.12 state and
the "terraform_remote_state" data source. Since "terraform" is the only
built-in provider this adds a very simple check to the parser so it
properly returns the builtin FQN.

* add tests to the addrs package
2020-06-08 08:27:36 -04:00
James Bardin 08128fe775 add omitempty to create_before_destroy
no need to litter the state file with "create_before_destroy":false
2020-05-18 11:31:36 -04:00
James Bardin 9fe87fe520 remove unused depends_on field and add test
The depends_on fields was deprecated during 0.12, and is no longer used.

Add a roundtrip test for the create_before_destroy status.
2020-05-14 15:46:08 -04:00
James Bardin 23f5fc3e5a thread create_before_destroy to the state file 2020-05-14 15:46:08 -04:00
James Bardin 15a95031e5 remove a few traces of states.EachMode 2020-04-30 09:22:15 -04:00
James Bardin f915c5d957 remove EachMode from resource state
Due to the fact that resources can transition between each modes, trying
to track the mode for a resource as a whole in state doesn't work,
because there may be instances with a mode different from the resource
as a whole. This is difficult for core to track, as this metadata being
changed as a side effect from multiple places often causes core to see
the incorrect mode when evaluating instances.

Since core can always determine the correct mode to evaluate from the
configuration, we don't need to interrogate the state to know the mode.
Once core no longer needs to reference EachMode from states, the
resource state can simply be a container for instances, and doesn't need
to try and track the "current" mode.
2020-04-30 09:22:14 -04:00
James Bardin 6c0f7703a6
Merge pull request #24697 from hashicorp/jbardin/get-module-data
Always return all module instances during evaluation
2020-04-22 09:49:45 -04:00
James Bardin 2c4c027a97 Add ModuleOutputs method to states
In order to efficiently build the module objects for evaluation, we need
to collect the outputs from a set of module instances. The ModuleOutputs
method will return a copy of the state outputs, while not requiring the
unnecessary copying of each entire module.
2020-04-13 17:59:09 -04:00
James Bardin e9eb8e04cc add AbsOutputAddrs to state outputs
We need all module instance outputs to build the objects for evaluation,
but there is no need to copy all the resource instances along with that.
This allows us to only return the output states, with enough information
to connect them with their module instances.
2020-04-13 16:37:59 -04:00
Mikhail Mazurskiy c9ce1f0c1f
Fix typos (#24531) 2020-04-13 08:34:40 -04:00
James Bardin a0f92f9d6a remove extra brackets in state string output 2020-04-09 15:36:16 -04:00
James Bardin 3ee9cf49ce missing wg.Wait in concurrent test 2020-04-08 10:55:20 -04:00
Kristin Laemmert c4873778c8 Mildwonkey/tests (#24548)
* helper/resource: remove provider resolver test
* repl tests passing
* helper/resource: add some extra shimming to ShimLegacyState

Some of the tests in helper/resource have to shim between legacy and
modern states. Terraform expects modern states to have the Provider set
for each resource (and not be a legacy provider). This PR adds a wrapper
around terraform.ShimLegacyState which iterates through the resources
in the state (after the shim) and adds the provider FQN.
2020-04-06 09:24:23 -07:00
Kristin Laemmert d95667f264 states/version3 upgrade: revert provider FQN changes (#24504)
This commit reverts an earlier change which automatically converted
provider strings to legacy provider FQNs. It has become apparent that a
state upgrade step will be required before upgrading to v0.13.
2020-04-06 09:24:23 -07:00
Martin Atkins 4061cbed38 internal/getproviders: A new shared model for provider requirements
We've been using the models from the "moduledeps" package to represent our
provider dependencies everywhere since the idea of provider dependencies
was introduced in Terraform 0.10, but that model is not convenient to use
for any use-case other than the "terraform providers" command that needs
individual-module-level detail.

To make things easier for new codepaths working with the new-style
provider installer, here we introduce a new model type
getproviders.Requirements which is based on the type the new installer was
already taking as its input. We have new methods in the states, configs,
and earlyconfig packages to produce values of this type, and a helper
to merge Requirements together so we can combine config-derived and
state-derived requirements together during installation.

The advantage of this new model over the moduledeps one is that all of
recursive module walking is done up front and we produce a simple, flat
structure that is more convenient for the main use-cases of selecting
providers for installation and then finding providers in the local cache
to use them for other operations.

This new model is _not_ suitable for implementing "terraform providers"
because it does not retain module-specific requirement details. Therefore
we will likely keep using moduledeps for "terraform providers" for now,
and then possibly at a later time consider specializing the moduledeps
logic for only what "terraform providers" needs, because it seems to be
the only use-case that needs to retain that level of detail.
2020-03-27 09:01:32 -07:00
James Bardin b3fc0dab94 use addrs.ConfigResource for dependency tracking
We can't get module instances during transformation, so we need to
reduce the Dependencies to using `addrs.ConfigResource` for now.
2020-03-25 17:03:06 -04:00
Kristin Laemmert ed1aebbeda
terraform: large refactor to use Provider from configs.Resource (#24396)
* terraform: large refactor to use Provider from configs.Resource

configs.Resource.ImpliedProvider() now returns a string; it is the
callers' responsibility to turn that into an addrs.Provider if needed.

GraphNodeProviderConsumer ProvidedBy() no longer returns nil (reverting
to earlier, pre-provider-fqn behavior): it will return either the
provider set in config, provider set in state, or the default provider.
2020-03-18 08:58:20 -04:00
James Bardin ea51b790bc states.Resource needs to record its module too
The ModuleInstance is known while building the state resource, but it's
not recorded. Since a resource may be retrieved via a ConfigResource
address, we need to know from which module instance it was loaded.
2020-03-16 11:16:23 -04:00
James Bardin 3729e6a705 update MaybeFixUpResourceInstanceAddressForCount
We need to fixup count by config path, not by absolute path, so update
MaybeFixUpResourceInstanceAddressForCount to work with
addrs.ConfigResource.
2020-03-16 11:16:23 -04:00
James Bardin e13eecbc5b finish provider ModuleInstance replacement 2020-03-11 14:19:52 -04:00
James Bardin 8497adcb6e AbsProviderConfig to use addrs.Module
Change ModuleInstance to Module in AbsProviderConfig, because providers
need to be handled before module expansion, and should not be used
defined inside an expanded module at all.

Renaming of the addrs type can happen later, when there's less work
in-flight around provider configuration.
2020-03-10 20:25:44 -04:00
James Bardin 654e880bb8
Merge pull request #24084 from hashicorp/jbardin/cbd-instance-state
Add CreateBeforeDestroy to instance state
2020-03-09 13:16:29 -04:00
Alisdair McDiarmid f54e2a62bf states: Fix coverage blips in statemgr/filesystem
Something non-deterministic in the test suite is causing a coverage blip
in a line in the filesystem state manager. This commit adds a test which
specifically covers that line, which hopefully pleases the Codecov robot.
2020-03-06 17:55:28 -05:00
James Bardin dd8ab5812e add CreateBeforeDestroy to state
Added to src and object, but still need serialization and tests.
2020-02-13 21:04:56 -05:00
Kristin Laemmert 47a16b0937
addrs: embed Provider in AbsProviderConfig instead of Type
a large refactor to addrs.AbsProviderConfig, embedding the addrs.Provider instead of a Type string. I've added and updated tests, added some Legacy functions to support older state formats and shims, and added a normalization step when reading v4 (current) state files (not the added tests under states/statefile/roundtrip which work with both current and legacy-style AbsProviderConfig strings).

The remaining 'fixme' and 'todo' comments are mostly going to be addressed in a subsequent PR and involve looking up a given local provider config's FQN. This is fine for now as we are only working with default assumption.
2020-02-13 15:32:58 -05:00
Martin Atkins 8b511524d6
Initial steps towards AbsProviderConfig/LocalProviderConfig separation (#23978)
* Introduce "Local" terminology for non-absolute provider config addresses

In a future change AbsProviderConfig and LocalProviderConfig are going to
become two entirely distinct types, rather than Abs embedding Local as
written here. This naming change is in preparation for that subsequent
work, which will also include introducing a new "ProviderConfig" type
that is an interface that AbsProviderConfig and LocalProviderConfig both
implement.

This is intended to be largely just a naming change to get started, so
we can deal with all of the messy renaming. However, this did also require
a slight change in modeling where the Resource.DefaultProviderConfig
method has become Resource.DefaultProvider returning a Provider address
directly, because this method doesn't have enough information to construct
a true and accurate LocalProviderConfig -- it would need to refer to the
configuration to know what this module is calling the provider it has
selected.

In order to leave a trail to follow for subsequent work, all of the
changes here are intended to ensure that remaining work will become
obvious via compile-time errors when all of the following changes happen:
- The concept of "legacy" provider addresses is removed from the addrs
  package, including removing addrs.NewLegacyProvider and
  addrs.Provider.LegacyString.
- addrs.AbsProviderConfig stops having addrs.LocalProviderConfig embedded
  in it and has an addrs.Provider and a string alias directly instead.
- The provider-schema-handling parts of Terraform core are updated to
  work with addrs.Provider to identify providers, rather than legacy
  strings.

In particular, there are still several codepaths here making legacy
provider address assumptions (in order to limit the scope of this change)
but I've made sure each one is doing something that relies on at least
one of the above changes not having been made yet.

* addrs: ProviderConfig interface

In a (very) few special situations in the main "terraform" package we need
to make runtime decisions about whether a provider config is absolute
or local.

We currently do that by exploiting the fact that AbsProviderConfig has
LocalProviderConfig nested inside of it and so in the local case we can
just ignore the wrapping AbsProviderConfig and use the embedded value.

In a future change we'll be moving away from that embedding and making
these two types distinct in order to represent that mapping between them
requires consulting a lookup table in the configuration, and so here we
introduce a new interface type ProviderConfig that can represent either
AbsProviderConfig or LocalProviderConfig decided dynamically at runtime.

This also includes the Config.ResolveAbsProviderAddr method that will
eventually be responsible for that local-to-absolute translation, so
that callers with access to the configuration can normalize to an
addrs.AbsProviderConfig given a non-nil addrs.ProviderConfig. That's
currently unused because existing callers are still relying on the
simplistic structural transform, but we'll switch them over in a later
commit.

* rename LocalType to LocalName

Co-authored-by: Kristin Laemmert <mildwonkey@users.noreply.github.com>
2020-01-31 08:23:07 -05:00
Kristin Laemmert 6541775ce4
addrs: roll back change to Type field in ProviderConfig (#23937) 2020-01-28 08:13:30 -05:00
Kristin Laemmert 67fc4dd5a1 configs: move ProviderConfigCompact[Str] from addrs to configs
The configs package is aware of provider name and type (which are the
same thing today, but expected to be two different things in a future
release), and should be the source of truth for a provider config
address.
2019-12-09 08:30:08 -05:00
Kristin Laemmert e3416124cc
addrs: replace "Type string" with "Type Provider" in ProviderConfig
* huge change to weave new addrs.Provider into addrs.ProviderConfig
* terraform: do not include an empty string in the returned Providers /
Provisioners
- Fixed a minor bug where results included an extra empty string
2019-12-06 08:00:18 -05:00
Pam Selle e4d2469698 Update comment 2019-12-04 12:08:34 -05:00
Pam Selle de953eca6f Fixup for allowing empty lists/sets of resources 2019-12-03 16:29:01 -05:00
Pam Selle 19b408e053 Only need this one call, don't call the meta func 2019-12-03 14:27:18 -05:00
Pam Selle 61ce1cb3ad Fix provider related tests 2019-12-03 14:27:18 -05:00
Pam Selle 01da7b4f7f Tested and confirmed we cannot delete this, so removed TODO 2019-12-03 14:27:18 -05:00
Pam Selle 87fdcd0064 Add a new method cause doing things in module is weird 2019-12-03 14:27:18 -05:00