Commit Graph

25847 Commits

Author SHA1 Message Date
Lee Trout cb0e20ca2b Add support for force pushing with the remote backend
Both differing serials and lineage protections should be bypassed
with the -force flag (in addition to resources).

Compared to other backends we aren’t just shipping over the state
bytes in a simple payload during the persistence phase of the push
command and the force flag added to the Go TFE client needs to be
specified at that time.

To prevent changing every method signature of PersistState of the
remote client I added an optional interface that provides a hook
to flag the Client as operating in a force push context. Changing
the method signature would be more explicit at the cost of not
being used anywhere else currently or the optional interface pattern
could be applied to the state itself so it could be upgraded to
support PersistState(force bool) only when needed.

Prior to this only the resources of the state were checked for
changes not the lineage or the serial. To bring this in line with
documented behavior noted above those attributes also have a “read”
counterpart just like state has. These are now checked along with
state to determine if the state as a whole is unchanged.

Tests were altered to table driven test format and testing was
expanded to include WriteStateForMigration and its interaction
with a ClientForcePusher type.
2020-05-06 12:07:43 -04:00
Lee Trout c2c38b2ad3 Add remote state test for serial and lineage changes
We only persist a new state if the actual state contents have
changed. This test demonstrates that behavior by calling write
and persist methods when either the lineage or serial have changed.
2020-05-04 11:48:50 -04:00
James Bardin 9debd341bc
Merge pull request #24807 from hashicorp/jbardin/remove-each-mode
Remove EachMode from state
2020-05-03 10:54:07 -04:00
Kristin Laemmert cca0526705
providercache: actually break out of the loop when a matching version is found (#24823) 2020-05-01 08:49:47 -04:00
Kristin Laemmert ce03f1255f
internal/providercache: fix error message for protocol mismatch (#24818)
There was a bug in the installer trying to pass a nil error.
2020-04-30 11:12:04 -04:00
James Bardin 2bfaddcf57 fix state mv to work without EachMode
The only situation where `state mv` needs to understand the each mode is
when with resource addresses that may reference a single instance, or a
group of for_each or count instances. In this case we can differentiate
the two by checking the existence of the NoKey instance key.
2020-04-30 09:22:15 -04:00
James Bardin 15a95031e5 remove a few traces of states.EachMode 2020-04-30 09:22:15 -04:00
James Bardin 98cf28b02d 2 more tests that weren't correct
Found 2 more tests that still had dangling empty modules, which are now
fixed.
2020-04-30 09:22:14 -04:00
James Bardin f915c5d957 remove EachMode from resource state
Due to the fact that resources can transition between each modes, trying
to track the mode for a resource as a whole in state doesn't work,
because there may be instances with a mode different from the resource
as a whole. This is difficult for core to track, as this metadata being
changed as a side effect from multiple places often causes core to see
the incorrect mode when evaluating instances.

Since core can always determine the correct mode to evaluate from the
configuration, we don't need to interrogate the state to know the mode.
Once core no longer needs to reference EachMode from states, the
resource state can simply be a container for instances, and doesn't need
to try and track the "current" mode.
2020-04-30 09:22:14 -04:00
Pam Selle 87bce5f9dd
Support reading module outputs in terraform console (#24808)
* Include eval in output walk

This allows outputs to be evaluated in the evalwalk,
impacting terraform console. Outputs are still not evaluated
for terraform console in the root module, so this has
no impact on writing to state (as child module outputs are not
written to state). Also adds test coverage to the console command,
including for evaluating locals (another use of the evalwalk)
2020-04-30 09:21:42 -04:00
James Bardin 67f66ee970
Merge pull request #24751 from hashicorp/jbardin/mod-resource-evaluation
Resource (and module) evaluate improvements
2020-04-30 09:21:15 -04:00
Alisdair McDiarmid 43030b21ac
Merge pull request #24783 from hashicorp/alisdair/implied-provider-lookup-by-localname
configs: Fix for resources with implied providers
2020-04-28 15:15:47 -04:00
Alisdair McDiarmid dcb8b45e0f configs: Fix for resources with implied providers
Previously, resources without explicit provider configuration (i.e. a
`provider =` attribute) would be assigned a default provider based upon
the resource type. For example, a resource `foo_bar` would be assigned
provider `hashicorp/foo`.

This behaviour did not work well with community or partner providers,
with sources configured in `terraform.required_providers` blocks. With
the following configuration:

    terraform {
      required_providers {
        foo = {
          source = "acme/foo"
        }
      }
    }

    resource foo_bar "a" { }

the resource would be configured with the `hashicorp/foo` provider.

This commit fixes this implied provider behaviour. First we look for a
provider with local name matching the resource type in the module's
required providers map. If one is found, this provider is assigned to
the resource. Otherwise, we still fall back to a default provider.
2020-04-28 14:54:31 -04:00
Robin Norwood d47f984e75
Add reference to new modules track (#24445) 2020-04-28 11:09:42 -07:00
Pam Selle 9957a6b42b
Update CHANGELOG.md 2020-04-28 12:27:10 -04:00
Daniel Dreier c5a2e27003
Merge pull request #24786 from hashicorp/danieldreier-23510-changelog
Update CHANGELOG.md for swift authentication
2020-04-28 09:04:16 -07:00
kmoe 8e7a4a6c21
Merge pull request #24670 from hashicorp/kmoe-cloudinit-provider
Cloud-init provider link
2020-04-28 10:03:46 +01:00
Daniel Dreier cc3c18ac35
Update CHANGELOG.md for swift authentication 2020-04-27 14:48:33 -07:00
Daniel Dreier 1170efb14a
Merge pull request #23510 from kayrus/new-swift-opts
backend/remote: Swift Authentication Update
2020-04-27 14:37:25 -07:00
Pam Selle 5956276590
Merge pull request #24780 from hashicorp/pselle/walkEval
Add expansion transformer to eval graph
2020-04-27 15:39:15 -04:00
Daniel Dreier 7aa97f415d
Fix "additoinal" typo in web site
Fixes 24769
2020-04-27 10:14:27 -07:00
Pam Selle 6ee42efe16 Add expansion transformer to eval graph
Add the expansion transformer to the eval graph,
which is used in rare scenarios which includes running
terraform console. Prevents panic when running terraform
console in contexts with module expansion
2020-04-27 13:09:08 -04:00
Alisdair McDiarmid c28044d232
Update CHANGELOG.md 2020-04-27 11:20:34 -04:00
Alisdair McDiarmid c1137430c4
Merge pull request #24763 from hashicorp/alisdair/required-providers-refactor
configs: Simplify required_providers blocks
2020-04-27 11:16:24 -04:00
kayrus 533c059378 Update dependencies 2020-04-25 08:41:54 +02:00
kayrus bd344f9d73 Sync auth options with upstream openstack provider 2020-04-25 08:39:33 +02:00
kayrus 50084f5b17 gofmt -w -s 2020-04-25 08:39:30 +02:00
Alisdair McDiarmid 9266e944fa
Merge pull request #24752 from hashicorp/alisdair/terraform-state-replace-provider-docs
website: Add docs for state replace-provider
2020-04-24 16:33:08 -04:00
Alisdair McDiarmid 7ca7b1f0fe configs: Simplify required_providers blocks
We now permit at most one `required_providers` block per module (except
for overrides). This prevents users (and Terraform) from struggling to
understand how to merge multiple `required_providers` configurations,
with `version` and `source` attributes split across multiple blocks.

Because only one `required_providers` block is permitted, there is no
need to concatenate version constraints and resolve them. This allows us
to simplify the structs used to represent provider requirements,
aligning more closely with other structs in this package.

This commit also fixes a semantic use-before-initialize bug, where
resources defined before a `required_providers` block would be unable to
use its source attribute. We achieve this by processing the module's
`required_providers` configuration (and overrides) before resources.

Overrides for `required_providers` work as before, replacing the entire
block per provider.
2020-04-24 13:44:08 -04:00
Brian Flad dad1262fb8
Update CHANGELOG for #24744 2020-04-24 12:23:37 -04:00
Brian Flad 2681ccf87f
deps: github.com/aws/aws-sdk-go@v1.30.12 [master] (#24745)
* update github.com/aws/aws-sdk-go to v1.30.9

* deps: github.com/aws/aws-sdk-go@v1.30.12

Reference: https://github.com/hashicorp/terraform/issues/24710
Reference: https://github.com/hashicorp/terraform/issues/24741

Changes:

```
NOTES:

* backend/s3: Region validation now automatically supports the new `af-south-1` (Africa (Cape Town)) region. For AWS operations to work in the new region, the region must be explicitly enabled as outlined in the [AWS Documentation](https://docs.aws.amazon.com/general/latest/gr/rande-manage.html#rande-manage-enable). When the region is not enabled, the Terraform S3 Backend will return errors during credential validation (e.g. `error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid`).

ENHANCEMENTS:

* backend/s3: Support automatic region validation for `af-south-1`
```

Updated via:

```console
$ go get github.com/aws/aws-sdk-go@v1.30.12
$ go mod tidy
$ go mod vendor
```

Output from acceptance testing:

```console
$ TF_ACC=1 go test -v ./backend/remote-state/s3 | grep '^--- '
--- PASS: TestBackend_impl (0.00s)
--- PASS: TestBackendConfig (1.68s)
--- PASS: TestBackendConfig_invalidKey (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyLength (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyEncoding (0.00s)
--- PASS: TestBackendConfig_conflictingEncryptionSchema (0.00s)
--- PASS: TestBackend (15.07s)
--- PASS: TestBackendLocked (26.40s)
--- PASS: TestBackendSSECustomerKey (16.99s)
--- PASS: TestBackendExtraPaths (12.05s)
--- PASS: TestBackendPrefixInWorkspace (5.55s)
--- PASS: TestKeyEnv (45.07s)
--- PASS: TestRemoteClient_impl (0.00s)
--- PASS: TestRemoteClient (5.39s)
--- PASS: TestRemoteClientLocks (14.30s)
--- PASS: TestForceUnlock (20.08s)
--- PASS: TestRemoteClient_clientMD5 (16.43s)
--- PASS: TestRemoteClient_stateChecksum (24.58s)
```

Co-authored-by: Nicola Senno <nicola.senno@workday.com>
2020-04-24 12:20:04 -04:00
Martin Atkins 7d494b1b03
Update CHANGELOG.md 2020-04-23 17:48:37 -07:00
Martin Atkins 3fc3cba66d
Update CHANGELOG.md 2020-04-23 16:35:39 -07:00
James Bardin 7e6d07ee46 create index-able types for validation
Since objects and tuples have fixed numbers of elements, we can't return
an unknown version of those during validation. While we could return a
DyanmicVal (which was used previously), that prevents the validation of
outputs and attributes in config references.

Instead, we can return a synthetic type made from a List or Map based
on the configuration, which will allow us to more precisely validate
indexes, attributes, and outputs.
2020-04-23 16:23:19 -04:00
James Bardin 91e243b878 we must evaluate to DynamicVal during Validate
Because tuple types have a fixed number of elements, and we may not know
the number of expanded instances, we can't use an unknown tuple type to
validate index expressions.
2020-04-23 16:23:19 -04:00
James Bardin 7290e28ca4 update GetModules
Update the GetModule evaluation method with details learned from
refactoring the GetResource method.
2020-04-23 16:23:19 -04:00
James Bardin 5fa90d2032 refactor GetResource based on GetModules
Since evaluation is driven only by the configuration (i.e. you can't
interpolate what's not in the config), the resource evaluation should
also follow configuration rather than state. Determining the each mode
solely from the config, and applying that to the state and changes
removes the need for EachMode in the resource state. This bypasses the
awkward dance around getting the correct EachMode set in and retrieved
from state during plan when it changes in the config.
2020-04-23 16:23:19 -04:00
James Bardin 0930f9cd97 use object and tuple for module eval
The outputs may be fed from dynamic types, so we need to use object,
tuple, and dynamic values.
2020-04-23 16:23:19 -04:00
Kristin Laemmert 320fcf4942
internal/getproviders: apply case normalizations in ParseMultiSourceMatchingPatterns (#24753)
* internal/getproviders: apply case normalizations in ParseMultiSourceMatchingPatterns

This is a very minor refactor which takes advantage of addrs.ParseProviderPart case normalization to normalize non-wildcard sources.
2020-04-23 14:50:47 -04:00
Martin Atkins 1ce3c60693
command/cliconfig: Explicit provider installation method configuration
This set of commits allows explicit configuration of provider installation methods
in the CLI config, overriding the implicit method selections.
2020-04-23 10:58:00 -07:00
Martin Atkins 622abf707d command/cliconfig: Remove redundant struct types
These were being used in an earlier iteration of the provider installation
configuration but it was all collapsed down into a single
ProviderInstallationMethod type later, making these redundant.
2020-04-23 10:52:01 -07:00
Martin Atkins dadec6ee9e fixup docs 2020-04-23 10:52:01 -07:00
Martin Atkins c6cbbcb79a website: Initial documentation for provider_installation in the CLI config
This is an initial draft of documentation for this new feature of the
CLI configuration. This is mainly intended as a placeholder for now,
because there are other documentation updates pending for the new provider
namespacing and installation scheme and we'll likely want to revise these
docs to better complement the broader documentation once it's written.
2020-04-23 10:52:01 -07:00
Martin Atkins 6b2050f42a main: Properly handle provider installation method exclusions
Previously we were incorrectly using the Include configuration for both
the include and exclude list, making the include portion totally
ineffective.
2020-04-23 10:52:01 -07:00
Martin Atkins 3167067029 command/e2etest: provider installation with explicit install methods
This exercises the ability to customize the installation methods used by
the provider plugin installer, in this case forcing the use of a custom
local directory with a result essentially the same as what happens when
you pass -plugin-dir to "terraform init".
2020-04-23 10:52:01 -07:00
Martin Atkins c7fe6b9160 command/cliconfig: handle provider_installation block in JSON syntax
The CLI config can be written in both native HCL and HCL JSON syntaxes, so
the provider_installation block must be expressible using JSON too. Our
previous checks to approximate HCL 2-level strictness were too strict for
HCL JSON where things are more ambiguous even in HCL 2, so this includes
some additional relaxations if we detect that we're decoding an AST
produced from a JSON file.

This is still subject to the quirky ways HCL 1 handles JSON though, so
the JSON value must be structured in a way that doesn't trigger HCL's
heuristics that try to guess what is a block and what is an attribute.
(This is the issue that HCL 2 fixes by always decoding using a schema;
there's more context on this in:
  https://log.martinatkins.me/2019/04/25/hcl-json/ )
2020-04-23 10:52:01 -07:00
Martin Atkins f5012c12da command/cliconfig: Installation methods, not installation sources
Unfortunately in the user model the noun "source" is already used for the
argument in the required_providers block to specify which provider to use,
so it's confusing to use the same noun to also refer to the method used to
obtain that provider.

In the hope of mitigating that confusion, here we use the noun "method",
as in "installation method", to talk about the decision between getting
a provider directly from its origin registry or getting it from some
mirror. This is distinct from the provider's "source", which is the
location where a provider _originates_ (prior to mirroring).

This noun is also not super awesome, but better than overloading an
existing term in the same feature.
2020-04-23 10:52:01 -07:00
Martin Atkins e872ec4461 command/cliconfig: Remove remnant extraArg checks in provider_installation
In the first pass of implementing this it was strict about what arguments
are allowed inside source blocks, but that was counter to our usual design
principles for CLI config where we tend to ignore unrecognized things to
allow for some limited kinds of future expansion without breaking
compatibility with older versions of Terraform that will be sharing the
same CLI configuration files with newer versions.

However, I'd removed the tracking of that prior to the initial commit. I
missed some leftover parts when doing that removal, so this cleans up the
rest of it.
2020-04-23 10:52:01 -07:00
Martin Atkins 8b75d1498f command/cliconfig: Use existing HTTP mirror source rather than new stub
An earlier commit added a redundant stub for a new network mirror source
that was already previously stubbed as HTTPMirrorSource.

This commit removes the unnecessary extra stub and changes the CLI config
handling to use it instead. Along the way this also switches to using a
full base URL rather than just a hostname for the mirror, because using
the usual "Terraform-native service discovery" protocol here doesn't isn't
as useful as in the places we normally use it (the mirror mechanism is
already serving as an indirection over the registry protocol) and using
a direct base URL will make it easier to deploy an HTTP mirror under
a path prefix on an existing static file server.
2020-04-23 10:52:01 -07:00
Martin Atkins b8856c677c cliconfig: Ignore config dir if TF_CLI_CONFIG_FILE envvar is set
When we originally introduced this environment variable it was intended to
solve for the use-case where a particular invocation of Terraform needs
a different CLI configuration than usual, such as if Terraform is being
run as part of an automated test suite or other sort of automated
situation with different needs than normal use.

However, we accidentally had it only override the original singleton CLI
config file, while leaving the CLI configuration directory still enabled.
Now we'll take the CLI configuration out of the equation too, so that only
the single specified configuration file and any other environment-sourced
settings will be included.
2020-04-23 10:52:01 -07:00