Commit Graph

732 Commits

Author SHA1 Message Date
Martin Atkins 23395a1022 providercache: Discard lock entries for unused providers
Previously we would only ever add new lock entries or update existing
ones. However, it's possible that over time a module may _cease_ using
a particular provider, at which point we ought to remove it from the lock
file so that operations won't fail when seeing that the provider cache
directory is inconsistent with the lock file.

Now the provider installer (EnsureProviderVersions) will remove any lock
file entries that relate to providers not included in the given
requirements, which therefore makes the resulting lock file properly match
the set of packages the installer wrote into the cache.

This does potentially mean that someone could inadvertently defeat the
lock by removing a provider dependency, running "terraform init", then
undoing that removal, and finally running "terraform init" again. However,
that seems relatively unlikely compared to the likelihood of removing
a provider and keeping it removed, and in the event it _did_ happen the
changes to the lock entry for that provider would be visible in the diff
of the provider lock file as usual, and so could be noticed in code
review just as for any other change to dependencies.
2021-12-17 15:30:21 -08:00
James Bardin 8c4031ef15 don't persist a nil state from Apply
Apply should not return a nil state to be persisted.
2021-12-17 14:00:59 -05:00
James Bardin 58e001c22a return graph errors from Context.Apply
errors from building during apply were lost
2021-12-17 14:00:57 -05:00
James Bardin f83ed441bb
Merge pull request #30189 from hashicorp/jbardin/validate-moves
Ignore unexpanded paths when validating move statements.
2021-12-17 13:57:24 -05:00
James Bardin b213386a73 InstancesForModule should not panic
instances.Set is only used after all instances have been processes, so
it should therefor only handle known instances and not panic when given
an address that traverses an unexpanded module.
2021-12-17 13:31:41 -05:00
James Bardin 371660ab8f cleanup panic output 2021-12-17 11:57:52 -05:00
James Bardin a5017bff2f failing test moved with target 2021-12-16 18:20:49 -05:00
James Bardin e22ab70e03
Merge pull request #30171 from hashicorp/jbardin/revert-validate-for-each
use `cty.DynamicVal` for expanded resources during validation
2021-12-15 08:57:43 -05:00
James Bardin 645fcc5f12 test for lookup regression during validation 2021-12-15 08:43:37 -05:00
James Bardin 71b9682e8c ensure there is always a valid return value 2021-12-14 18:02:57 -05:00
James Bardin d469e86331 revert 6b8b0617
Revert the evaluation change from #29862.
While returning a dynamic value for all expanded resources during
validation is not optimal, trying to work around this using unknown maps
and lists is causing other undesirable behaviors during evaluation.
2021-12-14 17:58:10 -05:00
Martin Atkins c4d46e7c6b getmodules: Re-allow git:: source with ref=COMMIT_ID
Earlier versions of this code allowed "ref" to take any value that would
be accepted by "git checkout" as a valid target of a symbolic ref. We
inadvertently accepted a breaking change to upstream go-getter that broke
that as part of introducing a shallow clone optimization, because shallow
clone requires selecting a single branch.

To restore the previous capabilities while retaining the "depth" argument,
here we accept a compromise where "ref" has the stronger requirement of
being a valid named ref in the remote repository if and only if "depth"
is set to a value greater than zero. If depth isn't set or is less than
one, we will do the old behavior of just cloning all of the refs in the
remote repository in full and then switching to refer to the selected
branch, tag, or naked commit ID as a separate step.

This includes a heuristic to generate an additional error message hint if
we get an error from "git clone" and it looks like the user might've been
trying to use "depth" and "ref=COMMIT" together. We can't recognize that
error accurately because it's only reported as human-oriented git command
output, but this heuristic should hopefully minimize situations where we
show it inappropriately.

For now this is a change in the Terraform repository directly, so that we
can expedite the fix to an already-reported regression. After this is
released I tend to also submit a similar set of changes to upstream
go-getter, at which point we can revert Terraform to using the upstream
getter.GitGetter instead of our own local fork.
2021-12-14 11:24:23 -08:00
Martin Atkins b0ff17ef2a getmodules: Inline our own fork of getter.GitGetter
This is a pragmatic temporary solution to allow us to more quickly resolve
an upstream regression in go-getter locally within Terraform, so that the
work to upstream it for other callers can happen asynchronously and with
less time pressure.

This commit doesn't yet include any changes to address the bug, and
instead aims to be functionally equivalent to getter.GitGetter. A
subsequent commit will then address the regression, so that the diff of
that commit will be easier to apply later to the upstream to get the same
effect there.
2021-12-14 11:24:23 -08:00
Chris Arcand 8b8fe2771f
Merge pull request #30142 from hashicorp/chrisarcand/remote-backend-no-workspaces-regression
command/meta_backend: Allow the remote backend to have no workspaces [again]
2021-12-14 09:58:50 -06:00
Chris Arcand 98978b3853 command/meta_backend: Allow the remote backend to have no workspaces [again]
A regression introduced in d72a413ef8

The comment explains, but TLDR: The remote backend actually *depended*
on being able to write it's backend state even though an 'error'
occurred (no workspaces).
2021-12-14 09:50:42 -06:00
Martin Atkins 096cddb4b7 command/format: Limitation of plans.ResourceInstanceDeleteBecauseNoModule
This is an explicit technical debt note that our plan renderer isn't able
to give a fully-specific hint in this particular case of deletion reason.

This reason code means that at least one of the module instance keys in
the resource's module path doesn't match an instance declared in the
configuration, but the plan data structure doesn't retain enough
information to know which is the first step in the path which refers to
a missing instance, and so we just always return the whole thing.

This would be confusing if we return module.foo[0].module.bar not being
in the configuration as a result of module.foo not using "count"; it would
be better to say "module.foo[0] is not in the configuration" instead.

It would be most ideal to handle all of the different situations that
ResourceInstanceDeleteBecauseWrongRepetition's rendering does, so that we
can go further and explain exactly _why_ that module instance isn't
declared anymore.

We can do neither of those things today because only the Terraform Core
"expander" component knows that information, and we've discarded that
by the time we get to rendering a plan. To fix this one day would require
preserving in the plan information about which module instances are
declared, as a separate sidecar data structure from which resource
instances we're taking actions on, and then using that to identify which
step in addr.Module here first selects an invalid instance.
2021-12-13 10:04:15 -05:00
Martin Atkins ec6fe93fa8 instances: Non-existing module instance has no resource instances
Previously we were treating it as a programming error to ask for the
instances of a resource inside an instance of a module that is declared
but whose declaration doesn't include the given instance key.

However, that's actually a valid situation which can arise if, for
example, the user has changed the repetition/expansion mode for an
existing module call and so now all of the resource instances addresses it
previously contained are "orphaned".

To represent that, we'll instead say that an invalid instance key of a
declared module behaves as if it contains no resource instances at all,
regardless of the configurations of any resources nested inside. This
then gives the result needed to successfully detect all of the former
resource instances as "orphaned" and plan to destroy them.

However, this then introduces a new case for
NodePlannableResourceInstanceOrphan.deleteActionReason to deal with: the
resource configuration still exists (because configuration isn't aware of
individual module/resource instances) but the module instance does not.
This actually allows us to resolve, at least partially, a previous missing
piece of explaining to the user why the resource instances are planned
for deletion in that case, finally allowing us to be explicit to the user
that it's because of the module instance being removed, which
internally we call plans.ResourceInstanceDeleteBecauseNoModule.

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2021-12-13 10:03:50 -05:00
Chris Arcand f521ba6cd7 Remove 'enhanced' backend type distinction
As explained in the changes: The 'enhanced' backend terminology, which
only truly pertains to the 'remote' backend with a single API (Terraform
Cloud/Enterprise's), has been found to be a confusing vestige which need
only be explained in the context of the 'remote' backend.

These changes reorient the explanation(s) of backends to pertain more
directly to their primary purpose, which is storage of state snapshots
(and not implementing operations).

That Terraform operations are still _implemented_ by the literal
`Backend` and `Enhanced` interfaces is inconsequential a user of
Terraform, an internal detail.
2021-12-07 16:29:51 -06:00
Chris Arcand f4f5b7c968
Merge pull request #30059 from hashicorp/barrettclark/explicit-local-empty-migrate-messaging
Cloud: Do not ask to migrate empty default workspace
2021-12-01 11:53:36 -06:00
Barrett Clark c706c8f92b Do not ask to migrate empty default workspace
When migrating from an explicit local backend to Terraform Cloud, we ask
if you want to migrate the state. If there is no state to migrate we
should not ask if they want to migrate the emptiness.
2021-12-01 11:43:41 -06:00
Barrett Clark e08a02e7bf Fixes Issue #29959, Apply w/o init error message
When going from a local backend to Terraform Cloud, if you skip the
`terraform init` and run `terraform apply` this will give the user more
clear instructions.
2021-12-01 11:28:35 -06:00
Luces Huayhuaca d72a413ef8
command/meta_backend: Prompt to select workspace before saving backend config (#29945)
When terraform detects that a user has no workspaces that map to their current configuration, it will prompt the user to create a new workspace and enter a value name. If the user ignores the prompt and exits it, the legacy backend (terraform.tfstate) will be left in a awkward state:

1. This saved backend config will show a diff for the JSON attributes "serial", "tags" and "hash"
2. "Terraform workspace list" will show an empty list
3. "Terraform apply" will run successfully using the previous workspace, from the previous config, not the one from the current saved backend config
4. The cloud config is not reflective of the current working directory

Solution: If the user exits the prompt, the saved backend config should not be updated because they did not select a new workspace. They are back at the beginning where they are force to re run the init cmd again before proceeding with new changes.
2021-12-01 08:53:47 -08:00
Martin Atkins affe2c3295 addrs: Expose the registry address parser's error messages
Previously we ended up losing all of the error message detail produced by
the registry address parser, because we treated any registry address
failure as cause to parse the address as a go-getter-style remote address
instead.

That led to terrible feedback in the situation where the user _was_
trying to write a module address but it was invalid in some way.

Although we can't really tighten this up in the default case due to our
compatibility promises, it's never been valid to use the "version"
argument with anything other than a registry address and so as a
compromise here we'll use the presence of "version" as a heuristic for
user intent to parse the source address as a registry address, and thus
we can return a registry-address-specific error message in that case and
thus give more direct feedback about what was wrong.

This unfortunately won't help someone trying to install from the registry
_without_ a version constraint, but I didn't want to let perfect be the
enemy of the good here, particularly since we recommend using version
constraints with registry modules anyway; indeed, that's one of the main
benefits of using a registry rather than a remote source directly.
2021-11-30 15:46:16 -08:00
kmoe 8f923cea08
Merge pull request #30048 from hashicorp/kmoe/http-backend-debug-log
backend/http: supply logging.LogOutput() to retryable client
2021-11-30 14:25:28 +00:00
Alisdair McDiarmid 8ec9ad0407
Merge pull request #30029 from hashicorp/alisdair/add-sensitive-marks-for-nested-attributes
configs: Add sensitive marks for nested attributes
2021-11-30 09:20:43 -05:00
Alisdair McDiarmid d24403799e
Merge pull request #30019 from hashicorp/alisdair/fix-provider-schema-for-deleted-resource-instance
core: Fix schema loading for deleted resources
2021-11-30 09:20:27 -05:00
Katy Moe b1c071bb57
pass hclog logger to retryable client 2021-11-30 00:09:01 +00:00
Chris Arcand b5af7b6c92
Merge pull request #29987 from hashicorp/chrisarcand/backend-flag-with-tfc
command/init: Add -cloud alias to -backend, adjust `init` help output
2021-11-29 08:06:08 -06:00
Alisdair McDiarmid fbed52a353 configs: Add sensitive marks for nested attributes
Object values returned from providers have their attributes marked as
sensitive based on the provider schema. This was not fully implemented
for nested attribute types, which is corrected in this commit.
2021-11-25 11:55:17 -05:00
Alisdair McDiarmid 5ddf5e1f7d core: Fix schema loading for deleted resources
Resource instances removed from the configuration would previously use
the implied provider address. This is correct for default providers, but
incorrect for those from other namespaces or hosts. The fix here is to
use the stored provider config if it is present.
2021-11-24 15:23:20 -05:00
Barrett Clark a6b56ad76f
Merge pull request #29997 from hashicorp/barrettclark/cloud-input-false
Cloud integration requires input for migrations
2021-11-23 16:22:27 -06:00
Barrett Clark 419676cb69 Cloud integration requires input for migrations
We cannot programmatically migrate workspaces to Terraform Cloud without
prompts, so `-input=false` should not be allowed in those cases.

There are 4 scenarios where we need input from a user to complete
migrating workspaces to Terraform Cloud.

1.) Migrate from a single local workspace to Terraform Cloud

* Terraform config for a local backend. Implicit local (no backend
  specified) is fine.
* `terraform init` and `terraform apply`
* Change the Terraform config to use the cloud block
* `terraform init -input=false`
* You should now see an error message

2.) Migrate from a remote backend with a prefix to Terraform Cloud with
  tags

* Create a workspace in Terraform Cloud manually. The name should
  include a prefix, like "app-one"
* Have the terraform config use `backend "remote"` with a prefix set to
  "app-"
* `terraform init` and `terraform apply`
* Update the Terraform config to use a cloud block with `tags
  = ["app"]`. There should not be a prefix defined in the config now.
* `terraform init -input=false`
* You should now see an error message

3.) Migrate from multiple local workspaces to a single Terraform Cloud
  workspace
* Create one or many local workspaces
* `terraform init` and `terraform apply` in each
* Change the Terraform config to use the cloud block
* `terraform init -input=false`
* You should now see an error message

4.) Migrate to Terraform Cloud and ask for a workspace name
* Create several local workspaces
* `terraform init` and `terraform apply` in each
* Change the Terraform config to use the cloud block with tags
* `terraform init -input=false`
* You should now see an error message
2021-11-23 10:39:42 -06:00
Luces Huayhuaca f63b6198ca
Create a function for logic that assigns value to initReason var after changing backend configuration (#29967)
* Create a function for logic that assigns value to initReason var after changing backend configuration

Create func determineInitReason() for logic block that assigns value to initReason var after changing backend/cloud configuration block or migrating to a different type of backend configuration. Also clarify 'cloud' configuration block message to say 'Terraform Cloud configuration block has changed' instead of 'Terraform Cloud configuration has changed'.
2021-11-22 13:32:34 -08:00
Alisdair McDiarmid a8b9d086b2 cli: Fix nested single and map diff rendering
Indentation was out by two spaces. This is now consistent with the other
displays of object/map values: four spaces for each nesting level.
2021-11-19 15:48:10 -05:00
Chris Arcand a0c7c1dc8d command/init: Adjust init help output
Some of the wording here needed adjusting with the change that backends
largely reflect state snapshot storage (removing 'enhanced'
designation), and that a 'backend' is not necessarily always present.
2021-11-19 14:40:12 -06:00
Chris Arcand 33d7c1e6d6 command/init: Add -cloud alias (of -backend)
This fixes an issue where a user could not disable initialization of the
'cloud' configuration block (As is possible with -backend=false), as
well as add some syntactic sugar around -backend by adding a mutually
exclusive -cloud alias.
2021-11-19 14:40:12 -06:00
Alisdair McDiarmid b562c4fb84 cli: Fix nested set diff rendering indentation 2021-11-19 13:06:34 -05:00
Alisdair McDiarmid 2d0349d9a4 cli: Fix diff for nested set unchanged elements
Unchanged elements in nested attributes backed by sets were previously
misrendered as empty objects. This commit removes the additional
brackets and adds a count of unchanged elements.
2021-11-19 11:53:36 -05:00
Krista LaFentres 2fc4e06890 Fix failing lint check for deprecated x/crypto/openpgp library 2021-11-18 14:59:29 -06:00
Barrett Clark 7aeaec9b48 Update tests for new prompts
* Remove double question to migrate
* Remove parallelism (it has been flaky)
* Add subtests to apply_auto_approve_test (it was overlooked before)
2021-11-17 14:20:44 -08:00
Martin Atkins 7d6d31eff8 command/init: Skip redundant state migration prompt in Cloud mode
The specialized Terraform Cloud migration process asks right up top
whether the user wants to migrate state, because there are various other
questions contingent on that answer.

Therefore we ought to just honor their earlier answer when we get to the
point of actually doing the state migration, rather than prompting again.

This is tricky because we're otherwise just reusing a codepath that's
common to both modes. Hopefully we can find a better way to do this in
a later commit, but for the moment our main motivation is minimizing risk
to the very next release.
2021-11-17 14:20:44 -08:00
Barrett Clark 27aa51687e Fix cloud e2e tests for updated migration flow 2021-11-17 14:20:44 -08:00
Martin Atkins bac59d2480 command/init: Be explicit that some options are not relevant for Cloud
There are a few command line options for "terraform init" which are only
relevant when working with traditional backends, with the Cloud
integration previously just mostly ignoring them, or sometimes misbehaving
slightly due to them creating an unreasonable situation.

Now we'll catch these and return explicit errors, in order to be clear
that these options are not needed nor supported in Cloud mode.
2021-11-17 14:20:44 -08:00
Martin Atkins c28b57b4d6 cloud/e2e: Additional context for failing to find expected output
This just gives a little extra information to work with when trying to
understand why a test failed. It doesn't change what any of the tests are
actually trying to test.
2021-11-17 14:20:44 -08:00
Martin Atkins e5172fea95 cloud: DetectConfigChangeType helper
This aims to encapsulate the somewhat-weird logic we currently use to
distinguish between the various "terraform init" situations involving
Terraform Cloud mode, in the hope of making codepaths that branch based
on this slightly easier to read.

This isn't yet used, but uses of it will follow in subsequent commits.
2021-11-17 14:20:44 -08:00
Tom Harvey a1c7c0dfa4
Update internal/backend/remote-state/azure/backend.go
Co-authored-by: Tom Bamford <tbamford@hashicorp.com>
2021-11-17 20:24:22 +01:00
tombuildsstuff ffe9a41468 internal/backend/remote-state/azure: adding tests covering MSAL 2021-11-17 18:45:44 +01:00
tombuildsstuff a2cad7d579 internal/backend/remote-state/azure: prefixing the existing tests with ADAL 2021-11-17 18:43:39 +01:00
tombuildsstuff 9f710558ce internal/backend/remote-state/azure: adding a feature-toggle to use MSAL 2021-11-17 18:42:01 +01:00
tombuildsstuff 27a34d6c2c internal/backend/remote-state/azure: updating the link to the client secret
This redirects to the registry so we should point there directly
2021-11-17 18:28:47 +01:00