Commit Graph

180 Commits

Author SHA1 Message Date
Martin Atkins 73dda868cc Move backend/ to internal/backend/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
James Bardin b11578da45 prevent -migrate-state with -reconfigure 2021-05-17 14:28:34 -04:00
James Bardin 1b48636b42 update init docs for -migrate-state 2021-05-17 12:41:54 -04:00
James Bardin edc2695d18 no longer automatically attempt state migration
Add `init -migrate-state` flag to indicate automatic state migration is
desired. This flag will be implied by the `-force-copy` flag, since that
would indicate state migration is expected.

If `init` encounters a change to the stored backend configuration, it
will now always return an error when neither `-reconfigure` or
`-migrate-state` is supplied.

Turn the most common legacy output strings into diagnostics, removing
the "see above text" error output.
2021-05-17 12:41:54 -04:00
Alisdair McDiarmid 786d99207e cli: Only rewrite provider locks file if changed
If the provider locks have not changed, there is no need to rewrite the
locks file. Preventing this needless rewrite should allow Terraform to
operate in a read-only directory, so long as the provider requirements
don't change.
2021-03-29 16:09:07 -04:00
Pam Selle 242f319638
Give suggestions & remind users to use required_providers when provider not in registry (#28014)
* Add helper suggestion when failed registry err

When someone has a failed registry error on init, remind them that
they should have required_providers in every module

* Give suggestion for a provider based on reqs

Suggest another provider on a registry error, from the list of
requirements we have on init. This skips the legacy lookup
process if there is a similar provider existing in requirements.
2021-03-11 08:54:18 -05:00
Masayuki Morita 31a5aa1878
command/init: Add a new flag `-lockfile=readonly` (#27630)
Fixes #27506

Add a new flag `-lockfile=readonly` to `terraform init`.
It would be useful to allow us to suppress dependency lockfile changes
explicitly.

The type of the `-lockfile` flag is string rather than bool, leaving
room for future extensions to other behavior variants.

The readonly mode suppresses lockfile changes, but should verify
checksums against the information already recorded. It should conflict
with the `-upgrade` flag.

Note: In the original use-case described in #27506, I would like to
suppress adding zh hashes, but a test code here suppresses adding h1
hashes because it's easy for testing.

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2021-03-09 11:12:00 -05:00
Alisdair McDiarmid 94340b5940 cli: Add reference to global options to help text 2021-02-22 09:25:56 -05:00
Alisdair McDiarmid ca23a096d8 cli: Remove legacy positional path arguments
Several commands continued to support the legacy positional path
argument to specify a working directory. This functionality has been
replaced with the global -chdir flag, which is specified before any
other arguments, including the sub-command name.

This commit removes support for the trailing path parameter from
most commands. The only command which still supports a path argument is
fmt, which also supports "-" to indicate receiving configuration from
standard input.

Any invocation of a command with an invalid trailing path parameter will
result in a short error message, pointing at the -chdir alternative.

There are many test updates in this commit, almost all of which are
migrations from using positional arguments to specify a working
directory. Because of the layer at which these tests run, we are unable
to use the -chdir argument, so the churn in test files is larger than
ideal. Sorry!
2021-02-02 13:21:26 -05:00
Brian Flad 70aac9c551
command: Use different warning messaging in init command and move above potentially confusing output (#27514)
Reference: https://github.com/hashicorp/terraform/issues/27459
Reference: https://github.com/hashicorp/terraform/pull/27507

This follows the change in the plan command to move the development overrides warning above other output from the command that may exhibit unexpected behavior.
2021-02-01 10:50:08 -05:00
Jonathan Hall 49ee3d3ef8 Grammar nit: "setup" as a verb should be spelled "set up" 2021-01-26 20:39:11 +01:00
Nick Fagerlund a60120477c Update links to CLI docs in code comments, messages, and readme 2021-01-22 12:22:21 -08:00
Pam Selle 0d87c66211
Merge pull request #27463 from hashicorp/pselle/get-plugins
Remove get-plugins flag
2021-01-11 15:02:22 -05:00
Pam Selle 801f8aa0b7
Merge pull request #27461 from hashicorp/pselle/verify-plugins
Remove verify-plugins flag
2021-01-11 14:57:45 -05:00
Pam Selle bbcf6ae20b Remove state lock flags from init 2021-01-11 12:02:03 -05:00
Pam Selle 5ad6100ff3 Remove get-plugins
Remove the no-op get-plugins flag
2021-01-11 11:47:23 -05:00
Pam Selle da5a28e6ae Remove verify-plugins flag
This flag does not do anything, and so this removes the mention in
the CLI docs and the parsing of the flag variable
2021-01-11 11:38:17 -05:00
Martin Atkins bab4979128 command/init: Remove the warnings about the "legacy" cache directory
We included these warnings in v0.14 after noticing that we'd accidentally
published some incorrect documentation about the purpose of the plugin
cache directory under .terraform/plugins. We switched to using
.terraform/providers instead so that we could treat any missing providers
that appear in the legacy directory as likely to be a result of following
that documentation, and thus produce this extra warning.

However, the further we get from v0.13 the more likely it is for this
warning to be a confusing false positive rather than something helpful,
and this is a non-trivial codepath requiring us to retain a concept that
we otherwise don't need (the "legacy cache dir"), so here we'll remove
those warnings and support code for v0.15 onwards.

These warnings were always accompanied by an error message saying that a
provider could not be found, and that error message remains after this
change. This just removes the "by the way..."-style warning we had been
emitting alongside the errors.
2020-12-11 08:00:15 -08:00
Martin Atkins 4b3e237668 command/init: Hint about providers in other namespaces
If a user forgets to specify the source address for a provider, Terraform
will assume they meant a provider in the registry.terraform.io/hashicorp/
namespace. If that ultimately doesn't exist, we'll now try to see if
there's some other provider source address recorded in the registry's
legacy provider lookup table, and suggest it if so.

The error message here is a terse one addressed primarily to folks who are
already somewhat familiar with provider source addresses and how to
specify them. Terraform v0.13 had a more elaborate version of this error
message which directed the user to try the v0.13 automatic upgrade tool,
but we no longer have that available in v0.14 and later so the user must
make the fix themselves.
2020-12-10 10:11:27 -08:00
Pam Selle b963ea8594 Update docs and add warning for -get-plugins
As of Terraform 0.13+, the get-plugins command has been
superceded by new provider installation mechanisms, and
general philosophy (providers are always installed, but
the sources may be customized). Updat the init command
to give users a warning if they are setting this flag,
to encourage them to remove it from their workflow, and
update relevant docs and docstrings as well
2020-12-07 14:13:52 -05:00
James Bardin 1c58c6ba48 command staticcheck 2020-12-02 13:59:19 -05:00
Alisdair McDiarmid c5c1f31db3 backend: Validate remote backend Terraform version
When using the enhanced remote backend, a subset of all Terraform
operations are supported. Of these, only plan and apply can be executed
on the remote infrastructure (e.g. Terraform Cloud). Other operations
run locally and use the remote backend for state storage.

This causes problems when the local version of Terraform does not match
the configured version from the remote workspace. If the two versions
are incompatible, an `import` or `state mv` operation can cause the
remote workspace to be unusable until a manual fix is applied.

To prevent this from happening accidentally, this commit introduces a
check that the local Terraform version and the configured remote
workspace Terraform version are compatible. This check is skipped for
commands which do not write state, and can also be disabled by the use
of a new command-line flag, `-ignore-remote-version`.

Terraform version compatibility is defined as:

- For all releases before 0.14.0, local must exactly equal remote, as
  two different versions cannot share state;
- 0.14.0 to 1.0.x are compatible, as we will not change the state
  version number until at least Terraform 1.1.0;
- Versions after 1.1.0 must have the same major and minor versions, as
  we will not change the state version number in a patch release.

If the two versions are incompatible, a diagnostic is displayed,
advising that the error can be suppressed with `-ignore-remote-version`.
When this flag is used, the diagnostic is still displayed, but as a
warning instead of an error.

Commands which will not write state can assert this fact by calling the
helper `meta.ignoreRemoteBackendVersionConflict`, which will disable the
checks. Those which can write state should instead call the helper
`meta.remoteBackendVersionCheck`, which will return diagnostics for
display.

In addition to these explicit paths for managing the version check, we
have an implicit check in the remote backend's state manager
initialization method. Both of the above helpers will disable this
check. This fallback is in place to ensure that future code paths which
access state cannot accidentally skip the remote version check.
2020-11-19 13:19:40 -05:00
Martin Atkins c94a6102df command: Improve consistency of the command short descriptions
The short description of our commands (as shown in the main help output
from "terraform") was previously very inconsistent, using different
tense/mood for different commands. Some of the commands were also using
some terminology choices inconsistent with how we currently talk about
the related ideas in our documentation.

Here I've tried to add some consistency by first rewriting them all in
the imperative mood (except the ones that just are just subcommand
groupings), and tweaking some of the terminology to hopefully gel better
with how we present similar ideas in our recently-updated docs.

While working on this I inevitably spotted some similar inconsistencies
in the longer-form help output of some of the commands. I've not reviewed
all of these for consistency, but I did update some where the wording
was either left inconsstent with the short form changes I'd made or
where the prose stood out to me as particularly inconsistent with our
current usual documentation language style.

All of this is subjective, so I expect we'll continue to tweak these over
time as we continue to develop our documentation writing style based on
user questions and feedback.
2020-10-26 09:55:21 -07:00
Martin Atkins 30204ecded command/cliconfig: Allow development overrides for providers
For normal provider installation we want to associate each provider with
a selected version number and find a suitable package for that version
that conforms to the official hashes for that release.

Those requirements are very onerous for a provider developer currently
testing a not-yet-released build, though. To allow for that case this new
CLI configuration feature allows overriding specific providers to refer
to give local filesystem directories.

Any provider overridden in this way is not subject to the usual
restrictions about selected versions or checksum conformance, and
activating an override won't cause any changes to the selections recorded
in the lock file because it's intended to be a temporary setting for one
developer only.

This is, in a sense, a spiritual successor of an old capability we had to
override specific plugins in the CLI configuration file. There were
some vestiges of that left in the main package and CLI config package
but nothing has actually been honoring them for several versions now and
so this commit removes them to avoid confusion with the new mechanism.
2020-10-16 14:31:15 -07:00
James Bardin 1ecd86d08e
Merge pull request #26155 from hashicorp/unused-env-var
Remove unused env var TF_SKIP_PROVIDER_VERIFY
2020-10-14 18:03:22 -04:00
Martin Atkins e70ab09bf1 command: new cache directory .terraform/providers for providers
Terraform v0.10 introduced .terraform/plugins as a cache directory for
automatically-installed plugins, Terraform v0.13 later reorganized the
directory structure inside but retained its purpose as a cache.

The local cache used to also serve as a record of specifically which
packages were selected in a particular working directory, with the intent
that a second run of "terraform init" would always select the same
packages again. That meant that in some sense it behaved a bit like a
local filesystem mirror directory, even though that wasn't its intended
purpose.

Due to some unfortunate miscommunications, somewhere a long the line we
published some documentation that _recommended_ using the cache directory
as if it were a filesystem mirror directory when working with Terraform
Cloud. That was really only working as an accident of implementation
details, and Terraform v0.14 is now going to break that because the source
of record for the currently-selected provider versions is now the
public-facing dependency lock file rather than the contents of an existing
local cache directory on disk.

After some consideration of how to move forward here, this commit
implements a compromise that tries to avoid silently doing anything
surprising while still giving useful guidance to folks who were previously
using the unsupported strategy. Specifically:

- The local cache directory will now be .terraform/providers rather than
  .terraform/plugins, because .terraform/plugins is effectively "poisoned"
  by the incorrect usage that we can't reliably distinguish from prior
  version correct usage.

- The .terraform/plugins directory is now the "legacy cache directory". It
  is intentionally _not_ now a filesystem mirror directory, because that
  would risk incorrectly interpreting providers automatically installed
  by Terraform v0.13 as if they were a local mirror, and thus upgrades
  and checksum fetches from the origin registry would be blocked.

- Because of the previous two points, someone who _was_ trying to use the
  legacy cache directory as a filesystem mirror would see installation
  fail for any providers they manually added to the legacy directory.

  To avoid leaving that user stumped as to what went wrong, there's a
  heuristic for the case where a non-official provider fails installation
  and yet we can see it in the legacy cache directory. If that heuristic
  matches then we'll produce a warning message hinting to move the
  provider under the terraform.d/plugins directory, which is a _correct_
  location for "bundled" provider plugins that belong only to a single
  configuration (as opposed to being installed globally on a system).

This does unfortunately mean that anyone who was following the
incorrectly-documented pattern will now encounter an error (and the
aforementioned warning hint) after upgrading to Terraform v0.14. This
seems like the safest compromise because Terraform can't automatically
infer the intent of files it finds in .terraform/plugins in order to
decide automatically how best to handle them.

The internals of the .terraform directory are always considered
implementation detail for a particular Terraform version and so switching
to a new directory for the _actual_ cache directory fits within our usual
set of guarantees, though it's definitely non-ideal in isolation but okay
when taken in the broader context of this problem, where the alternative
would be silent misbehavior when upgrading.
2020-10-14 07:53:41 -07:00
Martin Atkins 2611e08430 command/init: Mention using the lock file for provider selection
This probably isn't the best UI we could do here, but it's a placeholder
for now just to avoid making it seem like we're ignoring the lock file
and checking for new versions anyway.
2020-10-09 09:26:23 -07:00
Martin Atkins b3f5c7f1e6 command/init: Read, respect, and update provider dependency locks
This changes the approach used by the provider installer to remember
between runs which selections it has previously made, using the lock file
format implemented in internal/depsfile.

This means that version constraints in the configuration are considered
only for providers we've not seen before or when -upgrade mode is active.
2020-10-09 09:26:23 -07:00
Kristin Laemmert d2e999ba1f
remove unused code (#26503)
* remove unused code

I've removed the provider-specific code under registry, and unused nil
backend, and replaced a call to helper from backend/oss (the other
callers of that func are provisioners scheduled to be deprecated).

I also removed the Dockerfile, as our build process uses a different
file.

Finally I removed the examples directory, which had outdated examples
and links. There are better, actively maintained examples available.

* command: remove various unused bits

* test wasn't running

* backend: remove unused err
2020-10-07 11:00:06 -04:00
Pam Selle 1817c8ac3c
Merge pull request #26412 from OwenTuz/issue-26411-fix-failing-e2e-test-provider-not-found
Fix bad string match that was causing TestInitProviderNotFound to fail
2020-10-02 10:43:18 -04:00
Martin Atkins 59b116f7bf command/init: Remove support for legacy provider addresses
We no longer need to support 0.12-and-earlier-style provider addresses
because users should've upgraded their existing configurations and states
on Terraform 0.13 already.

For now this is only checked in the "init" command, because various test
shims are still relying on the idea of legacy providers the core layer.
However, rejecting these during init is sufficient grounds to avoid
supporting legacy provider addresses in the new dependency lock file
format, and thus sets the stage for a more severe removal of legacy
provider support in a later commit.
2020-09-30 08:54:57 -07:00
Owen Tuz 5453147381 Fix bad string match that was causing TestInitProviderNotFound to fail on Macs
The tmp directory path is longer on Macs than other systems and was wrapping
across lines when printed, breaking the string match in the test.

Fix suggested by @apparentlymart is to add two spaces before the leading
'-' when printing to prevent the diagnostic renderer wrapping the line.
2020-09-30 09:30:02 +01:00
Martin Atkins f53264d378 command/init: Better error message for provider unsupported platform
As we add and remove support for different target platforms over time,
there will be transition periods where the available platforms for
each provider will be different than the available platforms for Terraform
CLI itself.

In recognition of that possibility, here we add a more specialized error
message for that situation which tries to explain the problem a little
more clearly than the generic error message that came before it.

In an ideal world we'd be able to detect situations where a newer or older
version has support in a similar vein to what we do with provider protocol
incompatibilities, but we don't currently have logic to fetch the data
necessary to implement that, so this is instead a presentation-only change
aimed at allowing some possible near-future changes to the supported
target platforms.
2020-09-29 10:28:26 -07:00
Martin Atkins 0b734a2803 command: Make provider installation interruptible
In earlier commits we started to make the installation codepath
context-aware so that it could be canceled in the event of a SIGINT, but
we didn't complete wiring that through the API of the getproviders
package.

Here we make the getproviders.Source interface methods, along with some
other functions that can make network requests, take a context.Context
argument and act appropriately if that context is cancelled.

The main providercache.Installer.EnsureProviderVersions method now also
has some context-awareness so that it can abort its work early if its
context reports any sort of error. That avoids waiting for the process
to wind through all of the remaining iterations of the various loops,
logging each request failure separately, and instead returns just
a single aggregate "canceled" error.

We can then set things up in the "terraform init" and
"terraform providers mirror" commands so that the context will be
cancelled if we get an interrupt signal, allowing provider installation
to abort early while still atomically completing any local-side effects
that may have started.
2020-09-29 10:00:35 -07:00
Paul Tyng f3ff843ffd Remove unused env var TF_SKIP_PROVIDER_VERIFY 2020-09-10 09:03:56 -04:00
Alisdair McDiarmid 9f824c53a5 command: Better in-house provider install errors
When init attempts to install a legacy provider required by state and
fails, but another provider with the same type is successfully
installed, this almost definitely means that the user is migrating an
in-house provider. The solution here is to use the `terraform state
replace-provider` subcommand.

This commit makes that next step clearer, by detecting this specific
case, and displaying a list of commands to fix the existing state
provider references.
2020-09-01 14:02:19 -04:00
Alisdair McDiarmid fc7e467d19 command: Add redirect support to 0.13upgrade
If a provider changes namespace in the registry, we can detect this when
running the 0.13upgrade command. As long as there is a version matching
the user's constraints, we now use the provider's new source address.
Otherwise, warn the user that the provider has moved and a version
upgrade is necessary to move to it.
2020-08-31 14:53:35 -04:00
Alisdair McDiarmid f028b0a2bf command: Fix backend config schema validation
When applying a backend config override file, we must not check for the
presence of all required fields, as the override can be a partial
configuration. It is only valid to check for required fields after all
overrides have been merged, which init already does.
2020-08-26 10:50:47 -04:00
Alisdair McDiarmid c5d9935c0e
Merge pull request #25960 from hashicorp/alisdair/backend-config-override-fix
command: Fix backend config override validation
2020-08-24 10:14:16 -04:00
Alisdair McDiarmid 677aabc767 command: Fix backend config override validation
When loading a backend config override file, init was doing two things
wrong:

- First, if the file failed to parse, we accidentally didn't return,
  which caused a panic due to the parsed body being nil;
- Secondly, we were overzealous with the validation of the file,
  allowing only attributes. While most backend configs are attributes
  only, the enhanced remote backend body also contains a `workspaces`
  block, which we need to support here.

This commit fixes the first bug with an early return and adds test cases
for missing file and intentionally-blank filename (to clear the config).

We also add a schema validation for the backend block, based on the
backend schema itself. This requires constructing an HCL body schema so
that we can call `Content` and check for diagnostic errors.

The result is more useful errors when an invalid backend config override
file is used, while also supporting the enhanced remote backend config
fully.

Does not include tests specific to the remote backend, because the
mocking involved to allow the backend to fully initialize is too
involved to be worth it.
2020-08-21 16:21:13 -04:00
Alisdair McDiarmid b239570abb command: Always validate workspace name
The workspace name can be overridden by setting a TF_WORKSPACE
environment variable. If this is done, we should still validate the
resulting workspace name; otherwise, we could end up with an invalid and
unselectable workspace.

This change updates the Meta.Workspace function to return an error, and
handles that error wherever necessary.
2020-08-11 12:33:12 -04:00
Alisdair McDiarmid 53e587e1a6
Merge pull request #25504 from hashicorp/alisdair/post-install-cache-validate
Add post-install provider cache validation and error reporting
2020-07-09 14:55:17 -04:00
Martin Atkins 7909dd318d command/init: Specialized error message for host that isn't registry
If the user specifies a host that isn't a provider registry in a provider
source address then we'll print out some specialized error messages for
different variants of that situation.

In particular, this includes a special case for when the error is on the
hostname "github.com", in anticipation of folks incorrectly attempting to
use GitHub repository URLs (or Go-style module paths that happen to be
on GitHub) to specify providers, so we can give a more specific hint about
that.

This is just a different presentation of an existing error case that we
are already covering in the installer tests, so there are no new tests
here. We could in principle have a test covering the exact text of these
error messages, but we don't have much precedent for command package tests
covering that level of cosmetic detail.
2020-07-08 10:18:55 -07:00
Martin Atkins 80ab867e57 command/init: Remove special 0.12upgrade heuristic
For Terraform v0.12 we introduced a special loading mode where we would
use the 0.11-syntax-compatible "earlyconfig" package as a heuristic to
identify situations where it was likely that the user was trying to use
0.11-only syntax that the upgrade tool might help with.

However, as the language has moved on that is no longer a suitable
heuristic in Terraform 0.13 and later: other new additions to the
language can cause the main loader to disagree with earlyconfig, which
would lead us to give poor advice about how to respond.

Instead, we'll now return the same generic "there are errors" message in
all syntax error cases. We have an extra message for errors in this
case (as compared to other commands) because "terraform init" is usually
the first command a new user interacts with and so this message gives some
extra explanation about what "terraform init" will do with the
configuration once it's valid.

This also includes a reset control character in the output of the message
as part of our ongoing mission to stop Terraform printing out whole
paragraphs of colored text, which can often be hard to read for various
reasons.
2020-07-08 10:18:55 -07:00
Alisdair McDiarmid 87d1fb4006 command/init: Display provider validation errors
After installing providers, we validate the presence of an executable
file, and generate a selected versions lockfile. If this process fails,
notify the user. One possible cause for this is an invalid provider
package with a missing or misnamed executable file.
2020-07-07 15:20:20 -04:00
Kristin Laemmert df244b87c2
command/init: return an error with invalid -backend-config files (#25411)
* command/init: return an error with invalid -backend-config files

The -backend-config flag expects a set of key-value pairs or a file
containing key-value pairs. If the file instead contains a full backend
configuration block, it was silently ignored. This commit adds a check
for blocks in the file and returns an error if they are encountered.

Fixes #24845

* emphasize backend configuration file in docs
2020-06-26 12:49:31 -04:00
Kristin Laemmert 47e657c611
internal/getproviders: decode and return any registry warnings (#25337)
* internal/getproviders: decode and return any registry warnings

The public registry may include a list of warnings in the "versions"
response for any given provider. This PR adds support for warnings from
the registry and an installer event to return those warnings to the
user.
2020-06-25 10:49:48 -04:00
Alisdair McDiarmid 820ed48813 command: Fix init flags silent exit bug
When using `-flag=value` with Powershell, unquoted values are broken
into separate arguments. This means that the following command:

  terraform init -backend-config=./backend.conf

is interpreted by Terraform as:

  terraform init -backend-config= ./backend.conf

This results in an empty backend-config setting (which is semantically
valid!) followed by a custom configuration path (pointing at a file).

Due to a bug where we could exit without printing diagnostics, this
would result in a silent failure that was very difficult to diagnose.
2020-06-18 17:58:45 -04:00
Alisdair McDiarmid 9263b28e99 command/init: Improve diags for legacy providers
When initializing a configuration which refers to re-namespaced legacy
providers, we attempt to detect this and display a diagnostic message.
Previously this message would direct the user to run the 0.13upgrade
command, but without specifying in which directories.

This commit detects which modules are using the providers in question,
and for local modules displays a list of upgrade commands which specify
the source directories of these modules.

For remote modules, we display a separate list noting that they need to
be upgraded elsewhere, providing both the local module call name and the
module source address.
2020-06-12 09:57:01 -04:00
Alisdair McDiarmid ca40107066 command/init: Better diagnostics for provider 404s
Fetching a default namespace provider from the public registry can
result in 404 Not Found error. This might be caused by a previously-
default provider moving to a new namespace, which means that the
configuration needs to be upgraded to use an explicit provider source.

This commit adds a more detailed diagnostic for this situation,
suggesting that the intended provider might be in a new namespace. The
recommended course of action is to run the 0.13upgrade command to
generate the correct required_providers configuration.
2020-05-28 09:24:32 -04:00