I frequently see people attempting to ask questions about Terraform's
error and warning messages but either only copying part of the message or
accidentally copying a surrounding paragraph that isn't part of the
message.
While I'm sure some of these are just "careless" mistakes, I've also
noticed that this has sometimes overlapped with someone asking a question
whose answer is written directly in the part of the message they didn't
include when copying, and so I have a theory that our current output
doesn't create a good enough visual hierarchy for sighted users to
understand where the diagnostic messages start and end when we show them
in close proximity to other content, or to other diagnostic messages.
As a result, some folks fail to notice the relevant message that might've
answered their question.
I tried a few different experiments for different approaches here, such
as adding more horizontal rules to the output and coloring the detail
text differently, but the approach that felt like the nicest compromise
to me was what's implemented here, which is to add a vertical line
along the left edge of each diagnostic message, colored to match with the
typical color we use for each diagnostic severity. This means that the
diagnostics end up slightly indented from what's around them, and the
vertical line seems to help subtly signal how we intended the content
to be grouped together.
If you run the e2etests locally and use a configured plugin_cache_dir,
the test will leave a bad directory behind in your cache dir that causes
later `init`s to fail. To circumvent this, pass an explicity-empty CLI
config file.
This is a nicety for local developers and not necessarily required, but
it happens to me often enough that I'd like to fix it. It's probably not
a *bad* idea to pass an explicit cli config to all e2etests, honestly,
but this is the only one that causes active problems so I limited this
PR to that one test.
Here's the error which occurs on subsequent `init` if this test is run on a
machine that uses a plugin cache dir:
2020/10/13 10:41:05 [TRACE] providercache.fillMetaCache: error while scanning directory /Users/mildwonkey/.terraform.d/plugin-cache: failed to read metadata about /Users/mildwonkey/.terraform.d/plugin-cache/example.com/awesomecorp/happycloud/1.2.0/darwin_amd64: stat /Users/mildwonkey/.terraform.d/plugin-cache/example.com/awesomecorp/happycloud/1.2.0/darwin_amd64: no such file or directory
Terraform v0.10 introduced .terraform/plugins as a cache directory for
automatically-installed plugins, Terraform v0.13 later reorganized the
directory structure inside but retained its purpose as a cache.
The local cache used to also serve as a record of specifically which
packages were selected in a particular working directory, with the intent
that a second run of "terraform init" would always select the same
packages again. That meant that in some sense it behaved a bit like a
local filesystem mirror directory, even though that wasn't its intended
purpose.
Due to some unfortunate miscommunications, somewhere a long the line we
published some documentation that _recommended_ using the cache directory
as if it were a filesystem mirror directory when working with Terraform
Cloud. That was really only working as an accident of implementation
details, and Terraform v0.14 is now going to break that because the source
of record for the currently-selected provider versions is now the
public-facing dependency lock file rather than the contents of an existing
local cache directory on disk.
After some consideration of how to move forward here, this commit
implements a compromise that tries to avoid silently doing anything
surprising while still giving useful guidance to folks who were previously
using the unsupported strategy. Specifically:
- The local cache directory will now be .terraform/providers rather than
.terraform/plugins, because .terraform/plugins is effectively "poisoned"
by the incorrect usage that we can't reliably distinguish from prior
version correct usage.
- The .terraform/plugins directory is now the "legacy cache directory". It
is intentionally _not_ now a filesystem mirror directory, because that
would risk incorrectly interpreting providers automatically installed
by Terraform v0.13 as if they were a local mirror, and thus upgrades
and checksum fetches from the origin registry would be blocked.
- Because of the previous two points, someone who _was_ trying to use the
legacy cache directory as a filesystem mirror would see installation
fail for any providers they manually added to the legacy directory.
To avoid leaving that user stumped as to what went wrong, there's a
heuristic for the case where a non-official provider fails installation
and yet we can see it in the legacy cache directory. If that heuristic
matches then we'll produce a warning message hinting to move the
provider under the terraform.d/plugins directory, which is a _correct_
location for "bundled" provider plugins that belong only to a single
configuration (as opposed to being installed globally on a system).
This does unfortunately mean that anyone who was following the
incorrectly-documented pattern will now encounter an error (and the
aforementioned warning hint) after upgrading to Terraform v0.14. This
seems like the safest compromise because Terraform can't automatically
infer the intent of files it finds in .terraform/plugins in order to
decide automatically how best to handle them.
The internals of the .terraform directory are always considered
implementation detail for a particular Terraform version and so switching
to a new directory for the _actual_ cache directory fits within our usual
set of guarantees, though it's definitely non-ideal in isolation but okay
when taken in the broader context of this problem, where the alternative
would be silent misbehavior when upgrading.
This changes the approach used by the provider installer to remember
between runs which selections it has previously made, using the lock file
format implemented in internal/depsfile.
This means that version constraints in the configuration are considered
only for providers we've not seen before or when -upgrade mode is active.
We no longer need to support 0.12-and-earlier-style provider addresses
because users should've upgraded their existing configurations and states
on Terraform 0.13 already.
For now this is only checked in the "init" command, because various test
shims are still relying on the idea of legacy providers the core layer.
However, rejecting these during init is sufficient grounds to avoid
supporting legacy provider addresses in the new dependency lock file
format, and thus sets the stage for a more severe removal of legacy
provider support in a later commit.
The tmp directory path is longer on Macs than other systems and was wrapping
across lines when printed, breaking the string match in the test.
Fix suggested by @apparentlymart is to add two spaces before the leading
'-' when printing to prevent the diagnostic renderer wrapping the line.
* internal/getproviders: decode and return any registry warnings
The public registry may include a list of warnings in the "versions"
response for any given provider. This PR adds support for warnings from
the registry and an installer event to return those warnings to the
user.
Fetching a default namespace provider from the public registry can
result in 404 Not Found error. This might be caused by a previously-
default provider moving to a new namespace, which means that the
configuration needs to be upgraded to use an explicit provider source.
This commit adds a more detailed diagnostic for this situation,
suggesting that the intended provider might be in a new namespace. The
recommended course of action is to run the 0.13upgrade command to
generate the correct required_providers configuration.
This exercises the ability to customize the installation methods used by
the provider plugin installer, in this case forcing the use of a custom
local directory with a result essentially the same as what happens when
you pass -plugin-dir to "terraform init".
This more closely replicates the 0.12-and-earlier behavior, where having
at least one version of a provider installed locally would totally disable
any attempt to look for newer versions remotely.
This is just for the implicit default behavior. Assumption is that later
we'll have an explicit configuration mechanism that will allow the user
to specify exactly where to look for what, and thus avoid tricky
heuristics like this.
This restores some of the local search directories we used to include when
searching for provider plugins in Terraform 0.12 and earlier. The
directory structures we are expecting in these are different than before,
so existing directory contents will not be compatible without
restructuring, but we need to retain support for these local directories
so that users can continue to sideload third-party provider plugins until
the explicit, first-class provider mirrors configuration (in CLI config)
is implemented, at which point users will be able to override these to
whatever directories they want.
This also includes some new search directories that are specific to the
operating system where Terraform is running, following the documented
layout conventions of that platform. In particular, this follows the
XDG Base Directory specification on Unix systems, which has been a
somewhat-common request to better support "sideloading" of packages via
standard Linux distribution package managers and other similar mechanisms.
While it isn't strictly necessary to add that now, it seems ideal to do
all of the changes to our search directory layout at once so that our
documentation about this can cleanly distinguish "0.12 and earlier" vs.
"0.13 and later", rather than having to document a complex sequence of
smaller changes.
Because this behavior is a result of the integration of package main with
package command, this behavior is verified using an e2etest rather than
a unit test. That test, TestInitProvidersVendored, is also fixed here to
create a suitable directory structure for the platform where the test is
being run. This fixes TestInitProvidersVendored.
In the new design the ProviderSource is decided by package main, not by
the "command" package, and so making sure the vendor directory is included
is the responsibility of that package instead. Therefore we can no longer
test this at the "command" package level, but we'll retain a test for it
in e2etests to record that it isn't currently working, so that we have
a prompt to fix it before releasing.
Due to some incomplete rework of this function in an earlier commit, the
safety check for using the same directory as both the target and the
cache was inverted and was raising an error _unless_ they matched, rather
than _if_ they matched.
This change is verified by the e2etest TestInitProviders_pluginCache,
which is also updated to use the new-style cache directory layout as part
of this commit.
These tests make assertions against specific user-oriented output from the
"terraform init" command, but we've intentionally changed some of these
messages as part of introducing support for the decentralized provider
namespace.
The canonical location of the "template" provider is now in the hashicorp
namespace rather than the terraform-providers namespace, so the output
has changed to reflect that.
* internal/initwd: Allow deprecated relative module paths
In Terraform 0.11 we deprecated this form but didn't have any explicit
warning for it. Now we'll still accept it but generate a warning. In a
future major release we will drop this form altogether, since it is
ambiguous with registry module source addresses.
This codepath is covered by the command/e2etest suite.
* e2e: Skip copying .exists file, if present
We use this only in the "empty" test fixture in order to let git know that
the directory exists. We need to skip copying it so that we can test
"terraform init -from-module=...", which expects to find an empty
directory.
* command/e2etests: Re-enable and fix up the e2etest "acctests"
We disabled all of the tests that accessed remote services like the
Terraform Registry while they were being updated to support the new
protocols we now expect. With those services now in place, we can
re-enable these tests.
Some details of exactly what output we print, etc, have intentionally
changed since these tests were last updated.
* e2e: refactor for modern states and plans
* command/e2etest: re-enable e2etests and update for tf 0.12 compatibility
plugin/discovery: mkdirAll instead of mkdir when creating cache dir
As part of the 0.10 core/provider split we moved this provider, along with
all the others, out into its own repository.
In retrospect, the "terraform" provider doesn't really make sense to be
separated since it's just a thin wrapper around some core code anyway,
and so re-integrating it into core avoids the confusion that results when
Terraform Core and the terraform provider have inconsistent versions of
the backend code and dependencies.
There is no good reason to use a different version of the backend code
in the provider than in core, so this new "internal provider" mechanism
is stricter than the old one: it's not possible to use an external build
of this provider at all, and version constraints for it are rejected as
a result.
This provider is also run in-process rather than in a child process, since
again it's just a very thin wrapper around code that's already running
in Terraform core anyway, and so the process barrier between the two does
not create enough advantage to warrant the additional complexity.
This is a tough one to unit tests because the behavior is tangled up in
the code that hits releases.hashicorp.com, so we'll add this e2etest as
some extra insurance that this works end-to-end.
We already have good tests for the business logic around provider
installation, but the existing tests all stub out the main repository
server. This test completes that coverage by verifying that the installer
is able to run against the real repository and install an official release
of the template provider.
Martin Atkins