Commit Graph

129 Commits

Author SHA1 Message Date
Martin Atkins 773dd56b42 internal/depsfile: Introduce the concept of "non-lockable" providers
It doesn't make sense for a built-in provider to appear in a lock file
because built-in providers have no version independent of the version of
Terraform they are compiled into.

We also exclude legacy providers here, because they were supported only
as a transitional aid to enable the Terraform 0.13 upgrade process and
are not intended for explicit selection.

The provider installer will, once it's updated to understand dependency
locking, use this concept to decide which subset of its selections to
record in the dependency lock file for reference for future installation
requests.
2020-09-08 09:50:58 -07:00
Martin Atkins 98e2e69abb internal/depsfile: SaveLocksToFile implementation
This is an initial implementation of writing locks back to a file on disk.
This initial implementation is incomplete because it does not write the
changes to the new file atomically. We'll revisit that in a later commit
as we return to polish these codepaths, once we've proven out this
package's design by integrating it with Terraform's provider installer.
2020-09-08 09:50:58 -07:00
Martin Atkins 92723661d0 internal/depsfile: Loading locks from HCL files on disk
This is the initial implementation of the parser/decoder portion of the
new dependency lock file handler. It's currently dead code because the
caller isn't written yet. We'll continue to build out this functionality
here until we have the basic level of both load and save functionality
before introducing this into the provider installer codepath.
2020-09-08 09:50:58 -07:00
Martin Atkins 6993ecb0a6 internal/getproviders: VersionConstraintsString for "~> 2" input
The version constraint parser allows "~> 2", but it behavior is identical
to "~> 2.0". Due to a quirk of the constraint parser (caused by the fact
that it supports both Ruby-style and npm/cargo-style constraints), it
ends up returning "~> 2" with the minor version marked as "unconstrained"
rather than as zero, but that means the same thing as zero in this context
anyway and so we'll prefer to stringify as "~> 2.0" so that we can be
clearer about how Terraform is understanding that version constraint.
2020-09-08 09:50:58 -07:00
Alisdair McDiarmid 5587509bcf internal: Fix providercache test failures on macOS
For reasons that are unclear, these two tests just started failing on
macOS very recently. The failure looked like:

    PackageDir: strings.Join({
      "/",
+     "private/",
      "var/folders/3h/foobar/T/terraform-test-p",
      "rovidercache655312854/registry.terraform.io/hashicorp/null/2.0.0",
      "/windows_amd64",
    },

Speculating that the macOS temporary directory moved into the /private
directory, I added a couple of EvalSymlinks calls and the tests pass
again.

No other unit tests appear to be affected by this at the moment.
2020-09-04 16:09:57 -04:00
Alisdair McDiarmid fc7e467d19 command: Add redirect support to 0.13upgrade
If a provider changes namespace in the registry, we can detect this when
running the 0.13upgrade command. As long as there is a version matching
the user's constraints, we now use the provider's new source address.
Otherwise, warn the user that the provider has moved and a version
upgrade is necessary to move to it.
2020-08-31 14:53:35 -04:00
Alan D. Salewski f63c38d6a1
internal/initwd: allow tests to pass when $PWD contains symlinks (#26015)
Fixes #26014
2020-08-27 10:02:22 -04:00
Martin Atkins 2bd2a9a923 internal/getproviders: HTTPMirrorSource implementation
We previously had this just stubbed out because it was a stretch goal for
the v0.13.0 release and it ultimately didn't make it in.

Here we fill out the existing stub -- with a minor change to its interface
so it can access credentials -- with a client implementation that is
compatible with the directory structure produced by the
"terraform providers mirror" subcommand, were the result to be published
on a static file server.
2020-08-26 13:18:08 -07:00
Martin Atkins 146e983c36 internal/getproviders: package authenticator for our new-style hashes
Earlier we introduced a new package hashing mechanism that is compatible
with both packed and unpacked packages, because it's a hash of the
contents of the package rather than of the archive it's delivered in.
However, we were using that only for the local selections file and not
for any remote package authentication yet.

The provider network mirrors protocol includes new-style hashes as a step
towards transitioning over to the new hash format in all cases, so this
new authenticator is here in preparation for verifying the checksums of
packages coming from network mirrors, for mirrors that support them.

For now this leaves us in a kinda confusing situation where we have both
NewPackageHashAuthentication for the new style and
NewArchiveChecksumAuthentication for the old style, which for the moment
is represented only by a doc comment on the latter. Hopefully we can
remove NewArchiveChecksumAuthentication in a future commit, if we can
get the registry updated to use the new hashing format.
2020-08-26 13:18:08 -07:00
Alisdair McDiarmid 915f53af23 internal: Clean up package install temp file
The installFromHTTPURL function downloads a package to a temporary file,
then delegates to installFromLocalArchive to install it. We were
previously not deleting the temporary file afterwards. This commit fixes
that.
2020-08-25 08:35:32 -04:00
Martin Atkins ce67a818db internal/getproviders: Allow basedir for local search to be symlink
The SearchLocalDirectory function was intentionally written to only
support symlinks at the leaves so that it wouldn't risk getting into an
infinite loop traversing intermediate symlinks, but that rule was also
applying to the base directory itself.

It's pretty reasonable to put your local plugins in some location
Terraform wouldn't normally search (e.g. because you want to get them from
a shared filesystem mounted somewhere) and creating a symlink from one
of the locations Terraform _does_ search is a convenient way to help
Terraform find those without going all in on the explicit provider
installation methods configuration that is intended for more complicated
situations.

To allow for that, here we make a special exception for the base
directory, resolving that first before we do any directory walking.

In order to help with debugging a situation where there are for some
reason symlinks at intermediate levels inside the search tree, we also now
emit a WARN log line in that case to be explicit that symlinks are not
supported there and to hint to put the symlink at the top-level if you
want to use symlinks at all.

(The support for symlinks at the deepest level of search is not mentioned
in this message because we allow it primarily for our own cache linking
behavior.)
2020-08-18 14:28:50 -07:00
Alisdair McDiarmid 440543f427 internal/providercache: Fix bug when symlink fails
When installing a provider which is already cached, we attempt to create
a symlink from the install directory targeting the cache. If symlinking
fails due to missing OS/filesystem support, we instead want to copy the
cached provider.

The fallback code to do this would always fail, due to a missing target
directory. This commit fixes that. I was unable to find a way to add
automated tests around this, but I have manually verified the fix on
Windows 8.1.
2020-07-23 11:36:46 -04:00
Alisdair McDiarmid 3b1347ac1a providercache: Validate provider executable file
At the end of the EnsureProviderVersions process, we generate a lockfile
of the selected and installed provider versions. This includes a hash of
the unpacked provider directory.

When calculating this hash and generating the lockfile, we now also
verify that the provider directory contains a valid executable file. If
not, we return an error for this provider and trigger the installer's
HashPackageFailure event. Note that this event is not yet processed by
terraform init; that comes in the next commit.
2020-07-07 15:20:17 -04:00
Alisdair McDiarmid a18b531b14 getproviders: FakeInstallablePackageMeta filename
Add an optional execFilename argument to the test helper function
FakeInstallablePackageMeta, which allows the creation of invalid
packages.
2020-07-07 15:18:30 -04:00
Alisdair McDiarmid 8e87ccb689 providercache: Lazily detect executable file
Instead of searching the installed provider package directory for a
binary as we install it, we can lazily detect the executable as it is
required. Doing so allows us to separately report an invalid unpacked
package, giving the user more actionable error messages.
2020-07-07 15:18:27 -04:00
Kristin Laemmert 47e657c611
internal/getproviders: decode and return any registry warnings (#25337)
* internal/getproviders: decode and return any registry warnings

The public registry may include a list of warnings in the "versions"
response for any given provider. This PR adds support for warnings from
the registry and an installer event to return those warnings to the
user.
2020-06-25 10:49:48 -04:00
Kristin Laemmert 1b8f4566fa
internal/initwd: fix panics with relative submodules in DirFromModule (#25250)
* internal/initwd: fix panics with relative submodules in DirFromModule

There were two related issues here:

1. panic with any local module with submodules
1. panic with a relative directory that was above the workdir ("../")

The first panic was caused by the local installer looking up the root
module with the (nonexistant) key "root.", instead of "".

The second panic was caused by the installer trying to determine the
relative path from ".". This was fixed by detecting "." as the source
path and using the absolute path for the call to filepath.Rel.

Added test cases for both panics and updated the existing e2e tests with
the correct install paths.
2020-06-17 13:24:56 -04:00
Lars Lehtonen 3ddfa66ca4
internal/modsdir: Fix Dropped Error (#24600)
* internal/modsdir: fix dropped error

* fix typo to unmarshalling

Co-authored-by: Daniel Dreier <danieldreier@users.noreply.github.com>
2020-06-04 10:26:26 -04:00
Martin Atkins 85af77386c internal/getproviders: PackageFilePathForPackage
This is the equivalent of UnpackedDirectoryPathForPackage when working
with the packed directory layout. It returns a path to a .zip file with
a name that would be detected by SearchLocalDirectory as a
PackageLocalArchive package.
2020-06-01 14:49:43 -07:00
Martin Atkins 9489672d54 internal/getproviders: Package hashing for local filesystem packages
We previously had this functionality available for cached packages in the
providercache package. This moves the main implementation of this over
to the getproviders package and then implements it also for PackageMeta,
allowing us to compute hashes in a consistent way across both of our
representations of a provider package.

The new methods on PackageMeta will only be effective for packages in the
local filesystem because we need direct access to the contents in order
to produce the hash. Hopefully in future the registry protocol will be
able to also provide hashes using this content-based (rather than
archive-based) algorithm and then we'll be able to make this work for
PackageMeta referring to a package obtained from a registry too, but
hashes for local packages only are still useful for some cases right now,
such as generating mirror directories in the "terraform providers mirror"
command.
2020-06-01 14:49:43 -07:00
Alisdair McDiarmid ef28671b34
Merge pull request #24932 from hashicorp/signing-language
Modify language for reporting signing state
2020-05-28 09:09:34 -04:00
Paddy 5127f1ef8b
command: Unmanaged providers
This adds supports for "unmanaged" providers, or providers with process
lifecycles not controlled by Terraform. These providers are assumed to
be started before Terraform is launched, and are assumed to shut
themselves down after Terraform has finished running.

To do this, we must update the go-plugin dependency to v1.3.0, which
added support for the "test mode" plugin serving that powers all this.

As a side-effect of not needing to manage the process lifecycle anymore,
Terraform also no longer needs to worry about the provider's binary, as
it won't be used for anything anymore. Because of this, we can disable
the init behavior that concerns itself with downloading that provider's
binary, checking its version, and otherwise managing the binary.

This is all managed on a per-provider basis, so managed providers that
Terraform downloads, starts, and stops can be used in the same commands
as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable
is added, and is a JSON encoding of the provider's address to the
information we need to connect to it.

This change enables two benefits: first, delve and other debuggers can
now be attached to provider server processes, and Terraform can connect.
This allows for attaching debuggers to provider processes, which before
was difficult to impossible. Second, it allows the SDK test framework to
host the provider in the same process as the test driver, while running
a production Terraform binary against the provider. This allows for Go's
built-in race detector and test coverage tooling to work as expected in
provider tests.

Unmanaged providers are expected to work in the exact same way as
managed providers, with one caveat: Terraform kills provider processes
and restarts them once per graph walk, meaning multiple times during
most Terraform CLI commands. As unmanaged providers can't be killed by
Terraform, and have no visibility into graph walks, unmanaged providers
are likely to have differences in how their global mutable state behaves
when compared to managed providers. Namely, unmanaged providers are
likely to retain global state when managed providers would have reset
it. Developers relying on global state should be aware of this.
2020-05-26 17:48:57 -07:00
Paul Tyng 22ef5cc99c Modify language for reporting signing state
Be more explicit about the signing status of fetched plugins and provide documentation about the different signing options.
2020-05-26 13:14:05 -04:00
Alisdair McDiarmid 62d826e066 command/init: Use full config for provider reqs
Relying on the early config for provider requirements was necessary in
Terraform 0.12, to allow the 0.12upgrade command to run after init
installs providers.

However in 0.13, the same restrictions do not apply, and the detection
of provider requirements has changed. As a result, the early config
loader gives incorrect provider requirements in some circumstances,
such as those in the new test in this commit.

Therefore we are changing the init command to use the requirements found
by the full configuration loader. This also means that we can remove the
internal initwd CheckCoreVersionRequirements function.
2020-05-25 16:50:12 -04:00
Kristin Laemmert eead4c49fe command/init: add e2e tests for provider not found messages 2020-05-20 11:04:11 -04:00
Kristin Laemmert 8d28d73de3 getproviders: add a registry-specific error and modify output when a
provider is not found.

Previously a user would see the following error even if terraform was
only searching the local filesystem:

"provider registry registry.terraform.io does not have a provider named
...."

This PR adds a registry-specific error type and modifies the MultiSource
installer to check for registry errors. It will return the
registry-specific error message if there is one, but if not the error
message will list all locations searched.
2020-05-20 11:04:11 -04:00
Kristin Laemmert 0d620018fe
provider cache: log errors and validate dir exists (#24993)
* providercache: add logging for errors from getproviders.SearchLocalDirectory

providercache.fillMetaCache() was silently swallowing errors when
searching the cache directory. This commit logs the error without
changing the behavior otherwise.

* command/cliconfig: validate plugin cache dir exists

The plugin cache directory must exist for terraform to use it, so we
will add a check at the begining.
2020-05-19 15:32:36 -04:00
Kristin Laemmert d350818126
internal/getproviders: fix panic with invalid path parts (#24940)
* internal/getproviders: fix panic with invalid path parts

If the search path is missing a directory, the provider installer would
try to create an addrs.Provider with the wrong parts. For example if the
hostname was missing (as in the test case), it would call
addrs.NewProvider with (namespace, typename, version). This adds a
validation step for each part before calling addrs.NewProvider to avoid
the panic.
2020-05-13 13:16:09 -04:00
Alisdair McDiarmid 070c3018f8 internal/providercache: Remove unused retry events 2020-05-13 09:48:41 -04:00
Alisdair McDiarmid e27a36cafd internal/getproviders: Retry failed HTTP requests
This is a port of the retry/timeout logic added in #24260 and #24259,
using the same environment variables to configure the retry and timeout
settings.
2020-05-13 09:48:41 -04:00
Kristin Laemmert 60321b41e8
getproviders: move protocol compatibility functions into registry client (#24846)
* internal/registry source: return error if requested provider version protocols are not supported

* getproviders: move responsibility for protocol compatibility checks into the registry client

The original implementation had the providercache checking the provider
metadata for protocol compatibility, but this is only relevant for the
registry source so it made more sense to move the logic into
getproviders.

This also addresses an issue where we were pulling the metadata for
every provider version until we found one that was supported. I've
extended the registry client to unmarshal the protocols in
`ProviderVersions` so we can filter through that list, instead of
pulling each version's metadata.
2020-05-11 13:49:12 -04:00
Alisdair McDiarmid 62b0cbed12 internal: Fix LookupLegacyProvider
When looking up the namespace for a legacy provider source, we need to
use the /v1/providers/-/{name}/versions endpoint. For non-HashiCorp
providers, the /v1/providers/-/{name} endpoint returns a 404.

This commit updates the LegacyProviderDefaultNamespace method and the
mock registry servers accordingly.
2020-05-08 12:29:25 -04:00
Alisdair McDiarmid ae98bd12a7 command: Rework 0.13upgrade sub-command
This commit implements most of the intended functionality of the upgrade
command for rewriting configurations.

For a given module, it makes a list of all providers in use. Then it
attempts to detect the source address for providers without an explicit
source.

Once this step is complete, the tool rewrites the relevant configuration
files. This results in a single "required_providers" block for the
module, with a source for each provider.

Any providers for which the source cannot be detected (for example,
unofficial providers) will need a source to be defined by the user. The
tool writes an explanatory comment to the configuration to help with
this.
2020-05-07 11:38:55 -04:00
Kristin Laemmert cca0526705
providercache: actually break out of the loop when a matching version is found (#24823) 2020-05-01 08:49:47 -04:00
Kristin Laemmert ce03f1255f
internal/providercache: fix error message for protocol mismatch (#24818)
There was a bug in the installer trying to pass a nil error.
2020-04-30 11:12:04 -04:00
Kristin Laemmert 320fcf4942
internal/getproviders: apply case normalizations in ParseMultiSourceMatchingPatterns (#24753)
* internal/getproviders: apply case normalizations in ParseMultiSourceMatchingPatterns

This is a very minor refactor which takes advantage of addrs.ParseProviderPart case normalization to normalize non-wildcard sources.
2020-04-23 14:50:47 -04:00
Martin Atkins 1ce3c60693
command/cliconfig: Explicit provider installation method configuration
This set of commits allows explicit configuration of provider installation methods
in the CLI config, overriding the implicit method selections.
2020-04-23 10:58:00 -07:00
Martin Atkins 8b75d1498f command/cliconfig: Use existing HTTP mirror source rather than new stub
An earlier commit added a redundant stub for a new network mirror source
that was already previously stubbed as HTTPMirrorSource.

This commit removes the unnecessary extra stub and changes the CLI config
handling to use it instead. Along the way this also switches to using a
full base URL rather than just a hostname for the mirror, because using
the usual "Terraform-native service discovery" protocol here doesn't isn't
as useful as in the places we normally use it (the mirror mechanism is
already serving as an indirection over the registry protocol) and using
a direct base URL will make it easier to deploy an HTTP mirror under
a path prefix on an existing static file server.
2020-04-23 10:52:01 -07:00
Kristin Laemmert 84e9d86c25
Update installer_test.go
update now-exported function (fix bad PR)
2020-04-23 08:33:28 -04:00
Kristin Laemmert 21b9da5a02
internal/providercache: verify that the provider protocol version is compatible (#24737)
* internal/providercache: verify that the provider protocol version is
compatible

The public registry includes a list of supported provider protocol
versions for each provider version. This change adds verification of
support and adds a specific error message pointing users to the closest
matching version.
2020-04-23 08:21:56 -04:00
Alisdair McDiarmid 54abb87fb4 Fix broken test due to function rename 2020-04-22 10:52:49 -04:00
Martin Atkins c5bd783eba internal/getproviders: Stub NetworkMirrorSource
This is a placeholder for later implementation of a mirror source that
talks to a particular remote HTTP server and expects it to implement the
provider mirror protocol.
2020-04-21 16:27:48 -07:00
Kristin Laemmert a43f141f9e
tools/terraform-bundle: refactor to use new provider installer and provider directory layouts (#24629)
* tools/terraform-bundle: refactor to use new provider installer and
provider directory layouts

terraform-bundle now supports a "source" attribute for providers,
uses the new provider installer, and the archive it creates preserves
the new (required) directory hierarchy for providers, under a "plugins"
directory.

This is a breaking change in many ways: source is required for any
non-HashiCorp provider, locally-installed providers must be given a
source (can be arbitrary, see docs) and placed in the expected directory
hierarchy, and the unzipped archive is no longer flat; there is a new
"plugins" directory created with providers in the new directory layout.

This PR also extends the existing test to check the contents of the zip
file.

TODO: Re-enable e2e tests (currently suppressed with a t.Skip)
This commit includes an update to our travis configuration, so the terraform-bundle e2e tests run. It also turns off the e2e tests, which will fail until we have a terraform 0.13.* release under releases.hashicorp.com. We decided it was better to merge this now instead of waiting when we started seeing issues opened from users who built terraform-bundle from 0.13 and found it didn't work with 0.12 - better that they get an immediate error message from the binary directing them to build from the appropriate release.
2020-04-21 17:09:29 -04:00
Alisdair McDiarmid a5b3d497cc internal: Verify provider signatures on install
Providers installed from the registry are accompanied by a list of
checksums (the "SHA256SUMS" file), which is cryptographically signed to
allow package authentication. The process of verifying this has multiple
steps:

- First we must verify that the SHA256 hash of the package archive
  matches the expected hash. This could be done for local installations
  too, in the future.
- Next we ensure that the expected hash returned as part of the registry
  API response matches an entry in the checksum list.
- Finally we verify the cryptographic signature of the checksum list,
  using the public keys provided by the registry.

Each of these steps is implemented as a separate PackageAuthentication
type. The local archive installation mechanism uses only the archive
checksum authenticator, and the HTTP installation uses all three in the
order given.

The package authentication system now also returns a result value, which
is used by command/init to display the result of the authentication
process.

There are three tiers of signature, each of which is presented
differently to the user:

- Signatures from the embedded HashiCorp public key indicate that the
  provider is officially supported by HashiCorp;
- If the signing key is not from HashiCorp, it may have an associated
  trust signature, which indicates that the provider is from one of
  HashiCorp's trusted partners;
- Otherwise, if the signature is valid, this is a community provider.
2020-04-17 13:57:19 -04:00
Kristin Laemmert f09ae6f862
provider source tests: added test suite to exercise hyphenated providers (#24685) 2020-04-16 15:54:33 -04:00
Martin Atkins 6ec5e0fc88 internal/getproviders: Tests for MemoizeSource
Due to other pressures at the time this was implemented, it was tested
only indirectly through integration tests in other packages. This now
introduces tests for the two main entry points on MemoizeSource.
2020-04-07 08:55:52 -07:00
Martin Atkins c6535a0fa4 internal/getproviders: Tests for MultiSource
Due to other pressures at the time this was implemented, it was tested
only indirectly through integration tests in other packages.

This now introduces tests for the two main entry points on the
MultiSource, along with its provider-address pattern matching logic.

This does not yet include thorough tests for
ParseMultiSourceMatchingPatterns, because that function still needs some
adjustments to do the same case folding as for normal provider address
parsing, which will follow in a latter commit along with suitable tests.

With that said, the tests added here do _indirectly_ test the happy path
of ParseMultiSourceMatchingPatterns, so we have some incomplete testing
of that function in the meantime.
2020-04-07 08:55:13 -07:00
Martin Atkins 0ad4c1be2f internal/getproviders: Tidy up some confusion about package hashes
Earlier on in the stubbing of this package we realized that it wasn't
going to be possible to populate the authentication-related bits for all
packages because the relevant metadata just isn't available for packages
that are already local.

However, we just moved ahead with that awkward design at the time because
we needed to get other work done, and so we've been mostly producing
PackageMeta values with all-zeros hashes and just ignoring them entirely
as a temporary workaround.

This is a first step towards what is hopefully a more intuitive model:
authentication is an optional thing in a PackageMeta that is currently
populated only for packages coming from a registry.

So far this still just models checking a SHA256 hash, which is not a
sufficient set of checks for a real release but hopefully the "real"
implementation is a natural iteration of this starting point, and if not
then at least this interim step is a bit more honest about the fact that
Authentication will not be populated on every PackageMeta.
2020-04-06 16:31:23 -07:00
Alisdair McDiarmid 0f5a38b384 internal: Fix init provider lockfile test
The fake installable package meta used a ZIP archive which gave
different checksums between macOS and Linux targets. This commit removes
the target from the contents of this archive, and updates the golden
hash value in the test to match. This test should now pass on both
platforms.
2020-04-06 09:24:23 -07:00
Martin Atkins 958ea4f7d1 internal/providercache: Handle built-in providers
Built-in providers are special providers that are distributed as part of
Terraform CLI itself, rather than being installed separately. They always
live in the terraform.io/builtin/... namespace so it's easier to see that
they are special, and currently there is only one built-in provider named
"terraform".

Previous commits established the addressing scheme for built-in providers.
This commit makes the installer aware of them to the extent that it knows
not to try to install them the usual way and it's able to report an error
if the user requests a built-in provider that doesn't exist or tries to
impose a particular version constraint for a built-in provider.

For the moment the tests for this are the ones in the "command" package
because that's where the existing testing infrastructure for this
functionality lives. A later commit should add some more focused unit
tests here in the internal/providercache package, too.
2020-04-06 09:24:23 -07:00