* getproviders ParsePlatform: add check for invalid platform strings with too many parts
The existing logic would not catch things like a platform string containing multiple underscores. I've added an explicit check for exactly 2 parts and some basic tests to prove it.
* command/providers-lock: add tests
This commit adds some simple tests for the providers lock command. While adding this test I noticed that there was a mis-copied error message, so I replaced that with a more specific message. I also added .terraform.lock.hcl to our gitignore for hopefully obvious reasons.
getproviders.ParsePlatform: use parts in place of slice range, since it's available
* command: Providers mirror tests
The providers mirror command is already well tested in e2e tests, so this includes only the most absolutely basic test case.
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.
If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
When rendering a set of version constraints to a string, we normalize
partially-constrained versions. This means converting a version
like 2.68.* to 2.68.0.
Prior to this commit, this normalization was done after deduplication.
This could result in a version constraints string with duplicate
entries, if multiple partially-constrained versions are equivalent. This
commit fixes this by normalizing before deduplicating and sorting.
An earlier commit made this remove duplicates, which set the precedent
that this function is trying to canonically represent the _meaning_ of
the version constraints rather than exactly how they were expressed in
the configuration.
Continuing in that vein, now we'll also apply a consistent (though perhaps
often rather arbitrary) ordering to the terms, so that it doesn't change
due to irrelevant details like declarations being written in a different
order in the configuration.
The ordering here is intended to be reasonably intuitive for simple cases,
but constraint strings with many different constraints are hard to
interpret no matter how we order them so the main goal is consistency,
so those watching how the constraints change over time (e.g. in logs of
Terraform output, or in the dependency log file) will see fewer noisy
changes that don't actually mean anything.
A set of version constraints can contain duplicates. This can happen if
multiple identical constraints are specified throughout a configuration.
When rendering the set, it is confusing to display redundant
constraints. This commit changes the string renderer to only show the
first instance of a given constraint, and adds unit tests for this
function to cover this change.
This also fixes a bug with the locks file generation: previously, a
configuration with redundant constraints would result in this error on
second init:
Error: Invalid provider version constraints
on .terraform.lock.hcl line 6:
(source code not available)
The recorded version constraints for provider
registry.terraform.io/hashicorp/random must be written in normalized form:
"3.0.0".
If a configuration requires a partial provider version (with some parts
unspecified), Terraform considers this as a constrained-to-zero version.
For example, a version constraint of 1.2 will result in an attempt to
install version 1.2.0, even if 1.2.1 is available.
When writing the dependency locks file, we previously would write 1.2.*,
as this is the in-memory representation of 1.2. This would then cause an
error on re-reading the locks file, as this is not a valid constraint
format.
Instead, we now explicitly convert the constraint to its zero-filled
representation before writing the locks file. This ensures that it
correctly round-trips.
Because this change is made in getproviders.VersionConstraintsString, it
also affects the output of the providers sub-command.
As we continue iterating towards saving valid hashes for a package in a
depsfile lock file after installation and verifying them on future
installation, this prepares getproviders for the possibility of having
multiple valid hashes per package.
This will arise in future commits for two reasons:
- We will need to support both the legacy "zip hash" hashing scheme and
the new-style content-based hashing scheme because currently the
registry protocol is only able to produce the legacy scheme, but our
other installation sources prefer the content-based scheme. Therefore
packages will typically have a mixture of hashes of both types.
- Installing from an upstream registry will save the hashes for the
packages across all supported platforms, rather than just the current
platform, and we'll consider all of those valid for future installation
if we see both successful matching of the current platform checksum and
a signature verification for the checksums file as a whole.
This also includes some more preparation for the second case above in that
signatureAuthentication now supports AcceptableHashes and returns all of
the zip-based hashes it can find in the checksums file. This is a bit of
an abstraction leak because previously that authenticator considered its
"document" to just be opaque bytes, but we want to make sure that we can
only end up trusting _all_ of the hashes if we've verified that the
document is signed. Hopefully we'll make this better in a future commit
with some refactoring, but that's deferred for now in order to minimize
disruption to existing codepaths while we work towards a provider locking
MVP.
The logic for what constitutes a valid hash and how different hash schemes
are represented was starting to get sprawled over many different files and
packages.
Consistently with other cases where we've used named types to gather the
definition of a particular string into a single place and have the Go
compiler help us use it properly, this introduces both getproviders.Hash
representing a hash value and getproviders.HashScheme representing the
idea of a particular hash scheme.
Most of this changeset is updating existing uses of primitive strings to
uses of getproviders.Hash. The new type definitions are in
internal/getproviders/hash.go.
The "acceptable hashes" for a package is a set of hashes that the upstream
source considers to be good hashes for checking whether future installs
of the same provider version are considered to match this one.
Because the acceptable hashes are a package authentication concern and
they already need to be known (at least in part) to implement the
authenticators, here we add AcceptableHashes as an optional extra method
that an authenticator can implement.
Because these are hashes chosen by the upstream system, the caller must
make its own determination about their trustworthiness. The result of
authentication is likely to be an input to that, for example by
distrusting hashes produced by an authenticator that succeeds but doesn't
report having validated anything.
This is the initial implementation of the parser/decoder portion of the
new dependency lock file handler. It's currently dead code because the
caller isn't written yet. We'll continue to build out this functionality
here until we have the basic level of both load and save functionality
before introducing this into the provider installer codepath.
The version constraint parser allows "~> 2", but it behavior is identical
to "~> 2.0". Due to a quirk of the constraint parser (caused by the fact
that it supports both Ruby-style and npm/cargo-style constraints), it
ends up returning "~> 2" with the minor version marked as "unconstrained"
rather than as zero, but that means the same thing as zero in this context
anyway and so we'll prefer to stringify as "~> 2.0" so that we can be
clearer about how Terraform is understanding that version constraint.
* internal/getproviders: decode and return any registry warnings
The public registry may include a list of warnings in the "versions"
response for any given provider. This PR adds support for warnings from
the registry and an installer event to return those warnings to the
user.
This is the equivalent of UnpackedDirectoryPathForPackage when working
with the packed directory layout. It returns a path to a .zip file with
a name that would be detected by SearchLocalDirectory as a
PackageLocalArchive package.
* internal/registry source: return error if requested provider version protocols are not supported
* getproviders: move responsibility for protocol compatibility checks into the registry client
The original implementation had the providercache checking the provider
metadata for protocol compatibility, but this is only relevant for the
registry source so it made more sense to move the logic into
getproviders.
This also addresses an issue where we were pulling the metadata for
every provider version until we found one that was supported. I've
extended the registry client to unmarshal the protocols in
`ProviderVersions` so we can filter through that list, instead of
pulling each version's metadata.
Providers installed from the registry are accompanied by a list of
checksums (the "SHA256SUMS" file), which is cryptographically signed to
allow package authentication. The process of verifying this has multiple
steps:
- First we must verify that the SHA256 hash of the package archive
matches the expected hash. This could be done for local installations
too, in the future.
- Next we ensure that the expected hash returned as part of the registry
API response matches an entry in the checksum list.
- Finally we verify the cryptographic signature of the checksum list,
using the public keys provided by the registry.
Each of these steps is implemented as a separate PackageAuthentication
type. The local archive installation mechanism uses only the archive
checksum authenticator, and the HTTP installation uses all three in the
order given.
The package authentication system now also returns a result value, which
is used by command/init to display the result of the authentication
process.
There are three tiers of signature, each of which is presented
differently to the user:
- Signatures from the embedded HashiCorp public key indicate that the
provider is officially supported by HashiCorp;
- If the signing key is not from HashiCorp, it may have an associated
trust signature, which indicates that the provider is from one of
HashiCorp's trusted partners;
- Otherwise, if the signature is valid, this is a community provider.
Earlier on in the stubbing of this package we realized that it wasn't
going to be possible to populate the authentication-related bits for all
packages because the relevant metadata just isn't available for packages
that are already local.
However, we just moved ahead with that awkward design at the time because
we needed to get other work done, and so we've been mostly producing
PackageMeta values with all-zeros hashes and just ignoring them entirely
as a temporary workaround.
This is a first step towards what is hopefully a more intuitive model:
authentication is an optional thing in a PackageMeta that is currently
populated only for packages coming from a registry.
So far this still just models checking a SHA256 hash, which is not a
sufficient set of checks for a real release but hopefully the "real"
implementation is a natural iteration of this starting point, and if not
then at least this interim step is a bit more honest about the fact that
Authentication will not be populated on every PackageMeta.
We've been using the models from the "moduledeps" package to represent our
provider dependencies everywhere since the idea of provider dependencies
was introduced in Terraform 0.10, but that model is not convenient to use
for any use-case other than the "terraform providers" command that needs
individual-module-level detail.
To make things easier for new codepaths working with the new-style
provider installer, here we introduce a new model type
getproviders.Requirements which is based on the type the new installer was
already taking as its input. We have new methods in the states, configs,
and earlyconfig packages to produce values of this type, and a helper
to merge Requirements together so we can combine config-derived and
state-derived requirements together during installation.
The advantage of this new model over the moduledeps one is that all of
recursive module walking is done up front and we produce a simple, flat
structure that is more convenient for the main use-cases of selecting
providers for installation and then finding providers in the local cache
to use them for other operations.
This new model is _not_ suitable for implementing "terraform providers"
because it does not retain module-specific requirement details. Therefore
we will likely keep using moduledeps for "terraform providers" for now,
and then possibly at a later time consider specializing the moduledeps
logic for only what "terraform providers" needs, because it seems to be
the only use-case that needs to retain that level of detail.
These new functions allow command implementations to get hold of the
providercache objects and installation source object derived from the
current CLI configuration.
This is not tested yet, but it's a compilable strawman implementation of
the necessary sequence of events to coordinate all of the moving parts
of running a provider installation operation.
This will inevitably see more iteration in later commits as we complete
the surrounding parts and wire it up to be used by "terraform init". So
far, it's just dead code not called by any other package.
The Installer type will encapsulate the logic for running an entire
provider installation request: given a set of providers to install, it
will determine a method to obtain each of them (or detect that they are
already installed) and then take the necessary actions.
So far it doesn't do anything, but this stubs out an interface by which
the caller can request ongoing notifications during an installation
operation.
The existing functionality in this package deals with finding packages
that are either available for installation or already installed. In order
to support installation we also need to determine the location where a
package should be installed.
This lives in the getproviders package because that way all of the logic
related to the filesystem layout for local provider directories lives
together here where they can be maintained together more easily in future.
This is a basic implementation of FilesystemMirrorSource for now aimed
only at the specific use-case of scanning the cache of provider plugins
Terraform will keep under the ".terraform" directory, as part of our
interim provider installer implementation for Terraform 0.13.
The full functionality of this will grow out in later work when we
implement explicit local filesystem mirrors, but for now the goal is to
use this just to inspect the work done by the automatic installer once
we switch it to the new provider-FQN-aware directory structure.
The various FIXME comments in this are justified by the limited intended
scope of this initial implementation, and they should be resolved by
later work to use FilesystemMirrorSource explicitly for user-specified
provider package mirrors.
These are utility functions to ease processing of lists of PackageMeta
elsewhere, once we have functionality that works with multiple packages
at once. The local filesystem mirror source will be the first example of
this, so these methods are motivated mainly by its needs.
This is just to have a centralized set of logic for converting from a
platform string (like "linux_amd64") to a Platform object, so we can do
normalization and validation consistently.
Although we tend to return these in contexts where at least one of these
values is implied, being explicit means that PackageMeta values are
self-contained and less reliant on such external context.
Our local filesystem mirror mechanism will allow provider packages to be
given either in packed form as an archive directly downloaded to disk or
in an unpacked form where the archive is extracted.
Distinguishing these two cases in the concrete Location types will allow
callers to reliably select the mode chosen by the selected installation
source and handle it appropriately, rather than resorting to out-of-band
heuristics like checking whether the object is a directory or a file.
We intend to support installation both directly from origin registries and
from mirrors in the local filesystem or over the network. This Source
interface will serve as our abstraction over those three options, allowing
calling code to treat them all the same.
Our existing provider installer was originally built to work with
releases.hashicorp.com and later retrofitted to talk to the official
Terraform Registry. It also assumes a flat namespace of providers.
We're starting a new one here, copying and adapting code from the old one
as necessary, so that we can build out this new API while retaining all
of the existing functionality and then cut over to this new implementation
in a later step.
Here we're creating a foundational component for the new installer, which
is a mechanism to query for the available versions and download locations
of a particular provider.
Subsequent commits in this package will introduce other Source
implementations for installing from network and filesystem mirrors.
Kristin Laemmert