Commit Graph

183 Commits

Author SHA1 Message Date
James Bardin 76cb40005a add grpcwrap.Provisioner
Rename grpcwrap.New() to grpcwrap.Provider()
Add a grpcwrap function to create a test proivisioner plugin.
2020-12-02 12:45:00 -05:00
James Bardin 03d11f988e remove old test provider 2020-12-02 12:45:00 -05:00
James Bardin eee581ae54 simple provider for internal testing
Simple is a minimal provider implementation using the grpcwrap package,
intended for use in internal tests.
2020-12-02 12:44:59 -05:00
James Bardin 65340f51be wrap the terraform provider for grpc tests 2020-12-02 12:44:59 -05:00
James Bardin b1c104834e grpcwrap: convert provider interface to grpcserver
grpcwrap wraps a core providers.Interface and converts it to a
tfplugin5.ProviderServer to allow for easier protocol testing.
2020-12-02 12:44:59 -05:00
James Bardin 811a3a81e9 add test provider bin for e2etest
One e2etest still requires the bin, so add that back temporarily.
2020-12-02 12:16:36 -05:00
James Bardin b32362da9d internal/legacy/builtin/providers/test
Move the test provider into the interal/legacy directory, until we can
factor it out of the e2e test.
2020-12-02 12:16:35 -05:00
James Bardin 5bbac72a85 internal/legacy/helper/plugin
This is used internally for testing
2020-12-02 12:16:35 -05:00
James Bardin e4edce22ca internal/legacy/helper/schema
moving helper/schema into the ineternal/legacy tree
2020-12-02 12:16:35 -05:00
James Bardin a49e7eee8b internal/legacy/terraform
This is a partial copy of the terraform package to preserve the legacy
types for internal use.
2020-12-02 12:16:35 -05:00
James Bardin ccf19fbf5e regenerate plugin proto with correct tooling
The proto file is structurally unchanged, though it now requires a go
import path option.
2020-12-01 14:06:49 -05:00
James Bardin 7eb491719f normalize temp dir paths in tests
The temporary directory on some systems (most notably MacOS) contains
symlinks, which would not be recorded by the installer. In order to make
these paths comparable in the tests we need to eval the symlinks in
the paths before giving them to the installer.
2020-11-20 16:35:31 -05:00
James Bardin d52e17e111 hide provider crashes from panicwrap when logging
When logging is turned on, panicwrap will still see provider crashes and
falsely report them as core crashes, hiding the formatted provider
error. We can trick panicwrap by slightly obfuscating the error line.
2020-11-05 10:54:21 -05:00
Alisdair McDiarmid fd43bc7847
Merge pull request #26783 from hashicorp/alisdair/normalize-version-constraints-before-deduplication
getproviders: Normalize versions before dedupe
2020-11-04 09:42:19 -05:00
Alisdair McDiarmid 90c4f11b61 go get github.com/hashicorp/go-getter@v1.5.0
Includes fix for breaking upstream API change.
2020-11-02 10:46:24 -05:00
Alisdair McDiarmid b1bc0e5d92 getproviders: Normalize versions before dedupe
When rendering a set of version constraints to a string, we normalize
partially-constrained versions. This means converting a version
like 2.68.* to 2.68.0.

Prior to this commit, this normalization was done after deduplication.
This could result in a version constraints string with duplicate
entries, if multiple partially-constrained versions are equivalent. This
commit fixes this by normalizing before deduplicating and sorting.
2020-11-02 10:45:45 -05:00
Martin Atkins e6e0b6ee46 providercache: verify locked hashes for local package dirs
Previously we were only verifying locked hashes for local archive zip
files, but if we have non-ziphash hashes available then we can and should
also verify that a local directory matches at least one of them.

This does mean that folks using filesystem mirrors but yet also running
Terraform across multiple platforms will need to take some extra care to
ensure the hashes pass on all relevant platforms, which could mean using
"terraform providers lock" to pre-seed their lock files with hashes across
all platforms, or could mean using the "packed" directory layout for the
filesystem mirror so that Terraform will end up in the install-from-archive
codepath instead of this install-from-directory codepath, and can thus
verify ziphash too.

(There's no additional documentation about the above here because there's
already general information about this in the lock file documentation
due to some similar -- though not identical -- situations with network
mirrors.)
2020-10-28 07:46:45 -07:00
Martin Atkins 24d32e9ca2 providercache: More exhaustive testing of the main installer
We previously had some tests for some happy paths and a few specific
failures into an empty directory with no existing locks, but we didn't
have tests for the installer respecting existing lock file entries.

This is a start on a more exhaustive set of tests for the installer,
aiming to visit as many of the possible codepaths as we can reasonably
test using this mocking strategy. (Some other codepaths require different
underlying source implementations, etc, so we'll have to visit those in
other tests separately.)
2020-10-28 07:46:45 -07:00
Martin Atkins fc5a41b5e5 depsfile: Allow loading locks from a byte array in memory
This won't be a typical usage pattern for normal code, but will be useful
for tests that need to work with locks as input so that they don't need to
write out a temporary file on disk just to read it back in immediately.
2020-10-28 07:46:45 -07:00
Martin Atkins 430318e262 getproviders: Consistent ordering of terms in VersionConstraintsString
An earlier commit made this remove duplicates, which set the precedent
that this function is trying to canonically represent the _meaning_ of
the version constraints rather than exactly how they were expressed in
the configuration.

Continuing in that vein, now we'll also apply a consistent (though perhaps
often rather arbitrary) ordering to the terms, so that it doesn't change
due to irrelevant details like declarations being written in a different
order in the configuration.

The ordering here is intended to be reasonably intuitive for simple cases,
but constraint strings with many different constraints are hard to
interpret no matter how we order them so the main goal is consistency,
so those watching how the constraints change over time (e.g. in logs of
Terraform output, or in the dependency log file) will see fewer noisy
changes that don't actually mean anything.
2020-10-26 12:44:15 -07:00
Alisdair McDiarmid 8eed942482
Merge pull request #26678 from hashicorp/alisdair/suppress-duplicate-version-constraints
internal: Suppress duplicate version constraints
2020-10-26 11:27:51 -04:00
James Bardin 3225d9ac11 record all plugin panics, and print on main exit
Create a logger that will record any apparent crash output for later
processing.

If the cli command returns with a non-zero exit status, check for any
recorded crashes and add those to the output.
2020-10-26 09:34:03 -04:00
James Bardin f8893785f0 separate core and provider loggers
Now that hclog can independently set levels on related loggers, we can
separate the log levels for different subsystems in terraform.

This adds the new environment variables, `TF_LOG_CORE` and
`TF_LOG_PROVIDER`, which each take the same set of log level arguments,
and only applies to logs from that subsystem. This means that setting
`TF_LOG_CORE=level` will not show logs from providers, and
`TF_LOG_PROVIDER=level` will not show logs from core. The behavior of
`TF_LOG` alone does not change.

While it is not necessarily needed since the default is to disable logs,
there is also a new level argument of `off`, which reflects the
associated level in hclog.
2020-10-23 12:46:32 -04:00
Alisdair McDiarmid fe9a9fadfb internal: Suppress duplicate version constraints
A set of version constraints can contain duplicates. This can happen if
multiple identical constraints are specified throughout a configuration.

When rendering the set, it is confusing to display redundant
constraints. This commit changes the string renderer to only show the
first instance of a given constraint, and adds unit tests for this
function to cover this change.

This also fixes a bug with the locks file generation: previously, a
configuration with redundant constraints would result in this error on
second init:

Error: Invalid provider version constraints

  on .terraform.lock.hcl line 6:
  (source code not available)

The recorded version constraints for provider
registry.terraform.io/hashicorp/random must be written in normalized form:
"3.0.0".
2020-10-22 12:08:00 -04:00
James Bardin bc1a841d65
Merge pull request #26665 from hashicorp/jbardin/logging
Restore "crash.log" behavior and remove prefixedio
2020-10-22 10:34:51 -04:00
James Bardin c2af5333e8 use a log sink to capture logs for panicwrap
Use a separate log sink to always capture trace logs for the panicwrap
handler to write out in a crash log.

This requires creating a log file in the outer process and passing that
path to the child process to log to.
2020-10-21 17:29:07 -04:00
James Bardin b61488a8ba write traceback to log crash log 2020-10-21 17:29:07 -04:00
James Bardin 1d9d82973b move panicwrap handler to logging package 2020-10-21 13:47:16 -04:00
Alisdair McDiarmid 7a31e56cb7
Merge pull request #26637 from hashicorp/alisdair/fix-locksfile-unconstrained-versions
internal: Fix lockfile constraint output for 1.2.*
2020-10-21 12:05:19 -04:00
Martin Atkins 1fe314b70b getproviders: Don't log if EvalSymlinks changes nothing
Previously this codepath was generating a confusing message in the absense
of any symlinks, because filepath.EvalSymlinks returns a successful result
if the target isn't a symlink.

Now we'll emit the log line only if filepath.EvalSymlinks returns a
result that's different in a way that isn't purely syntactic (which
filepath.Clean would "fix").

The new message is a little more generic because technically we've not
actually ensured that a difference here was caused by a symlink and so
we shouldn't over-promise and generate something potentially misleading.
2020-10-21 07:49:11 -07:00
Martin Atkins 2332a7ab47 replacefile: don't create the temporary file in TMPDIR
ioutil.TempFile has a special case where an empty string for its dir
argument is interpreted as a request to automatically look up the system
temporary directory, which is commonly /tmp .

We don't want that behavior here because we're specifically trying to
create the temporary file in the same directory as the file we're hoping
to replace. If the file gets created in /tmp then it might be on a
different device and thus the later atomic rename won't work.

Instead, we'll add our own special case to explicitly use "." when the
given filename is in the current working directory. That overrides the
special automatic behavior of ioutil.TempFile and thus forces the
behavior we need.

This hadn't previously mattered for earlier callers of this code because
they were creating files in subdirectories, but this codepath was failing
for the dependency lock file due to it always being created directly
in the current working directory.

Unfortunately since this is a picky implementation detail I couldn't find
a good way to write a unit test for it without considerable refactoring.
Instead, I verified manually that the temporary filename wasn't in /tmp on
my Linux system, and hope that the comment inline will explain this
situation well enough to avoid an accidental regression in future
maintenence.
2020-10-20 16:29:59 -07:00
Alisdair McDiarmid 9576a5b2d8 internal: Fix lockfile constraint output for 1.2.*
If a configuration requires a partial provider version (with some parts
unspecified), Terraform considers this as a constrained-to-zero version.
For example, a version constraint of 1.2 will result in an attempt to
install version 1.2.0, even if 1.2.1 is available.

When writing the dependency locks file, we previously would write 1.2.*,
as this is the in-memory representation of 1.2. This would then cause an
error on re-reading the locks file, as this is not a valid constraint
format.

Instead, we now explicitly convert the constraint to its zero-filled
representation before writing the locks file. This ensures that it
correctly round-trips.

Because this change is made in getproviders.VersionConstraintsString, it
also affects the output of the providers sub-command.
2020-10-20 10:14:03 -04:00
James Bardin a3a20e0396 remove LevelFilter
We want to always be using the hclogger to filter whenever possible
2020-10-19 14:31:05 -04:00
James Bardin 0b31ffa587 use a single log writer
Use a single log writer instance for all std library logging.

Setup the std log writer in the logging package, and remove boilerplate
from test packages.
2020-10-19 14:29:54 -04:00
James Bardin abf6b9b378 get properly configured hcloggers
make sure plugins get hcloggers configured to match core
2020-10-19 14:29:54 -04:00
James Bardin 211edf5d75 use hclog as the default logger
Inject hclog as the default logger in the main binary.
2020-10-19 14:29:54 -04:00
James Bardin 6ca477f042 move helper/logging to internal
remove a dead code file too
2020-10-19 14:27:53 -04:00
Martin Atkins 0009768c7f internal/depsfile: Update the dependency lock file atomically
In this case, "atomic" means that there will be no situation where the
file contains only part of the newContent data, and therefore other
software monitoring the file for changes (using a mechanism like inotify)
won't encounter a truncated file.

It does _not_ mean that there can't be existing filehandles open against
the old version of the file. On Windows systems the write will fail in
that case, but on Unix systems the write will typically succeed but leave
the existing filehandles still pointing at the old version of the file.
They'll need to reopen the file in order to see the new content.
2020-10-14 08:01:19 -07:00
Martin Atkins 55e6f64977 internal/depsfile: Factor out our atomic file replacement logic
This originated in the cliconfig code to write out credentials files. The
Windows implementation of this in particular was quite onerous to get
right because it needs a very specific sequence of operations to avoid
running into exclusive file locks, and so by factoring this out with
only cosmetic modification we can avoid repeating all of that engineering
effort for other atomic file writing use-cases.
2020-10-14 08:01:19 -07:00
Martin Atkins 0bbbb9c64b configs: Experimental support for optional object type attributes
This builds on an experimental feature in the underlying cty library which
allows marking specific attribtues of an object type constraint as
optional, which in turn modifies how the cty conversion package handles
missing attributes in a source value: it will silently substitute a null
value of the appropriate type rather than returning an error.

In order to implement the experiment this commit temporarily forks the
HCL typeexpr extension package into a local internal/typeexpr package,
where I've extended the type constraint syntax to allow annotating object
type attributes as being optional using the HCL function call syntax.
If the experiment is successful -- both at the Terraform layer and in
the underlying cty library -- we'll likely send these modifications to
upstream HCL so that other HCL-based languages can potentially benefit
from this new capability.

Because it's experimental, the optional attribute modifier is allowed only
with an explicit opt-in to the module_variable_optional_attrs experiment.
2020-10-12 10:12:28 -07:00
Martin Atkins e1cf0ac801 internal/depsfile: Control how the "hashes" value is formatted
Previously we were just letting hclwrite do its default formatting
behavior here. The current behavior there isn't ideal anyway -- it puts
big data structures all on one line -- but even ignoring that our goal
for this file format is to keep things in a highly-normalized shape so
that diffs against the file are clear and easy to read.

With that in mind, here we directly control how we write that value into
the file, which means that later changes to hclwrite's list/set
presentation won't affect it, regardless of what form they take.
2020-10-09 09:26:23 -07:00
Martin Atkins 2611e08430 command/init: Mention using the lock file for provider selection
This probably isn't the best UI we could do here, but it's a placeholder
for now just to avoid making it seem like we're ignoring the lock file
and checking for new versions anyway.
2020-10-09 09:26:23 -07:00
Martin Atkins b3f5c7f1e6 command/init: Read, respect, and update provider dependency locks
This changes the approach used by the provider installer to remember
between runs which selections it has previously made, using the lock file
format implemented in internal/depsfile.

This means that version constraints in the configuration are considered
only for providers we've not seen before or when -upgrade mode is active.
2020-10-09 09:26:23 -07:00
Martin Atkins 4a1b081afb depsfile: Locks.Equal and Locks.Empty methods
These are helper functions to give the installation UI some hints about
whether the lock file has changed so that it can in turn give the user
advice about it. The UI-layer callers of these will follow in a later
commit.
2020-10-09 09:26:23 -07:00
Kristin Laemmert 04be220f5f deprecate helper/copy
helper/copy CopyDir was used heavily in tests. It differes from
internal/copydir in a few ways, the main one being that it creates the
dst directory while the internal version expected the dst to exist
(there are other differences, which is why I did not just switch tests
to using internal's CopyDir).

I moved the CopyDir func from helper/copy into command_test.go; I could
also have moved it into internal/copy and named it something like
CreateDirAndCopy so if that seems like a better option please let me
know.

helper/copy/CopyFile was used in a couple of spots so I moved it into
internal, at which point I thought it made more sense to rename the
package copy (instead of copydir).

There's also a `go mod tidy` included.
2020-10-08 08:42:16 -04:00
Kristin Laemmert 3933cbd491
remove LegacyProvider (#26433) 2020-10-05 08:33:49 -04:00
Martin Atkins 593cf7b4d5 didyoumean: move from "helper" to "internal"
This new-ish package ended up under "helper" during the 0.12 cycle for
want of some other place to put it, but in retrospect that was an odd
choice because the "helper/" tree is otherwise a bunch of legacy code from
when the SDK lived in this repository.

Here we move it over into the "internal" directory just to distance it
from the guidance of not using "helper/" packages in new projects;
didyoumean is a package we actively use as part of error message hints.
2020-10-02 13:35:07 -07:00
Martin Atkins 59b116f7bf command/init: Remove support for legacy provider addresses
We no longer need to support 0.12-and-earlier-style provider addresses
because users should've upgraded their existing configurations and states
on Terraform 0.13 already.

For now this is only checked in the "init" command, because various test
shims are still relying on the idea of legacy providers the core layer.
However, rejecting these during init is sufficient grounds to avoid
supporting legacy provider addresses in the new dependency lock file
format, and thus sets the stage for a more severe removal of legacy
provider support in a later commit.
2020-09-30 08:54:57 -07:00
Martin Atkins 0b734a2803 command: Make provider installation interruptible
In earlier commits we started to make the installation codepath
context-aware so that it could be canceled in the event of a SIGINT, but
we didn't complete wiring that through the API of the getproviders
package.

Here we make the getproviders.Source interface methods, along with some
other functions that can make network requests, take a context.Context
argument and act appropriately if that context is cancelled.

The main providercache.Installer.EnsureProviderVersions method now also
has some context-awareness so that it can abort its work early if its
context reports any sort of error. That avoids waiting for the process
to wind through all of the remaining iterations of the various loops,
logging each request failure separately, and instead returns just
a single aggregate "canceled" error.

We can then set things up in the "terraform init" and
"terraform providers mirror" commands so that the context will be
cancelled if we get an interrupt signal, allowing provider installation
to abort early while still atomically completing any local-side effects
that may have started.
2020-09-29 10:00:35 -07:00
Martin Atkins ef64df950c getproviders: Prepare for having multiple valid hashes per package
As we continue iterating towards saving valid hashes for a package in a
depsfile lock file after installation and verifying them on future
installation, this prepares getproviders for the possibility of having
multiple valid hashes per package.

This will arise in future commits for two reasons:
- We will need to support both the legacy "zip hash" hashing scheme and
  the new-style content-based hashing scheme because currently the
  registry protocol is only able to produce the legacy scheme, but our
  other installation sources prefer the content-based scheme. Therefore
  packages will typically have a mixture of hashes of both types.
- Installing from an upstream registry will save the hashes for the
  packages across all supported platforms, rather than just the current
  platform, and we'll consider all of those valid for future installation
  if we see both successful matching of the current platform checksum and
  a signature verification for the checksums file as a whole.

This also includes some more preparation for the second case above in that
signatureAuthentication now supports AcceptableHashes and returns all of
the zip-based hashes it can find in the checksums file. This is a bit of
an abstraction leak because previously that authenticator considered its
"document" to just be opaque bytes, but we want to make sure that we can
only end up trusting _all_ of the hashes if we've verified that the
document is signed. Hopefully we'll make this better in a future commit
with some refactoring, but that's deferred for now in order to minimize
disruption to existing codepaths while we work towards a provider locking
MVP.
2020-09-24 14:01:54 -07:00