Commit Graph

267 Commits

Author SHA1 Message Date
Paddy 0b5c4a6a2c
Accept TF_LOG=json to enable TRACE logs in JSON format
This is not currently a supported interface, but we plan to release
tool(s) that consume parts of it that are more dependable later,
separately from Terraform CLI itself.
2021-03-16 14:59:15 -07:00
Pam Selle 242f319638
Give suggestions & remind users to use required_providers when provider not in registry (#28014)
* Add helper suggestion when failed registry err

When someone has a failed registry error on init, remind them that
they should have required_providers in every module

* Give suggestion for a provider based on reqs

Suggest another provider on a registry error, from the list of
requirements we have on init. This skips the legacy lookup
process if there is a similar provider existing in requirements.
2021-03-11 08:54:18 -05:00
Masayuki Morita 31a5aa1878
command/init: Add a new flag `-lockfile=readonly` (#27630)
Fixes #27506

Add a new flag `-lockfile=readonly` to `terraform init`.
It would be useful to allow us to suppress dependency lockfile changes
explicitly.

The type of the `-lockfile` flag is string rather than bool, leaving
room for future extensions to other behavior variants.

The readonly mode suppresses lockfile changes, but should verify
checksums against the information already recorded. It should conflict
with the `-upgrade` flag.

Note: In the original use-case described in #27506, I would like to
suppress adding zh hashes, but a test code here suppresses adding h1
hashes because it's easy for testing.

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2021-03-09 11:12:00 -05:00
Kristin Laemmert 106bcd3bf0
update to match new default branch name (#27909) 2021-02-24 13:36:47 -05:00
Kristin Laemmert e5a7586559
moduletest: update provider to match recent provider interface changes (#27904)
this fixes a bad merge on my part
2021-02-24 12:29:45 -05:00
Kristin Laemmert ff05362d51
providers.Interface: rename ValidateDataSourceConfig to ValidateDataResourceConfig (#27874)
* providers.Interface: rename ValidateDataSourceConfig to
ValidateDataResourceConfig

This PR came about after renaming ValidateResourceTypeConfig to
ValidateResourceConfig: I now understand that we'd called it the former
instead of the latter to indicate that the function wasn't necessarily
operating on a resource that actually exists. A possibly-more-accurate
renaming of both functions might then be ValidateManagedResourceConfig
and ValidateDataResourceConfig.

The next commit will update the protocol (v6 only) as well; these are in
separate commits for reviewers and will get squashed together before
merging.

* extend renaming to protov6
2021-02-24 12:04:28 -05:00
Martin Atkins 7f78342953 command: Experimental "terraform test" command
This is just a prototype to gather some feedback in our ongoing research
on integration testing of Terraform modules. The hope is that by having a
command integrated into Terraform itself it'll be easier for interested
module authors to give it a try, and also easier for us to iterate quickly
based on feedback without having to coordinate across multiple codebases.

Everything about this is subject to change even in future patch releases.
Since it's a CLI command rather than a configuration language feature it's
not using the language experiments mechanism, but generates a warning
similar to the one language experiments generate in order to be clear that
backward compatibility is not guaranteed.
2021-02-22 14:21:45 -08:00
Martin Atkins 8330f8e991 internal/moduletest: Experimental module testing helpers
As part of ongoing research into Terraform testing we'd like to use an
experimental feature to validate our current understanding that expressing
tests as part of the Terraform language, as opposed to in some other
language run alongside, is a good and viable way to write practical
module integration tests.

This initial experimental incarnation of that idea is implemented as a
provider, just because that's an easier extension point for research
purposes than a first-class language feature would be. Whether this would
ultimately emerge as a provider similar to this or as custom language
constructs will be a matter for future research, if this first
experiment confirms that tests written in the Terraform language are the
best direction to take.

The previous incarnation of this experiment was an externally-developed
provider apparentlymart/testing, listed on the Terraform Registry. That
helped with showing that there are some useful tests that we can write
in the Terraform language, but integrating such a provider into Terraform
will allow us to make use of it in the also-experimental "terraform test"
command, which will follow in subsequent commits, to see how this might
fit into a development workflow.
2021-02-22 14:21:45 -08:00
Kristin Laemmert 7b7273e3aa
Add support for plugin protocol v6 (#27826)
* Add support for plugin protocol v6

This PR turns on support for plugin protocol v6. A provider can
advertise itself as supporting protocol version 6 and terraform will
use the correct client.

Todo:

The "unmanaged" providers functionality does not support protocol
version, so at the moment terraform will continue to assume that
"unmanaged" providers are on protocol v5. This will require some
upstream work on go-plugin (I believe).

I would like to convert the builtin providers to use protocol v6 in a
future PR; however it is not necessary until we remove protocol v6.

* add e2e test for using both plugin protocol versions

- copied grpcwrap and made a version that returns protocol v6 provider
- copied the test provider, provider-simple, and made a version that's
  using protocol v6 with the above fun
- added an e2etest
2021-02-22 10:22:45 -05:00
James Bardin 3da5d2bdf6
Merge pull request #27692 from alexwlchan/abolish-the-heirarchy
Correct the spelling of heirarchy/hierarchy throughout
2021-02-18 16:34:28 -05:00
Pam Selle e1c25337c7 Delete readme, add doc.go to helper/schema
Remove the README that had old user-facing information, replacing
it with a doc.go that describes the package and points to the
plugin SDK for external consumers.
2021-02-18 12:30:10 -05:00
Kristin Laemmert f6505870cc
Mildwonkey/providers interface renaming (#27805)
* providers.Interface: huge renamification

This commit renames a handful of functions in the providers.Interface to
match changes made in protocol v6. The following commit implements this
change across the rest of the codebase; I put this in a separate commit
for ease of reviewing and will squash these together when merging.

One noteworthy detail: protocol v6 removes the config from the
ValidateProviderConfigResponse, since it's never been used. I chose to
leave that in place in the interface until we deprecate support for
protocol v5 entirely.

Note that none of these changes impact current providers using protocol
v5; the protocol is unchanged. Only the translation layer between the
proto and terraform have changed.
2021-02-18 10:13:43 -05:00
Kristin Laemmert 7943312ebd
Merge pull request #27699 from hashicorp/mildwonkey/protocolv6
Mildwonkey/protocolv6
2021-02-16 14:51:53 -05:00
Martin Atkins f91a3b87c1 terminal: Helpers for doing fmt-ish operations on the streams
It's pretty common to want to apply the various fmt.Fprint... functions
to our two output streams, and so to make that much less noisy at the
callsite here we have a small number of very thin wrappers around the
underlying fmt package functionality.

Although we're aiming to not have too much abstraction in this "terminal"
package, this seems justified in that it is only a very thin wrapper
around functionality that most Go programmers are already familiar with,
and so the risk of this causing any surprises is low and the improvement
to readability of callers seems worth it.
2021-02-12 11:24:13 -08:00
Martin Atkins ffba8064ed terminal: StreamsForTesting helper
This is to allow convenient testing of functions that are designed to work
directly with *terminal.Streams or the individual stream objects inside.

Because the InputStream and OutputStream APIs expose directly an *os.File,
this does some extra work to set up OS-level pipes so we can capture the
output into local buffers to make test assertions against. The idea here
is to keep the tricky stuff we need for testing confined to the test
codepaths, so that the "real" codepaths don't end up needing to work
around abstractions that are otherwise unnecessary.
2021-02-10 11:51:56 -08:00
Pam Selle aa24bfec47 Emit ProviderAlreadyInstalled when provider installed
Emit the ProviderAlreadyInstalled event when we successfully verify
that we've already installed this provider and are skipping
installation
2021-02-09 11:08:49 -05:00
Kristin Laemmert f3a057eb35 plugin-protocol v6
This is the first commit for plugin protocol v6. This is currently
unused (dead) code; future commits will add the necessary conversion
packages, extend configschema, and modify the providers.Interface.

The new plugin protocol includes the following changes:

- A new field has been added to Attribute: NestedType. This will be the
  key new feature in plugin protocol v6
- Several massages were renamed for consistency with the verb-noun
  pattern seen in _most_ messages.
- The prepared_config has been removed from PrepareProviderConfig
  (renamed ValidateProviderConfig), as it has never been used.
- The provisioner service has been removed entirely. This has no impact
  on built-in provisioners. 3rd party provisioners are not supported by
  the SDK and are not included in this protocol at all.
2021-02-05 13:19:55 -05:00
Alex Chan 4ec80fa18d Correct the spelling of heirarchy/hierarchy throughout 2021-02-05 15:07:04 +00:00
Jonathan Hall 49ee3d3ef8 Grammar nit: "setup" as a verb should be spelled "set up" 2021-01-26 20:39:11 +01:00
Pam Selle aedca597dd Reuse installed target dir providers in init
In init, we can check to see if the target dir already has the
provider we are seeking and skip further querying/installing of
that provider.
2021-01-25 11:13:57 -05:00
Nick Fagerlund a60120477c Update links to CLI docs in code comments, messages, and readme 2021-01-22 12:22:21 -08:00
Pam Selle 084cec6346
Merge pull request #27447 from Filirom1/fix-27445
Providers: Fix init with broken link in plugin_cache_dir
2021-01-22 10:30:55 -05:00
James Bardin 9f2a6d33be move remaining helper packages to internal 2021-01-20 13:54:00 -05:00
Martin Atkins 17728c8fe8 internal/terminal: Interrogate and initialize the terminal, if any
This is a helper package that creates a very thin abstraction over
terminal setup, with the main goal being to deal with all of the extra
setup we need to do in order to get a UTF-8-supporting virtual terminal
on a Windows system.
2021-01-13 15:37:04 -08:00
Kristin Laemmert 86a63e8e39
remove unused legacy plugin package (#27493) 2021-01-13 16:13:08 -05:00
Romain Philibert 5ae5c99ba0 fix: do not fail when provider symlink is broken. Only print a warning.
fix https://github.com/hashicorp/terraform/issues/27445
2021-01-08 12:51:09 +01:00
Anthony Sottile 8cd72e51cb
fix file mode on lock file (#27205)
Signed-off-by: Anthony Sottile <asottile@umich.edu>
2020-12-18 11:51:59 -05:00
Martin Atkins 21d6fb5a37 depsfile: Don't panic when lock file is unreadable
Previously we were expecting that the *hcl.File would always be non-nil,
even in error cases. That isn't always true, so now we'll be more robust
about it and explicitly return an empty locks object in that case, along
with the error diagnostics.

In particular this avoids a panic in a strange situation where the user
created a directory where the lock file would normally go. There's no
meaning to such a directory, so it would always be a mistake and so now
we'll return an error message about it, rather than panicking as before.

The error message for the situation where the lock file is a directory is
currently not very specific, but since it's HCL responsible for generating
that message we can't really fix that at this layer. Perhaps in future
we can change HCL to have a specialized error message for that particular
error situation, but for the sake of this commit the goal is only to
stop the panic and return a normal error message.
2020-12-15 17:00:35 -08:00
Martin Atkins 4b3e237668 command/init: Hint about providers in other namespaces
If a user forgets to specify the source address for a provider, Terraform
will assume they meant a provider in the registry.terraform.io/hashicorp/
namespace. If that ultimately doesn't exist, we'll now try to see if
there's some other provider source address recorded in the registry's
legacy provider lookup table, and suggest it if so.

The error message here is a terse one addressed primarily to folks who are
already somewhat familiar with provider source addresses and how to
specify them. Terraform v0.13 had a more elaborate version of this error
message which directed the user to try the v0.13 automatic upgrade tool,
but we no longer have that available in v0.14 and later so the user must
make the fix themselves.
2020-12-10 10:11:27 -08:00
James Bardin 6529659455 internal/providercache: staticcheck 2020-12-02 13:59:20 -05:00
James Bardin d22e08bd32 internal/earlyconfig: staticcheck 2020-12-02 13:59:19 -05:00
James Bardin f0c3ddf5a1 internal/getproviders: staticcheck 2020-12-02 13:59:19 -05:00
James Bardin 276dfe634f internal/providercache: staticcheck 2020-12-02 13:59:19 -05:00
James Bardin 1f359bab03 internal/typeexpr: staticcheck
diagnostics were appended to diags shadowing the intended value.
2020-12-02 13:59:19 -05:00
James Bardin 76cb40005a add grpcwrap.Provisioner
Rename grpcwrap.New() to grpcwrap.Provider()
Add a grpcwrap function to create a test proivisioner plugin.
2020-12-02 12:45:00 -05:00
James Bardin 03d11f988e remove old test provider 2020-12-02 12:45:00 -05:00
James Bardin eee581ae54 simple provider for internal testing
Simple is a minimal provider implementation using the grpcwrap package,
intended for use in internal tests.
2020-12-02 12:44:59 -05:00
James Bardin 65340f51be wrap the terraform provider for grpc tests 2020-12-02 12:44:59 -05:00
James Bardin b1c104834e grpcwrap: convert provider interface to grpcserver
grpcwrap wraps a core providers.Interface and converts it to a
tfplugin5.ProviderServer to allow for easier protocol testing.
2020-12-02 12:44:59 -05:00
James Bardin 811a3a81e9 add test provider bin for e2etest
One e2etest still requires the bin, so add that back temporarily.
2020-12-02 12:16:36 -05:00
James Bardin b32362da9d internal/legacy/builtin/providers/test
Move the test provider into the interal/legacy directory, until we can
factor it out of the e2e test.
2020-12-02 12:16:35 -05:00
James Bardin 5bbac72a85 internal/legacy/helper/plugin
This is used internally for testing
2020-12-02 12:16:35 -05:00
James Bardin e4edce22ca internal/legacy/helper/schema
moving helper/schema into the ineternal/legacy tree
2020-12-02 12:16:35 -05:00
James Bardin a49e7eee8b internal/legacy/terraform
This is a partial copy of the terraform package to preserve the legacy
types for internal use.
2020-12-02 12:16:35 -05:00
James Bardin ccf19fbf5e regenerate plugin proto with correct tooling
The proto file is structurally unchanged, though it now requires a go
import path option.
2020-12-01 14:06:49 -05:00
James Bardin 7eb491719f normalize temp dir paths in tests
The temporary directory on some systems (most notably MacOS) contains
symlinks, which would not be recorded by the installer. In order to make
these paths comparable in the tests we need to eval the symlinks in
the paths before giving them to the installer.
2020-11-20 16:35:31 -05:00
James Bardin d52e17e111 hide provider crashes from panicwrap when logging
When logging is turned on, panicwrap will still see provider crashes and
falsely report them as core crashes, hiding the formatted provider
error. We can trick panicwrap by slightly obfuscating the error line.
2020-11-05 10:54:21 -05:00
Alisdair McDiarmid fd43bc7847
Merge pull request #26783 from hashicorp/alisdair/normalize-version-constraints-before-deduplication
getproviders: Normalize versions before dedupe
2020-11-04 09:42:19 -05:00
Alisdair McDiarmid 90c4f11b61 go get github.com/hashicorp/go-getter@v1.5.0
Includes fix for breaking upstream API change.
2020-11-02 10:46:24 -05:00
Alisdair McDiarmid b1bc0e5d92 getproviders: Normalize versions before dedupe
When rendering a set of version constraints to a string, we normalize
partially-constrained versions. This means converting a version
like 2.68.* to 2.68.0.

Prior to this commit, this normalization was done after deduplication.
This could result in a version constraints string with duplicate
entries, if multiple partially-constrained versions are equivalent. This
commit fixes this by normalizing before deduplicating and sorting.
2020-11-02 10:45:45 -05:00
Martin Atkins e6e0b6ee46 providercache: verify locked hashes for local package dirs
Previously we were only verifying locked hashes for local archive zip
files, but if we have non-ziphash hashes available then we can and should
also verify that a local directory matches at least one of them.

This does mean that folks using filesystem mirrors but yet also running
Terraform across multiple platforms will need to take some extra care to
ensure the hashes pass on all relevant platforms, which could mean using
"terraform providers lock" to pre-seed their lock files with hashes across
all platforms, or could mean using the "packed" directory layout for the
filesystem mirror so that Terraform will end up in the install-from-archive
codepath instead of this install-from-directory codepath, and can thus
verify ziphash too.

(There's no additional documentation about the above here because there's
already general information about this in the lock file documentation
due to some similar -- though not identical -- situations with network
mirrors.)
2020-10-28 07:46:45 -07:00
Martin Atkins 24d32e9ca2 providercache: More exhaustive testing of the main installer
We previously had some tests for some happy paths and a few specific
failures into an empty directory with no existing locks, but we didn't
have tests for the installer respecting existing lock file entries.

This is a start on a more exhaustive set of tests for the installer,
aiming to visit as many of the possible codepaths as we can reasonably
test using this mocking strategy. (Some other codepaths require different
underlying source implementations, etc, so we'll have to visit those in
other tests separately.)
2020-10-28 07:46:45 -07:00
Martin Atkins fc5a41b5e5 depsfile: Allow loading locks from a byte array in memory
This won't be a typical usage pattern for normal code, but will be useful
for tests that need to work with locks as input so that they don't need to
write out a temporary file on disk just to read it back in immediately.
2020-10-28 07:46:45 -07:00
Martin Atkins 430318e262 getproviders: Consistent ordering of terms in VersionConstraintsString
An earlier commit made this remove duplicates, which set the precedent
that this function is trying to canonically represent the _meaning_ of
the version constraints rather than exactly how they were expressed in
the configuration.

Continuing in that vein, now we'll also apply a consistent (though perhaps
often rather arbitrary) ordering to the terms, so that it doesn't change
due to irrelevant details like declarations being written in a different
order in the configuration.

The ordering here is intended to be reasonably intuitive for simple cases,
but constraint strings with many different constraints are hard to
interpret no matter how we order them so the main goal is consistency,
so those watching how the constraints change over time (e.g. in logs of
Terraform output, or in the dependency log file) will see fewer noisy
changes that don't actually mean anything.
2020-10-26 12:44:15 -07:00
Alisdair McDiarmid 8eed942482
Merge pull request #26678 from hashicorp/alisdair/suppress-duplicate-version-constraints
internal: Suppress duplicate version constraints
2020-10-26 11:27:51 -04:00
James Bardin 3225d9ac11 record all plugin panics, and print on main exit
Create a logger that will record any apparent crash output for later
processing.

If the cli command returns with a non-zero exit status, check for any
recorded crashes and add those to the output.
2020-10-26 09:34:03 -04:00
James Bardin f8893785f0 separate core and provider loggers
Now that hclog can independently set levels on related loggers, we can
separate the log levels for different subsystems in terraform.

This adds the new environment variables, `TF_LOG_CORE` and
`TF_LOG_PROVIDER`, which each take the same set of log level arguments,
and only applies to logs from that subsystem. This means that setting
`TF_LOG_CORE=level` will not show logs from providers, and
`TF_LOG_PROVIDER=level` will not show logs from core. The behavior of
`TF_LOG` alone does not change.

While it is not necessarily needed since the default is to disable logs,
there is also a new level argument of `off`, which reflects the
associated level in hclog.
2020-10-23 12:46:32 -04:00
Alisdair McDiarmid fe9a9fadfb internal: Suppress duplicate version constraints
A set of version constraints can contain duplicates. This can happen if
multiple identical constraints are specified throughout a configuration.

When rendering the set, it is confusing to display redundant
constraints. This commit changes the string renderer to only show the
first instance of a given constraint, and adds unit tests for this
function to cover this change.

This also fixes a bug with the locks file generation: previously, a
configuration with redundant constraints would result in this error on
second init:

Error: Invalid provider version constraints

  on .terraform.lock.hcl line 6:
  (source code not available)

The recorded version constraints for provider
registry.terraform.io/hashicorp/random must be written in normalized form:
"3.0.0".
2020-10-22 12:08:00 -04:00
James Bardin bc1a841d65
Merge pull request #26665 from hashicorp/jbardin/logging
Restore "crash.log" behavior and remove prefixedio
2020-10-22 10:34:51 -04:00
James Bardin c2af5333e8 use a log sink to capture logs for panicwrap
Use a separate log sink to always capture trace logs for the panicwrap
handler to write out in a crash log.

This requires creating a log file in the outer process and passing that
path to the child process to log to.
2020-10-21 17:29:07 -04:00
James Bardin b61488a8ba write traceback to log crash log 2020-10-21 17:29:07 -04:00
James Bardin 1d9d82973b move panicwrap handler to logging package 2020-10-21 13:47:16 -04:00
Alisdair McDiarmid 7a31e56cb7
Merge pull request #26637 from hashicorp/alisdair/fix-locksfile-unconstrained-versions
internal: Fix lockfile constraint output for 1.2.*
2020-10-21 12:05:19 -04:00
Martin Atkins 1fe314b70b getproviders: Don't log if EvalSymlinks changes nothing
Previously this codepath was generating a confusing message in the absense
of any symlinks, because filepath.EvalSymlinks returns a successful result
if the target isn't a symlink.

Now we'll emit the log line only if filepath.EvalSymlinks returns a
result that's different in a way that isn't purely syntactic (which
filepath.Clean would "fix").

The new message is a little more generic because technically we've not
actually ensured that a difference here was caused by a symlink and so
we shouldn't over-promise and generate something potentially misleading.
2020-10-21 07:49:11 -07:00
Martin Atkins 2332a7ab47 replacefile: don't create the temporary file in TMPDIR
ioutil.TempFile has a special case where an empty string for its dir
argument is interpreted as a request to automatically look up the system
temporary directory, which is commonly /tmp .

We don't want that behavior here because we're specifically trying to
create the temporary file in the same directory as the file we're hoping
to replace. If the file gets created in /tmp then it might be on a
different device and thus the later atomic rename won't work.

Instead, we'll add our own special case to explicitly use "." when the
given filename is in the current working directory. That overrides the
special automatic behavior of ioutil.TempFile and thus forces the
behavior we need.

This hadn't previously mattered for earlier callers of this code because
they were creating files in subdirectories, but this codepath was failing
for the dependency lock file due to it always being created directly
in the current working directory.

Unfortunately since this is a picky implementation detail I couldn't find
a good way to write a unit test for it without considerable refactoring.
Instead, I verified manually that the temporary filename wasn't in /tmp on
my Linux system, and hope that the comment inline will explain this
situation well enough to avoid an accidental regression in future
maintenence.
2020-10-20 16:29:59 -07:00
Alisdair McDiarmid 9576a5b2d8 internal: Fix lockfile constraint output for 1.2.*
If a configuration requires a partial provider version (with some parts
unspecified), Terraform considers this as a constrained-to-zero version.
For example, a version constraint of 1.2 will result in an attempt to
install version 1.2.0, even if 1.2.1 is available.

When writing the dependency locks file, we previously would write 1.2.*,
as this is the in-memory representation of 1.2. This would then cause an
error on re-reading the locks file, as this is not a valid constraint
format.

Instead, we now explicitly convert the constraint to its zero-filled
representation before writing the locks file. This ensures that it
correctly round-trips.

Because this change is made in getproviders.VersionConstraintsString, it
also affects the output of the providers sub-command.
2020-10-20 10:14:03 -04:00
James Bardin a3a20e0396 remove LevelFilter
We want to always be using the hclogger to filter whenever possible
2020-10-19 14:31:05 -04:00
James Bardin 0b31ffa587 use a single log writer
Use a single log writer instance for all std library logging.

Setup the std log writer in the logging package, and remove boilerplate
from test packages.
2020-10-19 14:29:54 -04:00
James Bardin abf6b9b378 get properly configured hcloggers
make sure plugins get hcloggers configured to match core
2020-10-19 14:29:54 -04:00
James Bardin 211edf5d75 use hclog as the default logger
Inject hclog as the default logger in the main binary.
2020-10-19 14:29:54 -04:00
James Bardin 6ca477f042 move helper/logging to internal
remove a dead code file too
2020-10-19 14:27:53 -04:00
Martin Atkins 0009768c7f internal/depsfile: Update the dependency lock file atomically
In this case, "atomic" means that there will be no situation where the
file contains only part of the newContent data, and therefore other
software monitoring the file for changes (using a mechanism like inotify)
won't encounter a truncated file.

It does _not_ mean that there can't be existing filehandles open against
the old version of the file. On Windows systems the write will fail in
that case, but on Unix systems the write will typically succeed but leave
the existing filehandles still pointing at the old version of the file.
They'll need to reopen the file in order to see the new content.
2020-10-14 08:01:19 -07:00
Martin Atkins 55e6f64977 internal/depsfile: Factor out our atomic file replacement logic
This originated in the cliconfig code to write out credentials files. The
Windows implementation of this in particular was quite onerous to get
right because it needs a very specific sequence of operations to avoid
running into exclusive file locks, and so by factoring this out with
only cosmetic modification we can avoid repeating all of that engineering
effort for other atomic file writing use-cases.
2020-10-14 08:01:19 -07:00
Martin Atkins 0bbbb9c64b configs: Experimental support for optional object type attributes
This builds on an experimental feature in the underlying cty library which
allows marking specific attribtues of an object type constraint as
optional, which in turn modifies how the cty conversion package handles
missing attributes in a source value: it will silently substitute a null
value of the appropriate type rather than returning an error.

In order to implement the experiment this commit temporarily forks the
HCL typeexpr extension package into a local internal/typeexpr package,
where I've extended the type constraint syntax to allow annotating object
type attributes as being optional using the HCL function call syntax.
If the experiment is successful -- both at the Terraform layer and in
the underlying cty library -- we'll likely send these modifications to
upstream HCL so that other HCL-based languages can potentially benefit
from this new capability.

Because it's experimental, the optional attribute modifier is allowed only
with an explicit opt-in to the module_variable_optional_attrs experiment.
2020-10-12 10:12:28 -07:00
Martin Atkins e1cf0ac801 internal/depsfile: Control how the "hashes" value is formatted
Previously we were just letting hclwrite do its default formatting
behavior here. The current behavior there isn't ideal anyway -- it puts
big data structures all on one line -- but even ignoring that our goal
for this file format is to keep things in a highly-normalized shape so
that diffs against the file are clear and easy to read.

With that in mind, here we directly control how we write that value into
the file, which means that later changes to hclwrite's list/set
presentation won't affect it, regardless of what form they take.
2020-10-09 09:26:23 -07:00
Martin Atkins 2611e08430 command/init: Mention using the lock file for provider selection
This probably isn't the best UI we could do here, but it's a placeholder
for now just to avoid making it seem like we're ignoring the lock file
and checking for new versions anyway.
2020-10-09 09:26:23 -07:00
Martin Atkins b3f5c7f1e6 command/init: Read, respect, and update provider dependency locks
This changes the approach used by the provider installer to remember
between runs which selections it has previously made, using the lock file
format implemented in internal/depsfile.

This means that version constraints in the configuration are considered
only for providers we've not seen before or when -upgrade mode is active.
2020-10-09 09:26:23 -07:00
Martin Atkins 4a1b081afb depsfile: Locks.Equal and Locks.Empty methods
These are helper functions to give the installation UI some hints about
whether the lock file has changed so that it can in turn give the user
advice about it. The UI-layer callers of these will follow in a later
commit.
2020-10-09 09:26:23 -07:00
Kristin Laemmert 04be220f5f deprecate helper/copy
helper/copy CopyDir was used heavily in tests. It differes from
internal/copydir in a few ways, the main one being that it creates the
dst directory while the internal version expected the dst to exist
(there are other differences, which is why I did not just switch tests
to using internal's CopyDir).

I moved the CopyDir func from helper/copy into command_test.go; I could
also have moved it into internal/copy and named it something like
CreateDirAndCopy so if that seems like a better option please let me
know.

helper/copy/CopyFile was used in a couple of spots so I moved it into
internal, at which point I thought it made more sense to rename the
package copy (instead of copydir).

There's also a `go mod tidy` included.
2020-10-08 08:42:16 -04:00
Kristin Laemmert 3933cbd491
remove LegacyProvider (#26433) 2020-10-05 08:33:49 -04:00
Martin Atkins 593cf7b4d5 didyoumean: move from "helper" to "internal"
This new-ish package ended up under "helper" during the 0.12 cycle for
want of some other place to put it, but in retrospect that was an odd
choice because the "helper/" tree is otherwise a bunch of legacy code from
when the SDK lived in this repository.

Here we move it over into the "internal" directory just to distance it
from the guidance of not using "helper/" packages in new projects;
didyoumean is a package we actively use as part of error message hints.
2020-10-02 13:35:07 -07:00
Martin Atkins 59b116f7bf command/init: Remove support for legacy provider addresses
We no longer need to support 0.12-and-earlier-style provider addresses
because users should've upgraded their existing configurations and states
on Terraform 0.13 already.

For now this is only checked in the "init" command, because various test
shims are still relying on the idea of legacy providers the core layer.
However, rejecting these during init is sufficient grounds to avoid
supporting legacy provider addresses in the new dependency lock file
format, and thus sets the stage for a more severe removal of legacy
provider support in a later commit.
2020-09-30 08:54:57 -07:00
Martin Atkins 0b734a2803 command: Make provider installation interruptible
In earlier commits we started to make the installation codepath
context-aware so that it could be canceled in the event of a SIGINT, but
we didn't complete wiring that through the API of the getproviders
package.

Here we make the getproviders.Source interface methods, along with some
other functions that can make network requests, take a context.Context
argument and act appropriately if that context is cancelled.

The main providercache.Installer.EnsureProviderVersions method now also
has some context-awareness so that it can abort its work early if its
context reports any sort of error. That avoids waiting for the process
to wind through all of the remaining iterations of the various loops,
logging each request failure separately, and instead returns just
a single aggregate "canceled" error.

We can then set things up in the "terraform init" and
"terraform providers mirror" commands so that the context will be
cancelled if we get an interrupt signal, allowing provider installation
to abort early while still atomically completing any local-side effects
that may have started.
2020-09-29 10:00:35 -07:00
Martin Atkins ef64df950c getproviders: Prepare for having multiple valid hashes per package
As we continue iterating towards saving valid hashes for a package in a
depsfile lock file after installation and verifying them on future
installation, this prepares getproviders for the possibility of having
multiple valid hashes per package.

This will arise in future commits for two reasons:
- We will need to support both the legacy "zip hash" hashing scheme and
  the new-style content-based hashing scheme because currently the
  registry protocol is only able to produce the legacy scheme, but our
  other installation sources prefer the content-based scheme. Therefore
  packages will typically have a mixture of hashes of both types.
- Installing from an upstream registry will save the hashes for the
  packages across all supported platforms, rather than just the current
  platform, and we'll consider all of those valid for future installation
  if we see both successful matching of the current platform checksum and
  a signature verification for the checksums file as a whole.

This also includes some more preparation for the second case above in that
signatureAuthentication now supports AcceptableHashes and returns all of
the zip-based hashes it can find in the checksums file. This is a bit of
an abstraction leak because previously that authenticator considered its
"document" to just be opaque bytes, but we want to make sure that we can
only end up trusting _all_ of the hashes if we've verified that the
document is signed. Hopefully we'll make this better in a future commit
with some refactoring, but that's deferred for now in order to minimize
disruption to existing codepaths while we work towards a provider locking
MVP.
2020-09-24 14:01:54 -07:00
Martin Atkins 6694cfaa0e getproviders: Add a real type Hash for package hashes
The logic for what constitutes a valid hash and how different hash schemes
are represented was starting to get sprawled over many different files and
packages.

Consistently with other cases where we've used named types to gather the
definition of a particular string into a single place and have the Go
compiler help us use it properly, this introduces both getproviders.Hash
representing a hash value and getproviders.HashScheme representing the
idea of a particular hash scheme.

Most of this changeset is updating existing uses of primitive strings to
uses of getproviders.Hash. The new type definitions are in
internal/getproviders/hash.go.
2020-09-24 14:01:54 -07:00
Martin Atkins 264a3cf031 depsfile: Flatten the "hashes" locks to a single set of strings
Although origin registries return specific [filename, hash] pairs, our
various different installation methods can't produce a structured mapping
from platform to hash without breaking changes.

Therefore, as a compromise, we'll continue to do platform-specific checks
against upstream data in the cases where that's possible (installation
from origin registry or network mirror) but we'll treat the lock file as
just a flat set of equally-valid hashes, at least one of which must match
after we've completed whatever checks we've made against the
upstream-provided checksums/signatures.

This includes only the minimal internal/getproviders updates required to
make this compile. A subsequent commit will update that package to
actually support the idea of verifying against multiple hashes.
2020-09-24 14:01:54 -07:00
Martin Atkins b2c0ccdf96 internal/getproviders: Allow PackageMeta to carry acceptable hashes
The "acceptable hashes" for a package is a set of hashes that the upstream
source considers to be good hashes for checking whether future installs
of the same provider version are considered to match this one.

Because the acceptable hashes are a package authentication concern and
they already need to be known (at least in part) to implement the
authenticators, here we add AcceptableHashes as an optional extra method
that an authenticator can implement.

Because these are hashes chosen by the upstream system, the caller must
make its own determination about their trustworthiness. The result of
authentication is likely to be an input to that, for example by
distrusting hashes produced by an authenticator that succeeds but doesn't
report having validated anything.
2020-09-24 14:01:54 -07:00
Martin Atkins e843097e52 internal/getproviders: Formalize the "ziphash" hashing scheme
This is the pre-existing hashing scheme that was initially built for
releases.hashicorp.com and then later reused for the provider registry
protocol, which takes a SHA256 hash of the official distribution .zip file
and formats it as lowercase hex.

This is a non-ideal hash scheme because it works only for
PackageLocalArchive locations, and so we can't verify package directories
on local disk against such hashes. However, the registry protocol is now
a compatibility constraint and so we're going to need to support this
hashing scheme for the foreseeable future.
2020-09-24 14:01:54 -07:00
Martin Atkins 773dd56b42 internal/depsfile: Introduce the concept of "non-lockable" providers
It doesn't make sense for a built-in provider to appear in a lock file
because built-in providers have no version independent of the version of
Terraform they are compiled into.

We also exclude legacy providers here, because they were supported only
as a transitional aid to enable the Terraform 0.13 upgrade process and
are not intended for explicit selection.

The provider installer will, once it's updated to understand dependency
locking, use this concept to decide which subset of its selections to
record in the dependency lock file for reference for future installation
requests.
2020-09-08 09:50:58 -07:00
Martin Atkins 98e2e69abb internal/depsfile: SaveLocksToFile implementation
This is an initial implementation of writing locks back to a file on disk.
This initial implementation is incomplete because it does not write the
changes to the new file atomically. We'll revisit that in a later commit
as we return to polish these codepaths, once we've proven out this
package's design by integrating it with Terraform's provider installer.
2020-09-08 09:50:58 -07:00
Martin Atkins 92723661d0 internal/depsfile: Loading locks from HCL files on disk
This is the initial implementation of the parser/decoder portion of the
new dependency lock file handler. It's currently dead code because the
caller isn't written yet. We'll continue to build out this functionality
here until we have the basic level of both load and save functionality
before introducing this into the provider installer codepath.
2020-09-08 09:50:58 -07:00
Martin Atkins 6993ecb0a6 internal/getproviders: VersionConstraintsString for "~> 2" input
The version constraint parser allows "~> 2", but it behavior is identical
to "~> 2.0". Due to a quirk of the constraint parser (caused by the fact
that it supports both Ruby-style and npm/cargo-style constraints), it
ends up returning "~> 2" with the minor version marked as "unconstrained"
rather than as zero, but that means the same thing as zero in this context
anyway and so we'll prefer to stringify as "~> 2.0" so that we can be
clearer about how Terraform is understanding that version constraint.
2020-09-08 09:50:58 -07:00
Alisdair McDiarmid 5587509bcf internal: Fix providercache test failures on macOS
For reasons that are unclear, these two tests just started failing on
macOS very recently. The failure looked like:

    PackageDir: strings.Join({
      "/",
+     "private/",
      "var/folders/3h/foobar/T/terraform-test-p",
      "rovidercache655312854/registry.terraform.io/hashicorp/null/2.0.0",
      "/windows_amd64",
    },

Speculating that the macOS temporary directory moved into the /private
directory, I added a couple of EvalSymlinks calls and the tests pass
again.

No other unit tests appear to be affected by this at the moment.
2020-09-04 16:09:57 -04:00
Alisdair McDiarmid fc7e467d19 command: Add redirect support to 0.13upgrade
If a provider changes namespace in the registry, we can detect this when
running the 0.13upgrade command. As long as there is a version matching
the user's constraints, we now use the provider's new source address.
Otherwise, warn the user that the provider has moved and a version
upgrade is necessary to move to it.
2020-08-31 14:53:35 -04:00
Alan D. Salewski f63c38d6a1
internal/initwd: allow tests to pass when $PWD contains symlinks (#26015)
Fixes #26014
2020-08-27 10:02:22 -04:00
Martin Atkins 2bd2a9a923 internal/getproviders: HTTPMirrorSource implementation
We previously had this just stubbed out because it was a stretch goal for
the v0.13.0 release and it ultimately didn't make it in.

Here we fill out the existing stub -- with a minor change to its interface
so it can access credentials -- with a client implementation that is
compatible with the directory structure produced by the
"terraform providers mirror" subcommand, were the result to be published
on a static file server.
2020-08-26 13:18:08 -07:00
Martin Atkins 146e983c36 internal/getproviders: package authenticator for our new-style hashes
Earlier we introduced a new package hashing mechanism that is compatible
with both packed and unpacked packages, because it's a hash of the
contents of the package rather than of the archive it's delivered in.
However, we were using that only for the local selections file and not
for any remote package authentication yet.

The provider network mirrors protocol includes new-style hashes as a step
towards transitioning over to the new hash format in all cases, so this
new authenticator is here in preparation for verifying the checksums of
packages coming from network mirrors, for mirrors that support them.

For now this leaves us in a kinda confusing situation where we have both
NewPackageHashAuthentication for the new style and
NewArchiveChecksumAuthentication for the old style, which for the moment
is represented only by a doc comment on the latter. Hopefully we can
remove NewArchiveChecksumAuthentication in a future commit, if we can
get the registry updated to use the new hashing format.
2020-08-26 13:18:08 -07:00
Alisdair McDiarmid 915f53af23 internal: Clean up package install temp file
The installFromHTTPURL function downloads a package to a temporary file,
then delegates to installFromLocalArchive to install it. We were
previously not deleting the temporary file afterwards. This commit fixes
that.
2020-08-25 08:35:32 -04:00
Martin Atkins ce67a818db internal/getproviders: Allow basedir for local search to be symlink
The SearchLocalDirectory function was intentionally written to only
support symlinks at the leaves so that it wouldn't risk getting into an
infinite loop traversing intermediate symlinks, but that rule was also
applying to the base directory itself.

It's pretty reasonable to put your local plugins in some location
Terraform wouldn't normally search (e.g. because you want to get them from
a shared filesystem mounted somewhere) and creating a symlink from one
of the locations Terraform _does_ search is a convenient way to help
Terraform find those without going all in on the explicit provider
installation methods configuration that is intended for more complicated
situations.

To allow for that, here we make a special exception for the base
directory, resolving that first before we do any directory walking.

In order to help with debugging a situation where there are for some
reason symlinks at intermediate levels inside the search tree, we also now
emit a WARN log line in that case to be explicit that symlinks are not
supported there and to hint to put the symlink at the top-level if you
want to use symlinks at all.

(The support for symlinks at the deepest level of search is not mentioned
in this message because we allow it primarily for our own cache linking
behavior.)
2020-08-18 14:28:50 -07:00
Alisdair McDiarmid 440543f427 internal/providercache: Fix bug when symlink fails
When installing a provider which is already cached, we attempt to create
a symlink from the install directory targeting the cache. If symlinking
fails due to missing OS/filesystem support, we instead want to copy the
cached provider.

The fallback code to do this would always fail, due to a missing target
directory. This commit fixes that. I was unable to find a way to add
automated tests around this, but I have manually verified the fix on
Windows 8.1.
2020-07-23 11:36:46 -04:00