Commit Graph

209 Commits

Author SHA1 Message Date
Kristin Laemmert 7b7273e3aa
Add support for plugin protocol v6 (#27826)
* Add support for plugin protocol v6

This PR turns on support for plugin protocol v6. A provider can
advertise itself as supporting protocol version 6 and terraform will
use the correct client.

Todo:

The "unmanaged" providers functionality does not support protocol
version, so at the moment terraform will continue to assume that
"unmanaged" providers are on protocol v5. This will require some
upstream work on go-plugin (I believe).

I would like to convert the builtin providers to use protocol v6 in a
future PR; however it is not necessary until we remove protocol v6.

* add e2e test for using both plugin protocol versions

- copied grpcwrap and made a version that returns protocol v6 provider
- copied the test provider, provider-simple, and made a version that's
  using protocol v6 with the above fun
- added an e2etest
2021-02-22 10:22:45 -05:00
James Bardin 3da5d2bdf6
Merge pull request #27692 from alexwlchan/abolish-the-heirarchy
Correct the spelling of heirarchy/hierarchy throughout
2021-02-18 16:34:28 -05:00
Pam Selle e1c25337c7 Delete readme, add doc.go to helper/schema
Remove the README that had old user-facing information, replacing
it with a doc.go that describes the package and points to the
plugin SDK for external consumers.
2021-02-18 12:30:10 -05:00
Kristin Laemmert f6505870cc
Mildwonkey/providers interface renaming (#27805)
* providers.Interface: huge renamification

This commit renames a handful of functions in the providers.Interface to
match changes made in protocol v6. The following commit implements this
change across the rest of the codebase; I put this in a separate commit
for ease of reviewing and will squash these together when merging.

One noteworthy detail: protocol v6 removes the config from the
ValidateProviderConfigResponse, since it's never been used. I chose to
leave that in place in the interface until we deprecate support for
protocol v5 entirely.

Note that none of these changes impact current providers using protocol
v5; the protocol is unchanged. Only the translation layer between the
proto and terraform have changed.
2021-02-18 10:13:43 -05:00
Kristin Laemmert 7943312ebd
Merge pull request #27699 from hashicorp/mildwonkey/protocolv6
Mildwonkey/protocolv6
2021-02-16 14:51:53 -05:00
Martin Atkins f91a3b87c1 terminal: Helpers for doing fmt-ish operations on the streams
It's pretty common to want to apply the various fmt.Fprint... functions
to our two output streams, and so to make that much less noisy at the
callsite here we have a small number of very thin wrappers around the
underlying fmt package functionality.

Although we're aiming to not have too much abstraction in this "terminal"
package, this seems justified in that it is only a very thin wrapper
around functionality that most Go programmers are already familiar with,
and so the risk of this causing any surprises is low and the improvement
to readability of callers seems worth it.
2021-02-12 11:24:13 -08:00
Martin Atkins ffba8064ed terminal: StreamsForTesting helper
This is to allow convenient testing of functions that are designed to work
directly with *terminal.Streams or the individual stream objects inside.

Because the InputStream and OutputStream APIs expose directly an *os.File,
this does some extra work to set up OS-level pipes so we can capture the
output into local buffers to make test assertions against. The idea here
is to keep the tricky stuff we need for testing confined to the test
codepaths, so that the "real" codepaths don't end up needing to work
around abstractions that are otherwise unnecessary.
2021-02-10 11:51:56 -08:00
Pam Selle aa24bfec47 Emit ProviderAlreadyInstalled when provider installed
Emit the ProviderAlreadyInstalled event when we successfully verify
that we've already installed this provider and are skipping
installation
2021-02-09 11:08:49 -05:00
Kristin Laemmert f3a057eb35 plugin-protocol v6
This is the first commit for plugin protocol v6. This is currently
unused (dead) code; future commits will add the necessary conversion
packages, extend configschema, and modify the providers.Interface.

The new plugin protocol includes the following changes:

- A new field has been added to Attribute: NestedType. This will be the
  key new feature in plugin protocol v6
- Several massages were renamed for consistency with the verb-noun
  pattern seen in _most_ messages.
- The prepared_config has been removed from PrepareProviderConfig
  (renamed ValidateProviderConfig), as it has never been used.
- The provisioner service has been removed entirely. This has no impact
  on built-in provisioners. 3rd party provisioners are not supported by
  the SDK and are not included in this protocol at all.
2021-02-05 13:19:55 -05:00
Alex Chan 4ec80fa18d Correct the spelling of heirarchy/hierarchy throughout 2021-02-05 15:07:04 +00:00
Jonathan Hall 49ee3d3ef8 Grammar nit: "setup" as a verb should be spelled "set up" 2021-01-26 20:39:11 +01:00
Pam Selle aedca597dd Reuse installed target dir providers in init
In init, we can check to see if the target dir already has the
provider we are seeking and skip further querying/installing of
that provider.
2021-01-25 11:13:57 -05:00
Nick Fagerlund a60120477c Update links to CLI docs in code comments, messages, and readme 2021-01-22 12:22:21 -08:00
Pam Selle 084cec6346
Merge pull request #27447 from Filirom1/fix-27445
Providers: Fix init with broken link in plugin_cache_dir
2021-01-22 10:30:55 -05:00
James Bardin 9f2a6d33be move remaining helper packages to internal 2021-01-20 13:54:00 -05:00
Martin Atkins 17728c8fe8 internal/terminal: Interrogate and initialize the terminal, if any
This is a helper package that creates a very thin abstraction over
terminal setup, with the main goal being to deal with all of the extra
setup we need to do in order to get a UTF-8-supporting virtual terminal
on a Windows system.
2021-01-13 15:37:04 -08:00
Kristin Laemmert 86a63e8e39
remove unused legacy plugin package (#27493) 2021-01-13 16:13:08 -05:00
Romain Philibert 5ae5c99ba0 fix: do not fail when provider symlink is broken. Only print a warning.
fix https://github.com/hashicorp/terraform/issues/27445
2021-01-08 12:51:09 +01:00
Anthony Sottile 8cd72e51cb
fix file mode on lock file (#27205)
Signed-off-by: Anthony Sottile <asottile@umich.edu>
2020-12-18 11:51:59 -05:00
Martin Atkins 21d6fb5a37 depsfile: Don't panic when lock file is unreadable
Previously we were expecting that the *hcl.File would always be non-nil,
even in error cases. That isn't always true, so now we'll be more robust
about it and explicitly return an empty locks object in that case, along
with the error diagnostics.

In particular this avoids a panic in a strange situation where the user
created a directory where the lock file would normally go. There's no
meaning to such a directory, so it would always be a mistake and so now
we'll return an error message about it, rather than panicking as before.

The error message for the situation where the lock file is a directory is
currently not very specific, but since it's HCL responsible for generating
that message we can't really fix that at this layer. Perhaps in future
we can change HCL to have a specialized error message for that particular
error situation, but for the sake of this commit the goal is only to
stop the panic and return a normal error message.
2020-12-15 17:00:35 -08:00
Martin Atkins 4b3e237668 command/init: Hint about providers in other namespaces
If a user forgets to specify the source address for a provider, Terraform
will assume they meant a provider in the registry.terraform.io/hashicorp/
namespace. If that ultimately doesn't exist, we'll now try to see if
there's some other provider source address recorded in the registry's
legacy provider lookup table, and suggest it if so.

The error message here is a terse one addressed primarily to folks who are
already somewhat familiar with provider source addresses and how to
specify them. Terraform v0.13 had a more elaborate version of this error
message which directed the user to try the v0.13 automatic upgrade tool,
but we no longer have that available in v0.14 and later so the user must
make the fix themselves.
2020-12-10 10:11:27 -08:00
James Bardin 6529659455 internal/providercache: staticcheck 2020-12-02 13:59:20 -05:00
James Bardin d22e08bd32 internal/earlyconfig: staticcheck 2020-12-02 13:59:19 -05:00
James Bardin f0c3ddf5a1 internal/getproviders: staticcheck 2020-12-02 13:59:19 -05:00
James Bardin 276dfe634f internal/providercache: staticcheck 2020-12-02 13:59:19 -05:00
James Bardin 1f359bab03 internal/typeexpr: staticcheck
diagnostics were appended to diags shadowing the intended value.
2020-12-02 13:59:19 -05:00
James Bardin 76cb40005a add grpcwrap.Provisioner
Rename grpcwrap.New() to grpcwrap.Provider()
Add a grpcwrap function to create a test proivisioner plugin.
2020-12-02 12:45:00 -05:00
James Bardin 03d11f988e remove old test provider 2020-12-02 12:45:00 -05:00
James Bardin eee581ae54 simple provider for internal testing
Simple is a minimal provider implementation using the grpcwrap package,
intended for use in internal tests.
2020-12-02 12:44:59 -05:00
James Bardin 65340f51be wrap the terraform provider for grpc tests 2020-12-02 12:44:59 -05:00
James Bardin b1c104834e grpcwrap: convert provider interface to grpcserver
grpcwrap wraps a core providers.Interface and converts it to a
tfplugin5.ProviderServer to allow for easier protocol testing.
2020-12-02 12:44:59 -05:00
James Bardin 811a3a81e9 add test provider bin for e2etest
One e2etest still requires the bin, so add that back temporarily.
2020-12-02 12:16:36 -05:00
James Bardin b32362da9d internal/legacy/builtin/providers/test
Move the test provider into the interal/legacy directory, until we can
factor it out of the e2e test.
2020-12-02 12:16:35 -05:00
James Bardin 5bbac72a85 internal/legacy/helper/plugin
This is used internally for testing
2020-12-02 12:16:35 -05:00
James Bardin e4edce22ca internal/legacy/helper/schema
moving helper/schema into the ineternal/legacy tree
2020-12-02 12:16:35 -05:00
James Bardin a49e7eee8b internal/legacy/terraform
This is a partial copy of the terraform package to preserve the legacy
types for internal use.
2020-12-02 12:16:35 -05:00
James Bardin ccf19fbf5e regenerate plugin proto with correct tooling
The proto file is structurally unchanged, though it now requires a go
import path option.
2020-12-01 14:06:49 -05:00
James Bardin 7eb491719f normalize temp dir paths in tests
The temporary directory on some systems (most notably MacOS) contains
symlinks, which would not be recorded by the installer. In order to make
these paths comparable in the tests we need to eval the symlinks in
the paths before giving them to the installer.
2020-11-20 16:35:31 -05:00
James Bardin d52e17e111 hide provider crashes from panicwrap when logging
When logging is turned on, panicwrap will still see provider crashes and
falsely report them as core crashes, hiding the formatted provider
error. We can trick panicwrap by slightly obfuscating the error line.
2020-11-05 10:54:21 -05:00
Alisdair McDiarmid fd43bc7847
Merge pull request #26783 from hashicorp/alisdair/normalize-version-constraints-before-deduplication
getproviders: Normalize versions before dedupe
2020-11-04 09:42:19 -05:00
Alisdair McDiarmid 90c4f11b61 go get github.com/hashicorp/go-getter@v1.5.0
Includes fix for breaking upstream API change.
2020-11-02 10:46:24 -05:00
Alisdair McDiarmid b1bc0e5d92 getproviders: Normalize versions before dedupe
When rendering a set of version constraints to a string, we normalize
partially-constrained versions. This means converting a version
like 2.68.* to 2.68.0.

Prior to this commit, this normalization was done after deduplication.
This could result in a version constraints string with duplicate
entries, if multiple partially-constrained versions are equivalent. This
commit fixes this by normalizing before deduplicating and sorting.
2020-11-02 10:45:45 -05:00
Martin Atkins e6e0b6ee46 providercache: verify locked hashes for local package dirs
Previously we were only verifying locked hashes for local archive zip
files, but if we have non-ziphash hashes available then we can and should
also verify that a local directory matches at least one of them.

This does mean that folks using filesystem mirrors but yet also running
Terraform across multiple platforms will need to take some extra care to
ensure the hashes pass on all relevant platforms, which could mean using
"terraform providers lock" to pre-seed their lock files with hashes across
all platforms, or could mean using the "packed" directory layout for the
filesystem mirror so that Terraform will end up in the install-from-archive
codepath instead of this install-from-directory codepath, and can thus
verify ziphash too.

(There's no additional documentation about the above here because there's
already general information about this in the lock file documentation
due to some similar -- though not identical -- situations with network
mirrors.)
2020-10-28 07:46:45 -07:00
Martin Atkins 24d32e9ca2 providercache: More exhaustive testing of the main installer
We previously had some tests for some happy paths and a few specific
failures into an empty directory with no existing locks, but we didn't
have tests for the installer respecting existing lock file entries.

This is a start on a more exhaustive set of tests for the installer,
aiming to visit as many of the possible codepaths as we can reasonably
test using this mocking strategy. (Some other codepaths require different
underlying source implementations, etc, so we'll have to visit those in
other tests separately.)
2020-10-28 07:46:45 -07:00
Martin Atkins fc5a41b5e5 depsfile: Allow loading locks from a byte array in memory
This won't be a typical usage pattern for normal code, but will be useful
for tests that need to work with locks as input so that they don't need to
write out a temporary file on disk just to read it back in immediately.
2020-10-28 07:46:45 -07:00
Martin Atkins 430318e262 getproviders: Consistent ordering of terms in VersionConstraintsString
An earlier commit made this remove duplicates, which set the precedent
that this function is trying to canonically represent the _meaning_ of
the version constraints rather than exactly how they were expressed in
the configuration.

Continuing in that vein, now we'll also apply a consistent (though perhaps
often rather arbitrary) ordering to the terms, so that it doesn't change
due to irrelevant details like declarations being written in a different
order in the configuration.

The ordering here is intended to be reasonably intuitive for simple cases,
but constraint strings with many different constraints are hard to
interpret no matter how we order them so the main goal is consistency,
so those watching how the constraints change over time (e.g. in logs of
Terraform output, or in the dependency log file) will see fewer noisy
changes that don't actually mean anything.
2020-10-26 12:44:15 -07:00
Alisdair McDiarmid 8eed942482
Merge pull request #26678 from hashicorp/alisdair/suppress-duplicate-version-constraints
internal: Suppress duplicate version constraints
2020-10-26 11:27:51 -04:00
James Bardin 3225d9ac11 record all plugin panics, and print on main exit
Create a logger that will record any apparent crash output for later
processing.

If the cli command returns with a non-zero exit status, check for any
recorded crashes and add those to the output.
2020-10-26 09:34:03 -04:00
James Bardin f8893785f0 separate core and provider loggers
Now that hclog can independently set levels on related loggers, we can
separate the log levels for different subsystems in terraform.

This adds the new environment variables, `TF_LOG_CORE` and
`TF_LOG_PROVIDER`, which each take the same set of log level arguments,
and only applies to logs from that subsystem. This means that setting
`TF_LOG_CORE=level` will not show logs from providers, and
`TF_LOG_PROVIDER=level` will not show logs from core. The behavior of
`TF_LOG` alone does not change.

While it is not necessarily needed since the default is to disable logs,
there is also a new level argument of `off`, which reflects the
associated level in hclog.
2020-10-23 12:46:32 -04:00
Alisdair McDiarmid fe9a9fadfb internal: Suppress duplicate version constraints
A set of version constraints can contain duplicates. This can happen if
multiple identical constraints are specified throughout a configuration.

When rendering the set, it is confusing to display redundant
constraints. This commit changes the string renderer to only show the
first instance of a given constraint, and adds unit tests for this
function to cover this change.

This also fixes a bug with the locks file generation: previously, a
configuration with redundant constraints would result in this error on
second init:

Error: Invalid provider version constraints

  on .terraform.lock.hcl line 6:
  (source code not available)

The recorded version constraints for provider
registry.terraform.io/hashicorp/random must be written in normalized form:
"3.0.0".
2020-10-22 12:08:00 -04:00