Commit Graph

732 Commits

Author SHA1 Message Date
Martin Atkins e81162c4e1 Move provisioners/ to internal/provisioners/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 1409f30f9c Move providers/ to internal/providers/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins bc31dbc179 Move instances/ to internal/instances/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 05caff2ca3 Move tfdiags/ to internal/tfdiags/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins ec85fb1960 Move communicator/ to internal/communicator/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 4c254cc2be Move httpclient/ to internal/httpclient/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 415ab31db8 Move e2e/ to internal/e2e/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 4c5b866dea Move registry/ to internal/registry/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 2bd2568ad8 Move repl/ to internal/repl/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 70eebe3521 Move dag/ to internal/dag/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins cf93aa8046 Move builtin/ to internal/builtin/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 73dda868cc Move backend/ to internal/backend/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
James Bardin feb7622846 monitor plugin std outputs for unexpected data
Once a plugin process is started, go-plugin will redirect the stdout and
stderr stream through a grpc service and provide those streams to the
client. This is rarely used, as it is prone to failing with races
because those same file descriptors are needed for the initial handshake
and logging setup, but data may be accidentally sent to these
nonetheless.

The usual culprits are stray `fmt.Print` usage where logging was
intended, or the configuration of a logger after the os.Stderr file
descriptor was replaced by go-plugin. These situations are very hard for
provider developers to debug since the data is discarded entirely.

While there may be improvements to be made in the go-plugin package to
configure this behavior, in the meantime we can add a simple monitoring
io.Writer to the streams which will surface th data as warnings in the
logs instead of writing it to `io.Discard`
2021-05-13 16:57:36 -04:00
James Bardin a4a6b40bee update getproviders tests for new key 2021-04-23 11:18:58 -04:00
James Bardin e1d790be61 update HC PublicKey 2021-04-23 11:18:58 -04:00
Paddy 0b5c4a6a2c
Accept TF_LOG=json to enable TRACE logs in JSON format
This is not currently a supported interface, but we plan to release
tool(s) that consume parts of it that are more dependable later,
separately from Terraform CLI itself.
2021-03-16 14:59:15 -07:00
Pam Selle 242f319638
Give suggestions & remind users to use required_providers when provider not in registry (#28014)
* Add helper suggestion when failed registry err

When someone has a failed registry error on init, remind them that
they should have required_providers in every module

* Give suggestion for a provider based on reqs

Suggest another provider on a registry error, from the list of
requirements we have on init. This skips the legacy lookup
process if there is a similar provider existing in requirements.
2021-03-11 08:54:18 -05:00
Masayuki Morita 31a5aa1878
command/init: Add a new flag `-lockfile=readonly` (#27630)
Fixes #27506

Add a new flag `-lockfile=readonly` to `terraform init`.
It would be useful to allow us to suppress dependency lockfile changes
explicitly.

The type of the `-lockfile` flag is string rather than bool, leaving
room for future extensions to other behavior variants.

The readonly mode suppresses lockfile changes, but should verify
checksums against the information already recorded. It should conflict
with the `-upgrade` flag.

Note: In the original use-case described in #27506, I would like to
suppress adding zh hashes, but a test code here suppresses adding h1
hashes because it's easy for testing.

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2021-03-09 11:12:00 -05:00
Kristin Laemmert 106bcd3bf0
update to match new default branch name (#27909) 2021-02-24 13:36:47 -05:00
Kristin Laemmert e5a7586559
moduletest: update provider to match recent provider interface changes (#27904)
this fixes a bad merge on my part
2021-02-24 12:29:45 -05:00
Kristin Laemmert ff05362d51
providers.Interface: rename ValidateDataSourceConfig to ValidateDataResourceConfig (#27874)
* providers.Interface: rename ValidateDataSourceConfig to
ValidateDataResourceConfig

This PR came about after renaming ValidateResourceTypeConfig to
ValidateResourceConfig: I now understand that we'd called it the former
instead of the latter to indicate that the function wasn't necessarily
operating on a resource that actually exists. A possibly-more-accurate
renaming of both functions might then be ValidateManagedResourceConfig
and ValidateDataResourceConfig.

The next commit will update the protocol (v6 only) as well; these are in
separate commits for reviewers and will get squashed together before
merging.

* extend renaming to protov6
2021-02-24 12:04:28 -05:00
Martin Atkins 7f78342953 command: Experimental "terraform test" command
This is just a prototype to gather some feedback in our ongoing research
on integration testing of Terraform modules. The hope is that by having a
command integrated into Terraform itself it'll be easier for interested
module authors to give it a try, and also easier for us to iterate quickly
based on feedback without having to coordinate across multiple codebases.

Everything about this is subject to change even in future patch releases.
Since it's a CLI command rather than a configuration language feature it's
not using the language experiments mechanism, but generates a warning
similar to the one language experiments generate in order to be clear that
backward compatibility is not guaranteed.
2021-02-22 14:21:45 -08:00
Martin Atkins 8330f8e991 internal/moduletest: Experimental module testing helpers
As part of ongoing research into Terraform testing we'd like to use an
experimental feature to validate our current understanding that expressing
tests as part of the Terraform language, as opposed to in some other
language run alongside, is a good and viable way to write practical
module integration tests.

This initial experimental incarnation of that idea is implemented as a
provider, just because that's an easier extension point for research
purposes than a first-class language feature would be. Whether this would
ultimately emerge as a provider similar to this or as custom language
constructs will be a matter for future research, if this first
experiment confirms that tests written in the Terraform language are the
best direction to take.

The previous incarnation of this experiment was an externally-developed
provider apparentlymart/testing, listed on the Terraform Registry. That
helped with showing that there are some useful tests that we can write
in the Terraform language, but integrating such a provider into Terraform
will allow us to make use of it in the also-experimental "terraform test"
command, which will follow in subsequent commits, to see how this might
fit into a development workflow.
2021-02-22 14:21:45 -08:00
Kristin Laemmert 7b7273e3aa
Add support for plugin protocol v6 (#27826)
* Add support for plugin protocol v6

This PR turns on support for plugin protocol v6. A provider can
advertise itself as supporting protocol version 6 and terraform will
use the correct client.

Todo:

The "unmanaged" providers functionality does not support protocol
version, so at the moment terraform will continue to assume that
"unmanaged" providers are on protocol v5. This will require some
upstream work on go-plugin (I believe).

I would like to convert the builtin providers to use protocol v6 in a
future PR; however it is not necessary until we remove protocol v6.

* add e2e test for using both plugin protocol versions

- copied grpcwrap and made a version that returns protocol v6 provider
- copied the test provider, provider-simple, and made a version that's
  using protocol v6 with the above fun
- added an e2etest
2021-02-22 10:22:45 -05:00
James Bardin 3da5d2bdf6
Merge pull request #27692 from alexwlchan/abolish-the-heirarchy
Correct the spelling of heirarchy/hierarchy throughout
2021-02-18 16:34:28 -05:00
Pam Selle e1c25337c7 Delete readme, add doc.go to helper/schema
Remove the README that had old user-facing information, replacing
it with a doc.go that describes the package and points to the
plugin SDK for external consumers.
2021-02-18 12:30:10 -05:00
Kristin Laemmert f6505870cc
Mildwonkey/providers interface renaming (#27805)
* providers.Interface: huge renamification

This commit renames a handful of functions in the providers.Interface to
match changes made in protocol v6. The following commit implements this
change across the rest of the codebase; I put this in a separate commit
for ease of reviewing and will squash these together when merging.

One noteworthy detail: protocol v6 removes the config from the
ValidateProviderConfigResponse, since it's never been used. I chose to
leave that in place in the interface until we deprecate support for
protocol v5 entirely.

Note that none of these changes impact current providers using protocol
v5; the protocol is unchanged. Only the translation layer between the
proto and terraform have changed.
2021-02-18 10:13:43 -05:00
Kristin Laemmert 7943312ebd
Merge pull request #27699 from hashicorp/mildwonkey/protocolv6
Mildwonkey/protocolv6
2021-02-16 14:51:53 -05:00
Martin Atkins f91a3b87c1 terminal: Helpers for doing fmt-ish operations on the streams
It's pretty common to want to apply the various fmt.Fprint... functions
to our two output streams, and so to make that much less noisy at the
callsite here we have a small number of very thin wrappers around the
underlying fmt package functionality.

Although we're aiming to not have too much abstraction in this "terminal"
package, this seems justified in that it is only a very thin wrapper
around functionality that most Go programmers are already familiar with,
and so the risk of this causing any surprises is low and the improvement
to readability of callers seems worth it.
2021-02-12 11:24:13 -08:00
Martin Atkins ffba8064ed terminal: StreamsForTesting helper
This is to allow convenient testing of functions that are designed to work
directly with *terminal.Streams or the individual stream objects inside.

Because the InputStream and OutputStream APIs expose directly an *os.File,
this does some extra work to set up OS-level pipes so we can capture the
output into local buffers to make test assertions against. The idea here
is to keep the tricky stuff we need for testing confined to the test
codepaths, so that the "real" codepaths don't end up needing to work
around abstractions that are otherwise unnecessary.
2021-02-10 11:51:56 -08:00
Pam Selle aa24bfec47 Emit ProviderAlreadyInstalled when provider installed
Emit the ProviderAlreadyInstalled event when we successfully verify
that we've already installed this provider and are skipping
installation
2021-02-09 11:08:49 -05:00
Kristin Laemmert f3a057eb35 plugin-protocol v6
This is the first commit for plugin protocol v6. This is currently
unused (dead) code; future commits will add the necessary conversion
packages, extend configschema, and modify the providers.Interface.

The new plugin protocol includes the following changes:

- A new field has been added to Attribute: NestedType. This will be the
  key new feature in plugin protocol v6
- Several massages were renamed for consistency with the verb-noun
  pattern seen in _most_ messages.
- The prepared_config has been removed from PrepareProviderConfig
  (renamed ValidateProviderConfig), as it has never been used.
- The provisioner service has been removed entirely. This has no impact
  on built-in provisioners. 3rd party provisioners are not supported by
  the SDK and are not included in this protocol at all.
2021-02-05 13:19:55 -05:00
Alex Chan 4ec80fa18d Correct the spelling of heirarchy/hierarchy throughout 2021-02-05 15:07:04 +00:00
Jonathan Hall 49ee3d3ef8 Grammar nit: "setup" as a verb should be spelled "set up" 2021-01-26 20:39:11 +01:00
Pam Selle aedca597dd Reuse installed target dir providers in init
In init, we can check to see if the target dir already has the
provider we are seeking and skip further querying/installing of
that provider.
2021-01-25 11:13:57 -05:00
Nick Fagerlund a60120477c Update links to CLI docs in code comments, messages, and readme 2021-01-22 12:22:21 -08:00
Pam Selle 084cec6346
Merge pull request #27447 from Filirom1/fix-27445
Providers: Fix init with broken link in plugin_cache_dir
2021-01-22 10:30:55 -05:00
James Bardin 9f2a6d33be move remaining helper packages to internal 2021-01-20 13:54:00 -05:00
Martin Atkins 17728c8fe8 internal/terminal: Interrogate and initialize the terminal, if any
This is a helper package that creates a very thin abstraction over
terminal setup, with the main goal being to deal with all of the extra
setup we need to do in order to get a UTF-8-supporting virtual terminal
on a Windows system.
2021-01-13 15:37:04 -08:00
Kristin Laemmert 86a63e8e39
remove unused legacy plugin package (#27493) 2021-01-13 16:13:08 -05:00
Romain Philibert 5ae5c99ba0 fix: do not fail when provider symlink is broken. Only print a warning.
fix https://github.com/hashicorp/terraform/issues/27445
2021-01-08 12:51:09 +01:00
Anthony Sottile 8cd72e51cb
fix file mode on lock file (#27205)
Signed-off-by: Anthony Sottile <asottile@umich.edu>
2020-12-18 11:51:59 -05:00
Martin Atkins 21d6fb5a37 depsfile: Don't panic when lock file is unreadable
Previously we were expecting that the *hcl.File would always be non-nil,
even in error cases. That isn't always true, so now we'll be more robust
about it and explicitly return an empty locks object in that case, along
with the error diagnostics.

In particular this avoids a panic in a strange situation where the user
created a directory where the lock file would normally go. There's no
meaning to such a directory, so it would always be a mistake and so now
we'll return an error message about it, rather than panicking as before.

The error message for the situation where the lock file is a directory is
currently not very specific, but since it's HCL responsible for generating
that message we can't really fix that at this layer. Perhaps in future
we can change HCL to have a specialized error message for that particular
error situation, but for the sake of this commit the goal is only to
stop the panic and return a normal error message.
2020-12-15 17:00:35 -08:00
Martin Atkins 4b3e237668 command/init: Hint about providers in other namespaces
If a user forgets to specify the source address for a provider, Terraform
will assume they meant a provider in the registry.terraform.io/hashicorp/
namespace. If that ultimately doesn't exist, we'll now try to see if
there's some other provider source address recorded in the registry's
legacy provider lookup table, and suggest it if so.

The error message here is a terse one addressed primarily to folks who are
already somewhat familiar with provider source addresses and how to
specify them. Terraform v0.13 had a more elaborate version of this error
message which directed the user to try the v0.13 automatic upgrade tool,
but we no longer have that available in v0.14 and later so the user must
make the fix themselves.
2020-12-10 10:11:27 -08:00
James Bardin 6529659455 internal/providercache: staticcheck 2020-12-02 13:59:20 -05:00
James Bardin d22e08bd32 internal/earlyconfig: staticcheck 2020-12-02 13:59:19 -05:00
James Bardin f0c3ddf5a1 internal/getproviders: staticcheck 2020-12-02 13:59:19 -05:00
James Bardin 276dfe634f internal/providercache: staticcheck 2020-12-02 13:59:19 -05:00
James Bardin 1f359bab03 internal/typeexpr: staticcheck
diagnostics were appended to diags shadowing the intended value.
2020-12-02 13:59:19 -05:00
James Bardin 76cb40005a add grpcwrap.Provisioner
Rename grpcwrap.New() to grpcwrap.Provider()
Add a grpcwrap function to create a test proivisioner plugin.
2020-12-02 12:45:00 -05:00
James Bardin 03d11f988e remove old test provider 2020-12-02 12:45:00 -05:00
James Bardin eee581ae54 simple provider for internal testing
Simple is a minimal provider implementation using the grpcwrap package,
intended for use in internal tests.
2020-12-02 12:44:59 -05:00
James Bardin 65340f51be wrap the terraform provider for grpc tests 2020-12-02 12:44:59 -05:00
James Bardin b1c104834e grpcwrap: convert provider interface to grpcserver
grpcwrap wraps a core providers.Interface and converts it to a
tfplugin5.ProviderServer to allow for easier protocol testing.
2020-12-02 12:44:59 -05:00
James Bardin 811a3a81e9 add test provider bin for e2etest
One e2etest still requires the bin, so add that back temporarily.
2020-12-02 12:16:36 -05:00
James Bardin b32362da9d internal/legacy/builtin/providers/test
Move the test provider into the interal/legacy directory, until we can
factor it out of the e2e test.
2020-12-02 12:16:35 -05:00
James Bardin 5bbac72a85 internal/legacy/helper/plugin
This is used internally for testing
2020-12-02 12:16:35 -05:00
James Bardin e4edce22ca internal/legacy/helper/schema
moving helper/schema into the ineternal/legacy tree
2020-12-02 12:16:35 -05:00
James Bardin a49e7eee8b internal/legacy/terraform
This is a partial copy of the terraform package to preserve the legacy
types for internal use.
2020-12-02 12:16:35 -05:00
James Bardin ccf19fbf5e regenerate plugin proto with correct tooling
The proto file is structurally unchanged, though it now requires a go
import path option.
2020-12-01 14:06:49 -05:00
James Bardin 7eb491719f normalize temp dir paths in tests
The temporary directory on some systems (most notably MacOS) contains
symlinks, which would not be recorded by the installer. In order to make
these paths comparable in the tests we need to eval the symlinks in
the paths before giving them to the installer.
2020-11-20 16:35:31 -05:00
James Bardin d52e17e111 hide provider crashes from panicwrap when logging
When logging is turned on, panicwrap will still see provider crashes and
falsely report them as core crashes, hiding the formatted provider
error. We can trick panicwrap by slightly obfuscating the error line.
2020-11-05 10:54:21 -05:00
Alisdair McDiarmid fd43bc7847
Merge pull request #26783 from hashicorp/alisdair/normalize-version-constraints-before-deduplication
getproviders: Normalize versions before dedupe
2020-11-04 09:42:19 -05:00
Alisdair McDiarmid 90c4f11b61 go get github.com/hashicorp/go-getter@v1.5.0
Includes fix for breaking upstream API change.
2020-11-02 10:46:24 -05:00
Alisdair McDiarmid b1bc0e5d92 getproviders: Normalize versions before dedupe
When rendering a set of version constraints to a string, we normalize
partially-constrained versions. This means converting a version
like 2.68.* to 2.68.0.

Prior to this commit, this normalization was done after deduplication.
This could result in a version constraints string with duplicate
entries, if multiple partially-constrained versions are equivalent. This
commit fixes this by normalizing before deduplicating and sorting.
2020-11-02 10:45:45 -05:00
Martin Atkins e6e0b6ee46 providercache: verify locked hashes for local package dirs
Previously we were only verifying locked hashes for local archive zip
files, but if we have non-ziphash hashes available then we can and should
also verify that a local directory matches at least one of them.

This does mean that folks using filesystem mirrors but yet also running
Terraform across multiple platforms will need to take some extra care to
ensure the hashes pass on all relevant platforms, which could mean using
"terraform providers lock" to pre-seed their lock files with hashes across
all platforms, or could mean using the "packed" directory layout for the
filesystem mirror so that Terraform will end up in the install-from-archive
codepath instead of this install-from-directory codepath, and can thus
verify ziphash too.

(There's no additional documentation about the above here because there's
already general information about this in the lock file documentation
due to some similar -- though not identical -- situations with network
mirrors.)
2020-10-28 07:46:45 -07:00
Martin Atkins 24d32e9ca2 providercache: More exhaustive testing of the main installer
We previously had some tests for some happy paths and a few specific
failures into an empty directory with no existing locks, but we didn't
have tests for the installer respecting existing lock file entries.

This is a start on a more exhaustive set of tests for the installer,
aiming to visit as many of the possible codepaths as we can reasonably
test using this mocking strategy. (Some other codepaths require different
underlying source implementations, etc, so we'll have to visit those in
other tests separately.)
2020-10-28 07:46:45 -07:00
Martin Atkins fc5a41b5e5 depsfile: Allow loading locks from a byte array in memory
This won't be a typical usage pattern for normal code, but will be useful
for tests that need to work with locks as input so that they don't need to
write out a temporary file on disk just to read it back in immediately.
2020-10-28 07:46:45 -07:00
Martin Atkins 430318e262 getproviders: Consistent ordering of terms in VersionConstraintsString
An earlier commit made this remove duplicates, which set the precedent
that this function is trying to canonically represent the _meaning_ of
the version constraints rather than exactly how they were expressed in
the configuration.

Continuing in that vein, now we'll also apply a consistent (though perhaps
often rather arbitrary) ordering to the terms, so that it doesn't change
due to irrelevant details like declarations being written in a different
order in the configuration.

The ordering here is intended to be reasonably intuitive for simple cases,
but constraint strings with many different constraints are hard to
interpret no matter how we order them so the main goal is consistency,
so those watching how the constraints change over time (e.g. in logs of
Terraform output, or in the dependency log file) will see fewer noisy
changes that don't actually mean anything.
2020-10-26 12:44:15 -07:00
Alisdair McDiarmid 8eed942482
Merge pull request #26678 from hashicorp/alisdair/suppress-duplicate-version-constraints
internal: Suppress duplicate version constraints
2020-10-26 11:27:51 -04:00
James Bardin 3225d9ac11 record all plugin panics, and print on main exit
Create a logger that will record any apparent crash output for later
processing.

If the cli command returns with a non-zero exit status, check for any
recorded crashes and add those to the output.
2020-10-26 09:34:03 -04:00
James Bardin f8893785f0 separate core and provider loggers
Now that hclog can independently set levels on related loggers, we can
separate the log levels for different subsystems in terraform.

This adds the new environment variables, `TF_LOG_CORE` and
`TF_LOG_PROVIDER`, which each take the same set of log level arguments,
and only applies to logs from that subsystem. This means that setting
`TF_LOG_CORE=level` will not show logs from providers, and
`TF_LOG_PROVIDER=level` will not show logs from core. The behavior of
`TF_LOG` alone does not change.

While it is not necessarily needed since the default is to disable logs,
there is also a new level argument of `off`, which reflects the
associated level in hclog.
2020-10-23 12:46:32 -04:00
Alisdair McDiarmid fe9a9fadfb internal: Suppress duplicate version constraints
A set of version constraints can contain duplicates. This can happen if
multiple identical constraints are specified throughout a configuration.

When rendering the set, it is confusing to display redundant
constraints. This commit changes the string renderer to only show the
first instance of a given constraint, and adds unit tests for this
function to cover this change.

This also fixes a bug with the locks file generation: previously, a
configuration with redundant constraints would result in this error on
second init:

Error: Invalid provider version constraints

  on .terraform.lock.hcl line 6:
  (source code not available)

The recorded version constraints for provider
registry.terraform.io/hashicorp/random must be written in normalized form:
"3.0.0".
2020-10-22 12:08:00 -04:00
James Bardin bc1a841d65
Merge pull request #26665 from hashicorp/jbardin/logging
Restore "crash.log" behavior and remove prefixedio
2020-10-22 10:34:51 -04:00
James Bardin c2af5333e8 use a log sink to capture logs for panicwrap
Use a separate log sink to always capture trace logs for the panicwrap
handler to write out in a crash log.

This requires creating a log file in the outer process and passing that
path to the child process to log to.
2020-10-21 17:29:07 -04:00
James Bardin b61488a8ba write traceback to log crash log 2020-10-21 17:29:07 -04:00
James Bardin 1d9d82973b move panicwrap handler to logging package 2020-10-21 13:47:16 -04:00
Alisdair McDiarmid 7a31e56cb7
Merge pull request #26637 from hashicorp/alisdair/fix-locksfile-unconstrained-versions
internal: Fix lockfile constraint output for 1.2.*
2020-10-21 12:05:19 -04:00
Martin Atkins 1fe314b70b getproviders: Don't log if EvalSymlinks changes nothing
Previously this codepath was generating a confusing message in the absense
of any symlinks, because filepath.EvalSymlinks returns a successful result
if the target isn't a symlink.

Now we'll emit the log line only if filepath.EvalSymlinks returns a
result that's different in a way that isn't purely syntactic (which
filepath.Clean would "fix").

The new message is a little more generic because technically we've not
actually ensured that a difference here was caused by a symlink and so
we shouldn't over-promise and generate something potentially misleading.
2020-10-21 07:49:11 -07:00
Martin Atkins 2332a7ab47 replacefile: don't create the temporary file in TMPDIR
ioutil.TempFile has a special case where an empty string for its dir
argument is interpreted as a request to automatically look up the system
temporary directory, which is commonly /tmp .

We don't want that behavior here because we're specifically trying to
create the temporary file in the same directory as the file we're hoping
to replace. If the file gets created in /tmp then it might be on a
different device and thus the later atomic rename won't work.

Instead, we'll add our own special case to explicitly use "." when the
given filename is in the current working directory. That overrides the
special automatic behavior of ioutil.TempFile and thus forces the
behavior we need.

This hadn't previously mattered for earlier callers of this code because
they were creating files in subdirectories, but this codepath was failing
for the dependency lock file due to it always being created directly
in the current working directory.

Unfortunately since this is a picky implementation detail I couldn't find
a good way to write a unit test for it without considerable refactoring.
Instead, I verified manually that the temporary filename wasn't in /tmp on
my Linux system, and hope that the comment inline will explain this
situation well enough to avoid an accidental regression in future
maintenence.
2020-10-20 16:29:59 -07:00
Alisdair McDiarmid 9576a5b2d8 internal: Fix lockfile constraint output for 1.2.*
If a configuration requires a partial provider version (with some parts
unspecified), Terraform considers this as a constrained-to-zero version.
For example, a version constraint of 1.2 will result in an attempt to
install version 1.2.0, even if 1.2.1 is available.

When writing the dependency locks file, we previously would write 1.2.*,
as this is the in-memory representation of 1.2. This would then cause an
error on re-reading the locks file, as this is not a valid constraint
format.

Instead, we now explicitly convert the constraint to its zero-filled
representation before writing the locks file. This ensures that it
correctly round-trips.

Because this change is made in getproviders.VersionConstraintsString, it
also affects the output of the providers sub-command.
2020-10-20 10:14:03 -04:00
James Bardin a3a20e0396 remove LevelFilter
We want to always be using the hclogger to filter whenever possible
2020-10-19 14:31:05 -04:00
James Bardin 0b31ffa587 use a single log writer
Use a single log writer instance for all std library logging.

Setup the std log writer in the logging package, and remove boilerplate
from test packages.
2020-10-19 14:29:54 -04:00
James Bardin abf6b9b378 get properly configured hcloggers
make sure plugins get hcloggers configured to match core
2020-10-19 14:29:54 -04:00
James Bardin 211edf5d75 use hclog as the default logger
Inject hclog as the default logger in the main binary.
2020-10-19 14:29:54 -04:00
James Bardin 6ca477f042 move helper/logging to internal
remove a dead code file too
2020-10-19 14:27:53 -04:00
Martin Atkins 0009768c7f internal/depsfile: Update the dependency lock file atomically
In this case, "atomic" means that there will be no situation where the
file contains only part of the newContent data, and therefore other
software monitoring the file for changes (using a mechanism like inotify)
won't encounter a truncated file.

It does _not_ mean that there can't be existing filehandles open against
the old version of the file. On Windows systems the write will fail in
that case, but on Unix systems the write will typically succeed but leave
the existing filehandles still pointing at the old version of the file.
They'll need to reopen the file in order to see the new content.
2020-10-14 08:01:19 -07:00
Martin Atkins 55e6f64977 internal/depsfile: Factor out our atomic file replacement logic
This originated in the cliconfig code to write out credentials files. The
Windows implementation of this in particular was quite onerous to get
right because it needs a very specific sequence of operations to avoid
running into exclusive file locks, and so by factoring this out with
only cosmetic modification we can avoid repeating all of that engineering
effort for other atomic file writing use-cases.
2020-10-14 08:01:19 -07:00
Martin Atkins 0bbbb9c64b configs: Experimental support for optional object type attributes
This builds on an experimental feature in the underlying cty library which
allows marking specific attribtues of an object type constraint as
optional, which in turn modifies how the cty conversion package handles
missing attributes in a source value: it will silently substitute a null
value of the appropriate type rather than returning an error.

In order to implement the experiment this commit temporarily forks the
HCL typeexpr extension package into a local internal/typeexpr package,
where I've extended the type constraint syntax to allow annotating object
type attributes as being optional using the HCL function call syntax.
If the experiment is successful -- both at the Terraform layer and in
the underlying cty library -- we'll likely send these modifications to
upstream HCL so that other HCL-based languages can potentially benefit
from this new capability.

Because it's experimental, the optional attribute modifier is allowed only
with an explicit opt-in to the module_variable_optional_attrs experiment.
2020-10-12 10:12:28 -07:00
Martin Atkins e1cf0ac801 internal/depsfile: Control how the "hashes" value is formatted
Previously we were just letting hclwrite do its default formatting
behavior here. The current behavior there isn't ideal anyway -- it puts
big data structures all on one line -- but even ignoring that our goal
for this file format is to keep things in a highly-normalized shape so
that diffs against the file are clear and easy to read.

With that in mind, here we directly control how we write that value into
the file, which means that later changes to hclwrite's list/set
presentation won't affect it, regardless of what form they take.
2020-10-09 09:26:23 -07:00
Martin Atkins 2611e08430 command/init: Mention using the lock file for provider selection
This probably isn't the best UI we could do here, but it's a placeholder
for now just to avoid making it seem like we're ignoring the lock file
and checking for new versions anyway.
2020-10-09 09:26:23 -07:00
Martin Atkins b3f5c7f1e6 command/init: Read, respect, and update provider dependency locks
This changes the approach used by the provider installer to remember
between runs which selections it has previously made, using the lock file
format implemented in internal/depsfile.

This means that version constraints in the configuration are considered
only for providers we've not seen before or when -upgrade mode is active.
2020-10-09 09:26:23 -07:00
Martin Atkins 4a1b081afb depsfile: Locks.Equal and Locks.Empty methods
These are helper functions to give the installation UI some hints about
whether the lock file has changed so that it can in turn give the user
advice about it. The UI-layer callers of these will follow in a later
commit.
2020-10-09 09:26:23 -07:00
Kristin Laemmert 04be220f5f deprecate helper/copy
helper/copy CopyDir was used heavily in tests. It differes from
internal/copydir in a few ways, the main one being that it creates the
dst directory while the internal version expected the dst to exist
(there are other differences, which is why I did not just switch tests
to using internal's CopyDir).

I moved the CopyDir func from helper/copy into command_test.go; I could
also have moved it into internal/copy and named it something like
CreateDirAndCopy so if that seems like a better option please let me
know.

helper/copy/CopyFile was used in a couple of spots so I moved it into
internal, at which point I thought it made more sense to rename the
package copy (instead of copydir).

There's also a `go mod tidy` included.
2020-10-08 08:42:16 -04:00
Kristin Laemmert 3933cbd491
remove LegacyProvider (#26433) 2020-10-05 08:33:49 -04:00
Martin Atkins 593cf7b4d5 didyoumean: move from "helper" to "internal"
This new-ish package ended up under "helper" during the 0.12 cycle for
want of some other place to put it, but in retrospect that was an odd
choice because the "helper/" tree is otherwise a bunch of legacy code from
when the SDK lived in this repository.

Here we move it over into the "internal" directory just to distance it
from the guidance of not using "helper/" packages in new projects;
didyoumean is a package we actively use as part of error message hints.
2020-10-02 13:35:07 -07:00
Martin Atkins 59b116f7bf command/init: Remove support for legacy provider addresses
We no longer need to support 0.12-and-earlier-style provider addresses
because users should've upgraded their existing configurations and states
on Terraform 0.13 already.

For now this is only checked in the "init" command, because various test
shims are still relying on the idea of legacy providers the core layer.
However, rejecting these during init is sufficient grounds to avoid
supporting legacy provider addresses in the new dependency lock file
format, and thus sets the stage for a more severe removal of legacy
provider support in a later commit.
2020-09-30 08:54:57 -07:00
Martin Atkins 0b734a2803 command: Make provider installation interruptible
In earlier commits we started to make the installation codepath
context-aware so that it could be canceled in the event of a SIGINT, but
we didn't complete wiring that through the API of the getproviders
package.

Here we make the getproviders.Source interface methods, along with some
other functions that can make network requests, take a context.Context
argument and act appropriately if that context is cancelled.

The main providercache.Installer.EnsureProviderVersions method now also
has some context-awareness so that it can abort its work early if its
context reports any sort of error. That avoids waiting for the process
to wind through all of the remaining iterations of the various loops,
logging each request failure separately, and instead returns just
a single aggregate "canceled" error.

We can then set things up in the "terraform init" and
"terraform providers mirror" commands so that the context will be
cancelled if we get an interrupt signal, allowing provider installation
to abort early while still atomically completing any local-side effects
that may have started.
2020-09-29 10:00:35 -07:00
Martin Atkins ef64df950c getproviders: Prepare for having multiple valid hashes per package
As we continue iterating towards saving valid hashes for a package in a
depsfile lock file after installation and verifying them on future
installation, this prepares getproviders for the possibility of having
multiple valid hashes per package.

This will arise in future commits for two reasons:
- We will need to support both the legacy "zip hash" hashing scheme and
  the new-style content-based hashing scheme because currently the
  registry protocol is only able to produce the legacy scheme, but our
  other installation sources prefer the content-based scheme. Therefore
  packages will typically have a mixture of hashes of both types.
- Installing from an upstream registry will save the hashes for the
  packages across all supported platforms, rather than just the current
  platform, and we'll consider all of those valid for future installation
  if we see both successful matching of the current platform checksum and
  a signature verification for the checksums file as a whole.

This also includes some more preparation for the second case above in that
signatureAuthentication now supports AcceptableHashes and returns all of
the zip-based hashes it can find in the checksums file. This is a bit of
an abstraction leak because previously that authenticator considered its
"document" to just be opaque bytes, but we want to make sure that we can
only end up trusting _all_ of the hashes if we've verified that the
document is signed. Hopefully we'll make this better in a future commit
with some refactoring, but that's deferred for now in order to minimize
disruption to existing codepaths while we work towards a provider locking
MVP.
2020-09-24 14:01:54 -07:00
Martin Atkins 6694cfaa0e getproviders: Add a real type Hash for package hashes
The logic for what constitutes a valid hash and how different hash schemes
are represented was starting to get sprawled over many different files and
packages.

Consistently with other cases where we've used named types to gather the
definition of a particular string into a single place and have the Go
compiler help us use it properly, this introduces both getproviders.Hash
representing a hash value and getproviders.HashScheme representing the
idea of a particular hash scheme.

Most of this changeset is updating existing uses of primitive strings to
uses of getproviders.Hash. The new type definitions are in
internal/getproviders/hash.go.
2020-09-24 14:01:54 -07:00
Martin Atkins 264a3cf031 depsfile: Flatten the "hashes" locks to a single set of strings
Although origin registries return specific [filename, hash] pairs, our
various different installation methods can't produce a structured mapping
from platform to hash without breaking changes.

Therefore, as a compromise, we'll continue to do platform-specific checks
against upstream data in the cases where that's possible (installation
from origin registry or network mirror) but we'll treat the lock file as
just a flat set of equally-valid hashes, at least one of which must match
after we've completed whatever checks we've made against the
upstream-provided checksums/signatures.

This includes only the minimal internal/getproviders updates required to
make this compile. A subsequent commit will update that package to
actually support the idea of verifying against multiple hashes.
2020-09-24 14:01:54 -07:00
Martin Atkins b2c0ccdf96 internal/getproviders: Allow PackageMeta to carry acceptable hashes
The "acceptable hashes" for a package is a set of hashes that the upstream
source considers to be good hashes for checking whether future installs
of the same provider version are considered to match this one.

Because the acceptable hashes are a package authentication concern and
they already need to be known (at least in part) to implement the
authenticators, here we add AcceptableHashes as an optional extra method
that an authenticator can implement.

Because these are hashes chosen by the upstream system, the caller must
make its own determination about their trustworthiness. The result of
authentication is likely to be an input to that, for example by
distrusting hashes produced by an authenticator that succeeds but doesn't
report having validated anything.
2020-09-24 14:01:54 -07:00
Martin Atkins e843097e52 internal/getproviders: Formalize the "ziphash" hashing scheme
This is the pre-existing hashing scheme that was initially built for
releases.hashicorp.com and then later reused for the provider registry
protocol, which takes a SHA256 hash of the official distribution .zip file
and formats it as lowercase hex.

This is a non-ideal hash scheme because it works only for
PackageLocalArchive locations, and so we can't verify package directories
on local disk against such hashes. However, the registry protocol is now
a compatibility constraint and so we're going to need to support this
hashing scheme for the foreseeable future.
2020-09-24 14:01:54 -07:00
Martin Atkins 773dd56b42 internal/depsfile: Introduce the concept of "non-lockable" providers
It doesn't make sense for a built-in provider to appear in a lock file
because built-in providers have no version independent of the version of
Terraform they are compiled into.

We also exclude legacy providers here, because they were supported only
as a transitional aid to enable the Terraform 0.13 upgrade process and
are not intended for explicit selection.

The provider installer will, once it's updated to understand dependency
locking, use this concept to decide which subset of its selections to
record in the dependency lock file for reference for future installation
requests.
2020-09-08 09:50:58 -07:00
Martin Atkins 98e2e69abb internal/depsfile: SaveLocksToFile implementation
This is an initial implementation of writing locks back to a file on disk.
This initial implementation is incomplete because it does not write the
changes to the new file atomically. We'll revisit that in a later commit
as we return to polish these codepaths, once we've proven out this
package's design by integrating it with Terraform's provider installer.
2020-09-08 09:50:58 -07:00
Martin Atkins 92723661d0 internal/depsfile: Loading locks from HCL files on disk
This is the initial implementation of the parser/decoder portion of the
new dependency lock file handler. It's currently dead code because the
caller isn't written yet. We'll continue to build out this functionality
here until we have the basic level of both load and save functionality
before introducing this into the provider installer codepath.
2020-09-08 09:50:58 -07:00
Martin Atkins 6993ecb0a6 internal/getproviders: VersionConstraintsString for "~> 2" input
The version constraint parser allows "~> 2", but it behavior is identical
to "~> 2.0". Due to a quirk of the constraint parser (caused by the fact
that it supports both Ruby-style and npm/cargo-style constraints), it
ends up returning "~> 2" with the minor version marked as "unconstrained"
rather than as zero, but that means the same thing as zero in this context
anyway and so we'll prefer to stringify as "~> 2.0" so that we can be
clearer about how Terraform is understanding that version constraint.
2020-09-08 09:50:58 -07:00
Alisdair McDiarmid 5587509bcf internal: Fix providercache test failures on macOS
For reasons that are unclear, these two tests just started failing on
macOS very recently. The failure looked like:

    PackageDir: strings.Join({
      "/",
+     "private/",
      "var/folders/3h/foobar/T/terraform-test-p",
      "rovidercache655312854/registry.terraform.io/hashicorp/null/2.0.0",
      "/windows_amd64",
    },

Speculating that the macOS temporary directory moved into the /private
directory, I added a couple of EvalSymlinks calls and the tests pass
again.

No other unit tests appear to be affected by this at the moment.
2020-09-04 16:09:57 -04:00
Alisdair McDiarmid fc7e467d19 command: Add redirect support to 0.13upgrade
If a provider changes namespace in the registry, we can detect this when
running the 0.13upgrade command. As long as there is a version matching
the user's constraints, we now use the provider's new source address.
Otherwise, warn the user that the provider has moved and a version
upgrade is necessary to move to it.
2020-08-31 14:53:35 -04:00
Alan D. Salewski f63c38d6a1
internal/initwd: allow tests to pass when $PWD contains symlinks (#26015)
Fixes #26014
2020-08-27 10:02:22 -04:00
Martin Atkins 2bd2a9a923 internal/getproviders: HTTPMirrorSource implementation
We previously had this just stubbed out because it was a stretch goal for
the v0.13.0 release and it ultimately didn't make it in.

Here we fill out the existing stub -- with a minor change to its interface
so it can access credentials -- with a client implementation that is
compatible with the directory structure produced by the
"terraform providers mirror" subcommand, were the result to be published
on a static file server.
2020-08-26 13:18:08 -07:00
Martin Atkins 146e983c36 internal/getproviders: package authenticator for our new-style hashes
Earlier we introduced a new package hashing mechanism that is compatible
with both packed and unpacked packages, because it's a hash of the
contents of the package rather than of the archive it's delivered in.
However, we were using that only for the local selections file and not
for any remote package authentication yet.

The provider network mirrors protocol includes new-style hashes as a step
towards transitioning over to the new hash format in all cases, so this
new authenticator is here in preparation for verifying the checksums of
packages coming from network mirrors, for mirrors that support them.

For now this leaves us in a kinda confusing situation where we have both
NewPackageHashAuthentication for the new style and
NewArchiveChecksumAuthentication for the old style, which for the moment
is represented only by a doc comment on the latter. Hopefully we can
remove NewArchiveChecksumAuthentication in a future commit, if we can
get the registry updated to use the new hashing format.
2020-08-26 13:18:08 -07:00
Alisdair McDiarmid 915f53af23 internal: Clean up package install temp file
The installFromHTTPURL function downloads a package to a temporary file,
then delegates to installFromLocalArchive to install it. We were
previously not deleting the temporary file afterwards. This commit fixes
that.
2020-08-25 08:35:32 -04:00
Martin Atkins ce67a818db internal/getproviders: Allow basedir for local search to be symlink
The SearchLocalDirectory function was intentionally written to only
support symlinks at the leaves so that it wouldn't risk getting into an
infinite loop traversing intermediate symlinks, but that rule was also
applying to the base directory itself.

It's pretty reasonable to put your local plugins in some location
Terraform wouldn't normally search (e.g. because you want to get them from
a shared filesystem mounted somewhere) and creating a symlink from one
of the locations Terraform _does_ search is a convenient way to help
Terraform find those without going all in on the explicit provider
installation methods configuration that is intended for more complicated
situations.

To allow for that, here we make a special exception for the base
directory, resolving that first before we do any directory walking.

In order to help with debugging a situation where there are for some
reason symlinks at intermediate levels inside the search tree, we also now
emit a WARN log line in that case to be explicit that symlinks are not
supported there and to hint to put the symlink at the top-level if you
want to use symlinks at all.

(The support for symlinks at the deepest level of search is not mentioned
in this message because we allow it primarily for our own cache linking
behavior.)
2020-08-18 14:28:50 -07:00
Alisdair McDiarmid 440543f427 internal/providercache: Fix bug when symlink fails
When installing a provider which is already cached, we attempt to create
a symlink from the install directory targeting the cache. If symlinking
fails due to missing OS/filesystem support, we instead want to copy the
cached provider.

The fallback code to do this would always fail, due to a missing target
directory. This commit fixes that. I was unable to find a way to add
automated tests around this, but I have manually verified the fix on
Windows 8.1.
2020-07-23 11:36:46 -04:00
Alisdair McDiarmid 3b1347ac1a providercache: Validate provider executable file
At the end of the EnsureProviderVersions process, we generate a lockfile
of the selected and installed provider versions. This includes a hash of
the unpacked provider directory.

When calculating this hash and generating the lockfile, we now also
verify that the provider directory contains a valid executable file. If
not, we return an error for this provider and trigger the installer's
HashPackageFailure event. Note that this event is not yet processed by
terraform init; that comes in the next commit.
2020-07-07 15:20:17 -04:00
Alisdair McDiarmid a18b531b14 getproviders: FakeInstallablePackageMeta filename
Add an optional execFilename argument to the test helper function
FakeInstallablePackageMeta, which allows the creation of invalid
packages.
2020-07-07 15:18:30 -04:00
Alisdair McDiarmid 8e87ccb689 providercache: Lazily detect executable file
Instead of searching the installed provider package directory for a
binary as we install it, we can lazily detect the executable as it is
required. Doing so allows us to separately report an invalid unpacked
package, giving the user more actionable error messages.
2020-07-07 15:18:27 -04:00
Kristin Laemmert 47e657c611
internal/getproviders: decode and return any registry warnings (#25337)
* internal/getproviders: decode and return any registry warnings

The public registry may include a list of warnings in the "versions"
response for any given provider. This PR adds support for warnings from
the registry and an installer event to return those warnings to the
user.
2020-06-25 10:49:48 -04:00
Kristin Laemmert 1b8f4566fa
internal/initwd: fix panics with relative submodules in DirFromModule (#25250)
* internal/initwd: fix panics with relative submodules in DirFromModule

There were two related issues here:

1. panic with any local module with submodules
1. panic with a relative directory that was above the workdir ("../")

The first panic was caused by the local installer looking up the root
module with the (nonexistant) key "root.", instead of "".

The second panic was caused by the installer trying to determine the
relative path from ".". This was fixed by detecting "." as the source
path and using the absolute path for the call to filepath.Rel.

Added test cases for both panics and updated the existing e2e tests with
the correct install paths.
2020-06-17 13:24:56 -04:00
Lars Lehtonen 3ddfa66ca4
internal/modsdir: Fix Dropped Error (#24600)
* internal/modsdir: fix dropped error

* fix typo to unmarshalling

Co-authored-by: Daniel Dreier <danieldreier@users.noreply.github.com>
2020-06-04 10:26:26 -04:00
Martin Atkins 85af77386c internal/getproviders: PackageFilePathForPackage
This is the equivalent of UnpackedDirectoryPathForPackage when working
with the packed directory layout. It returns a path to a .zip file with
a name that would be detected by SearchLocalDirectory as a
PackageLocalArchive package.
2020-06-01 14:49:43 -07:00
Martin Atkins 9489672d54 internal/getproviders: Package hashing for local filesystem packages
We previously had this functionality available for cached packages in the
providercache package. This moves the main implementation of this over
to the getproviders package and then implements it also for PackageMeta,
allowing us to compute hashes in a consistent way across both of our
representations of a provider package.

The new methods on PackageMeta will only be effective for packages in the
local filesystem because we need direct access to the contents in order
to produce the hash. Hopefully in future the registry protocol will be
able to also provide hashes using this content-based (rather than
archive-based) algorithm and then we'll be able to make this work for
PackageMeta referring to a package obtained from a registry too, but
hashes for local packages only are still useful for some cases right now,
such as generating mirror directories in the "terraform providers mirror"
command.
2020-06-01 14:49:43 -07:00
Alisdair McDiarmid ef28671b34
Merge pull request #24932 from hashicorp/signing-language
Modify language for reporting signing state
2020-05-28 09:09:34 -04:00
Paddy 5127f1ef8b
command: Unmanaged providers
This adds supports for "unmanaged" providers, or providers with process
lifecycles not controlled by Terraform. These providers are assumed to
be started before Terraform is launched, and are assumed to shut
themselves down after Terraform has finished running.

To do this, we must update the go-plugin dependency to v1.3.0, which
added support for the "test mode" plugin serving that powers all this.

As a side-effect of not needing to manage the process lifecycle anymore,
Terraform also no longer needs to worry about the provider's binary, as
it won't be used for anything anymore. Because of this, we can disable
the init behavior that concerns itself with downloading that provider's
binary, checking its version, and otherwise managing the binary.

This is all managed on a per-provider basis, so managed providers that
Terraform downloads, starts, and stops can be used in the same commands
as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable
is added, and is a JSON encoding of the provider's address to the
information we need to connect to it.

This change enables two benefits: first, delve and other debuggers can
now be attached to provider server processes, and Terraform can connect.
This allows for attaching debuggers to provider processes, which before
was difficult to impossible. Second, it allows the SDK test framework to
host the provider in the same process as the test driver, while running
a production Terraform binary against the provider. This allows for Go's
built-in race detector and test coverage tooling to work as expected in
provider tests.

Unmanaged providers are expected to work in the exact same way as
managed providers, with one caveat: Terraform kills provider processes
and restarts them once per graph walk, meaning multiple times during
most Terraform CLI commands. As unmanaged providers can't be killed by
Terraform, and have no visibility into graph walks, unmanaged providers
are likely to have differences in how their global mutable state behaves
when compared to managed providers. Namely, unmanaged providers are
likely to retain global state when managed providers would have reset
it. Developers relying on global state should be aware of this.
2020-05-26 17:48:57 -07:00
Paul Tyng 22ef5cc99c Modify language for reporting signing state
Be more explicit about the signing status of fetched plugins and provide documentation about the different signing options.
2020-05-26 13:14:05 -04:00
Alisdair McDiarmid 62d826e066 command/init: Use full config for provider reqs
Relying on the early config for provider requirements was necessary in
Terraform 0.12, to allow the 0.12upgrade command to run after init
installs providers.

However in 0.13, the same restrictions do not apply, and the detection
of provider requirements has changed. As a result, the early config
loader gives incorrect provider requirements in some circumstances,
such as those in the new test in this commit.

Therefore we are changing the init command to use the requirements found
by the full configuration loader. This also means that we can remove the
internal initwd CheckCoreVersionRequirements function.
2020-05-25 16:50:12 -04:00
Kristin Laemmert eead4c49fe command/init: add e2e tests for provider not found messages 2020-05-20 11:04:11 -04:00
Kristin Laemmert 8d28d73de3 getproviders: add a registry-specific error and modify output when a
provider is not found.

Previously a user would see the following error even if terraform was
only searching the local filesystem:

"provider registry registry.terraform.io does not have a provider named
...."

This PR adds a registry-specific error type and modifies the MultiSource
installer to check for registry errors. It will return the
registry-specific error message if there is one, but if not the error
message will list all locations searched.
2020-05-20 11:04:11 -04:00
Kristin Laemmert 0d620018fe
provider cache: log errors and validate dir exists (#24993)
* providercache: add logging for errors from getproviders.SearchLocalDirectory

providercache.fillMetaCache() was silently swallowing errors when
searching the cache directory. This commit logs the error without
changing the behavior otherwise.

* command/cliconfig: validate plugin cache dir exists

The plugin cache directory must exist for terraform to use it, so we
will add a check at the begining.
2020-05-19 15:32:36 -04:00
Kristin Laemmert d350818126
internal/getproviders: fix panic with invalid path parts (#24940)
* internal/getproviders: fix panic with invalid path parts

If the search path is missing a directory, the provider installer would
try to create an addrs.Provider with the wrong parts. For example if the
hostname was missing (as in the test case), it would call
addrs.NewProvider with (namespace, typename, version). This adds a
validation step for each part before calling addrs.NewProvider to avoid
the panic.
2020-05-13 13:16:09 -04:00
Alisdair McDiarmid 070c3018f8 internal/providercache: Remove unused retry events 2020-05-13 09:48:41 -04:00
Alisdair McDiarmid e27a36cafd internal/getproviders: Retry failed HTTP requests
This is a port of the retry/timeout logic added in #24260 and #24259,
using the same environment variables to configure the retry and timeout
settings.
2020-05-13 09:48:41 -04:00
Kristin Laemmert 60321b41e8
getproviders: move protocol compatibility functions into registry client (#24846)
* internal/registry source: return error if requested provider version protocols are not supported

* getproviders: move responsibility for protocol compatibility checks into the registry client

The original implementation had the providercache checking the provider
metadata for protocol compatibility, but this is only relevant for the
registry source so it made more sense to move the logic into
getproviders.

This also addresses an issue where we were pulling the metadata for
every provider version until we found one that was supported. I've
extended the registry client to unmarshal the protocols in
`ProviderVersions` so we can filter through that list, instead of
pulling each version's metadata.
2020-05-11 13:49:12 -04:00
Alisdair McDiarmid 62b0cbed12 internal: Fix LookupLegacyProvider
When looking up the namespace for a legacy provider source, we need to
use the /v1/providers/-/{name}/versions endpoint. For non-HashiCorp
providers, the /v1/providers/-/{name} endpoint returns a 404.

This commit updates the LegacyProviderDefaultNamespace method and the
mock registry servers accordingly.
2020-05-08 12:29:25 -04:00
Alisdair McDiarmid ae98bd12a7 command: Rework 0.13upgrade sub-command
This commit implements most of the intended functionality of the upgrade
command for rewriting configurations.

For a given module, it makes a list of all providers in use. Then it
attempts to detect the source address for providers without an explicit
source.

Once this step is complete, the tool rewrites the relevant configuration
files. This results in a single "required_providers" block for the
module, with a source for each provider.

Any providers for which the source cannot be detected (for example,
unofficial providers) will need a source to be defined by the user. The
tool writes an explanatory comment to the configuration to help with
this.
2020-05-07 11:38:55 -04:00
Kristin Laemmert cca0526705
providercache: actually break out of the loop when a matching version is found (#24823) 2020-05-01 08:49:47 -04:00
Kristin Laemmert ce03f1255f
internal/providercache: fix error message for protocol mismatch (#24818)
There was a bug in the installer trying to pass a nil error.
2020-04-30 11:12:04 -04:00
Kristin Laemmert 320fcf4942
internal/getproviders: apply case normalizations in ParseMultiSourceMatchingPatterns (#24753)
* internal/getproviders: apply case normalizations in ParseMultiSourceMatchingPatterns

This is a very minor refactor which takes advantage of addrs.ParseProviderPart case normalization to normalize non-wildcard sources.
2020-04-23 14:50:47 -04:00
Martin Atkins 1ce3c60693
command/cliconfig: Explicit provider installation method configuration
This set of commits allows explicit configuration of provider installation methods
in the CLI config, overriding the implicit method selections.
2020-04-23 10:58:00 -07:00
Martin Atkins 8b75d1498f command/cliconfig: Use existing HTTP mirror source rather than new stub
An earlier commit added a redundant stub for a new network mirror source
that was already previously stubbed as HTTPMirrorSource.

This commit removes the unnecessary extra stub and changes the CLI config
handling to use it instead. Along the way this also switches to using a
full base URL rather than just a hostname for the mirror, because using
the usual "Terraform-native service discovery" protocol here doesn't isn't
as useful as in the places we normally use it (the mirror mechanism is
already serving as an indirection over the registry protocol) and using
a direct base URL will make it easier to deploy an HTTP mirror under
a path prefix on an existing static file server.
2020-04-23 10:52:01 -07:00
Kristin Laemmert 84e9d86c25
Update installer_test.go
update now-exported function (fix bad PR)
2020-04-23 08:33:28 -04:00
Kristin Laemmert 21b9da5a02
internal/providercache: verify that the provider protocol version is compatible (#24737)
* internal/providercache: verify that the provider protocol version is
compatible

The public registry includes a list of supported provider protocol
versions for each provider version. This change adds verification of
support and adds a specific error message pointing users to the closest
matching version.
2020-04-23 08:21:56 -04:00
Alisdair McDiarmid 54abb87fb4 Fix broken test due to function rename 2020-04-22 10:52:49 -04:00
Martin Atkins c5bd783eba internal/getproviders: Stub NetworkMirrorSource
This is a placeholder for later implementation of a mirror source that
talks to a particular remote HTTP server and expects it to implement the
provider mirror protocol.
2020-04-21 16:27:48 -07:00
Kristin Laemmert a43f141f9e
tools/terraform-bundle: refactor to use new provider installer and provider directory layouts (#24629)
* tools/terraform-bundle: refactor to use new provider installer and
provider directory layouts

terraform-bundle now supports a "source" attribute for providers,
uses the new provider installer, and the archive it creates preserves
the new (required) directory hierarchy for providers, under a "plugins"
directory.

This is a breaking change in many ways: source is required for any
non-HashiCorp provider, locally-installed providers must be given a
source (can be arbitrary, see docs) and placed in the expected directory
hierarchy, and the unzipped archive is no longer flat; there is a new
"plugins" directory created with providers in the new directory layout.

This PR also extends the existing test to check the contents of the zip
file.

TODO: Re-enable e2e tests (currently suppressed with a t.Skip)
This commit includes an update to our travis configuration, so the terraform-bundle e2e tests run. It also turns off the e2e tests, which will fail until we have a terraform 0.13.* release under releases.hashicorp.com. We decided it was better to merge this now instead of waiting when we started seeing issues opened from users who built terraform-bundle from 0.13 and found it didn't work with 0.12 - better that they get an immediate error message from the binary directing them to build from the appropriate release.
2020-04-21 17:09:29 -04:00
Alisdair McDiarmid a5b3d497cc internal: Verify provider signatures on install
Providers installed from the registry are accompanied by a list of
checksums (the "SHA256SUMS" file), which is cryptographically signed to
allow package authentication. The process of verifying this has multiple
steps:

- First we must verify that the SHA256 hash of the package archive
  matches the expected hash. This could be done for local installations
  too, in the future.
- Next we ensure that the expected hash returned as part of the registry
  API response matches an entry in the checksum list.
- Finally we verify the cryptographic signature of the checksum list,
  using the public keys provided by the registry.

Each of these steps is implemented as a separate PackageAuthentication
type. The local archive installation mechanism uses only the archive
checksum authenticator, and the HTTP installation uses all three in the
order given.

The package authentication system now also returns a result value, which
is used by command/init to display the result of the authentication
process.

There are three tiers of signature, each of which is presented
differently to the user:

- Signatures from the embedded HashiCorp public key indicate that the
  provider is officially supported by HashiCorp;
- If the signing key is not from HashiCorp, it may have an associated
  trust signature, which indicates that the provider is from one of
  HashiCorp's trusted partners;
- Otherwise, if the signature is valid, this is a community provider.
2020-04-17 13:57:19 -04:00
Kristin Laemmert f09ae6f862
provider source tests: added test suite to exercise hyphenated providers (#24685) 2020-04-16 15:54:33 -04:00
Martin Atkins 6ec5e0fc88 internal/getproviders: Tests for MemoizeSource
Due to other pressures at the time this was implemented, it was tested
only indirectly through integration tests in other packages. This now
introduces tests for the two main entry points on MemoizeSource.
2020-04-07 08:55:52 -07:00
Martin Atkins c6535a0fa4 internal/getproviders: Tests for MultiSource
Due to other pressures at the time this was implemented, it was tested
only indirectly through integration tests in other packages.

This now introduces tests for the two main entry points on the
MultiSource, along with its provider-address pattern matching logic.

This does not yet include thorough tests for
ParseMultiSourceMatchingPatterns, because that function still needs some
adjustments to do the same case folding as for normal provider address
parsing, which will follow in a latter commit along with suitable tests.

With that said, the tests added here do _indirectly_ test the happy path
of ParseMultiSourceMatchingPatterns, so we have some incomplete testing
of that function in the meantime.
2020-04-07 08:55:13 -07:00
Martin Atkins 0ad4c1be2f internal/getproviders: Tidy up some confusion about package hashes
Earlier on in the stubbing of this package we realized that it wasn't
going to be possible to populate the authentication-related bits for all
packages because the relevant metadata just isn't available for packages
that are already local.

However, we just moved ahead with that awkward design at the time because
we needed to get other work done, and so we've been mostly producing
PackageMeta values with all-zeros hashes and just ignoring them entirely
as a temporary workaround.

This is a first step towards what is hopefully a more intuitive model:
authentication is an optional thing in a PackageMeta that is currently
populated only for packages coming from a registry.

So far this still just models checking a SHA256 hash, which is not a
sufficient set of checks for a real release but hopefully the "real"
implementation is a natural iteration of this starting point, and if not
then at least this interim step is a bit more honest about the fact that
Authentication will not be populated on every PackageMeta.
2020-04-06 16:31:23 -07:00
Alisdair McDiarmid 0f5a38b384 internal: Fix init provider lockfile test
The fake installable package meta used a ZIP archive which gave
different checksums between macOS and Linux targets. This commit removes
the target from the contents of this archive, and updates the golden
hash value in the test to match. This test should now pass on both
platforms.
2020-04-06 09:24:23 -07:00
Martin Atkins 958ea4f7d1 internal/providercache: Handle built-in providers
Built-in providers are special providers that are distributed as part of
Terraform CLI itself, rather than being installed separately. They always
live in the terraform.io/builtin/... namespace so it's easier to see that
they are special, and currently there is only one built-in provider named
"terraform".

Previous commits established the addressing scheme for built-in providers.
This commit makes the installer aware of them to the extent that it knows
not to try to install them the usual way and it's able to report an error
if the user requests a built-in provider that doesn't exist or tries to
impose a particular version constraint for a built-in provider.

For the moment the tests for this are the ones in the "command" package
because that's where the existing testing infrastructure for this
functionality lives. A later commit should add some more focused unit
tests here in the internal/providercache package, too.
2020-04-06 09:24:23 -07:00
Martin Atkins 7caf0b9246 addrs: ImpliedProviderForUnqualifiedType function
This encapsulates the logic for selecting an implied FQN for an
unqualified type name, which could either come from a local name used in
a module without specifying an explicit source for it or from the prefix
of a resource type on a resource that doesn't explicitly set "provider".

This replaces the previous behavior of just directly calling
NewDefaultProvider everywhere so that we can use a different implication
for the local name "terraform", to refer to the built-in terraform
provider rather than the stale one that's on registry.terraform.io for
compatibility with other Terraform versions.
2020-04-06 09:24:23 -07:00
Martin Atkins f35ebe2d65 internal/providercache: Fix incorrect logic in Installer.SetGlobalCacheDir
Due to some incomplete rework of this function in an earlier commit, the
safety check for using the same directory as both the target and the
cache was inverted and was raising an error _unless_ they matched, rather
than _if_ they matched.

This change is verified by the e2etest TestInitProviders_pluginCache,
which is also updated to use the new-style cache directory layout as part
of this commit.
2020-04-06 09:24:23 -07:00
Martin Atkins ff55e1a1cd internal/providercache: installFromLocalDir
We previously skipped this one because it wasn't strictly necessary for
replicating the old "terraform init" behavior, but we do need it to work
so that things like the -plugin-dir option can behave correctly.

Linking packages from other cache directories and installing from unpacked
directories are fundamentally the same operation because a cache directory
is really just a collection of unpacked packages, so here we refactor
the LinkFromOtherCache functionality to actually be in
installFromLocalDir, and LinkFromOtherCache becomes a wrapper for
the installFromLocalDir function that just calculates the source and
target directories automatically and invalidates the metaCache.
2020-04-06 09:24:23 -07:00
Martin Atkins f7072a8f29 internal/getproviders: Implement MultiSource
We previously had only a stub implementation for a totally-empty
MultiSource. Here we have an initial implementation of the full
functionality, which we'll need to support "terraform init -plugin-dir=..."
in a subsequent commit.
2020-04-06 09:24:23 -07:00
Martin Atkins 2ff4582be2 internal/providercache: Fix positions on selections file/dir
On Unix-derived systems a directory must be marked as "executable" in
order to be accessible, so our previous mode of 0660 here was unsufficient
and would cause a failure if it happened to be the installer that was
creating the plugins directory for the first time here.

Now we'll make it executable and readable for all but only writable by
the same user/group. For consistency, we also make the selections file
itself readable by everyone. In both cases, the umask we are run with may
further constrain these modes.
2020-04-06 09:24:23 -07:00
Martin Atkins 531de52dff internal/getproviders: MockSource and mock package metadata
These are some helpers to support unit testing in other packages, allowing
callers to exercise provider installation mechanisms without hitting any
real upstream source or having to prepare local package directories.

MockSource is a Source implementation that just scans over a provided
static list of packages and returns whatever matches.

FakePackageMeta is a shorthand for concisely constructing a
realistic-looking but uninstallable PackageMeta, probably for use with
MockSource.

FakeInstallablePackageMeta is similar to FakePackageMeta but also goes to
the trouble of creating a real temporary archive on local disk so that
the resulting package meta is pointing to something real on disk. This
makes the result more useful to the caller, but in return they get the
responsibility to clean up the temporary file once the test is over.

Nothing is using these yet.
2020-04-06 09:24:23 -07:00
Martin Atkins ae080481c0 internal/providercache: Installer records its selections in a file
Just as with the old installer mechanism, our goal is that explicit
provider installation is the only way that new provider versions can be
selected.

To achieve that, we conclude each call to EnsureProviderVersions by
writing a selections lock file into the target directory. A later caller
can then recall the selections from that file by calling SelectedPackages,
which both ensures that it selects the same set of versions and also
verifies that the checksums recorded by the installer still match.

This new selections.json file has a different layout than our old
plugins.json lock file. Not only does it use a different hashing algorithm
than before, we also record explicitly which version of each provider
was selected. In the old model, we'd repeat normal discovery when
reloading the lock file and then fail with a confusing error message if
discovery happened to select a different version, but now we'll be able
to distinguish between a package that's gone missing since installation
(which could previously have then selected a different available version)
from a package that has been modified.
2020-04-06 09:24:23 -07:00
Martin Atkins f6a7a4868b internal/providercache: Hashing of contents of cached packages
For the old-style provider cache directory model we hashed the individual
executable file for each provider. That's no longer appropriate because
we're giving each provider package a whole directory to itself where it
can potentially have many files.

This therefore introduces a new directory-oriented hashing algorithm, and
it's just using the Go Modules directory hashing algorithm directly
because that's already had its cross-platform quirks and other wrinkles
addressed during the Go Modules release process, and is now used
prolifically enough in Go codebases that breaking changes to the upstream
algorithm would be very expensive to the Go ecosystem.

This is also a bit of forward planning, anticipating that later we'll use
hashes in a top-level lock file intended to be checked in to user version
control, and then use those hashes also to verify packages _during_
installation, where we'd need to be able to hash unpacked zip files. The
Go Modules hashing algorithm is already implemented to consistently hash
both a zip file and an unpacked version of that zip file.
2020-04-06 09:24:23 -07:00
Martin Atkins 079b4cf7be internal/providercache: Clear the metadata cache during package install
This was previously happening during linking from another cache, but not
when installing an entirely new provider.
2020-04-06 09:24:23 -07:00
Martin Atkins 4061cbed38 internal/getproviders: A new shared model for provider requirements
We've been using the models from the "moduledeps" package to represent our
provider dependencies everywhere since the idea of provider dependencies
was introduced in Terraform 0.10, but that model is not convenient to use
for any use-case other than the "terraform providers" command that needs
individual-module-level detail.

To make things easier for new codepaths working with the new-style
provider installer, here we introduce a new model type
getproviders.Requirements which is based on the type the new installer was
already taking as its input. We have new methods in the states, configs,
and earlyconfig packages to produce values of this type, and a helper
to merge Requirements together so we can combine config-derived and
state-derived requirements together during installation.

The advantage of this new model over the moduledeps one is that all of
recursive module walking is done up front and we produce a simple, flat
structure that is more convenient for the main use-cases of selecting
providers for installation and then finding providers in the local cache
to use them for other operations.

This new model is _not_ suitable for implementing "terraform providers"
because it does not retain module-specific requirement details. Therefore
we will likely keep using moduledeps for "terraform providers" for now,
and then possibly at a later time consider specializing the moduledeps
logic for only what "terraform providers" needs, because it seems to be
the only use-case that needs to retain that level of detail.
2020-03-27 09:01:32 -07:00
Martin Atkins 537c1bedcf internal/providercache: LinkFromOtherCache removes target, not source
This was incorrectly removing the _source_ entry prior to creating the
symlink, therefore ending up with a dangling symlink and no source file.

This wasn't obvious before because the test case for LinkFromOtherCache
was also incorrectly named and therefore wasn't running. Fixing the name
of that test made this problem apparent.

The TestLinkFromOtherCache test case now ends up seeing the final resolved
directory rather than the symlink target, because of upstream changes
to the internal/getproviders filesystem scanning logic to handle symlinks
properly.
2020-03-25 13:50:00 -07:00
Martin Atkins eb25fe8b24 internal/getproviders: SearchLocalDirectory can handle symlinks
Previously this was failing to treat symlinks to directories as unpacked
layout, because our file info was only an Lstat result, not a full Stat.

Now we'll resolve the symlink first, allowing us to handle a symlink to
a directory. That's important because our internal/providercache behavior
is to symlink from one cache to another where possible.
2020-03-25 13:50:00 -07:00
Martin Atkins ad15459468 internal/{getproviders,providercache}: improved trace logging
There's a lot going on in these functions that can be hard to follow from
the outside, so we'll add some additional trace logging so that we can
more easily understand why things are behaving the way they are.
2020-03-25 13:50:00 -07:00
Martin Atkins 391ca0c991 internal/providercache: add windows test fixture
This was accidentally left out of an earlier commit due to our top-level
.gitignore file containing *.exe as an ignore pattern.
2020-03-25 13:50:00 -07:00
Martin Atkins 807267d1b5 internal/providercache: Installation from HTTP URLs and local archives
When a provider source produces an HTTP URL location we'll expect it to
resolve to a zip file, which we'll first download to a temporary
directory and then treat it like a local archive.

When a provider source produces a local archive path we'll expect it to
be a zip file and extract it into the target directory.

This does not yet include an implementation of installing from an
already-unpacked local directory. That will follow in a subsequent commit,
likely following a similar principle as in Dir.LinkFromOtherCache.
2020-03-25 11:29:48 -07:00
Martin Atkins 754b7ebb65 command: Expose providercache package objects for use elsewhere
These new functions allow command implementations to get hold of the
providercache objects and installation source object derived from the
current CLI configuration.
2020-03-25 11:29:48 -07:00
Martin Atkins e4d7a71d91 internal/getproviders: Exit early if MultiSource has no sources
The MultiSource isn't actually properly implemented yet, but this is a
minimal implementation just for the case where there are no underlying
sources at all, because we use an empty MultiSource as a placeholder
when a test in the "command" package fails to explicitly populate a
ProviderSource.
2020-03-25 11:29:48 -07:00
Martin Atkins 18dd0a396d internal/providercache: First pass of the actual install process
This is not tested yet, but it's a compilable strawman implementation of
the necessary sequence of events to coordinate all of the moving parts
of running a provider installation operation.

This will inevitably see more iteration in later commits as we complete
the surrounding parts and wire it up to be used by "terraform init". So
far, it's just dead code not called by any other package.
2020-03-25 11:29:48 -07:00
Martin Atkins 03155daf98 internal/providercache: Start to stub Installer type
The Installer type will encapsulate the logic for running an entire
provider installation request: given a set of providers to install, it
will determine a method to obtain each of them (or detect that they are
already installed) and then take the necessary actions.

So far it doesn't do anything, but this stubs out an interface by which
the caller can request ongoing notifications during an installation
operation.
2020-03-25 11:29:48 -07:00
Martin Atkins e4d14324e0 internal/providercache: Stub of Dir.InstallPackage method
This will eventually be responsible for actually retrieving a package from
a source and then installing it into the cache directory, but for the
moment it's just a stub to complete the proposed API, which I intend to
test in a subsequent commit by writing the full "Installer" API that will
encapsulate the full installation logic.
2020-03-25 11:29:48 -07:00
Martin Atkins 67ca067910 internal/providercache: Linking from one cache to another
When a system-wide shared plugin cache is configured, we'll want to make
use of entries already in the shared cache when populating a local
(configuration-specific) cache.

This new method LinkFromOtherCache encapsulates the work of placing a link
from one cache to another. If possible it will create a symlink, therefore
retaining a key advantage of configuring a shared plugin cache, but
otherwise we'll do a deep copy of the package directory from one cache
to the other.

Our old provider installer would always skip trying to create symlinks on
Windows because Go standard library support for os.Symlink on Windows
was inconsistent in older versions. However, os.Symlink can now create
symlinks using a new API introduced in a Windows 10 update and cleanly
fail if symlink creation is impossible, so it's safe for us to just
try to create the symlink and react if that produces an error, just as we
used to do on non-Windows systems when possibly creating symlinks on
filesystems that cannot support them.
2020-03-25 11:29:48 -07:00
Martin Atkins 514184cc9d internal/getproviders: Functions to determine installation directories
The existing functionality in this package deals with finding packages
that are either available for installation or already installed. In order
to support installation we also need to determine the location where a
package should be installed.

This lives in the getproviders package because that way all of the logic
related to the filesystem layout for local provider directories lives
together here where they can be maintained together more easily in future.
2020-03-25 11:29:48 -07:00
Martin Atkins 072c6d9aed internal/copydir: Factor out our recursive directory copy for reuse
We've previously been copying this function around so it could remain
unexported while being used in various packages. However, it's a
non-trivial function with lots of specific assumptions built into it, so
here we'll put it somewhere that other packages can depend on it _and_
document the assumptions it seems to be making for future reference.

As a bonus, this now uses os.SameFile to detect when two paths point to
the same physical file, instead of the slightly buggy local implementation
we had before which only worked on Unix systems and did not correctly
handle when the paths were on different physical devices.

The copy of the function I extracted here is the one from internal/initwd,
so this commit also includes the removal of that unexported version and
updating the callers in that package to use at at this new location.
2020-03-25 11:29:48 -07:00
Martin Atkins d13001830b providercache: A package to encapsulate management of provider cache dirs
Historically our logic to handle discovering and installing providers has
been spread across several different packages. This package is intended
to become the home of all logic related to what is now called "provider
cache directories", which means directories on local disk where Terraform
caches providers in a form that is ready to run.

That includes both logic related to interrogating items already in a cache
(included in this commit) and logic related to inserting new items into
the cache from upstream provider sources (to follow in later commits).

These new codepaths are focused on providers and do not include other
plugin types (provisioners and credentials helpers), because providers are
the only plugin type that is represented by a heirarchical, decentralized
namespace and the only plugin type that has an auto-installation protocol
defined. The existing codepaths will remain to support the handling of
the other plugin types that require manual installation and that use only
a flat, locally-defined namespace.
2020-03-25 11:29:48 -07:00
Martin Atkins 283b4d4cad internal/getproviders: Make local directory search a public API
Previously this was available by instantiating a throwaway
FilesystemMirrorSource, but that's pretty counter-intuitive for callers
that just want to do a one-off scan without retaining any ongoing state.

Now we expose SearchLocalDirectory as an exported function, and the
FilesystemMirrorSource then uses it as part of its implementation too.
Callers that just want to know what's available in a directory can call
SearchLocalDirectory directly.
2020-03-25 11:29:48 -07:00
Paddy e6592dc710
Add support for provider metadata to modules. (#22583)
Implement a new provider_meta block in the terraform block of modules, allowing provider-keyed metadata to be communicated from HCL to provider binaries.

Bundled in this change for minimal protocol version bumping is the addition of markdown support for attribute descriptions and the ability to indicate when an attribute is deprecated, so this information can be shown in the schema dump.

Co-authored-by: Paul Tyng <paul@paultyng.net>
2020-03-05 16:53:24 -08:00
Kim Ngo 8f5159ad54
Merge pull request #24260 from findkim/registry-retry
registry: retryable discovery requests
2020-03-05 10:00:08 -06:00
Martin Atkins 43c1ec69e2 build: Use Go 1.14
This implies some notable changes that will have a visible impact to
end-users of official Terraform releases:

- Terraform is no longer compatible with MacOS 10.10 Yosemite, and
  requires at least 10.11 El Capitan. (Relatedly, Go 1.14 is planned to be
  the last release to support El Capitan, so while that remains supported
  for now, it's notable that Terraform 0.13 is likely to be the last major
  release of Terraform supporting it, with 0.14 likely to further require
  MacOS 10.12 Sierra.)

- Terraform is no longer compatible with FreeBSD 10.x, which has reached
  end-of-life. Terraform now requires FreeBSD 11.2 or later.

- Terraform now supports TLS 1.3 when it makes connections to remote
  services such as backends and module registries. Although TLS 1.3 is
  backward-compatible in principle, some legacy systems reportedly work
  incorrectly when attempting to negotiate it. (This change does not
  affect outgoing requests made by provider plugins, though they will see
  a similar change in behavior once built with Go 1.13 or later.)

- Ed25519 certificates are now supported for TLS 1.2 and 1.3 connections.

- On UNIX systems where "use-vc" is set in resolv.conf, TCP will now be
  used for DNS resolution. This is unlikely to cause issues in practice
  because a system set up in this way can presumably already reach its
  nameservers over TCP (or else other applications would misbehave), but
  could potentially lead to lookup failures in unusual situations where a
  system only runs Terraform, has historically had "use-vc" in its
  configuration, but yet is blocked from reaching its configured
  nameservers over TCP.

- Some parts of Terraform now support Unicode 12.0 when working with
  strings. However, notably the Terraform Language itself continues to
  use the text segmentation tables from Unicode 9.0, which means it lacks
  up-to-date support for recognizing modern emoji combining forms as
  single characters. (We may wish to upgrade the text segmentation tables
  to Unicode 12.0 tables in a later commit, to restore consistency.)

This also includes some changes to the contents of "vendor", and
particularly to the format of vendor/modules.txt, per the changes to
vendoring in the Go 1.14 toolchain. This new syntax is activated by the
specification of "go 1.14" in the go.mod file.

Finally, the exact format of error messages from the net/http library has
changed since Go 1.12, and so a couple of our tests needed updates to
their expected error messages to match that.
2020-03-04 13:26:50 -08:00
Martin Atkins 8eff19e48f internal/getproviders: Initial implementation of FilesystemMirrorSource
This is a basic implementation of FilesystemMirrorSource for now aimed
only at the specific use-case of scanning the cache of provider plugins
Terraform will keep under the ".terraform" directory, as part of our
interim provider installer implementation for Terraform 0.13.

The full functionality of this will grow out in later work when we
implement explicit local filesystem mirrors, but for now the goal is to
use this just to inspect the work done by the automatic installer once
we switch it to the new provider-FQN-aware directory structure.

The various FIXME comments in this are justified by the limited intended
scope of this initial implementation, and they should be resolved by
later work to use FilesystemMirrorSource explicitly for user-specified
provider package mirrors.
2020-02-25 10:30:03 -05:00
Martin Atkins c073db09ea internal/getproviders: Sorting and filtering for lists of PackageMeta
These are utility functions to ease processing of lists of PackageMeta
elsewhere, once we have functionality that works with multiple packages
at once. The local filesystem mirror source will be the first example of
this, so these methods are motivated mainly by its needs.
2020-02-25 10:30:03 -05:00
Martin Atkins d82a36b6f5 internal/getproviders: ParsePlatform method
This is just to have a centralized set of logic for converting from a
platform string (like "linux_amd64") to a Platform object, so we can do
normalization and validation consistently.
2020-02-25 10:30:03 -05:00
Martin Atkins 3078d21c5f internal/getproviders: Include Provider and Version in PackageMeta
Although we tend to return these in contexts where at least one of these
values is implied, being explicit means that PackageMeta values are
self-contained and less reliant on such external context.
2020-02-25 10:30:03 -05:00
Pam Selle c249943360
Module Expansion: Part 2 (#24154)
* WIP: dynamic expand

* WIP: add variable and local support

* WIP: outputs

* WIP: Add referencer

* String representation, fixing tests it impacts

* Fixes TestContext2Apply_outputOrphanModule

* Fix TestContext2Apply_plannedDestroyInterpolatedCount

* Update DestroyOutputTransformer and associated types to reflect PlannableOutputs

* Remove comment about locals

* Remove module count enablement

* Removes allowing count for modules, and reverts the test,
while adding a Skip()'d test that works when you re-enable
the config

* update TargetDownstream signature to match master

* remove unnecessary method

Co-authored-by: James Bardin <j.bardin@gmail.com>
2020-02-24 17:42:32 -05:00
Martin Atkins 86f0b5191c addrs: Stronger validation and normalization of provider namespace/type
The provider FQN is becoming our primary identifier for a provider, so
it's important that we are clear about the equality rules for these
addresses and what characters are valid within them.

We previously had a basic regex permitting ASCII letters and digits for
validation and no normalization at all. We need to do at least case
folding and UTF-8 normalization because these names will appear in file
and directory names in case-insensitive filesystems and in repository
names such as on GitHub.

Since we're already using DNS-style normalization and validation rules
for the hostname part, rather than defining an entirely new set of rules
here we'll just treat the provider namespace and type as if they were
single labels in a DNS name. Aside from some internal consistency, that
also works out nicely because systems like GitHub use organization and
repository names as part of hostnames (e.g. with GitHub Pages) and so
tend to apply comparable constraints themselves.

This introduces the possibility of names containing letters from alphabets
other than the latin alphabet, and for latin letters with diacritics.
That's consistent with our introduction of similar support for identifiers
in the language in Terraform 0.12, and is intended to be more friendly to
Terraform users throughout the world that might prefer to name their
products using a different alphabet. This is also a further justification
for using the DNS normalization rules: modern companies tend to choose
product names that make good domain names, and now such names will be
usable as Terraform provider names too.
2020-02-18 15:42:09 -08:00
findkim d0e13320d5 registry: configurable retry client 2020-02-14 13:29:00 -06:00
Kristin Laemmert 7eed30595a
moduledeps: replace ProviderInstance with addrs.Provider (#24017)
* addrs: add ParseProviderSourceString function to parse fqns from
tfconfig-inspect
* moduledeps: use addrs.Provider instead of ProviderInstance
2020-02-05 09:27:32 -05:00
Pam Selle 9f374eb1fc
Merge pull request #23865 from loslocostacos/fix-win-tfe-pathing
Fix compatibility issue between Windows CLI and remote plan and apply
2020-01-22 12:04:41 -05:00
Martin Atkins 4d7122a0dd internal/getproviders: LookupLegacyProvider
This is a temporary helper so that we can potentially ship the new
provider installer without making a breaking change by relying on the
old default namespace lookup API on the default registry to find a proper
FQN for a legacy provider provider address during installation.

If it's given a non-legacy provider address then it just returns the given
address verbatim, so any codepath using it will also correctly handle
explicit full provider addresses. This also means it will automatically
self-disable once we stop using addrs.NewLegacyProvider in the config
loader, because there will therefore no longer be any legacy provider
addresses in the config to resolve. (They'll be "default" provider
addresses instead, assumed to be under registry.terraform.io/hashicorp/* )

It's not decided yet whether we will actually introduce the new provider
in a minor release, but even if we don't this API function will likely be
useful for a hypothetical automatic upgrade tool to introduce explicit
full provider addresses into existing modules that currently rely on
the equivalent to this lookup in the current provider installer.

This is dead code for now, but my intent is that it would either be called
as part of new provider installation to produce an address suitable to
pass to Source.AvailableVersions, or it would be called from the
aforementioned hypothetical upgrade tool.

Whatever happens, these functions can be removed no later than one whole
major release after the new provider installer is introduced, when
everyone's had the opportunity to update their legacy unqualified
addresses.
2020-01-22 09:02:22 -08:00
Steve Burns 26305a83a3 Remove unecessary conditions for module manifest read and write functions 2020-01-17 12:14:17 -07:00
Steve Burns 43b855743b Uniformity for module manifest dir read/write 2020-01-14 21:13:36 -07:00
Steven Burns 98e612ee50 Add condition for conversion of module path in manifest 2020-01-14 21:06:42 -07:00
Steve Burns 8bc0ef8fdc Attempt to change the output of module manifest 2020-01-14 20:42:00 -07:00
Martin Atkins 2aac8cf812 internal/getproviders: Distinguished packed vs. unpacked local packages
Our local filesystem mirror mechanism will allow provider packages to be
given either in packed form as an archive directly downloaded to disk or
in an unpacked form where the archive is extracted.

Distinguishing these two cases in the concrete Location types will allow
callers to reliably select the mode chosen by the selected installation
source and handle it appropriately, rather than resorting to out-of-band
heuristics like checking whether the object is a directory or a file.
2020-01-10 09:41:27 -08:00
Martin Atkins 92a58ec438 internal/getproviders: Stub out the two mirror sources
In a future commit, these implementations of Source will allow finding
and retrieving provider packages via local mirrors, both in the local
filesystem and over the network using an HTTP-based protocol.
2020-01-10 09:41:27 -08:00
Martin Atkins 773e38b49c internal/getproviders: Stub of MultiSource
This is an API stub for a component that will be added in a future commit
to support considering a number of different installation sources for each
provider. These will eventually be configurable in the CLI configuration,
allowing users to e.g. mirror certain providers within their own
infrastructure while still being able to go upstream for those that aren't
mirrored, or permit locally-mirrored providers only, etc.
2020-01-10 09:41:27 -08:00
Martin Atkins d695956061 internal/getproviders: MemoizeSource for local caching of source responses
Some sources make network requests that are likely to be slow, so this
wrapper type can cache previous responses for its lifetime in order to
speed up repeated requests for the same information.
2020-01-10 09:41:27 -08:00
Martin Atkins a77bc59c44 internal/getproviders: Package URL should always be absolute
Registries backed by static files are likely to use relative paths to
their archives for simplicity's sake, but we'll normalize them to be
absolute before returning because the caller wouldn't otherwise know what
to resolve the URLs relative to.
2020-01-10 09:41:27 -08:00
Martin Atkins c8f7223adb internal/getproviders: Source interface for generalization
We intend to support installation both directly from origin registries and
from mirrors in the local filesystem or over the network. This Source
interface will serve as our abstraction over those three options, allowing
calling code to treat them all the same.
2020-01-10 09:41:27 -08:00
Martin Atkins c76260e957 internal/getproviders: Query a provider registry
Our existing provider installer was originally built to work with
releases.hashicorp.com and later retrofitted to talk to the official
Terraform Registry. It also assumes a flat namespace of providers.

We're starting a new one here, copying and adapting code from the old one
as necessary, so that we can build out this new API while retaining all
of the existing functionality and then cut over to this new implementation
in a later step.

Here we're creating a foundational component for the new installer, which
is a mechanism to query for the available versions and download locations
of a particular provider.

Subsequent commits in this package will introduce other Source
implementations for installing from network and filesystem mirrors.
2020-01-10 09:41:27 -08:00
Kristin Laemmert 18dd1bb4d6
Mildwonkey/tfconfig upgrade (#23670)
* deps: bump terraform-config-inspect library
* configs: parse `version` in new required_providers block

With the latest version of `terraform-config-inspect`, the
required_providers attribute can now be a string or an object with
attributes "source" and "version". This change allows parsing the
version constraint from the new object while ignoring any given source attribute.
2020-01-10 11:54:53 -05:00
Kristin Laemmert 4045a6e5d0
initwd: cache registry responses for module versions and download URL (#23727)
* initwd: cache registry responses for module versions and download URL

Closes #23544
2020-01-07 15:03:23 -05:00
Martin Atkins 2b2ac1f6de configs: use local set of go-getter detectors
In an earlier change we switched to defining our own sets of detectors,
getters, etc for go-getter in order to insulate us from upstream changes
to those sets that might otherwise change the user-visible behavior of
Terraform's module installer.

However, we apparently neglected to actually refer to our local set of
detectors, and continued to refer to the upstream set. Here we catch up
with the latest detectors from upstream (taken from the version of
go-getter we currently have vendored) and start using that fixed set.

Currently we are maintaining these custom go-getter sets in two places
due to the configload vs. initwd distinction. That was already true for
goGetterGetters and goGetterDecompressors, and so I've preserved that for
now just to keep this change relatively simple; in later change it would
be nice to factor these "get with go getter" functions out into a shared
location which we can call from both configload and initwd.
2020-01-07 09:38:46 -08:00
Kristin Laemmert 68dfc3046d backend/remote-state: etcdv3, oss, and manta acc tests should fail
faster

The acceptance tests for etcdv3, oss and manta were not validating
required env variablea, chosing to assume that if one was running
acceptance tests they had already configured the credentials.

It was not always clear if this was a bug in the tests or the provider,
so I opted to make the tests fail faster when required attributes were
unset (or "").
2019-09-27 08:45:12 -04:00
Kristin Laemmert 69b6791ef1 internal/initwd: fix issues with tests and symlinks
filepath.EvalSymlinks is our friend! The code already does this, the
tests needed to be updated to do the same.
2019-09-25 16:27:17 -04:00
Radek Simko 3d94baecf6
Regenerate protobuf files under latest versions
Used protobuf 3.9.1 with protoc-gen-go 1.3.2
2019-09-05 14:36:15 +02:00
Radek Simko 5b9f2fafc8 Standardise directory name for test data 2019-06-30 10:16:15 +02:00
Pam Selle 59c5cc4788
Merge pull request #21254 from davewongillies/gcs
Add GCS source support for modules
2019-06-13 10:24:38 -04:00
James Bardin fa1f9be539 add tfplugin proto docs files
Add versioned tfplugin proto files to the docs directory, for easier
reference. The latest version starts as a symlink to the current
file used for generated the tfplugin package in ./internal/tfplugin5.

When changing the protocol version, the old file must be copied to
./docs/plugin-protocol/, and a new symlink created for the latest
version.
2019-06-07 15:47:46 -04:00
James Bardin 863bfdce06 set proto version 5.1
With the addition of the private field to ReadResource, we need to bump
the proto version to 5.1.
2019-06-07 15:11:11 -04:00
James Bardin e2b2f1bbbc add Private fields to ReadResource
Private data was previously created during Plan, and sent back to the
provider during Apply. This data also needs to be persisteded accross
Read calls, but rather than rely on core for that we can send the data
to the provider during Read to allow for more flexibilty.
2019-06-03 18:01:34 -04:00
Martin Atkins c5285021fa internal/initwd: Test that version is rejected for local modules
This was already working, but since that codepath is separate from the
go-getter install codepath it's helpful to have a separate test for it,
in addition to the existing one for go-getter modules.
2019-06-03 09:45:30 -07:00
David Gillies 8b45443b21
Add GCS source support for modules 2019-05-21 12:18:49 -07:00
Matt Morrison cbebb7cdf1 command/init: Don't panic if go-getter-fetched module has version constraint 2019-05-17 13:19:31 -07:00
Martin Atkins f5a7f45ffb internal/initwd: Fix module installation error
The "err" variable in the MaybeRelativePathErr condition was masking the
original err with nil in the "else" case of this branch, causing the
error message to be incomplete.

While here, also tweaked the wording to say "Could not download" rather
than "Error attempting to download", both to say the same thing in fewer
words and because the summary line above already starts with "Error:"
when we print out this message, so it looks weird to have both lines
start with the same word.
2019-05-14 07:38:06 -07:00
Kristin Laemmert 9327eedb04
internal/initwd: follow local module path symlink (#21063)
* internal/initwd: follow local module path symlink

Fixes #21060

While a previous commit fixed a problem when the local module directory
contained a symlink, it did not account for the possibility that the
entire directory was a symlink.
2019-04-24 08:19:27 -04:00
Martin Atkins 88e76fa9ef configs/configschema: Introduce the NestingGroup mode for blocks
In study of existing providers we've found a pattern we werent previously
accounting for of using a nested block type to represent a group of
arguments that relate to a particular feature that is always enabled but
where it improves configuration readability to group all of its settings
together in a nested block.

The existing NestingSingle was not a good fit for this because it is
designed under the assumption that the presence or absence of the block
has some significance in enabling or disabling the relevant feature, and
so for these always-active cases we'd generate a misleading plan where
the settings for the feature appear totally absent, rather than showing
the default values that will be selected.

NestingGroup is, therefore, a slight variation of NestingSingle where
presence vs. absence of the block is not distinguishable (it's never null)
and instead its contents are treated as unset when the block is absent.
This then in turn causes any default values associated with the nested
arguments to be honored and displayed in the plan whenever the block is
not explicitly configured.

The current SDK cannot activate this mode, but that's okay because its
"legacy type system" opt-out flag allows it to force a block to be
processed in this way anyway. We're adding this now so that we can
introduce the feature in a future SDK without causing a breaking change
to the protocol, since the set of possible block nesting modes is not
extensible.
2019-04-10 14:53:52 -07:00
Justin Downing 1e32ae243c grammatical updates to comments and docs (#20195) 2019-03-21 14:05:41 -07:00
Kristin Laemmert a15a4acf2f
configs/configupgrade: detect possible relative module sources (#20646)
* configs/configupgrade: detect possible relative module sources

If a module source appears to be a relative local path but does not have
a preceding ./, print a #TODO message for the user.

* internal/initwd: limit go-getter detectors to those supported by terraform
* internal/initwd: move isMaybeRelativeLocalPath check into getWithGoGetter

To avoid making two calls to getter.Detect, which potentially makes
non-trivial API calls, the "isMaybeRelativeLocalPath" check was moved to
a later step and a custom error type was added so user-friendly
diagnostics could be displayed in the event that a possible relative local
path was detected.
2019-03-13 11:17:14 -07:00
Kristin Laemmert bf04503f04
internal/initwd: identify possible relative module sources
Identify module sources that look like relative paths ("child" instead
of "./child", for example) and surface a helpful error.

Previously, such module sources would be passed to go-getter, which
would fail because it was expecting an absolute, or properly relative,
path. This commit moves the check for improper relative paths sooner so
a user-friendly error can be displayed.
2019-03-11 15:25:21 -07:00
Kristin Laemmert 0c5fd835ce
copyDir: detect if the module install path is a symlink to a directory (#20603)
configs/configload and internal/initwd both had a copyDir function that
would fail if the source directory contained a symlinked directory,
because the os.FileMode.IsDir() returns false for symlinks.

This PR adds a check for a symlink and copies that symlink in the
target directory. It handles symlinks for both files and directories
(with included tests).

Fixes #20539
2019-03-07 12:59:48 -08:00
Martin Atkins 31299e688d core: Allow legacy SDK to opt out of plan-time safety checks
Due to the inprecision of our shimming from the legacy SDK type system to
the new Terraform Core type system, the legacy SDK produces a number of
inconsistencies that produce only minor quirky behavior or broken
edge-cases. To retain compatibility with those existing weird behaviors,
the legacy SDK opts out of our safety checks.

The intent here is to allow existing providers to continue to do their
previous unsafe behaviors for now, accepting that this will allow certain
quirky bugs from previous releases to persist, and then gradually migrate
away from the legacy SDK and remove this opt-out on a per-resource basis
over time.

As with the apply-time safety check opt-out, this is reserved only for
the legacy SDK and must not be used in any new SDK implementations. We
still include any inconsistencies as warnings in the logs as an aid to
anyone debugging weird behavior, so that they can see situations where
blame may be misplaced in the user-visible error messages.
2019-02-11 17:26:49 -08:00
Martin Atkins 1530fe52f7 core: Legacy SDK providers opt out of our new apply result check
The shim layer for the legacy SDK type system is not precise enough to
guarantee it will produce identical results between plan and apply. In
particular, values that are null during plan will often become zero-valued
during apply.

To avoid breaking those existing providers while still allowing us to
introduce this check in the future, we'll introduce a rather-hacky new
flag that allows the legacy SDK to signal that it is the legacy SDK and
thus disable the check.

Once we start phasing out the legacy SDK in favor of one that natively
understands our new type system, we can stop setting this flag and thus
get the additional safety of this check without breaking any
previously-released providers.

No other SDK is permitted to set this flag, and we will remove it if we
ever introduce protocol version 6 in future, assuming that any provider
supporting that protocol will always produce consistent results.
2019-02-06 11:40:30 -08:00
Martin Atkins 86c02d5c35 command: "terraform init" can partially initialize for 0.12upgrade
There are a few constructs from 0.11 and prior that cause 0.12 parsing to
fail altogether, which previously created a chicken/egg problem because
we need to install the providers in order to run "terraform 0.12upgrade"
and thus fix the problem.

This changes "terraform init" to use the new "early configuration" loader
for module and provider installation. This is built on the more permissive
parser in the terraform-config-inspect package, and so it allows us to
read out the top-level blocks from the configuration while accepting
legacy HCL syntax.

In the long run this will let us do version compatibility detection before
attempting a "real" config load, giving us better error messages for any
future syntax additions, but in the short term the key thing is that it
allows us to install the dependencies even if the configuration isn't
fully valid.

Because backend init still requires full configuration, this introduces a
new mode of terraform init where it detects heuristically if it seems like
we need to do a configuration upgrade and does a partial init if so,
before finally directing the user to run "terraform 0.12upgrade" before
running any other commands.

The heuristic here is based on two assumptions:
- If the "early" loader finds no errors but the normal loader does, the
  configuration is likely to be valid for Terraform 0.11 but not 0.12.
- If there's already a version constraint in the configuration that
  excludes Terraform versions prior to v0.12 then the configuration is
  probably _already_ upgraded and so it's just a normal syntax error,
  even if the early loader didn't detect it.

Once the upgrade process is removed in 0.13.0 (users will be required to
go stepwise 0.11 -> 0.12 -> 0.13 to upgrade after that), some of this can
be simplified to remove that special mode, but the idea of doing the
dependency version checks against the liberal parser will remain valuable
to increase our chances of reporting version-based incompatibilities
rather than syntax errors as we add new features in future.
2019-01-14 11:33:21 -08:00
Martin Atkins 0c0a437bcb Move module install functionality over to internal/initwd 2019-01-14 11:33:21 -08:00
Martin Atkins e27e0ddc9e internal/init: Module installation functionality
This is an adaptation of the installation code from configs/configload,
now using the "earlyconfig" package instead of the "configs" package.

Module installation is an initialization-only process, with all other
commands assuming an already-initialized directory. Having it here can
therefore simplify the API of configs/configload, which can now focus only
on the problem of loading modules that have already been installed.

The old installer code in configs/configload is still in place for now
because the caller in "terraform init" isn't yet updated to use this.
2019-01-14 11:33:21 -08:00
Martin Atkins 53ee858851 internal/init: Package for helpers of the "terraform init" command
"terraform init" is quite a complex beast in relation to other commands
since almost everything it does is unique to it and thus not factored out
into other packages.

To get some of that sprawl out of the "command" package, this new internal
package will give us somewhere to put this init functionality that is
also useful for test code that needs to mimic the initialization behavior
against fixture directories.
2019-01-14 11:33:21 -08:00
Martin Atkins 0ce2add59f internal/modsdir: Factor out module manifest models to separate package 2019-01-14 11:33:21 -08:00
Martin Atkins 8ca1fcec51 internal/earlyconfig: Liberal config parsing for init
This is an alternative to the full config loader in the "configs" package
that is good for "early" use-cases like "terraform init", where we want
to find out what our dependencies are without getting tripped up on any
other errors that might be present.
2019-01-14 11:33:21 -08:00
Martin Atkins 4fe9632f09 plugin: Establish our current plugin protocol as version 5
The main significant change here is that the package name for the proto
definition is "tfplugin5", which is important because this name is part
of the wire protocol for references to types defined in our package.

Along with that, we also move the generated package into "internal" to
make it explicit that importing the generated Go package from elsewhere is
not the right approach for externally-implemented SDKs, which should
instead vendor the proto definition they are using and generate their
own stubs to ensure that the wire protocol is the only hard dependency
between Terraform Core and plugins.

After this is merged, any provider binaries built against our
helper/schema package will need to be rebuilt so that they use the new
"tfplugin5" package name instead of "proto".

In a future commit we will include more elaborate and organized
documentation on how an external codebase might make use of our RPC
interface definition to implement an SDK, but the primary concern here
is to ensure we have the right wire package name before release.
2018-11-19 09:56:41 -08:00