Commit Graph

98 Commits

Author SHA1 Message Date
James Bardin c07ce1cd4b move conversion functions into separate package
Managing which function need to be shared between the terraform plugin
and the helper plugin without creating cycles was becoming difficult.
Move all functions related to converting between terraform and proto
type into plugin/convert.
2018-10-16 19:11:09 -07:00
James Bardin 88bfbeb9c5 schema version changed type 2018-10-16 19:11:09 -07:00
James Bardin b403023841 serve the new version 5 grpc plugins
Use the new go-plugin version negotiation to server th appropriate
plugin type when the client requests protocol version 5.
2018-10-16 18:58:49 -07:00
James Bardin ff7d51a9b4 update tests to match new test func signature 2018-10-16 18:58:49 -07:00
Kristin Laemmert 5e7956f440 registry/response: rename Collection to ProviderVersionCollection for clarity 2018-10-16 18:58:49 -07:00
James Bardin c9d91dc64e remove old comment 2018-10-16 18:58:49 -07:00
Kristin Laemmert bdaf8290b4 registry/client: added a specific error if the registry client does not
support the requested service.
2018-10-16 18:58:49 -07:00
James Bardin c1d4a63fae GRPCProvider and GRPCProvisioner clients
Here we add the GRPCProvisioner and GRPCProvider which implement the
core provisioners.Interface and providers.Interface, and translate
betweeen the core types and the grpc protocol.
2018-10-16 18:58:49 -07:00
James Bardin 81bd3b09d6 generate mocks for some of the grpc interfaces
These will allow easier testing of the grpc endpoints in isolation.
Mocks are generated for ProviderClient, ProvisionerClient,
Provisioner_ProvisionResourceClient, and
Provisioner_ProvisionResourceServer using `go generate`
2018-10-16 18:58:49 -07:00
James Bardin e29053432e grpc proto fixes
Add missing connection field for ProvisionResource.

Add diagnostics to the ProvisionResource stream.
2018-10-16 18:58:49 -07:00
Kristin Laemmert ae015e0f75 plugin/discovery: add function to filter list of versions by those which
support the requested platform.

If the newest version of a provider which matches the version
constraints does not support the requested platform, filter the list of
available versions by platform support and try again.
2018-10-16 18:58:49 -07:00
Kristin Laemmert f83d5866fe plugin/discovery: removing deprecated functions 2018-10-16 18:58:49 -07:00
Kristin Laemmert ce5e66e178 plugin/discovery provider installer: download providers from the registry
Terraform will query the public registry at
https://registry.terraform.io for providers, instead of
https://releases.hashicorp.com.
2018-10-16 18:56:50 -07:00
James Bardin 8f51b2fba5 add grpc service definitions
This add the grpc protobuf service definitions for the new plugin
protocol, as well as the generated code.
2018-10-16 18:53:51 -07:00
Martin Atkins 4ed06a9227 terraform: HCL2-flavored module dependency resolver
For the moment this is just a lightly-adapted copy of
ModuleTreeDependencies named ConfigTreeDependencies, with the goal that
the two can live concurrently for the moment while not all callers are yet
updated and then we can drop ModuleTreeDependencies and its helper
functions altogether in a later commit.

This can then be used to make "terraform init" and "terraform providers"
work properly with the HCL2-powered configuration loader.
2018-10-16 18:44:26 -07:00
James Bardin 0a6938e90a update tests to match pre-release constraint rules 2018-10-16 18:24:47 -07:00
Sean Chittenden d749420a25
Fix drift caused from gofmt when running make dev and go 1.11.
A fresh checkout of `origin/master` does not build atm using the `dev`
target because `master` has not been formatted using `gofmt` from Go
1.11 (tis has been the case for a while if you've been running devel).

None of the drift in question is especially new but now that Go 1.11
has been released and gofmt's formatting guidelines have been updated,
it would be *really* nice if the code in `master` reflected the current
tooling in order to avoid having to fight this drift locally.

* 8mo: https://github.com/hashicorp/terraform/blame/master/backend/remote-state/s3/backend_test.go#L260-L261
* 6mo: https://github.com/hashicorp/terraform/blame/master/builtin/provisioners/chef/linux_provisioner_test.go#L124
* 1yr: 7cfeffe36b/command/init.go (L75-L76)
* 12d: 7cfeffe36b/command/meta_backend_test.go (L1437)
* 2yr: 7cfeffe36b/helper/schema/resource_timeout_test.go (L26)
* 4yr: 7cfeffe36b/helper/schema/schema_test.go (L2059)
* 1yr: 7cfeffe36b/plugin/discovery/get_test.go (L151)
2018-09-09 10:18:08 -07:00
Paul Tyng c868092d2d
Standardize http.Client creation with User-Agent 2018-02-28 12:09:50 -05:00
Masayuki Morita f440dba137 Standardize on log level "WARN" rather than "WARNING" 2018-01-16 18:05:26 -08:00
James Bardin 6cb4e14cf8 Use pooled http client for fetching providers
While the TLS handshakes are a fairly small overhead compared to
downloading the providers, clients in some situation are failing to
complete the TLS handshake in a timely manner. It's unclear if this is
because of heavily constrained clients are stalling while doing the
major crpto operations, or the edge servers are throttling repeated
requests from the same IPs.

This should allow reusing the open TLS connection to the release edge
servers during init.
2017-10-31 10:53:42 -04:00
Martin Atkins 183833affc core: terraform.ResourceProvider.GetSchema method
In order to parse provider, resource and data source configuration from
HCL2 config files, we need to know the relevant configuration schema.
This new method allows Terraform Core to request these from a provider.

This is a breaking change to this interface, so all of its implementers
in this package are updated too. This includes concrete implementations
of the new method in helper/schema that use the schema conversion code
added in an earlier commit to produce a configschema.Block automatically.

Plugins compiled against prior versions of helper/schema will not have
support for this method, and so calls to them will fail. Callers of
this new method will therefore need to sniff for support using the
SchemaAvailable field added to both ResourceType and DataSource.

This careful handling will need to persist until next time we increment
the plugin protocol version, at which point we can make the breaking
change of requiring this information to be available.
2017-10-17 07:23:41 -07:00
Martin Atkins 879899d434 plugin/discovery: plugin caching mechanism
For users that have metered or slow internet connections it is annoying
to have Terraform constantly re-downloading the same files when they
initialize many separate directories.

To help such users, here we add an opt-in mechanism to use a local
directory as a read-through cache. When enabled, any plugin download will
be skipped if a suitable file already exists in the cache directory. If
the desired plugin isn't in the cache, it will be downloaded into the
cache for use next time.

This mechanism also serves to reduce total disk usage by allowing
plugin files to be shared between many configurations, as long as the
target system isn't Windows and supports either hardlinks or symlinks.
2017-09-29 14:03:09 -07:00
Martin Atkins 12d6bc8c30 plugin/discovery: ignore non-files when discovering
If we encounter something that isn't a file -- for example, a dangling
symlink whose referent has been deleted -- we'll ignore it so that we
can either later produce a "no such plugin" error or auto-install a plugin
that will actually work.
2017-09-29 14:03:09 -07:00
James Bardin 3dd0f93aed Merge pull request #15816 from hashicorp/jbardin/plugin-logger
specify a logger for go-plugin
2017-08-15 14:55:11 -04:00
James Bardin 714df97a02 specify a logger for go-plugin
The go-plugin package now uses hclog. The default Logger has a level set
to Info, but all plugin output is relayed via Debug. Create a new named
logger for plugins with the level set to Trace so that all output comes
through.
2017-08-15 14:34:04 -04:00
Radek Simko 28b33c9299
plugin: Display version + source when initializing plugins 2017-08-15 19:29:13 +02:00
Radek Simko de3015cebf
Remove overly verbose DEBUG log 2017-08-03 16:11:45 +02:00
Martin Atkins ac250d2792 plugin/discovery: trim off .exe suffix when parsing filenames
On Windows systems the plugin binaries use a .exe suffix, which we were
misparsing as part of either the "v" or "x" parts of the filename.

This fixes #15578.
2017-07-18 10:00:23 -07:00
James Bardin 2b84e786e9 allow missing x-terraform-protocol-version
If the release site is missing the "x-terraform-protocol-version"
header, we should fetch the latest spec'ed release. Downloading the
wrong protocol version can't do any damage, and the version present is
more than likely compatible.
2017-07-14 15:52:41 -04:00
Martin Atkins 610fcb605e plugin/discovery: allow customizing the OS/arch for auto-install
Previously we forced only installing for the current GOOS and GOARCH. Now
we allow this to be optionally overridden, which allows building tools
that can, for example, populate a directory with plugins to run on a Linux
server while working on a Mac.
2017-07-05 10:02:05 -07:00
James Bardin beebd5cee3 improve discovery logging
Display whether we're looking for a provider or provisioner in the log
messages.
2017-06-27 11:09:54 -04:00
James Bardin e059eb80d1 add test for signature mismatch 2017-06-21 16:17:34 -04:00
James Bardin 2cca6f10fa return sig error when fetching a plugin
A signature error was being ignored while fetching a plugin
2017-06-21 12:57:53 -04:00
James Bardin c10f5caf05 Merge pull request #15345 from hashicorp/jbardin/plugin-sig
verify automatically downloaded plugins
2017-06-20 17:05:54 -04:00
Martin Atkins af2111f24e plugin/discovery: sentinel error values for Get errors
Some errors from Get are essentially user error, so we want to be able to
recognize them and give the user good feedback on how to proceed.

Although sentinel values are not an ideal solution to this, it's something
reasonably simple we can do to get this done without lots of refactoring.
2017-06-20 13:39:45 -07:00
James Bardin 13d835f996 test signature verification
use a published provider's checksum file and signature to check the
verification.
2017-06-20 13:14:30 -04:00
James Bardin 415d562d36 add signature verification
Fetch the SHA256SUMS file and verify it's signature before downloading
any plugins.

This embeds the hashicorp public key in the binary. If the publickey is
replaced, new releases will need to be cut anyway. A
--verify-plugin=false flag will be added to skip signature verification
in these cases.
2017-06-20 13:14:30 -04:00
James Bardin 2e57d284cb Merge pull request #15323 from hashicorp/jbardin/constraint-suggestion
remove "~> 0.0" constraint suggestions
2017-06-16 18:20:24 -04:00
James Bardin ec99b6910b remove "~> 0.0" constraint suggestions
Don't suggest constraints when the available plugin isn't versioned.

Add zero version const for comparisons.
2017-06-16 16:25:36 -04:00
James Bardin 6faace287d remove restriction on unversioned plugins
Discover unversioned plugins regarless of location.
2017-06-16 15:28:48 -04:00
James Bardin ba5b0dc609 mostly remove OS_ARCH knowledge from discovery
Since the command package also needs to know about the specific OS_ARCH
directories, remove the logic fom the discovery package.

This doesn't completely remove the knowledge of the path from discovery,
in order to maintain the current behavior of skipping legacy plugin
names within a new-style path.
2017-06-16 13:58:40 -04:00
James Bardin f723270e3e search the vendor directory for plugins
The default location for users to manually add plugins will be
./terraform.d/plugins/
2017-06-15 10:12:00 -04:00
Martin Atkins af4c82d151 plugin/discovery: Installer.PurgeUnused method
Given a map describing the chosen plugin for each provider name, this
method should purge any other plugins present in the local cache
directory.
2017-06-13 12:28:07 -07:00
Martin Atkins f753974bb3 plugin/discovery: Installer interface, and provider implementation
Previously we had a "getProvider" function type used to implement plugin
fetching. Here we replace that with an interface type, initially with
just a "Get" function.

For now this just simplifies the interface by allowing the target
directory and protocol version to be members of the struct rather than
passed as arguments.

A later change will extend this interface to also include a method to
purge unused plugins, so that upgrading frequently doesn't leave behind
a trail of unused executable files.
2017-06-13 12:28:07 -07:00
James Bardin 840978b2d5 udpate to plugin name convention
Names are case-insensitive, using lowercase by default.
2017-06-09 14:03:59 -07:00
James Bardin 1b201e67ea fix releases path and protocol header
Last minute change to the location of the binaries
2017-06-09 14:03:59 -07:00
James Bardin 283ae0c7c4 fix comment and add some logging in discovery 2017-06-09 14:03:59 -07:00
James Bardin ea3d9c86ba sort constraints for consistent output 2017-06-09 14:03:59 -07:00
Martin Atkins 4ba20f9c1c command init: show suggested constraints for unconstrained providers
When running "terraform init" with providers that are unconstrained, we
will now produce information to help the user update configuration to
constrain for the particular providers that were chosen, to prevent
inadvertently drifting onto a newer major release that might contain
breaking changes.

A ~> constraint is used here because pinning to a single specific version
is expected to create dependency hell when using child modules. By using
this constraint mode, which allows minor version upgrades, we avoid the
need for users to constantly adjust version constraints across many
modules, but make major version upgrades still be opt-in.

Any constraint at all in the configuration will prevent the display of
these suggestions, so users are free to use stronger or weaker constraints
if desired, ignoring the recommendation.
2017-06-09 14:03:59 -07:00
James Bardin dbbafbd43f clean up plugin fetching
We can filter the allowed versions and sort them before checking the
protocol version, that way we can just return the first one found
reducing network requests.
2017-06-09 14:03:59 -07:00