Commit Graph

147 Commits

Author SHA1 Message Date
James Bardin 88bfbeb9c5 schema version changed type 2018-10-16 19:11:09 -07:00
James Bardin b403023841 serve the new version 5 grpc plugins
Use the new go-plugin version negotiation to server th appropriate
plugin type when the client requests protocol version 5.
2018-10-16 18:58:49 -07:00
James Bardin ff7d51a9b4 update tests to match new test func signature 2018-10-16 18:58:49 -07:00
Kristin Laemmert 5e7956f440 registry/response: rename Collection to ProviderVersionCollection for clarity 2018-10-16 18:58:49 -07:00
James Bardin c9d91dc64e remove old comment 2018-10-16 18:58:49 -07:00
Kristin Laemmert bdaf8290b4 registry/client: added a specific error if the registry client does not
support the requested service.
2018-10-16 18:58:49 -07:00
James Bardin c1d4a63fae GRPCProvider and GRPCProvisioner clients
Here we add the GRPCProvisioner and GRPCProvider which implement the
core provisioners.Interface and providers.Interface, and translate
betweeen the core types and the grpc protocol.
2018-10-16 18:58:49 -07:00
James Bardin 81bd3b09d6 generate mocks for some of the grpc interfaces
These will allow easier testing of the grpc endpoints in isolation.
Mocks are generated for ProviderClient, ProvisionerClient,
Provisioner_ProvisionResourceClient, and
Provisioner_ProvisionResourceServer using `go generate`
2018-10-16 18:58:49 -07:00
James Bardin e29053432e grpc proto fixes
Add missing connection field for ProvisionResource.

Add diagnostics to the ProvisionResource stream.
2018-10-16 18:58:49 -07:00
Kristin Laemmert ae015e0f75 plugin/discovery: add function to filter list of versions by those which
support the requested platform.

If the newest version of a provider which matches the version
constraints does not support the requested platform, filter the list of
available versions by platform support and try again.
2018-10-16 18:58:49 -07:00
Kristin Laemmert f83d5866fe plugin/discovery: removing deprecated functions 2018-10-16 18:58:49 -07:00
Kristin Laemmert ce5e66e178 plugin/discovery provider installer: download providers from the registry
Terraform will query the public registry at
https://registry.terraform.io for providers, instead of
https://releases.hashicorp.com.
2018-10-16 18:56:50 -07:00
James Bardin 8f51b2fba5 add grpc service definitions
This add the grpc protobuf service definitions for the new plugin
protocol, as well as the generated code.
2018-10-16 18:53:51 -07:00
Martin Atkins 4ed06a9227 terraform: HCL2-flavored module dependency resolver
For the moment this is just a lightly-adapted copy of
ModuleTreeDependencies named ConfigTreeDependencies, with the goal that
the two can live concurrently for the moment while not all callers are yet
updated and then we can drop ModuleTreeDependencies and its helper
functions altogether in a later commit.

This can then be used to make "terraform init" and "terraform providers"
work properly with the HCL2-powered configuration loader.
2018-10-16 18:44:26 -07:00
James Bardin 0a6938e90a update tests to match pre-release constraint rules 2018-10-16 18:24:47 -07:00
Sean Chittenden d749420a25
Fix drift caused from gofmt when running make dev and go 1.11.
A fresh checkout of `origin/master` does not build atm using the `dev`
target because `master` has not been formatted using `gofmt` from Go
1.11 (tis has been the case for a while if you've been running devel).

None of the drift in question is especially new but now that Go 1.11
has been released and gofmt's formatting guidelines have been updated,
it would be *really* nice if the code in `master` reflected the current
tooling in order to avoid having to fight this drift locally.

* 8mo: https://github.com/hashicorp/terraform/blame/master/backend/remote-state/s3/backend_test.go#L260-L261
* 6mo: https://github.com/hashicorp/terraform/blame/master/builtin/provisioners/chef/linux_provisioner_test.go#L124
* 1yr: 7cfeffe36b/command/init.go (L75-L76)
* 12d: 7cfeffe36b/command/meta_backend_test.go (L1437)
* 2yr: 7cfeffe36b/helper/schema/resource_timeout_test.go (L26)
* 4yr: 7cfeffe36b/helper/schema/schema_test.go (L2059)
* 1yr: 7cfeffe36b/plugin/discovery/get_test.go (L151)
2018-09-09 10:18:08 -07:00
Paul Tyng c868092d2d
Standardize http.Client creation with User-Agent 2018-02-28 12:09:50 -05:00
Masayuki Morita f440dba137 Standardize on log level "WARN" rather than "WARNING" 2018-01-16 18:05:26 -08:00
James Bardin 6cb4e14cf8 Use pooled http client for fetching providers
While the TLS handshakes are a fairly small overhead compared to
downloading the providers, clients in some situation are failing to
complete the TLS handshake in a timely manner. It's unclear if this is
because of heavily constrained clients are stalling while doing the
major crpto operations, or the edge servers are throttling repeated
requests from the same IPs.

This should allow reusing the open TLS connection to the release edge
servers during init.
2017-10-31 10:53:42 -04:00
Martin Atkins 183833affc core: terraform.ResourceProvider.GetSchema method
In order to parse provider, resource and data source configuration from
HCL2 config files, we need to know the relevant configuration schema.
This new method allows Terraform Core to request these from a provider.

This is a breaking change to this interface, so all of its implementers
in this package are updated too. This includes concrete implementations
of the new method in helper/schema that use the schema conversion code
added in an earlier commit to produce a configschema.Block automatically.

Plugins compiled against prior versions of helper/schema will not have
support for this method, and so calls to them will fail. Callers of
this new method will therefore need to sniff for support using the
SchemaAvailable field added to both ResourceType and DataSource.

This careful handling will need to persist until next time we increment
the plugin protocol version, at which point we can make the breaking
change of requiring this information to be available.
2017-10-17 07:23:41 -07:00
Martin Atkins 879899d434 plugin/discovery: plugin caching mechanism
For users that have metered or slow internet connections it is annoying
to have Terraform constantly re-downloading the same files when they
initialize many separate directories.

To help such users, here we add an opt-in mechanism to use a local
directory as a read-through cache. When enabled, any plugin download will
be skipped if a suitable file already exists in the cache directory. If
the desired plugin isn't in the cache, it will be downloaded into the
cache for use next time.

This mechanism also serves to reduce total disk usage by allowing
plugin files to be shared between many configurations, as long as the
target system isn't Windows and supports either hardlinks or symlinks.
2017-09-29 14:03:09 -07:00
Martin Atkins 12d6bc8c30 plugin/discovery: ignore non-files when discovering
If we encounter something that isn't a file -- for example, a dangling
symlink whose referent has been deleted -- we'll ignore it so that we
can either later produce a "no such plugin" error or auto-install a plugin
that will actually work.
2017-09-29 14:03:09 -07:00
James Bardin 3dd0f93aed Merge pull request #15816 from hashicorp/jbardin/plugin-logger
specify a logger for go-plugin
2017-08-15 14:55:11 -04:00
James Bardin 714df97a02 specify a logger for go-plugin
The go-plugin package now uses hclog. The default Logger has a level set
to Info, but all plugin output is relayed via Debug. Create a new named
logger for plugins with the level set to Trace so that all output comes
through.
2017-08-15 14:34:04 -04:00
Radek Simko 28b33c9299
plugin: Display version + source when initializing plugins 2017-08-15 19:29:13 +02:00
Radek Simko de3015cebf
Remove overly verbose DEBUG log 2017-08-03 16:11:45 +02:00
Martin Atkins ac250d2792 plugin/discovery: trim off .exe suffix when parsing filenames
On Windows systems the plugin binaries use a .exe suffix, which we were
misparsing as part of either the "v" or "x" parts of the filename.

This fixes #15578.
2017-07-18 10:00:23 -07:00
James Bardin 2b84e786e9 allow missing x-terraform-protocol-version
If the release site is missing the "x-terraform-protocol-version"
header, we should fetch the latest spec'ed release. Downloading the
wrong protocol version can't do any damage, and the version present is
more than likely compatible.
2017-07-14 15:52:41 -04:00
Martin Atkins 610fcb605e plugin/discovery: allow customizing the OS/arch for auto-install
Previously we forced only installing for the current GOOS and GOARCH. Now
we allow this to be optionally overridden, which allows building tools
that can, for example, populate a directory with plugins to run on a Linux
server while working on a Mac.
2017-07-05 10:02:05 -07:00
James Bardin beebd5cee3 improve discovery logging
Display whether we're looking for a provider or provisioner in the log
messages.
2017-06-27 11:09:54 -04:00
James Bardin e059eb80d1 add test for signature mismatch 2017-06-21 16:17:34 -04:00
James Bardin 2cca6f10fa return sig error when fetching a plugin
A signature error was being ignored while fetching a plugin
2017-06-21 12:57:53 -04:00
James Bardin c10f5caf05 Merge pull request #15345 from hashicorp/jbardin/plugin-sig
verify automatically downloaded plugins
2017-06-20 17:05:54 -04:00
Martin Atkins af2111f24e plugin/discovery: sentinel error values for Get errors
Some errors from Get are essentially user error, so we want to be able to
recognize them and give the user good feedback on how to proceed.

Although sentinel values are not an ideal solution to this, it's something
reasonably simple we can do to get this done without lots of refactoring.
2017-06-20 13:39:45 -07:00
James Bardin 13d835f996 test signature verification
use a published provider's checksum file and signature to check the
verification.
2017-06-20 13:14:30 -04:00
James Bardin 415d562d36 add signature verification
Fetch the SHA256SUMS file and verify it's signature before downloading
any plugins.

This embeds the hashicorp public key in the binary. If the publickey is
replaced, new releases will need to be cut anyway. A
--verify-plugin=false flag will be added to skip signature verification
in these cases.
2017-06-20 13:14:30 -04:00
James Bardin 2e57d284cb Merge pull request #15323 from hashicorp/jbardin/constraint-suggestion
remove "~> 0.0" constraint suggestions
2017-06-16 18:20:24 -04:00
James Bardin ec99b6910b remove "~> 0.0" constraint suggestions
Don't suggest constraints when the available plugin isn't versioned.

Add zero version const for comparisons.
2017-06-16 16:25:36 -04:00
James Bardin 6faace287d remove restriction on unversioned plugins
Discover unversioned plugins regarless of location.
2017-06-16 15:28:48 -04:00
James Bardin ba5b0dc609 mostly remove OS_ARCH knowledge from discovery
Since the command package also needs to know about the specific OS_ARCH
directories, remove the logic fom the discovery package.

This doesn't completely remove the knowledge of the path from discovery,
in order to maintain the current behavior of skipping legacy plugin
names within a new-style path.
2017-06-16 13:58:40 -04:00
James Bardin f723270e3e search the vendor directory for plugins
The default location for users to manually add plugins will be
./terraform.d/plugins/
2017-06-15 10:12:00 -04:00
Martin Atkins af4c82d151 plugin/discovery: Installer.PurgeUnused method
Given a map describing the chosen plugin for each provider name, this
method should purge any other plugins present in the local cache
directory.
2017-06-13 12:28:07 -07:00
Martin Atkins f753974bb3 plugin/discovery: Installer interface, and provider implementation
Previously we had a "getProvider" function type used to implement plugin
fetching. Here we replace that with an interface type, initially with
just a "Get" function.

For now this just simplifies the interface by allowing the target
directory and protocol version to be members of the struct rather than
passed as arguments.

A later change will extend this interface to also include a method to
purge unused plugins, so that upgrading frequently doesn't leave behind
a trail of unused executable files.
2017-06-13 12:28:07 -07:00
James Bardin 840978b2d5 udpate to plugin name convention
Names are case-insensitive, using lowercase by default.
2017-06-09 14:03:59 -07:00
James Bardin 1b201e67ea fix releases path and protocol header
Last minute change to the location of the binaries
2017-06-09 14:03:59 -07:00
James Bardin 283ae0c7c4 fix comment and add some logging in discovery 2017-06-09 14:03:59 -07:00
James Bardin ea3d9c86ba sort constraints for consistent output 2017-06-09 14:03:59 -07:00
Martin Atkins 4ba20f9c1c command init: show suggested constraints for unconstrained providers
When running "terraform init" with providers that are unconstrained, we
will now produce information to help the user update configuration to
constrain for the particular providers that were chosen, to prevent
inadvertently drifting onto a newer major release that might contain
breaking changes.

A ~> constraint is used here because pinning to a single specific version
is expected to create dependency hell when using child modules. By using
this constraint mode, which allows minor version upgrades, we avoid the
need for users to constantly adjust version constraints across many
modules, but make major version upgrades still be opt-in.

Any constraint at all in the configuration will prevent the display of
these suggestions, so users are free to use stronger or weaker constraints
if desired, ignoring the recommendation.
2017-06-09 14:03:59 -07:00
James Bardin dbbafbd43f clean up plugin fetching
We can filter the allowed versions and sort them before checking the
protocol version, that way we can just return the first one found
reducing network requests.
2017-06-09 14:03:59 -07:00
James Bardin 48d37131e0 more tests for fetching providers
Extend the test reslease server to return the protocol version header
and a dummy zip file for the provider.

Test filtering the plugins by plugin protocol version and add a full
GetProvder test.
2017-06-09 14:03:59 -07:00
James Bardin 5f053a2e64 refactor GetProvider
Get provider needs to be provided with the plugin protocol version,
because it can't be imported here directly.

The plugin url types and methods were confusing; replace them with a few
functions to format the urls.
2017-06-09 14:03:59 -07:00
James Bardin 8ad67991a5 check protocol version on plugins
Verify that the plugin we're requesting has a compatible protocol
version.
2017-06-09 14:03:59 -07:00
James Bardin fdeb3d929c Add Versions.Sort
Sort versions from newest to oldest.
2017-06-09 14:03:59 -07:00
Martin Atkins 04bcece59c plugin/discovery: handle the -Xn suffix used by auto-installed plugins
This is used to mark the plugin protocol version. Currently we actually
just ignore this entirely, since only one protocol version exists anyway.
Later we will need to add checks here to ensure that we only pay attention
to plugins of the right version.
2017-06-09 14:03:59 -07:00
Martin Atkins e3401947a6 plugin/discovery: PluginRequirements can specify SHA256 digests
As well as constraining plugins by version number, we also want to be
able to pin plugins to use specific executables so that we can detect
drift in available plugins between commands.

This commit allows such requirements to be specified, but doesn't yet
specify any such requirements, nor validate them.
2017-06-09 14:03:59 -07:00
James Bardin 044ad5ef59 rename some Constraints methods per code review 2017-06-09 14:03:59 -07:00
James Bardin 211f5b5d6e add test for newestVersion 2017-06-09 14:03:59 -07:00
James Bardin 46190590cb change []*Version to []Version
Versions are used as values, so don't keep them as pointers here
2017-06-09 14:03:59 -07:00
James Bardin a547e7c2f0 add releases version listing test 2017-06-09 14:03:59 -07:00
James Bardin 2749946f5c basic plugin getter
Add discovery.GetProviders to fetch plugins from the relases site.

This is an early version, with no tests, that only (probably) fetches
plugins from the default location. The URLs are still subject to change,
and since there are no plugin releases, it doesn't work at all yet.
2017-06-09 14:03:59 -07:00
James Bardin fa49c69793 expose Version.NewerThan
NewerThan is useful outside of the package
2017-06-09 14:03:59 -07:00
James Bardin 7d2d951f27 Rename VersionSet to Constraints
VersionSet is a wrapper around version.Contraints, so rename it it as
such.
2017-06-09 14:03:59 -07:00
James Bardin 718ede0636 have Meta.Backend use a Config rather than loading
Instead of providing the a path in BackendOpts, provide a loaded
*config.Config instead. This reduces the number of places where
configuration is loaded.
2017-06-09 14:03:59 -07:00
Martin Atkins cfc08ae7e3 plugin/discovery: provide an AllVersions set
Various implied dependencies will accept all versions, so this is
convenient for populating dependency data structures for such implied
dependencies.
2017-06-09 14:03:59 -07:00
Martin Atkins a1e29ae290 plugin/discovery: use go-version instead of semver
The semver library we were using doesn't have support for a "pessimistic
constraint" where e.g. the user wants to accept only minor or patch
version upgrades. This is important for providers since users will
generally want to pin their dependencies to not inadvertantly accept
breaking changes.

So here we switch to hashicorp's home-grown go-version library, which
has the ~> constraint operator for this sort of constraint.

Given how much the old version object was already intruding into the
interface and creating dependency noise in callers, this also now wraps
the "raw" go-version objects in package-local structs, thus keeping the
details encapsulated and allowing callers to deal just with this package's
own types.
2017-06-09 14:03:59 -07:00
Martin Atkins 9b4f15c261 plugin: move Client function into plugin, from plugin/discovery
Having this as a method of PluginMeta felt most natural, but unfortunately
that means that discovery must depend on plugin and plugin in turn
depends on core Terraform, thus making the discovery package hard to use
without creating dependency cycles.

To resolve this, we invert the dependency and make the plugin package be
responsible for instantiating clients given a meta, using a top-level
function.
2017-06-09 14:03:59 -07:00
Martin Atkins bcd395e6bd plugin/discovery: Allow overriding the paths of certain plugin names
The .terraformrc file allows the user to override the executable location
for certain plugins. This mechanism allows us to retain that behavior for
a deprecation period by treating such executables as an unversioned
plugin for the given name and excluding all other candidates for that
name, thus ensuring that the override will "win".

Users must eventually transition away from using this mechanism and use
vendor directories instead, because these unversioned overrides will never
match for a provider referenced with non-zero version constraints.
2017-06-09 14:03:59 -07:00
Martin Atkins 551bc9c4a4 plugin/discovery: build plugin clients
These new methods ClientConfig and Client provide the bridge into the
main plugin infrastructure by configuring and instantiating (respectively)
a client object for the referenced plugin.

This stops short of getting the proxy object from the client since that
then requires referencing the interface for the plugin kind, which would
then create a dependency on the main terraform package which we'd rather
avoid here. It'll be the responsibility of the caller in the command
package to do the final wiring to get a provider instance out of a
provider plugin client.
2017-06-09 14:03:59 -07:00
Martin Atkins 0a08214a74 plugin/discovery: SHA256() method to get the hash of each plugin
This will be used later to verify that installed plugins match what's
available on the releases server.
2017-06-09 14:03:59 -07:00
Martin Atkins a94e9a4362 plugin/discovery: find plugins in a given set of directories
For now this supports both our old and new directory layouts, so we can
preserve compatibility with existing configurations until a future major
release where support for the old paths will be removed.

Currently this allows both styles in all given directories, which means we
support more variants than we actually intend to but this is accepted to
keep the interface simple such that we can easily remove the legacy
exception later. The documentation will reflect only the subset of
path layouts that we actually intend to support.
2017-06-09 14:03:59 -07:00
Martin Atkins 39deb1ff6b plugin/discovery: helpers for wrangling plugin versions
With forthcoming support for versioned plugins we need to be able to
answer questions like what versions of plugins are currently installed,
what's the newest version of a given plugin available, etc.

PluginMetaSet gives us a building block for this sort of plugin version
wrangling.
2017-06-09 14:03:59 -07:00
Mitchell Hashimoto 700c74bdd0
plugin: bump protocol version
We changed the type of InstanceState.Meta and this will break
pre-existing plugins. They simply have to recompile to work.
2017-03-01 14:42:33 -08:00
Mitchell Hashimoto 96884ec42d
plugin: bump the protocol version due to Provisioner change 2017-01-26 15:03:27 -08:00
Mitchell Hashimoto 2e894c4ef7
plugin: add ResourceProvisioner.Stop API 2017-01-26 15:03:27 -08:00
Mitchell Hashimoto b3124e16ca
plugin: bump protocol version
This is necessary since the TypeUnknown HIL handling in helper/schema
makes providers compiled WITHOUT TypeUnknown incompatible with the way
core handles unknown values.
2016-11-17 10:06:29 -08:00
Mitchell Hashimoto 045ddf3695
plugin: implement ResourceProvider.Stop 2016-10-25 12:08:36 -07:00
Mitchell Hashimoto 60140b28f4
Revert "Merge pull request #9536 from hashicorp/f-provider-stop"
This reverts commit c3a4cff133, reversing
changes made to 791a02e6e4.

This change requires plugin recompilation and we should hold off until a
minor release for that.
2016-10-25 12:00:36 -07:00
Mitchell Hashimoto 43b5818b55
plugin: implement ResourceProvider.Stop 2016-10-25 11:32:17 -07:00
Clint a9aaf44a87 fix make issues (supersedes #7868) (#7876)
* Fixing the make error or invalid data type for errorf and printf

* fix make errors
2016-07-29 21:05:57 +01:00
Martin Atkins 0e0e3d73af core: New ResourceProvider methods for data resources
This is a breaking change to the ResourceProvider interface that adds the
new operations relating to data sources.

DataSources, ValidateDataSource, ReadDataDiff and ReadDataApply are the
data source equivalents of Resources, Validate, Diff and Apply (respectively)
for managed resources.

The diff/apply model seems at first glance a rather strange workflow for
read-only resources, but implementing data resources in this way allows them
to fit cleanly into the standard plan/apply lifecycle in cases where the
configuration contains computed arguments and thus the read must be deferred
until apply time.

Along with breaking the interface, we also fix up the plugin client/server
and helper/schema implementations of it, which are all of the callers
used when provider plugins use helper/schema. This would be a breaking
change for any provider plugin that directly implements the provider
interface, but no known plugins do this and it is not recommended.

At the helper/schema layer the implementer sees ReadDataApply as a "Read",
as opposed to "Create" or "Update" as in the managed resource Apply
implementation. The planning mechanics are handled entirely within
helper/schema, so that complexity is hidden from the provider implementation
itself.
2016-05-14 08:26:36 -07:00
Mitchell Hashimoto ba84485c87
plugin: tests for ImportState 2016-05-11 13:02:35 -07:00
Mitchell Hashimoto a6b4bb4932
plugin: new API for ImportState 2016-05-11 13:02:35 -07:00
Mitchell Hashimoto 498e23162d
plugin: implement new interface methods 2016-05-11 13:02:34 -07:00
Mitchell Hashimoto 84214437b3 Use hashicorp/go-plugin for plugin system
This replaces this plugin system with the extracted hashicorp/go-plugin
library.

This doesn't introduce any new features such as binary flattening but
opens us up to that a lot more easily and removes a lot of code from TF
in favor of the upstream lib.

This will introduce a protocol change that will cause all existing
plugins to have to be recompiled to work properly. There is no actual
API changes so they just have to recompile, but it is technically
backwards incompatible.
2016-05-10 14:14:47 -04:00
Martin Atkins 8173cd25bb Demote some log lines to DEBUG.
Now that we support log line filtering (as of 0090c063) it's good to be
a bit more fussy about what log levels are assigned to different things.

Here we make a few things that are implementation details log as DEBUG,
and prevent spurious errors from EvalValidateCount where it was returning
an empty EvalValidateError rather than nil when everything was okay.
2015-10-11 10:45:33 -07:00
Sander van Harmelen c9e9e374bb Adding some abstractions for the communicators
This is needed as preperation for adding WinRM support. There is still
one error in the tests which needs another look, but other than that it
seems like were now ready to start working on the WinRM part…
2015-05-01 18:48:54 +02:00
Mitchell Hashimoto 04858e1a15 plugin: Client/Server uses new RPC client/server 2014-09-28 11:19:24 -07:00
Mitchell Hashimoto 0281c1108e rpc: Create the Client/Server 2014-09-28 10:49:27 -07:00
Mitchell Hashimoto 0de633abeb plugin: up API version to 2 2014-09-16 16:49:21 -07:00
Mitchell Hashimoto 08dbf4daf0 fmt 2014-08-30 17:02:11 -07:00
Mitchell Hashimoto fb9810ca5c website: document how to write providers using the new helper/schema 2014-08-26 21:31:53 -07:00
Alex Gaynor 46154ca1d3 Fixed a ton of typos in docs and comments 2014-08-07 00:19:56 -07:00
Mitchell Hashimoto e0f2197d6c plugin: better logging 2014-07-24 07:30:21 -07:00
Armon Dadgar 3849ca80ef plugin: Adding support for provisioners 2014-07-10 11:38:56 -07:00
Mitchell Hashimoto 0cda9a7539 some comments, package comments 2014-05-28 21:27:10 -07:00
Mitchell Hashimoto 8163d364c9 plugin: ResourceProviderFactory 2014-05-28 21:19:44 -07:00
Mitchell Hashimoto 951b7b18eb plugin: can execute plugins and communicate that way 2014-05-28 21:09:47 -07:00