Commit Graph

185 Commits

Author SHA1 Message Date
Martin Atkins bdcac8792d plugin: Use correct schema when marshaling imported resource objects
Previously we were using the type name requested in the import to select
the schema, but a provider is free to return additional objects of other
types as part of an import result, and so it's important that we perform
schema selection separately for each returned object.

If we don't do this, we get confusing downstream errors where the
resulting object decodes to the wrong type and breaks various invariants
expected by Terraform Core.

The testResourceImportOther test in the test provider didn't catch this
previously because it happened to have an identical schema to the other
resource type being imported. Now the schema is changed and also there's
a computed attribute we can set as part of the refresh phase to make sure
we're completing the Read call properly during import. Refresh was working
correctly, but we didn't have any tests for it as part of the import flow.
2019-02-01 15:22:54 -08:00
findkim 7816e61614 Bump installer protocol version to 5 and separate client and server protocol references 2019-01-16 15:07:57 -06:00
findkim abeb86072b Remove duplicate logging 2019-01-15 15:39:28 -06:00
findkim 6e0de3e3f5 Nest incompatible provider protocol error to include dynamic custom msg 2019-01-14 14:56:47 -06:00
findkim f75d83c1cd Prerelease versions are filtered for plugin protocol mismatches 2019-01-11 12:56:47 -06:00
findkim 368ac85a26 Add provider protocol compatibility UI err msg during registry discovery 2019-01-11 11:28:09 -06:00
findkim 43b22d4e14 Fix plugin protocol test 2019-01-09 15:20:33 -06:00
James Bardin 6f54bfaa7c send config during apply 2019-01-08 15:16:46 -05:00
James Bardin e706f34794 generate new plugin proto mocks 2018-12-10 17:16:13 -05:00
Martin Atkins 53926ea581 plugin: Fix incorrect trace log message in provider Close
Was incorrectly printing out "PlanResourceChange" instead of "Close".
2018-12-07 17:05:36 -08:00
James Bardin 1d4c8403a2 enable Auto mTLS in go-plugin 2018-12-05 20:30:16 -05:00
James Bardin 924b97238f Handle StateFuncs in provider shim
Any state modifying functions can only be run once during the plan-apply
cycle. When regenerating the Diff during ApplyResourceChange, strip out
all StateFunc and CustomizeDiff functions from the schema.

Thew NewExtra diff field was where config data that was modified by a
StateFunc was stored, and needs to be maintained between plan and apply.

During PlanResourceChange, store any NewExtra data from the Diff in the
PlannedPrivate data, and re-insert the NewExtra data into the Diff
generated during ApplyResourceChange.
2018-12-03 18:12:02 -05:00
Martin Atkins 72e279e6b2 providers: Consistently use int64 for schema versions
The rest of Terraform is still using uint64 for this in various spots, but
we'll update that gradually later. We use int64 here because that matches
what's used in our protobuf definition, and unsigned integers are not
portable across all of the protobuf target languages anyway.
2018-11-30 11:22:39 -08:00
findkim 5e06e39fcc Use registry alias to fetch providers 2018-11-28 11:54:39 -06:00
Justin Campbell 06825bf46d plugin/discovery: Add friendly gpg err msg
When GPG verification fails, display a helpful message to the user instead of the generic openpgp error.
2018-11-20 14:09:16 -05:00
Justin Campbell 9a8a74b9bb plugin/discovery: Print name before verification
This is so that any errors output from the checksum/signature verification show up in the expected place in the output.
2018-11-20 14:09:16 -05:00
Justin Campbell 495826444b plugin/discovery: Use GPG keys from Registry
When verifying the signature of the SHA256SUMS file, we have been
hardcoding HashiCorp's public GPG key and using it as the keyring.

Going forward, Terraform will get a list of valid public keys for a
provider from the Terraform Registry (registry.terraform.io), and use
them as the keyring for the openpgp verification func.
2018-11-20 14:09:16 -05:00
Martin Atkins 4fe9632f09 plugin: Establish our current plugin protocol as version 5
The main significant change here is that the package name for the proto
definition is "tfplugin5", which is important because this name is part
of the wire protocol for references to types defined in our package.

Along with that, we also move the generated package into "internal" to
make it explicit that importing the generated Go package from elsewhere is
not the right approach for externally-implemented SDKs, which should
instead vendor the proto definition they are using and generate their
own stubs to ensure that the wire protocol is the only hard dependency
between Terraform Core and plugins.

After this is merged, any provider binaries built against our
helper/schema package will need to be rebuilt so that they use the new
"tfplugin5" package name instead of "proto".

In a future commit we will include more elaborate and organized
documentation on how an external codebase might make use of our RPC
interface definition to implement an SDK, but the primary concern here
is to ensure we have the right wire package name before release.
2018-11-19 09:56:41 -08:00
Martin Atkins ec57927ea3 build: Take protoc out of the "go generate" path
Since protoc is not go-gettable, and most development tasks in Terraform
won't involve recompiling protoc files anyway, we'll use a separate
mechanism for these.

This way "go generate" only depends on things we can "go get" in the
"make tools" target.

In a later commit we should also in some way specify a particular version
of protoc to use so that we don't get "flapping" regenerations as
developers work with different versions, but the priority here is just to
make "make generate" minimally usable again to restore the dev workflow
documented in the README.

This also includes some updates that resulted from running "make generate"
and "make protobuf" after those Makefile changes were in place.
2018-10-18 10:39:20 -07:00
James Bardin 3bdd3b2a6e update plugin client with PrepareProviderConfig 2018-10-18 08:48:54 -04:00
James Bardin b173037eb3 update mock for PrepareProviderConfig 2018-10-18 08:48:54 -04:00
James Bardin ba472c7059 Update proto with new PrepareProviderConfig method 2018-10-18 08:48:54 -04:00
James Bardin 3a0c2f1b67 close grpc TestListener 2018-10-17 12:13:59 -04:00
Martin Atkins e25f79ed28 plugin/convert: Show approximate location context for all provider errors
Even if a provider doesn't indicate a specific attribute as the cause of
a resource operation error, we know the error relates to some aspect of
the resource, so we'll include that approximate information in the result
so that we don't produce user-hostile error messages with no context
whatsoever.

Later we can hopefully refine this to place the source range on the header
of the configuration block rather than on an empty part of the body, but
that'll require some more complex rework here and so for now we'll just
accept this as an interim state so that the user can at least figure out
which resource block the error is coming from.
2018-10-16 19:14:54 -07:00
Martin Atkins 3eaf2c308d plugin/discovery: Use mockos_mockarch when testing installation
If we don't override these then the tests can only pass on one platform.
2018-10-16 19:14:54 -07:00
James Bardin 0d4d572c39 start work on helper/resource test fixtures
The helper resource tests won't pass for now, as they use a
terraform.MockProvider which can't be used in the schema.Provider shims.
2018-10-16 19:14:11 -07:00
James Bardin 52c0032aed update provisioners for multiple processes
The "internal" provisioners are still run in a separate process, and
need to be updated to restart on each walk.
2018-10-16 19:14:11 -07:00
James Bardin a14fe7ae0c missing diagnostics in pugin protocol
Some call diagnostics were being missed
2018-10-16 19:14:11 -07:00
James Bardin 795161f615 update to start a new process for each plugin
Modify the plugin factories to create a new plugin process for each
individual plugin.
2018-10-16 19:14:11 -07:00
James Bardin 35b375d3ee check responses for nil values
While providers shouldn't return nil values, we need to protect against
them. Always assign valid null cty values when the response contains a
nil.
2018-10-16 19:14:11 -07:00
Martin Atkins 88551181d5 plugin: Fix tests
An earlier change made the tests not compile here. We now need to use the
legacyPluginMap function, since pluginMap has now been replaced with
helpers to produce new-style plugin _sets_.
2018-10-16 19:14:11 -07:00
Martin Atkins a7342de274 core: Properly handle no-op changes in plan
Previously we just left these out of the plan altogether, but in the new
plan types we intentionally include change information for every resource
instance, even if no changes are actually planned, to allow alternative
plan file viewers to show what isn't changing as well as what is.
2018-10-16 19:14:11 -07:00
Martin Atkins 9ec00468f8 plugin/convert: Convert cty.PathError to diagnostics with attribute path 2018-10-16 19:14:11 -07:00
Martin Atkins 5d744e6824 plugin: TRACE log for each GRPC-remoted plugin call 2018-10-16 19:14:11 -07:00
Martin Atkins 4bf19797f3 plugin: Fix deadlock in getSchema where mutex is never unlocked
On the second call here this would fail to release the lock.
2018-10-16 19:14:11 -07:00
Martin Atkins 5559d21855 plugin: Register v5 plugins only if present
Previously we would construct both provisioner and the provider objects if
either callback was set, but this is incorrect because a plugin should
actually set only one of these at a time, depending on what kind of plugin
it is.
2018-10-16 19:14:11 -07:00
Martin Atkins 44bc7519a6 terraform: More wiring in of new provider types
This doesn't actually work yet, but it builds and then panics in a pretty
satisfying way.
2018-10-16 19:12:54 -07:00
James Bardin c07ce1cd4b move conversion functions into separate package
Managing which function need to be shared between the terraform plugin
and the helper plugin without creating cycles was becoming difficult.
Move all functions related to converting between terraform and proto
type into plugin/convert.
2018-10-16 19:11:09 -07:00
James Bardin 88bfbeb9c5 schema version changed type 2018-10-16 19:11:09 -07:00
James Bardin b403023841 serve the new version 5 grpc plugins
Use the new go-plugin version negotiation to server th appropriate
plugin type when the client requests protocol version 5.
2018-10-16 18:58:49 -07:00
James Bardin ff7d51a9b4 update tests to match new test func signature 2018-10-16 18:58:49 -07:00
Kristin Laemmert 5e7956f440 registry/response: rename Collection to ProviderVersionCollection for clarity 2018-10-16 18:58:49 -07:00
James Bardin c9d91dc64e remove old comment 2018-10-16 18:58:49 -07:00
Kristin Laemmert bdaf8290b4 registry/client: added a specific error if the registry client does not
support the requested service.
2018-10-16 18:58:49 -07:00
James Bardin c1d4a63fae GRPCProvider and GRPCProvisioner clients
Here we add the GRPCProvisioner and GRPCProvider which implement the
core provisioners.Interface and providers.Interface, and translate
betweeen the core types and the grpc protocol.
2018-10-16 18:58:49 -07:00
James Bardin 81bd3b09d6 generate mocks for some of the grpc interfaces
These will allow easier testing of the grpc endpoints in isolation.
Mocks are generated for ProviderClient, ProvisionerClient,
Provisioner_ProvisionResourceClient, and
Provisioner_ProvisionResourceServer using `go generate`
2018-10-16 18:58:49 -07:00
James Bardin e29053432e grpc proto fixes
Add missing connection field for ProvisionResource.

Add diagnostics to the ProvisionResource stream.
2018-10-16 18:58:49 -07:00
Kristin Laemmert ae015e0f75 plugin/discovery: add function to filter list of versions by those which
support the requested platform.

If the newest version of a provider which matches the version
constraints does not support the requested platform, filter the list of
available versions by platform support and try again.
2018-10-16 18:58:49 -07:00
Kristin Laemmert f83d5866fe plugin/discovery: removing deprecated functions 2018-10-16 18:58:49 -07:00
Kristin Laemmert ce5e66e178 plugin/discovery provider installer: download providers from the registry
Terraform will query the public registry at
https://registry.terraform.io for providers, instead of
https://releases.hashicorp.com.
2018-10-16 18:56:50 -07:00
James Bardin 8f51b2fba5 add grpc service definitions
This add the grpc protobuf service definitions for the new plugin
protocol, as well as the generated code.
2018-10-16 18:53:51 -07:00
Martin Atkins 4ed06a9227 terraform: HCL2-flavored module dependency resolver
For the moment this is just a lightly-adapted copy of
ModuleTreeDependencies named ConfigTreeDependencies, with the goal that
the two can live concurrently for the moment while not all callers are yet
updated and then we can drop ModuleTreeDependencies and its helper
functions altogether in a later commit.

This can then be used to make "terraform init" and "terraform providers"
work properly with the HCL2-powered configuration loader.
2018-10-16 18:44:26 -07:00
James Bardin 0a6938e90a update tests to match pre-release constraint rules 2018-10-16 18:24:47 -07:00
Sean Chittenden d749420a25
Fix drift caused from gofmt when running make dev and go 1.11.
A fresh checkout of `origin/master` does not build atm using the `dev`
target because `master` has not been formatted using `gofmt` from Go
1.11 (tis has been the case for a while if you've been running devel).

None of the drift in question is especially new but now that Go 1.11
has been released and gofmt's formatting guidelines have been updated,
it would be *really* nice if the code in `master` reflected the current
tooling in order to avoid having to fight this drift locally.

* 8mo: https://github.com/hashicorp/terraform/blame/master/backend/remote-state/s3/backend_test.go#L260-L261
* 6mo: https://github.com/hashicorp/terraform/blame/master/builtin/provisioners/chef/linux_provisioner_test.go#L124
* 1yr: 7cfeffe36b/command/init.go (L75-L76)
* 12d: 7cfeffe36b/command/meta_backend_test.go (L1437)
* 2yr: 7cfeffe36b/helper/schema/resource_timeout_test.go (L26)
* 4yr: 7cfeffe36b/helper/schema/schema_test.go (L2059)
* 1yr: 7cfeffe36b/plugin/discovery/get_test.go (L151)
2018-09-09 10:18:08 -07:00
Paul Tyng c868092d2d
Standardize http.Client creation with User-Agent 2018-02-28 12:09:50 -05:00
Masayuki Morita f440dba137 Standardize on log level "WARN" rather than "WARNING" 2018-01-16 18:05:26 -08:00
James Bardin 6cb4e14cf8 Use pooled http client for fetching providers
While the TLS handshakes are a fairly small overhead compared to
downloading the providers, clients in some situation are failing to
complete the TLS handshake in a timely manner. It's unclear if this is
because of heavily constrained clients are stalling while doing the
major crpto operations, or the edge servers are throttling repeated
requests from the same IPs.

This should allow reusing the open TLS connection to the release edge
servers during init.
2017-10-31 10:53:42 -04:00
Martin Atkins 183833affc core: terraform.ResourceProvider.GetSchema method
In order to parse provider, resource and data source configuration from
HCL2 config files, we need to know the relevant configuration schema.
This new method allows Terraform Core to request these from a provider.

This is a breaking change to this interface, so all of its implementers
in this package are updated too. This includes concrete implementations
of the new method in helper/schema that use the schema conversion code
added in an earlier commit to produce a configschema.Block automatically.

Plugins compiled against prior versions of helper/schema will not have
support for this method, and so calls to them will fail. Callers of
this new method will therefore need to sniff for support using the
SchemaAvailable field added to both ResourceType and DataSource.

This careful handling will need to persist until next time we increment
the plugin protocol version, at which point we can make the breaking
change of requiring this information to be available.
2017-10-17 07:23:41 -07:00
Martin Atkins 879899d434 plugin/discovery: plugin caching mechanism
For users that have metered or slow internet connections it is annoying
to have Terraform constantly re-downloading the same files when they
initialize many separate directories.

To help such users, here we add an opt-in mechanism to use a local
directory as a read-through cache. When enabled, any plugin download will
be skipped if a suitable file already exists in the cache directory. If
the desired plugin isn't in the cache, it will be downloaded into the
cache for use next time.

This mechanism also serves to reduce total disk usage by allowing
plugin files to be shared between many configurations, as long as the
target system isn't Windows and supports either hardlinks or symlinks.
2017-09-29 14:03:09 -07:00
Martin Atkins 12d6bc8c30 plugin/discovery: ignore non-files when discovering
If we encounter something that isn't a file -- for example, a dangling
symlink whose referent has been deleted -- we'll ignore it so that we
can either later produce a "no such plugin" error or auto-install a plugin
that will actually work.
2017-09-29 14:03:09 -07:00
James Bardin 3dd0f93aed Merge pull request #15816 from hashicorp/jbardin/plugin-logger
specify a logger for go-plugin
2017-08-15 14:55:11 -04:00
James Bardin 714df97a02 specify a logger for go-plugin
The go-plugin package now uses hclog. The default Logger has a level set
to Info, but all plugin output is relayed via Debug. Create a new named
logger for plugins with the level set to Trace so that all output comes
through.
2017-08-15 14:34:04 -04:00
Radek Simko 28b33c9299
plugin: Display version + source when initializing plugins 2017-08-15 19:29:13 +02:00
Radek Simko de3015cebf
Remove overly verbose DEBUG log 2017-08-03 16:11:45 +02:00
Martin Atkins ac250d2792 plugin/discovery: trim off .exe suffix when parsing filenames
On Windows systems the plugin binaries use a .exe suffix, which we were
misparsing as part of either the "v" or "x" parts of the filename.

This fixes #15578.
2017-07-18 10:00:23 -07:00
James Bardin 2b84e786e9 allow missing x-terraform-protocol-version
If the release site is missing the "x-terraform-protocol-version"
header, we should fetch the latest spec'ed release. Downloading the
wrong protocol version can't do any damage, and the version present is
more than likely compatible.
2017-07-14 15:52:41 -04:00
Martin Atkins 610fcb605e plugin/discovery: allow customizing the OS/arch for auto-install
Previously we forced only installing for the current GOOS and GOARCH. Now
we allow this to be optionally overridden, which allows building tools
that can, for example, populate a directory with plugins to run on a Linux
server while working on a Mac.
2017-07-05 10:02:05 -07:00
James Bardin beebd5cee3 improve discovery logging
Display whether we're looking for a provider or provisioner in the log
messages.
2017-06-27 11:09:54 -04:00
James Bardin e059eb80d1 add test for signature mismatch 2017-06-21 16:17:34 -04:00
James Bardin 2cca6f10fa return sig error when fetching a plugin
A signature error was being ignored while fetching a plugin
2017-06-21 12:57:53 -04:00
James Bardin c10f5caf05 Merge pull request #15345 from hashicorp/jbardin/plugin-sig
verify automatically downloaded plugins
2017-06-20 17:05:54 -04:00
Martin Atkins af2111f24e plugin/discovery: sentinel error values for Get errors
Some errors from Get are essentially user error, so we want to be able to
recognize them and give the user good feedback on how to proceed.

Although sentinel values are not an ideal solution to this, it's something
reasonably simple we can do to get this done without lots of refactoring.
2017-06-20 13:39:45 -07:00
James Bardin 13d835f996 test signature verification
use a published provider's checksum file and signature to check the
verification.
2017-06-20 13:14:30 -04:00
James Bardin 415d562d36 add signature verification
Fetch the SHA256SUMS file and verify it's signature before downloading
any plugins.

This embeds the hashicorp public key in the binary. If the publickey is
replaced, new releases will need to be cut anyway. A
--verify-plugin=false flag will be added to skip signature verification
in these cases.
2017-06-20 13:14:30 -04:00
James Bardin 2e57d284cb Merge pull request #15323 from hashicorp/jbardin/constraint-suggestion
remove "~> 0.0" constraint suggestions
2017-06-16 18:20:24 -04:00
James Bardin ec99b6910b remove "~> 0.0" constraint suggestions
Don't suggest constraints when the available plugin isn't versioned.

Add zero version const for comparisons.
2017-06-16 16:25:36 -04:00
James Bardin 6faace287d remove restriction on unversioned plugins
Discover unversioned plugins regarless of location.
2017-06-16 15:28:48 -04:00
James Bardin ba5b0dc609 mostly remove OS_ARCH knowledge from discovery
Since the command package also needs to know about the specific OS_ARCH
directories, remove the logic fom the discovery package.

This doesn't completely remove the knowledge of the path from discovery,
in order to maintain the current behavior of skipping legacy plugin
names within a new-style path.
2017-06-16 13:58:40 -04:00
James Bardin f723270e3e search the vendor directory for plugins
The default location for users to manually add plugins will be
./terraform.d/plugins/
2017-06-15 10:12:00 -04:00
Martin Atkins af4c82d151 plugin/discovery: Installer.PurgeUnused method
Given a map describing the chosen plugin for each provider name, this
method should purge any other plugins present in the local cache
directory.
2017-06-13 12:28:07 -07:00
Martin Atkins f753974bb3 plugin/discovery: Installer interface, and provider implementation
Previously we had a "getProvider" function type used to implement plugin
fetching. Here we replace that with an interface type, initially with
just a "Get" function.

For now this just simplifies the interface by allowing the target
directory and protocol version to be members of the struct rather than
passed as arguments.

A later change will extend this interface to also include a method to
purge unused plugins, so that upgrading frequently doesn't leave behind
a trail of unused executable files.
2017-06-13 12:28:07 -07:00
James Bardin 840978b2d5 udpate to plugin name convention
Names are case-insensitive, using lowercase by default.
2017-06-09 14:03:59 -07:00
James Bardin 1b201e67ea fix releases path and protocol header
Last minute change to the location of the binaries
2017-06-09 14:03:59 -07:00
James Bardin 283ae0c7c4 fix comment and add some logging in discovery 2017-06-09 14:03:59 -07:00
James Bardin ea3d9c86ba sort constraints for consistent output 2017-06-09 14:03:59 -07:00
Martin Atkins 4ba20f9c1c command init: show suggested constraints for unconstrained providers
When running "terraform init" with providers that are unconstrained, we
will now produce information to help the user update configuration to
constrain for the particular providers that were chosen, to prevent
inadvertently drifting onto a newer major release that might contain
breaking changes.

A ~> constraint is used here because pinning to a single specific version
is expected to create dependency hell when using child modules. By using
this constraint mode, which allows minor version upgrades, we avoid the
need for users to constantly adjust version constraints across many
modules, but make major version upgrades still be opt-in.

Any constraint at all in the configuration will prevent the display of
these suggestions, so users are free to use stronger or weaker constraints
if desired, ignoring the recommendation.
2017-06-09 14:03:59 -07:00
James Bardin dbbafbd43f clean up plugin fetching
We can filter the allowed versions and sort them before checking the
protocol version, that way we can just return the first one found
reducing network requests.
2017-06-09 14:03:59 -07:00
James Bardin 48d37131e0 more tests for fetching providers
Extend the test reslease server to return the protocol version header
and a dummy zip file for the provider.

Test filtering the plugins by plugin protocol version and add a full
GetProvder test.
2017-06-09 14:03:59 -07:00
James Bardin 5f053a2e64 refactor GetProvider
Get provider needs to be provided with the plugin protocol version,
because it can't be imported here directly.

The plugin url types and methods were confusing; replace them with a few
functions to format the urls.
2017-06-09 14:03:59 -07:00
James Bardin 8ad67991a5 check protocol version on plugins
Verify that the plugin we're requesting has a compatible protocol
version.
2017-06-09 14:03:59 -07:00
James Bardin fdeb3d929c Add Versions.Sort
Sort versions from newest to oldest.
2017-06-09 14:03:59 -07:00
Martin Atkins 04bcece59c plugin/discovery: handle the -Xn suffix used by auto-installed plugins
This is used to mark the plugin protocol version. Currently we actually
just ignore this entirely, since only one protocol version exists anyway.
Later we will need to add checks here to ensure that we only pay attention
to plugins of the right version.
2017-06-09 14:03:59 -07:00
Martin Atkins e3401947a6 plugin/discovery: PluginRequirements can specify SHA256 digests
As well as constraining plugins by version number, we also want to be
able to pin plugins to use specific executables so that we can detect
drift in available plugins between commands.

This commit allows such requirements to be specified, but doesn't yet
specify any such requirements, nor validate them.
2017-06-09 14:03:59 -07:00
James Bardin 044ad5ef59 rename some Constraints methods per code review 2017-06-09 14:03:59 -07:00
James Bardin 211f5b5d6e add test for newestVersion 2017-06-09 14:03:59 -07:00
James Bardin 46190590cb change []*Version to []Version
Versions are used as values, so don't keep them as pointers here
2017-06-09 14:03:59 -07:00
James Bardin a547e7c2f0 add releases version listing test 2017-06-09 14:03:59 -07:00
James Bardin 2749946f5c basic plugin getter
Add discovery.GetProviders to fetch plugins from the relases site.

This is an early version, with no tests, that only (probably) fetches
plugins from the default location. The URLs are still subject to change,
and since there are no plugin releases, it doesn't work at all yet.
2017-06-09 14:03:59 -07:00
James Bardin fa49c69793 expose Version.NewerThan
NewerThan is useful outside of the package
2017-06-09 14:03:59 -07:00
James Bardin 7d2d951f27 Rename VersionSet to Constraints
VersionSet is a wrapper around version.Contraints, so rename it it as
such.
2017-06-09 14:03:59 -07:00