Commit Graph

158 Commits

Author SHA1 Message Date
James Bardin dcab82e897 send and receive Private through ReadResource
Send Private data blob through ReadResource as well. This will allow for
extra flexibility for future providers that may want to pass data out of
band through to their resource Read functions.
2019-06-03 18:08:26 -04:00
James Bardin 9a2a6d14bd Stop grpc server when running ACC tests
The grpc server does not shutdown when the listener is closed. Since
tests aren't run through go-plugin, which has a separate RPC Shutdown
channel to stop the server, we need to track and stop the server
directly.
2019-04-29 14:14:04 -04:00
Martin Atkins eed605ac05 [WIP] Re-enable the end-to-end tests (#20044)
* internal/initwd: Allow deprecated relative module paths

In Terraform 0.11 we deprecated this form but didn't have any explicit
warning for it. Now we'll still accept it but generate a warning. In a
future major release we will drop this form altogether, since it is
ambiguous with registry module source addresses.

This codepath is covered by the command/e2etest suite.

* e2e: Skip copying .exists file, if present

We use this only in the "empty" test fixture in order to let git know that
the directory exists. We need to skip copying it so that we can test
"terraform init -from-module=...", which expects to find an empty
directory.

* command/e2etests: Re-enable and fix up the e2etest "acctests"

We disabled all of the tests that accessed remote services like the
Terraform Registry while they were being updated to support the new
protocols we now expect. With those services now in place, we can
re-enable these tests.

Some details of exactly what output we print, etc, have intentionally
changed since these tests were last updated.

* e2e: refactor for modern states and plans

* command/e2etest: re-enable e2etests and update for tf 0.12 compatibility
plugin/discovery: mkdirAll instead of mkdir when creating cache dir
2019-04-29 13:03:24 -04:00
Radek Simko b171997b39
plugin/discovery: Fix quoting of provider name 2019-04-24 20:47:32 +01:00
Martin Atkins 88e76fa9ef configs/configschema: Introduce the NestingGroup mode for blocks
In study of existing providers we've found a pattern we werent previously
accounting for of using a nested block type to represent a group of
arguments that relate to a particular feature that is always enabled but
where it improves configuration readability to group all of its settings
together in a nested block.

The existing NestingSingle was not a good fit for this because it is
designed under the assumption that the presence or absence of the block
has some significance in enabling or disabling the relevant feature, and
so for these always-active cases we'd generate a misleading plan where
the settings for the feature appear totally absent, rather than showing
the default values that will be selected.

NestingGroup is, therefore, a slight variation of NestingSingle where
presence vs. absence of the block is not distinguishable (it's never null)
and instead its contents are treated as unset when the block is absent.
This then in turn causes any default values associated with the nested
arguments to be honored and displayed in the plan whenever the block is
not explicitly configured.

The current SDK cannot activate this mode, but that's okay because its
"legacy type system" opt-out flag allows it to force a block to be
processed in this way anyway. We're adding this now so that we can
introduce the feature in a future SDK without causing a breaking change
to the protocol, since the set of possible block nesting modes is not
extensible.
2019-04-10 14:53:52 -07:00
James Bardin 7df3275120
Merge pull request #20957 from hashicorp/jbardin/shim-schemas
New methods for obtaining CoreConfigSchema for shimming
2019-04-08 18:16:40 -04:00
James Bardin a3d58665ad use LegacyResourceSchema
rather than the previous .CoreConfigSchemaForShimming
2019-04-08 16:45:35 -04:00
Pam Selle 346e341ff2 Only display status link if public registry used
Updates to throw a specific error if using the defined public
registry vs. another registry.
2019-04-05 16:49:27 -04:00
James Bardin 2ebc2beda0 add 3rd param to mock call 2019-04-02 16:11:32 -04:00
James Bardin 746aac8bda increase grpc recv limit
Some providers may generate quite large schemas, and the internal
default grpc response size limit is 4MB. 64MB should cover most any use
case, and if we get providers nearing that we may want to consider a
finer-grained API to fetch individual resource schemas.
2019-04-02 15:52:54 -04:00
Pam Selle d72456d188 Add friendly error for when registry unresponsive
If the registry is unresponsive, you will now get an error
specific to this, rather than a misleading "provider unavailable" type
error. Also adds debug logging for when errors like this may occur
2019-03-27 14:39:14 -04:00
Justin Downing 1e32ae243c grammatical updates to comments and docs (#20195) 2019-03-21 14:05:41 -07:00
findkim 161fe47b34 plugin/discovery: improve providery discovery verification errors 2019-03-21 14:38:51 -05:00
findkim 2726282307 plugin/discovery: add context for provider discovery verification 2019-03-21 14:38:20 -05:00
findkim 1a32617d5e plugin/discovery: verify checksum matches Registry response 2019-03-21 12:31:31 -05:00
Justin Campbell e6316c9de6 plugin/discovery: Parse warnings from TF Registry
Terraform Registry (and other registry implementations) can now return
an array of warnings with the versions response. These warnings are now
displayed to the user during a `terraform init`.
2019-03-18 12:21:27 -04:00
Justin Campbell 24e13d8ec1 plugin/discovery: Return tfdiags from Get
Allows us to surface warnings to the user using the tfdiags interfaces.
2019-03-18 12:21:27 -04:00
Sander van Harmelen 973e2a7cf9 core: add a context to the UIInput interface 2019-03-08 10:24:40 +01:00
Kristin Laemmert 9402b86217
plugin/discovery: create target plugin directory if it does not exist (#20575)
If the user supplies a plugin cache dir, the provider installer was not
checking if target directory exists before linking the cached pluging.
2019-03-05 12:39:24 -08:00
Justin Campbell 0574eadb62 plugin/discovery: Verify sig against HashiCorp key
https://github.com/hashicorp/terraform/pull/19389 introduced a change to
the provider GPG signature verification process, and removed the
hardcoded HashiCorp GPG key.

While the changes were intended and are still planned for a future
release, we should still be verifying all providers in the TF 0.12.0
release against the HashiCorp GPG key until a more robust key
verification procedure is in place.

Fixes https://github.com/hashicorp/terraform/issues/20527
2019-03-01 14:05:51 -05:00
Martin Atkins 31299e688d core: Allow legacy SDK to opt out of plan-time safety checks
Due to the inprecision of our shimming from the legacy SDK type system to
the new Terraform Core type system, the legacy SDK produces a number of
inconsistencies that produce only minor quirky behavior or broken
edge-cases. To retain compatibility with those existing weird behaviors,
the legacy SDK opts out of our safety checks.

The intent here is to allow existing providers to continue to do their
previous unsafe behaviors for now, accepting that this will allow certain
quirky bugs from previous releases to persist, and then gradually migrate
away from the legacy SDK and remove this opt-out on a per-resource basis
over time.

As with the apply-time safety check opt-out, this is reserved only for
the legacy SDK and must not be used in any new SDK implementations. We
still include any inconsistencies as warnings in the logs as an aid to
anyone debugging weird behavior, so that they can see situations where
blame may be misplaced in the user-visible error messages.
2019-02-11 17:26:49 -08:00
Martin Atkins 1530fe52f7 core: Legacy SDK providers opt out of our new apply result check
The shim layer for the legacy SDK type system is not precise enough to
guarantee it will produce identical results between plan and apply. In
particular, values that are null during plan will often become zero-valued
during apply.

To avoid breaking those existing providers while still allowing us to
introduce this check in the future, we'll introduce a rather-hacky new
flag that allows the legacy SDK to signal that it is the legacy SDK and
thus disable the check.

Once we start phasing out the legacy SDK in favor of one that natively
understands our new type system, we can stop setting this flag and thus
get the additional safety of this check without breaking any
previously-released providers.

No other SDK is permitted to set this flag, and we will remove it if we
ever introduce protocol version 6 in future, assuming that any provider
supporting that protocol will always produce consistent results.
2019-02-06 11:40:30 -08:00
Martin Atkins a9274beaca build: Run "go generate" in modules mode
It seems that all of the tools we run here are now sufficiently
modules-aware to run without problems in modules mode, and indeed running
_not_ in modules mode was causing problems with locating packages in
mockgen.
2019-02-06 11:19:44 -08:00
Martin Atkins bdcac8792d plugin: Use correct schema when marshaling imported resource objects
Previously we were using the type name requested in the import to select
the schema, but a provider is free to return additional objects of other
types as part of an import result, and so it's important that we perform
schema selection separately for each returned object.

If we don't do this, we get confusing downstream errors where the
resulting object decodes to the wrong type and breaks various invariants
expected by Terraform Core.

The testResourceImportOther test in the test provider didn't catch this
previously because it happened to have an identical schema to the other
resource type being imported. Now the schema is changed and also there's
a computed attribute we can set as part of the refresh phase to make sure
we're completing the Read call properly during import. Refresh was working
correctly, but we didn't have any tests for it as part of the import flow.
2019-02-01 15:22:54 -08:00
findkim 7816e61614 Bump installer protocol version to 5 and separate client and server protocol references 2019-01-16 15:07:57 -06:00
findkim abeb86072b Remove duplicate logging 2019-01-15 15:39:28 -06:00
findkim 6e0de3e3f5 Nest incompatible provider protocol error to include dynamic custom msg 2019-01-14 14:56:47 -06:00
findkim f75d83c1cd Prerelease versions are filtered for plugin protocol mismatches 2019-01-11 12:56:47 -06:00
findkim 368ac85a26 Add provider protocol compatibility UI err msg during registry discovery 2019-01-11 11:28:09 -06:00
findkim 43b22d4e14 Fix plugin protocol test 2019-01-09 15:20:33 -06:00
James Bardin 6f54bfaa7c send config during apply 2019-01-08 15:16:46 -05:00
James Bardin e706f34794 generate new plugin proto mocks 2018-12-10 17:16:13 -05:00
Martin Atkins 53926ea581 plugin: Fix incorrect trace log message in provider Close
Was incorrectly printing out "PlanResourceChange" instead of "Close".
2018-12-07 17:05:36 -08:00
James Bardin 1d4c8403a2 enable Auto mTLS in go-plugin 2018-12-05 20:30:16 -05:00
James Bardin 924b97238f Handle StateFuncs in provider shim
Any state modifying functions can only be run once during the plan-apply
cycle. When regenerating the Diff during ApplyResourceChange, strip out
all StateFunc and CustomizeDiff functions from the schema.

Thew NewExtra diff field was where config data that was modified by a
StateFunc was stored, and needs to be maintained between plan and apply.

During PlanResourceChange, store any NewExtra data from the Diff in the
PlannedPrivate data, and re-insert the NewExtra data into the Diff
generated during ApplyResourceChange.
2018-12-03 18:12:02 -05:00
Martin Atkins 72e279e6b2 providers: Consistently use int64 for schema versions
The rest of Terraform is still using uint64 for this in various spots, but
we'll update that gradually later. We use int64 here because that matches
what's used in our protobuf definition, and unsigned integers are not
portable across all of the protobuf target languages anyway.
2018-11-30 11:22:39 -08:00
findkim 5e06e39fcc Use registry alias to fetch providers 2018-11-28 11:54:39 -06:00
Justin Campbell 06825bf46d plugin/discovery: Add friendly gpg err msg
When GPG verification fails, display a helpful message to the user instead of the generic openpgp error.
2018-11-20 14:09:16 -05:00
Justin Campbell 9a8a74b9bb plugin/discovery: Print name before verification
This is so that any errors output from the checksum/signature verification show up in the expected place in the output.
2018-11-20 14:09:16 -05:00
Justin Campbell 495826444b plugin/discovery: Use GPG keys from Registry
When verifying the signature of the SHA256SUMS file, we have been
hardcoding HashiCorp's public GPG key and using it as the keyring.

Going forward, Terraform will get a list of valid public keys for a
provider from the Terraform Registry (registry.terraform.io), and use
them as the keyring for the openpgp verification func.
2018-11-20 14:09:16 -05:00
Martin Atkins 4fe9632f09 plugin: Establish our current plugin protocol as version 5
The main significant change here is that the package name for the proto
definition is "tfplugin5", which is important because this name is part
of the wire protocol for references to types defined in our package.

Along with that, we also move the generated package into "internal" to
make it explicit that importing the generated Go package from elsewhere is
not the right approach for externally-implemented SDKs, which should
instead vendor the proto definition they are using and generate their
own stubs to ensure that the wire protocol is the only hard dependency
between Terraform Core and plugins.

After this is merged, any provider binaries built against our
helper/schema package will need to be rebuilt so that they use the new
"tfplugin5" package name instead of "proto".

In a future commit we will include more elaborate and organized
documentation on how an external codebase might make use of our RPC
interface definition to implement an SDK, but the primary concern here
is to ensure we have the right wire package name before release.
2018-11-19 09:56:41 -08:00
Martin Atkins ec57927ea3 build: Take protoc out of the "go generate" path
Since protoc is not go-gettable, and most development tasks in Terraform
won't involve recompiling protoc files anyway, we'll use a separate
mechanism for these.

This way "go generate" only depends on things we can "go get" in the
"make tools" target.

In a later commit we should also in some way specify a particular version
of protoc to use so that we don't get "flapping" regenerations as
developers work with different versions, but the priority here is just to
make "make generate" minimally usable again to restore the dev workflow
documented in the README.

This also includes some updates that resulted from running "make generate"
and "make protobuf" after those Makefile changes were in place.
2018-10-18 10:39:20 -07:00
James Bardin 3bdd3b2a6e update plugin client with PrepareProviderConfig 2018-10-18 08:48:54 -04:00
James Bardin b173037eb3 update mock for PrepareProviderConfig 2018-10-18 08:48:54 -04:00
James Bardin ba472c7059 Update proto with new PrepareProviderConfig method 2018-10-18 08:48:54 -04:00
James Bardin 3a0c2f1b67 close grpc TestListener 2018-10-17 12:13:59 -04:00
Martin Atkins e25f79ed28 plugin/convert: Show approximate location context for all provider errors
Even if a provider doesn't indicate a specific attribute as the cause of
a resource operation error, we know the error relates to some aspect of
the resource, so we'll include that approximate information in the result
so that we don't produce user-hostile error messages with no context
whatsoever.

Later we can hopefully refine this to place the source range on the header
of the configuration block rather than on an empty part of the body, but
that'll require some more complex rework here and so for now we'll just
accept this as an interim state so that the user can at least figure out
which resource block the error is coming from.
2018-10-16 19:14:54 -07:00
Martin Atkins 3eaf2c308d plugin/discovery: Use mockos_mockarch when testing installation
If we don't override these then the tests can only pass on one platform.
2018-10-16 19:14:54 -07:00
James Bardin 0d4d572c39 start work on helper/resource test fixtures
The helper resource tests won't pass for now, as they use a
terraform.MockProvider which can't be used in the schema.Provider shims.
2018-10-16 19:14:11 -07:00
James Bardin 52c0032aed update provisioners for multiple processes
The "internal" provisioners are still run in a separate process, and
need to be updated to restart on each walk.
2018-10-16 19:14:11 -07:00