Commit Graph

101 Commits

Author SHA1 Message Date
Paddy e6592dc710
Add support for provider metadata to modules. (#22583)
Implement a new provider_meta block in the terraform block of modules, allowing provider-keyed metadata to be communicated from HCL to provider binaries.

Bundled in this change for minimal protocol version bumping is the addition of markdown support for attribute descriptions and the ability to indicate when an attribute is deprecated, so this information can be shown in the schema dump.

Co-authored-by: Paul Tyng <paul@paultyng.net>
2020-03-05 16:53:24 -08:00
Alex Pilon 4bf43efcfd
move hcl2shim package to configs 2019-08-06 19:58:58 -04:00
James Bardin 75602df5ef Revert "Remove removed attribute from applied state"
This reverts commit 2e2a363052.
2019-06-28 11:51:52 -04:00
James Bardin d4b0788854 don't append nil elements to a diagnostic cty.Path
If a set element is nil in validateConfigNulls, we don't want to
append that element to the diagnostic path, since it doesn't offer any
useful info to the user.
2019-06-25 18:05:51 -04:00
James Bardin f362198467
Merge pull request #21806 from hashicorp/jbardin/lost-empty-strings
prevent an empty string from being lost
2019-06-25 12:32:06 -04:00
James Bardin 9365a2d97d private and timeout handling in grpc_provider
Load private data for read, so the resource can get it's configured
timeouts if they exist.

Ensure PlanResourceChange returns the saved private data when there is
an empty diff.

Handle the timeout decoding into Meta in the PlanResourceChange, so that
it's always there for later operations.
2019-06-19 22:48:15 -04:00
James Bardin 4dcda04d96 prevent an empty string from being lost
The helper/schema diff process loses empty strings, causing them to show
up as unset (null) during apply. Besides failing to show as set by
GetOk, the absence of the value also triggers the schema to insert a
default value again during apply.

It would also be be preferable if the defaults weren't re-evaluated
again during ApplyResourceChange, but that would require a more invasive
patch to the field readers, and ensuring the empty string is stored in
the plan should block the default.
2019-06-19 17:42:00 -04:00
James Bardin 8ae31aa2db normalize empty blocks during import
Like Upgrade, Import is another case where we don't have the context of
the configuration and need to ensure missing blocks are normalized.
2019-06-17 20:29:01 -04:00
James Bardin e6ee78555a
Merge pull request #21721 from hashicorp/jbardin/remove-new-removed
Remove removed attribute from applied state
2019-06-17 09:36:23 -04:00
James Bardin dbe22181ae Ensure all object attrs & empty blocks in upgrade
When upgrading from a flatmap state, unset blocks would not exist in the
state, while they will represented as empty in the new cty.Value. This
will cause an unexpected diff in the first plan after upgrade. This
situation may normally be applied with no impact, but some providers may
have unexpected behavior, and if the attributes force replacement it may
require manual alteration of the state to complete the upgrade.
2019-06-13 18:03:47 -04:00
James Bardin 2e2a363052 Remove removed attribute from applied state
When a Diff contains a NewRemoved attribute (which would have been null
in the planned state), the final value is often the "zero" value string
for the type, which the provider itself still applies to the state.
Rather than risking a change of behavior in helper/schema by fixing the
inconsistency, we'll remove the NewRemoved attributes after apply to
prevent further issues resulting from the change in planned value.
2019-06-13 17:29:25 -04:00
Brian Flad 94078f9029
helper/plugin: Allow missing MigrateState for provider flatmap state upgrades
Reference: https://github.com/terraform-providers/terraform-provider-aws/issues/8828

Prior to Terraform 0.12, providers were not required to implement the `MigrateState` function when increasing the `SchemaVersion` above `0`. Effectively there is no flatmap state difference between version 0 and defined `SchemaVersion` or lowest `StateUpgrader` `Version` (whichever is lowest) when `MigrateState` is undefined, so here we remove the error and increase the schema version automatically.
2019-06-05 23:23:45 -04:00
James Bardin a056b84cdd add delete timeout test
The timeout value is still not being persisted.

While it doesn't fix this issue, make sure to always return Private
during Plan.
2019-06-05 19:22:46 -04:00
James Bardin dcab82e897 send and receive Private through ReadResource
Send Private data blob through ReadResource as well. This will allow for
extra flexibility for future providers that may want to pass data out of
band through to their resource Read functions.
2019-06-03 18:08:26 -04:00
James Bardin c9e1d26c25 remove the legacy schema access
Having removed the methods, it is straightforward to mechanically update
this file to get rid of all references to the "legacy schema". There is
now only one config schema type to deal with in the sdk.
2019-05-14 18:12:57 -04:00
James Bardin ec65fb960d sdk: use core schema for json state upgrade
When handling the json state in UpgradeResourceState, the schema
must be what core uses, because that is the schema used for
encoding/decoding the json state.

When converting from flatmap to json state, the legacy schema will be
used to decode the flatmap to a cty value, but the resulting json will
be encoded using the CoreConfigSchema to match what core expects.
2019-05-14 17:59:45 -04:00
James Bardin cf61a689eb only hold back empty container changes in apply
When normalizing the state during read, if the resource was previously
imported, most nil-able values will be nil, and we need to prefer the
values returned by the latest Read operation. This didn't come up
before, because Read is usually working with a state create by plan and
Apply which has already shaped the state with the expected empty values.

Having the src value preferred only during Apply better follows the
intent of this function, which should allow Read to return whatever
values it deems necessary. Since Read and Plan use the same
normalization logic, the implied Read before plan should take care of any
perpetual diffs.
2019-05-13 19:04:25 -04:00
James Bardin b7ff04f1b6 remove extra attributes from state during upgrade
Terraform core would previously ignore unexpected attributes found in
the state, but since we now need to encode/decode the state according
the schema, the attributes must match the schema.

On any state upgrade, remove attributes no longer present in the schema
from the state. The only change this requires from providers is that
going forward removal of attribute is considered a schema change, and
requires an increment of the SchemaVersion in order to trigger the
removal of the attributes from state.
2019-05-10 18:05:41 -04:00
James Bardin 2fe0f9376a change "preferDst" to "apply" in normalizeNullVals
Inverting this and renaming it makes it align with the current use of
that boolean argument.
2019-04-23 12:59:30 -04:00
James Bardin 0696cf7245 don't retain removed map values
Make sure values removed from a map during apply are not copied into the
new map. The broken test is no longer valid in this case, and the
updated diff.Apply should prevent the case it used to cover.
2019-04-23 12:49:58 -04:00
James Bardin af8115dc9b removing the ~ set flag is no longer needed
The computed set sigil ~ should no longer appear in the diffs, because
the config will be cleaned before generating the diff.
2019-04-10 09:39:45 -04:00
James Bardin a3d58665ad use LegacyResourceSchema
rather than the previous .CoreConfigSchemaForShimming
2019-04-08 16:45:35 -04:00
James Bardin c5023c7702 cleanup after AsSingle removal 2019-04-08 16:45:35 -04:00
James Bardin 1a9c06d0f5 Revert "helper/schema: Implementation of the AsSingle mechanism"
This reverts commit 1987a92386.
2019-04-08 16:45:35 -04:00
James Bardin 7b67105407 don't strip new-computeds from plan diffs
Stripping these was a patch for some provider behavior which was fixed
in other ways, and is no longer needed.
Removing this allows us to implement correct CusomizeDiffFuncs in
providers so that they can mark fields with empty values as computed
during a plan.
2019-04-03 17:37:58 -04:00
James Bardin e024960c74 Revert "filter nulls when shimming a config"
This reverts commit 97bde5467c.
2019-04-03 13:52:56 -04:00
James Bardin f5395bd98a validate null values in shimmed configs
A list-like attribute containing null values will present a list to
helper/schema with nils, which can cause panics. Since null values were
not possible in configuration before HCL2 and not supported by the
legacy SDK, return an error to the user.
2019-04-03 11:10:24 -04:00
James Bardin 64c76be804 fixup lost collections containing unknowns
It turns out that collections containing only unknowns could be lost,
meaning there wasn't a direct correlation between the unknown and null
value which would have otherwise been restored.
2019-03-29 14:54:54 -04:00
James Bardin 86e30add98 fix unknowns added to maps by schemaMap
The legacy diff process inserts unknown values into an optional+computed
map. Fix these up in post-plan normalization process, by looking for
known strings that were changed to unknown.
2019-03-29 13:56:43 -04:00
James Bardin 009df443f7 restore lost unknowns during a planned update.
Because schema.ResourceDiff can't differentiate between unknown
values and new computed values, unknowns can be lost during an update.
If a planned value converted an unknown to a null, restore the unknown
so that it can be correctly replaced in the final plan.
2019-03-29 13:56:43 -04:00
Martin Atkins 135121562e helper/plugin: Implement Schema.SkipCoreTypeCheck
The previous commit added this flag but did not implement it. Here we
implement it by adjusting the shape of schema we return to Terraform Core
to mark the attribute as untyped and then ensure that gets handled
correctly on the SDK side.
2019-03-21 15:19:59 -07:00
Martin Atkins 1987a92386 helper/schema: Implementation of the AsSingle mechanism
The previous commit added a new flag to schema.Schema which is documented
to make a list with MaxItems: 1 be presented to Terraform Core as a single
value instead, giving a way to switch to non-list nested resources without
it being a breaking change for Terraform v0.11 users as long as it's done
prior to a provider's first v0.12-compatible release.

This is the implementation of that mechanism. It's intentionally
implemented as a suite of extra fixups rather than direct modifications to
existing shim code because we want to ensure that this has no effect
whatsoever on the result of a resource type that _isn't_ using AsSingle.

Although there is some small unit test coverage of the fixup steps here,
the primary testing for this is in the test provider since the integration
of all of these fixup steps in the correct order is the more important
result than any of the intermediate fixup steps.
2019-03-14 15:36:15 -07:00
James Bardin e080706e2e treat normalization during ReadResource like Plan
This will allow resources to return an unexpected change to set blocks
and attributes, otherwise we could mask these changes during
normalization.

Change the "plan" argument in normalizeNullValues to "preferDst" to more
accurately describe what the option is doing, since it no longer applies
only to PlanResourceChange.
2019-03-13 19:14:17 -04:00
James Bardin 6ecf9b143b we can normalize nulls in Read again
This should be the final change from removing the flatmap normalization.
Since we're no longer trying to a consistent zero or null value in the
flatmap config, rather we're trying to maintain the previously applied
value, ReadResource also needs to apply the normalizeNullValues step in
order to prevent unexpected diffs.
2019-03-12 16:00:25 -04:00
James Bardin 11ec3a420e remove normalizeFlatmapContainers
This method was added early on when the diff was being applied as the
legacy code would have done, which is no longer the case. Everything
that normalizeFlatmapContainers does should be covered by the
combination of the initial diff.Apply and the normalizeNullValues on the
final cty.Value.
2019-03-12 12:04:35 -04:00
James Bardin 9d4bb6ec14 stop removing empty flatmap containers
As we've improved the cty.Value normalization, we need to remove
normalization procedures from the flatmap handling. Keeping the empty
containers in the flatmap will prevent unexpected nils from being added
to some schema configurations
2019-03-11 15:14:29 -04:00
James Bardin 6cdf9ff566 Revert "normalize all objects read from the provider"
This reverts commit 209a0a460a.
2019-03-08 17:32:37 -05:00
James Bardin 209a0a460a normalize all objects read from the provider
Use objchange.NormalizeObjectFromLegacySDK to ensure that all objects
returned from the provider match what is expected based on the
configuration according to the schemas.
2019-03-06 14:09:04 -05:00
James Bardin 3600f59bb7
Merge pull request #20525 from hashicorp/jbardin/extra-set-value
remove the partially-known ~ set sigil in diffs
2019-03-05 16:50:02 -05:00
James Bardin 2b4d030a69 don't re-add removed list values even when planned
Providers were not strict (and were not forced to be) about customizing
the diff when a computed attribute needed to be updated during apply.
The fix we have in place to prevent loss of information during the
helper/schema apply process would add in single missing value back in.

The first place this was caught was when we attempt to fix up the
flatmapped attributes. The 1->0 count error is now better handled by our
cty.Value normalization step, so we can remove the special apply case
here altogether

The next place is in normalizeNullValues, and since the intent was to
re-insert missing zero-value lists and sets, adding a check for a length
of 0 protects us from adding in extra elements.

The new test fixture emulated common provider behavior of re-computing
values without customizing the diff. Since we can work around it, and
core will provider appropriate warnings, the shims should try to
maintain the legacy behavior.
2019-03-05 15:31:08 -05:00
James Bardin 47604c36c8 remove the partially-known ~ set sigil in diffs
The NewExtra values are stored outside the diff from plan, and the
original keys may not contain the ~ prefix. Adding the NewExtra back
into the diff with the mismatched key was causing an entire new set
element to be populated. Since this symbol isn't used to apply the diff
in helper/schema, we can simply strip them out.
2019-03-04 17:36:30 -05:00
James Bardin 33d5ddf291 remove empty timeouts blocks in copyTimeoutValues
The hcl2shims will always add in the timeouts block, because there's no
way to differentiate a null single block from an empty one in the
flatmapped state. Since we are only concerned with keeping the prior
timeouts value, always set the new value to null, and then copy over the
prior value if it exists.
2019-03-02 11:30:37 -05:00
James Bardin 49230f8198 existing fields cannot become computed during plan
Fields with no change can only become computed during initial creation.
2019-02-28 18:45:11 -05:00
James Bardin 9a39af5047 1->0 set changes non longer should happen in Read
The new normalization should make preventing those changes unnecessary,
and will also prevent extra empty elements from being added when
resources are refreshed.
2019-02-28 17:47:11 -05:00
James Bardin f9b62cb5fe
Merge pull request #20335 from hashicorp/jbardin/diff-apply
Diff apply needs to check for both types of containers keys
2019-02-13 19:33:34 -05:00
James Bardin c34c37fbd5 missed .% suffixes in diff.Apply
Diff.Apply checks for unneeded container count diffs, but was missing
the check for maps.

Add an early return for planning a destroy.
2019-02-13 19:09:46 -05:00
Martin Atkins fedbd6c3b8 helper/plugin: fix panic with empty objects in normalizeNullValues
cty.Value.AsValueMap can return nil if called on an empty map or object.
The logic above was dealing with that case for maps, but object types
were falling through into this codepath and panicking when trying to
assign a new key into the nil dstMap.

This also includes a bonus fix where we were calling ty.ElementType in
a switch case that accepts object types. Object types don't have a single
element type, so we can't call ElementType on those (that also panics)
but we _can_ use the type of the value we selected from src to construct
our placeholder null value.
2019-02-13 15:56:12 -08:00
Martin Atkins eb1346447f
Merge #20282: Enforce expected behaviors for provider PlanResourceChange
An exception remains for the legacy SDK, which does not meet all of these requirements.
2019-02-12 09:19:05 -08:00
Martin Atkins 31299e688d core: Allow legacy SDK to opt out of plan-time safety checks
Due to the inprecision of our shimming from the legacy SDK type system to
the new Terraform Core type system, the legacy SDK produces a number of
inconsistencies that produce only minor quirky behavior or broken
edge-cases. To retain compatibility with those existing weird behaviors,
the legacy SDK opts out of our safety checks.

The intent here is to allow existing providers to continue to do their
previous unsafe behaviors for now, accepting that this will allow certain
quirky bugs from previous releases to persist, and then gradually migrate
away from the legacy SDK and remove this opt-out on a per-resource basis
over time.

As with the apply-time safety check opt-out, this is reserved only for
the legacy SDK and must not be used in any new SDK implementations. We
still include any inconsistencies as warnings in the logs as an aid to
anyone debugging weird behavior, so that they can see situations where
blame may be misplaced in the user-visible error messages.
2019-02-11 17:26:49 -08:00
James Bardin 1bfc27817e process state even after provider.Apply errors
Terraform core expects a sane state even when the provider returns an
error. Make sure at the prior state is always the default value to
return, and then alway attempt to process any state returned by
provider.Apply.
2019-02-11 15:41:07 -05:00