Commit Graph

100 Commits

Author SHA1 Message Date
Martin Atkins 8112f589c1 configs/configupgrade: Pass through connection and provisioner blocks
This is a temporary implementation of these rules just so that these can
be passed through verbatim (rather than generating an error) while we
do testing of other features.

A subsequent commit will finish these with their own custom rulesets.
2018-12-05 10:25:01 -08:00
Martin Atkins 028b5ba34e configs/configupgrade: Upgrade depends_on in resources and outputs 2018-12-05 10:25:01 -08:00
Martin Atkins ef017345f1 configs/configupgrade: Upgrade the resource "lifecycle" nested block
The main tricky thing here is ignore_changes, which contains strings that
are better given as naked traversals in 0.12. We also handle here mapping
the old special case ["*"] value to the new "all" keyword.
2018-12-05 10:25:01 -08:00
Martin Atkins 4b52148262 configs/configupgrade: Upgrade provider addresses
Both resource blocks and module blocks contain references to providers
that are expressed as short-form provider addresses ("aws.foo" rather than
"provider.aws.foo").

These rules call for those to be unwrapped as naked identifiers during
upgrade, rather than appearing as quoted strings. This also introduces
some further rules for other simpler meta-arguments that are required
for the test fixtures for this feature.
2018-12-05 10:25:01 -08:00
Martin Atkins ea3b8b364c configs/configupgrade: Initial passthrough mapping for module blocks
Some further rules are required here to deal with the meta-arguments we
accept inside these blocks, but this is good enough to pass through most
module blocks using the standard attribute-expression-based mapping.
2018-12-05 10:25:01 -08:00
Martin Atkins 4b5d31d35d configs/configupgrade: Rules-based upgrade for "locals" block
Previously we were handling this one as a special case, effectively
duplicating most of the logic from upgradeBlockBody.

By doing some prior analysis of the block we can produce a "rules" that
just passes through all of the attributes as-is, allowing us to reuse
upgradeBlockBody. This is a little weird for the locals block since
everything in it is user-selected names, but this facility will also be
useful in a future commit for dealing with module blocks, which contain
a mixture of user-chosen and reserved argument names.
2018-12-05 10:25:01 -08:00
Martin Atkins 76221a3a4a configs/configupgrade: Retain any .tf.json files unchanged
We don't change JSON files at all and instead just emit a warning about
them since JSON files are usually generated rather than hand-written and
so any updates need to happen in the generator program rather than in its
output.

However, we do still need to copy them verbatim into the output map so
that we can keep track of them through any subsequent steps.
2018-12-04 11:37:39 -08:00
Martin Atkins 8490fc36f7 configs/configupgrade: Fix up references to counted/uncounted resources
Prior to v0.12 Terraform was liberal about these and allowed them to
mismatch, but now it's important to get this right so that resources
and resource instances can be used directly as object values, and so
we'll fix up any sloppy existing references so things keep working as
expected.

This is particularly important for the pattern of using count to create
conditional resources, since previously the "true" case would create one
instance and Terraform would accept an unindexed reference to that.
2018-12-04 11:37:39 -08:00
Martin Atkins ceedeb69a9 configs/configupgrade: Normalize and upgrade reference expressions
The reference syntax is not significantly changed, but there are some
minor additional restrictions on identifiers in HCL2 and as a special case
we need to rewrite references to data.terraform_remote_state .

Along with those mandatory upgrades, we will also switch references to
using normal index syntax where it's safe to do so, as part of
de-emphasizing the old strange integer attribute syntax (like foo.0.bar).
2018-12-04 11:37:39 -08:00
Martin Atkins e83976c008 configs/configupgrade: Print trailing comments inside blocks
Previously we were erroneously moving these out of their original block
into the surrounding body. Now we'll make sure to collect up any remaining
ad-hoc comments inside a nested block body before closing it.
2018-12-04 11:37:39 -08:00
Martin Atkins de0eb9ea30 configs/configupgrade: Distinguish data resources in analysis
Early on it looked like we wouldn't need to distinguish these since we
were only analyzing for provider types, but we're now leaning directly
on the resource addresses later on and so we need to make sure we produce
valid ones when data resources are present.
2018-12-04 11:37:39 -08:00
Martin Atkins 6596d031d9 configs/configupgrade: Convert block-as-attr to dynamic blocks
Users discovered that they could exploit some missing validation in
Terraform v0.11 and prior to treat block types as if they were attributes
and assign dynamic expressions to them, with some significant caveats and
gotchas resulting from the fact that this was never intended to work.

However, since such patterns are in use in the wild we'll convert them
to a dynamic block during upgrade. With only static analysis we must
unfortunately generate a very conservative, ugly dynamic block with
every possible argument set. Users ought to then clean up the generated
configuration after confirming which arguments are actually required.
2018-12-04 11:37:39 -08:00
Martin Atkins f96d702d4f configs/configupgrade: Upgrading of simple nested blocks 2018-12-04 11:37:39 -08:00
Martin Atkins 39c3e7112f configs/configupgrade: Use break to cancel default function rendering
We're using break elsewhere in here so it was weird to have a small set
of situations that return instead, which could then cause confusion for
future maintenance if a reader doesn't notice that control doesn't always
leave the outer switch statement.
2018-12-04 11:37:39 -08:00
Martin Atkins 48f1245e6b configs/configupgrade: Replace lookup(...) with index syntax
If lookup is being used with only two arguments then it is equivalent to
index syntax and more readable that way, so we'll replace it.

Ideally we'd do similarly for element(...) here but sadly we cannot
because we can't prove in static analysis that the user is not relying
on the modulo wraparound behavior of that function.
2018-12-04 11:37:39 -08:00
Martin Atkins 4927a4105b configs/configupgrade: Replace calls to list() and map()
We now have native language features for declaring tuples and objects,
which are the idiomatic way to construct sequence and mapping values that
can then be converted to list, set, and map types as needed.
2018-12-04 11:37:39 -08:00
Martin Atkins 8cf024d45a configs/configupgrade: Upgrade expressions nested in HCL list/object
In the old world, lists and maps could be created either using functions
in HIL or list/object constructs in HCL. Here we ensure that in the HCL
case we'll apply any required expression transformations to the individual
items within HCL's compound constructs.
2018-12-04 11:37:39 -08:00
Martin Atkins e49d993d89 configs/configupgrade: Decide on blank lines by ends of items
Previously we were using the line count difference between the start of
one item and the next to decide whether to insert a blank line between
two items, but that is incorrect for multi-line items.

Instead, we'll now count the difference from the final line of the
previous item to the first line of the next, as best we can with the
limited position info recorded by the HCL1 parser.
2018-12-04 11:37:39 -08:00
Martin Atkins bdb724562c configs/configupgrade: Test that map attrs as blocks are fixed
The old parser was forgiving in allowing the use of block syntax where a
map attribute was expected, but the new parser is not (in order to allow
for dynamic map keys, for expressions, etc) and so the upgrade tool must
fix these to use attribute syntax.
2018-12-04 11:37:39 -08:00
Martin Atkins 1aa368d0d8 configs/configupgrade: Add some logging and enable it for tests 2018-12-04 11:37:39 -08:00
Martin Atkins e8999eefdc configs/configupgrade: Populate the test provider schema
This will allow us to test some schema-sensitive migration rules.
2018-12-04 11:37:39 -08:00
Martin Atkins 8e594f32aa configs/configupgrade: Upgrade rules for the "terraform" block type
This includes the backend configuration, so we need to load the requested
backend and migrate the included configuration per that schema.
2018-12-04 11:37:39 -08:00
Martin Atkins 302b29557f configs/configupgrade: Pass through diagnostics from body upgrades 2018-12-04 11:37:39 -08:00
Martin Atkins c755745285 configs/configupgrade: Generalize migration of block bodies
The main area of interest in upgrading is dealing with special cases for
individual block items, so this generalization allows us to use the same
overall body-processing logic for everything but to specialize just how
individual items are dealt with, which we match by their names as given
in the original input source code.
2018-12-04 11:37:39 -08:00
Martin Atkins cf52e224f6 configs/configupgrade: Basic migration of provider blocks
This involved some refactoring of how block bodies are migrated, which
still needs some additional work to deal with meta-arguments but is now
at least partially generalized to support both resource and provider
blocks.
2018-12-04 11:37:39 -08:00
Radek Simko 9e5677adeb
Merge pull request #19453 from hashicorp/b-whole-body-diag-rendering
command/format: Fix rendering of attribute-agnostic diagnostics
2018-11-26 23:52:21 +00:00
Radek Simko 501fcd1e21
Fix tests after upgrading hcl 2018-11-26 23:38:37 +00:00
Martin Atkins bbbf22d8e4 configs/configschema: Block.StaticValidateTraversal method
This allows basic static validation of a traversal against a schema, to
verify that it represents a valid path through the structural parts of
the schema.

The main purpose of this is to produce better error messages (using our
knowledge of the schema) than we'd be able to achieve by just relying
on HCL expression evaluation errors. This is particularly important for
nested blocks because it may not be obvious whether one is represented
internally by a set or a list, and incorrect usage would otherwise produce
a confusing HCL-oriented error message.
2018-11-26 08:25:03 -08:00
Martin Atkins 0681935df5 configs: Reserve various names for future use
We want the forthcoming v0.12.0 release to be the last significant
breaking change to our main configuration constructs for a long time, but
not everything could be implemented in that release.

As a compromise then, we reserve various names we have some intent of
using in a future release so that such future uses will not be a further
breaking change later.

Some of these names are associated with specific short-term plans, while
others are reserved conservatively for possible later work and may be
"un-reserved" in a later release if we don't end up using them. The ones
that we expect to use in the near future were already being handled, so
we'll continue to decode them at the config layer but also produce an
error so that we don't get weird behavior downstream where the
corresponding features don't work yet.
2018-11-26 08:25:03 -08:00
James Bardin ebc9745788 fix "too many items" error message 2018-11-13 18:41:53 -05:00
Martin Atkins f001cb8654 command: Fix terraform init -from-module with relative paths
Since our new approach here works by installing with a synthetic module
configuration block, we need to treat relative paths as a special case
for two reasons:

- Relative paths in module addresses are relative to the file containing
  the call rather than the working directory, but -from-module uses the
  working directory (and the call is in a synthetic "file" anyway)

- We need to force Terraform to pass the path through to go-getter rather
  than just treating it as a relative reference, since we really do want
  a copy of the directory in this case, even if it is local.

To address both of these things, we'll detect a relative path and turn it
into an absolute path before beginning installation. This is a bit hacky,
but this is consistent with the general philosophy of the -from-module
implementation where it does hacky things so that the rest of the
installer code can be spared of dealing with its special cases.

This is covered by a couple of existing tests that run init -from-module,
including TestInit_fromModule_dstInSrc which now passes.
2018-11-09 09:48:03 -08:00
James Bardin b3a491d035 use correct block types in CoerceValue
When creating Null or Unknown values during CoerceValue, the the outer
block type was being used rather than the current block type.
2018-10-18 18:09:43 -04:00
Martin Atkins 1a654e9e1c configs/configupgrade: Disable the tests for now
The tests in here are illustrating that this package is not yet finished,
but we plan to run a release before we finish this and so we'll skip those
tests for now with the intent of reinstating this again once we return
to finish this up.
2018-10-16 19:14:11 -07:00
Martin Atkins 741d334ee4 command: Even more fixes for "apply" command tests 2018-10-16 19:14:11 -07:00
Martin Atkins 961056c08d configs/configupgrade: Use mock provider instead of test provider
The test provider comes with a lot of baggage since it's designed to be
used as a plugin, so instead we'll just use the mock provider
implementation directly, and so we can (in a later commit) configure it
appropriately for what our tests need here.
2018-10-16 19:14:11 -07:00
Martin Atkins 85aa8769db configs/configupgrade: Partially fix TestUpgradeValid
This is still not compileable because the test provider needs to be
updated to the new provider interface, but all the rest of the types are
now correct so we can update the test provider in a later commit to make
this work again.
2018-10-16 19:14:11 -07:00
Martin Atkins 7d0be84300 configload: Don't fail if one module is a child directory of another
Given a module foo and a module foo/bar, the previous code might
incorrectly treat "bar" as a file within "foo" rather than as a module
directory in its own right.
2018-10-16 19:14:11 -07:00
Martin Atkins 44bc7519a6 terraform: More wiring in of new provider types
This doesn't actually work yet, but it builds and then panics in a pretty
satisfying way.
2018-10-16 19:12:54 -07:00
Martin Atkins 549544f201 configschema: Handle nested blocks containing dynamic-typed attributes
We need to make the collection itself be a tuple or object rather than
list or map in this case, since otherwise all of the elements of the
collection are constrained to be of the same type and that isn't the
intent of a provider indicating that it accepts any type.
2018-10-16 19:11:09 -07:00
Martin Atkins 0317da9911 plans/objchange: logic for merging prior state with config
This produces a "proposed new state", which already has prior computed
values propagated into it (since that behavior is standard for all
resource types) but could be customized further by the provider to make
the "_planned_ new state".

In the process of implementing this it became clear that our configschema
DecoderSpec behavior is incorrect, since it's producing list values for
NestingList and map values for NestingMap. While that seems like it should
be right, we should actually be using tuple and object types respectively
to allow each block to have a different runtime type in situations where
an attribute is given the type cty.DynamicPseudoType. That's not fixed
here, and so without a further fix list and map blocks will panic here.
The DecoderSpec implementation will be fixed in a subsequent commit.
2018-10-16 19:11:09 -07:00
Martin Atkins a3403f2766 terraform: Ugly huge change to weave in new State and Plan types
Due to how often the state and plan types are referenced throughout
Terraform, there isn't a great way to switch them out gradually. As a
consequence, this huge commit gets us from the old world to a _compilable_
new world, but still has a large number of known test failures due to
key functionality being stubbed out.

The stubs here are for anything that interacts with providers, since we
now need to do the follow-up work to similarly replace the old
terraform.ResourceProvider interface with its replacement in the new
"providers" package. That work, along with work to fix the remaining
failing tests, will follow in subsequent commits.

The aim here was to replace all references to terraform.State and its
downstream types with states.State, terraform.Plan with plans.Plan,
state.State with statemgr.State, and switch to the new implementations of
the state and plan file formats. However, due to the number of times those
types are used, this also ended up affecting numerous other parts of core
such as terraform.Hook, the backend.Backend interface, and most of the CLI
commands.

Just as with 5861dbf3fc49b19587a31816eb06f511ab861bb4 before, I apologize
in advance to the person who inevitably just found this huge commit while
spelunking through the commit history.
2018-10-16 19:11:09 -07:00
Kristin Laemmert d1f27ce8c2 registry: renaming module-specific registry functions 2018-10-16 18:56:50 -07:00
James Bardin 8f295fcb22 CoerceValue should insert Null for unset attrs
This matches the decoder spec, where a value unset in the configuration
is always Null.
2018-10-16 18:53:51 -07:00
James Bardin 50e099ad10 don't set Unknown for computed values
Any value that is not set in the configuration should decode as a Null
value. Unknown should still be returned if a computed value expression
is unknown.
2018-10-16 18:53:51 -07:00
Martin Atkins d8bf3cc4e0 configschema: Fix ConfigSchema bugs with nested blocks
We were iterating over the wrong value to recursively coerce content for
nested blocks, and also incorrectly constructing the cty.Path used in
errors.
2018-10-16 18:53:51 -07:00
Martin Atkins 479c6b2466 move "configschema" from "config" to "configs"
The "config" package is no longer used and will be removed as part
of the 0.12 release cleanup. Since configschema is part of the
"new world" of configuration modelling, it makes more sense for
it to live as a subdirectory of the newer "configs" package.
2018-10-16 18:50:29 -07:00
Martin Atkins adb88eaa16 configupgrade: Analysis of input configuration
In order to properly migrate the contents of resource, data, provider and
provisioner blocks we will need the provider's schema in order to
understand what is expected, so we can resolve some ambiguities inherent
in the legacy HCL AST.

This includes an initial prototype of migrating the content of resource
blocks just to verify that the information is being gathered correctly.
As with the rest of the upgrade_native.go file, this will be reorganized
significantly once the basic end-to-end flow is established and we can
see how to organize this code better.
2018-10-16 18:50:29 -07:00
Martin Atkins f77e7a61b0 various: helpers for collecting necessary provider types
Since schemas are required to interpret provider, resource, and
provisioner attributes in configs, states, and plans, these helpers intend
to make it easier to gather up the the necessary provider types in order
to preload all of the needed schemas before beginning further processing.

Config.ProviderTypes returns directly the list of provider types, since
at this level further detail is not useful: we've not yet run the
provider allocation algorithm, and so the only thing we can reliably
extract here is provider types themselves.

State.ProviderAddrs and Plan.ProviderAddrs each return a list of
absolute provider addresses, which can then be turned into a list of
provider types using the new helper providers.AddressedTypesAbs.

Since we're already using configs.Config throughout core, this also
updates the terraform.LoadSchemas helper to use Config.ProviderTypes
to find the necessary providers, rather than implementing its own
discovery logic. states.State is not yet plumbed in, so we cannot yet
use State.ProviderAddrs to deal with the state but there's a TODO comment
to remind us to update that in a later commit when we swap out
terraform.State for states.State.

A later commit will probably refactor this further so that we can easily
obtain schema for the providers needed to interpret a plan too, but that
is deferred here because further work is required to make core work with
the new plan types first. At that point, terraform.LoadSchemas may become
providers.LoadSchemas with a different interface that just accepts lists
of provider and provisioner names that have been gathered by the caller
using these new helpers.
2018-10-16 18:50:29 -07:00
Martin Atkins ffe5f7c4e6 command: 0.12upgrade command
This is the frontend to the work-in-progress codepath for upgrading the
source code for a module written for Terraform v0.11 or earlier to use
the new syntax and idiom of v0.12.

The underlying upgrade code is not yet complete as of this commit, and
so the command is not yet very useful. We will continue to iterate on
the upgrade code in subsequent commits.
2018-10-16 18:50:29 -07:00
Martin Atkins 95b7b883a3 configupgrade: Basic expression formatting
This covers all of the expression node types in HIL's AST, and also
includes initial support for some of our top-level blocks so that we can
easily test that.

The initial implementations of the "variable" and "output" blocks are
pretty redundant and messy, so we can hopefully improve on these in a
later pass.
2018-10-16 18:50:29 -07:00