Commit Graph

65 Commits

Author SHA1 Message Date
Radek Simko 42ba7a3e00
configupgrade: Upgrade indexing of splat syntax 2019-04-26 23:27:32 +01:00
Radek Simko 1f5cadeec0
0.12upgrade: Return error for invalid reference 2019-04-26 23:27:31 +01:00
Martin Atkins d7f23f0beb configs/configupgrade: Don't panic if analyzer fails
Previously we were trying to access a field of the analysis object before
checking if analysis produced errors. The analysis function usually
returns a nil analysis on error, so this would result in a panic whenever
that happened.

Now we'll dereference the analysis object pointer only after checking for
errors, so we'll get a chance to report the analysis error to the user.
2019-04-17 10:08:54 -07:00
Martin Atkins 1bb47ab9a5 configs/configupgrade: Preserve comments on items in object exprs
The expression upgrade functionality mostly ignores comments because in
the old language the syntax prevented comments from appearing in the
middle of expressions, but there was one exception: object expressions.

Because HCL 1 used ObjectType both for blocks and for object expressions,
that is the one situation where something we consider to be an expression
could have inline attached comments in the old language.

We migrate these here so we don't lose these comments that don't appear
anywhere else. Other comments get gathered up into a general comments
set maintained inside the analysis object and so will be printed out as
required _between_ expressions, just as they did before.
2019-04-17 07:48:57 -07:00
Martin Atkins 88e76fa9ef configs/configschema: Introduce the NestingGroup mode for blocks
In study of existing providers we've found a pattern we werent previously
accounting for of using a nested block type to represent a group of
arguments that relate to a particular feature that is always enabled but
where it improves configuration readability to group all of its settings
together in a nested block.

The existing NestingSingle was not a good fit for this because it is
designed under the assumption that the presence or absence of the block
has some significance in enabling or disabling the relevant feature, and
so for these always-active cases we'd generate a misleading plan where
the settings for the feature appear totally absent, rather than showing
the default values that will be selected.

NestingGroup is, therefore, a slight variation of NestingSingle where
presence vs. absence of the block is not distinguishable (it's never null)
and instead its contents are treated as unset when the block is absent.
This then in turn causes any default values associated with the nested
arguments to be honored and displayed in the plan whenever the block is
not explicitly configured.

The current SDK cannot activate this mode, but that's okay because its
"legacy type system" opt-out flag allows it to force a block to be
processed in this way anyway. We're adding this now so that we can
introduce the feature in a future SDK without causing a breaking change
to the protocol, since the set of possible block nesting modes is not
extensible.
2019-04-10 14:53:52 -07:00
Martin Atkins 26c1e40ad7 configs/configupgrade: Normalize number literals to decimal
The v0.12 language supports numeric constants only in decimal notation, as
a simplification. For rare situations where a different base is more
appropriate, such as unix-style file modes, we've found it better for
providers to accept a string containing a representation in the
appropriate base, since that way the interpretation can be validated and
it will be displayed in the same way in the rendered plan diff, in
outputs, etc.

We use tv.Value() here to mimick how HCL 1 itself would have interpreted
these, and then format them back out in the canonical form, which
implicitly converts any non-decimal constants to decimal on the way
through.
2019-04-04 18:09:44 -07:00
Kristin Laemmert 3aa4ac43f9
configs/configupgrade: return if hil.Parse() produces an error. (#20920)
Fixes #20917
2019-04-03 14:20:59 -04:00
Martin Atkins 6c5819f910 configs/configupgrade: Prefer block syntax for list-of-object attributes
In order to preserve pre-v0.12 idiom for list-of-object attributes, we'll
prefer to use block syntax for them except for the special situation where
the user explicitly assigns an empty list, where attribute syntax is
required in order to allow existing provider logic to differentiate from
an implicit lack of blocks.
2019-04-01 13:30:24 -07:00
Kristin Laemmert da52170797
configupgrade: fix test (#20863) 2019-03-28 18:00:33 -04:00
Kristin Laemmert 1baa1b907e
configs/configupgrade: detect invalid resource names and print a TODO (#20856)
* configs/configupgrade: detect invalid resource names and print a TODO
message

In terraform 0.11 and prior it was possible to start a resource name
with a number. This is no longer valid, as the resource name would would
be ambiguous with number values in HCL expressions.

Fixes #19919

* Update configs/configupgrade/test-fixtures/valid/invalid-resource-name/want/resource.tf

Co-Authored-By: mildwonkey <mildwonkey@users.noreply.github.com>
2019-03-28 13:48:35 -04:00
Martin Atkins ea1d5f8fcb vendor: go get github.com/hashicorp/hcl2@master
This includes two upstream fixes:

- Handle explicit JSON "null" consistently during decode of JSON syntax.
- Properly detect the end of a "heredoc" when formatting to avoid messing
  up indentation of other lines following the heredoc.
2019-03-15 13:55:30 -07:00
Kristin Laemmert a15a4acf2f
configs/configupgrade: detect possible relative module sources (#20646)
* configs/configupgrade: detect possible relative module sources

If a module source appears to be a relative local path but does not have
a preceding ./, print a #TODO message for the user.

* internal/initwd: limit go-getter detectors to those supported by terraform
* internal/initwd: move isMaybeRelativeLocalPath check into getWithGoGetter

To avoid making two calls to getter.Detect, which potentially makes
non-trivial API calls, the "isMaybeRelativeLocalPath" check was moved to
a later step and a custom error type was added so user-friendly
diagnostics could be displayed in the event that a possible relative local
path was detected.
2019-03-13 11:17:14 -07:00
Martin Atkins b217624d83 config/configupgrade: Test to show that list unwrapping works for sets
This was already working but we had no tests to prove it.
2019-02-22 17:40:40 -08:00
Martin Atkins dd43926761 configs/configupgrade: Fix up uses of the .count pseudo-attribute
Terraform 0.11 and prior had an odd special case where a resource
attribute access for "count" would be resolved as the count for the
whole resource, rather than as an attribute of an individual instance as
for all other attributes.

Because Terraform 0.12 makes test_instance.foo appear as a list when count
is set (so it can be used in other expressions), it's no longer possible
to have an attribute in that position: lists don't have attributes.
Fortunately we don't really need that special case anymore since it
doesn't do anything we can't now do with the length(...) function.

This upgrade rule, then, detects references like test_instance.foo.count
and rewrites to length(test_instance.foo). As a special case, if
test_instance.foo doesn't have "count" set then it just rewrites as the
constant 1, which mimics what would've happened in that case in Terraform
0.11.
2019-02-22 16:18:53 -08:00
Martin Atkins 966eb39427 configs/configupgrade: Default arguments in "connection" blocks
Prior to Terraform v0.12 it was possible for a provider to secretly set
some default arguments for the "connection" block, which most commonly
included a hard-coded type of "ssh" and a value from "host".

In the interests of "explicit is better than implicit", Terraform 0.12 no
longer has this feature and instead requires connection settings to be
written explicitly in terms of the resource's exported attributes. For
compatibility though, the upgrade tool will insert expressions that are
as close as possible to the logic the provider formerly implemented, or
in a few rare cases a TF-UPGRADE-TODO comment to fix it up manually.

Some of the existing resource type implementations have incredibly
complicated implementations of selecting a single host IP address to use
and don't expose the result of that as an attribute, so for now we handle
those via a complicated Terraform language expression achieving the same
result. Ideally these providers would introduce a new attribute that
exports the same address formerly exported as the hostname before their
initial v0.12-compatible release, in which case we can simplify these to
just reference the attribute in question. That would be preferable also
because it would allow use of that exported attribute in other contexts,
such as in a null_resource provisioner somewhere else or in an output
to allow a caller to deal with the SSH part itself.
2019-02-22 12:32:56 -08:00
Martin Atkins ac6e0e42dc configs/configupgrade: Upgrade the bodies of "connection" blocks
This uses the fixed "superset" schema from the main terraform package to
apply our standard expression mapping, with the exception of "type" where
interpolation sequences are not supported due to the type being evaluated
early to retrieve the schema for decoding the rest.
2019-02-22 12:32:56 -08:00
Martin Atkins e2ef51800a configs/configupgrade: Upgrade the bodies of "provisioner" blocks
Aside from the two special meta-arguments "connection" and "provisioner"
this is just our standard mapping from schema to conversion rules, using
the provisioner's configuration schema.
2019-02-22 12:32:56 -08:00
Martin Atkins cdca8fbfe8 configs/configupgrade: Correct ignore_changes error message
Due to a copy-paste error, this was using the message from the providers
map in a "module" block.

This new message is not particularly helpful, but we should only see it
for a configuration that wouldn't have been valid in 0.11 either, and so
it's unlikely to be displayed.
2019-02-22 12:32:56 -08:00
Martin Atkins 0095a944cd configs/configupgrade: Include provisioner schemas in analysis
We'll need these to migrate any "provisioner" blocks we find in the input
configuration.
2019-02-22 12:32:56 -08:00
Martin Atkins fa0d6484df configs/configupgrade: Detect and fix element(...) usage with sets
Although sets do not have indexed elements, in Terraform 0.11 and earlier
element(...) would work with sets because we'd automatically convert them
to lists on entry to HIL -- with an arbitrary-but-consistent ordering --
and this return an arbitrary-but-consistent element from the list.

The element(...) function in Terraform 0.12 does not allow this because it
is not safe in general, but there was an existing pattern relying on this
in Terraform 0.11 configs which this upgrade rule is intended to preserve:

    resource "example" "example" {
      count = "${length(any_set_attribute)}"

      foo = "${element(any_set_attribute, count.index}"
    }

The above works because the exact indices assigned in the conversion are
irrelevant: we're just asking Terraform to create one resource for each
distinct element in the set.

This upgrade rule therefore inserts an explicit conversion to list if it
is able to successfully provide that the given expression will return a
set type:

    foo = "${element(tolist(any_set_attribute), count.index}"

This makes the conversion explicit, allowing users to decide if it is
safe and rework the configuration if not. Since our static type analysis
functionality focuses mainly on resource type attributes, in practice this
rule will only apply when the given expression is a statically-checkable
resource reference. Since sets are an SDK-only concept in Terraform 0.11
and earlier anyway, in practice that works out just right: it's not
possible for sets to appear anywhere else in older versions anyway.
2019-02-21 09:39:55 -08:00
Martin Atkins 085ac6d8ca configs/configupgrade: Test for removing commas between block arguments
The comma-separated syntax is now reserved only for object constructor
expressions in attribute values, so the upgrade tool rewrites block
arguments to be newline-separated instead.

This was already working but we didn't have an explicit test for it until
now.
2019-02-20 16:11:14 -08:00
Martin Atkins 54bb0b1e25 configs/configupgrade: Silently ignore and trim .% .# in ignore_changes
Prior to Terraform 0.12, ignore_changes was implemented in a
flatmap-oriented fashion and so users found that they could (and in fact,
were often forced to) use the internal .% and .# suffixes flatmap uses to
ignore changes to the number of elements in a list or map.

Terraform 0.12 no longer uses that representation, so we'll interpret
ignoring changes to the length as ignoring changes to the entire
collection. While this is not a totally-equivalent change, in practice
this pattern was most often used in conjunction with specific keys from a
map in order to _effectively_ ignore the entire map, even though Terraform
didn't really support that.
2019-02-20 15:58:44 -08:00
Martin Atkins 1d35233a03 configs/configupgrade: Fix up internal HIL conversion functions
HIL implemented its type conversions by rewriting its AST to include calls
to some undocumented builtin functions. Unfortunately those functions were
still explicitly callable if you could figure out the name for them, and
so they may have been used in the wild.

In particular, __builtin_StringToFloat was used as part of a workaround
for a HIL design flaw where it would prefer to convert strings to integers
rather than floats when performing arithmetic operations. This issue was,
indeed, the main reason for unifying int ant float into a single number
type in HCL. Since we published that as a suggested workaround, the
upgrade tool ought to fix it up.

The other cases have never been documented as a workaround, so they are
less likely to appear in the wild, but we might as well fix them up anyway
since we already have the conversion functions required to get the same
result in the new language.

To be safe/conservative, most of these convert to _two_ function calls
rather than just one, which ensures that these new expressions retain the
behavior of implicitly converting to the source type before running the
conversion. The new conversion functions only specify target type, and so
cannot guarantee identical results if the argument type does not exactly
match what was previously given as the parameter type in HIL.
2019-02-20 14:01:53 -08:00
Martin Atkins 154911688a configs/configupgrade: upgrade expressions inside heredocs
HEREDOC tokens are a little more fussy than normal string sequences
because we need to preserve the whitespace within them along with the
start and end markers while we upgrade any interpolated expressions inside.

We need to do some work locally here because the HCL heredoc processing
"does too much" and throws away information we need to do a faithful
upgrade.

We also need to contend with the fact that Terraform <=0.11 had an older
version of HCL that accidentally permitted a degenerate form of heredoc
where the marker was at the end of the final line, like this:

    degenerate = <<EOT
    this should never have workedEOT

When we migrate this, we'll introduce the additional newline that is now
required, which will unfortunately slightly change the result string to
include a newline when parsed by 0.12, and so we'll need to call this out
as a caveat in the upgrade guide.
2019-02-20 12:56:44 -08:00
Kristin Laemmert 5f8916b4fd
configs/configupgrade: do not panic on HEREDOCs. (#20281)
Previously, configupgrade would panic if it encountered a HEREDOC. For
the time being, we will simply print out the HEREDOC as-is.

Unfortunately, we discovered that terraform 0.11's version of HCL
allowed for HEREDOCs with the termination delimiter inline (instead of
on a newline, which is technically correct). Since 0.12configupgrade
needs to be bug-compatible with terraform 0.11, we must roll back to the
same version of HCL used in terraform 0.11.
2019-02-08 15:51:53 -08:00
Martin Atkins 954d38e870 lang: New file-hashing functions
In prior versions, we recommended using hash functions in conjunction with
the file function as an idiom for detecting changes to upstream blobs
without fetching and comparing the whole blob.

That approach relied on us being able to return raw binary data from
file(...). Since Terraform strings pass through intermediate
representations that are not binary-safe (e.g. the JSON state), there was
a risk of string corruption in prior versions which we have avoided for
0.12 by requiring that file(...) be used only with UTF-8 text files.

The specific case of returning a string and immediately passing it into
another function was not actually subject to that corruption risk, since
the HIL interpreter would just pass the string through verbatim, but this
is still now forbidden as a result of the stricter handling of file(...).

To avoid breaking these use-cases, here we introduce variants of the hash
functions a with "file" prefix that take a filename for a disk file to
hash rather than hashing the given string directly. The configuration
upgrade tool also now includes a rule to detect the documented idiom and
rewrite it into a single function call for one of these new functions.

This does cause a bit of function sprawl, but that seems preferable to
introducing more complex rules for when file(...) can and cannot read
binary files, making the behavior of these various functions easier to
understand in isolation.
2019-01-25 10:18:44 -08:00
Martin Atkins f93f7e5b5c configs/configupgrade: Remove redundant list brackets
In early versions of Terraform where the interpolation language didn't
have any real list support, list brackets around a single string was the
signal to split the string on a special uuid separator to produce a list
just in time for processing, giving expressions like this:

    foo = ["${test_instance.foo.*.id}"]

Logically this is weird because it looks like it should produce a list
of lists of strings. When we added real list support in Terraform 0.7 we
retained support for this behavior by trimming off extra levels of list
during evaluation, and inadvertently continued relying on this notation
for correct type checking.

During the Terraform 0.10 line we fixed the type checker bugs (a few
remaining issues notwithstanding) so that it was finally possible to
use the more intuitive form:

    foo = "${test_instance.foo.*.id}"

...but we continued trimming off extra levels of list for backward
compatibility.

Terraform 0.12 finally removes that compatibility shim, causing redundant
list brackets to be interpreted as a list of lists.

This upgrade rule attempts to identify situations that are relying on the
old compatibility behavior and trim off the redundant extra brackets. It's
not possible to do this fully-generally using only static analysis, but
we can gather enough information through or partial type inference
mechanism here to deal with the most common situations automatically and
produce a TF-UPGRADE-TODO comment for more complex scenarios where the
user intent isn't decidable with only static analysis.

In particular, this handles by far the most common situation of wrapping
list brackets around a splat expression like the first example above.
After this and the other upgrade rules are applied, the first example
above will become:

    foo = test_instance.foo.*.id
2018-12-07 17:05:36 -08:00
Martin Atkins d9603d5bc5 configs/configupgrade: Do type inference with input variables
By collecting information about the input variables during analysis, we
can return approximate type information for any references to those
variables in expressions.

Since Terraform 0.11 allowed maps of maps and lists of lists in certain
circumstances even though this was documented as forbidden, we
conservatively return collection types whose element types are unknown
here, which allows us to do shallow inference on them but will cause
us to get an incomplete result if any operations are performed on
elements of the list or map value.
2018-12-07 17:05:36 -08:00
Martin Atkins a2d9634dbf configs/configupgrade: Expression type inference
Although we can't do fully-precise type inference with access only to a
single module's configuration, we can do some approximate inference using
some clues within the module along with our resource type schemas.

This depends on HCL's ability to pass through type information even if the
input values are unknown, mapping our partial input type information into
partial output type information by evaluating the same expressions.

This will allow us to do some upgrades that require dynamic analysis to
fully decide, by giving us three outcomes: needed, not needed, or unknown.
If it's unknown then that'll be our prompt to emit a warning for the user
to make a decision.
2018-12-07 17:05:36 -08:00
Martin Atkins 8112f589c1 configs/configupgrade: Pass through connection and provisioner blocks
This is a temporary implementation of these rules just so that these can
be passed through verbatim (rather than generating an error) while we
do testing of other features.

A subsequent commit will finish these with their own custom rulesets.
2018-12-05 10:25:01 -08:00
Martin Atkins 028b5ba34e configs/configupgrade: Upgrade depends_on in resources and outputs 2018-12-05 10:25:01 -08:00
Martin Atkins ef017345f1 configs/configupgrade: Upgrade the resource "lifecycle" nested block
The main tricky thing here is ignore_changes, which contains strings that
are better given as naked traversals in 0.12. We also handle here mapping
the old special case ["*"] value to the new "all" keyword.
2018-12-05 10:25:01 -08:00
Martin Atkins 4b52148262 configs/configupgrade: Upgrade provider addresses
Both resource blocks and module blocks contain references to providers
that are expressed as short-form provider addresses ("aws.foo" rather than
"provider.aws.foo").

These rules call for those to be unwrapped as naked identifiers during
upgrade, rather than appearing as quoted strings. This also introduces
some further rules for other simpler meta-arguments that are required
for the test fixtures for this feature.
2018-12-05 10:25:01 -08:00
Martin Atkins ea3b8b364c configs/configupgrade: Initial passthrough mapping for module blocks
Some further rules are required here to deal with the meta-arguments we
accept inside these blocks, but this is good enough to pass through most
module blocks using the standard attribute-expression-based mapping.
2018-12-05 10:25:01 -08:00
Martin Atkins 4b5d31d35d configs/configupgrade: Rules-based upgrade for "locals" block
Previously we were handling this one as a special case, effectively
duplicating most of the logic from upgradeBlockBody.

By doing some prior analysis of the block we can produce a "rules" that
just passes through all of the attributes as-is, allowing us to reuse
upgradeBlockBody. This is a little weird for the locals block since
everything in it is user-selected names, but this facility will also be
useful in a future commit for dealing with module blocks, which contain
a mixture of user-chosen and reserved argument names.
2018-12-05 10:25:01 -08:00
Martin Atkins 76221a3a4a configs/configupgrade: Retain any .tf.json files unchanged
We don't change JSON files at all and instead just emit a warning about
them since JSON files are usually generated rather than hand-written and
so any updates need to happen in the generator program rather than in its
output.

However, we do still need to copy them verbatim into the output map so
that we can keep track of them through any subsequent steps.
2018-12-04 11:37:39 -08:00
Martin Atkins 8490fc36f7 configs/configupgrade: Fix up references to counted/uncounted resources
Prior to v0.12 Terraform was liberal about these and allowed them to
mismatch, but now it's important to get this right so that resources
and resource instances can be used directly as object values, and so
we'll fix up any sloppy existing references so things keep working as
expected.

This is particularly important for the pattern of using count to create
conditional resources, since previously the "true" case would create one
instance and Terraform would accept an unindexed reference to that.
2018-12-04 11:37:39 -08:00
Martin Atkins ceedeb69a9 configs/configupgrade: Normalize and upgrade reference expressions
The reference syntax is not significantly changed, but there are some
minor additional restrictions on identifiers in HCL2 and as a special case
we need to rewrite references to data.terraform_remote_state .

Along with those mandatory upgrades, we will also switch references to
using normal index syntax where it's safe to do so, as part of
de-emphasizing the old strange integer attribute syntax (like foo.0.bar).
2018-12-04 11:37:39 -08:00
Martin Atkins e83976c008 configs/configupgrade: Print trailing comments inside blocks
Previously we were erroneously moving these out of their original block
into the surrounding body. Now we'll make sure to collect up any remaining
ad-hoc comments inside a nested block body before closing it.
2018-12-04 11:37:39 -08:00
Martin Atkins de0eb9ea30 configs/configupgrade: Distinguish data resources in analysis
Early on it looked like we wouldn't need to distinguish these since we
were only analyzing for provider types, but we're now leaning directly
on the resource addresses later on and so we need to make sure we produce
valid ones when data resources are present.
2018-12-04 11:37:39 -08:00
Martin Atkins 6596d031d9 configs/configupgrade: Convert block-as-attr to dynamic blocks
Users discovered that they could exploit some missing validation in
Terraform v0.11 and prior to treat block types as if they were attributes
and assign dynamic expressions to them, with some significant caveats and
gotchas resulting from the fact that this was never intended to work.

However, since such patterns are in use in the wild we'll convert them
to a dynamic block during upgrade. With only static analysis we must
unfortunately generate a very conservative, ugly dynamic block with
every possible argument set. Users ought to then clean up the generated
configuration after confirming which arguments are actually required.
2018-12-04 11:37:39 -08:00
Martin Atkins f96d702d4f configs/configupgrade: Upgrading of simple nested blocks 2018-12-04 11:37:39 -08:00
Martin Atkins 39c3e7112f configs/configupgrade: Use break to cancel default function rendering
We're using break elsewhere in here so it was weird to have a small set
of situations that return instead, which could then cause confusion for
future maintenance if a reader doesn't notice that control doesn't always
leave the outer switch statement.
2018-12-04 11:37:39 -08:00
Martin Atkins 48f1245e6b configs/configupgrade: Replace lookup(...) with index syntax
If lookup is being used with only two arguments then it is equivalent to
index syntax and more readable that way, so we'll replace it.

Ideally we'd do similarly for element(...) here but sadly we cannot
because we can't prove in static analysis that the user is not relying
on the modulo wraparound behavior of that function.
2018-12-04 11:37:39 -08:00
Martin Atkins 4927a4105b configs/configupgrade: Replace calls to list() and map()
We now have native language features for declaring tuples and objects,
which are the idiomatic way to construct sequence and mapping values that
can then be converted to list, set, and map types as needed.
2018-12-04 11:37:39 -08:00
Martin Atkins 8cf024d45a configs/configupgrade: Upgrade expressions nested in HCL list/object
In the old world, lists and maps could be created either using functions
in HIL or list/object constructs in HCL. Here we ensure that in the HCL
case we'll apply any required expression transformations to the individual
items within HCL's compound constructs.
2018-12-04 11:37:39 -08:00
Martin Atkins e49d993d89 configs/configupgrade: Decide on blank lines by ends of items
Previously we were using the line count difference between the start of
one item and the next to decide whether to insert a blank line between
two items, but that is incorrect for multi-line items.

Instead, we'll now count the difference from the final line of the
previous item to the first line of the next, as best we can with the
limited position info recorded by the HCL1 parser.
2018-12-04 11:37:39 -08:00
Martin Atkins bdb724562c configs/configupgrade: Test that map attrs as blocks are fixed
The old parser was forgiving in allowing the use of block syntax where a
map attribute was expected, but the new parser is not (in order to allow
for dynamic map keys, for expressions, etc) and so the upgrade tool must
fix these to use attribute syntax.
2018-12-04 11:37:39 -08:00
Martin Atkins 1aa368d0d8 configs/configupgrade: Add some logging and enable it for tests 2018-12-04 11:37:39 -08:00
Martin Atkins e8999eefdc configs/configupgrade: Populate the test provider schema
This will allow us to test some schema-sensitive migration rules.
2018-12-04 11:37:39 -08:00