In order to remain backward compatible with some modules, we need to
handle subdirs during Load. This means duplicating part of the go-getter
code path for subDir handling so we can resolve any subDirs and globs
internally, while keeping the entire remote directory structure within
the file storage.
Test that we can get a subdirectory from a tarball (or any other
"packed" source that we support).
The 'tar-subdir-to-parent' test highlights a regression where the
subdirectory module references a module in its parent directory. This
breaks the intended use ofr the subdirectory and the implementation in
go-getter. We need to fix this in terraform, and possible plan warnings
and deprecations for this type of source.
The getter.FileDetector was intended to be the final detector, only
converting a path to a file URL and returning a true in all cases. We
want to check for a local module before checking the registry so no
local modules that happen to match a registry module are broken.
Wrap the getter.FileDetector to check the module source's existence
before delegating the search to the registry.
updating the key will cause the FolderStorage hash to change forcing
modules to be re-fetched. This is required because any configurations
using the subDir notation will have the configuration in the wrong
directory.
Terraform was redundantly handling `//dir` notation which should be
handled by go-getter. Rather than allowing go-getter to unpack a subdir
as expected, the subdir was stripped off and accessed through the module
configuration.
This scheme will no longer works now that go-getter supports `*`
subdirectories
(e.g. `//*` would be analogous to `tar --strip-components=1`).
Even though this allows Terraform to use go-getter's native unpacking,
detection is still done separately because Detect requires a `pwd` which
is dependent on the configuration directory and not known to the
global FolderStorage.
Add a getter.Detector for detecting registry modules and looking up
the download location of the latest version. This is essentially a
temporary API until constraint solving is supported by the registry, as
then we'll have to supply the full set of known contraints to the
registry at once for resolution and we will fetch specific versions of
modules.
There is some additional, early validation on the "count" meta-argument
that verifies that only suitable variable types are used, and adding local
values to this whitelist was missed in the initial implementation.
It seems that this somehow got lost in the commit/rebase shuffle and
wasn't caught by the tests that _did_ make it because they were all using
just one file.
As a result of this bug, locals would fail to work correctly in any
configuration with more than one .tf file.
Along with restoring the append/merge behavior, this also reworks some of
the tests to exercise the multi-file case as better insurance against
regressions of this sort in future.
This fixes#15969.
Go 1.9 adds this new function which, when called, marks the caller as
being a "helper function". Helper function stack frames are then skipped
when trying to find a line of test code to blame for a test failure, so
that the code in the main test function appears in the test failure output
rather than a line within the helper function itself.
This covers many -- but probaly not all -- of our test helpers across
various packages.
We added the description field in 0.9 but we never parsed it because we
didn't have a use for it. As we prepare to use this field, let's start
parsing it out
This escapes all characters that might have a special interpretation when embedded into a portion of a URL, including slashes, equals signs and ampersands.
Since Terraform's internals are not 8-bit clean (it assumes UTF-8
strings), we can't implement raw gzip directly. We're going to add
support where it makes sense for passing data to attributes as
base64 so that the result of this function can be used.
Previously we were using the "semver" library to parse version
constraints, but we switched over to go-version and encapsulated it
inside our own plugin/discovery package to reduce dependency sprawl in
the code.
This particular situation was missed when updating references to the new
path, which meant that our validation code disagreed with the rest of
the code about what is considered a valid version constraint string.
By using the correct function, we ensure that we catch early any invalid
versions.
Previously the logic for inferring a provider type from a resource name
was buried a utility function in the 'terraform' package. Instead here we
lift it up into the 'config' package where we can make broader use of it
and where it's easier to discover.
In future we will support version constraints on providers, so we're
reserving this attribute name that is currently not used by any builtin
providers.
For now using this will produce an error, since the rest of Terraform
(outside of the config parser) doesn't currently have this notion and we
don't want people to start trying to use it until its behavior is fully
defined and implemented.
It may be used by third-party providers, so this is a breaking change
worth warning about in CHANGELOG but one whose impact should be small.
Any third-party providers using this name should migrate to using a new
attribute name instead moving forward.
Prior to Terraform 0.7, lists in Terraform were just a shallow abstraction
on top of strings with a magic delimiter between items. Wrapping a single
string in brackets in the configuration was Terraform's prompt that it
needed to split the string on that delimiter during interpolation.
In 0.7, when first-class lists were added, this convention was preserved
by flattening lists-of-lists by one level when they were encountered in
configuration. However, there was an oversight in that change where it
did not correctly handle the case where the inner list was unknown.
In #14135 we removed some code that was flattening partially-unknown lists
into fully-unknown (untyped) values. This inadvertently exposed the missed
case from the previous paragraph, causing issues for list-wrapped splat
expressions with unknown members. While this worked fine for resources,
due to some fixup done inside helper/schema, this did not work for other
interpolation contexts such as module blocks.
Various attempts to fix this up and restore the flattening behavior
selectively were unsuccessful, due to a proliferation of assumptions all
over the core code that would be too risky to change just to fix this bug.
This change, then, takes the different approach of removing the
requirement that splats be presented inside list brackets. This
requirement didn't make much sense anymore anyway, since no other
list-returning expression had this constraint and so the rest of Terraform
was already successfully dealing with both cases.
This leaves us with two different scenarios:
- For resource arguments, existing normalization code in helper/schema
does its own flattening that preserves compatibility with the common
practice of using bracketed splats. This change proves this with a test
within the "test" provider that exercises the whole Terraform core and
helper/schema stack that assigns bracketed splats to list and set
attributes.
- For arguments in other blocks, such as in module callsites, the
interpolator's own flattening behavior applies to known lists,
preserving compatibility with configurations from before
partially-computed splats were possible, but those wishing to use
partially-computed splats are required to drop the surrounding brackets.
This is less concerning because this scenario was introduced only in
0.9.5, so the scope for breakage is limited to those who adopted this
new feature quickly after upgrading.
As of this commit, the recommendation is to stop using brackets around
splats but the old form continues to be supported for backward
compatibility. In a future _major_ version of Terraform we will probably
phase out this legacy form to improve consistency, but for now both
forms are acceptable at the expense of some (pre-existing) weird behavior
when _actual_ lists-of-lists are used.
This addresses #14521 by officially adopting the suggested workaround of
dropping the brackets around the splat. However, it doesn't yet allow
passing of a partially-unknown list between modules: that still violates
assumptions in Terraform's core, so for the moment partially-unknown lists
work only within a _single_ interpolation expression, and cannot be
passed around between expressions. Until more holistic work is done to
improve Terraform's type handling, passing a partially-unknown splat
through to a module will result in a fully-unknown list emerging on
the other side, just as was the case before #14135; this change just
addresses the fact that this was failing with an error in 0.9.5.
> This validation checks that there are now splat variables referencing ourself. This currently is not allowed.
=>
> This validation checks that there are no splat variables referencing ourself. This currently is not allowed.
We've been incorrectly validating (or not validating at all) the
requirement that certain blocks be followed by a name string, to prohibit
e.g. this:
variable {}
and:
variable = ""
Before this change we were catching this for most constructs only if
there were no _valid_ blocks of the same name in the same file. For
modules in particular, we were not catching this at all.
Now we detect this for all kinds of block (resources had a pre-existing
check, so aren't touched here) and produce a different error message
depending on which of the above incorrect forms are used.
This fixes#13575.
stringer has changed the boilerplate it generates in a recent version.
We'd previously updated to the new format but accientally rolled back
to the old while merging a long-running feature branch.
This restores us back to the new format again.
This new function allows using a search within one list to filter another list. For example, it can be used to find the ids of EC2 instances in a particular AZ.
The interface is made slightly awkward by the constraints of HIL's featureset.
#13847
This method mirrors that of config.Backend, so we can compare the
configration of a backend read from a config vs that of a backend read
from a state. This will prevent init from reinitializing when using
`-backend-config` options that match the existing state.
The variable validator assumes that any AST node it gets from an
interpolation walk is an indicator of an interpolation. Unfortunately,
back in f223be15 we changed the interpolation walker to emit a LiteralNode
as a way to signal that the result is a literal but not identical to the
input due to escapes.
The existence of this issue suggests a bit of a design smell in that the
interpolation walker interface at first glance appears to skip over all
literals, but it actually emits them in this one situation. In the long
run we should perhaps think about whether the abstraction is right here,
but this is a shallow, tactical change that fixes#13001.
golang/tools commit 23ca8a263 changed the format of the leading comment
to comply with some new standards discussed here:
https://golang.org/issue/13560
This is the result of running generate with the latest version of
stringer. Everyone working on Terraform will need to update stringer
after this is merged, to avoid reverting this:
go get -u golang.org/x/tools/cmd/stringer
Adds `basename` and `dirname` interpolation. I want to add a `stack` tag to our infrastructure, the value of which is set to `${basename(path.cwd)}`. We currently use `${replace(path.cwd, "/^.+\\//", "")}` instead, but this is extremeley unreadable. The existance of a `basename` function would be very useful for this use case.
I don't have an immediate use case for a `dirname` function, but it seemed reasonable to add it as well.
When configuration is read out of JSON, HCL assumes that empty levels of
objects can be flattened, but this removes too much to decode into a
config.Terraform struct.
Reconstruct the appropriate AST to decode the config struct.
Ensure that fields set in an earlier Terraform config block aren't
removed by Append when encountering another Terraform block. When
multiple blocks contain the same field, the later one still wins.
Fixes#12788
We would panic when referencing an output from an undefined module. The
panic above this is correct but in this case Load will not catch
interpolated variables that _reference_ an unloaded/undefined module.
Test included.
It can be tedious fixing a new module with many errors when Terraform
only outputs the first random error it encounters.
Accumulate all errors from validation, and format them for the user.
Fixes#11800
Type check the value of count so we don't panic on the conversion.
I wondered "why didn't we do this before?" There is no excuse for NOT
doing it at all but the reasoning was beacuse prior to the list/map work
in 0.7, the value couldn't be anything other than a string since any
primitive can turn into a string.
Regardless, we should've always done this.
This disables the computed value check for `count` during the validation
pass. This enables partial support for #3888 or #1497: as long as the
value is non-computed during the plan, complex values will work in
counts.
**Notably, this allows data source values to be present in counts!**
The "count" value can be disabled during validation safely because we
can treat it as if any field that uses `count.index` is computed for
validation. We then validate a single instance (as if `count = 1`) just
to make sure all required fields are set.
Fixes#11038
This is a **short term fix**.
Terraform core doesn't currently handle root modules named "root" well
because the prefix `[]string{"root"}` has special meaning and Terraform
core [currently] can't disambiguate between the root module and a module
named "root" in the root module.
This PR introduces a short term fix by simply disallowing root modules
named "root". This shouldn't break any BC because since 0.8.0 this
didn't work at all in many broken ways (including crashes).
Longer term, this should be fixed by removing the special prefix at all
and having empty paths be root. I started down this path but the core
changes necessary are far too scary for a patch release. We can aim for
0.9.
Fixes#4789
This improves the validation that valid provider aliases are used.
Previously, we required that provider aliases be defined in every module
they're used. This isn't correct because the alias may be used in a
parent module and inherited.
This removes that validation and creates the validation that a provider
alias must be defined in the used module or _any parent_. This allows
inheritance to work properly.
We've always had this type of validation for aliases because we believe
its a good UX tradeoff: typo-ing an alias is really painful, so we
require declaration of alias usage. It may add a small burden to
declare, but since relatively few aliases are used, it improves the
scenario where a user fat-fingers an alias name.
Fixes#10715
`config.Merge` was not updated to support a number of new features. This
updates the codepath to merge various fields, including the `terraform`
block which was the issue in #10715.
The `Merge` API is called when an `_override` file is present to _merge_
configurations. Normally configurations are _appended_. Only an override
file triggers a _merge_.
I started working on a generic library to do this automatically awhile
back but never finished it. This might motivate me to do so. In the
interest of getting a fix out though, we'll continue the manual
approach.
Fixes#10597
This disallows any names for variables, modules, etc. starting with
ints. This causes parse errors with the new HIL parser and actually
causes long term ambiguities if we allow this.
I've also updated the upgrade guide to note this as a backwards
compatibility and how people can fix this going forward.
We allow variables to have descriptions specified, as additional context
for a module user as to what should be provided for a given variable.
We previously lacked a similar mechanism for outputs. Since they too are
part of a module's public interface, it makes sense to be able to add
descriptions for these for symmetry's sake.
This change makes a "description" attribute valid within an "output"
configuration block and stores it within the configuration data structure,
but doesn't yet do anything further with it. For now this is useful only
for third-party tools that might parse a module's config to generate
user documentation; later we could expose the descriptions as part of
the "apply" output, but that is left for a separate change.
Fixes#10075Fixes#10013
When interpolating, we were only maintaining the last known slice index.
If you had sibling slices then you could lose your slice index when
exiting the slice. The resulting behavior was that no some runs the
computed key would be: "slice.0.attr" and on others would be
"slice.attr", the latter being incorrect.
We now maintain a list of slice indexes so that as we unnest, we
properly restore the old value.
Surprisingly unrelated to the graph but the shadow graph caught this
which is great. :)
Fixes#7774
This modifies the `import` command to load configuration files from the
pwd. This also augments the configuration loading section for the CLI to
have a new option (default false, same as old behavior) to
allow directories with no Terraform configurations.
For import, we allow directories with no Terraform configurations so
this option is set to true.
Fixes#7846
This changes from using the HCL decoder to manually decoding the
`variable` blocks within the configuration. This gives us a lot more
power to catch validation errors. This PR retains the same tests and
fixes one additional issue (covered by a test) in the case where a
variable has no named assigned.
Fixes#7607
An empty list is a valid value for formatlist which means to just return
an empty list as a result. The logic was somewhat convoluted here so I
cleaned that up a bit too. The function overall can definitely be
cleaned up a lot more but I left it mostly as-is to fix the bug.
This commit adds a new interpolation function, zipmap, which produces a
map given a list of string keys and a list of values of the same length
as the list of keys.
The name comes from the same operation in Clojure (and likely other
functional langauges).
This is the limitation of all lifecycle attributes currently. Right now,
interpolations are allowed through and the user ends up thinking it
should work. We should give an error.
In the future it should be possible to support some minimal set of
interpolations (static variables, data sources even perhaps) but for now
let's validate that this doesn't work.
This changes the key for the storage to be the _raw_ source from the
module, not the fully expanded source. Example: it'll be a relative path
instead of an absolute path.
This allows the ".terraform/modules" directory to be portable when
moving to other machines. This was a behavior that existed in <= 0.7.2
and was broken with #8398. This amends that and adds a test to verify.
As part of working on ResourceConfig.DeepCopy, Equal I updated
reflectwalk (to fix some issues in the new functions) but this
introduced more issues in other parts of Terraform. This update fixes
those.
Data sources should be able to support counts like a resource. We need
to remove "count" when we load the config because the key doesn't exist
in the schema, and the resource won't validate.
When a resource has only a single key set, the HCL parser treats that
key as part of the overall set of object keys. This isn't valid since
we expect resources to have exactly two keys. In this scenario, we have
to "unwrap" the keys back into a set of objects.
Set the default log package output to iotuil.Discard during tests if the
`-v` flag isn't set. If we are verbose, then apply the filter according
to the TF_LOG env variable.
The concat interpolation function now only accepts list arguments.
Strings are no longer supported, for concatenation or appending to
lists. All arguments must be a list, and single elements can be promoted
with the `list` interpolation function.
Fixes the following error when cross compiling:
```
--> freebsd/amd64 error: exit status 2
Stderr: # github.com/hashicorp/terraform/config/module
config/module/inode.go:18: cannot use st.Ino (type uint32) as type uint64 in return argument
```
* `map(key, value, ...)` - Returns a map consisting of the key/value pairs
specified as arguments. Every odd argument must be a string key, and every
even argument must have the same type as the other values specified.
Duplicate keys are not allowed. Examples:
* `map("hello", "world")`
* `map("us-east", list("a", "b", "c"), "us-west", list("b", "c", "d"))`
This will allow the concat interpolation function to accept lists of
lists, and lists of maps as well as strings. We still allow bare strings
for backwards compatibility, but remove some of the old comment wording
as it could cause confusion of this function with actual string
concatenation.
Since maps are now supported in the config, this removes the superfluous
(and failing) TestInterpolationFuncConcatListOfMaps.
Allow lists and maps within the list interpolation function via variable
interpolation. Since this requires setting the variadic type to TypeAny,
we check for non-heterogeneous lists in the callback.
The list() interpolation function provides a way to add support for list
literals (of strings) to HIL without having to invent new syntax for it
and modify the HIL parser.
It presents as a function, thus:
- list() -> []
- list("a") -> ["a"]
- list("a", "b") -> ["a", "b"]
Thanks to @wr0ngway for the idea of this approach, fixes#7460.
Part of the interpolation walk is to detect keys which involve computed
values and therefore cannot be resolved at this time. The interplation
walker keeps sufficient state to be able to populate the ResourceConfig
with a slice of such keys.
Previously they didn't take slice indexes into account, so in the
following case:
```
"services": []interface{}{
map[string]interface{}{
"elb": "___something computed___",
},
map[string]interface{}{
"elb": "___something else computed___",
},
map[string]interface{}{
"elb": "not computed",
},
}
```
Unknown keys would be populated as follows:
```
services.elb
services.elb
```
This is not sufficient information to be useful, as it is impossible to
distinguish which of the `services.elb`s are unknown vs not.
This commit therefore retains the slice indexes as part of the key for
unknown keys - producing for the example above:
```
services.0.elb
services.1.elb
```
When copying a config module, make sure the full path for src and dst
files don't match, and also check the inode in case we resolved a
different path to the same file.
Make a note about the unsafe usage of reusing a tempDir path.
Escaped quotes are no longer supported as HIL syntax (as of the last
update to HIL), so this commit changes the Terraform config-layer test
to verify the non-presence of this behaviour for 0.7.
The `concat()` interpolation function does not yet support types other
than strings / lists of strings. Make it an error message instead of a
panic when a list of non-primitives is supplied.
Fixes the panic in #7030
Dot indexing worked in the "regexps and strings" world of 0.6.x, but it
no longer works on the 0.7 series w/ proper List / Map types.
There is plenty of dot-indexed config out in the wild, so we need to do
what we can to point users to the new syntax.
Here is one place we can do it for user variables (`var.somemap`). We'll
also need to address Resource Variables and Module Variables in a
separate PR.
This fixes the panic in #7103 - a proper error message is now returned.
This commit changes config parsing from weak decoding lists and maps
into []string and map[string]string respectively to decode into
[]interface{} and map[string]interface{} respectively. This is in order
to take advantage of the work integrated in #7082 to defeat the backward
compatibility features of the mapstructure library.
Test coverage of loading empty variables and validating their default
types against expectation.
The mapstructure library has a regrettable backward compatibility
concern whereby a WeakDecode of []interface{}{} into a target of
map[string]interface{} yields an empty map rather than an error. One
possibility is to switch to using Decode instead of WeakDecode, but this
loses the nice handling of type conversion, requiring a large volume of
code to be added to Terraform or HIL in order to retain that behaviour.
Instead we add a DecodeHook to our usage of the mapstructure library
which checks for decoding []interface{}{} or []string{} into a map and
returns an error instead.
This has the effect of defeating the code added to retain backwards
compatibility in mapstructure, giving us the correct (for our
circumstances) behaviour of Decode for empty structures and the type
conversion of WeakDecode.
The code is identical to that in the HIL library, and packaged into a
helper.
Fixes#4474, where lookup() calls fail out the entire interpolation when
the provided key value is not found in the map. This will allow using
coalesce() along with lookup() to greatly improve module flexibility.
Since the data resource lifecycle contains no steps to deal with tainted
instances, we must make sure that they never get created.
Doing this out in the command layer is not the best, but this is currently
the only layer that has enough information to make this decision and so
this simple solution was preferred over a more disruptive refactoring,
under the assumption that this taint functionality eventually gets
reworked in terms of StateFilter anyway.
This allows ${data.TYPE.NAME.FIELD} interpolation syntax at the
configuration level, though since there is no special handling of them
in the core package this currently just acts as an alias for
${TYPE.NAME.FIELD}.
This allows the config loader to read "data" blocks from the config and
turn them into DataSource objects.
This just reads the data from the config file. It doesn't validate the
data nor do anything useful with it.
Previously resources were assumed to always support the full set of
create, read, update and delete operations, and Terraform's resource
management lifecycle.
Data sources introduce a new kind of resource that only supports the
"read" operation. To support this, a new "Mode" field is added to
the Resource concept within the config layer, which can be set to
ManagedResourceMode (to indicate the only mode previously possible) or
DataResourceMode (to indicate that only "read" is supported).
To support both managed and data resources in the tests, the
stringification of resources in config_string.go is adjusted slightly
to use the Id() method rather than the unusual type[name] serialization
from before, causing a simple mechanical adjustment to the loader tests'
expected result strings.
This commit adds support for native list variables and outputs, building
up on the previous change to state. Interpolation functions now return
native lists in preference to StringList.
List variables are defined like this:
variable "test" {
# This can also be inferred
type = "list"
default = ["Hello", "World"]
}
output "test_out" {
value = "${var.a_list}"
}
This results in the following state:
```
...
"outputs": {
"test_out": [
"hello",
"world"
]
},
...
```
And the result of terraform output is as follows:
```
$ terraform output
test_out = [
hello
world
]
```
Using the output name, an xargs-friendly representation is output:
```
$ terraform output test_out
hello
world
```
The output command also supports indexing into the list (with
appropriate range checking and no wrapping):
```
$ terraform output test_out 1
world
```
Along with maps, list outputs from one module may be passed as variables
into another, removing the need for the `join(",", var.list_as_string)`
and `split(",", var.list_as_string)` which was previously necessary in
Terraform configuration.
This commit also updates the tests and implementations of built-in
interpolation functions to take and return native lists where
appropriate.
A backwards compatibility note: previously the concat interpolation
function was capable of concatenating either strings or lists. The
strings use case was deprectated a long time ago but still remained.
Because we cannot return `ast.TypeAny` from an interpolation function,
this use case is no longer supported for strings - `concat` is only
capable of concatenating lists. This should not be a huge issue - the
type checker picks up incorrect parameters, and the native HIL string
concatenation - or the `join` function - can be used to replicate the
missing behaviour.
This changes the representation of maps in the interpolator from the
dotted flatmap form of a string variable named "var.variablename.key"
per map element to use native HIL maps instead.
This involves porting some of the interpolation functions in order to
keep the tests green, and adding support for map outputs.
There is one backwards incompatibility: as a result of an implementation
detail of maps, one could access an indexed map variable using the
syntax "${var.variablename.key}".
This is no longer possible - instead HIL native syntax -
"${var.variablename["key"]}" must be used. This was previously
documented, (though not heavily used) so it must be noted as a backward
compatibility issue for Terraform 0.7.
* core: Add support for marking outputs as sensitive
This commit allows an output to be marked "sensitive", in which case the
value is redacted in the post-refresh and post-apply list of outputs.
For example, the configuration:
```
variable "input" {
default = "Hello world"
}
output "notsensitive" {
value = "${var.input}"
}
output "sensitive" {
sensitive = true
value = "${var.input}"
}
```
Would result in the output:
```
terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
notsensitive = Hello world
sensitive = <sensitive>
```
The `terraform output` command continues to display the value as before.
Limitations: Note that sensitivity is not tracked internally, so if the
output is interpolated in another module into a resource, the value will
be displayed. The value is still present in the state.
hil.Eval() now returns (hil.EvaluationResult, error) instead of (value,
type, error). This commit updates the call sites, but retains all
previous behaviour. Tests are also updated.
These tests demonstrates a problem where the types to a module input are
not checked. For example, if a module - inner - defines a variable
"should_be_a_map" as a map, or with a default variable of map, we do not
fail if the user sets the variable value in the outer module to a string
value. This is also a problem in nested modules.
The implementation changes add a type checking step into the graph
evaluation process to ensure invalid types are not passed.
Fixes an interpolation race that was occurring when a tainted destroy
node and a primary destroy node both tried to interpolate a computed
count in their config. Since they were sharing a pointer to the _same_
config, depending on how the race played out one of them could catch the
config uninterpolated and would then throw a syntax error.
The `Copy()` tree implemented for this fix can probably be used
elsewhere - basically we should copy the config whenever we drop nodes
into the graph - but for now I'm just applying it to the place that
fixes this bug.
Fixes#4982 - Includes a test covering that race condition.
James Bardin