Commit Graph

119 Commits

Author SHA1 Message Date
Martin Atkins 954d38e870 lang: New file-hashing functions
In prior versions, we recommended using hash functions in conjunction with
the file function as an idiom for detecting changes to upstream blobs
without fetching and comparing the whole blob.

That approach relied on us being able to return raw binary data from
file(...). Since Terraform strings pass through intermediate
representations that are not binary-safe (e.g. the JSON state), there was
a risk of string corruption in prior versions which we have avoided for
0.12 by requiring that file(...) be used only with UTF-8 text files.

The specific case of returning a string and immediately passing it into
another function was not actually subject to that corruption risk, since
the HIL interpreter would just pass the string through verbatim, but this
is still now forbidden as a result of the stricter handling of file(...).

To avoid breaking these use-cases, here we introduce variants of the hash
functions a with "file" prefix that take a filename for a disk file to
hash rather than hashing the given string directly. The configuration
upgrade tool also now includes a rule to detect the documented idiom and
rewrite it into a single function call for one of these new functions.

This does cause a bit of function sprawl, but that seems preferable to
introducing more complex rules for when file(...) can and cannot read
binary files, making the behavior of these various functions easier to
understand in isolation.
2019-01-25 10:18:44 -08:00
Martin Atkins ecaaa91da9 lang/funcs: Factor out the various hash function implementations
These all follow the pattern of creating a hash and converting it to a
string using some encoding function, so we can write this implementation
only once and parameterize it with a hash factory function and an encoding
function.

This also includes a new test for the sha512 function, which was
previously missing a test and, it turns out, actually computing sha256
instead.
2019-01-25 10:18:44 -08:00
Martin Atkins 2f8f7d6f4d lang/funcs: Type conversion functions
It's not normally necessary to make explicit type conversions in Terraform
because the language implicitly converts as necessary, but explicit
conversions are useful in a few specialized cases:

- When defining output values for a reusable module, it may be desirable
  to force a "cleaner" output type than would naturally arise from a
  computation, such as forcing a string containing digits into a number.
- Our 0.12upgrade mechanism will use some of these to replace use of the
  undocumented, hidden type conversion functions in HIL, and force
  particular type interpretations in some tricky cases.
- We've found that type conversion functions can be useful as _temporary_
  workarounds for bugs in Terraform and in providers where implicit type
  conversion isn't working correctly or a type constraint isn't specified
  precisely enough for the automatic conversion behavior.

These all follow the same convention of being named "to" followed by a
short type name. Since we've had a long-standing convention of running all
the words together in lowercase in function names, we stick to that here
even though some of these names are quite strange, because these should
be rarely-used functions anyway.
2019-01-17 10:01:47 -08:00
Martin Atkins da51e72cbb lang/functions: set functions from cty
The sethaselement, setintersection, and setunion functions are defined in
the cty stdlib. Making them available in Terraform will make it easier to
work with sets, and complement the currently-Terraform-specific setproduct
function.

In the long run setproduct should probably move into the cty stdlib too,
but since it was submitted as a Terraform function originally we'll leave
it here now for simplicity's sake and reorganize later.
2019-01-16 09:57:16 -08:00
Martin Atkins edb5f82de1 lang/funcs: Convert the "setproduct" function to the new approach
In our new world it produces either a set of a tuple type or a list of a
tuple type, depending on the given argument types.

The resulting collection's element tuple type is decided by the element
types of the given collections, allowing type information to propagate
even if unknown values are present.
2019-01-16 09:57:16 -08:00
Martin Atkins d0e6a4c69a lang: Add "formatdate" function
We missed this one on a previous pass of bringing in most of the cty
stdlib functions.

This will resolve #17625 by allowing conversion from Terraform's
conventional RFC 3339 timestamps into various other formats.
2019-01-07 09:10:14 -08:00
Martin Atkins c753df6a93 lang/funcs: templatefile function
This function is similar to the template_file data source offered by the
template provider, but having it built in to the language makes it more
convenient to use, allowing templates to be rendered from files anywhere
an inline template would normally be allowed:

    user_data = templatefile("${path.module}/userdata.tmpl", {
      hostname = format("petserver%02d", count.index)
    })

Unlike the template_file data source, this function allows values of any
type in its variables map, passing them through verbatim to the template.
Its tighter integration with Terraform also allows it to return better
error messages with source location information from the template itself.

The template_file data source was originally created to work around the
fact that HIL didn't have any support for map values at the time, and
even once map support was added it wasn't very usable. With HCL2
expressions, there's little reason left to use a data source to render
a template; the only remaining reason left to use template_file is to
render a template that is constructed dynamically during the Terraform
run, which is a very rare need.
2018-12-21 08:06:14 -08:00
Martin Atkins 2be524d6ac core: Validate depends_on and ignore_changes traversals
Both depends_on and ignore_changes contain references to objects that we
can validate.

Historically Terraform has not validated these, instead just ignoring
references to non-existent objects. Since there is no reason to refer to
something that doesn't exist, we'll now verify this and return errors so
that users get explicit feedback on any typos they may have made, rather
than just wondering why what they added seems to have no effect.

This is particularly important for ignore_changes because users have
historically used strange values here to try to exploit the fact that
Terraform was resolving ignore_changes against a flatmap. This will give
them explicit feedback for any odd constructs that the configuration
upgrade tool doesn't know how to detect and fix.
2018-12-17 09:02:25 -08:00
Martin Atkins e63a1dfb96 lang: EvalExpr only convert if wantType is not dynamic
This actually seems to be a bug in the underlying cty Convert function
since converting to cty.DynamicPseudoType should always just return the
input verbatim, but it seems like it's actually converting unknown values
of any type to be cty.DynamicVal, losing the type information.

We should eventually fix this in cty too, but having this extra check in
the Terraform layer is harmless and allows us to make progress without
context-switching.
2018-12-07 17:05:36 -08:00
Martin Atkins 30497bbfb7 lang/funcs: zipmap accepts tuple of values and produces object
Now that our language supports tuple/object types in addition to list/map
types, it's convenient for zipmap to be able to produce an object type
given a tuple, since this makes it symmetrical with "keys" and "values"
such the the following identity holds for any map or object value "a"

    a == zipmap(keys(a), values(a))

When the values sequence is a tuple, the result has an object type whose
attribute types correspond to the given tuple types.

Since an object type has attribute names as part of its definition, there
is the additional constraint here that the result has an unknown type
(represented by the dynamic pseudo-type) if the given values is a tuple
and the given keys contains any unknown values. This isn't true for values
as a list because we can predict the resulting map element type using
just the list element type.
2018-11-28 07:45:43 -08:00
Martin Atkins 3b49028b77 core: Static-validate resource references against schemas
In the initial move to HCL2 we started relying only on full expression
evaluation to catch attribute errors, but that's not sufficient for
resource attributes in practice because during validation we can't know
yet whether a resource reference evaluates to a single object or to a
list of objects (if count is set).

To address this, here we reinstate some static validation of resource
references by analyzing directly the reference objects, disregarding any
instance index if present, and produce errors if the remaining subsequent
traversal steps do not correspond to items within the resource type
schema.

This also allows us to produce some more specialized error messages for
certain situations. In particular, we can recognize a reference like
aws_instance.foo.count, which in 0.11 and prior was a weird special case
for determining the count value of a resource block, and offer a helpful
error showing the new length(aws_instance.foo) usage pattern.

This eventually delegates to the static traversal validation logic that
was added to the configschema package in a previous commit, which also
includes some specialized error messages that distinguish between
attributes and block types in the schema so that the errors relate more
directly to constructs the user can see in the configuration.

In future we could potentially move more of the checks from the dynamic
schema construction step to the static validation step, but resources
are the reference type that most needs this immediately due to the
ambiguity caused by the instance indexing syntax. We can safely refactor
other reference types to be statically validated in later releases.

This is verified by two pre-existing context validate tests which we
temporarily disabled during earlier work (now re-enabled) and also by a
new validate test aimed specifically at the special case for the "count"
attribute.
2018-11-26 08:25:03 -08:00
Martin Atkins ecc42b838c lang/funcs: Fix crash and improve precision of keys/values functions
The "values" function wasn't producing consistently-ordered keys in its
result, leading to crashes. This fixes #19204.

While working on these functions anyway, this also improves slightly their
precision when working with object types, where we can produce a more
complete result for unknown values because the attribute names are part
of the type. We can also produce results for known maps that have unknown
elements; these unknowns will also appear in the values(...) result,
allowing them to propagate through expressions.

Finally, this adds a few more test cases to try different permutations
of empty and unknown values.
2018-11-06 08:33:49 -08:00
Martin Atkins 8c01cf7293 lang/funcs: Fix broken test for lookup function
When the value we're looking in has an object type, we need to know the
key in order to decide the result type. Therefore an object lookup with
an unknown key must produce cty.DynamicVal, not an unknown value with a
known type.
2018-11-06 08:33:49 -08:00
Martin Atkins 8f578c365f lang/funcs: Permit object types in the "length" function
The implementation already allowed this, so this was just an oversight in
the type checking function.

This fixes #19278.
2018-11-06 08:33:49 -08:00
Radek Simko b41cda7d7a
Return cty.DynamicPseudoType instead 2018-10-23 12:42:46 +02:00
Radek Simko 4856d81300
lang: Fix crash in lookup function 2018-10-22 12:58:47 +02:00
James Bardin 9e3c23d890 verify that all LengthInt calls are known
There were some funcs with LengthInt calls on unknown values
2018-10-18 19:19:59 -04:00
Radek Simko edaa4bbc82
lang: Add fileexists function 2018-10-17 10:18:07 +01:00
Kristin Laemmert fd77e56fd6 lookup will return a tuple type when passed an object 2018-10-16 19:14:54 -07:00
Kristin Laemmert f54ee830d3 lang/funcs: update values to accept object types 2018-10-16 19:14:54 -07:00
Kristin Laemmert 4ec904bca7 funcs/lang lookup: validate that argument is map or object type 2018-10-16 19:14:54 -07:00
Kristin Laemmert d1d0ede069 lang/funcs: return default value if provided when object lookup fails to find attr 2018-10-16 19:14:54 -07:00
Kristin Laemmert 46e168a682 lang/funcs: update lookup() to accept object-typed values for "map" arg 2018-10-16 19:14:54 -07:00
Martin Atkins c990c9f36d lang/funcs: Don't panic if coalescelist gets an unknown list 2018-10-16 19:14:11 -07:00
Martin Atkins db58b88c2d lang/funcs: short-circuit with unknown index and tuple collection
Since we need to know the index to know the result type for a tuple, we
need a special case here to deal with that situation and return
cty.DynamicVal; we can't predict the result type exactly until we know the
element type.
2018-10-16 19:14:11 -07:00
Martin Atkins efe631d9ec lang/funcs: in "sort", don't panic if given a null string
It is incorrect to use a null string, but that should be reported as an
error rather than a panic.
2018-10-16 19:14:11 -07:00
Martin Atkins 479c6b2466 move "configschema" from "config" to "configs"
The "config" package is no longer used and will be removed as part
of the 0.12 release cleanup. Since configschema is part of the
"new world" of configuration modelling, it makes more sense for
it to live as a subdirectory of the newer "configs" package.
2018-10-16 18:50:29 -07:00
Martin Atkins ccd90bcf35 lang/funcs: never include the private key in error output
This is based on c811440188 made against the
old "config" package implementations, but also catches a few other cases
where we would previously have printed the private key into the error
messages.
2018-10-16 18:50:29 -07:00
Kristin Laemmert c23a971ed1 minor fixes 2018-10-16 18:49:20 -07:00
Kristin Laemmert 0dbecc54c0 functions: ValuesFunc - cleanup return type 2018-10-16 18:49:20 -07:00
Kristin Laemmert a213c4a648 functions: add tests and support for unknown values 2018-10-16 18:49:20 -07:00
Kristin Laemmert d802d5c624 functions: pr feedback fixes 2018-10-16 18:49:20 -07:00
Kristin Laemmert ff4b3d763b functions: fix lookup()'s handling of numberical defaults 2018-10-16 18:49:20 -07:00
Kristin Laemmert 4f5c03339a functions: ZipmapFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert 605e23db6b various code review fixes 2018-10-16 18:49:20 -07:00
Kristin Laemmert 6463dd90e9 functions: TransposeFunc, SliceFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert 30671d85ad functions: MergeFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert aecd7b2e62 functions: LookupFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert 4d8c398f8e functions: KeysFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert 21daabe680 functions: MapFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert da02e0da4d functions: ListFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert 529c2c3cc9 functions: FlattenFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert 4dd3ffc127 porting matchkeys 2018-10-16 18:49:20 -07:00
Kristin Laemmert b979053361 general cleanup - addressing code review 2018-10-16 18:49:20 -07:00
Kristin Laemmert 498ffbf77b adding some test cases and tweaking implementation to address them 2018-10-16 18:49:20 -07:00
Kristin Laemmert 0cbcd75ebb port distinct and chunklist functions 2018-10-16 18:49:20 -07:00
Kristin Laemmert 8aac7587f7 port index and contains functions 2018-10-16 18:49:20 -07:00
Kristin Laemmert e697e7d733 port compact function 2018-10-16 18:49:20 -07:00
Kristin Laemmert 1901d5d184 port coalescelist function 2018-10-16 18:49:20 -07:00
Kristin Laemmert 04ac87747c base64decode: check that the decoded (not encoded) string is valid UTF-8 2018-10-16 18:49:20 -07:00
Kristin Laemmert 74f2d58b8b base64decode: check that the decoded (not encoded) string is valid UTF-8 2018-10-16 18:49:20 -07:00
Kristin Laemmert 6171ba3b8a base64decodeFunc now checks for valid UTF-8 2018-10-16 18:49:20 -07:00
Kristin Laemmert b6d3d69d3a port cidr functions 2018-10-16 18:49:20 -07:00
Kristin Laemmert 10ef61c71c porting many functions 2018-10-16 18:49:20 -07:00
Kristin Laemmert 602b59cdc4 porting functions 2018-10-16 18:49:20 -07:00
Kristin Laemmert 4ad3676934 port ceil function 2018-10-16 18:49:20 -07:00
Kristin Laemmert c4f4dddff5 porting crypto functions 2018-10-16 18:49:20 -07:00
Kristin Laemmert eb1d8b7909 cleanup comments for nicer godocs 2018-10-16 18:49:20 -07:00
Kristin Laemmert 9aa9b18658 porting crypto functions 2018-10-16 18:49:20 -07:00
Kristin Laemmert 1a5299efcb porting encoding functions 2018-10-16 18:49:20 -07:00
Kristin Laemmert d4e703a5c1 base64decode 2018-10-16 18:48:28 -07:00
Kristin Laemmert e30cb1a7dc typo 2018-10-16 18:48:28 -07:00
Kristin Laemmert a187c92f0e implement datetime functions 2018-10-16 18:48:28 -07:00
Kristin Laemmert 755b1e2497 implement pathexpand 2018-10-16 18:48:28 -07:00
Kristin Laemmert 2a2ffb6ef4 implement dirname function 2018-10-16 18:48:28 -07:00
Kristin Laemmert 8c1f0842b0 implement basename function 2018-10-16 18:48:28 -07:00
Martin Atkins 129f5fe74d lang/funcs: port some of Terraform's built-in functions
These implementations are adaptations of the existing implementations in
config/interpolate_funcs.go, updated to work with the cty API.

The set of functions chosen here was motivated mainly by what Terraform's
existing context tests depend on, so we can get the contexts tests back
into good shape before fleshing out the rest of these functions.
2018-10-16 18:48:28 -07:00
Martin Atkins c937c06a03 terraform: ugly huge change to weave in new HCL2-oriented types
Due to how deeply the configuration types go into Terraform Core, there
isn't a great way to switch out to HCL2 gradually. As a consequence, this
huge commit gets us from the old state to a _compilable_ new state, but
does not yet attempt to fix any tests and has a number of known missing
parts and bugs. We will continue to iterate on this in forthcoming
commits, heading back towards passing tests and making Terraform
fully-functional again.

The three main goals here are:
- Use the configuration models from the "configs" package instead of the
  older models in the "config" package, which is now deprecated and
  preserved only to help us write our migration tool.
- Do expression inspection and evaluation using the functionality of the
  new "lang" package, instead of the Interpolator type and related
  functionality in the main "terraform" package.
- Represent addresses of various objects using types in the addrs package,
  rather than hand-constructed strings. This is not critical to support
  the above, but was a big help during the implementation of these other
  points since it made it much more explicit what kind of address is
  expected in each context.

Since our new packages are built to accommodate some future planned
features that are not yet implemented (e.g. the "for_each" argument on
resources, "count"/"for_each" on modules), and since there's still a fair
amount of functionality still using old-style APIs, there is a moderate
amount of shimming here to connect new assumptions with old, hopefully in
a way that makes it easier to find and eliminate these shims later.

I apologize in advance to the person who inevitably just found this huge
commit while spelunking through the commit history.
2018-10-16 18:46:46 -07:00
Martin Atkins a16ca2ec53 lang: new package for the runtime parts of the config language
Whereas package "configs" deals with the static structure of the
configuration language, this new package "lang" deals with the dynamic
aspects such as expression evaluation.

So far this mainly consists of populating a hcl.EvalContext that contains
the values necessary to evaluate a block or an expression. There is also
special handling here for dynamic block generation using the HCL
"dynblock" extension, which is exposed in the public interface (rather
than hiding it as an implementation detail of EvalBlock) so that the
caller can then extract proper source locations for any result values
using the expanded body.

This also includes the beginnings of a replacement for the function table
handling that currently lives in the old "config" package, but most of
the functions are not yet ported and so this will expand in subsequent
commits.
2018-10-16 18:44:26 -07:00