Commit Graph

61 Commits

Author SHA1 Message Date
Martin Atkins 954d38e870 lang: New file-hashing functions
In prior versions, we recommended using hash functions in conjunction with
the file function as an idiom for detecting changes to upstream blobs
without fetching and comparing the whole blob.

That approach relied on us being able to return raw binary data from
file(...). Since Terraform strings pass through intermediate
representations that are not binary-safe (e.g. the JSON state), there was
a risk of string corruption in prior versions which we have avoided for
0.12 by requiring that file(...) be used only with UTF-8 text files.

The specific case of returning a string and immediately passing it into
another function was not actually subject to that corruption risk, since
the HIL interpreter would just pass the string through verbatim, but this
is still now forbidden as a result of the stricter handling of file(...).

To avoid breaking these use-cases, here we introduce variants of the hash
functions a with "file" prefix that take a filename for a disk file to
hash rather than hashing the given string directly. The configuration
upgrade tool also now includes a rule to detect the documented idiom and
rewrite it into a single function call for one of these new functions.

This does cause a bit of function sprawl, but that seems preferable to
introducing more complex rules for when file(...) can and cannot read
binary files, making the behavior of these various functions easier to
understand in isolation.
2019-01-25 10:18:44 -08:00
Martin Atkins ecaaa91da9 lang/funcs: Factor out the various hash function implementations
These all follow the pattern of creating a hash and converting it to a
string using some encoding function, so we can write this implementation
only once and parameterize it with a hash factory function and an encoding
function.

This also includes a new test for the sha512 function, which was
previously missing a test and, it turns out, actually computing sha256
instead.
2019-01-25 10:18:44 -08:00
Martin Atkins 2f8f7d6f4d lang/funcs: Type conversion functions
It's not normally necessary to make explicit type conversions in Terraform
because the language implicitly converts as necessary, but explicit
conversions are useful in a few specialized cases:

- When defining output values for a reusable module, it may be desirable
  to force a "cleaner" output type than would naturally arise from a
  computation, such as forcing a string containing digits into a number.
- Our 0.12upgrade mechanism will use some of these to replace use of the
  undocumented, hidden type conversion functions in HIL, and force
  particular type interpretations in some tricky cases.
- We've found that type conversion functions can be useful as _temporary_
  workarounds for bugs in Terraform and in providers where implicit type
  conversion isn't working correctly or a type constraint isn't specified
  precisely enough for the automatic conversion behavior.

These all follow the same convention of being named "to" followed by a
short type name. Since we've had a long-standing convention of running all
the words together in lowercase in function names, we stick to that here
even though some of these names are quite strange, because these should
be rarely-used functions anyway.
2019-01-17 10:01:47 -08:00
Martin Atkins edb5f82de1 lang/funcs: Convert the "setproduct" function to the new approach
In our new world it produces either a set of a tuple type or a list of a
tuple type, depending on the given argument types.

The resulting collection's element tuple type is decided by the element
types of the given collections, allowing type information to propagate
even if unknown values are present.
2019-01-16 09:57:16 -08:00
Martin Atkins c753df6a93 lang/funcs: templatefile function
This function is similar to the template_file data source offered by the
template provider, but having it built in to the language makes it more
convenient to use, allowing templates to be rendered from files anywhere
an inline template would normally be allowed:

    user_data = templatefile("${path.module}/userdata.tmpl", {
      hostname = format("petserver%02d", count.index)
    })

Unlike the template_file data source, this function allows values of any
type in its variables map, passing them through verbatim to the template.
Its tighter integration with Terraform also allows it to return better
error messages with source location information from the template itself.

The template_file data source was originally created to work around the
fact that HIL didn't have any support for map values at the time, and
even once map support was added it wasn't very usable. With HCL2
expressions, there's little reason left to use a data source to render
a template; the only remaining reason left to use template_file is to
render a template that is constructed dynamically during the Terraform
run, which is a very rare need.
2018-12-21 08:06:14 -08:00
Martin Atkins 30497bbfb7 lang/funcs: zipmap accepts tuple of values and produces object
Now that our language supports tuple/object types in addition to list/map
types, it's convenient for zipmap to be able to produce an object type
given a tuple, since this makes it symmetrical with "keys" and "values"
such the the following identity holds for any map or object value "a"

    a == zipmap(keys(a), values(a))

When the values sequence is a tuple, the result has an object type whose
attribute types correspond to the given tuple types.

Since an object type has attribute names as part of its definition, there
is the additional constraint here that the result has an unknown type
(represented by the dynamic pseudo-type) if the given values is a tuple
and the given keys contains any unknown values. This isn't true for values
as a list because we can predict the resulting map element type using
just the list element type.
2018-11-28 07:45:43 -08:00
Martin Atkins ecc42b838c lang/funcs: Fix crash and improve precision of keys/values functions
The "values" function wasn't producing consistently-ordered keys in its
result, leading to crashes. This fixes #19204.

While working on these functions anyway, this also improves slightly their
precision when working with object types, where we can produce a more
complete result for unknown values because the attribute names are part
of the type. We can also produce results for known maps that have unknown
elements; these unknowns will also appear in the values(...) result,
allowing them to propagate through expressions.

Finally, this adds a few more test cases to try different permutations
of empty and unknown values.
2018-11-06 08:33:49 -08:00
Martin Atkins 8c01cf7293 lang/funcs: Fix broken test for lookup function
When the value we're looking in has an object type, we need to know the
key in order to decide the result type. Therefore an object lookup with
an unknown key must produce cty.DynamicVal, not an unknown value with a
known type.
2018-11-06 08:33:49 -08:00
Martin Atkins 8f578c365f lang/funcs: Permit object types in the "length" function
The implementation already allowed this, so this was just an oversight in
the type checking function.

This fixes #19278.
2018-11-06 08:33:49 -08:00
Radek Simko b41cda7d7a
Return cty.DynamicPseudoType instead 2018-10-23 12:42:46 +02:00
Radek Simko 4856d81300
lang: Fix crash in lookup function 2018-10-22 12:58:47 +02:00
James Bardin 9e3c23d890 verify that all LengthInt calls are known
There were some funcs with LengthInt calls on unknown values
2018-10-18 19:19:59 -04:00
Radek Simko edaa4bbc82
lang: Add fileexists function 2018-10-17 10:18:07 +01:00
Kristin Laemmert fd77e56fd6 lookup will return a tuple type when passed an object 2018-10-16 19:14:54 -07:00
Kristin Laemmert f54ee830d3 lang/funcs: update values to accept object types 2018-10-16 19:14:54 -07:00
Kristin Laemmert 4ec904bca7 funcs/lang lookup: validate that argument is map or object type 2018-10-16 19:14:54 -07:00
Kristin Laemmert d1d0ede069 lang/funcs: return default value if provided when object lookup fails to find attr 2018-10-16 19:14:54 -07:00
Kristin Laemmert 46e168a682 lang/funcs: update lookup() to accept object-typed values for "map" arg 2018-10-16 19:14:54 -07:00
Martin Atkins c990c9f36d lang/funcs: Don't panic if coalescelist gets an unknown list 2018-10-16 19:14:11 -07:00
Martin Atkins db58b88c2d lang/funcs: short-circuit with unknown index and tuple collection
Since we need to know the index to know the result type for a tuple, we
need a special case here to deal with that situation and return
cty.DynamicVal; we can't predict the result type exactly until we know the
element type.
2018-10-16 19:14:11 -07:00
Martin Atkins efe631d9ec lang/funcs: in "sort", don't panic if given a null string
It is incorrect to use a null string, but that should be reported as an
error rather than a panic.
2018-10-16 19:14:11 -07:00
Martin Atkins ccd90bcf35 lang/funcs: never include the private key in error output
This is based on c811440188 made against the
old "config" package implementations, but also catches a few other cases
where we would previously have printed the private key into the error
messages.
2018-10-16 18:50:29 -07:00
Kristin Laemmert c23a971ed1 minor fixes 2018-10-16 18:49:20 -07:00
Kristin Laemmert 0dbecc54c0 functions: ValuesFunc - cleanup return type 2018-10-16 18:49:20 -07:00
Kristin Laemmert a213c4a648 functions: add tests and support for unknown values 2018-10-16 18:49:20 -07:00
Kristin Laemmert d802d5c624 functions: pr feedback fixes 2018-10-16 18:49:20 -07:00
Kristin Laemmert ff4b3d763b functions: fix lookup()'s handling of numberical defaults 2018-10-16 18:49:20 -07:00
Kristin Laemmert 4f5c03339a functions: ZipmapFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert 605e23db6b various code review fixes 2018-10-16 18:49:20 -07:00
Kristin Laemmert 6463dd90e9 functions: TransposeFunc, SliceFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert 30671d85ad functions: MergeFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert aecd7b2e62 functions: LookupFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert 4d8c398f8e functions: KeysFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert 21daabe680 functions: MapFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert da02e0da4d functions: ListFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert 529c2c3cc9 functions: FlattenFunc 2018-10-16 18:49:20 -07:00
Kristin Laemmert 4dd3ffc127 porting matchkeys 2018-10-16 18:49:20 -07:00
Kristin Laemmert b979053361 general cleanup - addressing code review 2018-10-16 18:49:20 -07:00
Kristin Laemmert 498ffbf77b adding some test cases and tweaking implementation to address them 2018-10-16 18:49:20 -07:00
Kristin Laemmert 0cbcd75ebb port distinct and chunklist functions 2018-10-16 18:49:20 -07:00
Kristin Laemmert 8aac7587f7 port index and contains functions 2018-10-16 18:49:20 -07:00
Kristin Laemmert e697e7d733 port compact function 2018-10-16 18:49:20 -07:00
Kristin Laemmert 1901d5d184 port coalescelist function 2018-10-16 18:49:20 -07:00
Kristin Laemmert 04ac87747c base64decode: check that the decoded (not encoded) string is valid UTF-8 2018-10-16 18:49:20 -07:00
Kristin Laemmert 74f2d58b8b base64decode: check that the decoded (not encoded) string is valid UTF-8 2018-10-16 18:49:20 -07:00
Kristin Laemmert 6171ba3b8a base64decodeFunc now checks for valid UTF-8 2018-10-16 18:49:20 -07:00
Kristin Laemmert b6d3d69d3a port cidr functions 2018-10-16 18:49:20 -07:00
Kristin Laemmert 10ef61c71c porting many functions 2018-10-16 18:49:20 -07:00
Kristin Laemmert 602b59cdc4 porting functions 2018-10-16 18:49:20 -07:00
Kristin Laemmert 4ad3676934 port ceil function 2018-10-16 18:49:20 -07:00