Commit Graph

115 Commits

Author SHA1 Message Date
Martin Atkins 135afaeb9c lang: "regex" and "regexall" functions
These existing upstream cty functions allow matching strings against
regular expression patterns, which can be useful if you need to consume
a non-standard string format that Terraform doesn't (and can't) have a
built-in function for.
2019-08-06 11:52:14 -07:00
Kristin Laemmert 2b14a6b332
lang/funcs: lookup() can work with maps of lists, maps and objects (#22269)
* lang/funcs: lookup() can work with maps of lists, maps and objects

lookup() can already handle aribtrary objects of (whatever) and should
handle maps of (whatever) similarly.
2019-08-01 15:52:52 -04:00
Pam Selle 360068b3cb
Merge pull request #21922 from pselle/resource_for_each
Resource for_each
2019-07-26 11:41:56 -04:00
James Bardin 6e222375c7 invlaid use of dynamic with attrs should not panic
Mistakenly using dynamic on an attribute will lead to a panic when
attempting to resolve variable references with a partial body, because
the dynamic blocks have yet to be expanded and validated. Check that the
block element type is actually an object before generating a schema.
2019-07-22 17:16:18 -04:00
Pam Selle 7d905f6777 Resource for_each 2019-07-22 10:51:16 -04:00
Radek Simko 5c30bafee0
Merge pull request #22119 from hashicorp/b-chunklist-empty-list
lang/funcs: Pass through empty list in chunklist
2019-07-18 20:59:47 +01:00
Radek Simko 736fda1613
lang/funcs: Pass through empty list in chunklist 2019-07-18 15:01:00 +01:00
Saverio Miroddi 983af7a183 Allow null values in Compact function
The function would previously panic when one or more null values were among the arguments.

The new behavior treats nulls as empty strings, therefore, it removes them.
2019-07-11 20:16:24 +02:00
Andreas Sommer 042aead714 lang/funcs: add "abspath" function (#21409) 2019-07-02 08:30:30 -04:00
Paul Tyng ebea080aca Add missing bool case
Fixes #20572
2019-06-24 15:27:59 -04:00
Pam Selle 1255ad6722 Don't allow nulls in calls to merge 2019-06-14 15:26:24 -04:00
Lars Eric Scheidler aa07806bfc lang/funcs: New "uuidv5" function
This generates name-based uuids, rather than pseudorandom uuids as with the
"uuid" function.
2019-06-07 14:38:22 -07:00
Martin Atkins 382e1ca821 lang: yamldecode and yamlencode functions
These follow the same principle as jsondecode and jsonencode, but use
YAML instead of JSON.

YAML has a much more complex information model than JSON, so we can only
support a subset of it during decoding, but hopefully the subset supported
here is a useful one.

Because there are many different ways to _generate_ YAML, the yamlencode
function is forced to make some decisions, and those decisions are likely
to affect compatibility with other real-world YAML parsers. Although the
format here is intended to be generic and compatible, we may find that
there are problems with it that'll we'll want to adjust for in a future
release, so yamlencode is therefore marked as experimental for now until
the underlying library is ready to commit to ongoing byte-for-byte
compatibility in serialization.

The main use-case here is met by yamldecode, which will allow reading in
files written in YAML format by humans for use in Terraform modules, in
situations where a higher-level input format than direct Terraform
language declarations is helpful.
2019-06-04 16:24:09 -07:00
Martin Atkins f9a73d48db lang: "range" function
This is similar to the function of the same name in Python, generating a
sequence of numbers as a list that can then be used in other
sequence-oriented operations.

The primary use-case for it is to turn a count expressed as a number into
a list of that length, which can then be iterated over or passed to a
collection function to produce that number of something else, as shown
in the example at the end of its documentation page.
2019-06-04 16:20:17 -07:00
Kristin Laemmert c163ba71db simplify 2019-06-04 11:54:26 -04:00
Kristin Laemmert 30a924e162 lang/funcs: `matchkeys` - unify type for keys and searchset
Added higher-level test for matchkeys to exercise mixing
types in searchset. This had to be in the functions tests so the HCL
auto conversion from tuple to list would occur.
2019-06-04 08:57:02 -04:00
Kristin Laemmert f2a14d7c18 lang/funcs: fix error when `matchkeys` encountered a variable
`matchkeys` was returning a (false) error if the searchset was a
variable, since then the type of the keylist and searchset parameters
would not match.

This does slightly change the behavior: previously matchkeys would
produce an error if the parameters were not of the same type, for e.g.
if searchset was a list of strings and keylist was a list of integers.
  This no longer produces an error.
2019-06-03 18:00:53 -04:00
James Bardin 0133dda311 allow empty dynamic blocks with ConfigModeAttr
If a dynamic block is evaluated zero times, the body content will
contain 0 blocks. Allow the probe for ConfigModeAttr to accept that no
blocks with a matching attribute should still be converted to a block if
they are called with dynamicExpand.
2019-05-31 16:31:24 -05:00
Martin Atkins 55794eb658 lang/funcs: Don't panic when distinct called with empty list 2019-05-31 11:44:31 -05:00
Martin Atkins 742deca3e9 lang/funcs: Short-circuit if start or end index is unknown
Previously the type-selection codepath for an input tuple referred
unconditionally to the start and end index values. In the Type callback,
only the types of the arguments are guaranteed to be known, so any access
to the values must be guarded with an .IsKnown check, or else the function
will not short-circuit properly when an unknown value is passed.

Now we will check the start and end indices are in range when we have
enough information to do so, but we'll return an approximate result if
either is unknown.
2019-05-17 08:48:02 -07:00
Kristin Laemmert b1d0b1383f
lang/funcs: remove sethaselement function and documentation (#21164)
`contains` and `sethaselement` are effectively the same function, and
`contains` works with `sets` thanks to automatic HCL conversion.
2019-05-02 10:47:19 -04:00
James Bardin 3ab496d4b1 allow sets and tuples in contains function
Sets are no longer going to be automatically converted, so we need to
handle those in contains.
2019-05-01 18:13:06 -04:00
James Bardin 19bddee11b more precise types handling in coalescelist
coalescelist should accept lists and tuples, and return a dynamic types
when the arguments are not homogeneous.
2019-05-01 18:03:10 -04:00
James Bardin d186d3a490 update slice test 2019-05-01 16:57:31 -04:00
James Bardin 359f057a16 allow chunklist to handle unknowns
Chunklist should be able to return chunks containing unknown values.
2019-05-01 16:57:31 -04:00
James Bardin 93ef015336 more precise type handling in flatten
FlattenFunc can return lists and tuples when individual elements are
unknown. Only return an unknown tuple if the number of elements cannot
be determined because it contains an unknown series.

Make sure flatten can handle non-series elements, which were previously
lost due to passing a slice value as the argument.
2019-05-01 16:57:31 -04:00
James Bardin 81e04c3050 more precise type handling in slice
When slicing a list containing unknown values, the list itself is known,
and slice should return the proper slice of that list.

Make SliceFunc return the correct type for slices of tuples, and
disallow slicing sets.
2019-05-01 16:57:02 -04:00
Kristin Laemmert 8520f4e4a5
* lang/funcs/flatten: accept sets and tuples, return tuples
* lang/funcs: flatten should handle sets and tuples

* flatten now returns a tuple
2019-05-01 10:19:40 -04:00
Martin Atkins 6adcc7ab73 vendor: go get github.com/zclconf/go-cty@master
cty now guarantees that sets of primitive values will iterate in a
reasonable order. Previously it was the caller's responsibility to deal
with that, but we invariably neglected to do so, causing inconsistent
ordering. Since cty prioritizes consistent behavior over performance, it
now imposes its own sort on set elements as part of iterating over them so
that calling applications don't have to worry so much about it.

This change also causes cty to consistently push unknown and null values
in sets to the end of iteration, where before that was undefined. This
means that our diff output will now consistently list additions before
removals when showing sets, rather than the ordering being undefined as
before.

The ordering of known, non-null, non-primitive values is still not
contractually fixed but remains consistent for a particular version of
cty.
2019-04-30 15:49:28 -07:00
Kristin Laemmert 394cf7f25e
lang/funcs: add acc tests for functions (#21112)
* lang/funcs: testing of functions through the lang package API
The function-specific unit tests do not cover the HCL conversion that happens when the functions are called in a terraform configuration. For e.g., HCL converts sets to lists before passing it to the function. This means that we could not test passing a set in the function _unit_ tests.
This adds a higher-level acceptance test, plus a check that every (pure) function has a test.

* website/docs: update function documentation
2019-04-29 13:11:28 -04:00
Kit Ewbank efc08de5d6 lang/funcs: add 'strrev' interpolation function. (#21091) 2019-04-24 14:52:39 -04:00
Kristin Laemmert d4669246c7
funcs/coalesce: return the first non-null, non-empty-string element from a sequence (#21002)
* funcs/coalesce: return the first non-null, non-empty element from a
sequence.

The go-cty coalesce function, which was originally used here, returns the
first non-null element from a sequence. Terraform 0.11's coalesce,
however, returns the first non-empty string from a list of strings.

This new coalesce function aims to preserve terraform's documented
functionality while adding support for additional argument types. The
tests include those in go-cty and adapted tests from the 0.11 version of
coalesce.

* website/docs: update coalesce function document
2019-04-12 13:57:52 -04:00
Martin Atkins ffcf9fc51b lang/blocktoattr: Apply fixup within already-fixed-up blocks too
When a top-level list-of-object contains an attribute that is also
list-of-object we need to do the fixup again inside the nested body (using
our synthetic attributes-only schema) so that the attr-as-blocks mechanism
can apply within the nested blocks too.
2019-04-05 14:22:41 -07:00
Martin Atkins 4d52999538 lang: EvalBlock should use ReferencesInBlock
Previously it was calling directly to hcldec.Variables, and thus missing
the special fixups we do inside ReferencesInBlock to deal with
Terraform-specific concerns such as our attribute-as-blocks preprocessing.
2019-04-03 15:57:17 -07:00
Martin Atkins 6c5819f910 configs/configupgrade: Prefer block syntax for list-of-object attributes
In order to preserve pre-v0.12 idiom for list-of-object attributes, we'll
prefer to use block syntax for them except for the special situation where
the user explicitly assigns an empty list, where attribute syntax is
required in order to allow existing provider logic to differentiate from
an implicit lack of blocks.
2019-04-01 13:30:24 -07:00
Martin Atkins 003317d7c8 lang: Detect references when a list/set attr is defined using blocks
For compatibility with documented patterns from existing providers we are
now allowing (via a pre-processing step) any attribute whose type is a
list-of-object or set-of-object type to optionally be assigned using one
or more blocks whose type is the attribute name.

The pre-processing functionality was implemented in previous commits but
we were not correctly detecting references within these blocks that are,
from the perspective of the primary schema, invalid. Now we'll use an
alternative implementation of variable detection that is able to apply the
same schema rewriting technique we used to implement the transform and
thus can find all of the references as if they were already in their
final locations.
2019-03-28 10:41:01 -07:00
Martin Atkins 8746e9e8ad lang/blocktoattr: ExpandedVariables function
Because we handle FixUpBlockAttrs after dynamic block expansion, when
resolving variables we unfortunately need to consider the possibility of
both dynamic block expansion _and_ the block attrs fixup.

To accommodate this we have a variant of dynblock.VariablesHCLDec that
instead walks using the configschema.Block representation of the schema
and applies the same opportunistic schema rewrite used by FixUpBlockAttrs
at each body encountered during the walk.
2019-03-28 10:41:01 -07:00
Martin Atkins 87786484ea lang/eval: Apply attr-as-nested-block fixup in EvalBlock
For any block content we evaluate dynamically via this API, we'll make a
special allowance for users to optionally write members of a list
attribute instead as a sequence of nested blocks, thus allowing some
existing provider features that were assuming this capability to continue
to support it after v0.12.

This should not be used for any new provider features, and should ideally
be eventually phased out so that there aren't two
similar-but-slightly-different syntaxes for saying the same thing.
2019-03-28 10:41:01 -07:00
Martin Atkins 6dcf8195b8 lang/blocktoattr: Selectively allow block syntax to be used for attributes
This preprocessing step allows users to use nested block syntax to specify
elements of an attribute that is defined as being a list or set of an
object type.

This restores part of the unintended flexibility permitted in Terraform
v0.11 so that we can work around a few tricky edges where provider
implementations were relying on Terraform's failure to validate this in
earlier versions.

For any body that is pre-processed using this new helper, we will
recognize when a configuration author uses nested block syntax with a
name that is specified in the schema as an attribute of a suitable type
and tweak the schema just in time before decoding to expect that usage
and then fix up the result on the way out to conform to the original
schema.

Achieving this requires an abstraction inversion because only Terraform's
high-level schema has enough information to decide how to rewrite the
incoming low-level schema. We must therefore here implement HCL's
lowest-level API interface in terms of the higher-level abstractions of
hcldec and Terraform's configschema.

Because of the abstraction inversion this fixup mechanism cannot be used
generally for arbitrary HCL bodies but we can use it carefully inside the
lang package where its own API can guarantee the necessary invariants for
this to work.
2019-03-28 10:41:01 -07:00
Martin Atkins 096b1bb87b lang/funcs: Port the "reverse" function from the old functions set
This has the same functionality as the "reverse" function that was
implemented in the "config" package, but adapted to the new language type
system.
2019-03-19 17:32:19 -07:00
Martin Atkins 2b1e650df2 lang/funcs: Fix out-of-bounds ArgError in templatefile function
The templatefile function only has two arguments, so ArgErrorf can be
called with only zero or one as the argument index. If we are out of
bounds then HCL itself will panic trying to build the error message for
this call when called as an HCL function.

Unfortunately there isn't really a great layer in Terraform to test for
this class of bug systematically, because we are currently testing these
functions directly rather than going through HCL to do it. For the moment
we'll just live with that, but if we see this class of error arise again
we might consider either reworking the tests in this package to work with
HCL expression source code instead of direct calls or adding some
additional tests elsewhere that do so.
2019-03-19 16:23:51 -07:00
Martin Atkins 50a101afbd lang: Consider "dynamic" blocks when resolving references
The hcldec package has no awareness of the dynamic block extension, so the
hcldec.Variables function misses any variables declared inside dynamic
blocks.

dynblock.VariablesHCLDec is a drop-in replacement for hcldec.Variables
that _is_ aware of dynamic blocks, returning all of the same variables
that hcldec would find naturally plus also any variables used inside
the dynamic block "for_each" and "labels" arguments and inside the
nested "content" block.
2019-03-19 10:04:45 -07:00
Martin Atkins 838a42d218 vendor: go get github.com/hashicorp/hcl2@master
This includes improved functionality for HCL's "dynamic block extension",
which will allow us (in a subsequent commit) to properly detect
dependencies inside nested "dynamic" blocks, where currently they get
missed.

For this commit though, we just upgrade HCL to a version that includes it
and make a small change to our "lang" package to align with an upstream
renaming.
2019-03-19 10:04:45 -07:00
Brian Flad 398fe8652d
config: Mention other file hashing functions when file() detects unsupported contents
Reference: https://github.com/hashicorp/terraform/issues/20664
2019-03-13 13:43:58 -04:00
Brian Flad 81bdaa8c38
lang/funcs: Fix filebase64sha256 function hashing algorithm
Reference: https://github.com/hashicorp/terraform/issues/20652

The implementation and testing were incorrectly referencing SHA-512 instead of SHA-256.
2019-03-12 12:59:36 -04:00
Martin Atkins f8a6f66be4 lang/funcs: Fix panic in "join" when an element is null
It is now a proper error message.
2019-02-07 14:35:13 -08:00
Martin Atkins 954d38e870 lang: New file-hashing functions
In prior versions, we recommended using hash functions in conjunction with
the file function as an idiom for detecting changes to upstream blobs
without fetching and comparing the whole blob.

That approach relied on us being able to return raw binary data from
file(...). Since Terraform strings pass through intermediate
representations that are not binary-safe (e.g. the JSON state), there was
a risk of string corruption in prior versions which we have avoided for
0.12 by requiring that file(...) be used only with UTF-8 text files.

The specific case of returning a string and immediately passing it into
another function was not actually subject to that corruption risk, since
the HIL interpreter would just pass the string through verbatim, but this
is still now forbidden as a result of the stricter handling of file(...).

To avoid breaking these use-cases, here we introduce variants of the hash
functions a with "file" prefix that take a filename for a disk file to
hash rather than hashing the given string directly. The configuration
upgrade tool also now includes a rule to detect the documented idiom and
rewrite it into a single function call for one of these new functions.

This does cause a bit of function sprawl, but that seems preferable to
introducing more complex rules for when file(...) can and cannot read
binary files, making the behavior of these various functions easier to
understand in isolation.
2019-01-25 10:18:44 -08:00
Martin Atkins ecaaa91da9 lang/funcs: Factor out the various hash function implementations
These all follow the pattern of creating a hash and converting it to a
string using some encoding function, so we can write this implementation
only once and parameterize it with a hash factory function and an encoding
function.

This also includes a new test for the sha512 function, which was
previously missing a test and, it turns out, actually computing sha256
instead.
2019-01-25 10:18:44 -08:00
Martin Atkins 2f8f7d6f4d lang/funcs: Type conversion functions
It's not normally necessary to make explicit type conversions in Terraform
because the language implicitly converts as necessary, but explicit
conversions are useful in a few specialized cases:

- When defining output values for a reusable module, it may be desirable
  to force a "cleaner" output type than would naturally arise from a
  computation, such as forcing a string containing digits into a number.
- Our 0.12upgrade mechanism will use some of these to replace use of the
  undocumented, hidden type conversion functions in HIL, and force
  particular type interpretations in some tricky cases.
- We've found that type conversion functions can be useful as _temporary_
  workarounds for bugs in Terraform and in providers where implicit type
  conversion isn't working correctly or a type constraint isn't specified
  precisely enough for the automatic conversion behavior.

These all follow the same convention of being named "to" followed by a
short type name. Since we've had a long-standing convention of running all
the words together in lowercase in function names, we stick to that here
even though some of these names are quite strange, because these should
be rarely-used functions anyway.
2019-01-17 10:01:47 -08:00
Martin Atkins da51e72cbb lang/functions: set functions from cty
The sethaselement, setintersection, and setunion functions are defined in
the cty stdlib. Making them available in Terraform will make it easier to
work with sets, and complement the currently-Terraform-specific setproduct
function.

In the long run setproduct should probably move into the cty stdlib too,
but since it was submitted as a Terraform function originally we'll leave
it here now for simplicity's sake and reorganize later.
2019-01-16 09:57:16 -08:00