Commit Graph

109 Commits

Author SHA1 Message Date
Alisdair McDiarmid 37006c5841 lang: Fix non-string key panics in map function
The map function assumed that the key arguments were strings, and would
panic if they were not.

After this commit, calling `map(1, 2)` will result in a map `{"1" = 1}`,
and calling `map(null, 1)` will result in a syntax error.

Fixes #23346, fixes #23043
2020-03-04 10:54:55 -05:00
James Bardin 0bd40fd496
Merge pull request #24265 from hashicorp/jbardin/cty-update
Update cty to v1.3.1 and migrate to cty stdlib function where applicable
2020-03-04 09:33:51 -05:00
James Bardin d999d43483 remove old funcs code 2020-03-03 15:23:58 -05:00
Martin Atkins 67d95b97ce lang/funcs: templatefile requires valid variable names
Previously the templatefile function would permit any arbitrary string as
a variable name, but due to the HCL template syntax it would be impossible
to refer to one that isn't a valid HCL identifier without causing an
HCL syntax error.

The HCL syntax errors are correct, but don't really point to the root
cause of the problem. Instead, we'll pre-verify that the variable names
are valid before we even try to render the template, and given a
specialized error message that refers to the vars argument expression as
the problematic part, which will hopefully make the resolution path
clearer for a user encountering this situation.

The syntax error still remains for situations where all of the variable
names are correct but e.g. the user made a typo referring to one, which
makes sense because in that case the problem _is_ inside the template.
2020-02-25 10:19:46 -05:00
Martin Atkins ec9f950b3f lang/funcs: Test actual error messages from templatefile
This function has a number of different error cases with hopefully-helpful
error messages for each, so it's good to test we're getting the error
message we were actually expecting in each case.
2020-02-25 10:19:46 -05:00
James Bardin 529271e0be update merge docs to match behavior 2020-02-05 15:47:36 -05:00
James Bardin f5bf9aa55d make the merge function more precise
This PR implements 2 changes to the merge function.
 - Rather than always defining the merge return type as dynamic, return
 a precise type when all argument types match, or all possible object
 attributes are known.
 - Always return a value containing all keys when the keys are known.
 This allows the use of merge output in for_each, even when keys are yet
 to be determined.
2020-02-05 13:46:08 -05:00
George Christou 91100c003c lang/funcs: Add more `trim*` functions (#23016)
* lang/funcs: Add `trim*` functions
2019-11-18 08:31:44 -05:00
Parviz 6eaf1c23f9 lang/funcs: don't panic when transpose produces empty result 2019-11-08 09:40:39 -08:00
Martin Atkins 39e609d5fd vendor: switch to HCL 2.0 in the HCL repository
Previously we were using the experimental HCL 2 repository, but now we'll
shift over to the v2 import path within the main HCL repository as part of
actually releasing HCL 2.0 as stable.

This is a mechanical search/replace to the new import paths. It also
switches to the v2.0.0 release of HCL, which includes some new code that
Terraform didn't previously have but should not change any behavior that
matters for Terraform's purposes.

For the moment the experimental HCL2 repository is still an indirect
dependency via terraform-config-inspect, so it remains in our go.sum and
vendor directories for the moment. Because terraform-config-inspect uses
a much smaller subset of the HCL2 functionality, this does still manage
to prune the vendor directory a little. A subsequent release of
terraform-config-inspect should allow us to completely remove that old
repository in a future commit.
2019-10-02 15:10:21 -07:00
Martin Atkins f84ab99b7d lang/funcs: cidrsubnets function
This is a companion to cidrsubnet that allows bulk-allocation of multiple
subnet addresses at once, with automatic numbering.

Unlike cidrsubnet, cidrsubnets allows each of the allocations to have a
different prefix length, and will pack the networks consecutively into the
given address space. cidrsubnets can potentially create more complicated
addressing schemes than cidrsubnet alone can, because it's able to take
into account the full set of requested prefix lengths rather than just
one at a time.
2019-09-20 15:58:01 -07:00
Jeet Parekh bcc69c05bb lang/funcs: parseint function 2019-09-17 15:33:22 -07:00
Pam Selle f9ebae749c
Merge pull request #22707 from vsimon/spell
docs: Minor spelling and typo fixes
2019-09-06 11:46:01 -04:00
Vicken Simonian 853a0e0677 docs: Minor spelling and typo fixes 2019-09-05 10:08:34 -07:00
Brian Flad 19cf34114f
lang/funcs: Switch fileset() function glob implementation to github.com/bmatcuk/doublestar to support additional glob patterns
This allows the usage of the glob patterns `**` and `{alternative1,...}` to simplify Terraform configuration logic for more complex file matching.
2019-08-30 20:22:03 -04:00
Brian Flad 769f626a0e
lang/funcs: Remove homedir.Expand() and refactor path trimming with filepath.Rel() in fileset() function
Reference: https://github.com/hashicorp/terraform/pull/22621#pullrequestreview-282259385
2019-08-30 20:19:44 -04:00
Brian Flad 820b79d91d
lang/funcs: Clarification update to MakeFileSetFunc comment for filepath.ToSlash usage 2019-08-30 20:19:44 -04:00
Brian Flad af7f6ef441
lang/funcs: Update fileset() function to include path as separate first argument, automatically trim the path argument from results, and ensure results are always canonical with forward slash path separators
Reference: https://github.com/hashicorp/terraform/pull/22523#pullrequestreview-279694703

These changes center around better function usability and consistency with other functions. The function has not yet been released, so these breaking changes can be applied safely.
2019-08-30 20:19:44 -04:00
Brian Flad d48d9ed766
lang/funcs: Add fileset function
Reference: https://github.com/hashicorp/terraform/issues/16697

Enumerates a set of regular file names from a given glob pattern. Implemented via the Go stdlib `path/filepath.Glob()` functionality. Notably, stdlib does not support `**` or `{}` extended patterns. See also: https://github.com/golang/go/issues/11862

To support the extended glob patterns, it will require adding a dependency on a third party library or adding our own matching code.
2019-08-20 04:50:01 -04:00
Alex Pilon e2eb70b737
remove satori/go.uuid in favor of google/uuid 2019-08-09 15:09:41 -04:00
Kristin Laemmert 2b14a6b332
lang/funcs: lookup() can work with maps of lists, maps and objects (#22269)
* lang/funcs: lookup() can work with maps of lists, maps and objects

lookup() can already handle aribtrary objects of (whatever) and should
handle maps of (whatever) similarly.
2019-08-01 15:52:52 -04:00
Radek Simko 5c30bafee0
Merge pull request #22119 from hashicorp/b-chunklist-empty-list
lang/funcs: Pass through empty list in chunklist
2019-07-18 20:59:47 +01:00
Radek Simko 736fda1613
lang/funcs: Pass through empty list in chunklist 2019-07-18 15:01:00 +01:00
Saverio Miroddi 983af7a183 Allow null values in Compact function
The function would previously panic when one or more null values were among the arguments.

The new behavior treats nulls as empty strings, therefore, it removes them.
2019-07-11 20:16:24 +02:00
Andreas Sommer 042aead714 lang/funcs: add "abspath" function (#21409) 2019-07-02 08:30:30 -04:00
Paul Tyng ebea080aca Add missing bool case
Fixes #20572
2019-06-24 15:27:59 -04:00
Pam Selle 1255ad6722 Don't allow nulls in calls to merge 2019-06-14 15:26:24 -04:00
Lars Eric Scheidler aa07806bfc lang/funcs: New "uuidv5" function
This generates name-based uuids, rather than pseudorandom uuids as with the
"uuid" function.
2019-06-07 14:38:22 -07:00
Kristin Laemmert c163ba71db simplify 2019-06-04 11:54:26 -04:00
Kristin Laemmert 30a924e162 lang/funcs: `matchkeys` - unify type for keys and searchset
Added higher-level test for matchkeys to exercise mixing
types in searchset. This had to be in the functions tests so the HCL
auto conversion from tuple to list would occur.
2019-06-04 08:57:02 -04:00
Kristin Laemmert f2a14d7c18 lang/funcs: fix error when `matchkeys` encountered a variable
`matchkeys` was returning a (false) error if the searchset was a
variable, since then the type of the keylist and searchset parameters
would not match.

This does slightly change the behavior: previously matchkeys would
produce an error if the parameters were not of the same type, for e.g.
if searchset was a list of strings and keylist was a list of integers.
  This no longer produces an error.
2019-06-03 18:00:53 -04:00
Martin Atkins 55794eb658 lang/funcs: Don't panic when distinct called with empty list 2019-05-31 11:44:31 -05:00
Martin Atkins 742deca3e9 lang/funcs: Short-circuit if start or end index is unknown
Previously the type-selection codepath for an input tuple referred
unconditionally to the start and end index values. In the Type callback,
only the types of the arguments are guaranteed to be known, so any access
to the values must be guarded with an .IsKnown check, or else the function
will not short-circuit properly when an unknown value is passed.

Now we will check the start and end indices are in range when we have
enough information to do so, but we'll return an approximate result if
either is unknown.
2019-05-17 08:48:02 -07:00
Kristin Laemmert b1d0b1383f
lang/funcs: remove sethaselement function and documentation (#21164)
`contains` and `sethaselement` are effectively the same function, and
`contains` works with `sets` thanks to automatic HCL conversion.
2019-05-02 10:47:19 -04:00
James Bardin 3ab496d4b1 allow sets and tuples in contains function
Sets are no longer going to be automatically converted, so we need to
handle those in contains.
2019-05-01 18:13:06 -04:00
James Bardin 19bddee11b more precise types handling in coalescelist
coalescelist should accept lists and tuples, and return a dynamic types
when the arguments are not homogeneous.
2019-05-01 18:03:10 -04:00
James Bardin 359f057a16 allow chunklist to handle unknowns
Chunklist should be able to return chunks containing unknown values.
2019-05-01 16:57:31 -04:00
James Bardin 93ef015336 more precise type handling in flatten
FlattenFunc can return lists and tuples when individual elements are
unknown. Only return an unknown tuple if the number of elements cannot
be determined because it contains an unknown series.

Make sure flatten can handle non-series elements, which were previously
lost due to passing a slice value as the argument.
2019-05-01 16:57:31 -04:00
James Bardin 81e04c3050 more precise type handling in slice
When slicing a list containing unknown values, the list itself is known,
and slice should return the proper slice of that list.

Make SliceFunc return the correct type for slices of tuples, and
disallow slicing sets.
2019-05-01 16:57:02 -04:00
Kristin Laemmert 8520f4e4a5
* lang/funcs/flatten: accept sets and tuples, return tuples
* lang/funcs: flatten should handle sets and tuples

* flatten now returns a tuple
2019-05-01 10:19:40 -04:00
Martin Atkins 6adcc7ab73 vendor: go get github.com/zclconf/go-cty@master
cty now guarantees that sets of primitive values will iterate in a
reasonable order. Previously it was the caller's responsibility to deal
with that, but we invariably neglected to do so, causing inconsistent
ordering. Since cty prioritizes consistent behavior over performance, it
now imposes its own sort on set elements as part of iterating over them so
that calling applications don't have to worry so much about it.

This change also causes cty to consistently push unknown and null values
in sets to the end of iteration, where before that was undefined. This
means that our diff output will now consistently list additions before
removals when showing sets, rather than the ordering being undefined as
before.

The ordering of known, non-null, non-primitive values is still not
contractually fixed but remains consistent for a particular version of
cty.
2019-04-30 15:49:28 -07:00
Kristin Laemmert 394cf7f25e
lang/funcs: add acc tests for functions (#21112)
* lang/funcs: testing of functions through the lang package API
The function-specific unit tests do not cover the HCL conversion that happens when the functions are called in a terraform configuration. For e.g., HCL converts sets to lists before passing it to the function. This means that we could not test passing a set in the function _unit_ tests.
This adds a higher-level acceptance test, plus a check that every (pure) function has a test.

* website/docs: update function documentation
2019-04-29 13:11:28 -04:00
Kristin Laemmert d4669246c7
funcs/coalesce: return the first non-null, non-empty-string element from a sequence (#21002)
* funcs/coalesce: return the first non-null, non-empty element from a
sequence.

The go-cty coalesce function, which was originally used here, returns the
first non-null element from a sequence. Terraform 0.11's coalesce,
however, returns the first non-empty string from a list of strings.

This new coalesce function aims to preserve terraform's documented
functionality while adding support for additional argument types. The
tests include those in go-cty and adapted tests from the 0.11 version of
coalesce.

* website/docs: update coalesce function document
2019-04-12 13:57:52 -04:00
Martin Atkins 096b1bb87b lang/funcs: Port the "reverse" function from the old functions set
This has the same functionality as the "reverse" function that was
implemented in the "config" package, but adapted to the new language type
system.
2019-03-19 17:32:19 -07:00
Martin Atkins 2b1e650df2 lang/funcs: Fix out-of-bounds ArgError in templatefile function
The templatefile function only has two arguments, so ArgErrorf can be
called with only zero or one as the argument index. If we are out of
bounds then HCL itself will panic trying to build the error message for
this call when called as an HCL function.

Unfortunately there isn't really a great layer in Terraform to test for
this class of bug systematically, because we are currently testing these
functions directly rather than going through HCL to do it. For the moment
we'll just live with that, but if we see this class of error arise again
we might consider either reworking the tests in this package to work with
HCL expression source code instead of direct calls or adding some
additional tests elsewhere that do so.
2019-03-19 16:23:51 -07:00
Brian Flad 398fe8652d
config: Mention other file hashing functions when file() detects unsupported contents
Reference: https://github.com/hashicorp/terraform/issues/20664
2019-03-13 13:43:58 -04:00
Brian Flad 81bdaa8c38
lang/funcs: Fix filebase64sha256 function hashing algorithm
Reference: https://github.com/hashicorp/terraform/issues/20652

The implementation and testing were incorrectly referencing SHA-512 instead of SHA-256.
2019-03-12 12:59:36 -04:00
Martin Atkins f8a6f66be4 lang/funcs: Fix panic in "join" when an element is null
It is now a proper error message.
2019-02-07 14:35:13 -08:00
Martin Atkins 954d38e870 lang: New file-hashing functions
In prior versions, we recommended using hash functions in conjunction with
the file function as an idiom for detecting changes to upstream blobs
without fetching and comparing the whole blob.

That approach relied on us being able to return raw binary data from
file(...). Since Terraform strings pass through intermediate
representations that are not binary-safe (e.g. the JSON state), there was
a risk of string corruption in prior versions which we have avoided for
0.12 by requiring that file(...) be used only with UTF-8 text files.

The specific case of returning a string and immediately passing it into
another function was not actually subject to that corruption risk, since
the HIL interpreter would just pass the string through verbatim, but this
is still now forbidden as a result of the stricter handling of file(...).

To avoid breaking these use-cases, here we introduce variants of the hash
functions a with "file" prefix that take a filename for a disk file to
hash rather than hashing the given string directly. The configuration
upgrade tool also now includes a rule to detect the documented idiom and
rewrite it into a single function call for one of these new functions.

This does cause a bit of function sprawl, but that seems preferable to
introducing more complex rules for when file(...) can and cannot read
binary files, making the behavior of these various functions easier to
understand in isolation.
2019-01-25 10:18:44 -08:00
Martin Atkins ecaaa91da9 lang/funcs: Factor out the various hash function implementations
These all follow the pattern of creating a hash and converting it to a
string using some encoding function, so we can write this implementation
only once and parameterize it with a hash factory function and an encoding
function.

This also includes a new test for the sha512 function, which was
previously missing a test and, it turns out, actually computing sha256
instead.
2019-01-25 10:18:44 -08:00