Commit Graph

173 Commits

Author SHA1 Message Date
Martin Atkins ae3c0c6a4a lang/funcs: Remove the deprecated "list" and "map" functions
Prior to Terraform 0.12 these two functions were the only way to construct
literal lists and maps (respectively) in HIL expressions. Terraform 0.12,
by switching to HCL 2, introduced first-class syntax for constructing
tuple and object values, which can then be converted into list and map
values using the tolist and tomap type conversion functions.

We marked both of these functions as deprecated in the Terraform v0.12
release and have since then mentioned in the docs that they will be
removed in a future Terraform version. The "terraform 0.12upgrade" tool
from Terraform v0.12 also included a rule to automatically rewrite uses
of these functions into equivalent new syntax.

The main motivation for removing these now is just to get this change made
prior to Terraform 1.0. as we'll be doing with various other deprecations.
However, a specific reason for these two functions in particular is that
their existence is what caused us to invent the idea of a "type expression"
as a distinct kind of expression in Terraform v0.12, and so removing them
now would allow potentially  unifying type expressions with value
expressions in a future release.

We do not have any current specific plans to make that change, but one
potential motivation for doing so would be to take another attempt at a
generalized "convert" function which takes a type as one of its arguments.
Our previous attempt to implement such a function was foiled by the fact
that Terraform's expression validator doesn't have any way to know to
treat one argument of a particular function as special, and so it was
generating incorrect error messages. We won't necessarily do that, but
having these "list" and "map" functions out of the way leaves the option
open.
2020-11-04 17:05:59 -08:00
James Bardin 7c4d00c04c allow path and terraform in self-block eval
We can insert the terraform and path values into EvalSelfBlock, since
these are static and always known during evaluation.
2020-11-02 14:00:58 -05:00
Martin Atkins e2c64bc255 core: Annotate for_each errors with expression info
Our diagnostics model allows for optionally annotating an error or warning
with information about the expression and eval context it was generated
from, which the diagnostic renderer for the UI will then use to give the
user some additional hints about what values may have contributed to the
error.

We previously didn't have those annotations on the results of evaluating
for_each expressions though, because in that case we were using the helper
function to evaluate an expression in one shot and thus we didn't ever
have a reference to the EvalContext in order to include it in the
diagnostic values.

Now, at the expense of having to handle the evaluation at a slightly lower
level of abstraction, we'll annotate all of the for_each error messages
with source expression information. This is valuable because we see users
often confused as to how their complex for_each expressions ended up being
invalid, and hopefully giving some information about what the inputs were
will allow more users to self-solve.
2020-10-29 09:07:48 -07:00
Arthur Burkart d4716a69e1
lang/funcs: "anytrue" function
This is an analog to the "alltrue" function, using OR as the reduce
operator rather than AND.

This also includes some simplification of the "alltrue" implementation
to implement it similarly as a sort of reduce operation with AND
as the reduce operator, but with the same effective behavior.
2020-10-23 13:52:48 -07:00
Martin Atkins 1dc4950bfa lang/funcs: Rename the base64 character encoding functions
These were initially introduced as functions with "encode" and "decode"
prefixes, but that doesn't match with our existing convention of putting
the encoding format first so that the encode and decode functions will
group together in a alphabetically-ordered function list.

"text" is not really a defined serialization format, but it's a short word
that hopefully represents well enough what these functions are aiming to
encode and decode, while being consistent with existing functions like
jsonencode/jsondecode, yamlencode/yamldecode, etc.

The "base64" at the end here is less convincing because there is precedent
for that modifier to appear both at the beginning and the end in our
existing function names. I chose to put it at the end here because that
seems to be our emergent convention for situations where the base64
encoding is a sort of secondary modifier alongside the primary purpose
of the function, as we see with "filebase64". (base64gzip is an exception
here, but it seems outvoted by the others.)
2020-10-21 10:56:56 -07:00
r0bnet 877399c631 lang/funcs: Functions for encoding text in specific character encodings 2020-10-21 10:39:43 -07:00
James Bardin dd8a8bdea1 add benchmark used for DecoderSpec
Not a great benchmark, but record it here for posterity. Practical
testing with the AWS provider gives us a 98% speedup for simple plans.
2020-10-14 09:19:26 -04:00
James Bardin c48af3f18b
Merge pull request #26470 from hashicorp/jbardin/inverse-destroy-references
Allow special-case evaluation of instances pending deletion.
2020-10-05 16:20:22 -04:00
OwenTuz bb39fafbe5
lang/funcs: lookup() will only treat map as unknown if it is wholly unknown (#26427)
Fix for issue #26320 - this allows us to derive known values from
partially known maps where we can, and may prevent unnecessary
destroy/rebuild cycles during apply in some cases.
2020-10-05 08:48:49 -04:00
James Bardin 64c3a8f550 lang.Scope.EvalSelfBlock
In order to properly evaluate a destroy provisioner, we cannot rely on
the usual evaluation context, because the resource has already been
removed from the state.

EvalSelfBlock evaluates an hcl.Body in the limited scope of a single
object as "self", with the added values of "count.index" and "each.key".
2020-10-02 11:25:21 -04:00
Arthur Burkart 6ed47c7241
lang/funcs: Add "alltrue" function (#25656)
This commit adds an `alltrue` function to Terraform configuration. A
reason we might want this function is because it will enable more
powerful custom variable validations. For example:

```hcl
variable "amis" {
  type = list(object({
    id = string
  }))

  validation {
    condition = (alltrue([
      for a in var.amis : length(a.id) > 4 && substr(a.id, 0, 4) == "ami-"
    ]))
    error_message = "The ID of at least one AMI was invalid."
  }
}
```
2020-09-22 09:06:42 -04:00
Alisdair McDiarmid c7568ccceb lang: Add test for jsonencode entity escaping
The encoding/json package escapes some HTML-specific characters to
prevent JSON from being misinterpreted when in the context of an HTML
document. Terraform 0.11 used this behaviour, and to preserve backwards
compatibility we are continuing to do so moving forward.

This commit adds an explicit test to document that this is intentional,
not a bug.
2020-09-04 09:18:52 -04:00
Kristin Laemmert e639d813ee add test cases and remove no-longer-needed validation 2020-07-08 13:53:54 -04:00
Kristin Laemmert 384cfaacf9 lang/funcs: refactor Subnet and Host functions to support 64-bit systems 2020-07-08 10:58:06 -04:00
Martin Atkins 8f77bd344e lang/funcs: Filesystem functions hint about dynamically-generated files
The functions that interact with the filesystem are, by design, restricted
to reading files that are distributed as a static part of the
configuration, and cannot be used to interact with files that are
generated dynamically by resources in the configuration.

However, new users have often yet developed a correct mental model of how
Terraform execution works and are confused by the terse error messages
these functions return. As an interim step to help some of those users,
this just adds some more commentary to the error message which gives a
vague, generic directive to look to attributes of the resource that is
generating the file, which should (if it's designed well) export
attributes that allow the resulting file to be used effectively with
common patterns, such as checksums of the content of the generated file.

The error message here is not particularly attractive due to the
limitations of the context where it's being returned from, but I'm
accepting that here in the interest of keeping this change simple, so we
can give a hint about a case that seems to frequently generate new-user
questions. We may iterate further on the presentation of this message
later.
2020-06-24 09:02:38 -07:00
Alisdair McDiarmid 80ca4db54d lang: Add regression test for yamldecode bug 2020-06-17 13:05:14 -04:00
Alisdair McDiarmid a4f5e04066 lang/funcs: Add support for OpenSSH RSA key format
Previously this function only supported the x509 RSA private key format.
More recent versions of OpenSSH default to generating a new PEM key
format, which this commit now supports using the x/crypto/ssh package.

Also improve the returned error messages for various invalid ciphertext
or invalid private key errors.
2020-06-03 10:50:38 -04:00
Daniel Dreier 0749e419de
Merge pull request #24484 from ctjhoa/fix_typo
fix typo in Base64DecodeFunc log
2020-05-04 16:11:47 -07:00
James Bardin 6c0f7703a6
Merge pull request #24697 from hashicorp/jbardin/get-module-data
Always return all module instances during evaluation
2020-04-22 09:49:45 -04:00
Noah Mercado d4d8812afa
Feature: Sum Function (#24666)
The sum function takes a list or set of numbers and returns the sum of those
numbers.
2020-04-15 14:27:06 -04:00
James Bardin 42cee86ee2 remove GetModuleInstanceOutput
There is no codepath that can use this any longer, since we need to
evaluate the modules as whole objects.

This means we're going to have to live for now with invalid module
output references returning "object" errors rather that "module".
2020-04-14 14:49:10 -04:00
James Bardin 600d4c3e1f eval Data needs to operate on whole modules
In order to be able to use module values, and handle operations like
possibly invalid module indexes in conditional statements, whole modules
must always be returned during evaluation.
2020-04-12 10:50:31 -04:00
Camille TJHOA d7f711f40e fix typo in Base64DecodeFunc log 2020-03-27 17:22:40 +00:00
Pam Selle f738f85241 Create non-specific ModuleCallOutput 2020-03-26 13:29:38 -04:00
Alisdair McDiarmid 37006c5841 lang: Fix non-string key panics in map function
The map function assumed that the key arguments were strings, and would
panic if they were not.

After this commit, calling `map(1, 2)` will result in a map `{"1" = 1}`,
and calling `map(null, 1)` will result in a syntax error.

Fixes #23346, fixes #23043
2020-03-04 10:54:55 -05:00
James Bardin 0bd40fd496
Merge pull request #24265 from hashicorp/jbardin/cty-update
Update cty to v1.3.1 and migrate to cty stdlib function where applicable
2020-03-04 09:33:51 -05:00
James Bardin d999d43483 remove old funcs code 2020-03-03 15:23:58 -05:00
James Bardin d0d85f909a convert cty func calls to stdlib 2020-03-03 15:23:58 -05:00
Martin Atkins 67d95b97ce lang/funcs: templatefile requires valid variable names
Previously the templatefile function would permit any arbitrary string as
a variable name, but due to the HCL template syntax it would be impossible
to refer to one that isn't a valid HCL identifier without causing an
HCL syntax error.

The HCL syntax errors are correct, but don't really point to the root
cause of the problem. Instead, we'll pre-verify that the variable names
are valid before we even try to render the template, and given a
specialized error message that refers to the vars argument expression as
the problematic part, which will hopefully make the resolution path
clearer for a user encountering this situation.

The syntax error still remains for situations where all of the variable
names are correct but e.g. the user made a typo referring to one, which
makes sense because in that case the problem _is_ inside the template.
2020-02-25 10:19:46 -05:00
Martin Atkins ec9f950b3f lang/funcs: Test actual error messages from templatefile
This function has a number of different error cases with hopefully-helpful
error messages for each, so it's good to test we're getting the error
message we were actually expecting in each case.
2020-02-25 10:19:46 -05:00
James Bardin a765d69fb0
Merge pull request #24032 from hashicorp/jbardin/map-funcs
make the merge function more precise
2020-02-12 10:51:05 -05:00
James Goodhouse 25bfe7337b
lang: add setsubtract function (#23424)
* add setdifference and setsubtract functions and docs
* remove setdifference as it is not implemented correct in underlying lib

* Update setintersection.html.md
* Update setproduct.html.md
* Update setunion.html.md
2020-02-06 12:49:11 -05:00
James Bardin 529271e0be update merge docs to match behavior 2020-02-05 15:47:36 -05:00
James Bardin f5bf9aa55d make the merge function more precise
This PR implements 2 changes to the merge function.
 - Rather than always defining the merge return type as dynamic, return
 a precise type when all argument types match, or all possible object
 attributes are known.
 - Always return a value containing all keys when the keys are known.
 This allows the use of merge output in for_each, even when keys are yet
 to be determined.
2020-02-05 13:46:08 -05:00
Martin Atkins 02576988c1 lang: "try" and "can" functions
These are intended to make it easier to work with arbitrary data
structures whose shape might not be known statically, such as the result
of jsondecode(...) or yamldecode(...) of data from a separate system.

For example, in an object value which has attributes that may or may not
be set we can concisely provide a fallback value to use when the attribute
isn't set:

    try(local.example.foo, "fallback-foo")

Using a "try to evaluate" model rather than explicit testing fits better
with the usual programming model of the Terraform language where values
are normally automatically converted to the necessary type where possible:
the given expression is subject to all of the same normal type conversions,
which avoids inadvertently creating a more restrictive evaluation model
as might happen if this were handled using checks like a hypothetical
isobject(...) function, etc.
2020-01-10 15:23:25 -08:00
George Christou 23fc68cc91
lang: Fix new `trim*` function mappings
Fixes #23413
2019-11-19 10:13:14 +00:00
George Christou 91100c003c lang/funcs: Add more `trim*` functions (#23016)
* lang/funcs: Add `trim*` functions
2019-11-18 08:31:44 -05:00
Parviz 6eaf1c23f9 lang/funcs: don't panic when transpose produces empty result 2019-11-08 09:40:39 -08:00
James Bardin 3839405d3d correctly evaluate self in for_each resources
The fallback type for GetResource from an EachMap is a cty.Object,
because resource schemas may contain dynamically typed attributes.
Check for an Object type in the evaluation of self, to use the proper
GetAttr method when extracting the value.
2019-10-29 12:44:42 -04:00
James Bardin 1ee851f256
Merge pull request #22846 from hashicorp/jbardin/evaluate-resource
Always evaluate resources in their entirety
2019-10-08 07:57:15 -04:00
James Bardin bfce78064b lang/eval: more evalContext fixups
self references do not need to be added to `managedResource`, and in
fact that could cause issues later if self is allowed in contexts other
than managed resources.

Coalesce 2 cases in the Referenceable switch, be take the
ContainingResource address of an instance beforehand.
2019-10-07 18:13:20 -04:00
Martin Atkins 39e609d5fd vendor: switch to HCL 2.0 in the HCL repository
Previously we were using the experimental HCL 2 repository, but now we'll
shift over to the v2 import path within the main HCL repository as part of
actually releasing HCL 2.0 as stable.

This is a mechanical search/replace to the new import paths. It also
switches to the v2.0.0 release of HCL, which includes some new code that
Terraform didn't previously have but should not change any behavior that
matters for Terraform's purposes.

For the moment the experimental HCL2 repository is still an indirect
dependency via terraform-config-inspect, so it remains in our go.sum and
vendor directories for the moment. Because terraform-config-inspect uses
a much smaller subset of the HCL2 functionality, this does still manage
to prune the vendor directory a little. A subsequent release of
terraform-config-inspect should allow us to completely remove that old
repository in a future commit.
2019-10-02 15:10:21 -07:00
Martin Atkins f84ab99b7d lang/funcs: cidrsubnets function
This is a companion to cidrsubnet that allows bulk-allocation of multiple
subnet addresses at once, with automatic numbering.

Unlike cidrsubnet, cidrsubnets allows each of the allocations to have a
different prefix length, and will pack the networks consecutively into the
given address space. cidrsubnets can potentially create more complicated
addressing schemes than cidrsubnet alone can, because it's able to take
into account the full set of requested prefix lengths rather than just
one at a time.
2019-09-20 15:58:01 -07:00
James Bardin d4031918d1 parse resource refs correctly
Now that we only evaluate whole resources, we can parse resource refs
correct as the resource, rather than an unknown instance.
2019-09-19 11:46:09 -04:00
James Bardin c49f976308 change lang eval to also only lookup resources
Continue only evaluating resource at a whole and push the indexing of
the resource down into the expression evaluation.

The exception here is that `self` must be an instance which must be
extracted from the resource. We now also add the entire resource to the
context, which was previously only partially populated with the self
referenced instance.
2019-09-19 11:46:09 -04:00
James Bardin 86bf674246 change GetResourceInstance to GetResource
In order to allow lazy evaluation of resource indexes, we can't index
resources immediately via GetResourceInstance. Change the evaluation to
always return whole Resources via GetResource, and index individual
instances during expression evaluation.

This will allow us to always check for invalid index errors rather than
returning an unknown value and ignoring it during apply.
2019-09-19 09:19:14 -04:00
Jeet Parekh bcc69c05bb lang/funcs: parseint function 2019-09-17 15:33:22 -07:00
Pam Selle f9ebae749c
Merge pull request #22707 from vsimon/spell
docs: Minor spelling and typo fixes
2019-09-06 11:46:01 -04:00
Vicken Simonian 853a0e0677 docs: Minor spelling and typo fixes 2019-09-05 10:08:34 -07:00
Brian Flad 19cf34114f
lang/funcs: Switch fileset() function glob implementation to github.com/bmatcuk/doublestar to support additional glob patterns
This allows the usage of the glob patterns `**` and `{alternative1,...}` to simplify Terraform configuration logic for more complex file matching.
2019-08-30 20:22:03 -04:00