lang: "try" and "can" functions

These are intended to make it easier to work with arbitrary data
structures whose shape might not be known statically, such as the result
of jsondecode(...) or yamldecode(...) of data from a separate system.

For example, in an object value which has attributes that may or may not
be set we can concisely provide a fallback value to use when the attribute
isn't set:

    try(local.example.foo, "fallback-foo")

Using a "try to evaluate" model rather than explicit testing fits better
with the usual programming model of the Terraform language where values
are normally automatically converted to the necessary type where possible:
the given expression is subject to all of the same normal type conversions,
which avoids inadvertently creating a more restrictive evaluation model
as might happen if this were handled using checks like a hypothetical
isobject(...) function, etc.
This commit is contained in:
Martin Atkins 2020-01-03 14:01:14 -08:00
parent 5771db6696
commit 02576988c1
8 changed files with 435 additions and 0 deletions

View File

@ -3,6 +3,7 @@ package lang
import ( import (
"fmt" "fmt"
"github.com/hashicorp/hcl/v2/ext/tryfunc"
ctyyaml "github.com/zclconf/go-cty-yaml" ctyyaml "github.com/zclconf/go-cty-yaml"
"github.com/zclconf/go-cty/cty" "github.com/zclconf/go-cty/cty"
"github.com/zclconf/go-cty/cty/function" "github.com/zclconf/go-cty/cty/function"
@ -39,6 +40,7 @@ func (s *Scope) Functions() map[string]function.Function {
"base64sha256": funcs.Base64Sha256Func, "base64sha256": funcs.Base64Sha256Func,
"base64sha512": funcs.Base64Sha512Func, "base64sha512": funcs.Base64Sha512Func,
"bcrypt": funcs.BcryptFunc, "bcrypt": funcs.BcryptFunc,
"can": tryfunc.CanFunc,
"ceil": funcs.CeilFunc, "ceil": funcs.CeilFunc,
"chomp": funcs.ChompFunc, "chomp": funcs.ChompFunc,
"cidrhost": funcs.CidrHostFunc, "cidrhost": funcs.CidrHostFunc,
@ -122,6 +124,7 @@ func (s *Scope) Functions() map[string]function.Function {
"trimprefix": funcs.TrimPrefixFunc, "trimprefix": funcs.TrimPrefixFunc,
"trimspace": funcs.TrimSpaceFunc, "trimspace": funcs.TrimSpaceFunc,
"trimsuffix": funcs.TrimSuffixFunc, "trimsuffix": funcs.TrimSuffixFunc,
"try": tryfunc.TryFunc,
"upper": stdlib.UpperFunc, "upper": stdlib.UpperFunc,
"urlencode": funcs.URLEncodeFunc, "urlencode": funcs.URLEncodeFunc,
"uuid": funcs.UUIDFunc, "uuid": funcs.UUIDFunc,

View File

@ -109,6 +109,27 @@ func TestFunctions(t *testing.T) {
}, },
}, },
"can": {
{
`can(true)`,
cty.True,
},
{
// Note: "can" only works with expressions that pass static
// validation, because it only gets an opportunity to run in
// that case. The following "works" (captures the error) because
// Terraform understands it as a reference to an attribute
// that does not exist during dynamic evaluation.
//
// "can" doesn't work with references that could never possibly
// be valid and are thus caught during static validation, such
// as an expression like "foo" alone which would be understood
// as an invalid resource reference.
`can({}.baz)`,
cty.False,
},
},
"ceil": { "ceil": {
{ {
`ceil(1.2)`, `ceil(1.2)`,
@ -865,6 +886,30 @@ func TestFunctions(t *testing.T) {
}, },
}, },
"try": {
{
// Note: "try" only works with expressions that pass static
// validation, because it only gets an opportunity to run in
// that case. The following "works" (captures the error) because
// Terraform understands it as a reference to an attribute
// that does not exist during dynamic evaluation.
//
// "try" doesn't work with references that could never possibly
// be valid and are thus caught during static validation, such
// as an expression like "foo" alone which would be understood
// as an invalid resource reference. That's okay because this
// function exists primarily to ease access to dynamically-typed
// structures that Terraform can't statically validate by
// definition.
`try({}.baz, "fallback")`,
cty.StringVal("fallback"),
},
{
`try("fallback")`,
cty.StringVal("fallback"),
},
},
"upper": { "upper": {
{ {
`upper("hello")`, `upper("hello")`,

View File

@ -0,0 +1,44 @@
# "Try" and "can" functions
This Go package contains two `cty` functions intended for use in an
`hcl.EvalContext` when evaluating HCL native syntax expressions.
The first function `try` attempts to evaluate each of its argument expressions
in order until one produces a result without any errors.
```hcl
try(non_existent_variable, 2) # returns 2
```
If none of the expressions succeed, the function call fails with all of the
errors it encountered.
The second function `can` is similar except that it ignores the result of
the given expression altogether and simply returns `true` if the expression
produced a successful result or `false` if it produced errors.
Both of these are primarily intended for working with deep data structures
which might not have a dependable shape. For example, we can use `try` to
attempt to fetch a value from deep inside a data structure but produce a
default value if any step of the traversal fails:
```hcl
result = try(foo.deep[0].lots.of["traversals"], null)
```
The final result to `try` should generally be some sort of constant value that
will always evaluate successfully.
## Using these functions
Languages built on HCL can make `try` and `can` available to user code by
exporting them in the `hcl.EvalContext` used for expression evaluation:
```go
ctx := &hcl.EvalContext{
Functions: map[string]function.Function{
"try": tryfunc.TryFunc,
"can": tryfunc.CanFunc,
},
}
```

View File

@ -0,0 +1,150 @@
// Package tryfunc contains some optional functions that can be exposed in
// HCL-based languages to allow authors to test whether a particular expression
// can succeed and take dynamic action based on that result.
//
// These functions are implemented in terms of the customdecode extension from
// the sibling directory "customdecode", and so they are only useful when
// used within an HCL EvalContext. Other systems using cty functions are
// unlikely to support the HCL-specific "customdecode" extension.
package tryfunc
import (
"errors"
"fmt"
"strings"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/ext/customdecode"
"github.com/zclconf/go-cty/cty"
"github.com/zclconf/go-cty/cty/function"
)
// TryFunc is a variadic function that tries to evaluate all of is arguments
// in sequence until one succeeds, in which case it returns that result, or
// returns an error if none of them succeed.
var TryFunc function.Function
// CanFunc tries to evaluate the expression given in its first argument.
var CanFunc function.Function
func init() {
TryFunc = function.New(&function.Spec{
VarParam: &function.Parameter{
Name: "expressions",
Type: customdecode.ExpressionClosureType,
},
Type: func(args []cty.Value) (cty.Type, error) {
v, err := try(args)
if err != nil {
return cty.NilType, err
}
return v.Type(), nil
},
Impl: func(args []cty.Value, retType cty.Type) (cty.Value, error) {
return try(args)
},
})
CanFunc = function.New(&function.Spec{
Params: []function.Parameter{
{
Name: "expression",
Type: customdecode.ExpressionClosureType,
},
},
Type: function.StaticReturnType(cty.Bool),
Impl: func(args []cty.Value, retType cty.Type) (cty.Value, error) {
return can(args[0])
},
})
}
func try(args []cty.Value) (cty.Value, error) {
if len(args) == 0 {
return cty.NilVal, errors.New("at least one argument is required")
}
// We'll collect up all of the diagnostics we encounter along the way
// and report them all if none of the expressions succeed, so that the
// user might get some hints on how to make at least one succeed.
var diags hcl.Diagnostics
for _, arg := range args {
closure := customdecode.ExpressionClosureFromVal(arg)
if dependsOnUnknowns(closure.Expression, closure.EvalContext) {
// We can't safely decide if this expression will succeed yet,
// and so our entire result must be unknown until we have
// more information.
return cty.DynamicVal, nil
}
v, moreDiags := closure.Value()
diags = append(diags, moreDiags...)
if moreDiags.HasErrors() {
continue // try the next one, if there is one to try
}
return v, nil // ignore any accumulated diagnostics if one succeeds
}
// If we fall out here then none of the expressions succeeded, and so
// we must have at least one diagnostic and we'll return all of them
// so that the user can see the errors related to whichever one they
// were expecting to have succeeded in this case.
//
// Because our function must return a single error value rather than
// diagnostics, we'll construct a suitable error message string
// that will make sense in the context of the function call failure
// diagnostic HCL will eventually wrap this in.
var buf strings.Builder
buf.WriteString("no expression succeeded:\n")
for _, diag := range diags {
if diag.Subject != nil {
buf.WriteString(fmt.Sprintf("- %s (at %s)\n %s\n", diag.Summary, diag.Subject, diag.Detail))
} else {
buf.WriteString(fmt.Sprintf("- %s\n %s\n", diag.Summary, diag.Detail))
}
}
buf.WriteString("\nAt least one expression must produce a successful result")
return cty.NilVal, errors.New(buf.String())
}
func can(arg cty.Value) (cty.Value, error) {
closure := customdecode.ExpressionClosureFromVal(arg)
if dependsOnUnknowns(closure.Expression, closure.EvalContext) {
// Can't decide yet, then.
return cty.UnknownVal(cty.Bool), nil
}
_, diags := closure.Value()
if diags.HasErrors() {
return cty.False, nil
}
return cty.True, nil
}
// dependsOnUnknowns returns true if any of the variables that the given
// expression might access are unknown values or contain unknown values.
//
// This is a conservative result that prefers to return true if there's any
// chance that the expression might derive from an unknown value during its
// evaluation; it is likely to produce false-positives for more complex
// expressions involving deep data structures.
func dependsOnUnknowns(expr hcl.Expression, ctx *hcl.EvalContext) bool {
for _, traversal := range expr.Variables() {
val, diags := traversal.TraverseAbs(ctx)
if diags.HasErrors() {
// If the traversal returned a definitive error then it must
// not traverse through any unknowns.
continue
}
if !val.IsWhollyKnown() {
// The value will be unknown if either it refers directly to
// an unknown value or if the traversal moves through an unknown
// collection. We're using IsWhollyKnown, so this also catches
// situations where the traversal refers to a compound data
// structure that contains any unknown values. That's important,
// because during evaluation the expression might evaluate more
// deeply into this structure and encounter the unknowns.
return true
}
}
return false
}

1
vendor/modules.txt vendored
View File

@ -360,6 +360,7 @@ github.com/hashicorp/hcl/v2/hclparse
github.com/hashicorp/hcl/v2/gohcl github.com/hashicorp/hcl/v2/gohcl
github.com/hashicorp/hcl/v2/ext/typeexpr github.com/hashicorp/hcl/v2/ext/typeexpr
github.com/hashicorp/hcl/v2/ext/dynblock github.com/hashicorp/hcl/v2/ext/dynblock
github.com/hashicorp/hcl/v2/ext/tryfunc
github.com/hashicorp/hcl/v2/ext/customdecode github.com/hashicorp/hcl/v2/ext/customdecode
github.com/hashicorp/hcl/v2/hcltest github.com/hashicorp/hcl/v2/hcltest
# github.com/hashicorp/hil v0.0.0-20190212112733-ab17b08d6590 # github.com/hashicorp/hil v0.0.0-20190212112733-ab17b08d6590

View File

@ -0,0 +1,67 @@
---
layout: "functions"
page_title: "can - Functions - Configuration Language"
sidebar_current: "docs-funcs-conversion-can"
description: |-
The can function tries to evaluate an expression given as an argument and
indicates whether the evaluation succeeded.
---
# `can` Function
-> **Note:** This page is about Terraform 0.12 and later. For Terraform 0.11 and
earlier, see
[0.11 Configuration Language: Interpolation Syntax](../../configuration-0-11/interpolation.html).
`can` evaluates the given expression and returns a boolean value indicating
whether the expression produced a result without any errors.
This is a special function that is able to catch errors produced when evaluating
its argument. This function should be used with care, considering many of the
same caveats that apply to [`try`](./try.html), to avoid writing configurations
that are hard to read and maintain.
For most situations it's better to use [`try`](./try.html), because it allows
for more concise definition of fallback values for failing expressions.
The `can` function can only catch and handle _dynamic_ errors resulting from
access to data that isn't known until runtime. It will not catch errors
relating to expressions that can be proven to be invalid for any input, such
as a malformed resource reference.
~> **Warning:** The `can` function is intended only for concise testing of the
presence of and types of object attributes. Although it can technically accept
any sort of expression, we recommend using it only with simple attribute
references and type conversion functions as shown in the [`try`](./try.html)
examples. Overuse of `can` to suppress errors will lead to a configuration that
is hard to understand and maintain.
## Examples
```
> local.foo
{
"bar" = "baz"
}
> can(local.foo.bar)
true
> can(local.foo.boop)
false
```
The `can` function will _not_ catch errors relating to constructs that are
provably invalid even before dynamic expression evaluation, such as a malformed
reference or a reference to a top-level object that has not been declared:
```
> can(local.nonexist)
Error: Reference to undeclared local value
A local value with the name "nonexist" has not been declared.
```
## Related Functions
* [`try`](./try.html), which tries evaluating a sequence of expressions and
returns the result of the first one that succeeds.

View File

@ -0,0 +1,117 @@
---
layout: "functions"
page_title: "try - Functions - Configuration Language"
sidebar_current: "docs-funcs-conversion-try"
description: |-
The try function tries to evaluate a sequence of expressions given as
arguments and returns the result of the first one that does not produce
any errors.
---
# `try` Function
-> **Note:** This page is about Terraform 0.12 and later. For Terraform 0.11 and
earlier, see
[0.11 Configuration Language: Interpolation Syntax](../../configuration-0-11/interpolation.html).
`try` evaluates all of its argument expressions in turn and returns the result
of the first one that does not produce any errors.
This is a special function that is able to catch errors produced when evaluating
its arguments, which is particularly useful when working with complex data
structures whose shape is not well-known at implementation time.
For example, if some data is retrieved from an external system in JSON or YAML
format and then decoded, the result may have attributes that are not guaranteed
to be set. We can use `try` to produce a normalized data structure which has
a predictable type that can therefore be used more conveniently elsewhere in
the configuration:
```hcl
locals {
raw_value = yamldecode("${path.module}/example.yaml")
normalized_value = {
name = tostring(try(local.raw_value.name, null))
groups = try(local.raw_value.groups, [])
}
}
```
With the above local value expressions, configuration elsewhere in the module
can refer to `local.normalized_value` attributes without the need to repeatedly
check for and handle absent attributes that would otherwise produce errors.
We can also use `try` to deal with situations where a value might be provided
in two different forms, allowing us to normalize to the most general form:
```hcl
variable "example" {
type = any
}
locals {
example = try(
[tostring(var.example)],
tolist(var.example),
)
}
```
The above permits `var.example` to be either a list or a single string. If it's
a single string then it'll be normalized to a single-element list containing
that string, again allowing expressions elsewhere in the configuration to just
assume that `local.example` is always a list.
This second example contains two expressions that can both potentially fail.
For example, if `var.example` were set to `{}` then it could be converted to
neither a string nor a list. If `try` exhausts all of the given expressions
without any succeeding, it will return an error describing all of the problems
it encountered.
We strongly suggest using `try` only in special local values whose expressions
perform normalization, so that the error handling is confined to a single
location in the module and the rest of the module can just use straightforward
references to the normalized structure and thus be more readable for future
maintainers.
The `try` function can only catch and handle _dynamic_ errors resulting from
access to data that isn't known until runtime. It will not catch errors
relating to expressions that can be proven to be invalid for any input, such
as a malformed resource reference.
~> **Warning:** The `try` function is intended only for concise testing of the
presence of and types of object attributes. Although it can technically accept
any sort of expression, we recommend using it only with simple attribute
references and type conversion functions as shown in the examples above.
Overuse of `try` to suppress errors will lead to a configuration that is hard
to understand and maintain.
## Examples
```
> local.foo
{
"bar" = "baz"
}
> try(local.foo.bar, "fallback")
baz
> try(local.foo.boop, "fallback")
fallback
```
The `try` function will _not_ catch errors relating to constructs that are
provably invalid even before dynamic expression evaluation, such as a malformed
reference or a reference to a top-level object that has not been declared:
```
> try(local.nonexist, "fallback")
Error: Reference to undeclared local value
A local value with the name "nonexist" has not been declared.
```
## Related Functions
* [`can`](./can.html), which tries evaluating an expression and returns a
boolean value indicating whether it succeeded.

View File

@ -448,6 +448,10 @@
<a href="#docs-funcs-conversion">Type Conversion Functions</a> <a href="#docs-funcs-conversion">Type Conversion Functions</a>
<ul class="nav" id="docs-funcs-conversion"> <ul class="nav" id="docs-funcs-conversion">
<li>
<a href="/docs/configuration/functions/can.html">can</a>
</li>
<li> <li>
<a href="/docs/configuration/functions/tobool.html">tobool</a> <a href="/docs/configuration/functions/tobool.html">tobool</a>
</li> </li>
@ -472,6 +476,10 @@
<a href="/docs/configuration/functions/tostring.html">tostring</a> <a href="/docs/configuration/functions/tostring.html">tostring</a>
</li> </li>
<li>
<a href="/docs/configuration/functions/try.html">try</a>
</li>
</ul> </ul>
</li> </li>