Commit Graph

140 Commits

Author SHA1 Message Date
Martin Atkins 70fed23be5 lang/funcs: File hashing functions stream data from disk
Previously our file hashing functions were backed by the same "read file
into memory" function we use for situations like "file" and "templatefile",
meaning that they'd read the entire file into memory first and then
calculate the hash from that buffer.

All of the hash implementations we use here can calculate hashes from a
sequence of smaller buffer writes though, so there's no actual need for
us to create a file-sized temporary buffer here.

This, then, is a small refactoring of our underlying function into two
parts, where one is responsible for deciding the actual filename to load
opening it, and the other is responsible for buffering the file into
memory. Our hashing functions can then use only the first function and
skip the second.

This then allows us to use io.Copy to stream from the file into the
hashing function in smaller chunks, possibly of a size chosen by the hash
function if it happens to implement io.ReaderFrom.

The new implementation is functionally equivalent to the old but should
use less temporary memory if the user passes a large file to one of the
hashing functions.
2021-05-12 09:28:31 -07:00
Alisdair McDiarmid dbe5272931 functions: Improve marks support for lookup
Several changes to lookup to improve how we handle marked values:

- If the entire collection is marked, preserve the marks on any result
  (whether successful or fallback)
- If a returned value from the collection is marked, preserve the marks
  from only that value, combined with any overall collection marks
- Retain marks on the fallback value when it is returned, combined with
  any overall collection marks
- Include marks on the key in the result, as otherwise the result it
  ends up selecting could imply what the sensitive value was
- Retain collection marks when returning an unknown value for a not
  wholly-known collection

See also https://github.com/zclconf/go-cty/pull/98
2021-05-07 12:55:40 -04:00
Alisdair McDiarmid e0c6b3fcda functions: Improve marks support for length
Similar to cty's implementation, we only need to preserve marks from the
value itself, not any nested values it may contain. This means that
taking the length of an umarked list with marked elements results in an
unmarked number.
2021-05-07 11:57:37 -04:00
Kristin Laemmert f6af7b4f7a
lang/funcs: add (console-only) TypeFunction (#28501)
* lang/funcs: add (console-only) TypeFunction

The type() function, which is only available for terraform console,
prints out the type of a given value. This is mainly intended for
debugging - it's nice to be able to print out terraform's understanding
of a complex variable.

This introduces a new field for Scope: ConsoleMode. When ConsoleMode is true, any additional functions intended for use in the console (only) may be added.
2021-04-23 10:29:50 -04:00
Kristin Laemmert 5ae4c2f92b lang/funcs: Defaults handling of marked arguments
Defaults will now preserve marks from non-null inputs and apply marks from any default values used. I've added tests for various structural types with marks, as well as some basic unknown cases.
2021-04-21 16:09:09 -04:00
Kristin Laemmert b34588ffca lang/funcs: TransposeFunc Tests
This commit adds test cases to TestTranspose to document how this function handles marks.

The short version is that any marks anywhere will be applied to the return value, be they marks on the input map or marks on elements (either the entire list of strings, or individual elemnets of those lists).
2021-04-21 16:09:09 -04:00
Martin Atkins 14336ae6f8 lang/funcs: Conversion functions can handle sensitive values
In order to avoid updating every one of our existing functions with
explicit support for sensitive values, there's a default rule in the
functions system which makes the result of a function sensitive if any
of its arguments contain sensitive values.

We were applying that default to the various type conversion functions,
like tomap and tolist, which meant that converting a complex-typed value
with a sensitive value anywhere inside it would result in a
wholly-sensitive result.

That's unnecessarily conservative because the cty conversion layer (which
these functions are wrapping) already knows how to handle sensitivity
in a more precise way. Therefore we can opt in to handling marked values
(which Terraform uses for sensitivity) here and the only special thing
we need to do is handle errors related to sensitive values differently,
so we won't print their values out literally in case of an error (and so
that the attempt to print them out literally won't panic trying to
extract the marked values).
2021-04-19 12:10:50 -07:00
Martin Atkins 140c613ae8 lang/funcs: "one" function
In the Terraform language we typically use lists of zero or one values in
some sense interchangably with single values that might be null, because
various Terraform language constructs are designed to work with
collections rather than with nullable values.

In Terraform v0.12 we made the splat operator [*] have a "special power"
of concisely converting from a possibly-null single value into a
zero-or-one list as a way to make that common operation more concise.

In a sense this "one" function is the opposite operation to that special
power: it goes from a zero-or-one collection (list, set, or tuple) to a
possibly-null single value.

This is a concise alternative to the following clunky conditional
expression, with the additional benefit that the following expression is
also not viable for set values, and it also properly handles the case
where there's unexpectedly more than one value:

    length(var.foo) != 0 ? var.foo[0] : null

Instead, we can write:

    one(var.foo)

As with the splat operator, this is a tricky tradeoff because it could be
argued that it's not something that'd be immediately intuitive to someone
unfamiliar with Terraform. However, I think that's justified given how
often zero-or-one collections arise in typical Terraform configurations.
Unlike the splat operator, it should at least be easier to search for its
name and find its documentation the first time you see it in a
configuration.

My expectation that this will become a common pattern is also my
justification for giving it a short, concise name. Arguably it could be
better named something like "oneornull", but that's a pretty clunky name
and I'm not convinced it really adds any clarity for someone who isn't
already familiar with it.
2021-04-12 15:32:03 -07:00
Alisdair McDiarmid c1f7193454 lang/funcs: Make nonsensitive more permissive
Calling the nonsensitive function with values which are not sensitive
will result in an error. This restriction was added with the goal of
preventing confusingly redundant use of this function.

Unfortunately, this breaks when using nonsensitive to reveal the value of
sensitive resource attributes. This is because the validate walk does
not (and cannot) mark attributes as sensitive based on the schema,
because the resource value itself is unknown.

This commit therefore alters this restriction such that it permits
nonsensitive unknown values, and adds a test case to cover this specific
scenario.
2021-04-12 12:31:59 -04:00
Martin Atkins 89b2405080 lang/funcs: "sensitive" and "nonsensitive" functions
These aim to allow hinting to Terraform about situations where it's not
able to automatically infer value sensitivity.

"nonsensitive" is for situations where Terraform's behavior is too
conservative, such as when a new value is derived from a sensitive value
in such a way that all of the sensitive content is removed.

"sensitive", on the other hand, is for situations where Terraform can't
otherwise infer that a value is sensitive. These situations should be
pretty rare in a module that's making effective use of sensitive input
variables and output values, but the documentation shows one example of
an uncommon situation where a more direct hint via this function would
be needed.

Both of these functions are aimed at only occasional use in unusual
situations. They are here for reasons of pragmatism, not because we
expect them to be used routinely or recommend their use.
2021-03-16 16:26:22 -07:00
Alisdair McDiarmid 7f97bd4489 functions: Fix missing defaults for objects/tuples
If no default is specified for a nested optional structural typed
attribute, the defaults function should just pass through its input.
Before this commit the function assumed that the fallback value was
always of the correct type, which would panic.
2021-03-12 18:14:14 -05:00
Alisdair McDiarmid 0bbe583eb8 functions: Fix defaults for null objects/tuples
When using defaults with a value which contains null objects or tuples,
we cannot continue to traverse the value and apply defaults. Instead,
when we find an attribute which is null, we return early and stop
processing this branch.
2021-03-12 08:23:41 -05:00
Alisdair McDiarmid 66f8d1c1c2 functions: Fix defaults mismatched types fallback
We allow primitive fallback values which have mismatched types, but only
if there is a conversion to the target type. Previously we would allow
unsafe conversions (e.g. string to bool), but later had no capacity to
return an error if the conversion failed due to the value of the
fallback being unable to convert to the target type.

This commit makes the more conservative requirement that default
fallback values must have a safe conversion.
2021-03-04 10:43:09 -05:00
Alisdair McDiarmid 178a9b32d7 functions: Fix defaults null collections panic
When applying default values to collection types, null collections in
the input should result in empty collections in the output.
2021-03-04 10:13:41 -05:00
Alisdair McDiarmid f27dae2ab7 lang: Improved robustness of sum function
Fixes error when calling sum with values not known until apply time.
Also allows sum to cope with numbers too large to represent in float64,
along with correctly handling errors when trying to sum opposing
infinities.
2020-12-10 17:13:56 -05:00
Alisdair McDiarmid df626e898c lang/funcs: Fix alltrue/anytrue with unknowns
The alltrue/anytrue functions did not correctly handle unknown values.
This commit changes these functions so that the result is unknown if:

- The list argument is unknown
- For alltrue: any elements are unknown
- For anytrue: any elements are unknown and no known elements are true

The last change is a little subtle, so there are test cases to cover it
specifically. Examples:

- anytrue(unknown) -> unknown
- anytrue(false, unknown) -> unknown
- anytrue(false, unknown, true) -> true
2020-12-10 11:36:14 -05:00
James Bardin e08422511e lang/funcs: staticcheck 2020-12-02 13:59:19 -05:00
Martin Atkins cec4578005 lang/funcs: Experimental "defaults" function
This is a new part of the existing module_variable_optional_attrs
experiment, because it's intended to complement the ability to declare
an input variable whose type constraint is an object type with optional
attributes. Module authors can use this to replace null values (that were
either explicitly set or implied by attribute omission) with other
non-null values of the same type.

This function is a bit more type-fussy than our functions typically are
because it's intended for use primarily with input variables that have
fully-specified type constraints, and thus it uses that type information
to help inform how the defaults data structure should be interpreted.

Other uses of this function will probably be harder today because it takes
a lot of extra annotation to build a value of a specific type if it isn't
passing through a variable type constraint. Perhaps later language
features for more general type conversion will make this more applicable,
but for now the more general form of this problem is better solved other
ways.
2020-11-13 17:27:20 -08:00
Martin Atkins ae3c0c6a4a lang/funcs: Remove the deprecated "list" and "map" functions
Prior to Terraform 0.12 these two functions were the only way to construct
literal lists and maps (respectively) in HIL expressions. Terraform 0.12,
by switching to HCL 2, introduced first-class syntax for constructing
tuple and object values, which can then be converted into list and map
values using the tolist and tomap type conversion functions.

We marked both of these functions as deprecated in the Terraform v0.12
release and have since then mentioned in the docs that they will be
removed in a future Terraform version. The "terraform 0.12upgrade" tool
from Terraform v0.12 also included a rule to automatically rewrite uses
of these functions into equivalent new syntax.

The main motivation for removing these now is just to get this change made
prior to Terraform 1.0. as we'll be doing with various other deprecations.
However, a specific reason for these two functions in particular is that
their existence is what caused us to invent the idea of a "type expression"
as a distinct kind of expression in Terraform v0.12, and so removing them
now would allow potentially  unifying type expressions with value
expressions in a future release.

We do not have any current specific plans to make that change, but one
potential motivation for doing so would be to take another attempt at a
generalized "convert" function which takes a type as one of its arguments.
Our previous attempt to implement such a function was foiled by the fact
that Terraform's expression validator doesn't have any way to know to
treat one argument of a particular function as special, and so it was
generating incorrect error messages. We won't necessarily do that, but
having these "list" and "map" functions out of the way leaves the option
open.
2020-11-04 17:05:59 -08:00
Arthur Burkart d4716a69e1
lang/funcs: "anytrue" function
This is an analog to the "alltrue" function, using OR as the reduce
operator rather than AND.

This also includes some simplification of the "alltrue" implementation
to implement it similarly as a sort of reduce operation with AND
as the reduce operator, but with the same effective behavior.
2020-10-23 13:52:48 -07:00
Martin Atkins 1dc4950bfa lang/funcs: Rename the base64 character encoding functions
These were initially introduced as functions with "encode" and "decode"
prefixes, but that doesn't match with our existing convention of putting
the encoding format first so that the encode and decode functions will
group together in a alphabetically-ordered function list.

"text" is not really a defined serialization format, but it's a short word
that hopefully represents well enough what these functions are aiming to
encode and decode, while being consistent with existing functions like
jsonencode/jsondecode, yamlencode/yamldecode, etc.

The "base64" at the end here is less convincing because there is precedent
for that modifier to appear both at the beginning and the end in our
existing function names. I chose to put it at the end here because that
seems to be our emergent convention for situations where the base64
encoding is a sort of secondary modifier alongside the primary purpose
of the function, as we see with "filebase64". (base64gzip is an exception
here, but it seems outvoted by the others.)
2020-10-21 10:56:56 -07:00
r0bnet 877399c631 lang/funcs: Functions for encoding text in specific character encodings 2020-10-21 10:39:43 -07:00
OwenTuz bb39fafbe5
lang/funcs: lookup() will only treat map as unknown if it is wholly unknown (#26427)
Fix for issue #26320 - this allows us to derive known values from
partially known maps where we can, and may prevent unnecessary
destroy/rebuild cycles during apply in some cases.
2020-10-05 08:48:49 -04:00
Arthur Burkart 6ed47c7241
lang/funcs: Add "alltrue" function (#25656)
This commit adds an `alltrue` function to Terraform configuration. A
reason we might want this function is because it will enable more
powerful custom variable validations. For example:

```hcl
variable "amis" {
  type = list(object({
    id = string
  }))

  validation {
    condition = (alltrue([
      for a in var.amis : length(a.id) > 4 && substr(a.id, 0, 4) == "ami-"
    ]))
    error_message = "The ID of at least one AMI was invalid."
  }
}
```
2020-09-22 09:06:42 -04:00
Kristin Laemmert e639d813ee add test cases and remove no-longer-needed validation 2020-07-08 13:53:54 -04:00
Kristin Laemmert 384cfaacf9 lang/funcs: refactor Subnet and Host functions to support 64-bit systems 2020-07-08 10:58:06 -04:00
Martin Atkins 8f77bd344e lang/funcs: Filesystem functions hint about dynamically-generated files
The functions that interact with the filesystem are, by design, restricted
to reading files that are distributed as a static part of the
configuration, and cannot be used to interact with files that are
generated dynamically by resources in the configuration.

However, new users have often yet developed a correct mental model of how
Terraform execution works and are confused by the terse error messages
these functions return. As an interim step to help some of those users,
this just adds some more commentary to the error message which gives a
vague, generic directive to look to attributes of the resource that is
generating the file, which should (if it's designed well) export
attributes that allow the resulting file to be used effectively with
common patterns, such as checksums of the content of the generated file.

The error message here is not particularly attractive due to the
limitations of the context where it's being returned from, but I'm
accepting that here in the interest of keeping this change simple, so we
can give a hint about a case that seems to frequently generate new-user
questions. We may iterate further on the presentation of this message
later.
2020-06-24 09:02:38 -07:00
Alisdair McDiarmid a4f5e04066 lang/funcs: Add support for OpenSSH RSA key format
Previously this function only supported the x509 RSA private key format.
More recent versions of OpenSSH default to generating a new PEM key
format, which this commit now supports using the x/crypto/ssh package.

Also improve the returned error messages for various invalid ciphertext
or invalid private key errors.
2020-06-03 10:50:38 -04:00
Daniel Dreier 0749e419de
Merge pull request #24484 from ctjhoa/fix_typo
fix typo in Base64DecodeFunc log
2020-05-04 16:11:47 -07:00
Noah Mercado d4d8812afa
Feature: Sum Function (#24666)
The sum function takes a list or set of numbers and returns the sum of those
numbers.
2020-04-15 14:27:06 -04:00
Camille TJHOA d7f711f40e fix typo in Base64DecodeFunc log 2020-03-27 17:22:40 +00:00
Alisdair McDiarmid 37006c5841 lang: Fix non-string key panics in map function
The map function assumed that the key arguments were strings, and would
panic if they were not.

After this commit, calling `map(1, 2)` will result in a map `{"1" = 1}`,
and calling `map(null, 1)` will result in a syntax error.

Fixes #23346, fixes #23043
2020-03-04 10:54:55 -05:00
James Bardin 0bd40fd496
Merge pull request #24265 from hashicorp/jbardin/cty-update
Update cty to v1.3.1 and migrate to cty stdlib function where applicable
2020-03-04 09:33:51 -05:00
James Bardin d999d43483 remove old funcs code 2020-03-03 15:23:58 -05:00
Martin Atkins 67d95b97ce lang/funcs: templatefile requires valid variable names
Previously the templatefile function would permit any arbitrary string as
a variable name, but due to the HCL template syntax it would be impossible
to refer to one that isn't a valid HCL identifier without causing an
HCL syntax error.

The HCL syntax errors are correct, but don't really point to the root
cause of the problem. Instead, we'll pre-verify that the variable names
are valid before we even try to render the template, and given a
specialized error message that refers to the vars argument expression as
the problematic part, which will hopefully make the resolution path
clearer for a user encountering this situation.

The syntax error still remains for situations where all of the variable
names are correct but e.g. the user made a typo referring to one, which
makes sense because in that case the problem _is_ inside the template.
2020-02-25 10:19:46 -05:00
Martin Atkins ec9f950b3f lang/funcs: Test actual error messages from templatefile
This function has a number of different error cases with hopefully-helpful
error messages for each, so it's good to test we're getting the error
message we were actually expecting in each case.
2020-02-25 10:19:46 -05:00
James Bardin 529271e0be update merge docs to match behavior 2020-02-05 15:47:36 -05:00
James Bardin f5bf9aa55d make the merge function more precise
This PR implements 2 changes to the merge function.
 - Rather than always defining the merge return type as dynamic, return
 a precise type when all argument types match, or all possible object
 attributes are known.
 - Always return a value containing all keys when the keys are known.
 This allows the use of merge output in for_each, even when keys are yet
 to be determined.
2020-02-05 13:46:08 -05:00
George Christou 91100c003c lang/funcs: Add more `trim*` functions (#23016)
* lang/funcs: Add `trim*` functions
2019-11-18 08:31:44 -05:00
Parviz 6eaf1c23f9 lang/funcs: don't panic when transpose produces empty result 2019-11-08 09:40:39 -08:00
Martin Atkins 39e609d5fd vendor: switch to HCL 2.0 in the HCL repository
Previously we were using the experimental HCL 2 repository, but now we'll
shift over to the v2 import path within the main HCL repository as part of
actually releasing HCL 2.0 as stable.

This is a mechanical search/replace to the new import paths. It also
switches to the v2.0.0 release of HCL, which includes some new code that
Terraform didn't previously have but should not change any behavior that
matters for Terraform's purposes.

For the moment the experimental HCL2 repository is still an indirect
dependency via terraform-config-inspect, so it remains in our go.sum and
vendor directories for the moment. Because terraform-config-inspect uses
a much smaller subset of the HCL2 functionality, this does still manage
to prune the vendor directory a little. A subsequent release of
terraform-config-inspect should allow us to completely remove that old
repository in a future commit.
2019-10-02 15:10:21 -07:00
Martin Atkins f84ab99b7d lang/funcs: cidrsubnets function
This is a companion to cidrsubnet that allows bulk-allocation of multiple
subnet addresses at once, with automatic numbering.

Unlike cidrsubnet, cidrsubnets allows each of the allocations to have a
different prefix length, and will pack the networks consecutively into the
given address space. cidrsubnets can potentially create more complicated
addressing schemes than cidrsubnet alone can, because it's able to take
into account the full set of requested prefix lengths rather than just
one at a time.
2019-09-20 15:58:01 -07:00
Jeet Parekh bcc69c05bb lang/funcs: parseint function 2019-09-17 15:33:22 -07:00
Pam Selle f9ebae749c
Merge pull request #22707 from vsimon/spell
docs: Minor spelling and typo fixes
2019-09-06 11:46:01 -04:00
Vicken Simonian 853a0e0677 docs: Minor spelling and typo fixes 2019-09-05 10:08:34 -07:00
Brian Flad 19cf34114f
lang/funcs: Switch fileset() function glob implementation to github.com/bmatcuk/doublestar to support additional glob patterns
This allows the usage of the glob patterns `**` and `{alternative1,...}` to simplify Terraform configuration logic for more complex file matching.
2019-08-30 20:22:03 -04:00
Brian Flad 769f626a0e
lang/funcs: Remove homedir.Expand() and refactor path trimming with filepath.Rel() in fileset() function
Reference: https://github.com/hashicorp/terraform/pull/22621#pullrequestreview-282259385
2019-08-30 20:19:44 -04:00
Brian Flad 820b79d91d
lang/funcs: Clarification update to MakeFileSetFunc comment for filepath.ToSlash usage 2019-08-30 20:19:44 -04:00
Brian Flad af7f6ef441
lang/funcs: Update fileset() function to include path as separate first argument, automatically trim the path argument from results, and ensure results are always canonical with forward slash path separators
Reference: https://github.com/hashicorp/terraform/pull/22523#pullrequestreview-279694703

These changes center around better function usability and consistency with other functions. The function has not yet been released, so these breaking changes can be applied safely.
2019-08-30 20:19:44 -04:00
Brian Flad d48d9ed766
lang/funcs: Add fileset function
Reference: https://github.com/hashicorp/terraform/issues/16697

Enumerates a set of regular file names from a given glob pattern. Implemented via the Go stdlib `path/filepath.Glob()` functionality. Notably, stdlib does not support `**` or `{}` extended patterns. See also: https://github.com/golang/go/issues/11862

To support the extended glob patterns, it will require adding a dependency on a third party library or adding our own matching code.
2019-08-20 04:50:01 -04:00