Commit Graph

29390 Commits

Author SHA1 Message Date
Martin Atkins 37b1413ab3 core: Handle root and child module input variables consistently
Previously we had a significant discrepancy between these two situations:
we wrote the raw root module variables directly into the EvalContext and
then applied type conversions only at expression evaluation time, while
for child modules we converted and validated the values while visiting
the variable graph node and wrote only the _final_ value into the
EvalContext.

This confusion seems to have been the root cause for #29899, where
validation rules for root module variables were being applied at the wrong
point in the process, prior to type conversion.

To fix that bug and also make similar mistakes less likely in the future,
I've made the root module variable handling more like the child module
variable handling in the following ways:
 - The "raw value" (exactly as given by the user) lives only in the graph
   node representing the variable, which mirrors how the _expression_
   for a child module variable lives in its graph node. This means that
   the flow for the two is the same except that there's no expression
   evaluation step for root module variables, because they arrive as
   constant values from the caller.
 - The set of variable values in the EvalContext is always only "final"
   values, after type conversion is complete. That in turn means we no
   longer need to do "just in time" conversion in
   evaluationStateData.GetInputVariable, and can just return the value
   exactly as stored, which is consistent with how we handle all other
   references between objects.

This diff is noisier than I'd like because of how much it takes to wire
a new argument (the raw variable values) through to the plan graph builder,
but those changes are pretty mechanical and the interesting logic lives
inside the plan graph builder itself, in NodeRootVariable, and
the shared helper functions in eval_variable.go.

While here I also took the opportunity to fix a historical API wart in
EvalContext, where SetModuleCallArguments was built to take a set of
variable values all at once but our current caller always calls with only
one at a time. That is now just SetModuleCallArgument singular, to match
with the new SetRootModuleArgument to deal with root module variables.
2022-01-10 12:26:54 -08:00
Martin Atkins 483c38aca1 core: Remove TestContext2Validate_PlanGraphBuilder
This test seems to be a holdover from the many-moons-ago switch from one
graph for all operations to separate graphs for plan and apply. It is
effectively just a copy of a subset of the content of the Context.Validate
function and is a maintainability hazard because it tends to lag behind
updates to that function unless changes there happen to make it fail.

This test doesn't cover anything that the other validate context tests
don't exercise as an implementation detail of calling Context.Validate,
so I've just removed it with no replacement.
2022-01-10 12:26:54 -08:00
Martin Atkins a579ae15b0
Update CHANGELOG.md 2022-01-10 12:24:53 -08:00
Martin Atkins 171e7ef6d9 core: Invalid for_each argument messaging improvements
Our original messaging here was largely just lifted from the equivalent
message for unknown values in "count", and it didn't really include any
specific advice on how to update a configuration to make for_each valid,
instead focusing only on the workaround of using the -target planning
option.

It's tough to pack in a fully-actionable suggestion here since unknown
values in for_each keys tends to be a gnarly architectural problem rather
than a local quirk -- when data flows between modules it can sometimes be
unclear whether it'll end up being used in a context which allows unknown
values.

I did my best to summarize the advice we've been giving in community forum
though, in the hope that more people will be able to address this for
themselves without asking for help, until we're one day able to smooth
this out better with a mechanism such as "partial apply".
2022-01-10 12:23:13 -08:00
Nick Fagerlund 1e9075b4fa
Merge pull request #30226 from hashicorp/nf/dec21-derandomize-dependencies
Sort dependencies when encoding `ResourceInstanceObject`
2022-01-07 15:26:07 -08:00
Laura Pacilio f4eb0ed24d
Merge pull request #30280 from Mukesh05/patch-2
Update local values definition
2022-01-07 10:22:57 -05:00
Mukesh Kumar 3ac334f267
Update website/docs/language/values/locals.mdx
Thank you. Your suggestion to use "instead of" makes the sentence even more easy to understand.

Co-authored-by: Laura Pacilio <83350965+laurapacilio@users.noreply.github.com>
2022-01-07 05:52:12 +05:30
Martin Atkins 4ec1feaa3d build: CGO_ENABLED when building for macOS
Normally when we cross-compile we do so without CGo, because we don't have
suitable C headers available for systems other than the host.

However, building for macOS on macOS is special because there are
sufficient headers available on darwin_amd64 to build for both
darwin_amd64 _and_ darwin_arm64. Also, we _must_ use CGo on macOS because
the system resolver is only available via darwin's libc, and so building
without CGo produces executables that don't resolve hostnames correctly.

This is a conditional in bash to avoid having to duplicate the entire
step. Perhaps later we'll find a more general version of this which can
avoid the special case, but this is sufficient for the moment.
2022-01-06 14:51:09 -08:00
Martin Atkins 087c2f06ee website: Documentation of how provisioners upload files
We recently made a change to how provisioners upload files in order to
address an unintended remote code execution vector when using SSH, which
revealed that we had not previously documented well enough the expected
contract for how provisioners upload files to remote systems, and so some
users were depending on unintended consequences of the the bug now fixed.

We are retaining the fix on security-related grounds, but this is a good
prompt to be clearer in the docs about what exactly Terraform is doing
when asked to upload files over SSH and WinRM, so users can understand
what is supported and write their configurations accordingly.

This also includes an additional section to the v1.1 upgrade guide, since
we apparently neglected to document this intentional breaking change in
the first draft of that page.

Of course, provisioners as a whole remain a last resort, and so we're
documenting this as hopefully a helpful aid to those who have no other
option, and not meaning in any way to recommend their use for any new
use-cases.
2022-01-06 13:04:55 -08:00
Laura Pacilio 6ef9cf652e
Merge pull request #30270 from addison-grant/patch-1
Update variables.mdx to fix typo
2022-01-06 11:14:46 -05:00
Laura Pacilio 662301610d
Merge pull request #30264 from minamijoyo/fix-typo-in-doc
docs: Fix typo in docs/plugin-protocol/releasing-new-version.md
2022-01-06 11:13:36 -05:00
Nick Fagerlund 05d0febf7f Relax test to focus on the behavior we care about (encoded == encoded)
The specific output order is meaningless, but it should always be the same after
two Encode() calls with identical (ignoring in-memory order) dependency sets.
2022-01-05 14:38:53 -08:00
Martin Atkins 28a6036cf2 build: Separate Linux distro package builds into separate job
In our build workflow we'll treat Linux distribution packaging (currently
.deb and .rpm packages) as a separate job, instead of embedding it into
the "build" job, so that this step can happen concurrently with the other
derived actions like the docker image build, and the e2etest runs.
2022-01-05 14:31:04 -08:00
Martin Atkins c1699ea80c build: Constrain permissions for the "build" workflow steps
This workflow only generates artifacts and doesn't need to modify anything
about the repository.
2022-01-05 14:31:04 -08:00
Martin Atkins 3bf758eaf5 build: Consolidate build-darwin job into just "build"
We can use an extra matrix dimension to select which execution environment
we'll use for each GOOS/GOARCH pair, and thus avoid duplicating the
job definition for darwin just to set runs-on: macos-latest for it.

This is not really an intended use of a matrix dimension because it's
directly related to the existing "goos" one, rather than being an
independent third dimension, but it doesn't matter in practice because
we're using the "include" option to specify exact combinations, and thus
we're not relying on the built-in functionality to generate all possible
matrix combinations.
2022-01-05 14:31:04 -08:00
Martin Atkins 218e55b23c build: Build docs source package as part of the release pipeline
This should eventually grow to be a step that actually verifies the
validity of the docs source prior to publishing the artifact that a
downstream publishing pipeline can consume, but for the moment it's really
just a placeholder since we have no such validation step and no downstream
pipeline consuming this artifact.

The general idea here is that the artifacts from this workflow should be
sufficient for all downstream release steps to occur without any direct
access to the Terraform CLI repository, and so this is intended to
eventually meet that ideal but as of this commit the website docs
publishing step _does_ still depend on direct access to this repository.
2022-01-05 14:31:04 -08:00
Martin Atkins b802db75d7 build: Build and run e2etest as part of the release build pipeline
This uses the decoupled build and run strategy to run the e2etests so that
we can arrange to run the tests against the real release packages produced
elsewhere in this workflow, rather than ones generated just in time by
the test harness.

The modifications to make-archive.sh here make it more consistent with its
originally-intended purpose of producing a harness for testing "real"
release executables. Our earlier compromise of making it include its own
terraform executable came from a desire to use that script as part of
manual cross-platform testing when we weren't yet set up to support
automation of those tests as we're doing here. That does mean, however,
that the terraform-e2etest package content must be combined with content
from a terraform release package in order to produce a valid contest for
running the tests.

We use a single job to cross-compile the test harness for all of the
supported platforms, because that build is relatively fast and so not
worth the overhead of matrix build, but then use a matrix build to
actually run the tests so that we can run them in a worker matching the
target platform.

We currently have access only to amd64 (x64) runners in GitHub Actions
and so for the moment this process is limited only to the subset of our
supported platforms which use that architecture.
2022-01-05 14:31:04 -08:00
Martin Atkins 6704f8c795 Initial work on a new package build process
For the moment this is just an experimental additional sidecar package
build process, separate from the one we really use for releases, so that
we can get some experience building in the GitHub Actions environment
before hopefully eventually switching to using the artifacts from this
process as the packages we'll release through the official release
channels.

It will react to any push to one of our release branches or to a release
tag by building official-release-like .zip, .deb, and .rpm packages, along
with Docker images, based on the content of the corresponding commit.

For the moment this doesn't actually produce _shippable_ packages because
in particular it doesn't know how to update our version/version.go file
to hard-code the correct version number. Once Go 1.18 is release and we've
upgraded to it we'll switch to using debug.ReadBuildInfo to determine
our version number at runtime and so no longer need to directly update
a source file for each release, but that functionality isn't yet available
in our current Go 1.17 release.
2022-01-05 14:31:04 -08:00
Alisdair McDiarmid df36a03be1 states: Add failing test for ordered dependencies 2022-01-05 14:24:03 -08:00
Alisdair McDiarmid 535da4ebc7
Merge pull request #30205 from hashicorp/alisdair/fix-show-plan-against-non-default-state
command/show: Disable plan state lineage checks
2022-01-05 12:03:28 -05:00
kmoe 73f5c7779f
Merge pull request #30292 from hashicorp/kmoe/dag-basicedge-hashcode
dags: fix BasicEdge pointer issue
2022-01-05 16:49:31 +00:00
Katy Moe f8fdb6de3f
do not use pointer addr strings as map keys in set
When creating a Set of BasicEdges, the Hashcode function is used to determine
map keys for the underlying set data structure.

The string hex representation of the two vertices' pointers is unsafe to use
as a map key, since these addresses may change between the time they are added
to the set and the time the set is operated on.

Instead we modify the Hashcode function to maintain the references to the
underlying vertices so they cannot be garbage collected during the lifetime
of the Set.
2022-01-05 11:28:47 +00:00
Dylan Staley a8bc7a0170
Merge pull request #30237 from hashicorp/ds.submodule-nav-main
Store website nav files (main)
2022-01-04 11:45:49 -08:00
James Bardin 9272ff2c29
Merge pull request #30286 from hashicorp/jbardin/dag
dag: minor cleanup
2022-01-04 12:51:21 -05:00
James Bardin 8bbba22f8c
Merge pull request #30253 from hashicorp/jbardin/move-graph
cleanup some move graph handling
2022-01-04 12:51:12 -05:00
Alisdair McDiarmid ef01d5d134
Merge pull request #30067 from hashicorp/alisdair/redact-sensitive-values-from-function-errors
lang/funcs: Redact sensitive values from function errors
2022-01-04 11:34:04 -05:00
James Bardin 344adb6c50 clarify dag comments
TransitiveReduction does not rely on having a single root, and only
must be free of cycles.

DepthFirstWalk and ReverseDepthFirstWalk do not do a topological sort,
so if order matters TransitiveReduction must be run first.
2022-01-04 10:07:31 -05:00
James Bardin fae68f166f Remove sorted walk functions
These two functions were left during a refactor to ensure the old
behavior of a sorted walk was still accessible in some manner. The
package has since been removed from any public API, and the sorted
versions are no longer called, so we can remove them.
2022-01-04 09:37:53 -05:00
James Bardin f46cf7b8bc cleanup some move graph handling
Create a separate `validateMoveStatementGraph` function so that
`ValidateMoves` and `ApplyMoves` both check the same conditions. Since
we're not using the builtin `graph.Validate` method, because we may have
multiple roots and want better cycle diagnostics, we need to add checks
for self references too. While multiple roots are an error enforced by
`Validate` for the concurrent walk, they are OK when using
`TransitiveReduction` and `ReverseDepthFirstWalk`, so we can skip that
check.

Apply moves must first use `TransitiveReduction` to reduce the graph,
otherwise nodes may be skipped if they are passed over by a transitive
edge.
2022-01-04 09:21:36 -05:00
James Bardin 22dc685052 check for nested module index changes
Changing only the index on a nested module will cause all nested moves
to create cycles, since their full addresses will match both the From
and To addresses. When building the dependency graph, check if the
parent is only changing the index of the containing module, and prevent
the backwards edge for the move.
2022-01-04 09:20:47 -05:00
James Bardin deb82daf2b find implied moves in nested modules
Implied moves in nested modules were being skipped
2022-01-04 09:20:47 -05:00
James Bardin 3d769b7282 IsModuleMoveReIndex
Add a method for checking if the From and To addresses in a move
statement are only changing the indexes of modules relative to the
statement module.

This is needed because move statement nested within the module will be
able to match against both the From and To addresses, causing cycles in
the order of move operations.
2022-01-04 09:20:47 -05:00
Martin Atkins 74761b2f8b getmodules: Use go-getter v1.5.10 and return to upstream GitGetter
There was an unintended regression in go-getter v1.5.9's GitGetter which
caused us to temporarily fork that particular getter into Terraform to
expedite a fix. However, upstream v1.5.10 now includes a
functionally-equivalent fix and so we can heal that fork by upgrading.

We'd also neglected to update the Module Sources docs when upgrading to
go-getter v1.5.9 originally and so we were missing documentation about the
new "depth" argument to enable shadow cloning, which I've added
retroactively here along with documenting its restriction of only
supporting named refs.

This new go-getter release also introduces a new credentials-passing
method for the Google Cloud Storage getter, and so we must incorporate
that into the Terraform-level documentation about module sources.
2022-01-03 11:44:16 -08:00
Mukesh Kumar 27cc49fe91
Update local values definition
The definition of local values used "it" many times, making the sentence not very clear.
2022-01-01 20:35:47 +05:30
Addison Grant 01be55b5c4
Update variables.mdx
Remove extraneous word "from" in "...use a sensitive value from as part of an..."
2021-12-27 10:25:31 -08:00
Masayuki Morita bdf683802d docs: Fix typo in docs/plugin-protocol/releasing-new-version.md 2021-12-27 14:29:02 +09:00
James Bardin 66b4d155b1
Merge pull request #30232 from hashicorp/jbardin/module-move-re-index
Handle move blocks within a module which is changing the index
2021-12-22 16:27:30 -05:00
James Bardin 75ef61c783 check for nested module index changes
Changing only the index on a nested module will cause all nested moves
to create cycles, since their full addresses will match both the From
and To addresses. When building the dependency graph, check if the
parent is only changing the index of the containing module, and prevent
the backwards edge for the move.
2021-12-22 16:15:04 -05:00
Martin Atkins e35c25da44 website: Try function documentation "provably" vs "probably" typo
This paragraph is trying to say that try only works for dynamic errors and
not for errors that are _not_ based on dynamic decision-making in
expressions.

I'm not sure if this typo was always here or if it was mistakenly "corrected"
at some point, but either way the word "probably" changes the meaning
of this sentence entirely, making it seem like Terraform is hedging
the likelihood of a problem rather than checking exactly for one.
2021-12-22 12:10:07 -08:00
Barrett Clark 296acdd961
Merge pull request #30020 from hashicorp/barrettclark/cloud-e2e-parallel
Cloud: Add parallelism back into the tests
2021-12-22 12:14:06 -06:00
Alisdair McDiarmid f772cb085e
Merge pull request #30233 from hashicorp/alisdair/move-nested-modules
refactoring: Move nested modules
2021-12-22 10:22:21 -05:00
Dylan Staley 31ed2d987c store website nav files 2021-12-21 14:52:51 -08:00
James Bardin a72d2d408d
Merge pull request #30095 from hashicorp/jbardin/invalid-provider-name
skip provider resolution when there are errors
2021-12-21 16:58:48 -05:00
James Bardin e761117562 find implied moves in nested modules
Implied moves in nested modules were being skipped
2021-12-21 16:49:25 -05:00
James Bardin 346418e31f IsModuleMoveReIndex
Add a method for checking if the From and To addresses in a move
statement are only changing the indexes of modules relative to the
statement module.

This is needed because move statement nested within the module will be
able to match against both the From and To addresses, causing cycles in
the order of move operations.
2021-12-21 16:49:25 -05:00
Alisdair McDiarmid d7ef123c12 refactoring: Move nested modules
When applying module `moved` statements by iterating through modules in
state, we previously required an exact match from the `moved`
statement's `from` field and the module address. This permitted moving
resources directly inside a module, but did not recur into module calls
within those moved modules.

This commit moves that exact match requirement so that it only applies
to `moved` statements targeting resources. In turn this allows nested
modules to be moved.
2021-12-21 16:25:06 -05:00
Laura Pacilio cd7277128f
Merge pull request #30223 from hashicorp/replace-flag-updates
Update taint command page to make alternative clearer
2021-12-21 10:42:46 -05:00
Nick Fagerlund 9b449bec99 Sort dependencies when encoding `ResourceInstanceObject`
Resource dependencies are by nature an unordered collection, but they're
persisted to state as a JSON array (in random order). This makes a mess for
`terraform apply -refresh-only`, which sees the new random order as a change
that requires the user to approve a state update.

(As an additional problem on top of that, the user interface for refresh-only
runs doesn't expect to see that as a type of change, so it says "no changes!
would you like to update to reflect these detected changes?")

This commit changes `ResourceInstanceObject.Encode()` to sort the in-memory
slice of dependencies (lexically, by address) before passing it on to be
compared and persisted. This appears to fix the observed UI issues with a
minimum of logic changes.
2021-12-20 21:46:39 -08:00
Barrett Clark d196d2870a Refactor cloud table test runs
As the cloud e2e tests evolved some common patters became apparent. This
standardizes and consolidates the patterns into a common test runner
that takes the table tests and runs them in parallel. Some tests also
needed to be converted to utilize table tests.
2021-12-20 16:36:06 -06:00
Laura Pacilio 2a530639f8 Tweaks 2021-12-20 16:01:03 -05:00