Commit Graph

29 Commits

Author SHA1 Message Date
Martin Atkins 4e74a7a4f1 initwd: Error message for local paths escaping module packages
Our module installer has a somewhat-informal idea of a "module package",
which is some external thing we can go fetch in order to add one or more
modules to the current configuration. Our documentation doesn't talk much
about it because most users seem to have found the distinction between
external and local modules pretty intuitive without us throwing a lot of
funny terminology at them, but there are some situations where the
distinction between a module and a module package are material to the
end-user.

One such situation is when using an absolute rather than relative
filesystem path: we treat that as an external package in order to make the
resulting working directory theoretically "portable" (although users can
do various other things to defeat that), and so Terraform will copy the
directory into .terraform/modules in the same way as it would download and
extract a remote archive package or clone a git repository.

A consequence of this, though, is that any relative paths called from
inside a module loaded from an absolute path will fail if they try to
traverse upward into the parent directory, because at runtime we're
actually running from a copy of the directory that's been taking out of
its original context.

A similar sort of situation can occur in a truly remote module package if
the author accidentally writes a "../" source path that traverses up out
of the package root, and so this commit introduces a special error message
for both situations that tries to be a bit clearer about there being a
package boundary and use that to explain why installation failed.

We would ideally have made escaping local references like that illegal in
the first place, but sadly we did not and so when we rebuilt the module
installer for Terraform v0.12 we ended up keeping the previous behavior of
just trying it and letting it succeed if there happened to somehow be a
matching directory at the given path, in order to remain compatible with
situations that had worked by coincidence rather than intention. For that
same reason, I've implemented this as a replacement error message we will
return only if local module installation was going to fail anyway, and
thus it only modifies the error message for some existing error situations
rather than introducing new error situations.

This also includes some light updates to the documentation to say a little
more about how Terraform treats absolute paths, though aiming not to get
too much into the weeds about module packages since it's something that
most users can get away with never knowing.
2021-05-27 11:00:43 -07:00
Martin Atkins 31349a9c3a Move configs/ to internal/configs/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins b9a93a0fe7 Move addrs/ to internal/addrs/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 05caff2ca3 Move tfdiags/ to internal/tfdiags/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 4c5b866dea Move registry/ to internal/registry/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
James Bardin 0b31ffa587 use a single log writer
Use a single log writer instance for all std library logging.

Setup the std log writer in the logging package, and remove boilerplate
from test packages.
2020-10-19 14:29:54 -04:00
James Bardin 6ca477f042 move helper/logging to internal
remove a dead code file too
2020-10-19 14:27:53 -04:00
Kristin Laemmert 04be220f5f deprecate helper/copy
helper/copy CopyDir was used heavily in tests. It differes from
internal/copydir in a few ways, the main one being that it creates the
dst directory while the internal version expected the dst to exist
(there are other differences, which is why I did not just switch tests
to using internal's CopyDir).

I moved the CopyDir func from helper/copy into command_test.go; I could
also have moved it into internal/copy and named it something like
CreateDirAndCopy so if that seems like a better option please let me
know.

helper/copy/CopyFile was used in a couple of spots so I moved it into
internal, at which point I thought it made more sense to rename the
package copy (instead of copydir).

There's also a `go mod tidy` included.
2020-10-08 08:42:16 -04:00
Alan D. Salewski f63c38d6a1
internal/initwd: allow tests to pass when $PWD contains symlinks (#26015)
Fixes #26014
2020-08-27 10:02:22 -04:00
Kristin Laemmert 1b8f4566fa
internal/initwd: fix panics with relative submodules in DirFromModule (#25250)
* internal/initwd: fix panics with relative submodules in DirFromModule

There were two related issues here:

1. panic with any local module with submodules
1. panic with a relative directory that was above the workdir ("../")

The first panic was caused by the local installer looking up the root
module with the (nonexistant) key "root.", instead of "".

The second panic was caused by the installer trying to determine the
relative path from ".". This was fixed by detecting "." as the source
path and using the absolute path for the call to filepath.Rel.

Added test cases for both panics and updated the existing e2e tests with
the correct install paths.
2020-06-17 13:24:56 -04:00
Alisdair McDiarmid 62d826e066 command/init: Use full config for provider reqs
Relying on the early config for provider requirements was necessary in
Terraform 0.12, to allow the 0.12upgrade command to run after init
installs providers.

However in 0.13, the same restrictions do not apply, and the detection
of provider requirements has changed. As a result, the early config
loader gives incorrect provider requirements in some circumstances,
such as those in the new test in this commit.

Therefore we are changing the init command to use the requirements found
by the full configuration loader. This also means that we can remove the
internal initwd CheckCoreVersionRequirements function.
2020-05-25 16:50:12 -04:00
Martin Atkins 072c6d9aed internal/copydir: Factor out our recursive directory copy for reuse
We've previously been copying this function around so it could remain
unexported while being used in various packages. However, it's a
non-trivial function with lots of specific assumptions built into it, so
here we'll put it somewhere that other packages can depend on it _and_
document the assumptions it seems to be making for future reference.

As a bonus, this now uses os.SameFile to detect when two paths point to
the same physical file, instead of the slightly buggy local implementation
we had before which only worked on Unix systems and did not correctly
handle when the paths were on different physical devices.

The copy of the function I extracted here is the one from internal/initwd,
so this commit also includes the removal of that unexported version and
updating the callers in that package to use at at this new location.
2020-03-25 11:29:48 -07:00
findkim d0e13320d5 registry: configurable retry client 2020-02-14 13:29:00 -06:00
Kristin Laemmert 4045a6e5d0
initwd: cache registry responses for module versions and download URL (#23727)
* initwd: cache registry responses for module versions and download URL

Closes #23544
2020-01-07 15:03:23 -05:00
Martin Atkins 2b2ac1f6de configs: use local set of go-getter detectors
In an earlier change we switched to defining our own sets of detectors,
getters, etc for go-getter in order to insulate us from upstream changes
to those sets that might otherwise change the user-visible behavior of
Terraform's module installer.

However, we apparently neglected to actually refer to our local set of
detectors, and continued to refer to the upstream set. Here we catch up
with the latest detectors from upstream (taken from the version of
go-getter we currently have vendored) and start using that fixed set.

Currently we are maintaining these custom go-getter sets in two places
due to the configload vs. initwd distinction. That was already true for
goGetterGetters and goGetterDecompressors, and so I've preserved that for
now just to keep this change relatively simple; in later change it would
be nice to factor these "get with go getter" functions out into a shared
location which we can call from both configload and initwd.
2020-01-07 09:38:46 -08:00
Kristin Laemmert 68dfc3046d backend/remote-state: etcdv3, oss, and manta acc tests should fail
faster

The acceptance tests for etcdv3, oss and manta were not validating
required env variablea, chosing to assume that if one was running
acceptance tests they had already configured the credentials.

It was not always clear if this was a bug in the tests or the provider,
so I opted to make the tests fail faster when required attributes were
unset (or "").
2019-09-27 08:45:12 -04:00
Kristin Laemmert 69b6791ef1 internal/initwd: fix issues with tests and symlinks
filepath.EvalSymlinks is our friend! The code already does this, the
tests needed to be updated to do the same.
2019-09-25 16:27:17 -04:00
Radek Simko 5b9f2fafc8 Standardise directory name for test data 2019-06-30 10:16:15 +02:00
Pam Selle 59c5cc4788
Merge pull request #21254 from davewongillies/gcs
Add GCS source support for modules
2019-06-13 10:24:38 -04:00
Martin Atkins c5285021fa internal/initwd: Test that version is rejected for local modules
This was already working, but since that codepath is separate from the
go-getter install codepath it's helpful to have a separate test for it,
in addition to the existing one for go-getter modules.
2019-06-03 09:45:30 -07:00
David Gillies 8b45443b21
Add GCS source support for modules 2019-05-21 12:18:49 -07:00
Matt Morrison cbebb7cdf1 command/init: Don't panic if go-getter-fetched module has version constraint 2019-05-17 13:19:31 -07:00
Martin Atkins f5a7f45ffb internal/initwd: Fix module installation error
The "err" variable in the MaybeRelativePathErr condition was masking the
original err with nil in the "else" case of this branch, causing the
error message to be incomplete.

While here, also tweaked the wording to say "Could not download" rather
than "Error attempting to download", both to say the same thing in fewer
words and because the summary line above already starts with "Error:"
when we print out this message, so it looks weird to have both lines
start with the same word.
2019-05-14 07:38:06 -07:00
Kristin Laemmert 9327eedb04
internal/initwd: follow local module path symlink (#21063)
* internal/initwd: follow local module path symlink

Fixes #21060

While a previous commit fixed a problem when the local module directory
contained a symlink, it did not account for the possibility that the
entire directory was a symlink.
2019-04-24 08:19:27 -04:00
Kristin Laemmert a15a4acf2f
configs/configupgrade: detect possible relative module sources (#20646)
* configs/configupgrade: detect possible relative module sources

If a module source appears to be a relative local path but does not have
a preceding ./, print a #TODO message for the user.

* internal/initwd: limit go-getter detectors to those supported by terraform
* internal/initwd: move isMaybeRelativeLocalPath check into getWithGoGetter

To avoid making two calls to getter.Detect, which potentially makes
non-trivial API calls, the "isMaybeRelativeLocalPath" check was moved to
a later step and a custom error type was added so user-friendly
diagnostics could be displayed in the event that a possible relative local
path was detected.
2019-03-13 11:17:14 -07:00
Kristin Laemmert bf04503f04
internal/initwd: identify possible relative module sources
Identify module sources that look like relative paths ("child" instead
of "./child", for example) and surface a helpful error.

Previously, such module sources would be passed to go-getter, which
would fail because it was expecting an absolute, or properly relative,
path. This commit moves the check for improper relative paths sooner so
a user-friendly error can be displayed.
2019-03-11 15:25:21 -07:00
Kristin Laemmert 0c5fd835ce
copyDir: detect if the module install path is a symlink to a directory (#20603)
configs/configload and internal/initwd both had a copyDir function that
would fail if the source directory contained a symlinked directory,
because the os.FileMode.IsDir() returns false for symlinks.

This PR adds a check for a symlink and copies that symlink in the
target directory. It handles symlinks for both files and directories
(with included tests).

Fixes #20539
2019-03-07 12:59:48 -08:00
Martin Atkins 86c02d5c35 command: "terraform init" can partially initialize for 0.12upgrade
There are a few constructs from 0.11 and prior that cause 0.12 parsing to
fail altogether, which previously created a chicken/egg problem because
we need to install the providers in order to run "terraform 0.12upgrade"
and thus fix the problem.

This changes "terraform init" to use the new "early configuration" loader
for module and provider installation. This is built on the more permissive
parser in the terraform-config-inspect package, and so it allows us to
read out the top-level blocks from the configuration while accepting
legacy HCL syntax.

In the long run this will let us do version compatibility detection before
attempting a "real" config load, giving us better error messages for any
future syntax additions, but in the short term the key thing is that it
allows us to install the dependencies even if the configuration isn't
fully valid.

Because backend init still requires full configuration, this introduces a
new mode of terraform init where it detects heuristically if it seems like
we need to do a configuration upgrade and does a partial init if so,
before finally directing the user to run "terraform 0.12upgrade" before
running any other commands.

The heuristic here is based on two assumptions:
- If the "early" loader finds no errors but the normal loader does, the
  configuration is likely to be valid for Terraform 0.11 but not 0.12.
- If there's already a version constraint in the configuration that
  excludes Terraform versions prior to v0.12 then the configuration is
  probably _already_ upgraded and so it's just a normal syntax error,
  even if the early loader didn't detect it.

Once the upgrade process is removed in 0.13.0 (users will be required to
go stepwise 0.11 -> 0.12 -> 0.13 to upgrade after that), some of this can
be simplified to remove that special mode, but the idea of doing the
dependency version checks against the liberal parser will remain valuable
to increase our chances of reporting version-based incompatibilities
rather than syntax errors as we add new features in future.
2019-01-14 11:33:21 -08:00
Martin Atkins 0c0a437bcb Move module install functionality over to internal/initwd 2019-01-14 11:33:21 -08:00