Commit Graph

25719 Commits

Author SHA1 Message Date
Pam Selle f738f85241 Create non-specific ModuleCallOutput 2020-03-26 13:29:38 -04:00
James Bardin 34cab3bc99
Merge pull request #24461 from hashicorp/jbardin/eval-context-path
Module Expansion Activate!
2020-03-26 12:45:23 -04:00
James Bardin cec989d660 comment update and remove extra Name method
This name method won't be called in the full graph, and remove it to
prevent confusion with the parent node in logs.
2020-03-26 11:52:41 -04:00
James Bardin 4f1692cfaf comment updates 2020-03-25 17:03:06 -04:00
James Bardin 5810261add don't log path in EvalRaw
eval nodes no longer always have a context path
2020-03-25 17:03:06 -04:00
James Bardin 04a117b2a1 module expansion test
simplify the test a bit and add a few more combinations to the config
2020-03-25 17:03:06 -04:00
James Bardin cd045f6f4e enable count and for_each in configuration 2020-03-25 17:03:06 -04:00
James Bardin 8eb3f2cf52 orphan resources needs to use AbsResource
The expand logic was separated into
nodeExpandRefreshableManagedResource, but the orphan logic wasn't
updated.
2020-03-25 17:03:06 -04:00
James Bardin 7f0199bab0 cleanup some expanders 2020-03-25 17:03:06 -04:00
James Bardin 2474b87ff4 remove UnkeyedInstanceShim from some provider nodes
Remove the shims where they aren't necessary from the Init and Close
provider nodes. This also removed some provider path checks from the
builtin eval context, which cannot be resolved since the context may not
be created with a ModuleInstance path.
2020-03-25 17:03:06 -04:00
James Bardin b3fc0dab94 use addrs.ConfigResource for dependency tracking
We can't get module instances during transformation, so we need to
reduce the Dependencies to using `addrs.ConfigResource` for now.
2020-03-25 17:03:06 -04:00
James Bardin 0afa3710fd create refresh node expanders 2020-03-25 17:03:06 -04:00
James Bardin 74d85aa956 Add Path to more nodes that require it. 2020-03-25 17:03:06 -04:00
James Bardin 0b85eeab38 NewNodeAbstractResource accepts a ResourceConfig
Use the new addrs type here.

Also remove the uniqueMap from the config transformer. We enforce
uniqueness during config loading, and this is more likely to have false
positives due to stringification than anything.
2020-03-25 17:03:06 -04:00
James Bardin 23cebc5205 create nodeExpandApplyableResource
Resources also need to be expanded during apply, which cannot be done
via EvalTree due to the lack of EvalContext.
2020-03-25 17:03:06 -04:00
James Bardin 40f09027f0 expand planned resources
While the Expander itself now handles the recursive expansion of
modules, Resources themselves still need to be expanded twice, because
the evaluation of the Resource, which entails evaluating the for_each or
count expressions, is separate from the ResourceInstance expansion.

Add a nodeExpandPlannableResource to do handle this expansion to allow
all NodePlannableResources to call EvalWriteResourceState with an
absolute address.
2020-03-25 17:03:06 -04:00
James Bardin 0b025d74e5 add EvalContext.WithPath
As the Graph is walked, the current way to set the context path was to
have the walker return a context from EnterPath. This required that
every node know it's absolute path, which can no longer be the case
during plan when modules have not been expanded.

This introduces a new method called WithPath, which returns a copy of
the context with the internal path updated to reflect the method
argument. Any use of the EvalContext that requires knowing the path will
now panic if it wasn't explicitly set to ensure that evaluations always
occur in the correct path.

Add EvalContext to the GraphWalker interface.
EvalContext returns an EvalContext that has not yet set a path. This
will allow us to enforce that all context operations requiring a module
instance path will require that a path be explicitly set rather than
evaluating within the wrong path.
2020-03-25 17:03:06 -04:00
Martin Atkins 537c1bedcf internal/providercache: LinkFromOtherCache removes target, not source
This was incorrectly removing the _source_ entry prior to creating the
symlink, therefore ending up with a dangling symlink and no source file.

This wasn't obvious before because the test case for LinkFromOtherCache
was also incorrectly named and therefore wasn't running. Fixing the name
of that test made this problem apparent.

The TestLinkFromOtherCache test case now ends up seeing the final resolved
directory rather than the symlink target, because of upstream changes
to the internal/getproviders filesystem scanning logic to handle symlinks
properly.
2020-03-25 13:50:00 -07:00
Martin Atkins eb25fe8b24 internal/getproviders: SearchLocalDirectory can handle symlinks
Previously this was failing to treat symlinks to directories as unpacked
layout, because our file info was only an Lstat result, not a full Stat.

Now we'll resolve the symlink first, allowing us to handle a symlink to
a directory. That's important because our internal/providercache behavior
is to symlink from one cache to another where possible.
2020-03-25 13:50:00 -07:00
Martin Atkins ad15459468 internal/{getproviders,providercache}: improved trace logging
There's a lot going on in these functions that can be hard to follow from
the outside, so we'll add some additional trace logging so that we can
more easily understand why things are behaving the way they are.
2020-03-25 13:50:00 -07:00
Martin Atkins 391ca0c991 internal/providercache: add windows test fixture
This was accidentally left out of an earlier commit due to our top-level
.gitignore file containing *.exe as an ignore pattern.
2020-03-25 13:50:00 -07:00
Pam Selle 34de1453ce (build) temporarily disable codecov while debugging travis 2020-03-25 15:37:38 -04:00
James Bardin c7e2bace18
Merge pull request #24454 from hashicorp/jbardin/expand-resource
Expander.ExpandResource cannot expand all modules
2020-03-25 15:34:31 -04:00
Martin Atkins 807267d1b5 internal/providercache: Installation from HTTP URLs and local archives
When a provider source produces an HTTP URL location we'll expect it to
resolve to a zip file, which we'll first download to a temporary
directory and then treat it like a local archive.

When a provider source produces a local archive path we'll expect it to
be a zip file and extract it into the target directory.

This does not yet include an implementation of installing from an
already-unpacked local directory. That will follow in a subsequent commit,
likely following a similar principle as in Dir.LinkFromOtherCache.
2020-03-25 11:29:48 -07:00
Martin Atkins 754b7ebb65 command: Expose providercache package objects for use elsewhere
These new functions allow command implementations to get hold of the
providercache objects and installation source object derived from the
current CLI configuration.
2020-03-25 11:29:48 -07:00
Martin Atkins e4d7a71d91 internal/getproviders: Exit early if MultiSource has no sources
The MultiSource isn't actually properly implemented yet, but this is a
minimal implementation just for the case where there are no underlying
sources at all, because we use an empty MultiSource as a placeholder
when a test in the "command" package fails to explicitly populate a
ProviderSource.
2020-03-25 11:29:48 -07:00
Martin Atkins 18dd0a396d internal/providercache: First pass of the actual install process
This is not tested yet, but it's a compilable strawman implementation of
the necessary sequence of events to coordinate all of the moving parts
of running a provider installation operation.

This will inevitably see more iteration in later commits as we complete
the surrounding parts and wire it up to be used by "terraform init". So
far, it's just dead code not called by any other package.
2020-03-25 11:29:48 -07:00
Martin Atkins 03155daf98 internal/providercache: Start to stub Installer type
The Installer type will encapsulate the logic for running an entire
provider installation request: given a set of providers to install, it
will determine a method to obtain each of them (or detect that they are
already installed) and then take the necessary actions.

So far it doesn't do anything, but this stubs out an interface by which
the caller can request ongoing notifications during an installation
operation.
2020-03-25 11:29:48 -07:00
Martin Atkins e4d14324e0 internal/providercache: Stub of Dir.InstallPackage method
This will eventually be responsible for actually retrieving a package from
a source and then installing it into the cache directory, but for the
moment it's just a stub to complete the proposed API, which I intend to
test in a subsequent commit by writing the full "Installer" API that will
encapsulate the full installation logic.
2020-03-25 11:29:48 -07:00
Martin Atkins 67ca067910 internal/providercache: Linking from one cache to another
When a system-wide shared plugin cache is configured, we'll want to make
use of entries already in the shared cache when populating a local
(configuration-specific) cache.

This new method LinkFromOtherCache encapsulates the work of placing a link
from one cache to another. If possible it will create a symlink, therefore
retaining a key advantage of configuring a shared plugin cache, but
otherwise we'll do a deep copy of the package directory from one cache
to the other.

Our old provider installer would always skip trying to create symlinks on
Windows because Go standard library support for os.Symlink on Windows
was inconsistent in older versions. However, os.Symlink can now create
symlinks using a new API introduced in a Windows 10 update and cleanly
fail if symlink creation is impossible, so it's safe for us to just
try to create the symlink and react if that produces an error, just as we
used to do on non-Windows systems when possibly creating symlinks on
filesystems that cannot support them.
2020-03-25 11:29:48 -07:00
Martin Atkins 514184cc9d internal/getproviders: Functions to determine installation directories
The existing functionality in this package deals with finding packages
that are either available for installation or already installed. In order
to support installation we also need to determine the location where a
package should be installed.

This lives in the getproviders package because that way all of the logic
related to the filesystem layout for local provider directories lives
together here where they can be maintained together more easily in future.
2020-03-25 11:29:48 -07:00
Martin Atkins 072c6d9aed internal/copydir: Factor out our recursive directory copy for reuse
We've previously been copying this function around so it could remain
unexported while being used in various packages. However, it's a
non-trivial function with lots of specific assumptions built into it, so
here we'll put it somewhere that other packages can depend on it _and_
document the assumptions it seems to be making for future reference.

As a bonus, this now uses os.SameFile to detect when two paths point to
the same physical file, instead of the slightly buggy local implementation
we had before which only worked on Unix systems and did not correctly
handle when the paths were on different physical devices.

The copy of the function I extracted here is the one from internal/initwd,
so this commit also includes the removal of that unexported version and
updating the callers in that package to use at at this new location.
2020-03-25 11:29:48 -07:00
Martin Atkins d13001830b providercache: A package to encapsulate management of provider cache dirs
Historically our logic to handle discovering and installing providers has
been spread across several different packages. This package is intended
to become the home of all logic related to what is now called "provider
cache directories", which means directories on local disk where Terraform
caches providers in a form that is ready to run.

That includes both logic related to interrogating items already in a cache
(included in this commit) and logic related to inserting new items into
the cache from upstream provider sources (to follow in later commits).

These new codepaths are focused on providers and do not include other
plugin types (provisioners and credentials helpers), because providers are
the only plugin type that is represented by a heirarchical, decentralized
namespace and the only plugin type that has an auto-installation protocol
defined. The existing codepaths will remain to support the handling of
the other plugin types that require manual installation and that use only
a flat, locally-defined namespace.
2020-03-25 11:29:48 -07:00
Martin Atkins 283b4d4cad internal/getproviders: Make local directory search a public API
Previously this was available by instantiating a throwaway
FilesystemMirrorSource, but that's pretty counter-intuitive for callers
that just want to do a one-off scan without retaining any ongoing state.

Now we expose SearchLocalDirectory as an exported function, and the
FilesystemMirrorSource then uses it as part of its implementation too.
Callers that just want to know what's available in a directory can call
SearchLocalDirectory directly.
2020-03-25 11:29:48 -07:00
James Bardin 14ec3e6078 Expander.ExpandResource cannot expand all modules
ExpandResource must only check the specific ModuleInstances in the
requested path, because the resource may not have been registered yet in
all module instances.
2020-03-25 11:58:23 -04:00
James Bardin c49fd39c2a
Merge pull request #24444 from hashicorp/jbardin/expand-resource
Expander.ExpandResource should use AbsResource
2020-03-24 12:55:02 -04:00
James Bardin 1b3f5beeec udpate core to work with new ExpandResource
This also calls ExpandModuleResource in one location, because the logic
is not yet updated to handle actual module expansion, but that will be
fixed in a forthcoming PR.
2020-03-24 12:13:28 -04:00
James Bardin 0d58537bf2 Expander.ExpandResource
It turns out that within terraform resource expansion will normally
happen with an absolute address. This is because in order to evaluate
the expansion expression, we need to already have the context within a
module instance.

This leaves the existing ExpandResource logic in place as
ExpandModuleResource since it's working, and in case we do find a
location where it's useful to get a full expansion (which may be needed
for instance dependency tracking)

Reword some of the resource related arguments and comments, as they were
copied from the module methods and not entirely accurate.
2020-03-24 12:12:41 -04:00
Kristin Laemmert 976d2980b2
docs/contributing: add note about codecov (#24434) 2020-03-23 15:48:31 -04:00
Kristin Laemmert 5b427ec648
addrs: Provider.IsLegacy and IsDefault functions
* add IsLegacy and IsDefault funcs to addrs.Provider
* add some test coverage
2020-03-23 09:50:35 -04:00
Adam Leskis 4f85a1a6ba
website: fix simple typo (#24415) 2020-03-23 08:55:08 -04:00
Kristin Laemmert 4f141ae365
rename provider-specific functions (#24417)
missingPlugins was hard-coded to work only with provider plugins, so I
renamed it to clarify the usage.

Also renamed a test provider from greater_than to greater-than as the
underscore is an invalid provider name character and this will become a
hard error in the near future.
2020-03-20 13:59:59 -04:00
Kristin Laemmert c8d64846ad
Mildwonkey/ps import (#24412)
* import: remove Config from ImportOpts

`Config` in ImportOpts was any provider configuration provided by the
user on the command line. This option has already been removed in favor
of only taking the provider from the configuration loaded in the current
context.

* terrafrom: add Config to ImportStateTransformer and refactor Transform
to get the resource provider FQN from the Config
2020-03-20 08:15:29 -04:00
Alisdair McDiarmid 9d0c8c5970
Remove changelog entry for fix already in 0.12.24 2020-03-19 10:40:58 -04:00
Alisdair McDiarmid 51ea5d6b2f
Merge pull request #24411 from hashicorp/alisdair/registry-retry-panic
registry: Fix panic when server is unreachable
2020-03-19 10:37:30 -04:00
Alisdair McDiarmid 1c1df6dc50 registry: Fix panic when server is unreachable
Non-HTTP errors previously resulted in a panic due to dereferencing the
resp pointer while it was nil, as part of rendering the error message.
This commit changes the error message formatting to cope with a nil
response, and extends test coverage.

Fixes #24384
2020-03-19 10:20:10 -04:00
Kristin Laemmert 7d6d653b4f
Update CHANGELOG.md 2020-03-19 08:04:12 -04:00
Kristin Laemmert 5f313a65ad
command: remove 0.12upgrade (#24403)
* command: remove 0.12upgrade and related `configupgrade` library
* leave deprecation warning for 0.12upgrade to point users to v0.12
2020-03-19 08:01:16 -04:00
Nick Fagerlund 109c4bf6ef website: Remove links to the getting started guide's old location
Since these links were in the soon-to-be-deprecated 0.11 language section, I
think we can just remove them without needing to find an equivalent link.
2020-03-18 14:20:03 -07:00
Alisdair McDiarmid 420e22ece4 Update CHANGELOG.md 2020-03-18 09:11:44 -04:00