Commit Graph

25768 Commits

Author SHA1 Message Date
Martin Atkins 7caf0b9246 addrs: ImpliedProviderForUnqualifiedType function
This encapsulates the logic for selecting an implied FQN for an
unqualified type name, which could either come from a local name used in
a module without specifying an explicit source for it or from the prefix
of a resource type on a resource that doesn't explicitly set "provider".

This replaces the previous behavior of just directly calling
NewDefaultProvider everywhere so that we can use a different implication
for the local name "terraform", to refer to the built-in terraform
provider rather than the stale one that's on registry.terraform.io for
compatibility with other Terraform versions.
2020-04-06 09:24:23 -07:00
Kristin Laemmert 27a794062e Mildwonkey/command tests (#24535)
* command: refactor testBackendState to write states.State

testBackendState was using the older terraform.State format, which is no
longer sufficient for most tests since the state upgrader does not
encode provider FQNs automatically. Users will run `terraform
0.13upgrade` to update their state to include provider FQNs in
resources, but tests need to use the modern state format instead of
relying on the automatic upgrade.

* plan tests passing
* graph tests passing
* json packages test update
* command test updates
* update show test fixtures
* state show tests passing
2020-04-06 09:24:23 -07:00
Martin Atkins de6c9ccec1 command/init: Move "vendored provider" test to e2etests
In the new design the ProviderSource is decided by package main, not by
the "command" package, and so making sure the vendor directory is included
is the responsibility of that package instead. Therefore we can no longer
test this at the "command" package level, but we'll retain a test for it
in e2etests to record that it isn't currently working, so that we have
a prompt to fix it before releasing.
2020-04-06 09:24:23 -07:00
Martin Atkins f35ebe2d65 internal/providercache: Fix incorrect logic in Installer.SetGlobalCacheDir
Due to some incomplete rework of this function in an earlier commit, the
safety check for using the same directory as both the target and the
cache was inverted and was raising an error _unless_ they matched, rather
than _if_ they matched.

This change is verified by the e2etest TestInitProviders_pluginCache,
which is also updated to use the new-style cache directory layout as part
of this commit.
2020-04-06 09:24:23 -07:00
Martin Atkins 14d456372a command/e2etest: Update expected output for new plugin installer
These tests make assertions against specific user-oriented output from the
"terraform init" command, but we've intentionally changed some of these
messages as part of introducing support for the decentralized provider
namespace.
2020-04-06 09:24:23 -07:00
Kristin Laemmert 0af09b23ca command: apply and most of import tests passing 2020-04-06 09:24:23 -07:00
Martin Atkins fe9105abfe command: Fix TestInit_getProviderMissing
This one just needs a tweak for the new wording of the error message.
2020-04-06 09:24:23 -07:00
Martin Atkins add7006de6 command: Fix TestInit_pluginDirProviders and _pluginDirProvidersDoesNotGet
Both of these are attempting to test -plugin-dir, which means we need some
additional help to populate some suitable directories for -plugin-dir to
refer to. The new installFakeProviderPackagesElsewhere helper generalizes
the earlier installFakeProviderPackages to allow installing fake provider
packages to an arbitrary other directory.
2020-04-06 09:24:23 -07:00
Martin Atkins ff55e1a1cd internal/providercache: installFromLocalDir
We previously skipped this one because it wasn't strictly necessary for
replicating the old "terraform init" behavior, but we do need it to work
so that things like the -plugin-dir option can behave correctly.

Linking packages from other cache directories and installing from unpacked
directories are fundamentally the same operation because a cache directory
is really just a collection of unpacked packages, so here we refactor
the LinkFromOtherCache functionality to actually be in
installFromLocalDir, and LinkFromOtherCache becomes a wrapper for
the installFromLocalDir function that just calculates the source and
target directories automatically and invalidates the metaCache.
2020-04-06 09:24:23 -07:00
Martin Atkins c81eebe0ac command/init: Fix TestInit_providerSource
This test is focused on making sure that the required_providers syntax
is working, so the rewritten version does not include any special handling
of pre-installed packages or "vendored" packages. Pre-installed plugins
are tested in other tests such as TestInit_getUpgradePlugins.
2020-04-06 09:24:23 -07:00
Martin Atkins 14701b8300 command/init: Fix TestInit_getUpgradePlugins
This test now requires a bit of a different approach because it was
previously directly constructing a cache directory but we now use a
different directory layout.

Rather than manually constructing the new heirarchical directory layout
(which would've required a lot more inline code), this introduces a helper
function installFakeProviderPackages that installs a fake provider package
directly into the local cache directory associated with a Meta object,
with the correct directory layout.
2020-04-06 09:24:23 -07:00
Martin Atkins c4fb22863c command/init: Support -plugin-dir again
This is a slightly different approach than we used to take for this
option: rather than disabling the installer and causing all future
commands to look elsewhere for plugins, we'll now leave the installer
enabled by constrain it to only look at the given directories.

This is overall simpler because it doesn't require any special tracking
of the plugin directories for subsequent commands. Instead, the selections
file generated by the installer will record the versions it selected from
the specified directories, and we'll link them in to the local cache just
as we would normally so that other commands don't need to do anything
special to select the right plugins in either case.
2020-04-06 09:24:23 -07:00
Martin Atkins f7072a8f29 internal/getproviders: Implement MultiSource
We previously had only a stub implementation for a totally-empty
MultiSource. Here we have an initial implementation of the full
functionality, which we'll need to support "terraform init -plugin-dir=..."
in a subsequent commit.
2020-04-06 09:24:23 -07:00
Martin Atkins 2ff4582be2 internal/providercache: Fix positions on selections file/dir
On Unix-derived systems a directory must be marked as "executable" in
order to be accessible, so our previous mode of 0660 here was unsufficient
and would cause a failure if it happened to be the installer that was
creating the plugins directory for the first time here.

Now we'll make it executable and readable for all but only writable by
the same user/group. For consistency, we also make the selections file
itself readable by everyone. In both cases, the umask we are run with may
further constrain these modes.
2020-04-06 09:24:23 -07:00
Martin Atkins d40085f374 command: Make the tests compile again
They still aren't passing, but this is just enough updating to make the
test program compile successfully after the refactoring related to
provider installation. They are now using the mock provider source offered
by the getproviders package, which is similar but not totally identical
to the idea of mocking the entire installer as these tests used to do, and
so many of them need further adjustment to still be testing what they
intended to test under this new architecture.

Subsequent commits will gradually repair the failing tests.
2020-04-06 09:24:23 -07:00
Martin Atkins 531de52dff internal/getproviders: MockSource and mock package metadata
These are some helpers to support unit testing in other packages, allowing
callers to exercise provider installation mechanisms without hitting any
real upstream source or having to prepare local package directories.

MockSource is a Source implementation that just scans over a provided
static list of packages and returns whatever matches.

FakePackageMeta is a shorthand for concisely constructing a
realistic-looking but uninstallable PackageMeta, probably for use with
MockSource.

FakeInstallablePackageMeta is similar to FakePackageMeta but also goes to
the trouble of creating a real temporary archive on local disk so that
the resulting package meta is pointing to something real on disk. This
makes the result more useful to the caller, but in return they get the
responsibility to clean up the temporary file once the test is over.

Nothing is using these yet.
2020-04-06 09:24:23 -07:00
Kristin Laemmert 32062b00a2 backend/local: refactor tests with modern state and default providers (#24524) 2020-04-06 09:24:23 -07:00
Kristin Laemmert 3f6ce3c588 Mildwonkey/tests (#24522)
* terraform: add helper functions for creating test state

testSetResourceInstanceCurrent and testSetResourceInstanceTainted are
wrapper functions around states.Module.SetResourceInstanceCurrent()
used to set a resource in state. They work with current, non-deposed
resources with no dependencies.

testSetResourceInstanceDeposed can be used to set a desosed resource in state.

* terraform: update all tests to use modern providers and state
2020-04-06 09:24:23 -07:00
Kristin Laemmert e683a6adef Mildwonkey/terraform tests (targeting integration branch) (#24513)
* configs: remove `Legacy*` Provider functions, switch to default
* terraform context test updates
2020-04-06 09:24:23 -07:00
Kristin Laemmert d95667f264 states/version3 upgrade: revert provider FQN changes (#24504)
This commit reverts an earlier change which automatically converted
provider strings to legacy provider FQNs. It has become apparent that a
state upgrade step will be required before upgrading to v0.13.
2020-04-06 09:24:23 -07:00
Martin Atkins a4280caf57 terraform: Remove some addrs.Provider.LegacyString uses
These are cases where we were using the legacy string only to produce a
message to the user or to write to the log. It's enough to make some
basic Terraform commands like "terraform validate" not panic and get far
enough along to see that provider startup is working.
2020-04-06 09:24:23 -07:00
Martin Atkins 549aede792 Remove terraform.ResourceProvider, use providercache.Installer instead
Back when we first introduced provider versioning in Terraform 0.10, we
did the provider version resolution in terraform.NewContext because we
weren't sure yet how exactly our versioning model was going to play out
(whether different versions could be selected per provider configuration,
for example) and because we were building around the limitations of our
existing filesystem-based plugin discovery model.

However, the new installer codepath is new able to do all of the
selections up front during installation, so we don't need such a heavy
inversion of control abstraction to get this done: the command package can
select the exact provider versions and pass their factories directly
to terraform.NewContext as a simple static map.

The result of this commit is that CLI commands other than "init" are now
able to consume the local cache directory and selections produced by the
installation process in "terraform init", passing all of the selected
providers down to the terraform.NewContext function for use in
implementing the main operations.

This commit is just enough to get the providers passing into the
terraform.Context. There's still plenty more to do here, including to
repair all of the tests this change has additionally broken.
2020-04-06 09:24:23 -07:00
Martin Atkins 5aa2e5ec8c addrs: "built-in" provider namespace
The introduction of a heirarchical addressing scheme for providers gives
us an opportunity to make more explicit the special case of "built-in"
providers.

Thus far we've just had a special case in the "command" package that the
provider named "terraform" is handled differently than all others, though
there's nothing especially obvious about that in the UI.

Moving forward we'll put such "built-in" providers under the special
namespace terraform.io/builtin/terraform, which will be visible in the UI
as being different than the other providers and we can use the namespace
itself (rather than a particular name) as the trigger for our special-case
behaviors around built-in plugins.

We have no plans to introduce any built-in providers other than
"terraform" in the foreseeable future, so any others will produce an
error.

This commit just establishes the addressing convention, without making use
of it anywhere yet. Subsequent commits will make the provider installer
and resolver codepaths aware of it, replacing existing checks for the
provider just being called "terraform".
2020-04-06 09:24:23 -07:00
Martin Atkins ae080481c0 internal/providercache: Installer records its selections in a file
Just as with the old installer mechanism, our goal is that explicit
provider installation is the only way that new provider versions can be
selected.

To achieve that, we conclude each call to EnsureProviderVersions by
writing a selections lock file into the target directory. A later caller
can then recall the selections from that file by calling SelectedPackages,
which both ensures that it selects the same set of versions and also
verifies that the checksums recorded by the installer still match.

This new selections.json file has a different layout than our old
plugins.json lock file. Not only does it use a different hashing algorithm
than before, we also record explicitly which version of each provider
was selected. In the old model, we'd repeat normal discovery when
reloading the lock file and then fail with a confusing error message if
discovery happened to select a different version, but now we'll be able
to distinguish between a package that's gone missing since installation
(which could previously have then selected a different available version)
from a package that has been modified.
2020-04-06 09:24:23 -07:00
Martin Atkins f6a7a4868b internal/providercache: Hashing of contents of cached packages
For the old-style provider cache directory model we hashed the individual
executable file for each provider. That's no longer appropriate because
we're giving each provider package a whole directory to itself where it
can potentially have many files.

This therefore introduces a new directory-oriented hashing algorithm, and
it's just using the Go Modules directory hashing algorithm directly
because that's already had its cross-platform quirks and other wrinkles
addressed during the Go Modules release process, and is now used
prolifically enough in Go codebases that breaking changes to the upstream
algorithm would be very expensive to the Go ecosystem.

This is also a bit of forward planning, anticipating that later we'll use
hashes in a top-level lock file intended to be checked in to user version
control, and then use those hashes also to verify packages _during_
installation, where we'd need to be able to hash unpacked zip files. The
Go Modules hashing algorithm is already implemented to consistently hash
both a zip file and an unpacked version of that zip file.
2020-04-06 09:24:23 -07:00
Martin Atkins 48bf00a7e2 vendor: go get golang.org/x/mod
We'll be using this for its directory hashing algorithm, as used in go.sum
in Go modules, and applying it also to Terraform provider packages.
2020-04-06 09:24:23 -07:00
Martin Atkins 079b4cf7be internal/providercache: Clear the metadata cache during package install
This was previously happening during linking from another cache, but not
when installing an entirely new provider.
2020-04-06 09:24:23 -07:00
Martin Atkins f113a7c22d command/init: Collect provider dependencies using our new helpers
This produces a value shaped the way the provider installer expects
without the need for further flattening and preprocessing.
2020-04-06 09:24:23 -07:00
Martin Atkins e6df3905c9 command/init: Generate progress output during provider installation
This restores some sequential event log output similar to what we had in
the previous implementation of plugin installation.
2020-04-06 09:24:23 -07:00
Martin Atkins 94e1ac2d07 command: Minimal integration of new provider installer in "init"
There's still a lot of work to do here around both the UX and the
follow-up steps that need to happen after installation completes, but this
is enough to faciliate some initial end-to-end testing of the new-style
install process.
2020-04-06 09:24:23 -07:00
Petros Kolyvas f6ac9e361b Merge branch 'contribution-codeowners' of github:pkolyvas/terraform into contribution-codeowners 2020-04-06 12:13:49 -04:00
Petros Kolyvas 77ebf266f3 Restored the note about our community guidelines 2020-04-06 12:13:46 -04:00
Petros Kolyvas 6ff988a423
Apply suggestions from code review
Alisdair's fixes.

Co-Authored-By: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2020-04-06 12:10:27 -04:00
Petros Kolyvas 10437b4b15
Update .github/CONTRIBUTING.md
Co-Authored-By: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2020-04-06 12:09:22 -04:00
Pam Selle 3ac0410fe2
Merge pull request #24568 from hashicorp/remove-travis
remove travis file
2020-04-06 10:57:28 -04:00
Alvin Huang 2d2369e3b1 add better error message when 'make fmtcheck generate' fails 2020-04-06 10:52:31 -04:00
Alvin Huang 21b78a15be remove travis script file 2020-04-06 10:49:28 -04:00
Alvin Huang a201e09a75 remove travis file 2020-04-06 10:31:53 -04:00
James Bardin fd4342668a
Merge pull request #24565 from hashicorp/jbardin/validate-module-expansion
Create module validation nodes
2020-04-06 09:24:12 -04:00
James Bardin 6da8542dc4
Merge pull request #24562 from hashicorp/jbardin/panicwrapped
don't call os.NewFile on unknown FDs
2020-04-06 09:14:45 -04:00
James Bardin e23aa02560 modules expansion validate test 2020-04-06 09:13:43 -04:00
James Bardin 73492fd2d5 add module expansion to validation
We cannot evaluate expansion during validation, since the values may not
be known at that time.

Inject a nodeValidateModule, using the "Concrete" pattern used for other
node types during graph building. This node will always evaluate to a
single module instance, so that we have a valid context within which to
evaluate all sub resources.
2020-04-05 12:13:48 -04:00
James Bardin 5fda76e31d simplify module expansion eval
Make the expansion logic easier to follow, keeping the evaluation and
registration local to switch cases. We don't validate anything between
count or for_each (config loading should handle that), and we don't need
to keep relying on the count == -1 sentinel value.
2020-04-05 11:07:38 -04:00
James Bardin cd65b28da0 don't call os.NewFile on unknown FDs
os.NewFile was called on file descriptors 3, 4, and 5 during every init,
in case this process happened to be running inside panicwrap. If the
runtime has already chosen one of these file descriptors to use
internally, starting polling on them can cause the runtime to crash.

Initialize the file descriptors lazily, only if we know that they belong
to us, after Wrapped is checked.
2020-04-04 15:46:19 -04:00
Petros Kolyvas ad15e4a116
A few more spacing issues in MD format for GH 2020-04-03 16:15:09 -04:00
Petros Kolyvas c7cae98076
GitHub hates me 2020-04-03 16:14:08 -04:00
Petros Kolyvas cd54300621 Why won't you h4 maintainers. 2020-04-03 16:13:36 -04:00
Petros Kolyvas db5878ff04 More updates and fixes, some grammatical, some clarity issues, one markdown bugaboo 2020-04-03 16:11:42 -04:00
Petros Kolyvas 68155c23ac Merge branch 'contribution-codeowners' of github:pkolyvas/terraform into contribution-codeowners 2020-04-03 16:06:11 -04:00
Petros Kolyvas ca90de14ec Updated Contribution Guide TOC 2020-04-03 16:04:55 -04:00