Commit Graph

25909 Commits

Author SHA1 Message Date
Martin Atkins 2ff4582be2 internal/providercache: Fix positions on selections file/dir
On Unix-derived systems a directory must be marked as "executable" in
order to be accessible, so our previous mode of 0660 here was unsufficient
and would cause a failure if it happened to be the installer that was
creating the plugins directory for the first time here.

Now we'll make it executable and readable for all but only writable by
the same user/group. For consistency, we also make the selections file
itself readable by everyone. In both cases, the umask we are run with may
further constrain these modes.
2020-04-06 09:24:23 -07:00
Martin Atkins d40085f374 command: Make the tests compile again
They still aren't passing, but this is just enough updating to make the
test program compile successfully after the refactoring related to
provider installation. They are now using the mock provider source offered
by the getproviders package, which is similar but not totally identical
to the idea of mocking the entire installer as these tests used to do, and
so many of them need further adjustment to still be testing what they
intended to test under this new architecture.

Subsequent commits will gradually repair the failing tests.
2020-04-06 09:24:23 -07:00
Martin Atkins 531de52dff internal/getproviders: MockSource and mock package metadata
These are some helpers to support unit testing in other packages, allowing
callers to exercise provider installation mechanisms without hitting any
real upstream source or having to prepare local package directories.

MockSource is a Source implementation that just scans over a provided
static list of packages and returns whatever matches.

FakePackageMeta is a shorthand for concisely constructing a
realistic-looking but uninstallable PackageMeta, probably for use with
MockSource.

FakeInstallablePackageMeta is similar to FakePackageMeta but also goes to
the trouble of creating a real temporary archive on local disk so that
the resulting package meta is pointing to something real on disk. This
makes the result more useful to the caller, but in return they get the
responsibility to clean up the temporary file once the test is over.

Nothing is using these yet.
2020-04-06 09:24:23 -07:00
Kristin Laemmert 32062b00a2 backend/local: refactor tests with modern state and default providers (#24524) 2020-04-06 09:24:23 -07:00
Kristin Laemmert 3f6ce3c588 Mildwonkey/tests (#24522)
* terraform: add helper functions for creating test state

testSetResourceInstanceCurrent and testSetResourceInstanceTainted are
wrapper functions around states.Module.SetResourceInstanceCurrent()
used to set a resource in state. They work with current, non-deposed
resources with no dependencies.

testSetResourceInstanceDeposed can be used to set a desosed resource in state.

* terraform: update all tests to use modern providers and state
2020-04-06 09:24:23 -07:00
Kristin Laemmert e683a6adef Mildwonkey/terraform tests (targeting integration branch) (#24513)
* configs: remove `Legacy*` Provider functions, switch to default
* terraform context test updates
2020-04-06 09:24:23 -07:00
Kristin Laemmert d95667f264 states/version3 upgrade: revert provider FQN changes (#24504)
This commit reverts an earlier change which automatically converted
provider strings to legacy provider FQNs. It has become apparent that a
state upgrade step will be required before upgrading to v0.13.
2020-04-06 09:24:23 -07:00
Martin Atkins a4280caf57 terraform: Remove some addrs.Provider.LegacyString uses
These are cases where we were using the legacy string only to produce a
message to the user or to write to the log. It's enough to make some
basic Terraform commands like "terraform validate" not panic and get far
enough along to see that provider startup is working.
2020-04-06 09:24:23 -07:00
Martin Atkins 549aede792 Remove terraform.ResourceProvider, use providercache.Installer instead
Back when we first introduced provider versioning in Terraform 0.10, we
did the provider version resolution in terraform.NewContext because we
weren't sure yet how exactly our versioning model was going to play out
(whether different versions could be selected per provider configuration,
for example) and because we were building around the limitations of our
existing filesystem-based plugin discovery model.

However, the new installer codepath is new able to do all of the
selections up front during installation, so we don't need such a heavy
inversion of control abstraction to get this done: the command package can
select the exact provider versions and pass their factories directly
to terraform.NewContext as a simple static map.

The result of this commit is that CLI commands other than "init" are now
able to consume the local cache directory and selections produced by the
installation process in "terraform init", passing all of the selected
providers down to the terraform.NewContext function for use in
implementing the main operations.

This commit is just enough to get the providers passing into the
terraform.Context. There's still plenty more to do here, including to
repair all of the tests this change has additionally broken.
2020-04-06 09:24:23 -07:00
Martin Atkins 5aa2e5ec8c addrs: "built-in" provider namespace
The introduction of a heirarchical addressing scheme for providers gives
us an opportunity to make more explicit the special case of "built-in"
providers.

Thus far we've just had a special case in the "command" package that the
provider named "terraform" is handled differently than all others, though
there's nothing especially obvious about that in the UI.

Moving forward we'll put such "built-in" providers under the special
namespace terraform.io/builtin/terraform, which will be visible in the UI
as being different than the other providers and we can use the namespace
itself (rather than a particular name) as the trigger for our special-case
behaviors around built-in plugins.

We have no plans to introduce any built-in providers other than
"terraform" in the foreseeable future, so any others will produce an
error.

This commit just establishes the addressing convention, without making use
of it anywhere yet. Subsequent commits will make the provider installer
and resolver codepaths aware of it, replacing existing checks for the
provider just being called "terraform".
2020-04-06 09:24:23 -07:00
Martin Atkins ae080481c0 internal/providercache: Installer records its selections in a file
Just as with the old installer mechanism, our goal is that explicit
provider installation is the only way that new provider versions can be
selected.

To achieve that, we conclude each call to EnsureProviderVersions by
writing a selections lock file into the target directory. A later caller
can then recall the selections from that file by calling SelectedPackages,
which both ensures that it selects the same set of versions and also
verifies that the checksums recorded by the installer still match.

This new selections.json file has a different layout than our old
plugins.json lock file. Not only does it use a different hashing algorithm
than before, we also record explicitly which version of each provider
was selected. In the old model, we'd repeat normal discovery when
reloading the lock file and then fail with a confusing error message if
discovery happened to select a different version, but now we'll be able
to distinguish between a package that's gone missing since installation
(which could previously have then selected a different available version)
from a package that has been modified.
2020-04-06 09:24:23 -07:00
Martin Atkins f6a7a4868b internal/providercache: Hashing of contents of cached packages
For the old-style provider cache directory model we hashed the individual
executable file for each provider. That's no longer appropriate because
we're giving each provider package a whole directory to itself where it
can potentially have many files.

This therefore introduces a new directory-oriented hashing algorithm, and
it's just using the Go Modules directory hashing algorithm directly
because that's already had its cross-platform quirks and other wrinkles
addressed during the Go Modules release process, and is now used
prolifically enough in Go codebases that breaking changes to the upstream
algorithm would be very expensive to the Go ecosystem.

This is also a bit of forward planning, anticipating that later we'll use
hashes in a top-level lock file intended to be checked in to user version
control, and then use those hashes also to verify packages _during_
installation, where we'd need to be able to hash unpacked zip files. The
Go Modules hashing algorithm is already implemented to consistently hash
both a zip file and an unpacked version of that zip file.
2020-04-06 09:24:23 -07:00
Martin Atkins 48bf00a7e2 vendor: go get golang.org/x/mod
We'll be using this for its directory hashing algorithm, as used in go.sum
in Go modules, and applying it also to Terraform provider packages.
2020-04-06 09:24:23 -07:00
Martin Atkins 079b4cf7be internal/providercache: Clear the metadata cache during package install
This was previously happening during linking from another cache, but not
when installing an entirely new provider.
2020-04-06 09:24:23 -07:00
Martin Atkins f113a7c22d command/init: Collect provider dependencies using our new helpers
This produces a value shaped the way the provider installer expects
without the need for further flattening and preprocessing.
2020-04-06 09:24:23 -07:00
Martin Atkins e6df3905c9 command/init: Generate progress output during provider installation
This restores some sequential event log output similar to what we had in
the previous implementation of plugin installation.
2020-04-06 09:24:23 -07:00
Martin Atkins 94e1ac2d07 command: Minimal integration of new provider installer in "init"
There's still a lot of work to do here around both the UX and the
follow-up steps that need to happen after installation completes, but this
is enough to faciliate some initial end-to-end testing of the new-style
install process.
2020-04-06 09:24:23 -07:00
Petros Kolyvas f6ac9e361b Merge branch 'contribution-codeowners' of github:pkolyvas/terraform into contribution-codeowners 2020-04-06 12:13:49 -04:00
Petros Kolyvas 77ebf266f3 Restored the note about our community guidelines 2020-04-06 12:13:46 -04:00
Petros Kolyvas 6ff988a423
Apply suggestions from code review
Alisdair's fixes.

Co-Authored-By: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2020-04-06 12:10:27 -04:00
Petros Kolyvas 10437b4b15
Update .github/CONTRIBUTING.md
Co-Authored-By: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2020-04-06 12:09:22 -04:00
Pam Selle 3ac0410fe2
Merge pull request #24568 from hashicorp/remove-travis
remove travis file
2020-04-06 10:57:28 -04:00
Alvin Huang 2d2369e3b1 add better error message when 'make fmtcheck generate' fails 2020-04-06 10:52:31 -04:00
Alvin Huang 21b78a15be remove travis script file 2020-04-06 10:49:28 -04:00
Alvin Huang a201e09a75 remove travis file 2020-04-06 10:31:53 -04:00
James Bardin fd4342668a
Merge pull request #24565 from hashicorp/jbardin/validate-module-expansion
Create module validation nodes
2020-04-06 09:24:12 -04:00
James Bardin 6da8542dc4
Merge pull request #24562 from hashicorp/jbardin/panicwrapped
don't call os.NewFile on unknown FDs
2020-04-06 09:14:45 -04:00
James Bardin e23aa02560 modules expansion validate test 2020-04-06 09:13:43 -04:00
James Bardin 73492fd2d5 add module expansion to validation
We cannot evaluate expansion during validation, since the values may not
be known at that time.

Inject a nodeValidateModule, using the "Concrete" pattern used for other
node types during graph building. This node will always evaluate to a
single module instance, so that we have a valid context within which to
evaluate all sub resources.
2020-04-05 12:13:48 -04:00
James Bardin 5fda76e31d simplify module expansion eval
Make the expansion logic easier to follow, keeping the evaluation and
registration local to switch cases. We don't validate anything between
count or for_each (config loading should handle that), and we don't need
to keep relying on the count == -1 sentinel value.
2020-04-05 11:07:38 -04:00
James Bardin cd65b28da0 don't call os.NewFile on unknown FDs
os.NewFile was called on file descriptors 3, 4, and 5 during every init,
in case this process happened to be running inside panicwrap. If the
runtime has already chosen one of these file descriptors to use
internally, starting polling on them can cause the runtime to crash.

Initialize the file descriptors lazily, only if we know that they belong
to us, after Wrapped is checked.
2020-04-04 15:46:19 -04:00
Petros Kolyvas ad15e4a116
A few more spacing issues in MD format for GH 2020-04-03 16:15:09 -04:00
Petros Kolyvas c7cae98076
GitHub hates me 2020-04-03 16:14:08 -04:00
Petros Kolyvas cd54300621 Why won't you h4 maintainers. 2020-04-03 16:13:36 -04:00
Petros Kolyvas db5878ff04 More updates and fixes, some grammatical, some clarity issues, one markdown bugaboo 2020-04-03 16:11:42 -04:00
Petros Kolyvas 68155c23ac Merge branch 'contribution-codeowners' of github:pkolyvas/terraform into contribution-codeowners 2020-04-03 16:06:11 -04:00
Petros Kolyvas ca90de14ec Updated Contribution Guide TOC 2020-04-03 16:04:55 -04:00
Petros Kolyvas 942b4df545 Added a provisioners section and a maintainers section 2020-04-03 16:00:19 -04:00
James Bardin 60e407a59b
Merge pull request #24539 from hashicorp/jbardin/expanded-orphans
handle orphaned instances in expanded modules
2020-04-03 13:16:29 -04:00
James Bardin 2c5e2d6b5b add missing action check in orphan test 2020-04-03 11:28:31 -04:00
James Bardin 512830bb7b update CHANGELOG.md 2020-04-03 10:07:47 -04:00
James Bardin 4e0b6d5467
Update terraform/node_module_expand.go
Co-Authored-By: Kristin Laemmert <mildwonkey@users.noreply.github.com>
2020-04-03 09:30:25 -04:00
James Bardin 841b0f66c8 rename evalModuleRoot
use evalCloseModule to match the parent node name
2020-04-02 16:04:05 -04:00
James Bardin 69b74ba62b add new root node to test output 2020-04-02 16:00:36 -04:00
James Bardin 0dcca1bc37 make the root node a nodeCloseModule for root
Replace the graphNodeRoot for the main graph with a nodeCloseModule for
the root module. USe a new transformer as well, so as to not change any
behavior of DynamicExpand graphs.

Closing out the root module like we do with sub modules means we no
longer need the OrphanResourceTransformer, or the NodeDestroyResource.
The old resource destroy logic has mostly moved into the instance nodes,
and the remaining resource node was just for cleanup, which need to be
done again by the module since there isn't always a NodeDestroyResource
to be evaluated.

The more-correct state caused a few tests to fail, which need to be
cleaned up to match the state without empty resource husks.
2020-04-02 16:00:36 -04:00
James Bardin 0e8cf5783e fix new graph builder test output 2020-04-02 16:00:36 -04:00
James Bardin 1a1ace5930 prune unused values based on behavior
remove the hard-coded types from PruneUnusedValuesTransformer
2020-04-02 16:00:36 -04:00
James Bardin c0bca9d5e9 check for the correct types when pruning values
There is not one more non-dependent type to look for when pruning unused
values. This fixes the oversight, but still leaves the ugly concrete
type checking which we need to remove.
2020-04-02 16:00:36 -04:00
James Bardin 2df7127943 add nodeCloseModule
During plan, anything dependent on a module can connect to the module
expansion node, because all instance nodes are created during
DynamicExpand. During apply the instance nodes are created from the
diff, so we need a root module to terminate the logical module subgraph.

Besides providing an anchor for the completion of a module, the
nodeCloseModule can also be used to cleanup the orphan resource and
module placeholders in the state.
2020-04-02 16:00:36 -04:00
James Bardin 026a45a390 remove abstract resource node from destroy node
NodeDestroyResource does not require a provider, and to avoid this a
temporary GraphNodeNoProvider was used to differentiate it from other
resource nodes. We can now de-couple the destroy node from the abstract
resource which was adding the ProvidedBy method, and remove the
NoProvider method.
2020-04-02 16:00:35 -04:00