Commit Graph

22884 Commits

Author SHA1 Message Date
Sander van Harmelen 5b0b7d0a78
Update CHANGELOG.md 2018-11-20 22:52:22 +01:00
Sander van Harmelen 5e2af5ddff
Update CHANGELOG.md 2018-11-20 22:51:57 +01:00
Sander van Harmelen ae6ddc29ae
Merge pull request #19378 from hashicorp/svh/f-remote-state
backend/remote: change how to fall back to a local backend
2018-11-20 22:47:56 +01:00
Sander van Harmelen a17f317025 Change how to fall back from remote to local backend
In order to support free organizations, we need a way to load the `remote` backend and then, depending on the used offering/plan, enable or disable remote operations.

In other words, we should be able to dynamically fall back to the `local` backend if needed, after first configuring the `remote` backend.

To make this works we need to change the way this was done previously when the env var `TF_FORCE_LOCAL_BACKEND` was set. The clear difference of course being that the env var would be available on startup, while the used offering/plan is only known after being able to connect to TFE.
2018-11-20 22:25:52 +01:00
Sander van Harmelen e7fd446f74
Update CHANGELOG.md 2018-11-20 21:17:48 +01:00
Martin Atkins 67041486c5
build: Travis-CI build should use vendor directory when building for Windows 2018-11-20 11:59:48 -08:00
Justin Campbell 97bdeddabe
Update CHANGELOG.md 2018-11-20 14:32:59 -05:00
Justin Campbell bf301d7abd
Merge pull request #19389 from hashicorp/f-registry-provider-signing-keys
Terraform Registry: Use signing keys provided from the Registry
2018-11-20 14:28:41 -05:00
Justin Campbell c993e9bed6 registry/response: Add protocols to DL resp 2018-11-20 14:09:16 -05:00
Justin Campbell 06825bf46d plugin/discovery: Add friendly gpg err msg
When GPG verification fails, display a helpful message to the user instead of the generic openpgp error.
2018-11-20 14:09:16 -05:00
Justin Campbell 9a8a74b9bb plugin/discovery: Print name before verification
This is so that any errors output from the checksum/signature verification show up in the expected place in the output.
2018-11-20 14:09:16 -05:00
Justin Campbell 495826444b plugin/discovery: Use GPG keys from Registry
When verifying the signature of the SHA256SUMS file, we have been
hardcoding HashiCorp's public GPG key and using it as the keyring.

Going forward, Terraform will get a list of valid public keys for a
provider from the Terraform Registry (registry.terraform.io), and use
them as the keyring for the openpgp verification func.
2018-11-20 14:09:16 -05:00
Sander van Harmelen 17787c943a
Update CHANGELOG.md 2018-11-20 17:57:07 +01:00
Sander van Harmelen 77844852e6 go-mod: update go-tfe 2018-11-20 17:06:19 +01:00
Sander van Harmelen d58d10e204
Merge pull request #19411 from hashicorp/svh/f-state-push
command/state: lock when pushing state
2018-11-20 16:26:08 +01:00
Sander van Harmelen 79a9a15879 command/state: lock when pushing state
Next to adding the locking for the `state push` command, this commit also fixes a small bug where the lock would not be propertly released when running the `state show` command.

And finally it renames some variables in the `[un]taint` code in order to try to standardize the var names of a few frequently used variables (e.g. statemgr.Full, states.State, states.SyncState).
2018-11-20 11:15:16 +01:00
James Bardin 407fcc0385
Merge pull request #19409 from hashicorp/jbardin/terraform-tests
fixes for the remaining tests
2018-11-19 19:19:24 -05:00
James Bardin f375691819 add missing key-value from test 2018-11-19 18:58:29 -05:00
James Bardin 0b7be2d0e3 fixes for the remaining tests
It's possible that a computed collection could be handled by the
attribute name, rather than the index count value.

Use a new testDiffFn for some tests, which don't work with the old
function that can't determine `computed` without the schema.
2018-11-19 18:56:50 -05:00
Martin Atkins 884aa387b8 command: Use vendoring when building helper programs in tests
In a couple places in tests we execute a child "go build" to make a helper
program. Now that we're running in module mode, "go build" will normally
default to downloading and caching dependencies, which we don't want
because we're still using vendoring for the moment.

Therefore we need to instruct these child builds to use vendoring too,
avoiding the need to download all of the dependencies and ensuring that
we'll be building with the same dependencies that we'd use for a normal
build.
2018-11-19 11:41:52 -08:00
Martin Atkins 5b676059a4
Update CHANGELOG.md 2018-11-19 10:04:01 -08:00
Martin Atkins 6606b525ec
Update CHANGELOG.md 2018-11-19 10:03:17 -08:00
Martin Atkins 4fe9632f09 plugin: Establish our current plugin protocol as version 5
The main significant change here is that the package name for the proto
definition is "tfplugin5", which is important because this name is part
of the wire protocol for references to types defined in our package.

Along with that, we also move the generated package into "internal" to
make it explicit that importing the generated Go package from elsewhere is
not the right approach for externally-implemented SDKs, which should
instead vendor the proto definition they are using and generate their
own stubs to ensure that the wire protocol is the only hard dependency
between Terraform Core and plugins.

After this is merged, any provider binaries built against our
helper/schema package will need to be rebuilt so that they use the new
"tfplugin5" package name instead of "proto".

In a future commit we will include more elaborate and organized
documentation on how an external codebase might make use of our RPC
interface definition to implement an SDK, but the primary concern here
is to ensure we have the right wire package name before release.
2018-11-19 09:56:41 -08:00
Martin Atkins 1ff9a54020 build: Temporarily disable the website tests for Travis-CI
There are some known broken links right now because this repository has
been updated ahead of some necessary changes in the terraform-website
repository. We'll need to wait until the end of the v0.12 release process
to re-enable this because the terraform-website repository is currently
set up for the v0.11 content and will continue to be until v0.12.0 final
is ready for release.
2018-11-19 09:02:35 -08:00
Martin Atkins c133de863b build: Update most things for Go 1.11 modules
We're still using vendoring for now until we get _all_ of our tooling
updated, so the main idea here is to force use of the vendor directory
when running tests and building for development so we can quickly find
situations where we forget to run "go mod vendor".

We also setting GO111MODULE=off for installation of tools. Right now this
is the best way to install a tool in GOBIN without also interfering with
go.mod and go.sum, until a better pattern for managing tool dependencies
is devised by the Go team.

Finally, we run "go mod download" before launching "gox" in the main
build process, to prime the local module cache once so that the concurrent
"go build" processes won't race to populate it redundantly. This means
that we'll be producing final builds from the module cache rather than
from vendor as with everything else -- there's currently no way to tell
gox to use -mod=vendor -- but that should be fine in practice since
our go.sum file will ensure that we get the exact sources we expect in
the module cache before building.
2018-11-19 09:02:35 -08:00
Martin Atkins 5255e85238 "go fmt" fixups
Apparently my editor is still not reliably formatting on save, so I missed
a few formatting quirks in these files.
2018-11-19 09:02:35 -08:00
Martin Atkins 73c9521a04 command/e2etest: Temporarily disable tests that access network
Several of these tests rely on external services (e.g. Terraform Registry)
that have not yet been updated to support the needs of Terraform v0.12.0,
so for now we'll skip all of these tests and wait until those systems have
been updated.

This should be removed before Terraform v0.12.0 final to enable these
tests to be used as part of pre-release smoke testing.
2018-11-19 09:02:35 -08:00
Martin Atkins 300eceeb25 plans/planfile: fix TestRoundtrip
This was broken by an earlier change to verify the Terraform version
number when reading a state file. To fix it, we'll use our current version
in our constructed file which should then match when it's read back in.
2018-11-19 09:02:35 -08:00
Martin Atkins 6cb3b0f4cf states/statemgr: Local locks meta is near output path, not input path
This was a mistake while adapting this code from the old state.LocalState.
Since the lock is held on the output file (s.path) the metadata should
live adjacent to that rather than being built from the read path
(s.readPath) that is used only as the initial snapshot on first
instantiation.

This also includes more logging, continuing the trend of other recent
commits in these files. The local state behavior is sufficiently complex
that these trace logs are a great help in debugging issues such as this
one with the wrong files being used or actions being taken in the wrong
order.
2018-11-19 09:02:35 -08:00
Martin Atkins 37bc187f95 command: "terraform output" mustn't panic when no state is present
This is verified by TestOutput_noArgs.
2018-11-19 09:02:35 -08:00
Martin Atkins 53b5b95ef5 command: Fix TestRefresh_backup
The local filesystem state manager no longer creates backup files eagerly,
instead creating them only if on first write there is already a snapshot
present in the target file.

Therefore for this test to exercise the codepaths it intends to we must
create an initial state snapshot for it to overwrite, creating the backup
in the process.

There are several other tests for this behavior elsewhere, so this test
is primarily to verify that the refresh command is configuring the backend
appropriately to get the backups written in the desired location.
2018-11-19 09:02:35 -08:00
Martin Atkins 762a173c7f command: Fix TestRefresh_outPath
We now only create a backup state file if the given output file already
exists, which it does not in this test.

(The behavior of creating the backup files is already covered by other
tests, so no need for this one go out of its way to do it.)
2018-11-19 09:02:35 -08:00
Martin Atkins e2ba90fdfa command: Fix TestMetaBackend_planLocalMatch
We now don't create a local state backup until the first snapshot write,
so we don't expect there to be a backup file until the end of the test.
(There is already a check at the end there, unmodified by this change.)
2018-11-19 09:02:35 -08:00
Martin Atkins 48601d261d states/statemgr: In Filesystem, back up output file, not input file
The filesystem backend has the option of using a different file for its
initial read.

Previously we were incorrectly writing the contents of that file out into
the backup file, rather than the prior contents of the output file. Now
we will always read the output file in RefreshState in order to decide
what we will back up but then we will optionally additionally read the
input file and prefer its content as the "current" state snapshot.

This is verified by command.TestMetaBackend_planLocalStatePath and
TestMetaBackend_configureNew, which are both now passing.
2018-11-19 09:02:35 -08:00
Martin Atkins 27abd9c6b8 command: Fix TestMetaBackend_localDoesNotDeleteLocal
The changes to how we handle setting the state path on the local backend
broke the heuristic we were using here for detecting migration from one
local backend to another with the same state path, which would by default
end up deleting the state altogether after migration.

We now use the StatePaths method to do this, which takes into account
both the default values and any settings that have been set.

Additionally this addresses a flaw in the old method which could
potentially have deleted all non-default workspace state files if the
"path" setting were changed without also changing the "workspace_dir"
setting. This new approach is conservative because it will preserve all
of the files if any one overlaps.
2018-11-19 09:02:35 -08:00
Martin Atkins be79bf0412 command: Fix TestPlan_outBackend
In an earlier change we fixed the "backendFromConfig" codepath to be
able to properly detect changes to the -backend-config arguments during
"terraform init", but this detection is too strict for the normal case
of running an operation in a previously-initialized directory.

Before any of the recent changes, the logic here was to selectively update
the hash to include -backend-config settings in the init case. Since
that late hash recalculation was confusing, here we take the alternative
path of using the hash only in the normal case and full value comparison
in the init case. Treating both of these cases separately makes things
marginally easier to follow here.
2018-11-19 09:02:35 -08:00
Martin Atkins b316e4ab56 command: Fix TestImport_remoteState
The import command was imposing the default state path at the CLI level,
rather than leaving that to be handled by the backend. As a result, the
output state was always forced to be terraform.tfstate, regardless of
the backend settings.
2018-11-19 09:02:35 -08:00
Martin Atkins 2b9f92be31 command: Partially fix TestMetaBackend_planLocalStatePath
This test is testing some strange implementation details of the old
local backend which do not hold with the new filesystem state manager.
Specifically, it was expecting state to be read from the stateOutPath
rather than the statePath, which makes no sense here because the backend
is configured to read from the default terraform.tfstate file (which does
not exist.)

There is another problem with this test which will be addressed in a
subsequent commit.
2018-11-19 09:02:35 -08:00
Martin Atkins aecb66d3db command: Fix TestMetaBackend_configuredChangeCopy_multiToNoDefaultWithoutDefault
As part of integrating the new "remote" backend we relaxed the requirement
that a "default" workspace must exist in all backends and now skip
migrating empty workspace states to avoid creating unnecessary "default"
workspaces when switching between backends that require it and backends
that don't, such as when switching from the local backend (which always
has a "default" workspace) to Terraform Enterprise.
2018-11-19 09:02:35 -08:00
Martin Atkins ec27526cc3 command: Fix TestMetaBackend_configuredChangeCopy_multiToMulti
This was failing because we now handle the settings for the local backend
a little differently as a result of decoding it with the HCL2 machinery.

Specifically, the backend.State* fields are now assumed to be what is
given in configuration, and any CLI overrides are maintained separately
in OverrideState* fields so that they can be imposed "just in time" in
StatePaths.

This is particularly important because OverrideStatePath (when set) is
used regardless of workspace name, while StatePath is a suitable value
only for the "default" workspace, with others needing to be constructed
from StateWorkspaceDir instead.
2018-11-19 09:02:35 -08:00
Martin Atkins 6c7cecfbd8 command: More logging during migration
This just finishes off the logging added in earlier commits to get all
the way through to the actual migration call.
2018-11-19 09:02:35 -08:00
Martin Atkins f39a5d0962 command: Fix various TestMetaBackend-prefix tests
Our new state model has a different implementation of "empty" that doesn't
consider lineage/serial, so we need to have some actual content in these
state fixtures to avoid them being skipped during state migrations.
2018-11-19 09:02:35 -08:00
Martin Atkins 22c84c71a4 command: Use statemgr.Import and statemgr.Export for state push and pull
We previously hacked around the import/export functionality being missing
in the statemgr layer after refactoring, but now it's been reintroduced
to fix functionality elsewhere we should use the centralized Import and
Export functions to ensure consistent behavior.

In particular, this pushes the logic for checking lineage and serial
during push down into the state manager itself, which is better because
all other details about lineage and serial are managed within the state
managers.
2018-11-19 09:02:35 -08:00
Martin Atkins 985b414dca states/statemgr: Fix the Filesystem state manager tests
Now that we're verifying the Terraform version during state loading, we
need to force a particular Terraform version to use during these tests.
2018-11-19 09:02:35 -08:00
Martin Atkins aacbe1d14b command: Fix TestMetaBackend_configureNewWithState
This test was initially failing because its fixture had a state which our
new state models consider to be "empty", and thus it was not migrated.

After fixing that (by adding an output to the fixture), this revealed a
bug that the lineage was not being persisted through the migration. This
is fixed by using the statemgr.Migrate method instead of writing via the
normal Writer interface, which allows two cooperating state managers to
properly transfer the lineage and serial along with the state snapshot.
2018-11-19 09:02:35 -08:00
Martin Atkins 24046ab833 command: More TRACE logging for the Backend instantiation codepaths 2018-11-19 09:02:35 -08:00
Martin Atkins 94510bc1b9 states/statemgr: Migrate, Import, and Export functions
In our recent refactoring of the state manager interfaces we made serial
and lineage management the responsibility of the state managers
themselves, not exposing them at all to most callers, and allowing for
simple state managers that don't implement them at all.

However, we do have some specific cases where we need to preserve these
properly when available, such as migration between backends, and the
"terraform state push" and "terraform state pull" commands.

These new functions and their associated optional interface allow the
logic here to be captured in one place and access via some simple
calls. Separating this from the main interface leaves things simple for
the normal uses of state managers.

Since these functions are mostly just thin wrappers around other
functionality, they are not yet well-tested directly, but will be
indirectly tested through the tests of their callers. A subsequent commit
will add more unit tests here.
2018-11-19 09:02:35 -08:00
Martin Atkins 2293391241 command: Fix TestMetaBackend_emptyWithExplicitState
This test was incorrectly updated in a previous iteration, with it
creating a modified state to write but then not actually writing it,
writing an empty test state instead.

This made the test fail because a backup state file is created only if
the new state snapshot is different to the old when written.
2018-11-19 09:02:35 -08:00
Radek Simko da5ae6e382
Merge pull request #19382 from hashicorp/b-tfdiags-enhancements
tfdiags: Detect more complex cfgs in ElaborateFromConfigBody
2018-11-17 14:27:24 +00:00
James Bardin e4270993be remove unused value 2018-11-16 15:28:30 -05:00