Commit Graph

216 Commits

Author SHA1 Message Date
Lee Trout cb0e20ca2b Add support for force pushing with the remote backend
Both differing serials and lineage protections should be bypassed
with the -force flag (in addition to resources).

Compared to other backends we aren’t just shipping over the state
bytes in a simple payload during the persistence phase of the push
command and the force flag added to the Go TFE client needs to be
specified at that time.

To prevent changing every method signature of PersistState of the
remote client I added an optional interface that provides a hook
to flag the Client as operating in a force push context. Changing
the method signature would be more explicit at the cost of not
being used anywhere else currently or the optional interface pattern
could be applied to the state itself so it could be upgraded to
support PersistState(force bool) only when needed.

Prior to this only the resources of the state were checked for
changes not the lineage or the serial. To bring this in line with
documented behavior noted above those attributes also have a “read”
counterpart just like state has. These are now checked along with
state to determine if the state as a whole is unchanged.

Tests were altered to table driven test format and testing was
expanded to include WriteStateForMigration and its interaction
with a ClientForcePusher type.
2020-05-06 12:07:43 -04:00
Lee Trout c2c38b2ad3 Add remote state test for serial and lineage changes
We only persist a new state if the actual state contents have
changed. This test demonstrates that behavior by calling write
and persist methods when either the lineage or serial have changed.
2020-05-04 11:48:50 -04:00
aqche 3578a5d80a
state: update local unlock err (#24320) 2020-03-17 14:01:51 -04:00
Pam Selle 9631e4c73d
Merge pull request #20571 from sergkondr/fix_misspelling
fix misspelling
2019-08-13 17:13:13 -04:00
Martin Atkins 85eda8a059 state/remote: Don't persist snapshot for unchanged state
Previously we would write to the backend for every call to PersistState,
even if nothing changed since the last write, but update the serial only
if the state had changed.

The Terraform Cloud & Enterprise state storage have a simple safety check
that any future write with an already-used lineage and serial must be
byte-for-byte identical. StatesMarshalEqual is intended to detect that,
but it only actually detects changes the state itself, and not changes
to the snapshot metadata.

Because we write the current Terraform version into the snapshot metadata
during serialization, we'd previously have an issue where if the first
state write after upgrading Terraform to a new version happened to change
nothing about the state content then we'd write a new snapshot that
differed only by Terraform version, and Terraform Cloud/Enterprise would
then reject it.

The snapshot header is discarded immediately after decoding, so we can't
use information from it when deciding whether to increment the serial.
The next best thing is to skip sending no-op snapshot updates to the state
client in the first place.

These writes are unnecessary anyway, and state storage owners have asked
us in the past to elide these to avoid generating noise in their version
logs, so we'll also finally meet those requests as a nice side-effect of
this change.

We didn't previously have tests for the full flow of retrieving and then
successively updating persisted state snapshots, so this includes a test
which covers that logic and includes an assertion that a no-op update does
not get written to the state client.
2019-06-20 06:18:40 -07:00
Martin Atkins 2124089e14 state/remote: Switch to statemgr interfaces in test
These statemgr interfaces are the new names for the older interfaces in
the "state" package. These types alias each other so it doesn't really
matter which we use, but the "state" package is deprecated and we intend
to eventually remove it, so this is a further step in that direction.
2019-06-20 06:18:40 -07:00
Sergey Kondrashov 43e7a7b552 fix misspelling 2019-03-05 16:12:52 +03:00
James Bardin 43b2c9a1d0 remove single rand source to prevent races
This shouldn't really be an issue in normal usage, but some of the
backend tests will trigger a race here.
2019-02-21 20:45:41 -05:00
Martin Atkins 94510bc1b9 states/statemgr: Migrate, Import, and Export functions
In our recent refactoring of the state manager interfaces we made serial
and lineage management the responsibility of the state managers
themselves, not exposing them at all to most callers, and allowing for
simple state managers that don't implement them at all.

However, we do have some specific cases where we need to preserve these
properly when available, such as migration between backends, and the
"terraform state push" and "terraform state pull" commands.

These new functions and their associated optional interface allow the
logic here to be captured in one place and access via some simple
calls. Separating this from the main interface leaves things simple for
the normal uses of state managers.

Since these functions are mostly just thin wrappers around other
functionality, they are not yet well-tested directly, but will be
indirectly tested through the tests of their callers. A subsequent commit
will add more unit tests here.
2018-11-19 09:02:35 -08:00
Sander van Harmelen a7b8cc8fe3 Do not clear out a previous set state when refreshing
In the case when no state existed remotely and a new one was created locally, we don’t want to blow away the new local state when refreshing.
2018-11-05 11:25:34 +01:00
Sander van Harmelen af1a471a05 command/state: update and fix the state list command 2018-10-19 16:31:12 +02:00
Martin Atkins 72a7947cb7 state/remote: Un-stub TestStateRace
The underlying bug that was causing this test to hang has since been
fixed.
2018-10-16 19:14:54 -07:00
Martin Atkins 78f1d1d1c0 state/remote: Don't hang in PersistState
We were calling from PersistState into RefreshState, but RefreshState is
protected by the same lock as PersistState and so the call would deadlock.

Instead, we introduce a new entry point refreshState which can be used
when already holding the lock.
2018-10-16 19:14:11 -07:00
Martin Atkins 477ea0d360 state/remote: Make tests compile and run to completion
One of the tests was hanging, so for now it's stubbed out until we can
get to the bottom of the hang on a subsequent pass.
2018-10-16 19:14:11 -07:00
Martin Atkins 4bdaffb586 state: Remove tests for obsolete components
Most of the functionality in this package has been obsoleted by equivalent
symbols in states/statemgr. We're keeping this package around for the
moment mainly just to house the type aliases in state.go.

The tests in here no longer work because they are written against the old
APIs. We will eventually remove the components these are testing too, but
we're keeping them around for the moment so that we don't break the build
for some leftover callers that are still depending on these.
2018-10-16 19:14:11 -07:00
Martin Atkins aaf405b662 backend/remote-state: Get all the backend tests building again
The state manager refactoring in an earlier commit was reflected in the
implementations of these backends, but not in their tests. This gets us
back to a state where the backend tests will compile, and gets _most_ of
them passing again, with a few exceptions that will be addressed in a
subsequent commit.
2018-10-16 19:14:11 -07:00
Martin Atkins a3403f2766 terraform: Ugly huge change to weave in new State and Plan types
Due to how often the state and plan types are referenced throughout
Terraform, there isn't a great way to switch them out gradually. As a
consequence, this huge commit gets us from the old world to a _compilable_
new world, but still has a large number of known test failures due to
key functionality being stubbed out.

The stubs here are for anything that interacts with providers, since we
now need to do the follow-up work to similarly replace the old
terraform.ResourceProvider interface with its replacement in the new
"providers" package. That work, along with work to fix the remaining
failing tests, will follow in subsequent commits.

The aim here was to replace all references to terraform.State and its
downstream types with states.State, terraform.Plan with plans.Plan,
state.State with statemgr.State, and switch to the new implementations of
the state and plan file formats. However, due to the number of times those
types are used, this also ended up affecting numerous other parts of core
such as terraform.Hook, the backend.Backend interface, and most of the CLI
commands.

Just as with 5861dbf3fc49b19587a31816eb06f511ab861bb4 before, I apologize
in advance to the person who inevitably just found this huge commit while
spelunking through the commit history.
2018-10-16 19:11:09 -07:00
James Bardin 979faa5dbe move artifactory remote state to backend 2018-10-16 18:23:14 -07:00
James Bardin 178eb6076e remove legacy remote file code
This code is not referenced at all
2018-10-16 18:22:37 -07:00
James Bardin 18ef072325 move legacy http remote state to a backend 2018-10-16 18:22:37 -07:00
James Bardin 3595bd3f81 remove dead legacy gcs code 2018-10-16 18:21:51 -07:00
James Bardin fe527ec9d7 move legacy etcd remote state to a backend 2018-10-16 18:21:51 -07:00
Sander van Harmelen 7fb2d1b8de Implement the Enterprise enhanced remote backend 2018-08-03 22:22:55 +02:00
James Bardin 9bf9280c03
Merge pull request #17431 from oscr/use-seekstart
Use io.SeekStart instead of deprecated os.SEEK_SET
2018-04-04 15:27:24 -04:00
James Bardin a84b4a669a use the open state file for refresh when possible
Only open a new file descriptor for RefreshState if we haven't written a
state, and don't have the correct state open already. This prevents
windows from failing to refresh a locked state.
2018-03-19 18:17:43 -04:00
James Bardin e10a7917e6 add failing test for windows state locks
Refreshing a locked state on windows could return nil if the read path
was locked, no state was yet written, and the read path is the same as
the write path.

Add a test that locks then refreshes a newly initialized state struct.
2018-03-19 17:09:53 -04:00
Paul Tyng c868092d2d
Standardize http.Client creation with User-Agent 2018-02-28 12:09:50 -05:00
Oscar Utbult 7434cd0a81 Use io.SeekStart instead of deprecated os.SEEK_SET 2018-02-25 23:11:40 +01:00
jwa 3d4642719f simulate a webdav backend 2018-01-19 02:30:21 +00:00
jwa b3d432e6bc allow HTTP 201 & 204 when storing remote state
Apache mod_dav returns 201 (created) and 204 (no content) during PUTs;
treat these as valid responses when using Apache as a http backend.
2018-01-11 22:09:59 +00:00
stack72 1fd0f803e4 Migrate Manta Remote state to be a backend
This PR changes manta from being a legacy remote state client to a new backend type. This also includes creating a simple lock within manta

This PR also unifies the way the triton client is configured (the schema) and also uses the same env vars to set the backend up

It is important to note that if the remote state path does not exist, then the backend will create that path. This means the user doesn't need to fall into a chicken and egg situation of creating the directory in advance before interacting with it
2017-10-30 18:36:50 +02:00
Florian Forster b09f121f86 state/remote: The "gcs" client has been superseeded by the "gcs" backend. 2017-10-27 16:52:21 -04:00
James Bardin 36b8be43e8 use the new version package
Update all references to the version values to use the new package.
The VersionString function was left in the terraform package
specifically for the aws provider, which is vendored. We can remove that
last call once the provider is updated.
2017-10-19 21:48:08 -04:00
Martin Atkins c12d64f340 Use t.Helper() in our test helpers
Go 1.9 adds this new function which, when called, marks the caller as
being a "helper function". Helper function stack frames are then skipped
when trying to find a line of test code to blame for a test failure, so
that the code in the main test function appears in the test failure output
rather than a line within the helper function itself.

This covers many -- but probaly not all -- of our test helpers across
various packages.
2017-08-28 09:59:30 -07:00
Martin Atkins 2e0c1d07ae Merge #15793: Locking support in HTTP remote backend 2017-08-22 09:43:38 -07:00
Ross McFarland d889ac38b0 Change remote/http store to update, more consistent with doc 2017-08-20 05:59:04 -07:00
Ross McFarland 510563b67f Fully test conf handling in httpFactory 2017-08-19 11:17:25 -07:00
Ross McFarland 69546c4b33 Pass at much more flexible remote/http backend
- Configurable Put (store) method, default POST to preserve behavior
- Configurable Lock method & address
- Configurable Unlock method & address

More thorough testing still needed, but this if functional
2017-08-19 10:31:47 -07:00
James Bardin 68da0390b7 remove legacy azure remote state code
The implementation has been moved to a backend.
2017-08-17 12:57:53 -04:00
Peter McAtominey f9e8e54835 backend: convert Azure remote state to a backend
Added locking support via blob leasing (requires that an empty state is
created before any lock can be acquired.

Added support for "environments" in much the same way as the S3 backend.
2017-08-17 16:32:17 +01:00
Ross McFarland 6cdea5af5d Clean up code and make the state/http behavior more consistent 2017-08-13 09:49:08 -07:00
Ross McFarland ce4d9fb3c2 Add tests for state/http with locking 2017-08-13 09:16:42 -07:00
Ross McFarland 1d38569c91 Add Lock/Unlock support to remote/http 2017-08-13 07:39:22 -07:00
James Bardin 32ae05c342 fix strict remote.State lineage check
We can't check lineage in the remote state instance, because we may need
to overwrite a state with a new lineage. Whil it's tempting to add an
optional interface for this, like OverwriteState(), optional interfaces
are never _really_ optional, and will have to be implemented by any
wrapper types as well.

Another solution may be to add a State.Supersedes field to indicate that
we intend to replace an existing state, but that may not be worth the
extra check either.
2017-08-01 19:34:22 -04:00
James Bardin fba5decae5 update TestState helper
In practice, States must all implement the full interface, so checking
for each method set only leaves gaps where tests could be skipped.
Change the helper to only accept a full state.State implementation.

Add some Lineage, Version, and TFVersion checks to TestState to avoid
regressions.

Compare the copy test against the immediate State returnedm rather than
our previous "current" state.

Check that the states round-trip and still marhsal identically via
MarshalEqual.
2017-07-05 17:18:12 -04:00
Martin Atkins 4d53eaa6df state: more robust handling of state Serial
Previously we relied on a constellation of coincidences for everything to
work out correctly with state serials. In particular, callers needed to
be very careful about mutating states (or not) because many different bits
of code shared pointers to the same objects.

Here we move to a model where all of the state managers always use
distinct instances of state, copied when WriteState is called. This means
that they are truly a snapshot of the state as it was at that call, even
if the caller goes on mutating the state that was passed.

We also adjust the handling of serials so that the state managers ignore
any serials in incoming states and instead just treat each Persist as
the next version after what was most recently Refreshed.

(An exception exists for when nothing has been refreshed, e.g. because
we are writing a state to a location for the first time. In that case
we _do_ trust the caller, since the given state is either a new state
or it's a copy of something we're migrating from elsewhere with its
state and lineage intact.)

The intent here is to allow the rest of Terraform to not worry about
serials and state identity, and instead just treat the state as a mutable
structure. We'll just snapshot it occasionally, when WriteState is called,
and deal with serials _only_ at persist time.

This is intended as a more robust version of #15423, which was a quick
hotfix to an issue that resulted from our previous slopping handling
of state serials but arguably makes the problem worse by depending on
an additional coincidental behavior of the local backend's apply
implementation.
2017-07-05 12:34:30 -07:00
Gavin Williams 96e6bf89ec remote/swift: Remove un-used code 2017-06-27 21:18:43 +01:00
Gavin Williams ee2e390f85 remote/swift: Migrate Swift remote state to a backend
Move the Swift State from a legacy remote state to an official backend.
Add `container` and `archive_container` configuration variables, and deprecate `path` and `archive_path` variables.

Future improvements: Add support for locking and environments.
2017-06-13 22:04:01 +01:00
James Bardin 808b504bcf rename openstack provider for swift remote state 2017-06-12 13:43:51 -04:00
Tom Harvey 13583b4b8b provider/azurerm: Upgrading to AutoRest v8 (#15006)
* Upgrading to AutoRest 8

* Upgrading to AutoRest v8

* Updating the Remote State to v8
2017-06-02 15:30:41 +01:00