Commit Graph

204 Commits

Author SHA1 Message Date
Martin Atkins 78f1d1d1c0 state/remote: Don't hang in PersistState
We were calling from PersistState into RefreshState, but RefreshState is
protected by the same lock as PersistState and so the call would deadlock.

Instead, we introduce a new entry point refreshState which can be used
when already holding the lock.
2018-10-16 19:14:11 -07:00
Martin Atkins 477ea0d360 state/remote: Make tests compile and run to completion
One of the tests was hanging, so for now it's stubbed out until we can
get to the bottom of the hang on a subsequent pass.
2018-10-16 19:14:11 -07:00
Martin Atkins 4bdaffb586 state: Remove tests for obsolete components
Most of the functionality in this package has been obsoleted by equivalent
symbols in states/statemgr. We're keeping this package around for the
moment mainly just to house the type aliases in state.go.

The tests in here no longer work because they are written against the old
APIs. We will eventually remove the components these are testing too, but
we're keeping them around for the moment so that we don't break the build
for some leftover callers that are still depending on these.
2018-10-16 19:14:11 -07:00
Martin Atkins aaf405b662 backend/remote-state: Get all the backend tests building again
The state manager refactoring in an earlier commit was reflected in the
implementations of these backends, but not in their tests. This gets us
back to a state where the backend tests will compile, and gets _most_ of
them passing again, with a few exceptions that will be addressed in a
subsequent commit.
2018-10-16 19:14:11 -07:00
Martin Atkins a3403f2766 terraform: Ugly huge change to weave in new State and Plan types
Due to how often the state and plan types are referenced throughout
Terraform, there isn't a great way to switch them out gradually. As a
consequence, this huge commit gets us from the old world to a _compilable_
new world, but still has a large number of known test failures due to
key functionality being stubbed out.

The stubs here are for anything that interacts with providers, since we
now need to do the follow-up work to similarly replace the old
terraform.ResourceProvider interface with its replacement in the new
"providers" package. That work, along with work to fix the remaining
failing tests, will follow in subsequent commits.

The aim here was to replace all references to terraform.State and its
downstream types with states.State, terraform.Plan with plans.Plan,
state.State with statemgr.State, and switch to the new implementations of
the state and plan file formats. However, due to the number of times those
types are used, this also ended up affecting numerous other parts of core
such as terraform.Hook, the backend.Backend interface, and most of the CLI
commands.

Just as with 5861dbf3fc49b19587a31816eb06f511ab861bb4 before, I apologize
in advance to the person who inevitably just found this huge commit while
spelunking through the commit history.
2018-10-16 19:11:09 -07:00
James Bardin 979faa5dbe move artifactory remote state to backend 2018-10-16 18:23:14 -07:00
James Bardin 178eb6076e remove legacy remote file code
This code is not referenced at all
2018-10-16 18:22:37 -07:00
James Bardin 18ef072325 move legacy http remote state to a backend 2018-10-16 18:22:37 -07:00
James Bardin 3595bd3f81 remove dead legacy gcs code 2018-10-16 18:21:51 -07:00
James Bardin fe527ec9d7 move legacy etcd remote state to a backend 2018-10-16 18:21:51 -07:00
Sander van Harmelen 7fb2d1b8de Implement the Enterprise enhanced remote backend 2018-08-03 22:22:55 +02:00
James Bardin 9bf9280c03
Merge pull request #17431 from oscr/use-seekstart
Use io.SeekStart instead of deprecated os.SEEK_SET
2018-04-04 15:27:24 -04:00
James Bardin a84b4a669a use the open state file for refresh when possible
Only open a new file descriptor for RefreshState if we haven't written a
state, and don't have the correct state open already. This prevents
windows from failing to refresh a locked state.
2018-03-19 18:17:43 -04:00
James Bardin e10a7917e6 add failing test for windows state locks
Refreshing a locked state on windows could return nil if the read path
was locked, no state was yet written, and the read path is the same as
the write path.

Add a test that locks then refreshes a newly initialized state struct.
2018-03-19 17:09:53 -04:00
Paul Tyng c868092d2d
Standardize http.Client creation with User-Agent 2018-02-28 12:09:50 -05:00
Oscar Utbult 7434cd0a81 Use io.SeekStart instead of deprecated os.SEEK_SET 2018-02-25 23:11:40 +01:00
jwa 3d4642719f simulate a webdav backend 2018-01-19 02:30:21 +00:00
jwa b3d432e6bc allow HTTP 201 & 204 when storing remote state
Apache mod_dav returns 201 (created) and 204 (no content) during PUTs;
treat these as valid responses when using Apache as a http backend.
2018-01-11 22:09:59 +00:00
stack72 1fd0f803e4 Migrate Manta Remote state to be a backend
This PR changes manta from being a legacy remote state client to a new backend type. This also includes creating a simple lock within manta

This PR also unifies the way the triton client is configured (the schema) and also uses the same env vars to set the backend up

It is important to note that if the remote state path does not exist, then the backend will create that path. This means the user doesn't need to fall into a chicken and egg situation of creating the directory in advance before interacting with it
2017-10-30 18:36:50 +02:00
Florian Forster b09f121f86 state/remote: The "gcs" client has been superseeded by the "gcs" backend. 2017-10-27 16:52:21 -04:00
James Bardin 36b8be43e8 use the new version package
Update all references to the version values to use the new package.
The VersionString function was left in the terraform package
specifically for the aws provider, which is vendored. We can remove that
last call once the provider is updated.
2017-10-19 21:48:08 -04:00
Martin Atkins c12d64f340 Use t.Helper() in our test helpers
Go 1.9 adds this new function which, when called, marks the caller as
being a "helper function". Helper function stack frames are then skipped
when trying to find a line of test code to blame for a test failure, so
that the code in the main test function appears in the test failure output
rather than a line within the helper function itself.

This covers many -- but probaly not all -- of our test helpers across
various packages.
2017-08-28 09:59:30 -07:00
Martin Atkins 2e0c1d07ae Merge #15793: Locking support in HTTP remote backend 2017-08-22 09:43:38 -07:00
Ross McFarland d889ac38b0 Change remote/http store to update, more consistent with doc 2017-08-20 05:59:04 -07:00
Ross McFarland 510563b67f Fully test conf handling in httpFactory 2017-08-19 11:17:25 -07:00
Ross McFarland 69546c4b33 Pass at much more flexible remote/http backend
- Configurable Put (store) method, default POST to preserve behavior
- Configurable Lock method & address
- Configurable Unlock method & address

More thorough testing still needed, but this if functional
2017-08-19 10:31:47 -07:00
James Bardin 68da0390b7 remove legacy azure remote state code
The implementation has been moved to a backend.
2017-08-17 12:57:53 -04:00
Peter McAtominey f9e8e54835 backend: convert Azure remote state to a backend
Added locking support via blob leasing (requires that an empty state is
created before any lock can be acquired.

Added support for "environments" in much the same way as the S3 backend.
2017-08-17 16:32:17 +01:00
Ross McFarland 6cdea5af5d Clean up code and make the state/http behavior more consistent 2017-08-13 09:49:08 -07:00
Ross McFarland ce4d9fb3c2 Add tests for state/http with locking 2017-08-13 09:16:42 -07:00
Ross McFarland 1d38569c91 Add Lock/Unlock support to remote/http 2017-08-13 07:39:22 -07:00
James Bardin 32ae05c342 fix strict remote.State lineage check
We can't check lineage in the remote state instance, because we may need
to overwrite a state with a new lineage. Whil it's tempting to add an
optional interface for this, like OverwriteState(), optional interfaces
are never _really_ optional, and will have to be implemented by any
wrapper types as well.

Another solution may be to add a State.Supersedes field to indicate that
we intend to replace an existing state, but that may not be worth the
extra check either.
2017-08-01 19:34:22 -04:00
James Bardin fba5decae5 update TestState helper
In practice, States must all implement the full interface, so checking
for each method set only leaves gaps where tests could be skipped.
Change the helper to only accept a full state.State implementation.

Add some Lineage, Version, and TFVersion checks to TestState to avoid
regressions.

Compare the copy test against the immediate State returnedm rather than
our previous "current" state.

Check that the states round-trip and still marhsal identically via
MarshalEqual.
2017-07-05 17:18:12 -04:00
Martin Atkins 4d53eaa6df state: more robust handling of state Serial
Previously we relied on a constellation of coincidences for everything to
work out correctly with state serials. In particular, callers needed to
be very careful about mutating states (or not) because many different bits
of code shared pointers to the same objects.

Here we move to a model where all of the state managers always use
distinct instances of state, copied when WriteState is called. This means
that they are truly a snapshot of the state as it was at that call, even
if the caller goes on mutating the state that was passed.

We also adjust the handling of serials so that the state managers ignore
any serials in incoming states and instead just treat each Persist as
the next version after what was most recently Refreshed.

(An exception exists for when nothing has been refreshed, e.g. because
we are writing a state to a location for the first time. In that case
we _do_ trust the caller, since the given state is either a new state
or it's a copy of something we're migrating from elsewhere with its
state and lineage intact.)

The intent here is to allow the rest of Terraform to not worry about
serials and state identity, and instead just treat the state as a mutable
structure. We'll just snapshot it occasionally, when WriteState is called,
and deal with serials _only_ at persist time.

This is intended as a more robust version of #15423, which was a quick
hotfix to an issue that resulted from our previous slopping handling
of state serials but arguably makes the problem worse by depending on
an additional coincidental behavior of the local backend's apply
implementation.
2017-07-05 12:34:30 -07:00
Gavin Williams 96e6bf89ec remote/swift: Remove un-used code 2017-06-27 21:18:43 +01:00
Gavin Williams ee2e390f85 remote/swift: Migrate Swift remote state to a backend
Move the Swift State from a legacy remote state to an official backend.
Add `container` and `archive_container` configuration variables, and deprecate `path` and `archive_path` variables.

Future improvements: Add support for locking and environments.
2017-06-13 22:04:01 +01:00
James Bardin 808b504bcf rename openstack provider for swift remote state 2017-06-12 13:43:51 -04:00
Tom Harvey 13583b4b8b provider/azurerm: Upgrading to AutoRest v8 (#15006)
* Upgrading to AutoRest 8

* Upgrading to AutoRest v8

* Updating the Remote State to v8
2017-06-02 15:30:41 +01:00
Tom Harvey df2875045d provider/azurerm: Upgrading to v10 of the Azure SDK (#14004)
provider/azurerm: Upgrading to v10.0.2 of the Azure SDK
2017-06-01 10:18:22 +01:00
James Bardin 4866f35645 add mutexes to Local, Backup, and InmemState 2017-05-25 11:20:52 -04:00
James Bardin f0f2220abb add mutexes to remote.State 2017-05-25 11:20:52 -04:00
Gavin Williams c63ad9c0f8 state/remote/swift: Support Openstack request logging (#13583)
* provider/openstack: Expose LogRoundTripper fields externally

* state/remote/swift: Add support for debugging Openstack calls using
OS_DEBUG env variable.

* provider/openstack: Update LogRoundTripper to log headers aswell as body.

* Add `RedactHeaders` function in order to redact sensitive http Headers.
Refactor `logRequest` and `logResponse` to use `RedactHeaders` func.
2017-04-15 17:11:28 +03:00
James Bardin eeddc3f8ea add some nil checks for unexpected lock failures 2017-04-05 09:06:47 -04:00
James Bardin af2e289212 remove Sleep from TestLockWithContext 2017-04-03 14:46:22 -04:00
James Bardin d1460d8c82 test LockWithContext 2017-04-03 11:50:19 -04:00
James Bardin 93b1dd6323 give LockWithContext a little backoff
Backoff the Lock calls exponentially, to a reasonable limit.
2017-04-01 18:01:49 -04:00
James Bardin 3f0dcd1308 Have the clistate Lock use LockWithContext
- Have the ui Lock helper use state.LockWithContext.
- Rename the message package to clistate, since that's how it's imported
  everywhere.
- Use a more idiomatic placement of the Context in the LockWithContext
  args.
2017-04-01 17:09:20 -04:00
James Bardin 826771a830 add state.LockWithContext
LockWithContext will retry a lock until the context expires or is
cancelled. This will let us implement a `-lock-timeout` flag, and make
use of existing contexts when applicable.
2017-04-01 17:01:55 -04:00
James Bardin 75458a182d remove extra state.Locker assertions
All states are lockers, so get rid of extra asertions.
2017-04-01 17:01:45 -04:00
James Bardin bf6384a163 All states are lockers
Since moving to the new backends, all states (except InmemState) are
Lockers. Add the methods to the State interface to remove a heap of
assertion checks.
2017-04-01 17:01:12 -04:00