Commit Graph

186 Commits

Author SHA1 Message Date
stack72 1fd0f803e4 Migrate Manta Remote state to be a backend
This PR changes manta from being a legacy remote state client to a new backend type. This also includes creating a simple lock within manta

This PR also unifies the way the triton client is configured (the schema) and also uses the same env vars to set the backend up

It is important to note that if the remote state path does not exist, then the backend will create that path. This means the user doesn't need to fall into a chicken and egg situation of creating the directory in advance before interacting with it
2017-10-30 18:36:50 +02:00
Florian Forster b09f121f86 state/remote: The "gcs" client has been superseeded by the "gcs" backend. 2017-10-27 16:52:21 -04:00
James Bardin 36b8be43e8 use the new version package
Update all references to the version values to use the new package.
The VersionString function was left in the terraform package
specifically for the aws provider, which is vendored. We can remove that
last call once the provider is updated.
2017-10-19 21:48:08 -04:00
Martin Atkins c12d64f340 Use t.Helper() in our test helpers
Go 1.9 adds this new function which, when called, marks the caller as
being a "helper function". Helper function stack frames are then skipped
when trying to find a line of test code to blame for a test failure, so
that the code in the main test function appears in the test failure output
rather than a line within the helper function itself.

This covers many -- but probaly not all -- of our test helpers across
various packages.
2017-08-28 09:59:30 -07:00
Martin Atkins 2e0c1d07ae Merge #15793: Locking support in HTTP remote backend 2017-08-22 09:43:38 -07:00
Ross McFarland d889ac38b0 Change remote/http store to update, more consistent with doc 2017-08-20 05:59:04 -07:00
Ross McFarland 510563b67f Fully test conf handling in httpFactory 2017-08-19 11:17:25 -07:00
Ross McFarland 69546c4b33 Pass at much more flexible remote/http backend
- Configurable Put (store) method, default POST to preserve behavior
- Configurable Lock method & address
- Configurable Unlock method & address

More thorough testing still needed, but this if functional
2017-08-19 10:31:47 -07:00
James Bardin 68da0390b7 remove legacy azure remote state code
The implementation has been moved to a backend.
2017-08-17 12:57:53 -04:00
Peter McAtominey f9e8e54835 backend: convert Azure remote state to a backend
Added locking support via blob leasing (requires that an empty state is
created before any lock can be acquired.

Added support for "environments" in much the same way as the S3 backend.
2017-08-17 16:32:17 +01:00
Ross McFarland 6cdea5af5d Clean up code and make the state/http behavior more consistent 2017-08-13 09:49:08 -07:00
Ross McFarland ce4d9fb3c2 Add tests for state/http with locking 2017-08-13 09:16:42 -07:00
Ross McFarland 1d38569c91 Add Lock/Unlock support to remote/http 2017-08-13 07:39:22 -07:00
James Bardin 32ae05c342 fix strict remote.State lineage check
We can't check lineage in the remote state instance, because we may need
to overwrite a state with a new lineage. Whil it's tempting to add an
optional interface for this, like OverwriteState(), optional interfaces
are never _really_ optional, and will have to be implemented by any
wrapper types as well.

Another solution may be to add a State.Supersedes field to indicate that
we intend to replace an existing state, but that may not be worth the
extra check either.
2017-08-01 19:34:22 -04:00
James Bardin fba5decae5 update TestState helper
In practice, States must all implement the full interface, so checking
for each method set only leaves gaps where tests could be skipped.
Change the helper to only accept a full state.State implementation.

Add some Lineage, Version, and TFVersion checks to TestState to avoid
regressions.

Compare the copy test against the immediate State returnedm rather than
our previous "current" state.

Check that the states round-trip and still marhsal identically via
MarshalEqual.
2017-07-05 17:18:12 -04:00
Martin Atkins 4d53eaa6df state: more robust handling of state Serial
Previously we relied on a constellation of coincidences for everything to
work out correctly with state serials. In particular, callers needed to
be very careful about mutating states (or not) because many different bits
of code shared pointers to the same objects.

Here we move to a model where all of the state managers always use
distinct instances of state, copied when WriteState is called. This means
that they are truly a snapshot of the state as it was at that call, even
if the caller goes on mutating the state that was passed.

We also adjust the handling of serials so that the state managers ignore
any serials in incoming states and instead just treat each Persist as
the next version after what was most recently Refreshed.

(An exception exists for when nothing has been refreshed, e.g. because
we are writing a state to a location for the first time. In that case
we _do_ trust the caller, since the given state is either a new state
or it's a copy of something we're migrating from elsewhere with its
state and lineage intact.)

The intent here is to allow the rest of Terraform to not worry about
serials and state identity, and instead just treat the state as a mutable
structure. We'll just snapshot it occasionally, when WriteState is called,
and deal with serials _only_ at persist time.

This is intended as a more robust version of #15423, which was a quick
hotfix to an issue that resulted from our previous slopping handling
of state serials but arguably makes the problem worse by depending on
an additional coincidental behavior of the local backend's apply
implementation.
2017-07-05 12:34:30 -07:00
Gavin Williams 96e6bf89ec remote/swift: Remove un-used code 2017-06-27 21:18:43 +01:00
Gavin Williams ee2e390f85 remote/swift: Migrate Swift remote state to a backend
Move the Swift State from a legacy remote state to an official backend.
Add `container` and `archive_container` configuration variables, and deprecate `path` and `archive_path` variables.

Future improvements: Add support for locking and environments.
2017-06-13 22:04:01 +01:00
James Bardin 808b504bcf rename openstack provider for swift remote state 2017-06-12 13:43:51 -04:00
Tom Harvey 13583b4b8b provider/azurerm: Upgrading to AutoRest v8 (#15006)
* Upgrading to AutoRest 8

* Upgrading to AutoRest v8

* Updating the Remote State to v8
2017-06-02 15:30:41 +01:00
Tom Harvey df2875045d provider/azurerm: Upgrading to v10 of the Azure SDK (#14004)
provider/azurerm: Upgrading to v10.0.2 of the Azure SDK
2017-06-01 10:18:22 +01:00
James Bardin 4866f35645 add mutexes to Local, Backup, and InmemState 2017-05-25 11:20:52 -04:00
James Bardin f0f2220abb add mutexes to remote.State 2017-05-25 11:20:52 -04:00
Gavin Williams c63ad9c0f8 state/remote/swift: Support Openstack request logging (#13583)
* provider/openstack: Expose LogRoundTripper fields externally

* state/remote/swift: Add support for debugging Openstack calls using
OS_DEBUG env variable.

* provider/openstack: Update LogRoundTripper to log headers aswell as body.

* Add `RedactHeaders` function in order to redact sensitive http Headers.
Refactor `logRequest` and `logResponse` to use `RedactHeaders` func.
2017-04-15 17:11:28 +03:00
James Bardin eeddc3f8ea add some nil checks for unexpected lock failures 2017-04-05 09:06:47 -04:00
James Bardin af2e289212 remove Sleep from TestLockWithContext 2017-04-03 14:46:22 -04:00
James Bardin d1460d8c82 test LockWithContext 2017-04-03 11:50:19 -04:00
James Bardin 93b1dd6323 give LockWithContext a little backoff
Backoff the Lock calls exponentially, to a reasonable limit.
2017-04-01 18:01:49 -04:00
James Bardin 3f0dcd1308 Have the clistate Lock use LockWithContext
- Have the ui Lock helper use state.LockWithContext.
- Rename the message package to clistate, since that's how it's imported
  everywhere.
- Use a more idiomatic placement of the Context in the LockWithContext
  args.
2017-04-01 17:09:20 -04:00
James Bardin 826771a830 add state.LockWithContext
LockWithContext will retry a lock until the context expires or is
cancelled. This will let us implement a `-lock-timeout` flag, and make
use of existing contexts when applicable.
2017-04-01 17:01:55 -04:00
James Bardin 75458a182d remove extra state.Locker assertions
All states are lockers, so get rid of extra asertions.
2017-04-01 17:01:45 -04:00
James Bardin bf6384a163 All states are lockers
Since moving to the new backends, all states (except InmemState) are
Lockers. Add the methods to the State interface to remove a heap of
assertion checks.
2017-04-01 17:01:12 -04:00
James Bardin 9f5cf2b105 convert S3 remote state to a backend
Move the S3 State from a legacy remote state to an official backend.

This increases test coverage, uses a set schema for configuration, and
will allow new backend features to be implemented for the S3 state, e.g.
"environments".
2017-03-22 10:59:37 -04:00
Mitchell Hashimoto 1daff7a826
backend/consul: support "lock" option to disable locking
This adds a "lock" config (default true) to allow users to optionally
disable state locking with Consul. This is necessary if the token given
doesn't have session permission and is necessary for backwards
compatibility.
2017-03-14 17:59:10 -07:00
Ash Berlin 70de22253a Fix error message in Azure state backend (#12424)
The error led me to try adding `resource_group` but the code wanted `resource_group_name`. This fixes the error message to match the code.
2017-03-04 20:27:37 +02:00
Peter McAtominey cbed463b44 state/azure: add environment option for non-public cloud usage (#12364) 2017-03-02 04:58:24 +00:00
Mitchell Hashimoto 942572b574
backend/init: add atlas, remove legacy atlas remote state 2017-03-01 13:29:16 -08:00
James Bardin 7f40f90c4c remove the legacy Inmem remote state
moved to a new backend
2017-02-20 14:50:31 -05:00
James Bardin a4283d23f6 fix windows locking
Don't create a new windows handle for the lock.
2017-02-17 13:29:48 -05:00
James Bardin c0dda6a7b3 test failure for windows local locks 2017-02-17 13:29:47 -05:00
Mitchell Hashimoto 03f6c650ca
state/remote: ClientLocker is just a Client that is a state.Locker 2017-02-15 14:27:02 -08:00
Mitchell Hashimoto 9451acc5da
state/remote: add unit test to verify s3 is a ClientLocker as well 2017-02-15 14:25:53 -08:00
Mitchell Hashimoto efe754183b
state/remote: export ClientLocker, test for implementation
This adds unit tests (that will fail at compile time) if various structs
don't implement the right interfaces for locking
2017-02-15 14:20:59 -08:00
James Bardin ec00564be6 Clean up LockInfo and LockError and use them
Gove LockInfo a Marshal method for easy serialization, and a String
method for more readable output.

Have the state.Locker implementations use LockError when possible to
return LockInfo and an error.
2017-02-15 14:44:43 -05:00
James Bardin 888af93356 Have S3 check the lockID on Unlock
This needs to be improved to happen in a transaction, but it gets the
implementation passing the new tests.
2017-02-15 14:44:43 -05:00
James Bardin 67bbebce08 Have consul state reutrn the lock ID
The lock ID isn't used because the lock is tied to the client, but
return the lock ID to match the behavior of other locks.
2017-02-15 14:44:43 -05:00
James Bardin 08cff7cc13 have LocalState check Lock ID on Unlock
Have LocalState store and check the lock ID, and strictly enforce
unlocking with the correct ID.

This isn't required for local lock correctness, as we track the file
descriptor to unlock, but it does provide a varification that locking
and unlocking is done correctly throughout terraform.
2017-02-15 14:41:55 -05:00
James Bardin 6aa1066f7c Store and use the correct IDs in TestRemoteLocks 2017-02-15 14:41:55 -05:00
James Bardin f5ed8cd288 Use NewLockInfo to get a pre-populated value
Using NewLockInfo ensure we start with all required fields filled.
2017-02-15 14:41:55 -05:00
James Bardin 52b2343672 make state test pass with new state.Locker 2017-02-15 14:41:55 -05:00