Commit Graph

115 Commits

Author SHA1 Message Date
James Bardin 9f5cf2b105 convert S3 remote state to a backend
Move the S3 State from a legacy remote state to an official backend.

This increases test coverage, uses a set schema for configuration, and
will allow new backend features to be implemented for the S3 state, e.g.
"environments".
2017-03-22 10:59:37 -04:00
Ash Berlin 70de22253a Fix error message in Azure state backend (#12424)
The error led me to try adding `resource_group` but the code wanted `resource_group_name`. This fixes the error message to match the code.
2017-03-04 20:27:37 +02:00
Peter McAtominey cbed463b44 state/azure: add environment option for non-public cloud usage (#12364) 2017-03-02 04:58:24 +00:00
Mitchell Hashimoto 942572b574
backend/init: add atlas, remove legacy atlas remote state 2017-03-01 13:29:16 -08:00
James Bardin 7f40f90c4c remove the legacy Inmem remote state
moved to a new backend
2017-02-20 14:50:31 -05:00
Mitchell Hashimoto 03f6c650ca
state/remote: ClientLocker is just a Client that is a state.Locker 2017-02-15 14:27:02 -08:00
Mitchell Hashimoto 9451acc5da
state/remote: add unit test to verify s3 is a ClientLocker as well 2017-02-15 14:25:53 -08:00
Mitchell Hashimoto efe754183b
state/remote: export ClientLocker, test for implementation
This adds unit tests (that will fail at compile time) if various structs
don't implement the right interfaces for locking
2017-02-15 14:20:59 -08:00
James Bardin ec00564be6 Clean up LockInfo and LockError and use them
Gove LockInfo a Marshal method for easy serialization, and a String
method for more readable output.

Have the state.Locker implementations use LockError when possible to
return LockInfo and an error.
2017-02-15 14:44:43 -05:00
James Bardin 888af93356 Have S3 check the lockID on Unlock
This needs to be improved to happen in a transaction, but it gets the
implementation passing the new tests.
2017-02-15 14:44:43 -05:00
James Bardin 67bbebce08 Have consul state reutrn the lock ID
The lock ID isn't used because the lock is tied to the client, but
return the lock ID to match the behavior of other locks.
2017-02-15 14:44:43 -05:00
James Bardin 6aa1066f7c Store and use the correct IDs in TestRemoteLocks 2017-02-15 14:41:55 -05:00
James Bardin f5ed8cd288 Use NewLockInfo to get a pre-populated value
Using NewLockInfo ensure we start with all required fields filled.
2017-02-15 14:41:55 -05:00
James Bardin 6d32b70637 Make S3 remote state pass tests
TODO: update S3Client to make full use of the state.Locker interface
2017-02-15 14:41:55 -05:00
James Bardin 0ad6f08204 Make remote state test run
Make them compile against the new interface.
The tests will be updated later to check new behavior.
2017-02-15 14:41:55 -05:00
James Bardin 4d00c29706 Merge pull request #11836 from hashicorp/jbardin/state-locking
Add locking during backend configuration
2017-02-09 18:30:10 -05:00
Daren Desjardins 5718294386 state/remote/s3: Fix Bug with Assume Role for Federated IAM Account (#10067)
* Enable remote s3 state support for assume role

- provide role_arn in backend config to enable assume role

Fixes #8739

* Check for errors after obtaining credentials
2017-02-09 16:54:49 +00:00
James Bardin 5c2e945b3c Fix format string and typos 2017-02-09 10:27:42 -05:00
James Bardin 14d965722e Use single state.LockInfo struct
Remove redundant structures
2017-02-08 11:34:31 -05:00
James Bardin 9b76f6e138 Move TestRemoteLocks to state/remote
This was legacy remote state client and backends can use this test
function without an import cycle.
2017-02-08 11:25:52 -05:00
James Bardin a2b5811f50 Remove "expires" from lock info.
We are not going to handle lock expiration, at least at this time, so
remove the Expires fields to avoid any confusion.
2017-02-03 14:55:21 -05:00
James Bardin 10f6d7f30f Add locking for s3 state
Use a DynamoDB table to coodinate state locking in S3.

We use a simple strategy here, defining a key containing the value of
the bucket/key of the state file as the lock. If the keys exists, the
locks fails.

TODO: decide if locks should automatically be expired, or require manual
intervention.
2017-01-30 17:16:57 -05:00
James Bardin 35307d5a60 Add remote state locking
In order to provide lockign for remote states, the Cache state,
remote.State need to expose Lock and Unlock methods. The actual locking
will be done by the remote.Client, which can implement the same
state.Locker methods.
2017-01-30 17:16:57 -05:00
Mitchell Hashimoto d3633ab368
state/remote: remove consul since we converted that 2017-01-26 14:33:50 -08:00
Ben Slusky e3b20c3508 Basic authentication for HTTP remote state backend 2017-01-19 21:01:50 -05:00
Aleksander Modzelewski 2627e4bd8d Add token authentication to Swift remote 2016-12-20 18:22:48 +01:00
Joe Topjian 8ecc6fda0a Merge pull request #10055 from fatmcgav/swift_add_version_expire
state/remote/swift: Add support for versioning state file in swift, and expiring objects
2016-11-22 09:57:49 -07:00
Gavin Williams c77ed69a4e state/remote/swift: Add support for versioning state file in swift, and expiring object after period of time.
Update website docs with changes
2016-11-22 16:46:10 +00:00
Peter McAtominey 507efcb180 state/azure: support passing of lease ID when writing storage blob (#10115)
Also fixed tests failing auth caused by getStorageAccountAccessKey returning the
key name rather than the value

TF_ACC= go test ./state/remote -v -run=TestAz -timeout=10m -parallel=4
=== RUN   TestAzureClient_impl
--- PASS: TestAzureClient_impl (0.00s)
=== RUN   TestAzureClient
2016/11/18 13:57:34 [DEBUG] New state was assigned lineage "96037426-f95e-45c3-9183-6c39b49f590b"
2016/11/18 13:57:34 [TRACE] Preserving existing state lineage "96037426-f95e-45c3-9183-6c39b49f590b"
--- PASS: TestAzureClient (130.60s)
=== RUN   TestAzureClientEmptyLease
2016/11/18 13:59:44 [DEBUG] New state was assigned lineage "d9997445-1ebf-4b2c-b4df-15ae152f6417"
2016/11/18 13:59:44 [TRACE] Preserving existing state lineage "d9997445-1ebf-4b2c-b4df-15ae152f6417"
--- PASS: TestAzureClientEmptyLease (128.15s)
=== RUN   TestAzureClientLease
2016/11/18 14:01:55 [DEBUG] New state was assigned lineage "85912a12-2e0e-464c-9886-8add39ea3a87"
2016/11/18 14:01:55 [TRACE] Preserving existing state lineage "85912a12-2e0e-464c-9886-8add39ea3a87"
--- PASS: TestAzureClientLease (138.09s)
PASS
ok  	github.com/hashicorp/terraform/state/remote	397.111s
2016-11-18 17:26:25 +02:00
Gavin Williams bf8612b9b7 state/remote/swift: Enhancements to support full set of Openstack configuration options, plus SSL certs. Documentation updated to support 2016-11-03 07:36:18 +00:00
Gavin Williams ac07430124 state/remote/swift: Update to use Gophercloud/gophercloud, restructure to support insecure TLS, added Keystone v3 auth params
Update swift remote documentation
2016-11-01 16:55:05 +00:00
Cameron Watters b4eb63d710 state/remote: New provider - manta
- add remote state provider backed by Joyent's Manta
- add documentation of Manta remote state provider
- explicitly check for passphrase-protected SSH keys, which are currently
  unsupported, and generate a more helpful error (borrowed from Packer's
  solution to the same problem):
  https://github.com/mitchellh/packer/blob/master/common/ssh/key.go#L27
2016-09-14 20:44:52 +01:00
James Nugent 8d0a68e1d4 state/remote: Officially Support local backend
This is a rework of pull request #6213 submitted by @joshuaspence,
adjusted to work with the remote state data source. We also add
a deprecation warning for people using the unsupported API, and retain
the ability to refer to "_local" as well as "local" for users in a mixed
version environment.
2016-09-03 15:42:40 -07:00
Ian Duffy 767914bbdc [GH-1275] Support for AWS access via IAMs AssumeRole functionality
This commit enables terraform to utilise the assume role functionality
of sts to execute commands with different privileges than the API
keys specified.

Signed-off-by: Ian Duffy <ian@ianduffy.ie>
2016-09-02 10:22:57 -07:00
James Bardin cdb80f68a8 Ensure better state normalization
Fix checksum issue with remote state

If we read a state file with "null" objects in a module and they become
initialized to an empty map the state file may be written out with empty
objects rather than "null", changing the checksum. If we can detect
this, increment the serial number to prevent a conflict in atlas.

Our fakeAtlas test server now needs to decode the state directly rather
than using the ReadState function, so as to be able to read the state
unaltered.

The terraform.State data structures have initialization spread out
throughout the package. More thoroughly initialize State during
ReadState, and add a call to init() during WriteState as another
normalization safeguard.

Expose State.init through an exported Init() method, so that a new State
can be completely realized outside of the terraform package.
Additionally, the internal init now completely walks all internal state
structures ensuring that all maps and slices are initialized.  While it
was mentioned before that the `init()` methods are problematic with too
many call sites, expanding this out better exposes the entry points that
will need to be refactored later for improved concurrency handling.

The State structures had a mix of `omitempty` fields. Remove omitempty
for all maps and slices as part of this normalization process. Make
Lineage mandatory, which is now explicitly set in some tests.
2016-08-12 11:09:50 -04:00
Guido Bakker ee9ebe849f Support setting datacenter when using consul remote state (#8102)
* Support setting datacenter when using consul remote state

Change-Id: I8c03f4058e9373f0de8fde7ce291ec552321cc60

* Add documentation for setting datacenter when using consul remote state

Change-Id: Ia62feea7a910a76308f0a5e7f9505c9a210e0339
2016-08-11 22:58:20 +12:00
James Bardin a1a501a26a Merge pull request #8053 from hashicorp/jbardin/atlas-retry-policy
don't retry the atlas requests with the wrong cert
2016-08-10 17:09:00 -04:00
James Bardin df0c795b39 Don't retry the atlas requests with the wrong cert
This probably won't recover, so abort immediately.
Requires retryablehttp CheckRetry patch.
2016-08-10 13:45:30 -04:00
Renier Morales c2bcb5fbe5 Skip IAM/STS validation and metadata check (#7874)
* Skip IAM/STS validation and metadata check

* Skip IAM/STS identity validation - For environments or other api
  implementations where there are no IAM/STS endpoints available, this
  option lets you opt out from that provider initialization step.
* Skip metdata api check - For environments in which you know ahead of
  time there isn't going to be a metadta api endpoint, this option lets
  you opt out from that check to save time.

* Allow iam/sts initialization even if skipping account/cred validation

(#7874)

* Split out skip of IAM validation into credentials and account id

(#7874)
2016-08-10 15:10:34 +01:00
James Bardin 0a3714eaac Don't send access_token in request params
Always send the access_token in the X-Atlas-Token header.
2016-08-05 11:44:14 -04:00
James Bardin 74813821ec Add remote state init test
Verify that a remote state file is correctly initialized in the same
manner as used by the `terraform remote config`
2016-07-07 16:24:38 -04:00
James Bardin 3622bfddd6 Revert #7464 and allow an empty state
Revert back to using a nil state. The external usage of the state shoudl
always check the Empty() method.
2016-07-07 16:19:58 -04:00
James Bardin 24f6d3fe98 Return an error when there's no remote state
When refreshing remote state, indicate when no state file was found with
an ErrRemoteStateNotFound error. This prevents us from inadvertantly
getting a nil state into a terraform.State where we assume there's
always a root module.
2016-07-01 18:44:23 -04:00
Paul Stack 079e1f9a56 provider/azurerm: Bump azure-sdk-for-go to 3.0.0-beta (#7420)
provider/azurerm: Bump azure-sdk-for-go to 3.0.0-beta
2016-06-30 15:36:08 +01:00
Paul Stack 32e1a32476 remote: Rename mas to azure (#7114)
MAS wasn't obvious it was Azure so renamed it to Azure
2016-06-10 20:04:40 +02:00
Maxime Bury c98f391bee Add basic implementation for remote state on azure (#7064)
* Add basic implementation for remote state on azure

* Don't auto-provision the container

* Fix compilation errors

* Add factory to the remote map

* Add documentation

* Add acceptance tests
2016-06-10 19:27:57 +02:00
James Nugent 706ccb7dfe core: Introduce state v3 and upgrade process
This commit makes the current Terraform state version 3 (previously 2),
and a migration process as part of reading v2 state. For the most part
this is unnecessary: helper/schema will deal with upgrading state for
providers written with that framework. However, for providers which
implemented the resource model directly, this gives a best-efforts
attempt at lossless upgrade.

The heuristics used to change the count of a map from the .# key to the
.% key are as follows:

    - if the flat map contains any non-numeric keys, we treat it as a
      map
    - if the map is empty it must be computed or optional, so we remove
      it from state

There is a known edge condition: maps with all-numeric keys are
indistinguishable from sets without access to the schema. They will need
manual conversion or may result in spurious diffs.
2016-06-09 10:49:49 +01:00
Matt Morrison cbfb4d8b86 remote state: Add GCS provider for remote state 2016-05-31 13:42:57 -05:00
James Nugent 3ea3c657b5 core: Use OutputState in JSON instead of map
This commit forward ports the changes made for 0.6.17, in order to store
the type and sensitive flag against outputs.

It also refactors the logic of the import for V0 to V1 state, and
fixes up the call sites of the new format for outputs in V2 state.

Finally we fix up tests which did not previously set a state version
where one is required.
2016-05-18 13:25:20 -05:00
Joseph Anthony Pasquale Holsten 546fb94265 atlas: update test err msg to reflect real timeout 2016-05-12 15:30:17 -07:00