Commit Graph

280 Commits

Author SHA1 Message Date
James Bardin 0c91d227fa
Merge branch 'master' into master 2019-06-10 15:50:59 -04:00
Ivan Kalita 5b6b1663ef backend/http: implement retries for the http backend (#19702)
Fixes #19619
2019-06-05 16:12:07 -04:00
Mateusz Gozdek b6de825723 backend/gcs: make Unlock error message more clear 2019-05-27 20:07:14 -07:00
Stephen Buergler fc5b186e8d Don't leak so many connections in the pg backend
This change fixes an error I get:
Error: pq: too many connections for role "asdf"
because I can only have so many connections.
2019-04-30 23:34:51 -05:00
He Guimin 3f44dd9dec Add tablestore config to store state lock 2019-04-11 07:11:10 +08:00
He Guimin b887d44712 Add Alibaba Cloud backend OSS with lock 2019-04-11 07:06:43 +08:00
Justin Downing 1e32ae243c grammatical updates to comments and docs (#20195) 2019-03-21 14:05:41 -07:00
Stefan Schmidt 3ca1253796 backend/gcs: Mark the unused attributes project and region as removed.
These two attributes are not used since aec45e6967.
2019-03-19 17:29:04 -07:00
Mars Hall 34fa67c6b0 Correct pg backend error info 2019-03-05 11:37:08 -08:00
Mars Hall ccf549a62d Stricter pg backend locking during workspace creation 2019-03-05 11:19:56 -08:00
Mars Hall 31c9776d55 Switch pg backend to session-level advisory locking, to avoid rollback of partial state updates 2019-03-04 17:01:08 -08:00
Brian Flad e34a97abca
backend/s3: Add debug logging and user agent
Porting over previous behavior from terraform-provider-aws.
2019-02-27 12:59:08 -05:00
James Bardin eada955721
Merge pull request #19070 from mars/postgres-backend
Postgres backend
2019-02-26 20:20:01 -05:00
Mars Hall 6d9e2048f4 Skip more pg backend acceptance tests 2019-02-26 16:50:36 -08:00
Mars Hall 920e7a7acc Mutex pg backend‘s database transaction 2019-02-26 13:45:16 -08:00
Mars Hall c6bf3442fc Lock() in pg backend now returns LockError for retryable errors 2019-02-26 13:20:40 -08:00
Mars Hall a075a68c58 Correct name of pg backend‘s default schema 2019-02-26 12:59:31 -08:00
James Bardin dd16455a79 fix s3 default region to match provider 2019-02-25 20:27:21 -05:00
Mars Hall 2621f95bd2 📚 doc corrections 2019-02-25 16:30:30 -08:00
Mars Hall b9a91b7c1e Switch pg backend to use native Postgres locks 2019-02-25 16:05:53 -08:00
James Bardin c814f2da37 Change backend.ValidateConfig to PrepareConfig
This mirrors the change made for providers, so that default values can
be inserted into the config by the backend implementation. This is only
the interface and method name changes, it does not yet add any default
values.
2019-02-25 18:37:20 -05:00
James Bardin aebfecf871
Merge pull request #20211 from yanndegat/swift_remote_backend
Swift remote backend support for workspaces & locking
2019-02-25 12:58:59 -05:00
Mars Hall 8cb2943b6b pg backend tests run when TF_ACC is set (no more TF_PG_TEST) 2019-02-22 14:42:24 -08:00
Mars Hall c8b8227967 Compatibility fixes for 0.12.0 2019-02-22 13:54:41 -08:00
Mars Hall d4d798c25e Fix to use existing pg indexes 2019-02-22 13:51:57 -08:00
Mars Hall 9feda7aab3 Correct code comment for pg 2019-02-22 13:51:57 -08:00
Mars Hall 9043005cfb Postgres database backend 2019-02-22 13:48:49 -08:00
James Bardin 31ca293777 fix slash handling around workspace_key_prefix
The handling of slashes was broken around listing workspaces in
workspace_key_prefix. While it worked in most places by splitting an
extra time around the spurious slashes, it failed in the case that the
prefix ended with a slash of its own.

A test was temporarily added to verify that the backend works with the
unusual keys, but rather than risking silent breakage around prefixes
with trailing slashes, we also add validation to prevent users from
entering keys with trailing slashes at all.
2019-02-22 10:29:10 -05:00
James Bardin f6af61f990 fix test that never worked
The 0.12 update left this test broken. Make sure we're working with the
correct objects here.
2019-02-22 10:14:53 -05:00
Brian Flad 185a3304e5
Merge pull request #20374 from hashicorp/td-backend-s3-dependencies
backend/s3: Switch from github.com/terraform-providers/terraform-provider-aws to github.com/hashicorp/aws-sdk-go-base
2019-02-20 08:42:07 -08:00
Brian Flad a41e545198
backend/s3: Reinstate region validation and update copy for skip_requesting_account_id deprecation message 2019-02-20 08:29:09 -08:00
Brian Flad 43f12bbfe0
backend/s3: Update copy for skip_get_ec2_platforms and skip_requesting_account_id argument deprecations 2019-02-19 10:54:53 -08:00
Sander van Harmelen 54736b068b backend/remote: use `state.v2` for remote state only
The API surface area is much smaller when we use the remote backend for remote state only.

So in order to try and prevent any backwards incompatibilities when TF runs inside of TFE, we’ve split up the discovery services into `state.v2` (which can be used for remote state only configurations, so when running in TFE) and `tfe.v2.1` (which can be used for all remote configurations).
2019-02-19 10:59:51 +01:00
Brian Flad 1aaac172b0
backend/s3: Switch from github.com/terraform-providers/terraform-provider-aws to github.com/hashicorp/aws-sdk-go-base
Output from acceptance testing (no new failures):

```
--- PASS: TestBackend_impl (0.00s)
--- PASS: TestBackendConfig (0.37s)
--- PASS: TestBackendConfig_invalidKey (0.00s)
--- PASS: TestBackend (3.26s)
--- PASS: TestBackendLocked (6.80s)
--- FAIL: TestBackendExtraPaths (2.32s)
--- PASS: TestBackendPrefixInWorkspace (2.06s)
--- PASS: TestKeyEnv (8.20s)
--- PASS: TestRemoteClient_impl (0.00s)
--- PASS: TestRemoteClient (2.42s)
--- PASS: TestRemoteClientLocks (6.33s)
--- PASS: TestForceUnlock (13.31s)
--- PASS: TestRemoteClient_clientMD5 (11.75s)
--- PASS: TestRemoteClient_stateChecksum (10.07s)
```
2019-02-18 02:30:30 -05:00
Yann DEGAT e70b8928e9 remote/backend/swift: Add support for workspaces & locking 2019-02-11 11:13:35 +01:00
Radek Simko a7f0722729
backend/swift: Fix interface after upgrade 2019-02-08 11:03:03 +00:00
Radek Simko 6365fffe4f
Fix interface after upgrade 2019-02-01 10:48:56 +00:00
Brian Flad ed37d07632
backend/s3: Configure AWS Client MaxRetries and provide enhanced S3 NoSuchBucket error message
The AWS Go SDK automatically provides a default request retryer with exponential backoff that is invoked via setting `MaxRetries` or leaving it `nil` will default to 3. The terraform-aws-provider `config.Client()` sets `MaxRetries` to 0 unless explicitly configured above 0. Previously, we were not overriding this behavior by setting the configuration and therefore not invoking the default request retryer.

The default retryer already handles HTTP error codes above 500, including S3's InternalError response, so the extraneous handling can be removed. This will also start automatically retrying many additional cases, such as temporary networking issues or other retryable AWS service responses.

Changes:
* s3/backend: Add `max_retries` argument
* s3/backend: Enhance S3 NoSuchBucket error to include additional information
2019-01-09 13:01:37 -05:00
Brian Flad 058434d28b
Merge pull request #19571 from hashicorp/f-backend-s3-other-endpoints
backend/s3: Support DynamoDB, IAM, and STS endpoint configurations
2018-12-10 19:28:01 -05:00
Tom Harvey 383bc98f5c
backend/azurerm: Support for authenticating using the Azure CLI (#19465)
* Upgrading to 2.0.0 of github.com/hashicorp/go-azure-helpers

* Support for authenticating using Azure CLI

* backend/azurerm: support for authenticating using the Azure CLI
2018-12-10 21:23:30 +00:00
Brian Flad 9a3b02cd6c
backend/s3: Support DynamoDB, IAM, and STS endpoint configurations
This change enables a few related use cases:
* AWS has partitions outside Commercial, GovCloud (US), and China, which are the only endpoints automatically handled by the AWS Go SDK. DynamoDB locking and credential verification can not currently be enabled in those regions.
* Allows usage of any DynamoDB-compatible API for state locking
* Allows usage of any IAM/STS-compatible API for credential verification
2018-12-07 03:10:51 -05:00
Tom Harvey 6d4f702467
backend/azurerm: support for custom resource manager endpoints (#19460)
* backend/azurerm: removing the `arm_` prefix from keys

* removing the deprecated fields test because the deprecation makes it fail

* authentication: support for custom resource manager endpoints

* Adding debug prefixes to the log statements
2018-11-26 14:42:16 +01:00
Tom Harvey d580f30e03
backend/azurerm: removing the `arm_` prefix from keys (#19448)
* backend/azurerm: removing the `arm_` prefix from keys

* removing the deprecated fields test because the deprecation makes it fail
2018-11-26 11:19:43 +01:00
Tom Harvey 96b1c951fa
backend/azurerm: support for authenticating via SAS Tokens (#19440)
* adding acceptance tests for msi auth

* including the resource group name in the tests

* backend/azurerm: support for authenticating using a SAS Token

* resolving merge conflicts

* moving the defer to prior to the error
2018-11-22 18:02:33 +01:00
Tom Harvey e437feadc2
backend/azurerm: fixing a bug where locks couldn't be unlocked (#19441)
Fixes #17046
2018-11-22 17:47:35 +01:00
Tom Harvey c928962f44
backend/azurerm: support for authenticating via msi (#19433)
* backend/azurerm: support for authenticating via msi

* adding acceptance tests for msi auth

* including the resource group name in the tests

* support for using the test client via msi
2018-11-22 16:52:27 +01:00
Tom Harvey 0ec109bdc0
backend/azurerm: upgrading the SDK / support for proxies (#19414)
* vendor updates

- updating to v21.3.0 of github.com/Azure/azure-sdk-for-go
- updating to v10.15.4 of github.com/Azure/go-autorest
- vendoring github.com/hashicorp/go-azure-helpers @ 0.1.1

* backend/azurerm: refactoring to use the new auth package

- refactoring the backend to use a shared client via the new auth package
- adding tests covering both Service Principal and Access Key auth
- support for authenticating using a proxy
- rewriting the backend documentation to include examples of both authentication types

* switching to use the build-in logging function

* documenting it's also possible to retrieve the access key from an env var
2018-11-21 22:06:03 +01:00
Sander van Harmelen 52a1b22f7a Implement the remote enhanced backend
This is a refactored version of the `remote` backend that was initially added to Terraform v0.11.8 which should now be compatible with v0.12.0.
2018-11-06 16:29:46 +01:00
Sander van Harmelen 48ef7ecfa6 Updates after running `make fmt` with Go v1.11.1 2018-10-17 14:11:08 -07:00
Martin Atkins 8565c7460e backend/remote-state/inmem: remove FIXME guards on tests that no longer hang 2018-10-16 19:14:11 -07:00
Martin Atkins 20727945ee backend/remote-state/inmem: enable optional logging in tests 2018-10-16 19:14:11 -07:00
Martin Atkins aaf405b662 backend/remote-state: Get all the backend tests building again
The state manager refactoring in an earlier commit was reflected in the
implementations of these backends, but not in their tests. This gets us
back to a state where the backend tests will compile, and gets _most_ of
them passing again, with a few exceptions that will be addressed in a
subsequent commit.
2018-10-16 19:14:11 -07:00
Martin Atkins a3403f2766 terraform: Ugly huge change to weave in new State and Plan types
Due to how often the state and plan types are referenced throughout
Terraform, there isn't a great way to switch them out gradually. As a
consequence, this huge commit gets us from the old world to a _compilable_
new world, but still has a large number of known test failures due to
key functionality being stubbed out.

The stubs here are for anything that interacts with providers, since we
now need to do the follow-up work to similarly replace the old
terraform.ResourceProvider interface with its replacement in the new
"providers" package. That work, along with work to fix the remaining
failing tests, will follow in subsequent commits.

The aim here was to replace all references to terraform.State and its
downstream types with states.State, terraform.Plan with plans.Plan,
state.State with statemgr.State, and switch to the new implementations of
the state and plan file formats. However, due to the number of times those
types are used, this also ended up affecting numerous other parts of core
such as terraform.Hook, the backend.Backend interface, and most of the CLI
commands.

Just as with 5861dbf3fc49b19587a31816eb06f511ab861bb4 before, I apologize
in advance to the person who inevitably just found this huge commit while
spelunking through the commit history.
2018-10-16 19:11:09 -07:00
Martin Atkins 5782357c28 backend: Update interface and implementations for new config loader
The new config loader requires some steps to happen in a different
order, particularly in regard to knowing the schema in order to
decode the configuration.

Here we lean directly on the configschema package, rather than
on helper/schema.Backend as before, because it's generally
sufficient for our needs here and this prepares us for the
helper/schema package later moving out into its own repository
to seed a "plugin SDK".
2018-10-16 18:39:12 -07:00
James Bardin 979faa5dbe move artifactory remote state to backend 2018-10-16 18:23:14 -07:00
James Bardin 18ef072325 move legacy http remote state to a backend 2018-10-16 18:22:37 -07:00
James Bardin fe527ec9d7 move legacy etcd remote state to a backend 2018-10-16 18:21:51 -07:00
Sean Chittenden d749420a25
Fix drift caused from gofmt when running make dev and go 1.11.
A fresh checkout of `origin/master` does not build atm using the `dev`
target because `master` has not been formatted using `gofmt` from Go
1.11 (tis has been the case for a while if you've been running devel).

None of the drift in question is especially new but now that Go 1.11
has been released and gofmt's formatting guidelines have been updated,
it would be *really* nice if the code in `master` reflected the current
tooling in order to avoid having to fight this drift locally.

* 8mo: https://github.com/hashicorp/terraform/blame/master/backend/remote-state/s3/backend_test.go#L260-L261
* 6mo: https://github.com/hashicorp/terraform/blame/master/builtin/provisioners/chef/linux_provisioner_test.go#L124
* 1yr: 7cfeffe36b/command/init.go (L75-L76)
* 12d: 7cfeffe36b/command/meta_backend_test.go (L1437)
* 2yr: 7cfeffe36b/helper/schema/resource_timeout_test.go (L26)
* 4yr: 7cfeffe36b/helper/schema/schema_test.go (L2059)
* 1yr: 7cfeffe36b/plugin/discovery/get_test.go (L151)
2018-09-09 10:18:08 -07:00
Sander van Harmelen 77b9fad7f0 backend/manta: deprecate camelcase attribute name
This change is needed (next to making it consistent as its the only attribute name that uses camelcase) because passing that attribute objectName in the update `terraform_remote_state` resource, ended up giving me this error: https://github.com/hashicorp/terraform/blob/master/helper/schema/schema.go#L736
2018-08-29 20:22:04 +02:00
Sander van Harmelen 495d1ea350 Use New() instead of `once.Do(b.init)` 2018-08-03 11:29:11 +02:00
stack72 df9b2913ae backend/manta: Add support for manta backend tiered path structure
This was already added to triton-go and is now making its way to
the manta backend

```
% acctests backend/remote-state/manta
=== RUN   TestBackend_impl
--- PASS: TestBackend_impl (0.00s)
=== RUN   TestBackend
--- PASS: TestBackend (27.36s)
=== RUN   TestBackendLocked
--- PASS: TestBackendLocked (16.24s)
=== RUN   TestRemoteClient_impl
--- PASS: TestRemoteClient_impl (0.00s)
=== RUN   TestRemoteClient
--- PASS: TestRemoteClient (3.40s)
=== RUN   TestRemoteClientLocks
--- PASS: TestRemoteClientLocks (7.17s)
PASS
ok  	github.com/hashicorp/terraform/backend/remote-state/manta
```
2018-04-03 17:32:10 +03:00
stack72 a6f76aa0af backend/manta: Manta Backend was not dealing with a ResourceNotFound
Fixes: #17314

We now deal correctly with the creation of the state file - we were
not dealing well with a ResourceNotFound error

Now that this has been changed around, we try and create the statefile
and if there is an error, we look for an existing statefile - previously
this was not the order of operations
2018-03-12 17:27:57 +02:00
Paul Tyng c868092d2d
Standardize http.Client creation with User-Agent 2018-02-28 12:09:50 -05:00
James Bardin 12bda53558 Revert "create clistate.Locker interface"
This reverts commit e88bd74bb7.
2018-02-23 11:31:13 -05:00
James Bardin e88bd74bb7 create clistate.Locker interface
Simplify the use of clistate.Lock by creating a clistate.Locker
instance, which stores the context of locking a state, to allow unlock
to be called without knowledge of how the state was locked.

This alows the backend code to bring the needed UI methods to the point
where the state is locked, and still unlock the state from an outer
scope.
2018-02-23 11:21:21 -05:00
James Bardin 57cbcbe99d update remote state tests to use new test helpers 2018-02-20 21:05:58 -05:00
James Bardin 112c37458f report the proper lock id from a state lock error
Fix the now failing state unlock test by reporting the correct ID.

The ID used by GCS is the generation number of the info object, which
isn't known until the info is already written out. While we can't get
the correct ID from the info data for the error rmessage, we can update
it with the generation number after it's read.
2018-02-20 20:54:01 -05:00
James Bardin 1fee5ae172 update gcs to use the new backend tests 2018-02-20 20:36:08 -05:00
James Bardin a5ed7d0ae4
Merge pull request #17086 from hashicorp/jbardin/GH-17083
only split the prefix string once from the s3 key
2018-01-17 18:51:43 -05:00
stack72 e9476c6765 backend/manta: Update manta dependencies
Internally, triton-go has changed how it handles errors. We can now get rid of
checking strings for errors, and we have introduced an errors library that
wraps some of the major errors we encounter and test for
2018-01-17 11:34:02 -08:00
Masayuki Morita f440dba137 Standardize on log level "WARN" rather than "WARNING" 2018-01-16 18:05:26 -08:00
James Bardin 43d000625d only split the prefix string once from the s3 key
Ensure that the prefix is only split off a single time when the
workspace_key_prefix is a substring of the workspace or key name.
2018-01-11 10:08:02 -05:00
Paddy e4cdbd6c9f
Merge pull request #16936 from negz/gcskeys
Support 'customer supplied encryption keys' in the GCS backend
2018-01-09 01:17:35 -08:00
James Bardin 7c703f2ab2
Merge pull request #16932 from rv-jmaggio/master
Fixing issues with workspace_key_prefix for s3 backend
2018-01-05 13:24:40 -05:00
James Bardin dd5882e5b2
Merge pull request #16992 from hashicorp/jbardin/go1.10-prep
go1.10 prep
2018-01-03 15:41:41 -05:00
goraxe fb6b349e58 backend/s3: Allow forcing use of the S3 path-style API
This is useful when the underlying API is an S3-workalike service, rather than official S3.
2018-01-03 12:15:47 -08:00
Paul Stack 191cf283d5 backend/manta: Support Triton RBAC
Triton Manta allows an account other than the main triton account to be used via RBAC.

Here we expose the SDC_USER / TRITON_USER options to the backend so that a user can be specified.
2018-01-03 12:12:46 -08:00
James Bardin 4b49a323c3 go fmt
slight change to go fmt coming in 0.10
2017-12-26 13:26:38 -05:00
rv-jmaggio b313ce80c4 Changing prefix for empty workspace prefix 2017-12-19 13:14:31 -05:00
Henry Bell 56357c0ab7 Fix typo ('depracted' -> 'deprecated') 2017-12-19 16:43:06 +00:00
rv-jmaggio bef64cfe91 Fixing implementation for empty string and making acceptance test work 2017-12-19 09:31:53 -05:00
rv-jmaggio b02a1c8a46 clarifying tests and using SplitN in implementation 2017-12-18 16:24:34 -05:00
Nic Cope 011841124b Support 'customer supplied encryption keys' in the GCS backend
https://cloud.google.com/storage/docs/encryption#customer-supplied

GCS state created using customer supplied encryption keys can only be read or
modified using the same key.
2017-12-17 19:27:52 -08:00
rv-jmaggio 7f8d686074 refactor and add a test 2017-12-15 21:04:15 -05:00
rv-jmaggio 0889c118a8 Fixing issues with workspace_key_prefix 2017-12-15 17:50:36 -05:00
James Bardin 52eced589f accept a path or contents for credentials
Match the operation of the google provider, by accepting either a file
path or contents for both `credentials` and `GOOGLE_CREDENTIALS`
2017-12-06 18:33:59 -05:00
James Bardin 9dea2f78d4 create unique buckets for each test, and clean up
This creates a unique bucket name for each test, so that the tests in
parallel don't collide, and buckets left over from interrupted tests
don't cause future failures.

Also make sure that buckets are removed, regardless of content.
2017-12-06 18:33:59 -05:00
James Bardin aec45e6967 backends should never create unmanaged resources
The backend was creating bucket named in the configuration if it didn't
exist. We don't allow other backends to do this, because these are not
managed resources that terraform can control.
2017-12-06 18:32:41 -05:00
Kaveh Mousavi Zamani 7507e3cd21 backend/gcs: fix locking issue when used with terraform_remote_state
Previously there was a problem with double-locking when using the GCS backend with the terraform_remote_state data source.

Here we adjust the locking methodology to avoid that problem.
2017-12-06 09:36:16 -08:00
Sander van Harmelen aaedf255c3 backend/s3: allow skipping the region check (#16757)
Without the possibility to skip this check, it’s not possible to use a custom region with a third-party service that mimicks the S3 API.
2017-12-05 11:19:36 -08:00
Nic Cope 27ba7de77d backend/gcs: automatically add trailing slash to prefix
The backend doesn't function correctly if a trailing slash is not present, due to how workspaces are enumerated.
2017-11-08 11:24:36 -08:00
Malik Bougacha ed0d57d1b7 Get the enviroment from the environment variable (#16326)
* Get the enviroment from the enviroment variable

* typo
2017-11-06 16:59:48 +00:00
stack72 1fd0f803e4 Migrate Manta Remote state to be a backend
This PR changes manta from being a legacy remote state client to a new backend type. This also includes creating a simple lock within manta

This PR also unifies the way the triton client is configured (the schema) and also uses the same env vars to set the backend up

It is important to note that if the remote state path does not exist, then the backend will create that path. This means the user doesn't need to fall into a chicken and egg situation of creating the directory in advance before interacting with it
2017-10-30 18:36:50 +02:00
James Bardin cf54ca3b0f update tests for new consul packages
Reuse the running consul server for all tests.

Update the lostLockConnection package, since the api client should no
longer lose a lock immediately on network errors.
2017-10-29 12:40:44 -04:00
James Bardin 3a03d3683e update consul api packages
This is from a commit just after the v1.0.0 release, because it removes
the Porter service dependency for tests. The client api package was not
changed.
2017-10-28 19:44:10 -04:00
James Bardin 55089e472d
Merge pull request #16484 from hashicorp/f-gcloud-backend
Convert gcloud backend
2017-10-27 17:23:22 -04:00
Florian Forster afa13a3d8e backend/remote-state/gcs: Move toBucketName to the tests. 2017-10-27 16:52:21 -04:00
Florian Forster 3023ae2813 backend/remote-state/gcs: Include project ID in bucket names when testing.
Since bucket names must be *globally* unique. By including the project
ID in the bucket name we ensure that people don't step on each other's
feet when testing.
2017-10-27 16:52:21 -04:00
Florian Forster dcb84ee2c2 backend/remote-state/gcs: Sanitize bucket names. 2017-10-27 16:52:21 -04:00
Florian Forster 454d6bbe2a backend/remote-state/gcs: Delete test buckets after tests complete.
This way tests clean up after themselves and don't leak buckets.
2017-10-27 16:52:21 -04:00