Commit Graph

541 Commits

Author SHA1 Message Date
kayrus d06609dd23 Swift backend: add application credential support 2019-06-12 17:23:13 +02:00
Joe Topjian d8343aa95f backend/swift: Authentication updates
Support for cross-domain authentication has been added and mapping
environment variables to the correct domain settings has been
fixed.

In addition, support for clouds.yaml files has been added.
2019-06-12 08:05:54 -07:00
James Bardin 0c91d227fa
Merge branch 'master' into master 2019-06-10 15:50:59 -04:00
Ivan Kalita 5b6b1663ef backend/http: implement retries for the http backend (#19702)
Fixes #19619
2019-06-05 16:12:07 -04:00
Martin Atkins b1213f7f6c backend/local: don't panic when an instance has only a deposed object
This unusual situation isn't supposed to arise in normal use, but it can
come up in practice in some edge-case scenarios where Terraform fails in
a severe way during a create_before_destroy.

Some earlier versions of Terraform also had bugs in their handling of
deposed objects, so this may also arise if upgrading from one of those
older versions with some leftover deposed objects in the state.
2019-06-04 09:23:29 -07:00
Mateusz Gozdek b6de825723 backend/gcs: make Unlock error message more clear 2019-05-27 20:07:14 -07:00
David Celis 5ccccfb91f
Remove pre-alpha cost estimation logic 2019-05-20 16:48:38 -07:00
Radek Simko 8a6d1d62b6
stringer: Regenerate files with latest version 2019-05-13 15:34:27 +01:00
Stephen Buergler fc5b186e8d Don't leak so many connections in the pg backend
This change fixes an error I get:
Error: pq: too many connections for role "asdf"
because I can only have so many connections.
2019-04-30 23:34:51 -05:00
Sander van Harmelen 394f20f59c backend/remote: do not unlock after a failed upload
When changes are made and we failed to upload the state, we should not
try to unlock the workspace. Leaving the workspace locked is a good
indication something went wrong and also prevents other changes from
being applied before the newest state is properly uploaded.

Additionally we now output the lock ID when a lock or force-unlock
action failed.
2019-04-29 21:23:33 +02:00
Paul Thrasher 151c91ffda
use scanner for reading logs
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-04-25 11:17:08 -07:00
Paul Thrasher 0e27a8862f
remove duplicate remote output line
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-04-25 10:52:19 -07:00
Sander van Harmelen bb12206bca Fixup the tests 2019-04-25 10:32:00 +02:00
Sander van Harmelen 7cf744241a Do not use a scanner to read the logs
Using a scanner can cause issues when reading long lines. Also make sure we return the error correctly while planning.
2019-04-25 09:51:52 +02:00
Paul Thrasher c7a023a95c
update test for new go-tfe version
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-04-24 16:46:17 -07:00
Paul Thrasher 53f977bee2
TFCE api correct pluralization
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-04-24 16:44:00 -07:00
Paul Thrasher e479bd5dc3
update to latest go-tfe
same version number but pointing to a new sha

Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-04-24 16:44:00 -07:00
Sander van Harmelen 90bc237b7b Prevent a panic caused by writing to a nil map 2019-04-11 14:34:14 +02:00
He Guimin 3f44dd9dec Add tablestore config to store state lock 2019-04-11 07:11:10 +08:00
He Guimin b887d44712 Add Alibaba Cloud backend OSS with lock 2019-04-11 07:06:43 +08:00
Sander van Harmelen 39a95e4222 backend/remote: correctly load remote variables
When using `terraform console` in combination with the remote backend, variables defined in Terraform Enterprise were load loaded correctly.
2019-03-28 17:23:48 +01:00
Sander van Harmelen 57f6e01830 backend/local: preserve serial and lineage on failure
When failing to write the state, the local backend writes the state to a local file called `errrored.tfstate`. Previously it would do so by creating a new state file which would use a new serial and lineage. By exorting the existing state file and directly assigning the new state, the serial and lineage are preserved.
2019-03-27 16:15:16 +01:00
Justin Downing 1e32ae243c grammatical updates to comments and docs (#20195) 2019-03-21 14:05:41 -07:00
Stefan Schmidt 3ca1253796 backend/gcs: Mark the unused attributes project and region as removed.
These two attributes are not used since aec45e6967.
2019-03-19 17:29:04 -07:00
Sander van Harmelen 9f6a126293 backend/remote: check for external updates 2019-03-08 19:18:07 +01:00
Sander van Harmelen 4628fbcc65
Merge pull request #20615 from hashicorp/svh/f-input-context
core: add a context to the UIInput interface
2019-03-08 18:44:28 +01:00
Sander van Harmelen e75e845804
Merge pull request #20588 from hashicorp/svh/f-retry
backend/remote: also retry on server errors
2019-03-08 18:17:29 +01:00
Sander van Harmelen 973e2a7cf9 core: add a context to the UIInput interface 2019-03-08 10:24:40 +01:00
Sander van Harmelen 0232d84a0d backend/remote: also retry on server errors
Enably retrying on server errors in the updated `go-tfe` client and add a retry log hook for writing retry messages to the CLI.
2019-03-06 13:36:06 +01:00
Kristin Laemmert b9d8e96e0c
command/plan: plan output should indicate if a resource is being (#20580)
replaced because the instance was tainted.
2019-03-05 16:18:55 -08:00
Martin Atkins 709487b4f1 backend: Cap number of "undeclared variable" warnings at four
For users who in previous versions have relied on our lack of checking for
whether variables are declared, they may previously have seen an
overwhelming number of warnings when running Terraform v0.12.

Here we cap that number at three specific warnings and then one general
warning, so we can still give a specific source location for the first
couple (for users who have genuinely made a typo) but summarize away a
large number for those who are seeing this because they've not yet
migrated to using environment variables.
2019-03-05 15:42:08 -08:00
Mars Hall 34fa67c6b0 Correct pg backend error info 2019-03-05 11:37:08 -08:00
Mars Hall ccf549a62d Stricter pg backend locking during workspace creation 2019-03-05 11:19:56 -08:00
Sergey Kondrashov 43e7a7b552 fix misspelling 2019-03-05 16:12:52 +03:00
Mars Hall 31c9776d55 Switch pg backend to session-level advisory locking, to avoid rollback of partial state updates 2019-03-04 17:01:08 -08:00
Brian Flad e34a97abca
backend/s3: Add debug logging and user agent
Porting over previous behavior from terraform-provider-aws.
2019-02-27 12:59:08 -05:00
Sander van Harmelen 63e2dcef8a
Merge pull request #20481 from hashicorp/svh/b-exit-code
backend/remote: exit with 1 when a run is canceled
2019-02-27 07:52:20 +01:00
James Bardin eada955721
Merge pull request #19070 from mars/postgres-backend
Postgres backend
2019-02-26 20:20:01 -05:00
Mars Hall 6d9e2048f4 Skip more pg backend acceptance tests 2019-02-26 16:50:36 -08:00
Mars Hall 920e7a7acc Mutex pg backend‘s database transaction 2019-02-26 13:45:16 -08:00
Mars Hall c6bf3442fc Lock() in pg backend now returns LockError for retryable errors 2019-02-26 13:20:40 -08:00
Mars Hall a075a68c58 Correct name of pg backend‘s default schema 2019-02-26 12:59:31 -08:00
Sander van Harmelen 01f17fa0ca backend/remote: exit with 1 when a run is canceled 2019-02-26 21:00:07 +01:00
James Bardin dd16455a79 fix s3 default region to match provider 2019-02-25 20:27:21 -05:00
Mars Hall 2621f95bd2 📚 doc corrections 2019-02-25 16:30:30 -08:00
Mars Hall b9a91b7c1e Switch pg backend to use native Postgres locks 2019-02-25 16:05:53 -08:00
James Bardin c814f2da37 Change backend.ValidateConfig to PrepareConfig
This mirrors the change made for providers, so that default values can
be inserted into the config by the backend implementation. This is only
the interface and method name changes, it does not yet add any default
values.
2019-02-25 18:37:20 -05:00
Sander van Harmelen 435b82d6d0
Merge pull request #20460 from hashicorp/svh/f-permissions
backend/remote: use the can-queue-apply permission
2019-02-25 19:32:20 +01:00
James Bardin aebfecf871
Merge pull request #20211 from yanndegat/swift_remote_backend
Swift remote backend support for workspaces & locking
2019-02-25 12:58:59 -05:00
Sander van Harmelen 58961026a2 backend/remote: use the can-queue-apply permission
Previously we checked can-update in order to determine if a user had the
required permissions to apply a run, but that wasn't sufficient. So we
added a new permission, can-queue-apply, that we now use instead.
2019-02-25 14:45:40 +01:00
Mars Hall 8cb2943b6b pg backend tests run when TF_ACC is set (no more TF_PG_TEST) 2019-02-22 14:42:24 -08:00
Mars Hall c8b8227967 Compatibility fixes for 0.12.0 2019-02-22 13:54:41 -08:00
Mars Hall d4d798c25e Fix to use existing pg indexes 2019-02-22 13:51:57 -08:00
Mars Hall 9feda7aab3 Correct code comment for pg 2019-02-22 13:51:57 -08:00
Mars Hall 6d7e8268e7 Include pg backend in main init 2019-02-22 13:51:57 -08:00
Mars Hall 9043005cfb Postgres database backend 2019-02-22 13:48:49 -08:00
James Bardin 31ca293777 fix slash handling around workspace_key_prefix
The handling of slashes was broken around listing workspaces in
workspace_key_prefix. While it worked in most places by splitting an
extra time around the spurious slashes, it failed in the case that the
prefix ended with a slash of its own.

A test was temporarily added to verify that the backend works with the
unusual keys, but rather than risking silent breakage around prefixes
with trailing slashes, we also add validation to prevent users from
entering keys with trailing slashes at all.
2019-02-22 10:29:10 -05:00
James Bardin f6af61f990 fix test that never worked
The 0.12 update left this test broken. Make sure we're working with the
correct objects here.
2019-02-22 10:14:53 -05:00
James Bardin 6cc3e1d0bd move init error to where it is generated
The init error was output deep in the backend by detecting a
special ResourceProviderError and formatted directly to the CLI.

Create some Diagnostics closer to where the problem is detected, and
passed that back through the normal diagnostic flow. While the output
isn't as nice yet, this restores the helpful error message and makes the
code easier to maintain. Better formatting can be handled later.
2019-02-20 14:18:37 -05:00
Brian Flad 185a3304e5
Merge pull request #20374 from hashicorp/td-backend-s3-dependencies
backend/s3: Switch from github.com/terraform-providers/terraform-provider-aws to github.com/hashicorp/aws-sdk-go-base
2019-02-20 08:42:07 -08:00
Brian Flad a41e545198
backend/s3: Reinstate region validation and update copy for skip_requesting_account_id deprecation message 2019-02-20 08:29:09 -08:00
Sander van Harmelen 43eb7c08e0
Merge pull request #20379 from hashicorp/svh/f-state-service
backend/remote: use `state.v2` for remote state only
2019-02-20 10:48:05 +01:00
Brian Flad 43f12bbfe0
backend/s3: Update copy for skip_get_ec2_platforms and skip_requesting_account_id argument deprecations 2019-02-19 10:54:53 -08:00
Sander van Harmelen 54736b068b backend/remote: use `state.v2` for remote state only
The API surface area is much smaller when we use the remote backend for remote state only.

So in order to try and prevent any backwards incompatibilities when TF runs inside of TFE, we’ve split up the discovery services into `state.v2` (which can be used for remote state only configurations, so when running in TFE) and `tfe.v2.1` (which can be used for all remote configurations).
2019-02-19 10:59:51 +01:00
Brian Flad 1aaac172b0
backend/s3: Switch from github.com/terraform-providers/terraform-provider-aws to github.com/hashicorp/aws-sdk-go-base
Output from acceptance testing (no new failures):

```
--- PASS: TestBackend_impl (0.00s)
--- PASS: TestBackendConfig (0.37s)
--- PASS: TestBackendConfig_invalidKey (0.00s)
--- PASS: TestBackend (3.26s)
--- PASS: TestBackendLocked (6.80s)
--- FAIL: TestBackendExtraPaths (2.32s)
--- PASS: TestBackendPrefixInWorkspace (2.06s)
--- PASS: TestKeyEnv (8.20s)
--- PASS: TestRemoteClient_impl (0.00s)
--- PASS: TestRemoteClient (2.42s)
--- PASS: TestRemoteClientLocks (6.33s)
--- PASS: TestForceUnlock (13.31s)
--- PASS: TestRemoteClient_clientMD5 (11.75s)
--- PASS: TestRemoteClient_stateChecksum (10.07s)
```
2019-02-18 02:30:30 -05:00
Yann DEGAT e70b8928e9 remote/backend/swift: Add support for workspaces & locking 2019-02-11 11:13:35 +01:00
Martin Atkins 6eb7bfbdfb
Merge #20265: Don't presume unknown for values unset in config
This changes the contract for `PlanResourceChange` so that the provider is now responsible
for populating all default values during plan, including inserting any unknown values for
defaults it will fill in at apply time.
2019-02-08 13:54:55 -08:00
Martin Atkins e3618f915b backend/local: Fix mock provider in tests
We've changed the contract for PlanResourceChange to now require the
provider to populate any default values (including unknowns) it wants to
set for computed arguments, so our mock provider here now needs to be a
little more complex to deal with that.

This fixes several of the tests in this package. A minor change to
TestLocal_applyEmptyDirDestroy was required to make it properly configure
the mock provider so PlanResourceChange can access the schema.
2019-02-08 12:48:32 -08:00
Radek Simko e6777105b7
Merge pull request #20275 from hashicorp/vendor-openstack-upgrade
vendor: Bump openstack provider to v1.15.0
2019-02-08 16:22:35 +00:00
Sander van Harmelen aefbec63b1 backend/remote: update the test logic 2019-02-08 16:56:37 +01:00
Sander van Harmelen 3b80f69eec
Merge pull request #20242 from hashicorp/svh/b-scanner-buffer-v0.12
backend/remote: fix bufio.Scanner: token too long
2019-02-08 16:52:22 +01:00
Radek Simko a7f0722729
backend/swift: Fix interface after upgrade 2019-02-08 11:03:03 +00:00
Sander van Harmelen 47a00ea34b backend/remote: cleanup test connections
Cleanup test connection to prevent file descriptor issues when running the tests on a Mac.
2019-02-07 09:55:19 +01:00
Sander van Harmelen 5249d0fe83 backend/remote: fix bufio.Scanner: token too long 2019-02-07 09:54:43 +01:00
Radek Simko 6365fffe4f
Fix interface after upgrade 2019-02-01 10:48:56 +00:00
Sander van Harmelen 6dd9f22780
Merge pull request #20086 from hashicorp/svh/f-service-constraints
backend/remote: make sure we show the correct error
2019-01-24 20:22:20 +01:00
Sander van Harmelen 1e4c20686e backend/remote: make sure we show the correct error
Previously we would show two errors when there was a version constraint
error. But of course one is enough.
2019-01-23 15:09:42 +01:00
Martin Atkins 10bf4c763b backend: Undeclared variables in -var-file is a warning, not an error
In Terraform 0.11 and earlier we just silently ignored undeclared
variables in -var-file and the automatically-loaded .tfvars files. This
was a bad user experience for anyone who made a typo in a variable name
and got no feedback about it, so we made this an error for 0.12.

However, several users are now relying on the silent-ignore behavior for
automation scenarios where they pass the same .tfvars file to all
configurations in their organization and expect Terraform to ignore any
settings that are not relevant to a specific configuration. We never
intentionally supported that, but we don't want to immediately break that
workflow during 0.12 upgrade.

As a compromise, then, we'll make this a warning for v0.12.0 that contains
a deprecation notice suggesting to move to using environment variables
for this "cross-configuration variables" use-case. We don't produce errors
for undeclared variables in environment variables, even though that
potentially causes the same UX annoyance as ignoring them in vars files,
because environment variables are assumed to live in the user's session
and this it would be very inconvenient to have to unset such variables
when moving between directories. Their "ambientness" makes them a better
fit for these automatically-assigned general variable values that may or
may not be used by a particular configuration.

This can revert to being an error in a future major release, after users
have had the opportunity to migrate their automation solutions over to
use environment variables.

We don't seem to have any tests covering this specific situation right
now. That isn't ideal, but this change is so straightforward that it would
be relatively expensive to build new targeted test cases for it and so
I instead just hand-tested that it is indeed now producing a warning where
we were previously producing an error. Hopefully if there is any more
substantial work done on this codepath in future that will be our prompt
to add some unit tests for this.
2019-01-22 07:46:51 -08:00
Sander van Harmelen 8937fedb76 backend/remote: fix a small typo 2019-01-18 19:40:57 +01:00
Sander van Harmelen e08a7e979e backend/remote: use the correct test operation 2019-01-15 16:13:16 +01:00
Martin Atkins 0c0a437bcb Move module install functionality over to internal/initwd 2019-01-14 11:33:21 -08:00
Brian Flad ed37d07632
backend/s3: Configure AWS Client MaxRetries and provide enhanced S3 NoSuchBucket error message
The AWS Go SDK automatically provides a default request retryer with exponential backoff that is invoked via setting `MaxRetries` or leaving it `nil` will default to 3. The terraform-aws-provider `config.Client()` sets `MaxRetries` to 0 unless explicitly configured above 0. Previously, we were not overriding this behavior by setting the configuration and therefore not invoking the default request retryer.

The default retryer already handles HTTP error codes above 500, including S3's InternalError response, so the extraneous handling can be removed. This will also start automatically retrying many additional cases, such as temporary networking issues or other retryable AWS service responses.

Changes:
* s3/backend: Add `max_retries` argument
* s3/backend: Enhance S3 NoSuchBucket error to include additional information
2019-01-09 13:01:37 -05:00
Sander van Harmelen bba03384d5 backend/remote: log early to indicate execution started 2019-01-08 17:06:24 +01:00
Sander van Harmelen 7b51af72b2 backend/remote: compare versions without the prerelease 2018-12-19 19:06:22 +01:00
Sander van Harmelen 55b6153b04 backend/remote: fix an error that prevents checking constraints 2018-12-15 21:36:47 +01:00
Sander van Harmelen 2ef8315885
Merge pull request #19659 from hashicorp/svh/f-check-constraints
backend/remote: return detailed incompatibility info
2018-12-14 21:26:02 +01:00
Sander van Harmelen 8f04e93739 backend/remote: return detailed incompatibility info 2018-12-14 21:11:41 +01:00
Radek Simko f64978b64c
backend/local: Render CBD replacement (+/-) correctly (#19642)
* backend/local: Render CBD replacement (+/-) correctly

* command/format: Use IsReplace helper function
2018-12-14 13:45:47 +00:00
Sander van Harmelen da8e02eb2e
Merge pull request #19647 from hashicorp/svh/f-versions
core:  add a method to the disco package retrieve version constraints
2018-12-14 12:38:52 +01:00
Sander van Harmelen 268c0f85ce Add a method to retrieve version contraints 2018-12-14 12:17:31 +01:00
Radek Simko 3ab4739ba4
backend/local: Avoid rendering data sources on destroy 2018-12-12 18:21:49 +00:00
Radek Simko e55079abc9
backend/local: Fix incorrect destroy/update count on apply 2018-12-11 18:06:53 +00:00
Brian Flad 058434d28b
Merge pull request #19571 from hashicorp/f-backend-s3-other-endpoints
backend/s3: Support DynamoDB, IAM, and STS endpoint configurations
2018-12-10 19:28:01 -05:00
Tom Harvey 383bc98f5c
backend/azurerm: Support for authenticating using the Azure CLI (#19465)
* Upgrading to 2.0.0 of github.com/hashicorp/go-azure-helpers

* Support for authenticating using Azure CLI

* backend/azurerm: support for authenticating using the Azure CLI
2018-12-10 21:23:30 +00:00
Sander van Harmelen a5a2156584 core: enhance service discovery
This PR improves the error handling so we can provide better feedback about any service discovery errors that occured.

Additionally it adds logic to test for specific versions when discovering a service using `service.vN`. This will enable more informational errors which can indicate any version incompatibilities.
2018-12-10 20:52:05 +01:00
Brian Flad 9a3b02cd6c
backend/s3: Support DynamoDB, IAM, and STS endpoint configurations
This change enables a few related use cases:
* AWS has partitions outside Commercial, GovCloud (US), and China, which are the only endpoints automatically handled by the AWS Go SDK. DynamoDB locking and credential verification can not currently be enabled in those regions.
* Allows usage of any DynamoDB-compatible API for state locking
* Allows usage of any IAM/STS-compatible API for credential verification
2018-12-07 03:10:51 -05:00
Sander van Harmelen 9062d887b8 backend/remote: use entitlements to select backends
Use the entitlements to a) determine if the organization exists, and b) as a means to select which backend to use (the local backend with remote state, or the remote backend).
2018-12-05 12:29:08 +01:00
Kristin Laemmert 67aa944166 backend/local: decode variables with cty.DynamicPseudoType
Variables values are marshalled with an explicit type of
cty.DynamicPseudoType, but were being decoded using `Implied Type` to
try and guess the type. This was causing errors because `Implied Type`
does not expect to find a late-bound value.
2018-11-30 15:15:28 -08:00
Martin Atkins 12572e97bc core: Automatically upgrade resource instance states on read
If an instance object in state has an earlier schema version number then
it is likely that the schema we're holding won't be able to decode the
raw data that is stored. Instead, we must ask the provider to upgrade it
for us first, which might also include translating it from flatmap form
if it was last updated with a Terraform version earlier than v0.12.

This ends up being a "seam" between our use of int64 for schema versions
in the providers package and uint64 everywhere else. We intend to
standardize on int64 everywhere eventually, but for now this remains
consistent with existing usage in each layer to keep the type conversion
noise contained here and avoid mass-updates to other Terraform components
at this time.

This also includes a minor change to the test helpers for the
backend/local package, which were inexplicably setting a SchemaVersion of
1 on the basic test state but setting the mock schema version to zero,
creating an invalid situation where the state would need to be downgraded.
2018-11-30 11:22:39 -08:00
Sander van Harmelen fe05609c5e backend/remote: support the new force-unlock API
Add support for the new `force-unlock` API and at the same time improve
performance a bit by reducing the amount of API calls made when using
the remote backend for state storage only.
2018-11-30 19:39:18 +01:00