Commit Graph

584 Commits

Author SHA1 Message Date
Radek Simko 8a6d1d62b6
stringer: Regenerate files with latest version 2019-05-13 15:34:27 +01:00
Stephen Buergler fc5b186e8d Don't leak so many connections in the pg backend
This change fixes an error I get:
Error: pq: too many connections for role "asdf"
because I can only have so many connections.
2019-04-30 23:34:51 -05:00
Sander van Harmelen 394f20f59c backend/remote: do not unlock after a failed upload
When changes are made and we failed to upload the state, we should not
try to unlock the workspace. Leaving the workspace locked is a good
indication something went wrong and also prevents other changes from
being applied before the newest state is properly uploaded.

Additionally we now output the lock ID when a lock or force-unlock
action failed.
2019-04-29 21:23:33 +02:00
Paul Thrasher 151c91ffda
use scanner for reading logs
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-04-25 11:17:08 -07:00
Paul Thrasher 0e27a8862f
remove duplicate remote output line
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-04-25 10:52:19 -07:00
Sander van Harmelen bb12206bca Fixup the tests 2019-04-25 10:32:00 +02:00
Sander van Harmelen 7cf744241a Do not use a scanner to read the logs
Using a scanner can cause issues when reading long lines. Also make sure we return the error correctly while planning.
2019-04-25 09:51:52 +02:00
Paul Thrasher c7a023a95c
update test for new go-tfe version
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-04-24 16:46:17 -07:00
Paul Thrasher 53f977bee2
TFCE api correct pluralization
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-04-24 16:44:00 -07:00
Paul Thrasher e479bd5dc3
update to latest go-tfe
same version number but pointing to a new sha

Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-04-24 16:44:00 -07:00
Sander van Harmelen 90bc237b7b Prevent a panic caused by writing to a nil map 2019-04-11 14:34:14 +02:00
He Guimin 3f44dd9dec Add tablestore config to store state lock 2019-04-11 07:11:10 +08:00
He Guimin b887d44712 Add Alibaba Cloud backend OSS with lock 2019-04-11 07:06:43 +08:00
Sander van Harmelen 39a95e4222 backend/remote: correctly load remote variables
When using `terraform console` in combination with the remote backend, variables defined in Terraform Enterprise were load loaded correctly.
2019-03-28 17:23:48 +01:00
Sander van Harmelen 57f6e01830 backend/local: preserve serial and lineage on failure
When failing to write the state, the local backend writes the state to a local file called `errrored.tfstate`. Previously it would do so by creating a new state file which would use a new serial and lineage. By exorting the existing state file and directly assigning the new state, the serial and lineage are preserved.
2019-03-27 16:15:16 +01:00
Justin Downing 1e32ae243c grammatical updates to comments and docs (#20195) 2019-03-21 14:05:41 -07:00
Stefan Schmidt 3ca1253796 backend/gcs: Mark the unused attributes project and region as removed.
These two attributes are not used since aec45e6967.
2019-03-19 17:29:04 -07:00
Sander van Harmelen 9f6a126293 backend/remote: check for external updates 2019-03-08 19:18:07 +01:00
Sander van Harmelen 4628fbcc65
Merge pull request #20615 from hashicorp/svh/f-input-context
core: add a context to the UIInput interface
2019-03-08 18:44:28 +01:00
Sander van Harmelen e75e845804
Merge pull request #20588 from hashicorp/svh/f-retry
backend/remote: also retry on server errors
2019-03-08 18:17:29 +01:00
Sander van Harmelen 973e2a7cf9 core: add a context to the UIInput interface 2019-03-08 10:24:40 +01:00
Sander van Harmelen 0232d84a0d backend/remote: also retry on server errors
Enably retrying on server errors in the updated `go-tfe` client and add a retry log hook for writing retry messages to the CLI.
2019-03-06 13:36:06 +01:00
Kristin Laemmert b9d8e96e0c
command/plan: plan output should indicate if a resource is being (#20580)
replaced because the instance was tainted.
2019-03-05 16:18:55 -08:00
Martin Atkins 709487b4f1 backend: Cap number of "undeclared variable" warnings at four
For users who in previous versions have relied on our lack of checking for
whether variables are declared, they may previously have seen an
overwhelming number of warnings when running Terraform v0.12.

Here we cap that number at three specific warnings and then one general
warning, so we can still give a specific source location for the first
couple (for users who have genuinely made a typo) but summarize away a
large number for those who are seeing this because they've not yet
migrated to using environment variables.
2019-03-05 15:42:08 -08:00
Mars Hall 34fa67c6b0 Correct pg backend error info 2019-03-05 11:37:08 -08:00
Mars Hall ccf549a62d Stricter pg backend locking during workspace creation 2019-03-05 11:19:56 -08:00
Sergey Kondrashov 43e7a7b552 fix misspelling 2019-03-05 16:12:52 +03:00
Mars Hall 31c9776d55 Switch pg backend to session-level advisory locking, to avoid rollback of partial state updates 2019-03-04 17:01:08 -08:00
Brian Flad e34a97abca
backend/s3: Add debug logging and user agent
Porting over previous behavior from terraform-provider-aws.
2019-02-27 12:59:08 -05:00
Sander van Harmelen 63e2dcef8a
Merge pull request #20481 from hashicorp/svh/b-exit-code
backend/remote: exit with 1 when a run is canceled
2019-02-27 07:52:20 +01:00
James Bardin eada955721
Merge pull request #19070 from mars/postgres-backend
Postgres backend
2019-02-26 20:20:01 -05:00
Mars Hall 6d9e2048f4 Skip more pg backend acceptance tests 2019-02-26 16:50:36 -08:00
Mars Hall 920e7a7acc Mutex pg backend‘s database transaction 2019-02-26 13:45:16 -08:00
Mars Hall c6bf3442fc Lock() in pg backend now returns LockError for retryable errors 2019-02-26 13:20:40 -08:00
Mars Hall a075a68c58 Correct name of pg backend‘s default schema 2019-02-26 12:59:31 -08:00
Sander van Harmelen 01f17fa0ca backend/remote: exit with 1 when a run is canceled 2019-02-26 21:00:07 +01:00
James Bardin dd16455a79 fix s3 default region to match provider 2019-02-25 20:27:21 -05:00
Mars Hall 2621f95bd2 📚 doc corrections 2019-02-25 16:30:30 -08:00
Mars Hall b9a91b7c1e Switch pg backend to use native Postgres locks 2019-02-25 16:05:53 -08:00
James Bardin c814f2da37 Change backend.ValidateConfig to PrepareConfig
This mirrors the change made for providers, so that default values can
be inserted into the config by the backend implementation. This is only
the interface and method name changes, it does not yet add any default
values.
2019-02-25 18:37:20 -05:00
Sander van Harmelen 435b82d6d0
Merge pull request #20460 from hashicorp/svh/f-permissions
backend/remote: use the can-queue-apply permission
2019-02-25 19:32:20 +01:00
James Bardin aebfecf871
Merge pull request #20211 from yanndegat/swift_remote_backend
Swift remote backend support for workspaces & locking
2019-02-25 12:58:59 -05:00
Sander van Harmelen 58961026a2 backend/remote: use the can-queue-apply permission
Previously we checked can-update in order to determine if a user had the
required permissions to apply a run, but that wasn't sufficient. So we
added a new permission, can-queue-apply, that we now use instead.
2019-02-25 14:45:40 +01:00
Mars Hall 8cb2943b6b pg backend tests run when TF_ACC is set (no more TF_PG_TEST) 2019-02-22 14:42:24 -08:00
Mars Hall c8b8227967 Compatibility fixes for 0.12.0 2019-02-22 13:54:41 -08:00
Mars Hall d4d798c25e Fix to use existing pg indexes 2019-02-22 13:51:57 -08:00
Mars Hall 9feda7aab3 Correct code comment for pg 2019-02-22 13:51:57 -08:00
Mars Hall 6d7e8268e7 Include pg backend in main init 2019-02-22 13:51:57 -08:00
Mars Hall 9043005cfb Postgres database backend 2019-02-22 13:48:49 -08:00
James Bardin 31ca293777 fix slash handling around workspace_key_prefix
The handling of slashes was broken around listing workspaces in
workspace_key_prefix. While it worked in most places by splitting an
extra time around the spurious slashes, it failed in the case that the
prefix ended with a slash of its own.

A test was temporarily added to verify that the backend works with the
unusual keys, but rather than risking silent breakage around prefixes
with trailing slashes, we also add validation to prevent users from
entering keys with trailing slashes at all.
2019-02-22 10:29:10 -05:00
James Bardin f6af61f990 fix test that never worked
The 0.12 update left this test broken. Make sure we're working with the
correct objects here.
2019-02-22 10:14:53 -05:00
James Bardin 6cc3e1d0bd move init error to where it is generated
The init error was output deep in the backend by detecting a
special ResourceProviderError and formatted directly to the CLI.

Create some Diagnostics closer to where the problem is detected, and
passed that back through the normal diagnostic flow. While the output
isn't as nice yet, this restores the helpful error message and makes the
code easier to maintain. Better formatting can be handled later.
2019-02-20 14:18:37 -05:00
Brian Flad 185a3304e5
Merge pull request #20374 from hashicorp/td-backend-s3-dependencies
backend/s3: Switch from github.com/terraform-providers/terraform-provider-aws to github.com/hashicorp/aws-sdk-go-base
2019-02-20 08:42:07 -08:00
Brian Flad a41e545198
backend/s3: Reinstate region validation and update copy for skip_requesting_account_id deprecation message 2019-02-20 08:29:09 -08:00
Sander van Harmelen 43eb7c08e0
Merge pull request #20379 from hashicorp/svh/f-state-service
backend/remote: use `state.v2` for remote state only
2019-02-20 10:48:05 +01:00
Brian Flad 43f12bbfe0
backend/s3: Update copy for skip_get_ec2_platforms and skip_requesting_account_id argument deprecations 2019-02-19 10:54:53 -08:00
Sander van Harmelen 54736b068b backend/remote: use `state.v2` for remote state only
The API surface area is much smaller when we use the remote backend for remote state only.

So in order to try and prevent any backwards incompatibilities when TF runs inside of TFE, we’ve split up the discovery services into `state.v2` (which can be used for remote state only configurations, so when running in TFE) and `tfe.v2.1` (which can be used for all remote configurations).
2019-02-19 10:59:51 +01:00
Brian Flad 1aaac172b0
backend/s3: Switch from github.com/terraform-providers/terraform-provider-aws to github.com/hashicorp/aws-sdk-go-base
Output from acceptance testing (no new failures):

```
--- PASS: TestBackend_impl (0.00s)
--- PASS: TestBackendConfig (0.37s)
--- PASS: TestBackendConfig_invalidKey (0.00s)
--- PASS: TestBackend (3.26s)
--- PASS: TestBackendLocked (6.80s)
--- FAIL: TestBackendExtraPaths (2.32s)
--- PASS: TestBackendPrefixInWorkspace (2.06s)
--- PASS: TestKeyEnv (8.20s)
--- PASS: TestRemoteClient_impl (0.00s)
--- PASS: TestRemoteClient (2.42s)
--- PASS: TestRemoteClientLocks (6.33s)
--- PASS: TestForceUnlock (13.31s)
--- PASS: TestRemoteClient_clientMD5 (11.75s)
--- PASS: TestRemoteClient_stateChecksum (10.07s)
```
2019-02-18 02:30:30 -05:00
Yann DEGAT e70b8928e9 remote/backend/swift: Add support for workspaces & locking 2019-02-11 11:13:35 +01:00
Martin Atkins 6eb7bfbdfb
Merge #20265: Don't presume unknown for values unset in config
This changes the contract for `PlanResourceChange` so that the provider is now responsible
for populating all default values during plan, including inserting any unknown values for
defaults it will fill in at apply time.
2019-02-08 13:54:55 -08:00
Martin Atkins e3618f915b backend/local: Fix mock provider in tests
We've changed the contract for PlanResourceChange to now require the
provider to populate any default values (including unknowns) it wants to
set for computed arguments, so our mock provider here now needs to be a
little more complex to deal with that.

This fixes several of the tests in this package. A minor change to
TestLocal_applyEmptyDirDestroy was required to make it properly configure
the mock provider so PlanResourceChange can access the schema.
2019-02-08 12:48:32 -08:00
Radek Simko e6777105b7
Merge pull request #20275 from hashicorp/vendor-openstack-upgrade
vendor: Bump openstack provider to v1.15.0
2019-02-08 16:22:35 +00:00
Sander van Harmelen aefbec63b1 backend/remote: update the test logic 2019-02-08 16:56:37 +01:00
Sander van Harmelen 3b80f69eec
Merge pull request #20242 from hashicorp/svh/b-scanner-buffer-v0.12
backend/remote: fix bufio.Scanner: token too long
2019-02-08 16:52:22 +01:00
Radek Simko a7f0722729
backend/swift: Fix interface after upgrade 2019-02-08 11:03:03 +00:00
Sander van Harmelen 47a00ea34b backend/remote: cleanup test connections
Cleanup test connection to prevent file descriptor issues when running the tests on a Mac.
2019-02-07 09:55:19 +01:00
Sander van Harmelen 5249d0fe83 backend/remote: fix bufio.Scanner: token too long 2019-02-07 09:54:43 +01:00
Radek Simko 6365fffe4f
Fix interface after upgrade 2019-02-01 10:48:56 +00:00
Sander van Harmelen 6dd9f22780
Merge pull request #20086 from hashicorp/svh/f-service-constraints
backend/remote: make sure we show the correct error
2019-01-24 20:22:20 +01:00
Sander van Harmelen 1e4c20686e backend/remote: make sure we show the correct error
Previously we would show two errors when there was a version constraint
error. But of course one is enough.
2019-01-23 15:09:42 +01:00
Martin Atkins 10bf4c763b backend: Undeclared variables in -var-file is a warning, not an error
In Terraform 0.11 and earlier we just silently ignored undeclared
variables in -var-file and the automatically-loaded .tfvars files. This
was a bad user experience for anyone who made a typo in a variable name
and got no feedback about it, so we made this an error for 0.12.

However, several users are now relying on the silent-ignore behavior for
automation scenarios where they pass the same .tfvars file to all
configurations in their organization and expect Terraform to ignore any
settings that are not relevant to a specific configuration. We never
intentionally supported that, but we don't want to immediately break that
workflow during 0.12 upgrade.

As a compromise, then, we'll make this a warning for v0.12.0 that contains
a deprecation notice suggesting to move to using environment variables
for this "cross-configuration variables" use-case. We don't produce errors
for undeclared variables in environment variables, even though that
potentially causes the same UX annoyance as ignoring them in vars files,
because environment variables are assumed to live in the user's session
and this it would be very inconvenient to have to unset such variables
when moving between directories. Their "ambientness" makes them a better
fit for these automatically-assigned general variable values that may or
may not be used by a particular configuration.

This can revert to being an error in a future major release, after users
have had the opportunity to migrate their automation solutions over to
use environment variables.

We don't seem to have any tests covering this specific situation right
now. That isn't ideal, but this change is so straightforward that it would
be relatively expensive to build new targeted test cases for it and so
I instead just hand-tested that it is indeed now producing a warning where
we were previously producing an error. Hopefully if there is any more
substantial work done on this codepath in future that will be our prompt
to add some unit tests for this.
2019-01-22 07:46:51 -08:00
Sander van Harmelen 8937fedb76 backend/remote: fix a small typo 2019-01-18 19:40:57 +01:00
Sander van Harmelen e08a7e979e backend/remote: use the correct test operation 2019-01-15 16:13:16 +01:00
Martin Atkins 0c0a437bcb Move module install functionality over to internal/initwd 2019-01-14 11:33:21 -08:00
Brian Flad ed37d07632
backend/s3: Configure AWS Client MaxRetries and provide enhanced S3 NoSuchBucket error message
The AWS Go SDK automatically provides a default request retryer with exponential backoff that is invoked via setting `MaxRetries` or leaving it `nil` will default to 3. The terraform-aws-provider `config.Client()` sets `MaxRetries` to 0 unless explicitly configured above 0. Previously, we were not overriding this behavior by setting the configuration and therefore not invoking the default request retryer.

The default retryer already handles HTTP error codes above 500, including S3's InternalError response, so the extraneous handling can be removed. This will also start automatically retrying many additional cases, such as temporary networking issues or other retryable AWS service responses.

Changes:
* s3/backend: Add `max_retries` argument
* s3/backend: Enhance S3 NoSuchBucket error to include additional information
2019-01-09 13:01:37 -05:00
Sander van Harmelen bba03384d5 backend/remote: log early to indicate execution started 2019-01-08 17:06:24 +01:00
Sander van Harmelen 7b51af72b2 backend/remote: compare versions without the prerelease 2018-12-19 19:06:22 +01:00
Sander van Harmelen 55b6153b04 backend/remote: fix an error that prevents checking constraints 2018-12-15 21:36:47 +01:00
Sander van Harmelen 2ef8315885
Merge pull request #19659 from hashicorp/svh/f-check-constraints
backend/remote: return detailed incompatibility info
2018-12-14 21:26:02 +01:00
Sander van Harmelen 8f04e93739 backend/remote: return detailed incompatibility info 2018-12-14 21:11:41 +01:00
Radek Simko f64978b64c
backend/local: Render CBD replacement (+/-) correctly (#19642)
* backend/local: Render CBD replacement (+/-) correctly

* command/format: Use IsReplace helper function
2018-12-14 13:45:47 +00:00
Sander van Harmelen da8e02eb2e
Merge pull request #19647 from hashicorp/svh/f-versions
core:  add a method to the disco package retrieve version constraints
2018-12-14 12:38:52 +01:00
Sander van Harmelen 268c0f85ce Add a method to retrieve version contraints 2018-12-14 12:17:31 +01:00
Radek Simko 3ab4739ba4
backend/local: Avoid rendering data sources on destroy 2018-12-12 18:21:49 +00:00
Radek Simko e55079abc9
backend/local: Fix incorrect destroy/update count on apply 2018-12-11 18:06:53 +00:00
Brian Flad 058434d28b
Merge pull request #19571 from hashicorp/f-backend-s3-other-endpoints
backend/s3: Support DynamoDB, IAM, and STS endpoint configurations
2018-12-10 19:28:01 -05:00
Tom Harvey 383bc98f5c
backend/azurerm: Support for authenticating using the Azure CLI (#19465)
* Upgrading to 2.0.0 of github.com/hashicorp/go-azure-helpers

* Support for authenticating using Azure CLI

* backend/azurerm: support for authenticating using the Azure CLI
2018-12-10 21:23:30 +00:00
Sander van Harmelen a5a2156584 core: enhance service discovery
This PR improves the error handling so we can provide better feedback about any service discovery errors that occured.

Additionally it adds logic to test for specific versions when discovering a service using `service.vN`. This will enable more informational errors which can indicate any version incompatibilities.
2018-12-10 20:52:05 +01:00
Brian Flad 9a3b02cd6c
backend/s3: Support DynamoDB, IAM, and STS endpoint configurations
This change enables a few related use cases:
* AWS has partitions outside Commercial, GovCloud (US), and China, which are the only endpoints automatically handled by the AWS Go SDK. DynamoDB locking and credential verification can not currently be enabled in those regions.
* Allows usage of any DynamoDB-compatible API for state locking
* Allows usage of any IAM/STS-compatible API for credential verification
2018-12-07 03:10:51 -05:00
Sander van Harmelen 9062d887b8 backend/remote: use entitlements to select backends
Use the entitlements to a) determine if the organization exists, and b) as a means to select which backend to use (the local backend with remote state, or the remote backend).
2018-12-05 12:29:08 +01:00
Kristin Laemmert 67aa944166 backend/local: decode variables with cty.DynamicPseudoType
Variables values are marshalled with an explicit type of
cty.DynamicPseudoType, but were being decoded using `Implied Type` to
try and guess the type. This was causing errors because `Implied Type`
does not expect to find a late-bound value.
2018-11-30 15:15:28 -08:00
Martin Atkins 12572e97bc core: Automatically upgrade resource instance states on read
If an instance object in state has an earlier schema version number then
it is likely that the schema we're holding won't be able to decode the
raw data that is stored. Instead, we must ask the provider to upgrade it
for us first, which might also include translating it from flatmap form
if it was last updated with a Terraform version earlier than v0.12.

This ends up being a "seam" between our use of int64 for schema versions
in the providers package and uint64 everywhere else. We intend to
standardize on int64 everywhere eventually, but for now this remains
consistent with existing usage in each layer to keep the type conversion
noise contained here and avoid mass-updates to other Terraform components
at this time.

This also includes a minor change to the test helpers for the
backend/local package, which were inexplicably setting a SchemaVersion of
1 on the basic test state but setting the mock schema version to zero,
creating an invalid situation where the state would need to be downgraded.
2018-11-30 11:22:39 -08:00
Sander van Harmelen fe05609c5e backend/remote: support the new force-unlock API
Add support for the new `force-unlock` API and at the same time improve
performance a bit by reducing the amount of API calls made when using
the remote backend for state storage only.
2018-11-30 19:39:18 +01:00
Sander van Harmelen 4c878db8fd
Merge pull request #19464 from hashicorp/svh/f-context
backend/remote: implement the Local interface
2018-11-28 20:16:58 +01:00
Sander van Harmelen a061725132
Merge pull request #19403 from hashicorp/svh/f-policies
backend/remote: also show policies when there are no changes
2018-11-28 16:30:31 +01:00
Martin Atkins 168d84b3c4 core: Make resource type schema versions visible to callers
Previously we were fetching these from the provider but then immediately
discarding the version numbers because the schema API had nowhere to put
them.

To avoid a late-breaking change to the internal structure of
terraform.ProviderSchema (which is constructed directly all over the
tests) we're retaining the resource type schemas in a new map alongside
the existing one with the same keys, rather than just switching to
using the providers.Schema struct directly there.

The methods that return resource type schemas now return two arguments,
intentionally creating a little API friction here so each new caller can
be reminded to think about whether they need to do something with the
schema version, though it can be ignored by many callers.

Since this was a breaking change to the Schemas API anyway, this also
fixes another API wart where there was a separate method for fetching
managed vs. data resource types and thus every caller ended up having a
switch statement on "mode". Now we just accept mode as an argument and
do the switch statement within the single SchemaForResourceType method.
2018-11-27 15:53:54 -08:00
Sander van Harmelen 35d9ce3f92 backend/remote: implement the Local interface 2018-11-26 20:50:25 +01:00
Tom Harvey 6d4f702467
backend/azurerm: support for custom resource manager endpoints (#19460)
* backend/azurerm: removing the `arm_` prefix from keys

* removing the deprecated fields test because the deprecation makes it fail

* authentication: support for custom resource manager endpoints

* Adding debug prefixes to the log statements
2018-11-26 14:42:16 +01:00
Tom Harvey d580f30e03
backend/azurerm: removing the `arm_` prefix from keys (#19448)
* backend/azurerm: removing the `arm_` prefix from keys

* removing the deprecated fields test because the deprecation makes it fail
2018-11-26 11:19:43 +01:00
Tom Harvey 96b1c951fa
backend/azurerm: support for authenticating via SAS Tokens (#19440)
* adding acceptance tests for msi auth

* including the resource group name in the tests

* backend/azurerm: support for authenticating using a SAS Token

* resolving merge conflicts

* moving the defer to prior to the error
2018-11-22 18:02:33 +01:00
Tom Harvey e437feadc2
backend/azurerm: fixing a bug where locks couldn't be unlocked (#19441)
Fixes #17046
2018-11-22 17:47:35 +01:00
Tom Harvey c928962f44
backend/azurerm: support for authenticating via msi (#19433)
* backend/azurerm: support for authenticating via msi

* adding acceptance tests for msi auth

* including the resource group name in the tests

* support for using the test client via msi
2018-11-22 16:52:27 +01:00
Tom Harvey 0ec109bdc0
backend/azurerm: upgrading the SDK / support for proxies (#19414)
* vendor updates

- updating to v21.3.0 of github.com/Azure/azure-sdk-for-go
- updating to v10.15.4 of github.com/Azure/go-autorest
- vendoring github.com/hashicorp/go-azure-helpers @ 0.1.1

* backend/azurerm: refactoring to use the new auth package

- refactoring the backend to use a shared client via the new auth package
- adding tests covering both Service Principal and Access Key auth
- support for authenticating using a proxy
- rewriting the backend documentation to include examples of both authentication types

* switching to use the build-in logging function

* documenting it's also possible to retrieve the access key from an env var
2018-11-21 22:06:03 +01:00
Sander van Harmelen 4561c80c1d Also show policies when there are no changes
This behavior was recently updated in the TFE UI, so lets follow that behavior here as well.
2018-11-21 11:34:59 +01:00
Sander van Harmelen a17f317025 Change how to fall back from remote to local backend
In order to support free organizations, we need a way to load the `remote` backend and then, depending on the used offering/plan, enable or disable remote operations.

In other words, we should be able to dynamically fall back to the `local` backend if needed, after first configuring the `remote` backend.

To make this works we need to change the way this was done previously when the env var `TF_FORCE_LOCAL_BACKEND` was set. The clear difference of course being that the env var would be available on startup, while the used offering/plan is only known after being able to connect to TFE.
2018-11-20 22:25:52 +01:00
Martin Atkins 27abd9c6b8 command: Fix TestMetaBackend_localDoesNotDeleteLocal
The changes to how we handle setting the state path on the local backend
broke the heuristic we were using here for detecting migration from one
local backend to another with the same state path, which would by default
end up deleting the state altogether after migration.

We now use the StatePaths method to do this, which takes into account
both the default values and any settings that have been set.

Additionally this addresses a flaw in the old method which could
potentially have deleted all non-default workspace state files if the
"path" setting were changed without also changing the "workspace_dir"
setting. This new approach is conservative because it will preserve all
of the files if any one overlaps.
2018-11-19 09:02:35 -08:00
Martin Atkins ec27526cc3 command: Fix TestMetaBackend_configuredChangeCopy_multiToMulti
This was failing because we now handle the settings for the local backend
a little differently as a result of decoding it with the HCL2 machinery.

Specifically, the backend.State* fields are now assumed to be what is
given in configuration, and any CLI overrides are maintained separately
in OverrideState* fields so that they can be imposed "just in time" in
StatePaths.

This is particularly important because OverrideStatePath (when set) is
used regardless of workspace name, while StatePath is a suitable value
only for the "default" workspace, with others needing to be constructed
from StateWorkspaceDir instead.
2018-11-19 09:02:35 -08:00
Sander van Harmelen 04439595ab Make the Atlas backend work after updating depencies
Newer versions of the retryablehttp package use a context, so we need to
add that in our custom `CheckRetry` function.

In addition I removed the `return true, nil` to continue retrying in
case of an error, and instead directly call the `DefaultRetryPolicy`.
This is because the `DefaultRetryPolicy` will now also take the context
into consideration.
2018-11-15 20:34:31 +01:00
Sander van Harmelen 52a1b22f7a Implement the remote enhanced backend
This is a refactored version of the `remote` backend that was initially added to Terraform v0.11.8 which should now be compatible with v0.12.0.
2018-11-06 16:29:46 +01:00
Sander van Harmelen b62a22ab62 Add a VariableSourceType for names .tfvars files
This new source type should be used for variables loaded from .tfvars files that were explicitly passed as command line arguments (e.g. -var-file=foo.tfvars)
2018-11-05 19:29:34 +01:00
Sander van Harmelen b846c5f653 Make the test compile again
Not sure if these checks still make sense, but without this change the test don’t compile for the related packages.
2018-10-19 15:29:14 +02:00
Sander van Harmelen 48ef7ecfa6 Updates after running `make fmt` with Go v1.11.1 2018-10-17 14:11:08 -07:00
James Bardin 0a0eece15c add backend cli options after configuration
The cli should override the config
2018-10-17 09:41:58 -04:00
Martin Atkins 541952bb8f Revert some work that happened since v0.12-dev branched
This work was done against APIs that were already changed in the branch
before work began, and so it doesn't apply to the v0.12 development work.

To allow v0.12 to merge down to master, we'll revert this work out for now
and then re-introduce equivalent functionality in later commits that works
against the new APIs.
2018-10-16 19:48:28 -07:00
Martin Atkins 98bbd560b5 command: Fix most (but not all) "terraform plan" tests 2018-10-16 19:14:11 -07:00
Martin Atkins 741d334ee4 command: Even more fixes for "apply" command tests 2018-10-16 19:14:11 -07:00
Martin Atkins c5940f2438 backend/local: Increase log verbosity for backend context construction
There are several steps here and a number of them can include reaching out
to remote servers or executing local processes, so it's helpful to have
some trace logs to better narrow down causes of errors and hangs during
this step.
2018-10-16 19:14:11 -07:00
Martin Atkins a6f399517b backend/local: reinstate additional steps in plan file processing
In earlier refactoring we skipped implementing prior state safety checks,
propagating the target addresses from plan, and verifying that all of
the providers are exactly the same from the plan being created.

This change reinstates those checks, including a new error message for
the "stale plan" situation.
2018-10-16 19:14:11 -07:00
Martin Atkins 91d2de6a25 backend/local: Stub out remaining planfile todos with errors
This is just to make sure they show up later when we are working on the
tests, so we can be sure not to forget to address them.
2018-10-16 19:14:11 -07:00
Martin Atkins fa3b4fc9de backend/local: Populate changes into a context built from planfile 2018-10-16 19:14:11 -07:00
Martin Atkins cbc548eb36 command: Do CLI init of backend loaded from plan
If we don't do this, we can't produce any output when applying a saved
plan file.

Here we also introduce a check to the local backend's ReportResult
function so that it won't panic if CLI init is skipped, although that
will no longer happen in the apply-from-file case due to the change
described in the previous paragraph.
2018-10-16 19:14:11 -07:00
Martin Atkins 2b80df0163 backend/local: Require caller to set PlanOutBackend with PlanOutPath
We can't generate a valid plan file without a backend configuration to
write into it, but it's the responsibility of the caller (the command
package) to manage the backend configuration mechanism, so we require it
to tell us what to write here.

This feels a little strange because the backend in principle knows its
own config, but in practice the backend only knows the _processed_ version
of the config, not the raw configuration value that was used to configure
it.
2018-10-16 19:14:11 -07:00
Kristin Laemmert 2d3cb87789 backend/local tests tests tests
converted the existing testPlanState() from terraform.State to
states.State to fix various plan tests.

reverted the "bandaid" in plans/planfile/tfplan.go - at this moment the
backend tests do not include backend configuration, and so the planfile
package can write the plan file but not read it back in. That will be
revisted in a separate track of work.
2018-10-16 19:14:11 -07:00
Kristin Laemmert 6a37ee9277 backend/local: more tests passing
I have no confidence in the change to plans/planfile/tfplan.go. The
tests were passing an empty backend config, which planfile  was able to
write to a file but not read from the same file. This change let me move
past that and it did not break any tests in the planfile package, but I
am concerned that it introduces undesired behavior.
2018-10-16 19:14:11 -07:00
Kristin Laemmert 56b879d0c0 backend/local: updated DiffFn adn ReadResourceFn for new models 2018-10-16 19:14:11 -07:00
Kristin Laemmert 64f696d9b3 backend/local refresh tests refactor 2018-10-16 19:14:11 -07:00
Kristin Laemmert e84def1e29 backend/local: test fixes
for TestLocal_applyBackendFail, config is relative to the tests' wd
2018-10-16 19:14:11 -07:00
Kristin Laemmert d865df7338 revert to explicitly declare schema for TestLocal_applyEmptyDirDestroy 2018-10-16 19:14:11 -07:00
Kristin Laemmert 739bd5ef0d backend/local tests 2018-10-16 19:14:11 -07:00
Martin Atkins 1c007473ba backend: TestBackendConfig to allow nil config body
This is not normally considered valid, but since this is a test helper
we will allow it and just treat it as an empty body.
2018-10-16 19:14:11 -07:00
Martin Atkins 8565c7460e backend/remote-state/inmem: remove FIXME guards on tests that no longer hang 2018-10-16 19:14:11 -07:00
Martin Atkins 20727945ee backend/remote-state/inmem: enable optional logging in tests 2018-10-16 19:14:11 -07:00
Martin Atkins aaf405b662 backend/remote-state: Get all the backend tests building again
The state manager refactoring in an earlier commit was reflected in the
implementations of these backends, but not in their tests. This gets us
back to a state where the backend tests will compile, and gets _most_ of
them passing again, with a few exceptions that will be addressed in a
subsequent commit.
2018-10-16 19:14:11 -07:00
Martin Atkins b0016e9cf6 command: Allow tests to run to completion without panics or hangs
There are still 160 test failures as of this commit, but at least the test
program can run to completion and list out all the failures.
2018-10-16 19:14:11 -07:00
Kristin Laemmert 3da04ef9fc backend/local: adding some informative comments to commented-out tests 2018-10-16 19:14:11 -07:00
Kristin Laemmert c661157999 plans/objchange: further harden ProposedNewObject against ~weird~
incoming values

Addresses an odd state where the priorV of an object to be changed is
known but null.

While this situation should not happen, it seemed prudent to ensure that
core is resilient to providers sending incorrect values (which might
also occur with manually edited state).
2018-10-16 19:14:11 -07:00
Kristin Laemmert 2808df48ec backend/local WIP commit - fixing tests 2018-10-16 19:14:11 -07:00
Kristin Laemmert fbe959ae6e backend/local: fix panic in tests
update the function names in testDelegateBackend to match what was being
called in TestLocal_multiStateBackend (matching the test behavior in
master)
2018-10-16 19:14:11 -07:00
Martin Atkins a43b7df282 core: Handle forced-create_before_destroy during the plan walk
Previously we used a single plan action "Replace" to represent both the
destroy-before-create and the create-before-destroy variants of replacing.
However, this forces the apply graph builder to jump through a lot of
hoops to figure out which nodes need it forced on and rebuild parts of
the graph to represent that.

If we instead decide between these two cases at plan time, the actual
determination of it is more straightforward because each resource is
represented by only one node in the plan graph, and then we can ensure
we put the right nodes in the graph during DiffTransformer and thus avoid
the logic for dealing with deposed instances being spread across various
different transformers and node types.

As a nice side-effect, this also allows us to show the difference between
destroy-then-create and create-then-destroy in the rendered diff in the
CLI, although this change doesn't fully implement that yet.
2018-10-16 19:14:11 -07:00
Martin Atkins 5390fb1eed backend/local: Don't count outputs for choosing diff action symbols
We're not yet showing outputs in the rendered diff, so it doesn't make
sense to count them for the purpose of deciding which change action
symbols to include in the legend.
2018-10-16 19:14:11 -07:00
Martin Atkins 20318ca193 backend/local: Sort planned resource changes before rendering them
As in the old plan renderer, they are ordered by the natural ordering of
the resource addresses.
2018-10-16 19:14:11 -07:00
Martin Atkins 239a54ad6f command: initial structural diff rendering
This is a light adaptation of our earlier prototype of structural diff
rendering, as a starting point for what we'll actually ship. This is not
consistent with the latest mocks, so will need some additional work before
it is ready, but integrating this allows us to at least see the plan
contents while fixing up remaining issues elsewhere.
2018-10-16 19:14:11 -07:00
Martin Atkins a6b5980a4f backend/local: Bail immediately if terraform.Context can't be created 2018-10-16 19:14:11 -07:00
Martin Atkins 44bc7519a6 terraform: More wiring in of new provider types
This doesn't actually work yet, but it builds and then panics in a pretty
satisfying way.
2018-10-16 19:12:54 -07:00
Martin Atkins a3403f2766 terraform: Ugly huge change to weave in new State and Plan types
Due to how often the state and plan types are referenced throughout
Terraform, there isn't a great way to switch them out gradually. As a
consequence, this huge commit gets us from the old world to a _compilable_
new world, but still has a large number of known test failures due to
key functionality being stubbed out.

The stubs here are for anything that interacts with providers, since we
now need to do the follow-up work to similarly replace the old
terraform.ResourceProvider interface with its replacement in the new
"providers" package. That work, along with work to fix the remaining
failing tests, will follow in subsequent commits.

The aim here was to replace all references to terraform.State and its
downstream types with states.State, terraform.Plan with plans.Plan,
state.State with statemgr.State, and switch to the new implementations of
the state and plan file formats. However, due to the number of times those
types are used, this also ended up affecting numerous other parts of core
such as terraform.Hook, the backend.Backend interface, and most of the CLI
commands.

Just as with 5861dbf3fc49b19587a31816eb06f511ab861bb4 before, I apologize
in advance to the person who inevitably just found this huge commit while
spelunking through the commit history.
2018-10-16 19:11:09 -07:00
Martin Atkins 479c6b2466 move "configschema" from "config" to "configs"
The "config" package is no longer used and will be removed as part
of the 0.12 release cleanup. Since configschema is part of the
"new world" of configuration modelling, it makes more sense for
it to live as a subdirectory of the newer "configs" package.
2018-10-16 18:50:29 -07:00
Martin Atkins ebc6238bee backend/local: Update tests for changes in "terraform" package
This mainly consists of adding schema information to the mock providers.
2018-10-16 18:49:20 -07:00
Martin Atkins c937c06a03 terraform: ugly huge change to weave in new HCL2-oriented types
Due to how deeply the configuration types go into Terraform Core, there
isn't a great way to switch out to HCL2 gradually. As a consequence, this
huge commit gets us from the old state to a _compilable_ new state, but
does not yet attempt to fix any tests and has a number of known missing
parts and bugs. We will continue to iterate on this in forthcoming
commits, heading back towards passing tests and making Terraform
fully-functional again.

The three main goals here are:
- Use the configuration models from the "configs" package instead of the
  older models in the "config" package, which is now deprecated and
  preserved only to help us write our migration tool.
- Do expression inspection and evaluation using the functionality of the
  new "lang" package, instead of the Interpolator type and related
  functionality in the main "terraform" package.
- Represent addresses of various objects using types in the addrs package,
  rather than hand-constructed strings. This is not critical to support
  the above, but was a big help during the implementation of these other
  points since it made it much more explicit what kind of address is
  expected in each context.

Since our new packages are built to accommodate some future planned
features that are not yet implemented (e.g. the "for_each" argument on
resources, "count"/"for_each" on modules), and since there's still a fair
amount of functionality still using old-style APIs, there is a moderate
amount of shimming here to connect new assumptions with old, hopefully in
a way that makes it easier to find and eliminate these shims later.

I apologize in advance to the person who inevitably just found this huge
commit while spelunking through the commit history.
2018-10-16 18:46:46 -07:00
Martin Atkins ebafa51723 command: Various updates for the new backend package API
This is a rather-messy, complex change to get the "command" package
building again against the new backend API that was updated for
the new configuration loader.

A lot of this is mechanical rewriting to the new API, but
meta_config.go and meta_backend.go in particular saw some major
changes to interface with the new loader APIs and to deal with
the change in order of steps in the backend API.
2018-10-16 18:44:26 -07:00
Martin Atkins 5782357c28 backend: Update interface and implementations for new config loader
The new config loader requires some steps to happen in a different
order, particularly in regard to knowing the schema in order to
decode the configuration.

Here we lean directly on the configschema package, rather than
on helper/schema.Backend as before, because it's generally
sufficient for our needs here and this prepares us for the
helper/schema package later moving out into its own repository
to seed a "plugin SDK".
2018-10-16 18:39:12 -07:00