Commit Graph

565 Commits

Author SHA1 Message Date
Kristin Laemmert 47a16b0937
addrs: embed Provider in AbsProviderConfig instead of Type
a large refactor to addrs.AbsProviderConfig, embedding the addrs.Provider instead of a Type string. I've added and updated tests, added some Legacy functions to support older state formats and shims, and added a normalization step when reading v4 (current) state files (not the added tests under states/statefile/roundtrip which work with both current and legacy-style AbsProviderConfig strings).

The remaining 'fixme' and 'todo' comments are mostly going to be addressed in a subsequent PR and involve looking up a given local provider config's FQN. This is fine for now as we are only working with default assumption.
2020-02-13 15:32:58 -05:00
Li Kexian 76e5b446ba
backend/cos: Add TencentCloud backend cos with lock (#22540)
* add TencentCloud COS backend for remote state

* add vendor of dependence

* fixed error not handle and remove default value for prefix argument

* get appid from TF_COS_APPID environment variables
2020-02-13 11:37:11 -05:00
Alisdair McDiarmid f15e58adf8 Improve remote backend missing token error
Prompt the user to run terraform login to generate and store a token for
the configured remote backend.
2020-02-05 13:25:29 -05:00
Kristin Laemmert 80ab551867
terraform: use addrs.Provider as map keys for provider schemas (#24002)
This is a stepping-stone PR for the provider source project. In this PR
"legcay-stype" FQNs are created from the provider name string. Future
work involves encoding the FQN directly in the AbsProviderConfig and
removing the calls to addrs.NewLegacyProvider().
2020-02-03 08:18:04 -05:00
Martin Atkins 8b511524d6
Initial steps towards AbsProviderConfig/LocalProviderConfig separation (#23978)
* Introduce "Local" terminology for non-absolute provider config addresses

In a future change AbsProviderConfig and LocalProviderConfig are going to
become two entirely distinct types, rather than Abs embedding Local as
written here. This naming change is in preparation for that subsequent
work, which will also include introducing a new "ProviderConfig" type
that is an interface that AbsProviderConfig and LocalProviderConfig both
implement.

This is intended to be largely just a naming change to get started, so
we can deal with all of the messy renaming. However, this did also require
a slight change in modeling where the Resource.DefaultProviderConfig
method has become Resource.DefaultProvider returning a Provider address
directly, because this method doesn't have enough information to construct
a true and accurate LocalProviderConfig -- it would need to refer to the
configuration to know what this module is calling the provider it has
selected.

In order to leave a trail to follow for subsequent work, all of the
changes here are intended to ensure that remaining work will become
obvious via compile-time errors when all of the following changes happen:
- The concept of "legacy" provider addresses is removed from the addrs
  package, including removing addrs.NewLegacyProvider and
  addrs.Provider.LegacyString.
- addrs.AbsProviderConfig stops having addrs.LocalProviderConfig embedded
  in it and has an addrs.Provider and a string alias directly instead.
- The provider-schema-handling parts of Terraform core are updated to
  work with addrs.Provider to identify providers, rather than legacy
  strings.

In particular, there are still several codepaths here making legacy
provider address assumptions (in order to limit the scope of this change)
but I've made sure each one is doing something that relies on at least
one of the above changes not having been made yet.

* addrs: ProviderConfig interface

In a (very) few special situations in the main "terraform" package we need
to make runtime decisions about whether a provider config is absolute
or local.

We currently do that by exploiting the fact that AbsProviderConfig has
LocalProviderConfig nested inside of it and so in the local case we can
just ignore the wrapping AbsProviderConfig and use the embedded value.

In a future change we'll be moving away from that embedding and making
these two types distinct in order to represent that mapping between them
requires consulting a lookup table in the configuration, and so here we
introduce a new interface type ProviderConfig that can represent either
AbsProviderConfig or LocalProviderConfig decided dynamically at runtime.

This also includes the Config.ResolveAbsProviderAddr method that will
eventually be responsible for that local-to-absolute translation, so
that callers with access to the configuration can normalize to an
addrs.AbsProviderConfig given a non-nil addrs.ProviderConfig. That's
currently unused because existing callers are still relying on the
simplistic structural transform, but we'll switch them over in a later
commit.

* rename LocalType to LocalName

Co-authored-by: Kristin Laemmert <mildwonkey@users.noreply.github.com>
2020-01-31 08:23:07 -05:00
Kristin Laemmert 6541775ce4
addrs: roll back change to Type field in ProviderConfig (#23937) 2020-01-28 08:13:30 -05:00
Daniel Dreier 05c9407986
Merge pull request #23293 from kmfk/hide-values-undeclared-var-warning
Avoid leaking sensitive values in undeclared variable warnings
2020-01-15 16:08:19 -08:00
Pam Selle 37d16b2f79 Deletions from backend local 2020-01-13 15:30:46 -05:00
Pam Selle bb3d513c4f Some deletions from static analysis 2020-01-13 15:22:34 -05:00
Pam Selle a93298bd14
Merge pull request #23656 from hashicorp/paddy_gcs_backend_env_var
Add a backend-specific env var for the GCS backend.
2019-12-17 07:30:41 -05:00
Pam Selle 41675543c8
Merge pull request #23627 from alrs/fix-backend-remote-state-dropped-test-errs
Fix backend remote state dropped test errors
2019-12-12 15:21:29 -05:00
Paddy Carver b8752c7610 Add a backend-specific env var for the GCS backend.
Right now, the only environment variable available is the same
environment variable that will be picked up by the GCP provider. Users
would like to be able to store state in separate projects or accounts or
otherwise authenticate to the provider with a service account that
doesn't have access to the state. This seems like a reasonable enough
practice to me, and the solution seems straightforward--offer an
environment variable that doesn't mean anything to the provider to
configure the backend credentials. I've added GOOGLE_BACKEND_CREDENTIALS
to manage just the backend credentials, and documented it appropriately.
2019-12-12 03:35:39 -08:00
Pam Selle d45a4d33bb
Merge pull request #22963 from rekahsoft/s3-backend-pagination
Allow s3 backends to contain more then 1000 workspaces
2019-12-10 13:11:45 -05:00
Pam Selle 45fa6a3145 Fix S3 test 2019-12-10 12:48:08 -05:00
Lars Lehtonen 77ffb4a0eb
backend/remote-state/http: fix dropped test errors 2019-12-10 07:36:54 -08:00
Lars Lehtonen 2845896fc9
backend/remote-state/http: drop unused assertError() 2019-12-10 07:33:22 -08:00
Kristin Laemmert e3416124cc
addrs: replace "Type string" with "Type Provider" in ProviderConfig
* huge change to weave new addrs.Provider into addrs.ProviderConfig
* terraform: do not include an empty string in the returned Providers /
Provisioners
- Fixed a minor bug where results included an extra empty string
2019-12-06 08:00:18 -05:00
Martin Atkins 30bf83cdeb helper/logging: Bring the LevelFilter into our own codebase
In order to make this work reasonably we can't avoid using some funny
heuristics, which are somewhat reasonable to apply within the context of
Terraform itself but would not be good to add to the general "logutils".

Specifically, this is adding the additional heuristic that lines starting
with spaces are continuation lines and so should inherit the log level
of the most recent non-continuation line.
2019-12-05 15:22:03 -08:00
Kristin Laemmert 9891d0354a
providers: use addrs.Provider as map keys for provider.Factory (#23548)
* terraform/context: use new addrs.Provider as map key in provider factories
* added NewLegacyProviderType and LegacyString funcs to make it explicit that these are temporary placeholders

This PR introduces a new concept, provider fully-qualified name (FQN), encapsulated by the `addrs.Provider` struct.
2019-12-04 11:30:20 -05:00
Martin Atkins 6db3cf8e5b vendor: upgrade Azure SDK and Azure/go-autorest
This involves some minor changes to the "azure" backend code to account
for upstream breaking changes.
2019-12-02 10:04:49 -08:00
James Bardin 6caa5d23e2 fix diagnostics handling
Located all non-test paths where a Diagnostic type was assigned to an
error variable.
2019-11-21 09:14:50 -05:00
Robert Tillery af77d1d22c backend/remote: Filter environment variables when loading context (#23358)
* backend/remote: Filter environment variables when loading context

Following up on #23122, the remote system (Terraform Cloud or
Enterprise) serves environment and Terraform variables using a single
type of object. We only should load Terraform variables into the
Terraform context.

Fixes https://github.com/hashicorp/terraform/issues/23283.
2019-11-13 11:34:09 -05:00
Lars Lehtonen 1219acf467 backend/atlas: fix use of T.Fatal() inside goroutine 2019-11-06 17:11:15 -08:00
Joaquín Fernández Campo d2bc7c25b9 backend/artifactory: Honor HTTP_PROXY and HTTPS_PROXY environment variables 2019-11-06 14:18:33 -08:00
Martin Atkins 9a62ab3014 command: "terraform show" renders plans like "terraform plan"
During the Terraform 0.12 work we briefly had a partial update of the old
Terraform 0.11 (and prior) diff renderer that could work with the new
plan structure, but could produce only partial results.

We switched to the new plan implementation prior to release, but the
"terraform show" command was left calling into the old partial
implementation, and thus produced incomplete results when rendering a
saved plan.

Here we instead use the plan rendering logic from the "terraform plan"
command, making the output of both identical.

Unfortunately, due to the current backend architecture that logic lives
inside the local backend package, and it contains some business logic
around state and schema wrangling that would make it inappropriate to move
wholesale into the command/format package. To allow for a low-risk fix to
the "terraform show" output, here we avoid some more severe refactoring by
just exporting the rendering functionality in a way that allows the
"terraform show" command to call into it.

In future we'd like to move all of the code that actually writes to the
output into the "command" package so that the roles of these components
are better segregated, but that is too big a change to block fixing this
issue.
2019-11-06 06:53:32 -08:00
Keith Kirk 9ba211c04e Avoid leaking sensitive values in undeclared warnings 2019-11-05 17:32:57 -08:00
He Guimin bfae627112 add a new field ecs_role_name to support more scenario 2019-11-02 00:09:46 +08:00
Martin Atkins 8f27409007 backend/remote: Support HCL variable values in local operations
For remote operations, the remote system (Terraform Cloud or Enterprise)
writes the stored variable values into a .tfvars file before running the
remote copy of Terraform CLI.

By contrast, for operations that only run locally (like
"terraform import"), we fetch the stored variable values from the remote
API and add them into the set of available variables directly as part
of creating the local execution context.

Previously in the local-only case we were assuming that all stored
variables are strings, which isn't true: the Terraform Cloud/Enterprise UI
allows users to specify that a particular variable is given as an HCL
expression, in which case the correct behavior is to parse and evaluate
the expression to obtain the final value.

This also addresses a related issue whereby previously we were forcing
all sensitive values to be represented as a special string "<sensitive>".
That leads to type checking errors for any variable specified as having
a type other than string, so instead here we use an unknown value as a
placeholder so that type checking can pass.

Unpopulated sensitive values may cause errors downstream though, so we'll
also produce a warning for each of them to let the user know that those
variables are not available for local-only operations. It's a warning
rather than an error so that operations that don't rely on known values
for those variables can potentially complete successfully.

This can potentially produce errors in situations that would've been
silently ignored before: if a remote variable is marked as being HCL
syntax but is not valid HCL then it will now fail parsing at this early
stage, whereas previously it would've just passed through as a string
and failed only if the operation tried to interpret it as a non-string.
However, in situations like these the remote operations like
"terraform plan" would already have been failing with an equivalent
error message anyway, so it's unlikely that any existing workspace that
is being used for routine operations would have such a broken
configuration.
2019-10-31 09:45:50 -07:00
Martin Atkins 9f9f22091e backend/remote: Unlock workspace if Context fails
Previously any error case in the Context method would cause us to leave
the remote workspace locked on exit, requiring manual action to unlock it.
2019-10-31 09:45:50 -07:00
Martin Atkins b10f058cbb backend/remote: Only load variables if we're going to use them
Some commands don't use variables at all or use them in a way that doesn't
require them to all be fully valid and consistent. For those, we don't
want to fetch variable values from the remote system and try to validate
them because that's wasteful and likely to cause unnecessary error
messages.

Furthermore, the variables endpoint in Terraform Cloud and Enterprise only
works for personal access tokens, so it's important that we don't assume
we can _always_ use it. If we do, then we'll see problems when commands
are run inside Terraform Cloud and Enterprise remote execution contexts,
where the variables map always comes back as empty.
2019-10-18 11:31:19 -07:00
Martin Atkins a8d01e3940 backend/remote: Report invalid variables only remotely
The remote backend uses backend.ParseVariableValues locally only to decide
if the user seems to be trying to use -var or -var-file options locally,
since those are not supported for the remote backend.

Other than detecting those, we don't actually have any need to use the
results of backend.ParseVariableValues, and so it's better for us to
ignore any errors it produces itself and prefer to just send a
potentially-invalid request to the remote system and let the remote system
be responsible for validating it.

This then avoids issues caused by the fact that when remote operations are
in use the local system does not have all of the required context: it
can't see which environment variables will be set in the remote execution
context nor which variables the remote system will set using its own
generated -var-file based on the workspace stored variables.
2019-10-18 11:31:19 -07:00
Radek Simko 7860f55e4f
Version tools per Go convention under tools.go 2019-10-17 22:23:39 +02:00
Pam Selle a5f2f4cbba Update messaging for .terraformignore 2019-10-17 11:12:00 -04:00
Radek Simko 32f9722d9d
Replace import paths & set UA string where necessary 2019-10-11 22:40:54 +01:00
Martin Atkins 8664749b59 backend: Allow certain commands to opt out of required variable checks
Terraform Core expects all variables to be set, but for some ancillary
commands it's fine for them to just be set to placeholders because the
variable values themselves are not key to the command's functionality
as long as the terraform.Context is still self-consistent.

For such commands, rather than prompting for interactive input for
required variables we'll just stub them out as unknowns to reflect that
they are placeholders for values that a user would normally need to
provide.

This achieves a similar effect to how these commands behaved before, but
without the tendency to produce a slightly invalid terraform.Context that
would fail in strange ways when asked to run certain operations.
2019-10-10 10:07:01 -07:00
Martin Atkins e21f0fa61e backend/local: Handle interactive prompts for variables in UI layer
During the 0.12 work we intended to move all of the variable value
collection logic into the UI layer (command package and backend packages)
and present them all together as a unified data structure to Terraform
Core. However, we didn't quite succeed because the interactive prompts
for unset required variables were still being handled _after_ calling
into Terraform Core.

Here we complete that earlier work by moving the interactive prompts for
variables out into the UI layer too, thus allowing us to handle final
validation of the variables all together in one place and do so in the UI
layer where we have the most context still available about where all of
these values are coming from.

This allows us to fix a problem where previously disabling input with
-input=false on the command line could cause Terraform Core to receive an
incomplete set of variable values, and fail with a bad error message.

As a consequence of this refactoring, the scope of terraform.Context.Input
is now reduced to only gathering provider configuration arguments. Ideally
that too would move into the UI layer somehow in a future commit, but
that's a problem for another day.
2019-10-10 10:07:01 -07:00
Martin Atkins 39e609d5fd vendor: switch to HCL 2.0 in the HCL repository
Previously we were using the experimental HCL 2 repository, but now we'll
shift over to the v2 import path within the main HCL repository as part of
actually releasing HCL 2.0 as stable.

This is a mechanical search/replace to the new import paths. It also
switches to the v2.0.0 release of HCL, which includes some new code that
Terraform didn't previously have but should not change any behavior that
matters for Terraform's purposes.

For the moment the experimental HCL2 repository is still an indirect
dependency via terraform-config-inspect, so it remains in our go.sum and
vendor directories for the moment. Because terraform-config-inspect uses
a much smaller subset of the HCL2 functionality, this does still manage
to prune the vendor directory a little. A subsequent release of
terraform-config-inspect should allow us to completely remove that old
repository in a future commit.
2019-10-02 15:10:21 -07:00
Collin J. Doering 6d838ed76c
Allow s3 backends to contain more then 1000 workspaces
* backend/remote-state/s3/backend_state.go: Prior to this commit, the terraform s3 backend did
  not paginate calls to s3 when finding workspaces, which resulted in workspaces 'disappearing'
  once they are switched away from, even though the state file still exists. This is due to the
  ListBucket operation defaulting MaxItems to 1000, so terraform s3 backends that contained
  more then 1000 workspaces did not function as expected. This rectifies this situation by
  paginating calls to s3 when finding workspaces.

Signed-off-by: Collin J. Doering <collin@rekahsoft.ca>
2019-10-01 11:46:34 -04:00
Paul Thrasher f2ef8ef317
slow down tfce polling to 1s
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-09-30 14:22:15 -07:00
Paul Thrasher 4e308ef362
typos. some code, some text.
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-09-30 13:44:23 -07:00
Paul Thrasher d2eaffabea
vendor latest go-tfe
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-09-27 15:13:40 -07:00
Paul Thrasher d3fc3dee6e
clean up go mod for go-tfe
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-09-27 14:16:41 -07:00
Paul Thrasher e2831b3d1e
tfce test additions
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-09-27 14:16:41 -07:00
Paul Thrasher bc2d888b7b
update to go-tfe 0.3.23
this fixes the attr types

Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-09-27 14:16:05 -07:00
Paul Thrasher 39817ac476
cost estimation status polling
Properly wait for cost estimation to finish running before outputting
the results. Waits 500 milliseconds between checks, rather than backing
off exponentially, because we are not in a run queue. At the point we're
waiting, we expect cost estimation to be run in a timely manner.
2019-09-27 14:16:05 -07:00
Paul Thrasher 979bba0f32
tfce mock fix
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-09-27 14:11:49 -07:00
Kristin Laemmert 006ef022e6
Merge pull request #22918 from hashicorp/mildwonkey/to-e2e-or-not-e2e
Sundry fixes to e2e tests
2019-09-27 08:46:04 -04:00
Kristin Laemmert 68dfc3046d backend/remote-state: etcdv3, oss, and manta acc tests should fail
faster

The acceptance tests for etcdv3, oss and manta were not validating
required env variablea, chosing to assume that if one was running
acceptance tests they had already configured the credentials.

It was not always clear if this was a bug in the tests or the provider,
so I opted to make the tests fail faster when required attributes were
unset (or "").
2019-09-27 08:45:12 -04:00
Kristin Laemmert 5300f85a79 backend/manta: fix panic when insecure_skip_tls_verify was not set
The DefaultFunc for insecure_skip_tls_verify was sending an empty string
instead of a bool. Fixes to default to `false`.
2019-09-26 10:08:29 -04:00
Paddy 9b24ef7870
Merge pull request #21772 from luis-silva/21680/GCS_OAUTH
Add OAuth2 token support for GCS backend
2019-09-25 16:09:35 -07:00