Commit Graph

100 Commits

Author SHA1 Message Date
Alisdair McDiarmid 53e587e1a6
Merge pull request #25504 from hashicorp/alisdair/post-install-cache-validate
Add post-install provider cache validation and error reporting
2020-07-09 14:55:17 -04:00
Martin Atkins 80ab867e57 command/init: Remove special 0.12upgrade heuristic
For Terraform v0.12 we introduced a special loading mode where we would
use the 0.11-syntax-compatible "earlyconfig" package as a heuristic to
identify situations where it was likely that the user was trying to use
0.11-only syntax that the upgrade tool might help with.

However, as the language has moved on that is no longer a suitable
heuristic in Terraform 0.13 and later: other new additions to the
language can cause the main loader to disagree with earlyconfig, which
would lead us to give poor advice about how to respond.

Instead, we'll now return the same generic "there are errors" message in
all syntax error cases. We have an extra message for errors in this
case (as compared to other commands) because "terraform init" is usually
the first command a new user interacts with and so this message gives some
extra explanation about what "terraform init" will do with the
configuration once it's valid.

This also includes a reset control character in the output of the message
as part of our ongoing mission to stop Terraform printing out whole
paragraphs of colored text, which can often be hard to read for various
reasons.
2020-07-08 10:18:55 -07:00
Alisdair McDiarmid 87d1fb4006 command/init: Display provider validation errors
After installing providers, we validate the presence of an executable
file, and generate a selected versions lockfile. If this process fails,
notify the user. One possible cause for this is an invalid provider
package with a missing or misnamed executable file.
2020-07-07 15:20:20 -04:00
Kristin Laemmert df244b87c2
command/init: return an error with invalid -backend-config files (#25411)
* command/init: return an error with invalid -backend-config files

The -backend-config flag expects a set of key-value pairs or a file
containing key-value pairs. If the file instead contains a full backend
configuration block, it was silently ignored. This commit adds a check
for blocks in the file and returns an error if they are encountered.

Fixes #24845

* emphasize backend configuration file in docs
2020-06-26 12:49:31 -04:00
Alisdair McDiarmid b6739829e7 command: Fix 0.13upgrade to preserve more comments
Previously, any comments inside the required provider configuration for
a given provider would be wiped out upon rerunning the 0.13upgrade
command. This commit attempts to preserve those comments if the existing
entry is semantically equivalent to the entry we are about to write.
2020-06-24 15:54:46 -04:00
Kristin Laemmert b611bd7209 reproduction test 2020-06-12 15:39:55 -04:00
Alisdair McDiarmid 08b735984a
Merge pull request #25191 from hashicorp/alisdair/better-provider-upgrade-hints-on-init
command/init: Improve diags for legacy providers
2020-06-12 12:31:33 -04:00
Martin Atkins 17feb2abfc vendor: go get github.com/apparentlymart/go-versions@v1.0.0
This new version permits omitting the space between the operator and the
boundary in a ruby-style version constraint, like ">1.0.0" instead of
"> 1.0.0".
2020-06-12 08:45:14 -07:00
Alisdair McDiarmid 9263b28e99 command/init: Improve diags for legacy providers
When initializing a configuration which refers to re-namespaced legacy
providers, we attempt to detect this and display a diagnostic message.
Previously this message would direct the user to run the 0.13upgrade
command, but without specifying in which directories.

This commit detects which modules are using the providers in question,
and for local modules displays a list of upgrade commands which specify
the source directories of these modules.

For remote modules, we display a separate list noting that they need to
be upgraded elsewhere, providing both the local module call name and the
module source address.
2020-06-12 09:57:01 -04:00
Kristin Laemmert 5450e8515d
command/013upgrade: detect builtin terraform provider (#25215)
* command/013upgrade: detect builtin terraform provider
2020-06-11 14:10:47 -04:00
Alisdair McDiarmid 1c1e4a4de0 command/providers: Show provider requirements tree
Providers can be required from multiple sources. The previous
implementation of the providers sub-command displayed only a flat list
of provider requirements, which made it difficult to see which modules
required each provider.

This commit reintroduces the tree display of provider requirements, and
adds a separate output block for providers required by existing state.
2020-06-09 14:21:53 -04:00
Alisdair McDiarmid ca40107066 command/init: Better diagnostics for provider 404s
Fetching a default namespace provider from the public registry can
result in 404 Not Found error. This might be caused by a previously-
default provider moving to a new namespace, which means that the
configuration needs to be upgraded to use an explicit provider source.

This commit adds a more detailed diagnostic for this situation,
suggesting that the intended provider might be in a new namespace. The
recommended course of action is to run the 0.13upgrade command to
generate the correct required_providers configuration.
2020-05-28 09:24:32 -04:00
Alisdair McDiarmid 62d826e066 command/init: Use full config for provider reqs
Relying on the early config for provider requirements was necessary in
Terraform 0.12, to allow the 0.12upgrade command to run after init
installs providers.

However in 0.13, the same restrictions do not apply, and the detection
of provider requirements has changed. As a result, the early config
loader gives incorrect provider requirements in some circumstances,
such as those in the new test in this commit.

Therefore we are changing the init command to use the requirements found
by the full configuration loader. This also means that we can remove the
internal initwd CheckCoreVersionRequirements function.
2020-05-25 16:50:12 -04:00
Alisdair McDiarmid 1fdcbc4825 command: Fix 0.13upgrade bug with multiple blocks
If a configuration had multiple blocks in the versions.tf file, it would
be added to the `rewritePaths` list multiple times. We would then remove
it from this slice, but only once, and so the output file would later be
rewritten to remove the required providers block.

This commit uses a set instead of a list to prevent this case, and adds
a regression test.
2020-05-07 20:11:44 -04:00
Alisdair McDiarmid a740b739e0 command: Change 0.13upgrade default to versions.tf
Instead of using providers.tf as the default output file for the
upgrader, we now default to versions.tf. This means that if the
configuration has no `required_providers` blocks at all, or has
multiple, the provider version requirements will be stored in the
versions.tf file.

We now also update the versions.tf file to set a `required_version`
attribute in the first `terraform` block, with value ">= 0.13". This
is similar to the behaviour of the 0.12upgrade command, and signals that
the configuration should not be used with older versions of Terraform.
2020-05-07 15:45:48 -04:00
Alisdair McDiarmid 01a3376ead command: Check required_version before upgrading
If a configuration has a version constraint which prevents use with
Terraform 0.13, the upgrade command should exit before making any
changes.
2020-05-07 15:45:48 -04:00
Alisdair McDiarmid ae98bd12a7 command: Rework 0.13upgrade sub-command
This commit implements most of the intended functionality of the upgrade
command for rewriting configurations.

For a given module, it makes a list of all providers in use. Then it
attempts to detect the source address for providers without an explicit
source.

Once this step is complete, the tool rewrites the relevant configuration
files. This results in a single "required_providers" block for the
module, with a source for each provider.

Any providers for which the source cannot be detected (for example,
unofficial providers) will need a source to be defined by the user. The
tool writes an explanatory comment to the configuration to help with
this.
2020-05-07 11:38:55 -04:00
Mike Morris 9568de6b90
command: Fix OAuth2 PKCE arguments
Providers like Okta and AWS Cognito expect that the PKCE challenge
uses base64 URL Encoding without any padding (base64.RawURLEncoding)

Additionally, Okta strictly adheres to section 4.2 of RFC 7636 and
requires that the unencoded key for the PKCE data is at least 43
characters in length.
2020-05-05 09:58:48 -07:00
Pam Selle 87bce5f9dd
Support reading module outputs in terraform console (#24808)
* Include eval in output walk

This allows outputs to be evaluated in the evalwalk,
impacting terraform console. Outputs are still not evaluated
for terraform console in the root module, so this has
no impact on writing to state (as child module outputs are not
written to state). Also adds test coverage to the console command,
including for evaluating locals (another use of the evalwalk)
2020-04-30 09:21:42 -04:00
Kristin Laemmert f09ae6f862
provider source tests: added test suite to exercise hyphenated providers (#24685) 2020-04-16 15:54:33 -04:00
Kristin Laemmert 269d511481 command/providers: refactor with new provider types and functions
The providers command has been refactored to use the modern provider types and
ProviderRequirements() functions. This resulted in a breaking change to
the output: it no longer outputs the providers by module and no longer
prints `(inherited)` or `(from state)` to show why a provider is
included. We decided that at this time it was best to stick with the
existing functions and make this change, but if we get feedback from the
community we will revisit.

Additional tests to exercise providers in modules and providers from
state have been included.
2020-04-10 15:08:10 -04:00
Kristin Laemmert 0a5fb40fdf configs: include provider configs in ProviderRequirements()
This PR adds iteration through any provider configuration blocks in the
config in addProviderRequirements().

A stale comment (of mine!) would leave one expecting the
module.ProviderRequirements to include any requirements from provider
configs. The comment was inaccurate and has been updated.
2020-04-10 15:08:10 -04:00
Kristin Laemmert 5852a5c33d Mildwonkey/last tests (#24553)
* show text fixture update
* temporarily disable providers tests
2020-04-06 09:24:23 -07:00
Martin Atkins 958ea4f7d1 internal/providercache: Handle built-in providers
Built-in providers are special providers that are distributed as part of
Terraform CLI itself, rather than being installed separately. They always
live in the terraform.io/builtin/... namespace so it's easier to see that
they are special, and currently there is only one built-in provider named
"terraform".

Previous commits established the addressing scheme for built-in providers.
This commit makes the installer aware of them to the extent that it knows
not to try to install them the usual way and it's able to report an error
if the user requests a built-in provider that doesn't exist or tries to
impose a particular version constraint for a built-in provider.

For the moment the tests for this are the ones in the "command" package
because that's where the existing testing infrastructure for this
functionality lives. A later commit should add some more focused unit
tests here in the internal/providercache package, too.
2020-04-06 09:24:23 -07:00
Kristin Laemmert 27a794062e Mildwonkey/command tests (#24535)
* command: refactor testBackendState to write states.State

testBackendState was using the older terraform.State format, which is no
longer sufficient for most tests since the state upgrader does not
encode provider FQNs automatically. Users will run `terraform
0.13upgrade` to update their state to include provider FQNs in
resources, but tests need to use the modern state format instead of
relying on the automatic upgrade.

* plan tests passing
* graph tests passing
* json packages test update
* command test updates
* update show test fixtures
* state show tests passing
2020-04-06 09:24:23 -07:00
Kristin Laemmert 4f141ae365
rename provider-specific functions (#24417)
missingPlugins was hard-coded to work only with provider plugins, so I
renamed it to clarify the usage.

Also renamed a test provider from greater_than to greater-than as the
underscore is an invalid provider name character and this will become a
hard error in the near future.
2020-03-20 13:59:59 -04:00
Kristin Laemmert 5f313a65ad
command: remove 0.12upgrade (#24403)
* command: remove 0.12upgrade and related `configupgrade` library
* leave deprecation warning for 0.12upgrade to point users to v0.12
2020-03-19 08:01:16 -04:00
Alisdair McDiarmid 3b0b29ef52 command: Add scaffold for 0.13upgrade command 2020-03-16 12:50:24 -04:00
Kristin Laemmert add16fc67b
jsonstate: sort child modules by address for consistency (#24329)
* jsonstate: sort child modules by address for consistency
2020-03-09 15:57:14 -04:00
Paddy e6592dc710
Add support for provider metadata to modules. (#22583)
Implement a new provider_meta block in the terraform block of modules, allowing provider-keyed metadata to be communicated from HCL to provider binaries.

Bundled in this change for minimal protocol version bumping is the addition of markdown support for attribute descriptions and the ability to indicate when an attribute is deprecated, so this information can be shown in the schema dump.

Co-authored-by: Paul Tyng <paul@paultyng.net>
2020-03-05 16:53:24 -08:00
Kristin Laemmert 7f1b0a4681
command/jsonstate: fix inconsistency with resource address (#24256)
* command/jsonstate: fix inconsistency with resource address

Resource addresses in state output were not including index for
instances created with for_each or count, while the index was appearing
in the plan output. This PR fixes that inconsistency, adds tests, and
updates the existing tests.

Fixes #24110

* add tests showing expected prior state resource addressing
* added example of show json state output with modules
2020-03-05 08:13:45 -05:00
Pam Selle c249943360
Module Expansion: Part 2 (#24154)
* WIP: dynamic expand

* WIP: add variable and local support

* WIP: outputs

* WIP: Add referencer

* String representation, fixing tests it impacts

* Fixes TestContext2Apply_outputOrphanModule

* Fix TestContext2Apply_plannedDestroyInterpolatedCount

* Update DestroyOutputTransformer and associated types to reflect PlannableOutputs

* Remove comment about locals

* Remove module count enablement

* Removes allowing count for modules, and reverts the test,
while adding a Skip()'d test that works when you re-enable
the config

* update TargetDownstream signature to match master

* remove unnecessary method

Co-authored-by: James Bardin <j.bardin@gmail.com>
2020-02-24 17:42:32 -05:00
Alisdair McDiarmid f0cf57e5d3 Remove unused import-provider-mismatch fixture 2020-02-14 14:49:26 -05:00
Alisdair McDiarmid 701d095808 command: Fix stale lock when exiting early
If an error occurs on creating the context for console or import, we
would fail to unlock the state. Fix this by unlocking slightly earlier.
Affects console and import commands.

Fixes #23318
2020-02-14 14:48:30 -05:00
Kristin Laemmert 47a16b0937
addrs: embed Provider in AbsProviderConfig instead of Type
a large refactor to addrs.AbsProviderConfig, embedding the addrs.Provider instead of a Type string. I've added and updated tests, added some Legacy functions to support older state formats and shims, and added a normalization step when reading v4 (current) state files (not the added tests under states/statefile/roundtrip which work with both current and legacy-style AbsProviderConfig strings).

The remaining 'fixme' and 'todo' comments are mostly going to be addressed in a subsequent PR and involve looking up a given local provider config's FQN. This is fine for now as we are only working with default assumption.
2020-02-13 15:32:58 -05:00
Kristin Laemmert 80862f3436
command/import: attach references before validating provider (#22862)
There was an order-of-operations bug where the import graph builder was
validating that the provider did not have any resource references before
references were actually being attached. This PR fixes the order of
operations and adds a test (in the command package).

Fixes #22804
2020-02-12 14:00:08 -05:00
Alisdair McDiarmid c77cfaafc2 Add token validation for manual terraform login
When a token is pasted by the user, we make a request to the
TFE API /account/details endpoint to verify its validity. If successful,
we display the logged-in username as confirmation. If not, we refuse to
store the invalid token and display an error message.

This commit also trims whitespace from around the pasted value, to
reduce the likelihood of a copy & paste error.
2020-02-04 17:24:25 -05:00
Kristin Laemmert 18dd1bb4d6
Mildwonkey/tfconfig upgrade (#23670)
* deps: bump terraform-config-inspect library
* configs: parse `version` in new required_providers block

With the latest version of `terraform-config-inspect`, the
required_providers attribute can now be a string or an object with
attributes "source" and "version". This change allows parsing the
version constraint from the new object while ignoring any given source attribute.
2020-01-10 11:54:53 -05:00
Kristin Laemmert 4d8fde3d6f
command: use backend config from state when backend=false is used. (#23802)
* command: use backend config from state when backend=false is used.

When a user runs `terraform init --backend=false`, terraform should
inspect the state for a previously-configured backend, and use that
backend, ignoring any backend config in the current configuration. If no
backend is configured or there is no state, return a local backend.

Fixes #16593
2020-01-07 15:07:06 -05:00
Kristin Laemmert 4b10a6e1bf
command/jsonplan: fix bug with nested modules output (#23092)
`marshalPlannedValues` builds a map of modules to their children in
order to output the resource changes in a tree. The map was built from
the list of resource changes. However if a module had no resources
itself, and only called another module (a very normal case), that module
would not get added to the map causing none of its children to be
output in `planned_values`.

This PR adds a walk up through a given module's ancestors to ensure that
each module, even those without resources, would be added.
2019-10-17 11:33:04 -04:00
Kristin Laemmert 54661ec1df
command/import: fix error during import when implied provider was not used (#22855)
* command/import: properly use `-provider` supplied on the command line

The import command now attaches the provider configuration in the resource
instance, if set. That config is attached to the NodeAbstractResource
during the import graph building. This prevents errors when the implied
provider is not actually in the configuration at all, which may happen
when a configuration is using the `-beta` version of a provider (and
only that `-beta` version).

* command/import: fix variable reassignment and update docs

Fixes #22564
2019-09-20 10:02:42 -04:00
Martin Atkins 8381112a5c command: Tests for the "terraform login" command
These run against a stub OAuth server implementation, verifying that we
are able to run an end-to-end login transaction for both the authorization
code and the password grant types.

This includes adding support for authorization code grants to our stub
OAuth server implementation; it previously supported only the password
grant type.
2019-09-09 11:15:24 -07:00
Martin Atkins a1e387a0e5 command: A stub OAuth server implementation for login testing 2019-09-09 11:15:24 -07:00
James Bardin e4640a43d7 fix show-json test to expect error from nested mod
One of the show json command tests expected no error when presented with
an invalid configuration in a nested module. Modify the test created in
PR #21569 so that it can still verify there is no panic, but now expect
an error from init.
2019-07-16 21:30:04 -04:00
Radek Simko 5b9f2fafc8 Standardise directory name for test data 2019-06-30 10:16:15 +02:00
James Bardin f2e496a14c Have backend operations properly unlock state
Make sure unlock is called with the correct LockID during operations
2017-02-15 14:41:55 -05:00
James Bardin f5ed8cd288 Use NewLockInfo to get a pre-populated value
Using NewLockInfo ensure we start with all required fields filled.
2017-02-15 14:41:55 -05:00
James Bardin 4f0c465187 make command tests pass with new state.Locker 2017-02-15 14:41:55 -05:00
James Bardin f3e4c05250 build the statelocker binary before running
this way we can signal it directly to amke sure it exits cleanly.
2017-02-03 15:59:24 -05:00
James Bardin fb60b6f6f2 Add separate program for locking state files
Depending on the implementation, local state locks may be reentrant
within the same process. Use a separate process to test locked state
files.
2017-02-03 15:31:21 -05:00