Commit Graph

84 Commits

Author SHA1 Message Date
Alisdair McDiarmid 62d826e066 command/init: Use full config for provider reqs
Relying on the early config for provider requirements was necessary in
Terraform 0.12, to allow the 0.12upgrade command to run after init
installs providers.

However in 0.13, the same restrictions do not apply, and the detection
of provider requirements has changed. As a result, the early config
loader gives incorrect provider requirements in some circumstances,
such as those in the new test in this commit.

Therefore we are changing the init command to use the requirements found
by the full configuration loader. This also means that we can remove the
internal initwd CheckCoreVersionRequirements function.
2020-05-25 16:50:12 -04:00
Kristin Laemmert 21b9da5a02
internal/providercache: verify that the provider protocol version is compatible (#24737)
* internal/providercache: verify that the provider protocol version is
compatible

The public registry includes a list of supported provider protocol
versions for each provider version. This change adds verification of
support and adds a specific error message pointing users to the closest
matching version.
2020-04-23 08:21:56 -04:00
Alisdair McDiarmid a5b3d497cc internal: Verify provider signatures on install
Providers installed from the registry are accompanied by a list of
checksums (the "SHA256SUMS" file), which is cryptographically signed to
allow package authentication. The process of verifying this has multiple
steps:

- First we must verify that the SHA256 hash of the package archive
  matches the expected hash. This could be done for local installations
  too, in the future.
- Next we ensure that the expected hash returned as part of the registry
  API response matches an entry in the checksum list.
- Finally we verify the cryptographic signature of the checksum list,
  using the public keys provided by the registry.

Each of these steps is implemented as a separate PackageAuthentication
type. The local archive installation mechanism uses only the archive
checksum authenticator, and the HTTP installation uses all three in the
order given.

The package authentication system now also returns a result value, which
is used by command/init to display the result of the authentication
process.

There are three tiers of signature, each of which is presented
differently to the user:

- Signatures from the embedded HashiCorp public key indicate that the
  provider is officially supported by HashiCorp;
- If the signing key is not from HashiCorp, it may have an associated
  trust signature, which indicates that the provider is from one of
  HashiCorp's trusted partners;
- Otherwise, if the signature is valid, this is a community provider.
2020-04-17 13:57:19 -04:00
Kristin Laemmert f09ae6f862
provider source tests: added test suite to exercise hyphenated providers (#24685) 2020-04-16 15:54:33 -04:00
Alisdair McDiarmid 0f5a38b384 internal: Fix init provider lockfile test
The fake installable package meta used a ZIP archive which gave
different checksums between macOS and Linux targets. This commit removes
the target from the contents of this archive, and updates the golden
hash value in the test to match. This test should now pass on both
platforms.
2020-04-06 09:24:23 -07:00
Martin Atkins 958ea4f7d1 internal/providercache: Handle built-in providers
Built-in providers are special providers that are distributed as part of
Terraform CLI itself, rather than being installed separately. They always
live in the terraform.io/builtin/... namespace so it's easier to see that
they are special, and currently there is only one built-in provider named
"terraform".

Previous commits established the addressing scheme for built-in providers.
This commit makes the installer aware of them to the extent that it knows
not to try to install them the usual way and it's able to report an error
if the user requests a built-in provider that doesn't exist or tries to
impose a particular version constraint for a built-in provider.

For the moment the tests for this are the ones in the "command" package
because that's where the existing testing infrastructure for this
functionality lives. A later commit should add some more focused unit
tests here in the internal/providercache package, too.
2020-04-06 09:24:23 -07:00
Martin Atkins de6c9ccec1 command/init: Move "vendored provider" test to e2etests
In the new design the ProviderSource is decided by package main, not by
the "command" package, and so making sure the vendor directory is included
is the responsibility of that package instead. Therefore we can no longer
test this at the "command" package level, but we'll retain a test for it
in e2etests to record that it isn't currently working, so that we have
a prompt to fix it before releasing.
2020-04-06 09:24:23 -07:00
Martin Atkins fe9105abfe command: Fix TestInit_getProviderMissing
This one just needs a tweak for the new wording of the error message.
2020-04-06 09:24:23 -07:00
Martin Atkins add7006de6 command: Fix TestInit_pluginDirProviders and _pluginDirProvidersDoesNotGet
Both of these are attempting to test -plugin-dir, which means we need some
additional help to populate some suitable directories for -plugin-dir to
refer to. The new installFakeProviderPackagesElsewhere helper generalizes
the earlier installFakeProviderPackages to allow installing fake provider
packages to an arbitrary other directory.
2020-04-06 09:24:23 -07:00
Martin Atkins c81eebe0ac command/init: Fix TestInit_providerSource
This test is focused on making sure that the required_providers syntax
is working, so the rewritten version does not include any special handling
of pre-installed packages or "vendored" packages. Pre-installed plugins
are tested in other tests such as TestInit_getUpgradePlugins.
2020-04-06 09:24:23 -07:00
Martin Atkins 14701b8300 command/init: Fix TestInit_getUpgradePlugins
This test now requires a bit of a different approach because it was
previously directly constructing a cache directory but we now use a
different directory layout.

Rather than manually constructing the new heirarchical directory layout
(which would've required a lot more inline code), this introduces a helper
function installFakeProviderPackages that installs a fake provider package
directly into the local cache directory associated with a Meta object,
with the correct directory layout.
2020-04-06 09:24:23 -07:00
Martin Atkins d40085f374 command: Make the tests compile again
They still aren't passing, but this is just enough updating to make the
test program compile successfully after the refactoring related to
provider installation. They are now using the mock provider source offered
by the getproviders package, which is similar but not totally identical
to the idea of mocking the entire installer as these tests used to do, and
so many of them need further adjustment to still be testing what they
intended to test under this new architecture.

Subsequent commits will gradually repair the failing tests.
2020-04-06 09:24:23 -07:00
Kristin Laemmert 4f141ae365
rename provider-specific functions (#24417)
missingPlugins was hard-coded to work only with provider plugins, so I
renamed it to clarify the usage.

Also renamed a test provider from greater_than to greater-than as the
underscore is an invalid provider name character and this will become a
hard error in the near future.
2020-03-20 13:59:59 -04:00
Kristin Laemmert 5f313a65ad
command: remove 0.12upgrade (#24403)
* command: remove 0.12upgrade and related `configupgrade` library
* leave deprecation warning for 0.12upgrade to point users to v0.12
2020-03-19 08:01:16 -04:00
Kristin Laemmert 5901952882
command: tests should not leave dirs behind thank you (#24340) 2020-03-10 16:32:22 -04:00
Kristin Laemmert 92f427779d
command/show: fix issue with show and aliased provider (#23848)
The formatter in `command/format/state.go`, when formatting a resource
with an aliased provider, was looking for a schema with the alias (ie,
test.foo), but the schemas are only listed by provider type (test).
Update the state formatter to lookup schemas by provider type only.

Some of the show tests (and a couple others) were not properly cleaning
up the created tmpdirs, so I fixed those. Also, the show tests are using
a statefile named `state.tfstate`, but were not passing that path to the
show command, so we were getting some false positives (a `show` command
that returns `no state` exits 0).

Fixes #21462
2020-01-13 15:10:00 -05:00
Kristin Laemmert 18dd1bb4d6
Mildwonkey/tfconfig upgrade (#23670)
* deps: bump terraform-config-inspect library
* configs: parse `version` in new required_providers block

With the latest version of `terraform-config-inspect`, the
required_providers attribute can now be a string or an object with
attributes "source" and "version". This change allows parsing the
version constraint from the new object while ignoring any given source attribute.
2020-01-10 11:54:53 -05:00
Kristin Laemmert a16e1fc0a1
command/init: omit a warning if -backend-config is used with no backend (#22164)
* command/init: omit a warning if -backend-config is used with no backend
block

Terraform would silently accept - and swallow - `-backend-config` on the
CLI when there was no `backend` block. Since it is mostly expected to
override existing backend configuration, terraform
should omit a warning if there is no backend configuration to
override.

If the user intended to override the default (local) backend
configuration, they can first add a `backend` block to the `terraform` block to silence the warning (or just ignore it):

```hcl
terraform {
  backend "local" {}
}
```
2019-07-23 08:08:28 -04:00
James Bardin 06dfc4abd8 allow setting -backend-config='' to unset override
There is currently no way to unset -backend-config during init, since
not setting that option assumes the user will use the saved config.
Allow setting `-backend-config=""` to specify no overrides.
2019-05-29 12:58:04 -05:00
James Bardin c017149b31 don't store prepared backend config
The backend gets to "prepare" the configuration before Configure is
called, in order to validate the values and insert defaults. We don't
want to store this value in the "config state", because it will often
not match the raw config after it is prepared, forcing unecessary
backend migrations during init.

Since PrepareConfig is always called before Configure, we can store the
config value directly, and assume that it will be prepared in the same
manner each time.
2019-05-24 14:51:18 -04:00
James Bardin ee9a618369 don't migrate backend during init without override
If the backend config hashes match during init, and there are no new
backend override options, then we assume the existing config is OK.
Since init should be idempotent, we should be able to run init with no
options or config changes, and not effect the backends at all.
2019-05-24 11:31:04 -04:00
James Bardin cefc927e48 failing test for backend re-init 2019-05-23 18:21:52 -04:00
Martin Atkins 63fa1ac418 vendor: go get github.com/hashicorp/hcl2@master
This includes a small fix to ensure the parser doesn't produce an invalid
body for block parsing syntax errors, and instead produces an incomplete
result that calling applications like Terraform can still analyze.

The problem here was affecting our version-constraint-sniffing code, which
intentionally tried to find a core version constraint even if there's a
syntax error so that it can report that a new version of Terraform is a
likely cause of the syntax error. It was working in most cases, unless
it was the "terraform" block itself that contained the error, because then
we'd try to analyze a broken hcl.Block with a nil body.

This includes a new test for "terraform init" that exercises this
recovery codepath.
2019-05-14 15:37:46 -07:00
Justin Campbell 24e13d8ec1 plugin/discovery: Return tfdiags from Get
Allows us to surface warnings to the user using the tfdiags interfaces.
2019-03-18 12:21:27 -04:00
Radek Simko 37f397dded
command/test: Clarify failure (add full path) 2019-01-29 22:16:41 +00:00
Martin Atkins 86c02d5c35 command: "terraform init" can partially initialize for 0.12upgrade
There are a few constructs from 0.11 and prior that cause 0.12 parsing to
fail altogether, which previously created a chicken/egg problem because
we need to install the providers in order to run "terraform 0.12upgrade"
and thus fix the problem.

This changes "terraform init" to use the new "early configuration" loader
for module and provider installation. This is built on the more permissive
parser in the terraform-config-inspect package, and so it allows us to
read out the top-level blocks from the configuration while accepting
legacy HCL syntax.

In the long run this will let us do version compatibility detection before
attempting a "real" config load, giving us better error messages for any
future syntax additions, but in the short term the key thing is that it
allows us to install the dependencies even if the configuration isn't
fully valid.

Because backend init still requires full configuration, this introduces a
new mode of terraform init where it detects heuristically if it seems like
we need to do a configuration upgrade and does a partial init if so,
before finally directing the user to run "terraform 0.12upgrade" before
running any other commands.

The heuristic here is based on two assumptions:
- If the "early" loader finds no errors but the normal loader does, the
  configuration is likely to be valid for Terraform 0.11 but not 0.12.
- If there's already a version constraint in the configuration that
  excludes Terraform versions prior to v0.12 then the configuration is
  probably _already_ upgraded and so it's just a normal syntax error,
  even if the early loader didn't detect it.

Once the upgrade process is removed in 0.13.0 (users will be required to
go stepwise 0.11 -> 0.12 -> 0.13 to upgrade after that), some of this can
be simplified to remove that special mode, but the idea of doing the
dependency version checks against the liberal parser will remain valuable
to increase our chances of reporting version-based incompatibilities
rather than syntax errors as we add new features in future.
2019-01-14 11:33:21 -08:00
James Bardin 149ccd929e missing commits from 19688 2018-12-18 16:54:09 -08:00
Martin Atkins 8c54da0ad2 command: TestInit_fromModule_explicitDest guard against other tests
Some other test is leaving behind a terraform.tfstate after it concludes,
which can cause this test to fail in a strange way due to picking up
extra provider requirements from that state.

This check doesn't fix that problem, but it at least makes the test fail
in a more helpful way to avoid time wasted trying to debug this test when
it's some other test that actually has the bug.
2018-11-12 15:19:55 -08:00
Martin Atkins 544c2932ce command: Fix TestInit_checkRequiredVersion
In prior refactoring we lost the required core version check from
"terraform init", which we restore here.

Additionally, this test used to have an incorrect name that suggested it
was testing something in the "getProvider" codepath, but version checking
happens regardless of what other options are selected.
2018-11-12 15:19:55 -08:00
Martin Atkins 9ba399bca8 command: Fix TestInit_getProvider
After all of the refactoring we were no longer checking the Terraform
version field in a state file, causing this test to fail.

This restores that check, though with a slightly different error message.
2018-11-12 15:19:55 -08:00
Martin Atkins f6d468ffd5 command: Fix TestInit_inputFalse
This test was using old-style state files as its input, differing only by
lineage. Since lineages are now managed within the state manager itself,
the test can't use that to distinguish the two files and so we put a
different output in each one instead.

This also introduces some TRACE logging to the migration codepaths.
There's some hard-to-follow control flow here and so this extra logging
helps to understand the reason for a particular outcome, and since this
codepath is visited only in "terraform init" anyway it doesn't hurt to
be a bit more verbose here.
2018-11-12 15:19:55 -08:00
Martin Atkins 297b3b8830 command: Fix TestInit_backendReinitConfigToExtra
This test was re-using the same InitCommand value to run multiple times,
which is not realistic. Since we now cache configuration source code
inside command.Meta on load, it's important that we use a fresh
InitCommand instance here so it'll see the modified configuration file
we've left on disk.
2018-11-09 09:48:03 -08:00
Martin Atkins 7b77e20bdc command: Fix several "terraform init" tests 2018-10-16 19:14:11 -07:00
Martin Atkins a3403f2766 terraform: Ugly huge change to weave in new State and Plan types
Due to how often the state and plan types are referenced throughout
Terraform, there isn't a great way to switch them out gradually. As a
consequence, this huge commit gets us from the old world to a _compilable_
new world, but still has a large number of known test failures due to
key functionality being stubbed out.

The stubs here are for anything that interacts with providers, since we
now need to do the follow-up work to similarly replace the old
terraform.ResourceProvider interface with its replacement in the new
"providers" package. That work, along with work to fix the remaining
failing tests, will follow in subsequent commits.

The aim here was to replace all references to terraform.State and its
downstream types with states.State, terraform.Plan with plans.Plan,
state.State with statemgr.State, and switch to the new implementations of
the state and plan file formats. However, due to the number of times those
types are used, this also ended up affecting numerous other parts of core
such as terraform.Hook, the backend.Backend interface, and most of the CLI
commands.

Just as with 5861dbf3fc49b19587a31816eb06f511ab861bb4 before, I apologize
in advance to the person who inevitably just found this huge commit while
spelunking through the commit history.
2018-10-16 19:11:09 -07:00
Martin Atkins ebafa51723 command: Various updates for the new backend package API
This is a rather-messy, complex change to get the "command" package
building again against the new backend API that was updated for
the new configuration loader.

A lot of this is mechanical rewriting to the new API, but
meta_config.go and meta_backend.go in particular saw some major
changes to interface with the new loader APIs and to deal with
the change in order of steps in the backend API.
2018-10-16 18:44:26 -07:00
James Bardin e980156451 cleanup temp files from command tests
Rather than try to modify all the hundreds of calls to the temp helper
functions, and cleanup the temp files at every call site, have all tests
work within a single temp directory that is removed at the end of
TestMain.
2018-03-28 13:08:38 -04:00
James Bardin 61c2be3e95
Merge pull request #16969 from hashicorp/jbardin/reset-plugin-dir
allow init to reset -plugin-dir
2018-01-08 16:51:47 -05:00
James Bardin 504ea578ee
Merge pull request #16920 from hashicorp/jbardin/init-future-state
check state version during init
2018-01-05 16:54:41 -05:00
James Bardin 5a975d9997 add test for internal plugins with -plugin-dir
-plugin-dir was short-circuiting the discovery for internal plugins
2018-01-05 11:51:09 -05:00
James Bardin ba84faf4e1 remove the plugin path only for an empty string
To avoid breaking automation where plugin-path was assumed to be set
permanently, only remove the plugin-path record if it was explicitly set
to and empty string.
2018-01-04 16:49:45 -05:00
James Bardin 79e985366f allow init to reset -plugin-dir
Remove the recorded -plugin-dir during init if the flag is not provided.
2017-12-21 11:21:07 -05:00
James Bardin 7d2da9865e inputFalse test should attempt migration and check error
Make sure the init inputFalse test actually errors from missing input,
since skipping input will still fail later during provider
initialization. We need to make sure there are two different states that
aren't a noop for migration, and reset the command struct for each run.

Also verify that we don't go into an infinite loop if there is no input.
2017-12-18 18:39:21 -05:00
James Bardin ea4cb6a20e check state version during init
The init command needs to parse the state to resolve providers, but
changes to the state format can cause that to fail with difficult to
understand errors. Check the terraform version during init and provide
the same error that would be returned by plan or apply.
2017-12-15 11:17:59 -05:00
James Bardin 1bf64ec788 add "Updating" output and fix output tests 2017-11-03 10:28:08 -04:00
Sunny 2d849f8650 command/init: check required_version
Previously we were checking required_version only during "real" operations, and not during initialization. Catching it during init is better because that's the first command users run on a new working directory.
2017-08-28 11:25:16 -07:00
James Bardin fa20d43d80 test loading of Meta.PluginOverrides
These are currently being skipped in discovery
2017-08-09 11:13:54 -04:00
Martin Atkins 8a7a0a7459 command: terraform init -from-module=...
This restores the earlier behavior of the first positional argument to
terraform init in 0.9, but as a command line option.

The positional argument was removed to improve consistency with other
commands that take a working directory as their first positional argument.
It was originally intended that this functionality would return in a
later release along with some other general improvements to Terraform's
module handling, but we're introducing here an interim solution that
uses the existing module source concept, to allow for easier porting of
workflows that previously depended on the automatic copy behavior.

In a future release this feature may change again as the module
improvements design firms up, but we expect it to be broadly compatible
with this temporary state.
2017-07-28 15:23:29 -07:00
Martin Atkins 40e2fbb8e9 command: init to allow plugin init without backend init
Previously init would crash if given these options:
    -backend=false -get-plugins=true

This is because the state is used as a source of provider dependency
information, and we need to instantiate the backend to get the state.

To avoid the crash, we now use the following adjusted behavior:

- if -backend=true, we behave as before

- if -backend=false, we instead try to instantiate the backend the same
  way any other command would, without modifying its configuration

- if we're able to instantiate the backend, we use it to fetch state
  for dependency resolution purposes

- if the backend is not instantiable then we assume it's not yet
  configured and proceed with a nil state, which may cause us to see an
  incomplete picture of the dependencies but still allows the install
  to succeed. Subsequently running "terraform plan" will not work until
  the backend is (re-)initialized, so the incomplete picture of required
  plugins is safe.
2017-06-21 11:43:28 -07:00
Martin Atkins 9c2fe3456b command: purge unused plugins as a side effect of plugin installation
Previously we only did this when _upgrading_, but that's unnecessarily
specific and confusing since e.g. plugins can get upgraded implicitly by
constraint changes, which would not then trigger the purge process.

Instead, we'll assume that the user is able to easily re-download plugins
that were purged here, or if they need more specific guarantees they will
manage manually a plugin directory and disable the auto-install behavior
using `-plugin-dir`.
2017-06-20 13:40:03 -07:00
James Bardin 270eedd4b8 always pass in the full plugin path to dicovery
Discovery no longer tries to walk into OS_ARCH dirs, so always pass in
the full search path.
2017-06-16 14:09:47 -04:00