Commit Graph

175 Commits

Author SHA1 Message Date
Kristin Laemmert a33a613703 command/init: add debug messages when -plugin-dir is set 2020-05-20 11:04:11 -04:00
Kristin Laemmert 862dc36884
command/init: include config loader diagnostics in output (#24921)
Previously the diagnostics from the config loaders (earlyconfig and
regular) were only appended to the overall diags if an error was found.
This adds all diagnostics from the regular config loader so that any
generated warnings will be displayed, even if there are no errors.

I did not add the `earlyconfig` warnings since they will be displayed if
there is an error and are likely to be duplicated by the config loader.
2020-05-12 08:39:12 -04:00
Kristin Laemmert 60321b41e8
getproviders: move protocol compatibility functions into registry client (#24846)
* internal/registry source: return error if requested provider version protocols are not supported

* getproviders: move responsibility for protocol compatibility checks into the registry client

The original implementation had the providercache checking the provider
metadata for protocol compatibility, but this is only relevant for the
registry source so it made more sense to move the logic into
getproviders.

This also addresses an issue where we were pulling the metadata for
every provider version until we found one that was supported. I've
extended the registry client to unmarshal the protocols in
`ProviderVersions` so we can filter through that list, instead of
pulling each version's metadata.
2020-05-11 13:49:12 -04:00
Kristin Laemmert 0b76100da0
init: return proper config errors (#24865)
Fixed a bug where we were returning earlyConfDiags instead of confDiags.
2020-05-05 10:08:05 -04:00
Kristin Laemmert ce03f1255f
internal/providercache: fix error message for protocol mismatch (#24818)
There was a bug in the installer trying to pass a nil error.
2020-04-30 11:12:04 -04:00
Alisdair McDiarmid a5b3d497cc internal: Verify provider signatures on install
Providers installed from the registry are accompanied by a list of
checksums (the "SHA256SUMS" file), which is cryptographically signed to
allow package authentication. The process of verifying this has multiple
steps:

- First we must verify that the SHA256 hash of the package archive
  matches the expected hash. This could be done for local installations
  too, in the future.
- Next we ensure that the expected hash returned as part of the registry
  API response matches an entry in the checksum list.
- Finally we verify the cryptographic signature of the checksum list,
  using the public keys provided by the registry.

Each of these steps is implemented as a separate PackageAuthentication
type. The local archive installation mechanism uses only the archive
checksum authenticator, and the HTTP installation uses all three in the
order given.

The package authentication system now also returns a result value, which
is used by command/init to display the result of the authentication
process.

There are three tiers of signature, each of which is presented
differently to the user:

- Signatures from the embedded HashiCorp public key indicate that the
  provider is officially supported by HashiCorp;
- If the signing key is not from HashiCorp, it may have an associated
  trust signature, which indicates that the provider is from one of
  HashiCorp's trusted partners;
- Otherwise, if the signature is valid, this is a community provider.
2020-04-17 13:57:19 -04:00
Martin Atkins 297a3a5db9 command/init: Restore the unconstrained provider warnings
When a provider dependency is implicit rather than explicit, or otherwise
when version constraints are lacking, we produce a warning recommending
the addition of explicit version constraints in the configuration.

This restores the warning functionality from previous Terraform versions,
adapting it slightly to account for the new provider FQN syntax and to
recommend using a required_providers block rather than version constraints
in "provider" blocks, because the latter is no longer recommended in the
documentation.
2020-04-06 09:24:23 -07:00
Alisdair McDiarmid b233aa39e9 addrs: Simplify presentation of provider FQNs
The provider fully-qualified name string used in configuration is very
long, and since most providers are hosted in the public registry, most
of that length is redundant. This commit adds and uses a `ForDisplay`
method, which simplifies the presentation of provider FQNs.

If the hostname is the default hostname, we now display only the
namespace and type. This is only used in UI, but should still be
unambiguous, as it matches the FQN string parsing behaviour.
2020-04-06 09:24:23 -07:00
Martin Atkins 958ea4f7d1 internal/providercache: Handle built-in providers
Built-in providers are special providers that are distributed as part of
Terraform CLI itself, rather than being installed separately. They always
live in the terraform.io/builtin/... namespace so it's easier to see that
they are special, and currently there is only one built-in provider named
"terraform".

Previous commits established the addressing scheme for built-in providers.
This commit makes the installer aware of them to the extent that it knows
not to try to install them the usual way and it's able to report an error
if the user requests a built-in provider that doesn't exist or tries to
impose a particular version constraint for a built-in provider.

For the moment the tests for this are the ones in the "command" package
because that's where the existing testing infrastructure for this
functionality lives. A later commit should add some more focused unit
tests here in the internal/providercache package, too.
2020-04-06 09:24:23 -07:00
Martin Atkins c4fb22863c command/init: Support -plugin-dir again
This is a slightly different approach than we used to take for this
option: rather than disabling the installer and causing all future
commands to look elsewhere for plugins, we'll now leave the installer
enabled by constrain it to only look at the given directories.

This is overall simpler because it doesn't require any special tracking
of the plugin directories for subsequent commands. Instead, the selections
file generated by the installer will record the versions it selected from
the specified directories, and we'll link them in to the local cache just
as we would normally so that other commands don't need to do anything
special to select the right plugins in either case.
2020-04-06 09:24:23 -07:00
Martin Atkins 549aede792 Remove terraform.ResourceProvider, use providercache.Installer instead
Back when we first introduced provider versioning in Terraform 0.10, we
did the provider version resolution in terraform.NewContext because we
weren't sure yet how exactly our versioning model was going to play out
(whether different versions could be selected per provider configuration,
for example) and because we were building around the limitations of our
existing filesystem-based plugin discovery model.

However, the new installer codepath is new able to do all of the
selections up front during installation, so we don't need such a heavy
inversion of control abstraction to get this done: the command package can
select the exact provider versions and pass their factories directly
to terraform.NewContext as a simple static map.

The result of this commit is that CLI commands other than "init" are now
able to consume the local cache directory and selections produced by the
installation process in "terraform init", passing all of the selected
providers down to the terraform.NewContext function for use in
implementing the main operations.

This commit is just enough to get the providers passing into the
terraform.Context. There's still plenty more to do here, including to
repair all of the tests this change has additionally broken.
2020-04-06 09:24:23 -07:00
Martin Atkins f113a7c22d command/init: Collect provider dependencies using our new helpers
This produces a value shaped the way the provider installer expects
without the need for further flattening and preprocessing.
2020-04-06 09:24:23 -07:00
Martin Atkins e6df3905c9 command/init: Generate progress output during provider installation
This restores some sequential event log output similar to what we had in
the previous implementation of plugin installation.
2020-04-06 09:24:23 -07:00
Martin Atkins 94e1ac2d07 command: Minimal integration of new provider installer in "init"
There's still a lot of work to do here around both the UX and the
follow-up steps that need to happen after installation completes, but this
is enough to faciliate some initial end-to-end testing of the new-style
install process.
2020-04-06 09:24:23 -07:00
Alisdair McDiarmid 67203dade8 command: Simplify Meta.process helper method
After some refactoring, this helper method had an unused argument (vars)
and an always-nil error return value. This commit cleans this up.
2020-04-01 15:01:08 -04:00
Kristin Laemmert 4f141ae365
rename provider-specific functions (#24417)
missingPlugins was hard-coded to work only with provider plugins, so I
renamed it to clarify the usage.

Also renamed a test provider from greater_than to greater-than as the
underscore is an invalid provider name character and this will become a
hard error in the near future.
2020-03-20 13:59:59 -04:00
Kristin Laemmert 5f313a65ad
command: remove 0.12upgrade (#24403)
* command: remove 0.12upgrade and related `configupgrade` library
* leave deprecation warning for 0.12upgrade to point users to v0.12
2020-03-19 08:01:16 -04:00
Martin Atkins f899f5aa42 command/init: new error message for signature verification failure
Our initial Terraform 0.13.0 release will continue to support only the
hard-coded official HashiCorp signing key, with support for other keys to
follow in a later release once the trust infrastructure is in place to
support that.

This change is intended to (marginally) improve the UX for a possible
future situation where a HashiCorp-distributed provider makes a released
signed with a new key and a prior version of Terraform ends up trying to
install it due to incorrect version constraints. With this new text we
hope to give the user a better prompt for onward troubleshooting, but
in a sort of hedging way because we have not yet finalized the details of
how new keys might roll out in practice.

Hopefully a user seeing this message would consult the release notes for
Terraform itself and for the provider in question and find some
as-yet-undetermined information about how to proceed.

If the decentralized trust model design comes together before the v0.13.0
release then we may make further amendments here to prepare for that, but
that work should not block the v0.13.0 release if other work concludes
first.
2020-02-25 10:31:54 -05:00
Kristin Laemmert 4d8fde3d6f
command: use backend config from state when backend=false is used. (#23802)
* command: use backend config from state when backend=false is used.

When a user runs `terraform init --backend=false`, terraform should
inspect the state for a previously-configured backend, and use that
backend, ignoring any backend config in the current configuration. If no
backend is configured or there is no state, return a local backend.

Fixes #16593
2020-01-07 15:07:06 -05:00
Kristin Laemmert f8a23da480 explicitly create legacy-style provider 2019-12-09 15:17:47 -05:00
Kristin Laemmert efafadbe5e command: rename choosePlugins to chooseProviders to clarify scope of function 2019-12-09 09:47:09 -05:00
Kristin Laemmert 9891d0354a
providers: use addrs.Provider as map keys for provider.Factory (#23548)
* terraform/context: use new addrs.Provider as map key in provider factories
* added NewLegacyProviderType and LegacyString funcs to make it explicit that these are temporary placeholders

This PR introduces a new concept, provider fully-qualified name (FQN), encapsulated by the `addrs.Provider` struct.
2019-12-04 11:30:20 -05:00
Kristin Laemmert 6728e521c1
addrs: rename Provider Name to more accurate Provider Type (#23449) 2019-12-02 15:32:31 -05:00
Martin Atkins 39e609d5fd vendor: switch to HCL 2.0 in the HCL repository
Previously we were using the experimental HCL 2 repository, but now we'll
shift over to the v2 import path within the main HCL repository as part of
actually releasing HCL 2.0 as stable.

This is a mechanical search/replace to the new import paths. It also
switches to the v2.0.0 release of HCL, which includes some new code that
Terraform didn't previously have but should not change any behavior that
matters for Terraform's purposes.

For the moment the experimental HCL2 repository is still an indirect
dependency via terraform-config-inspect, so it remains in our go.sum and
vendor directories for the moment. Because terraform-config-inspect uses
a much smaller subset of the HCL2 functionality, this does still manage
to prune the vendor directory a little. A subsequent release of
terraform-config-inspect should allow us to completely remove that old
repository in a future commit.
2019-10-02 15:10:21 -07:00
Kristin Laemmert 120bb0a66c
plugin/discovery: use new addrs.ProviderType in place of a provider typeName string (#22724)
This is a relatively small change meant to lay the foundation for
future enhancements to providers' address.
2019-09-09 16:59:50 -04:00
Kristin Laemmert a16e1fc0a1
command/init: omit a warning if -backend-config is used with no backend (#22164)
* command/init: omit a warning if -backend-config is used with no backend
block

Terraform would silently accept - and swallow - `-backend-config` on the
CLI when there was no `backend` block. Since it is mostly expected to
override existing backend configuration, terraform
should omit a warning if there is no backend configuration to
override.

If the user intended to override the default (local) backend
configuration, they can first add a `backend` block to the `terraform` block to silence the warning (or just ignore it):

```hcl
terraform {
  backend "local" {}
}
```
2019-07-23 08:08:28 -04:00
Alex Pilon 0450f487fa
move IsEmptyDir to configs package 2019-07-18 13:07:10 -04:00
James Bardin 06dfc4abd8 allow setting -backend-config='' to unset override
There is currently no way to unset -backend-config during init, since
not setting that option assumes the user will use the saved config.
Allow setting `-backend-config=""` to specify no overrides.
2019-05-29 12:58:04 -05:00
Pam Selle 346e341ff2 Only display status link if public registry used
Updates to throw a specific error if using the defined public
registry vs. another registry.
2019-04-05 16:49:27 -04:00
Pam Selle ff7245f27c Add status link to make message more helpful 2019-03-27 16:22:37 -04:00
Pam Selle d72456d188 Add friendly error for when registry unresponsive
If the registry is unresponsive, you will now get an error
specific to this, rather than a misleading "provider unavailable" type
error. Also adds debug logging for when errors like this may occur
2019-03-27 14:39:14 -04:00
findkim 161fe47b34 plugin/discovery: improve providery discovery verification errors 2019-03-21 14:38:51 -05:00
Justin Campbell 24e13d8ec1 plugin/discovery: Return tfdiags from Get
Allows us to surface warnings to the user using the tfdiags interfaces.
2019-03-18 12:21:27 -04:00
James Bardin c814f2da37 Change backend.ValidateConfig to PrepareConfig
This mirrors the change made for providers, so that default values can
be inserted into the config by the backend implementation. This is only
the interface and method name changes, it does not yet add any default
values.
2019-02-25 18:37:20 -05:00
findkim 7816e61614 Bump installer protocol version to 5 and separate client and server protocol references 2019-01-16 15:07:57 -06:00
Kim Ngo 41a2376915
Merge pull request #19981 from findkim/colorize-protocol-mismatch-msg
Condense protocol mismatch UI error message
2019-01-14 15:12:09 -06:00
findkim 6e0de3e3f5 Nest incompatible provider protocol error to include dynamic custom msg 2019-01-14 14:56:47 -06:00
Martin Atkins 86c02d5c35 command: "terraform init" can partially initialize for 0.12upgrade
There are a few constructs from 0.11 and prior that cause 0.12 parsing to
fail altogether, which previously created a chicken/egg problem because
we need to install the providers in order to run "terraform 0.12upgrade"
and thus fix the problem.

This changes "terraform init" to use the new "early configuration" loader
for module and provider installation. This is built on the more permissive
parser in the terraform-config-inspect package, and so it allows us to
read out the top-level blocks from the configuration while accepting
legacy HCL syntax.

In the long run this will let us do version compatibility detection before
attempting a "real" config load, giving us better error messages for any
future syntax additions, but in the short term the key thing is that it
allows us to install the dependencies even if the configuration isn't
fully valid.

Because backend init still requires full configuration, this introduces a
new mode of terraform init where it detects heuristically if it seems like
we need to do a configuration upgrade and does a partial init if so,
before finally directing the user to run "terraform 0.12upgrade" before
running any other commands.

The heuristic here is based on two assumptions:
- If the "early" loader finds no errors but the normal loader does, the
  configuration is likely to be valid for Terraform 0.11 but not 0.12.
- If there's already a version constraint in the configuration that
  excludes Terraform versions prior to v0.12 then the configuration is
  probably _already_ upgraded and so it's just a normal syntax error,
  even if the early loader didn't detect it.

Once the upgrade process is removed in 0.13.0 (users will be required to
go stepwise 0.11 -> 0.12 -> 0.13 to upgrade after that), some of this can
be simplified to remove that special mode, but the idea of doing the
dependency version checks against the liberal parser will remain valuable
to increase our chances of reporting version-based incompatibilities
rather than syntax errors as we add new features in future.
2019-01-14 11:33:21 -08:00
Sander van Harmelen ef9054562e commands: make sure the correct flagset is used
A lot of commands used `c.Meta.flagSet()` to create the initial flagset for the command, while quite a few of them didn’t actually use or support the flags that are then added.

So I updated a few commands to use `flag.NewFlagSet()` instead to only add the flags that are actually needed/supported.

Additionally this prevents a few commands from using locking while they actually don’t need locking (as locking is enabled as a default in `c.Meta.flagSet()`.
2018-11-23 16:13:34 +01:00
Martin Atkins 544c2932ce command: Fix TestInit_checkRequiredVersion
In prior refactoring we lost the required core version check from
"terraform init", which we restore here.

Additionally, this test used to have an incorrect name that suggested it
was testing something in the "getProvider" codepath, but version checking
happens regardless of what other options are selected.
2018-11-12 15:19:55 -08:00
Martin Atkins 8b603e4877 command: Name the Terraform Registry specifically in error message
When we originally wrote this message we struggled a bit for how to refer
to the releases server without writing an awkwardly-ungrammatical
sentence, and so "the official repository" became a placeholder name for
it.

Now that we'll be looking in Terraform Registry this gives us a nice
proper noun to use. This message will need to evolve more as our
integration with the registry gets more sophisticated, but for now this
works.
2018-11-09 09:48:03 -08:00
Sander van Harmelen 52a1b22f7a Implement the remote enhanced backend
This is a refactored version of the `remote` backend that was initially added to Terraform v0.11.8 which should now be compatible with v0.12.0.
2018-11-06 16:29:46 +01:00
Martin Atkins 541952bb8f Revert some work that happened since v0.12-dev branched
This work was done against APIs that were already changed in the branch
before work began, and so it doesn't apply to the v0.12 development work.

To allow v0.12 to merge down to master, we'll revert this work out for now
and then re-introduce equivalent functionality in later commits that works
against the new APIs.
2018-10-16 19:48:28 -07:00
Martin Atkins 44bc7519a6 terraform: More wiring in of new provider types
This doesn't actually work yet, but it builds and then panics in a pretty
satisfying way.
2018-10-16 19:12:54 -07:00
Martin Atkins a3403f2766 terraform: Ugly huge change to weave in new State and Plan types
Due to how often the state and plan types are referenced throughout
Terraform, there isn't a great way to switch them out gradually. As a
consequence, this huge commit gets us from the old world to a _compilable_
new world, but still has a large number of known test failures due to
key functionality being stubbed out.

The stubs here are for anything that interacts with providers, since we
now need to do the follow-up work to similarly replace the old
terraform.ResourceProvider interface with its replacement in the new
"providers" package. That work, along with work to fix the remaining
failing tests, will follow in subsequent commits.

The aim here was to replace all references to terraform.State and its
downstream types with states.State, terraform.Plan with plans.Plan,
state.State with statemgr.State, and switch to the new implementations of
the state and plan file formats. However, due to the number of times those
types are used, this also ended up affecting numerous other parts of core
such as terraform.Hook, the backend.Backend interface, and most of the CLI
commands.

Just as with 5861dbf3fc49b19587a31816eb06f511ab861bb4 before, I apologize
in advance to the person who inevitably just found this huge commit while
spelunking through the commit history.
2018-10-16 19:11:09 -07:00
Kristin Laemmert ce5e66e178 plugin/discovery provider installer: download providers from the registry
Terraform will query the public registry at
https://registry.terraform.io for providers, instead of
https://releases.hashicorp.com.
2018-10-16 18:56:50 -07:00
Martin Atkins 479c6b2466 move "configschema" from "config" to "configs"
The "config" package is no longer used and will be removed as part
of the 0.12 release cleanup. Since configschema is part of the
"new world" of configuration modelling, it makes more sense for
it to live as a subdirectory of the newer "configs" package.
2018-10-16 18:50:29 -07:00
Martin Atkins c937c06a03 terraform: ugly huge change to weave in new HCL2-oriented types
Due to how deeply the configuration types go into Terraform Core, there
isn't a great way to switch out to HCL2 gradually. As a consequence, this
huge commit gets us from the old state to a _compilable_ new state, but
does not yet attempt to fix any tests and has a number of known missing
parts and bugs. We will continue to iterate on this in forthcoming
commits, heading back towards passing tests and making Terraform
fully-functional again.

The three main goals here are:
- Use the configuration models from the "configs" package instead of the
  older models in the "config" package, which is now deprecated and
  preserved only to help us write our migration tool.
- Do expression inspection and evaluation using the functionality of the
  new "lang" package, instead of the Interpolator type and related
  functionality in the main "terraform" package.
- Represent addresses of various objects using types in the addrs package,
  rather than hand-constructed strings. This is not critical to support
  the above, but was a big help during the implementation of these other
  points since it made it much more explicit what kind of address is
  expected in each context.

Since our new packages are built to accommodate some future planned
features that are not yet implemented (e.g. the "for_each" argument on
resources, "count"/"for_each" on modules), and since there's still a fair
amount of functionality still using old-style APIs, there is a moderate
amount of shimming here to connect new assumptions with old, hopefully in
a way that makes it easier to find and eliminate these shims later.

I apologize in advance to the person who inevitably just found this huge
commit while spelunking through the commit history.
2018-10-16 18:46:46 -07:00
Martin Atkins 4ed06a9227 terraform: HCL2-flavored module dependency resolver
For the moment this is just a lightly-adapted copy of
ModuleTreeDependencies named ConfigTreeDependencies, with the goal that
the two can live concurrently for the moment while not all callers are yet
updated and then we can drop ModuleTreeDependencies and its helper
functions altogether in a later commit.

This can then be used to make "terraform init" and "terraform providers"
work properly with the HCL2-powered configuration loader.
2018-10-16 18:44:26 -07:00
Martin Atkins ebafa51723 command: Various updates for the new backend package API
This is a rather-messy, complex change to get the "command" package
building again against the new backend API that was updated for
the new configuration loader.

A lot of this is mechanical rewriting to the new API, but
meta_config.go and meta_backend.go in particular saw some major
changes to interface with the new loader APIs and to deal with
the change in order of steps in the backend API.
2018-10-16 18:44:26 -07:00
Sean Chittenden d749420a25
Fix drift caused from gofmt when running make dev and go 1.11.
A fresh checkout of `origin/master` does not build atm using the `dev`
target because `master` has not been formatted using `gofmt` from Go
1.11 (tis has been the case for a while if you've been running devel).

None of the drift in question is especially new but now that Go 1.11
has been released and gofmt's formatting guidelines have been updated,
it would be *really* nice if the code in `master` reflected the current
tooling in order to avoid having to fight this drift locally.

* 8mo: https://github.com/hashicorp/terraform/blame/master/backend/remote-state/s3/backend_test.go#L260-L261
* 6mo: https://github.com/hashicorp/terraform/blame/master/builtin/provisioners/chef/linux_provisioner_test.go#L124
* 1yr: 7cfeffe36b/command/init.go (L75-L76)
* 12d: 7cfeffe36b/command/meta_backend_test.go (L1437)
* 2yr: 7cfeffe36b/helper/schema/resource_timeout_test.go (L26)
* 4yr: 7cfeffe36b/helper/schema/schema_test.go (L2059)
* 1yr: 7cfeffe36b/plugin/discovery/get_test.go (L151)
2018-09-09 10:18:08 -07:00
Sander van Harmelen 7fb2d1b8de Implement the Enterprise enhanced remote backend 2018-08-03 22:22:55 +02:00
Sander van Harmelen 179b32d426 Add a `CredentialsForHost` method to disco.Disco
By adding this method you now only have to pass a `*disco.Disco` object around in order to do discovery and use any configured credentials for the discovered hosts.

Of course you can also still pass around both a `*disco.Disco` and a `auth.CredentialsSource` object if there is a need or a reason for that!
2018-08-03 11:29:11 +02:00
Sander van Harmelen 495d1ea350 Use New() instead of `once.Do(b.init)` 2018-08-03 11:29:11 +02:00
James Bardin 504ea578ee
Merge pull request #16920 from hashicorp/jbardin/init-future-state
check state version during init
2018-01-05 16:54:41 -05:00
James Bardin fe3aff91f3 missingPlugins should not return internal plugins
Filter the internal plugins out beforehand, so we don't attempt to
locate them at all during init.
2018-01-05 11:52:11 -05:00
James Bardin ea4cb6a20e check state version during init
The init command needs to parse the state to resolve providers, but
changes to the state format can cause that to fail with difficult to
understand errors. Check the terraform version during init and provide
the same error that would be returned by plan or apply.
2017-12-15 11:17:59 -05:00
Martin Atkins 9a5c865040 command: validate config as part of loading it
Previously we required callers to separately call .Validate on the root
module to determine if there were any value errors, but we did that
inconsistently and would thus see crashes in some cases where later code
would try to use invalid configuration as if it were valid.

Now we run .Validate automatically after config loading, returning the
resulting diagnostics. Since we return a diagnostics here, it's possible
to return both warnings and errors.

We return the loaded module even if it's invalid, so callers are free to
ignore returned errors and try to work with the config anyway, though they
will need to be defensive against invalid configuration themselves in
that case.

As a result of this, all of the commands that load configuration now need
to use diagnostic printing to signal errors. For the moment this just
allows us to return potentially-multiple config errors/warnings in full
fidelity, but also sets us up for later when more subsystems are able
to produce rich diagnostics so we can show them all together.

Finally, this commit also removes some stale, commented-out code for the
"legacy" (pre-0.8) graph implementation, which has not been available
for some time.
2017-12-07 14:28:43 -08:00
Martin Atkins ba0514106a return tfdiags.Diagnostics from validation methods
Validation is the best time to return detailed diagnostics
to the user since we're much more likely to have source
location information, etc than we are in later operations.

This change doesn't actually add any detail to the messages
yet, but it changes the interface so that we can gradually
introduce more detailed diagnostics over time.

While here there are some minor adjustments to some of the
messages to improve their consistency with terminology we
use elsewhere.
2017-11-28 11:15:29 -08:00
Martin Atkins 3b180d7f8a
Merge #16543: Bring the "terraform" provider back into core 2017-11-03 12:08:31 -07:00
Martin Atkins d4ee58ce59 Re-integrate the "terraform" provider into the main binary
As part of the 0.10 core/provider split we moved this provider, along with
all the others, out into its own repository.

In retrospect, the "terraform" provider doesn't really make sense to be
separated since it's just a thin wrapper around some core code anyway,
and so re-integrating it into core avoids the confusion that results when
Terraform Core and the terraform provider have inconsistent versions of
the backend code and dependencies.

There is no good reason to use a different version of the backend code
in the provider than in core, so this new "internal provider" mechanism
is stricter than the old one: it's not possible to use an external build
of this provider at all, and version constraints for it are rejected as
a result.

This provider is also run in-process rather than in a child process, since
again it's just a very thin wrapper around code that's already running
in Terraform core anyway, and so the process barrier between the two does
not create enough advantage to warrant the additional complexity.
2017-11-03 11:36:31 -07:00
James Bardin 9c334fe012 update init output
Change "Downloading" to 'Initializing" to match the provider loading
dialog.

List each module being loaded.

If a regisry module is being downloaded, list the registry host, and the
version discovered.

Show the source string from the config that is being fetched, rather
than the go-getter url. The full source can be found in the logs for
debugging.

Add much more extensive logging
2017-11-03 10:28:07 -04:00
James Bardin f64851242e use Storage.GetModule for init -from-module
This will enable copying any module source into a target directory.
2017-10-27 19:27:20 -04:00
Martin Atkins 5cd00a13ec command: use new diagnostics output for config errors
This uses the new diagnostics printer for config-related errors in the
main five commands that deal with config.

The immediate motivation for this is to allow HCL2-produced diagnostics
to be printed out in their full fidelity, though it also slightly changes
the presentation of other errors so that they are not presented in all
red text, which can be hard to read on some terminals.
2017-10-06 11:46:07 -07:00
Martin Atkins 5c0670fdf4 command: allow callers to activate a plugin cache
Here we add a new caller-settable field to command.Meta that activates
a read-through cache directory for plugin installation.
2017-09-29 14:03:09 -07:00
Martin Atkins 2ed9aa6077 command: autocomplete data for "terraform init" 2017-09-26 14:01:13 -07:00
Martin Atkins 0fe43c8977 cli: allow disabling "next steps" message in terraform plan
In #15884 we adjusted the plan output to give an explicit command to run
to apply a plan, whereas before this command was just alluded to in the
prose.

Since releasing that, we've got good feedback that it's confusing to
include such instructions when Terraform is running in a workflow
automation tool, because such tools usually abstract away exactly what
commands are run and require users to take different actions to
proceed through the workflow.

To accommodate such environments while retaining helpful messages for
normal CLI usage, here we introduce a new environment variable
TF_IN_AUTOMATION which, when set to a non-empty value, is a hint to
Terraform that it isn't being run in an interactive command shell and
it should thus tone down the "next steps" messaging.

The documentation for this setting is included as part of the "...in
automation" guide since it's not generally useful in other cases. We also
intentionally disclaim comprehensive support for this since we want to
avoid creating an extreme number of "if running in automation..."
codepaths that would increase the testing matrix and hurt maintainability.

The focus is specifically on the output of the three commands we give in
the automation guide, which at present means the following two situations:

* "terraform init" does not include the final paragraphs that suggest
  running "terraform plan" and tell you in what situations you might need
  to re-run "terraform init".
* "terraform plan" does not include the final paragraphs that either
  warn about not specifying "-out=..." or instruct to run
  "terraform apply" with the generated plan file.
2017-09-14 10:51:41 -07:00
Sunny 2d849f8650 command/init: check required_version
Previously we were checking required_version only during "real" operations, and not during initialization. Catching it during init is better because that's the first command users run on a new working directory.
2017-08-28 11:25:16 -07:00
Radek Simko aabab16069
command/init: Remove extra whitespace 2017-08-17 09:43:09 +02:00
Radek Simko 28b33c9299
plugin: Display version + source when initializing plugins 2017-08-15 19:29:13 +02:00
Martin Atkins 8a7a0a7459 command: terraform init -from-module=...
This restores the earlier behavior of the first positional argument to
terraform init in 0.9, but as a command line option.

The positional argument was removed to improve consistency with other
commands that take a working directory as their first positional argument.
It was originally intended that this functionality would return in a
later release along with some other general improvements to Terraform's
module handling, but we're introducing here an interim solution that
uses the existing module source concept, to allow for easier porting of
workflows that previously depended on the automatic copy behavior.

In a future release this feature may change again as the module
improvements design firms up, but we expect it to be broadly compatible
with this temporary state.
2017-07-28 15:23:29 -07:00
Robert Liebowitz 006744bfe0 Use all tfvars files in working directory
As a side effect, several commands that previously did not have a failure
state can now fail during meta-parameter processing.
2017-07-05 17:24:17 -07:00
Jake Champlin 9944ea6886
core: Skip provider checksum validation based on env var
Skips checksum validation if the `TF_SKIP_PROVIDER_VERIFY` environment variable is set. Undocumented variable, as the primary goal is to significantly improve the local provider development workflow.
2017-07-03 13:59:13 -04:00
Martin Atkins 40e2fbb8e9 command: init to allow plugin init without backend init
Previously init would crash if given these options:
    -backend=false -get-plugins=true

This is because the state is used as a source of provider dependency
information, and we need to instantiate the backend to get the state.

To avoid the crash, we now use the following adjusted behavior:

- if -backend=true, we behave as before

- if -backend=false, we instead try to instantiate the backend the same
  way any other command would, without modifying its configuration

- if we're able to instantiate the backend, we use it to fetch state
  for dependency resolution purposes

- if the backend is not instantiable then we assume it's not yet
  configured and proceed with a nil state, which may cause us to see an
  incomplete picture of the dependencies but still allows the install
  to succeed. Subsequently running "terraform plan" will not work until
  the backend is (re-)initialized, so the incomplete picture of required
  plugins is safe.
2017-06-21 11:43:28 -07:00
Martin Atkins 3dc7d618f7 command: avoid empty string constraints in plugin install errors
This takes care of a few dangling cases where we were still stringifying
empty version constraints, which creates confusing error messages due to
it stringing as the empty string.

For the "no suitable versions available" message, we fall back on the
"provider not found" message if no versions were found even though it's
unconstrained. This should only happen in an edge case where the
provider's index page exists on the releases server but no versions are
yet present.

For the message about plugin protocol versions, this again is an edge
case since with no constraints this should happen only if we release
an incompatible Terraform version but don't release a new version of the
plugin that's compatible. In this case we just show the constraint as
"(any version)" to make sure we always show _something_.
2017-06-21 09:18:17 -07:00
James Bardin c10f5caf05 Merge pull request #15345 from hashicorp/jbardin/plugin-sig
verify automatically downloaded plugins
2017-06-20 17:05:54 -04:00
Martin Atkins 9c2fe3456b command: purge unused plugins as a side effect of plugin installation
Previously we only did this when _upgrading_, but that's unnecessarily
specific and confusing since e.g. plugins can get upgraded implicitly by
constraint changes, which would not then trigger the purge process.

Instead, we'll assume that the user is able to easily re-download plugins
that were purged here, or if they need more specific guarantees they will
manage manually a plugin directory and disable the auto-install behavior
using `-plugin-dir`.
2017-06-20 13:40:03 -07:00
Martin Atkins d48dcbb4a5 command: more-helpful error messages from plugin installation
Now we are able to recognize and handle a few special error situations
from plugin installation with more verbose error messages that give the
user better feedback on how to proceed.
2017-06-20 13:39:45 -07:00
James Bardin 0a47228065 add -verify-plugins flag to init
This provides the user with the option to disable plugin signature
verification.
2017-06-20 13:14:31 -04:00
James Bardin da385c4268 Don't show plugin init message if there are none
Skip the message and plugin initialization if there are none in the
config.
2017-06-19 12:08:42 -04:00
James Bardin ec99b6910b remove "~> 0.0" constraint suggestions
Don't suggest constraints when the available plugin isn't versioned.

Add zero version const for comparisons.
2017-06-16 16:25:36 -04:00
James Bardin be2069ac81 add -plugin-dir option
The -plugin-dir option lets the user specify custom search paths for
plugins. This overrides all other plugin search paths, and prevents the
auto-installation of plugins.

We also make sure that the availability of plugins is always checked
during init, even if -get-plugins=false or -plugin-dir is set.
2017-06-15 15:23:16 -04:00
James Bardin 000e860706 Add plugin dir scaffolding
add pluginDir to command.Meta, the flag to initialize it, and the
methods to save and restore it.
2017-06-15 14:26:12 -04:00
James Bardin 55bf19e548 always write to dataDir in the current directory
Now that init can take a directory for configuration, the old behavior
of writing the .terraform data directory into the target path no longer
makes sense. Don't change the dataDir field during init, and write to
the default location.

Clean up all references to Meta.dataDir, and only use the getter method
in case we chose to dynamically override this at some point.
2017-06-14 15:14:26 -04:00
Martin Atkins 6979a07754 command: init -upgrade for provider plugins
Now when -upgrade is provided to "terraform init" (and plugin installation
isn't disabled) it will:

- ignore the contents of the auto-install plugin directory when deciding
  what is "available", thus causing anything there to be reinstalled,
  possibly at a newer version.
- if installation completes successfully, purge from the auto-install
  plugin directory any plugin-looking files that aren't in the set of
  chosen plugins.

As before, plugins outside of the auto-install directory are able to
take precedence over the auto-install ones, and these will never be
upgraded nor purged.

The thinking here is that the auto-install directory is an implementation
detail directly managed by Terraform, and so it's Terraform's
responsibility to automatically keep it clean as plugins are upgraded.

We don't yet have the -plugin-dir option implemented, but once it is it
should circumvent all of this behavior and just expect providers to be
already available in the given directory, meaning that nothing will be
auto-installed, -upgraded or -purged.
2017-06-13 12:28:07 -07:00
Martin Atkins f753974bb3 plugin/discovery: Installer interface, and provider implementation
Previously we had a "getProvider" function type used to implement plugin
fetching. Here we replace that with an interface type, initially with
just a "Get" function.

For now this just simplifies the interface by allowing the target
directory and protocol version to be members of the struct rather than
passed as arguments.

A later change will extend this interface to also include a method to
purge unused plugins, so that upgrading frequently doesn't leave behind
a trail of unused executable files.
2017-06-13 12:28:07 -07:00
Gavin Williams 5834333ea3 command: terraform get -upgrade
As of this commit this just upgrades modules, but this option will also
later upgrade plugins and indeed anything else that's being downloaded and
installed as part of the init.
2017-06-13 12:28:07 -07:00
James Bardin a30007b41d Merge pull request #15032 from hashicorp/jbardin/init
change init args to remove source copy
2017-06-12 09:06:32 -04:00
Martin Atkins 418a8a8bc9 command + backend: rename various API objects to "Workspace" terminology
We're shifting terminology from "environment" to "workspace". This takes
care of some of the main internal API surface that was using the old
terminology, though is not intended to be entirely comprehensive and is
mainly just to minimize the amount of confusion for maintainers as we
continue moving towards eliminating the old terminology.
2017-06-09 16:26:25 -07:00
James Bardin 2bfcdbbd08 change init args to remove source copy
When init was modified in 0.9 to initialize a terraform working
directory, the legacy behavior was kept to copy or fetch module sources.
This left the init command without the ability that the plan and apply
commands have to target a specific directory for the operation.

This commit removes the legacy behavior altogether, and allows init to
target a directory for initialization, bringing it into parity with plan
and apply. If one want to copy a module to the target or current
directory, that will have to be done manually before calling init. We
can later reintroduce fetching modules with init without breaking this
new behavior, by adding the source as an optional second argument.

The unit tests testing the copying of sources with init have been
removed, as well as some out of date (and commented out) init tests
regarding remote states.
2017-06-09 17:50:14 -04:00
James Bardin a529b64cc8 better init error output
Provide log-form message when a provider isn't found, along with the
desired constraints.
2017-06-09 14:03:59 -07:00
Martin Atkins 766f8e5d64 command init: remove confusing uses of "environment" in the usage
"environment" is a very overloaded term, so here we prefer to use the
term "working directory" to talk about a local directory where operations
are executed on a given Terraform configuration.
2017-06-09 14:03:59 -07:00
Martin Atkins 3c429b3628 command init: show log output for each provider plugin downloaded
Each provider plugin will take at least a few seconds to download, so
providing feedback about each one should make users feel less like
Terraform has hung.

Ideally we'd show ongoing progress during the download, but that's not
possible without re-working go-getter, so we'll accept this as an interim
solution for now.
2017-06-09 14:03:59 -07:00
Martin Atkins 4ba20f9c1c command init: show suggested constraints for unconstrained providers
When running "terraform init" with providers that are unconstrained, we
will now produce information to help the user update configuration to
constrain for the particular providers that were chosen, to prevent
inadvertently drifting onto a newer major release that might contain
breaking changes.

A ~> constraint is used here because pinning to a single specific version
is expected to create dependency hell when using child modules. By using
this constraint mode, which allows minor version upgrades, we avoid the
need for users to constantly adjust version constraints across many
modules, but make major version upgrades still be opt-in.

Any constraint at all in the configuration will prevent the display of
these suggestions, so users are free to use stronger or weaker constraints
if desired, ignoring the recommendation.
2017-06-09 14:03:59 -07:00
James Bardin e0f2235f66 update init command with new GetProvider signature
GetProvider needs the plugin protocol version to be passed in
2017-06-09 14:03:59 -07:00
Martin Atkins 032f71f1ff command: produce provider lock file during "terraform init"
Once we've installed the necessary plugins, we'll do one more walk of
the available plugins and record the SHA256 hashes of all of the plugins
we select in the provider lock file.

The file we write here gets read when we're building ContextOpts to
initialize the main terraform context, so any command that works with
the context will then fail if any of the provider binaries change.
2017-06-09 14:03:59 -07:00
Martin Atkins e3401947a6 plugin/discovery: PluginRequirements can specify SHA256 digests
As well as constraining plugins by version number, we also want to be
able to pin plugins to use specific executables so that we can detect
drift in available plugins between commands.

This commit allows such requirements to be specified, but doesn't yet
specify any such requirements, nor validate them.
2017-06-09 14:03:59 -07:00
James Bardin 4c32cd432a make getProvider pluggable 2017-06-09 14:03:59 -07:00
James Bardin 2749946f5c basic plugin getter
Add discovery.GetProviders to fetch plugins from the relases site.

This is an early version, with no tests, that only (probably) fetches
plugins from the default location. The URLs are still subject to change,
and since there are no plugin releases, it doesn't work at all yet.
2017-06-09 14:03:59 -07:00
James Bardin 718ede0636 have Meta.Backend use a Config rather than loading
Instead of providing the a path in BackendOpts, provide a loaded
*config.Config instead. This reduces the number of places where
configuration is loaded.
2017-06-09 14:03:59 -07:00