Commit Graph

205 Commits

Author SHA1 Message Date
Martin Atkins 86f0b5191c addrs: Stronger validation and normalization of provider namespace/type
The provider FQN is becoming our primary identifier for a provider, so
it's important that we are clear about the equality rules for these
addresses and what characters are valid within them.

We previously had a basic regex permitting ASCII letters and digits for
validation and no normalization at all. We need to do at least case
folding and UTF-8 normalization because these names will appear in file
and directory names in case-insensitive filesystems and in repository
names such as on GitHub.

Since we're already using DNS-style normalization and validation rules
for the hostname part, rather than defining an entirely new set of rules
here we'll just treat the provider namespace and type as if they were
single labels in a DNS name. Aside from some internal consistency, that
also works out nicely because systems like GitHub use organization and
repository names as part of hostnames (e.g. with GitHub Pages) and so
tend to apply comparable constraints themselves.

This introduces the possibility of names containing letters from alphabets
other than the latin alphabet, and for latin letters with diacritics.
That's consistent with our introduction of similar support for identifiers
in the language in Terraform 0.12, and is intended to be more friendly to
Terraform users throughout the world that might prefer to name their
products using a different alphabet. This is also a further justification
for using the DNS normalization rules: modern companies tend to choose
product names that make good domain names, and now such names will be
usable as Terraform provider names too.
2020-02-18 15:42:09 -08:00
Kristin Laemmert 6e2618d9be terraform: ProviderTransform gets provider fqn from module
Added configs.Module.ProviderForLocalProviderConfig which allows
terraform.ProviderTransformer to get the provider FQN from the module,
instead of assuming NewLegacyProvider.
2020-02-14 15:41:31 -08:00
Kristin Laemmert 72ec579807
configs/decodeRequiredProviders test: before comparing slices (#24114) 2020-02-14 08:26:14 -05:00
James Bardin 37e60cd883 fix comment text 2020-02-13 20:46:48 -05:00
James Bardin 84d1f5c688 convert destroy provisioner warnings to errors 2020-02-13 15:42:10 -05:00
Kristin Laemmert 47a16b0937
addrs: embed Provider in AbsProviderConfig instead of Type
a large refactor to addrs.AbsProviderConfig, embedding the addrs.Provider instead of a Type string. I've added and updated tests, added some Legacy functions to support older state formats and shims, and added a normalization step when reading v4 (current) state files (not the added tests under states/statefile/roundtrip which work with both current and legacy-style AbsProviderConfig strings).

The remaining 'fixme' and 'todo' comments are mostly going to be addressed in a subsequent PR and involve looking up a given local provider config's FQN. This is fine for now as we are only working with default assumption.
2020-02-13 15:32:58 -05:00
Kristin Laemmert 851e6dcdbb
configs: added map to configs.Module for provider local name lookup (#24039)
* configs: added map of ProviderLocalNames to configs.Module

We will need to lookup any user-supplied local names for a given FQN.
This PR adds a map of ProviderLocalNames to the Module, along with
adding tests for this and for decodeRequiredProvidersBlock.

This also introduces the appearance of support for a required_provider
"source" attribute, but ignores any user-supplied source and instead
continues to assume that addrs.NewLegacyProvider is the way to go.
2020-02-11 13:17:37 -05:00
Kristin Laemmert 7eed30595a
moduledeps: replace ProviderInstance with addrs.Provider (#24017)
* addrs: add ParseProviderSourceString function to parse fqns from
tfconfig-inspect
* moduledeps: use addrs.Provider instead of ProviderInstance
2020-02-05 09:27:32 -05:00
Kristin Laemmert 80ab551867
terraform: use addrs.Provider as map keys for provider schemas (#24002)
This is a stepping-stone PR for the provider source project. In this PR
"legcay-stype" FQNs are created from the provider name string. Future
work involves encoding the FQN directly in the AbsProviderConfig and
removing the calls to addrs.NewLegacyProvider().
2020-02-03 08:18:04 -05:00
Martin Atkins 8b511524d6
Initial steps towards AbsProviderConfig/LocalProviderConfig separation (#23978)
* Introduce "Local" terminology for non-absolute provider config addresses

In a future change AbsProviderConfig and LocalProviderConfig are going to
become two entirely distinct types, rather than Abs embedding Local as
written here. This naming change is in preparation for that subsequent
work, which will also include introducing a new "ProviderConfig" type
that is an interface that AbsProviderConfig and LocalProviderConfig both
implement.

This is intended to be largely just a naming change to get started, so
we can deal with all of the messy renaming. However, this did also require
a slight change in modeling where the Resource.DefaultProviderConfig
method has become Resource.DefaultProvider returning a Provider address
directly, because this method doesn't have enough information to construct
a true and accurate LocalProviderConfig -- it would need to refer to the
configuration to know what this module is calling the provider it has
selected.

In order to leave a trail to follow for subsequent work, all of the
changes here are intended to ensure that remaining work will become
obvious via compile-time errors when all of the following changes happen:
- The concept of "legacy" provider addresses is removed from the addrs
  package, including removing addrs.NewLegacyProvider and
  addrs.Provider.LegacyString.
- addrs.AbsProviderConfig stops having addrs.LocalProviderConfig embedded
  in it and has an addrs.Provider and a string alias directly instead.
- The provider-schema-handling parts of Terraform core are updated to
  work with addrs.Provider to identify providers, rather than legacy
  strings.

In particular, there are still several codepaths here making legacy
provider address assumptions (in order to limit the scope of this change)
but I've made sure each one is doing something that relies on at least
one of the above changes not having been made yet.

* addrs: ProviderConfig interface

In a (very) few special situations in the main "terraform" package we need
to make runtime decisions about whether a provider config is absolute
or local.

We currently do that by exploiting the fact that AbsProviderConfig has
LocalProviderConfig nested inside of it and so in the local case we can
just ignore the wrapping AbsProviderConfig and use the embedded value.

In a future change we'll be moving away from that embedding and making
these two types distinct in order to represent that mapping between them
requires consulting a lookup table in the configuration, and so here we
introduce a new interface type ProviderConfig that can represent either
AbsProviderConfig or LocalProviderConfig decided dynamically at runtime.

This also includes the Config.ResolveAbsProviderAddr method that will
eventually be responsible for that local-to-absolute translation, so
that callers with access to the configuration can normalize to an
addrs.AbsProviderConfig given a non-nil addrs.ProviderConfig. That's
currently unused because existing callers are still relying on the
simplistic structural transform, but we'll switch them over in a later
commit.

* rename LocalType to LocalName

Co-authored-by: Kristin Laemmert <mildwonkey@users.noreply.github.com>
2020-01-31 08:23:07 -05:00
Kristin Laemmert 6541775ce4
addrs: roll back change to Type field in ProviderConfig (#23937) 2020-01-28 08:13:30 -05:00
James Bardin 35107acc46 allow "terraform" in destroy provisioners
The values supplied in terraform are static, and do not create any
dependencies.
2020-01-15 11:39:01 -05:00
James Bardin c376905adc it's safe for destroy provisioners to access path
The path values are statically loaded, and do not create any
dependencies that could cause problems with destroy provisioners.
2020-01-13 16:44:44 -05:00
Kristin Laemmert 272cb44d3d
configs: extend module.ProviderRequirements to include the addrs.Provider instead of just version constraints. (#23843)
Renamed file.ProviderRequirements to file.RequiredProviders to match the
name of the block in the configuration. file.RequiredProviders contains
the contents of the file(s); module.ProviderRequirements contains the
parsed and merged provider requirements.

Extended decodeRequiredProvidersBlock to parse the new provider source
syntax (version only, it will ignore any other attributes).

Added some tests; swapped deep.Equal with cmp.Equal in the
terraform/module_dependencies_test.go because deep was not catching
incorrect constraints.
2020-01-13 11:31:47 -05:00
Martin Atkins ff4ea042c2 config: Allow module authors to specify validation rules for variables
The existing "type" argument allows specifying a type constraint that
allows for some basic validation, but often there are more constraints on
a variable value than just its type.

This new feature (requiring an experiment opt-in for now, while we refine
it) allows specifying arbitrary validation rules for any variable which
can then cause custom error messages to be returned when a caller provides
an inappropriate value.

    variable "example" {
      validation {
        condition = var.example != "nope"
        error_message = "Example value must not be \"nope\"."
      }
    }

The core parts of this are designed to do as little new work as possible
when no validations are specified, and thus the main new checking codepath
here can therefore only run when the experiment is enabled in order to
permit having validations.
2020-01-10 15:23:25 -08:00
Kristin Laemmert 18dd1bb4d6
Mildwonkey/tfconfig upgrade (#23670)
* deps: bump terraform-config-inspect library
* configs: parse `version` in new required_providers block

With the latest version of `terraform-config-inspect`, the
required_providers attribute can now be a string or an object with
attributes "source" and "version". This change allows parsing the
version constraint from the new object while ignoring any given source attribute.
2020-01-10 11:54:53 -05:00
Martin Atkins 2b2ac1f6de configs: use local set of go-getter detectors
In an earlier change we switched to defining our own sets of detectors,
getters, etc for go-getter in order to insulate us from upstream changes
to those sets that might otherwise change the user-visible behavior of
Terraform's module installer.

However, we apparently neglected to actually refer to our local set of
detectors, and continued to refer to the upstream set. Here we catch up
with the latest detectors from upstream (taken from the version of
go-getter we currently have vendored) and start using that fixed set.

Currently we are maintaining these custom go-getter sets in two places
due to the configload vs. initwd distinction. That was already true for
goGetterGetters and goGetterDecompressors, and so I've preserved that for
now just to keep this change relatively simple; in later change it would
be nice to factor these "get with go getter" functions out into a shared
location which we can call from both configload and initwd.
2020-01-07 09:38:46 -08:00
Kristin Laemmert 49fc53d1d1
provider source enhancements
* configs: move ProviderConfigCompact[Str] from addrs to configs

The configs package is aware of provider name and type (which are the
same thing today, but expected to be two different things in a future
release), and should be the source of truth for a provider config
address. This is an intermediate step; the next step will change the returned types to something based in the configs package.

* command: rename choosePlugins to chooseProviders to clarify scope of function

* use `Provider.LegacyString()` (instead of `Provider.Type`) consistently
* explicitly create legacy-style provider (continuing from above change)
2019-12-11 08:35:55 -05:00
Martin Atkins b90fb25321 experiments: a mechanism for opt-in experimental language features
Traditionally we've preferred to release new language features in major
releases only, because we can then use the beta cycle to gather feedback
on the feature and learn about any usability challenges or other
situations we didn't consider during our design in time to make those
changes before inclusion in a stable release.

This "experiments" feature is intended to decouple the feedback cycle for
new features from the major release rhythm, and thus allow us to release
new features in minor releases by first releasing them as experimental for
a minor release or two, adjust for any feedback gathered during that
period, and then finally remove the experiment gate and enable the feature
for everyone.

The intended model here is that anything behind an experiment gate is
subject to breaking changes even in patch releases, and so any module
using these experimental features will be broken by a future Terraform
upgrade.

The behavior implemented here is:

- Recognize a new "experiments" setting in the "terraform" block which
  allows module authors to explicitly opt in to experimental features.

  terraform {
    experiments = [resource_for_each]
  }

- Generate a warning whenever loading a module that has experiments
  enabled, to avoid accidentally depending on experimental features and
  thus risking unexpected breakage on next Terraform upgrade.

- We check the enabled experiments against the configuration at module
  load time, which means that experiments are scoped to a particular
  module. Enabling an experiment in one module does not automatically
  enable it in any other module.

This experiments mechanism is itself an experiment, and so I'd like to
use the resource for_each feature to trial it. Because any configuration
using experiments is subject to breaking changes, we are free to adjust
this experiments feature in future releases as we see fit, but once
for_each is shipped without an experiment gate we'll be blocked from
making significant changes to it until the next major release at least.
2019-12-10 09:27:05 -08:00
Kristin Laemmert 67fc4dd5a1 configs: move ProviderConfigCompact[Str] from addrs to configs
The configs package is aware of provider name and type (which are the
same thing today, but expected to be two different things in a future
release), and should be the source of truth for a provider config
address.
2019-12-09 08:30:08 -05:00
James Bardin 2fdf984cce update destroy provisioner warning text
Make it a little more user-oriented
2019-12-06 10:20:23 -05:00
Kristin Laemmert e3416124cc
addrs: replace "Type string" with "Type Provider" in ProviderConfig
* huge change to weave new addrs.Provider into addrs.ProviderConfig
* terraform: do not include an empty string in the returned Providers /
Provisioners
- Fixed a minor bug where results included an extra empty string
2019-12-06 08:00:18 -05:00
James Bardin 6817c844bc
Merge pull request #23559 from hashicorp/jbardin/deprecate-destroy-references
deprecation warning for destroy provisioner refs
2019-12-05 18:07:48 -05:00
Kristin Laemmert 9891d0354a
providers: use addrs.Provider as map keys for provider.Factory (#23548)
* terraform/context: use new addrs.Provider as map key in provider factories
* added NewLegacyProviderType and LegacyString funcs to make it explicit that these are temporary placeholders

This PR introduces a new concept, provider fully-qualified name (FQN), encapsulated by the `addrs.Provider` struct.
2019-12-04 11:30:20 -05:00
James Bardin 8547603ff5 deprecation warning for destroy provisioner refs
Add deprecation warning for references from destroy provisioners or
their connections to external resources or values. In order to ensure
resource destruction can be completed correctly, destroy nodes must be
able to evaluate with only their instance state.

We have sufficient information to validate destroy-time provisioners
early on during the config loading process. Later on these can be
converted to hard errors, and only allow self, count.index, and each.key
in destroy provisioners. Limited the provisioner and block evaluation
scope later on is tricky, but if the references can never be loaded,
then they will never be encountered during evaluation.
2019-12-04 11:14:37 -05:00
James Bardin 6caa5d23e2 fix diagnostics handling
Located all non-test paths where a Diagnostic type was assigned to an
error variable.
2019-11-21 09:14:50 -05:00
Kristin Laemmert 2c78414724
configs/configupgrade: do not panic when int value is out of range (#23394)
`terraform 0.12upgrade` assumes that the configuration has passed 0.11
init, but did not explicitly check that the configuration was valid.
Certain issues would not get caught because the configuration was
syntactically valid. In this case, int or float values out of range
resulted in a panic from `Value()`.

Since running a 0.11 validate command is a breaking change, this PR
merely moves the `Value()` logic for ints and floats into `configupgrade` so
the error can be returned to the user, instead of causing a panic.
2019-11-15 11:02:59 -05:00
Martin Atkins 91752f02da configs: Warn for deprecated interpolation and quoted type constraints
Following on from de652e22a26b, this introduces deprecation warnings for
when an attribute value expression is a template with only a single
interpolation sequence, and for variable type constraints given in quotes.

As with the previous commit, we allowed these deprecated forms with no
warning for a few releases after v0.12.0 to ensure that folks who need to
write cross-compatible modules for a while during upgrading would be able
to do so, but we're now marking these as explicitly deprecated to guide
users towards the new idiomatic forms.

The "terraform 0.12upgrade" tool would've already updated configurations
to not hit these warnings for those who had pre-existing configurations
written for Terraform 0.11.

The main target audience for these warnings are newcomers to Terraform who
are learning from existing examples already published in various spots on
the wider internet that may be showing older Terraform syntax, since those
folks will not be running their configurations through the upgrade tool.
These warnings will hopefully guide them towards modern Terraform usage
during their initial experimentation, and thus reduce the chances of
inadvertently adopting the less-readable legacy usage patterns in
greenfield projects.
2019-11-13 07:55:55 -08:00
Calle Pettersson 73d574913d configs: Mention leading underscores in invalid identifier message 2019-11-11 16:22:33 -08:00
Martin Atkins 79dc808614 configs: Emit warnings for deprecated quoted references/keywords
Terraform 0.12.0 removed the need for putting references and keywords
in quotes, but we disabled the deprecation warnings for the initial
release in order to avoid creating noise for folks who were intentionally
attempting to maintain modules that were cross-compatible with both
Terraform 0.11 and Terraform 0.12.

However, with Terraform 0.12 now more widely used, the lack of these
warnings seems to be causing newcomers to copy the quoted versions from
existing examples on the internet, which is perpetuating the old and
confusing quoted form in newer configurations.

In preparation for phasing out these deprecated forms altogether in a
future major release, and for the shorter-term benefit of giving better
feedback to newcomers when they are learning from outdated examples, we'll
now re-enable those deprecation warnings, and be explicit that the old
forms are intended for removal in a future release.

In order to properly test this, we establish a new set of test
configurations that explicitly mark which warnings they are expecting and
verify that they do indeed produce those expected warnings. We also
verify that the "success" tests do _not_ produce warnings, while removing
the ones that were previously written to succeed but have their warnings
ignored.
2019-11-11 10:17:34 -08:00
Pam Selle cca36025d6
Merge pull request #22946 from hashicorp/kmoe/copy_dir_dotfiles
remove dotfile exclusion in copyDir
2019-10-24 12:01:42 -04:00
Radek Simko 7860f55e4f
Version tools per Go convention under tools.go 2019-10-17 22:23:39 +02:00
Radek Simko 32f9722d9d
Replace import paths & set UA string where necessary 2019-10-11 22:40:54 +01:00
Martin Atkins e21f0fa61e backend/local: Handle interactive prompts for variables in UI layer
During the 0.12 work we intended to move all of the variable value
collection logic into the UI layer (command package and backend packages)
and present them all together as a unified data structure to Terraform
Core. However, we didn't quite succeed because the interactive prompts
for unset required variables were still being handled _after_ calling
into Terraform Core.

Here we complete that earlier work by moving the interactive prompts for
variables out into the UI layer too, thus allowing us to handle final
validation of the variables all together in one place and do so in the UI
layer where we have the most context still available about where all of
these values are coming from.

This allows us to fix a problem where previously disabling input with
-input=false on the command line could cause Terraform Core to receive an
incomplete set of variable values, and fail with a bad error message.

As a consequence of this refactoring, the scope of terraform.Context.Input
is now reduced to only gathering provider configuration arguments. Ideally
that too would move into the UI layer somehow in a future commit, but
that's a problem for another day.
2019-10-10 10:07:01 -07:00
James Bardin 1ee851f256
Merge pull request #22846 from hashicorp/jbardin/evaluate-resource
Always evaluate resources in their entirety
2019-10-08 07:57:15 -04:00
Martin Atkins 39e609d5fd vendor: switch to HCL 2.0 in the HCL repository
Previously we were using the experimental HCL 2 repository, but now we'll
shift over to the v2 import path within the main HCL repository as part of
actually releasing HCL 2.0 as stable.

This is a mechanical search/replace to the new import paths. It also
switches to the v2.0.0 release of HCL, which includes some new code that
Terraform didn't previously have but should not change any behavior that
matters for Terraform's purposes.

For the moment the experimental HCL2 repository is still an indirect
dependency via terraform-config-inspect, so it remains in our go.sum and
vendor directories for the moment. Because terraform-config-inspect uses
a much smaller subset of the HCL2 functionality, this does still manage
to prune the vendor directory a little. A subsequent release of
terraform-config-inspect should allow us to completely remove that old
repository in a future commit.
2019-10-02 15:10:21 -07:00
Katy Moe 2c44a7aacb
remove dotfile exclusion in copyDir
copyDir is used in configload/getter.go to copy previously downloaded modules instead of using the go-getter client every time. The go-getter client downloads dotfiles, but copyDir did not copy dotfiles, leading to inconsistent behaviour when reusing the same module source.
2019-09-30 13:11:57 +01:00
James Bardin 86bf674246 change GetResourceInstance to GetResource
In order to allow lazy evaluation of resource indexes, we can't index
resources immediately via GetResourceInstance. Change the evaluation to
always return whole Resources via GetResource, and index individual
instances during expression evaluation.

This will allow us to always check for invalid index errors rather than
returning an unknown value and ignoring it during apply.
2019-09-19 09:19:14 -04:00
James Bardin 13e2e10577 fix Min/Max validation during decoding
We can only validate MinItems >= 1 (equiv to "Required") during
decoding, as dynamic blocks each only decode as a single block. MaxItems
cannot be validated at all, also because of dynamic blocks, which may
have any number of blocks in the config.
2019-08-20 10:13:21 -04:00
James Bardin 731d4226d3 do not validate Min/Max Items in CoerceValue
Due to both the nature of dynamic blocks, and the need for resources to
sometimes communicate incomplete values, we cannot validate MinItems and
MaxItems in CoerceValue.
2019-08-20 10:13:15 -04:00
James Bardin 3e03213485
Merge pull request #22478 from hashicorp/jbardin/coerce-value
don't validate Min/Max block vals in CoerceValue
2019-08-16 15:58:53 -04:00
Daniel Schroeder 628b50f900 configs/configupgrade: typo fix (#22469) 2019-08-16 10:38:38 -04:00
James Bardin 554cedab8a don't validate Min/Max block vals in CoerceValue
A provider may not have the data to fill in required block values in all
cases during the resource Read operation. This is more common in import,
because there is no initial configuration or state, and it's possible
some values are only provided in the configuration.

The original intent of MinItems and MaxItems in the schema was to
enforce configuration constraints, not to enforce what the resource
could save in the state.  Since the configuration is already statically
validated, and the Schema is validated against the configuration in a
separate step, we can drop these extra validation constraints in
CoerceValue and relax it to only ensure the types conform to what is
expected.
2019-08-15 10:02:39 -04:00
Pam Selle 9631e4c73d
Merge pull request #20571 from sergkondr/fix_misspelling
fix misspelling
2019-08-13 17:13:13 -04:00
Alex Pilon 4bf43efcfd
move hcl2shim package to configs 2019-08-06 19:58:58 -04:00
James Bardin 5878527732
Merge pull request #22221 from hashicorp/jbardin/min-items
MinItems with dynamic blocks
2019-07-29 09:45:42 -07:00
James Bardin 67dbd6d345 don't check MinItems with unknowns in blocks
If a block was defined via "dynamic", there will be only one block value
until the expansion is known. Since we can't detect dynamic blocks at
this point, don't verify MinItems while there are unknown values in the
config.

The decoder spec can also only check for existence of a block, so limit
the check to 0 or 1.
2019-07-27 11:50:28 -07:00
Pam Selle 360068b3cb
Merge pull request #21922 from pselle/resource_for_each
Resource for_each
2019-07-26 11:41:56 -04:00
Pam Selle e7d8ac5ad7 Remove panic, update comment 2019-07-26 11:22:10 -04:00
Thayne McCombs 7c678d104f Add support for for_each for data blocks.
This also fixes a few things with resource for_each:

It makes validation more like validation for count.

It makes sure the index is stored in the state properly.
2019-07-25 16:59:06 -04:00