Commit Graph

23106 Commits

Author SHA1 Message Date
Sander van Harmelen 35d9ce3f92 backend/remote: implement the Local interface 2018-11-26 20:50:25 +01:00
Sander van Harmelen e68377b1f8
Merge pull request #19431 from hashicorp/svh/f-flagsets
commands: make sure the correct flagset is used
2018-11-26 20:44:50 +01:00
Martin Atkins 3b49028b77 core: Static-validate resource references against schemas
In the initial move to HCL2 we started relying only on full expression
evaluation to catch attribute errors, but that's not sufficient for
resource attributes in practice because during validation we can't know
yet whether a resource reference evaluates to a single object or to a
list of objects (if count is set).

To address this, here we reinstate some static validation of resource
references by analyzing directly the reference objects, disregarding any
instance index if present, and produce errors if the remaining subsequent
traversal steps do not correspond to items within the resource type
schema.

This also allows us to produce some more specialized error messages for
certain situations. In particular, we can recognize a reference like
aws_instance.foo.count, which in 0.11 and prior was a weird special case
for determining the count value of a resource block, and offer a helpful
error showing the new length(aws_instance.foo) usage pattern.

This eventually delegates to the static traversal validation logic that
was added to the configschema package in a previous commit, which also
includes some specialized error messages that distinguish between
attributes and block types in the schema so that the errors relate more
directly to constructs the user can see in the configuration.

In future we could potentially move more of the checks from the dynamic
schema construction step to the static validation step, but resources
are the reference type that most needs this immediately due to the
ambiguity caused by the instance indexing syntax. We can safely refactor
other reference types to be statically validated in later releases.

This is verified by two pre-existing context validate tests which we
temporarily disabled during earlier work (now re-enabled) and also by a
new validate test aimed specifically at the special case for the "count"
attribute.
2018-11-26 08:25:03 -08:00
Martin Atkins bbbf22d8e4 configs/configschema: Block.StaticValidateTraversal method
This allows basic static validation of a traversal against a schema, to
verify that it represents a valid path through the structural parts of
the schema.

The main purpose of this is to produce better error messages (using our
knowledge of the schema) than we'd be able to achieve by just relying
on HCL expression evaluation errors. This is particularly important for
nested blocks because it may not be obvious whether one is represented
internally by a set or a list, and incorrect usage would otherwise produce
a confusing HCL-oriented error message.
2018-11-26 08:25:03 -08:00
Martin Atkins 1226e77999 core: Remove GraphSemanticChecker, etc
These overly-general interfaces are no longer used anywhere, and their
presence in the important-sounding semantics.go file was a distracting
red herring.

We'd previously replaced the one checker in here with a simple helper
function for checking input variables, and that's arguably more at home
with all of the other InputValue functionality in variables.go, and that
allows us to remove semantics.go (and its associated test file) altogether
and make room for some forthcoming new files for static validation.
2018-11-26 08:25:03 -08:00
Martin Atkins 0681935df5 configs: Reserve various names for future use
We want the forthcoming v0.12.0 release to be the last significant
breaking change to our main configuration constructs for a long time, but
not everything could be implemented in that release.

As a compromise then, we reserve various names we have some intent of
using in a future release so that such future uses will not be a further
breaking change later.

Some of these names are associated with specific short-term plans, while
others are reserved conservatively for possible later work and may be
"un-reserved" in a later release if we don't end up using them. The ones
that we expect to use in the near future were already being handled, so
we'll continue to decode them at the config layer but also produce an
error so that we don't get weird behavior downstream where the
corresponding features don't work yet.
2018-11-26 08:25:03 -08:00
Tom Harvey 3259e969cb
Updating to include #19460 2018-11-26 14:42:55 +01:00
Tom Harvey 6d4f702467
backend/azurerm: support for custom resource manager endpoints (#19460)
* backend/azurerm: removing the `arm_` prefix from keys

* removing the deprecated fields test because the deprecation makes it fail

* authentication: support for custom resource manager endpoints

* Adding debug prefixes to the log statements
2018-11-26 14:42:16 +01:00
Tom Harvey 26509550e4
Updating to include #19448 2018-11-26 11:20:48 +01:00
Tom Harvey d580f30e03
backend/azurerm: removing the `arm_` prefix from keys (#19448)
* backend/azurerm: removing the `arm_` prefix from keys

* removing the deprecated fields test because the deprecation makes it fail
2018-11-26 11:19:43 +01:00
Sander van Harmelen ef9054562e commands: make sure the correct flagset is used
A lot of commands used `c.Meta.flagSet()` to create the initial flagset for the command, while quite a few of them didn’t actually use or support the flags that are then added.

So I updated a few commands to use `flag.NewFlagSet()` instead to only add the flags that are actually needed/supported.

Additionally this prevents a few commands from using locking while they actually don’t need locking (as locking is enabled as a default in `c.Meta.flagSet()`.
2018-11-23 16:13:34 +01:00
Tom Harvey 9ada53547f
Updating to include #19440 2018-11-22 18:03:06 +01:00
Tom Harvey 96b1c951fa
backend/azurerm: support for authenticating via SAS Tokens (#19440)
* adding acceptance tests for msi auth

* including the resource group name in the tests

* backend/azurerm: support for authenticating using a SAS Token

* resolving merge conflicts

* moving the defer to prior to the error
2018-11-22 18:02:33 +01:00
Tom Harvey a0e5ebf3b3
Updating to include #19441 2018-11-22 17:48:15 +01:00
Tom Harvey e437feadc2
backend/azurerm: fixing a bug where locks couldn't be unlocked (#19441)
Fixes #17046
2018-11-22 17:47:35 +01:00
Tom Harvey fc75cb7e3c
Updating to include #19433 2018-11-22 16:53:37 +01:00
Tom Harvey c928962f44
backend/azurerm: support for authenticating via msi (#19433)
* backend/azurerm: support for authenticating via msi

* adding acceptance tests for msi auth

* including the resource group name in the tests

* support for using the test client via msi
2018-11-22 16:52:27 +01:00
Tom Harvey 9f15381fec
Updating to include #19414 2018-11-21 22:07:01 +01:00
Tom Harvey 0ec109bdc0
backend/azurerm: upgrading the SDK / support for proxies (#19414)
* vendor updates

- updating to v21.3.0 of github.com/Azure/azure-sdk-for-go
- updating to v10.15.4 of github.com/Azure/go-autorest
- vendoring github.com/hashicorp/go-azure-helpers @ 0.1.1

* backend/azurerm: refactoring to use the new auth package

- refactoring the backend to use a shared client via the new auth package
- adding tests covering both Service Principal and Access Key auth
- support for authenticating using a proxy
- rewriting the backend documentation to include examples of both authentication types

* switching to use the build-in logging function

* documenting it's also possible to retrieve the access key from an env var
2018-11-21 22:06:03 +01:00
Sander van Harmelen 4561c80c1d Also show policies when there are no changes
This behavior was recently updated in the TFE UI, so lets follow that behavior here as well.
2018-11-21 11:34:59 +01:00
Nick Fagerlund 7d5db9522f
website: Fix plugin path on Windows (#19423)
...and one other reference to the application data directory.

Context:

https://docs.microsoft.com/en-us/windows/desktop/shell/knownfolderid#folderid_roamingappdata

In newer Windows versions, the folder accessible as `%APPDATA%` (and via various
APIs) is actually at something like "documents and settings\user\application
data\roaming", while earlier versions omit the "\roaming" part of the path. This
means you can confuse people by referring to the "application data" directory by
its human name, because "roaming" is the real application data directory, but it
looks like a subdirectory of "application data".

Thus, it's less confusing to just use the `%APPDATA%` variable, with the added
benefit that you can copy and paste the path and it'll just work in most places.
2018-11-20 16:54:18 -08:00
Sander van Harmelen 5b0b7d0a78
Update CHANGELOG.md 2018-11-20 22:52:22 +01:00
Sander van Harmelen 5e2af5ddff
Update CHANGELOG.md 2018-11-20 22:51:57 +01:00
Sander van Harmelen ae6ddc29ae
Merge pull request #19378 from hashicorp/svh/f-remote-state
backend/remote: change how to fall back to a local backend
2018-11-20 22:47:56 +01:00
Sander van Harmelen a17f317025 Change how to fall back from remote to local backend
In order to support free organizations, we need a way to load the `remote` backend and then, depending on the used offering/plan, enable or disable remote operations.

In other words, we should be able to dynamically fall back to the `local` backend if needed, after first configuring the `remote` backend.

To make this works we need to change the way this was done previously when the env var `TF_FORCE_LOCAL_BACKEND` was set. The clear difference of course being that the env var would be available on startup, while the used offering/plan is only known after being able to connect to TFE.
2018-11-20 22:25:52 +01:00
Sander van Harmelen e7fd446f74
Update CHANGELOG.md 2018-11-20 21:17:48 +01:00
Martin Atkins 67041486c5
build: Travis-CI build should use vendor directory when building for Windows 2018-11-20 11:59:48 -08:00
Justin Campbell 97bdeddabe
Update CHANGELOG.md 2018-11-20 14:32:59 -05:00
Justin Campbell bf301d7abd
Merge pull request #19389 from hashicorp/f-registry-provider-signing-keys
Terraform Registry: Use signing keys provided from the Registry
2018-11-20 14:28:41 -05:00
Justin Campbell c993e9bed6 registry/response: Add protocols to DL resp 2018-11-20 14:09:16 -05:00
Justin Campbell 06825bf46d plugin/discovery: Add friendly gpg err msg
When GPG verification fails, display a helpful message to the user instead of the generic openpgp error.
2018-11-20 14:09:16 -05:00
Justin Campbell 9a8a74b9bb plugin/discovery: Print name before verification
This is so that any errors output from the checksum/signature verification show up in the expected place in the output.
2018-11-20 14:09:16 -05:00
Justin Campbell 495826444b plugin/discovery: Use GPG keys from Registry
When verifying the signature of the SHA256SUMS file, we have been
hardcoding HashiCorp's public GPG key and using it as the keyring.

Going forward, Terraform will get a list of valid public keys for a
provider from the Terraform Registry (registry.terraform.io), and use
them as the keyring for the openpgp verification func.
2018-11-20 14:09:16 -05:00
Sander van Harmelen 17787c943a
Update CHANGELOG.md 2018-11-20 17:57:07 +01:00
Sander van Harmelen 77844852e6 go-mod: update go-tfe 2018-11-20 17:06:19 +01:00
Sander van Harmelen d58d10e204
Merge pull request #19411 from hashicorp/svh/f-state-push
command/state: lock when pushing state
2018-11-20 16:26:08 +01:00
Sander van Harmelen 79a9a15879 command/state: lock when pushing state
Next to adding the locking for the `state push` command, this commit also fixes a small bug where the lock would not be propertly released when running the `state show` command.

And finally it renames some variables in the `[un]taint` code in order to try to standardize the var names of a few frequently used variables (e.g. statemgr.Full, states.State, states.SyncState).
2018-11-20 11:15:16 +01:00
James Bardin 407fcc0385
Merge pull request #19409 from hashicorp/jbardin/terraform-tests
fixes for the remaining tests
2018-11-19 19:19:24 -05:00
James Bardin f375691819 add missing key-value from test 2018-11-19 18:58:29 -05:00
James Bardin 0b7be2d0e3 fixes for the remaining tests
It's possible that a computed collection could be handled by the
attribute name, rather than the index count value.

Use a new testDiffFn for some tests, which don't work with the old
function that can't determine `computed` without the schema.
2018-11-19 18:56:50 -05:00
Martin Atkins 884aa387b8 command: Use vendoring when building helper programs in tests
In a couple places in tests we execute a child "go build" to make a helper
program. Now that we're running in module mode, "go build" will normally
default to downloading and caching dependencies, which we don't want
because we're still using vendoring for the moment.

Therefore we need to instruct these child builds to use vendoring too,
avoiding the need to download all of the dependencies and ensuring that
we'll be building with the same dependencies that we'd use for a normal
build.
2018-11-19 11:41:52 -08:00
Martin Atkins 5b676059a4
Update CHANGELOG.md 2018-11-19 10:04:01 -08:00
Martin Atkins 6606b525ec
Update CHANGELOG.md 2018-11-19 10:03:17 -08:00
Martin Atkins 4fe9632f09 plugin: Establish our current plugin protocol as version 5
The main significant change here is that the package name for the proto
definition is "tfplugin5", which is important because this name is part
of the wire protocol for references to types defined in our package.

Along with that, we also move the generated package into "internal" to
make it explicit that importing the generated Go package from elsewhere is
not the right approach for externally-implemented SDKs, which should
instead vendor the proto definition they are using and generate their
own stubs to ensure that the wire protocol is the only hard dependency
between Terraform Core and plugins.

After this is merged, any provider binaries built against our
helper/schema package will need to be rebuilt so that they use the new
"tfplugin5" package name instead of "proto".

In a future commit we will include more elaborate and organized
documentation on how an external codebase might make use of our RPC
interface definition to implement an SDK, but the primary concern here
is to ensure we have the right wire package name before release.
2018-11-19 09:56:41 -08:00
Martin Atkins 1ff9a54020 build: Temporarily disable the website tests for Travis-CI
There are some known broken links right now because this repository has
been updated ahead of some necessary changes in the terraform-website
repository. We'll need to wait until the end of the v0.12 release process
to re-enable this because the terraform-website repository is currently
set up for the v0.11 content and will continue to be until v0.12.0 final
is ready for release.
2018-11-19 09:02:35 -08:00
Martin Atkins c133de863b build: Update most things for Go 1.11 modules
We're still using vendoring for now until we get _all_ of our tooling
updated, so the main idea here is to force use of the vendor directory
when running tests and building for development so we can quickly find
situations where we forget to run "go mod vendor".

We also setting GO111MODULE=off for installation of tools. Right now this
is the best way to install a tool in GOBIN without also interfering with
go.mod and go.sum, until a better pattern for managing tool dependencies
is devised by the Go team.

Finally, we run "go mod download" before launching "gox" in the main
build process, to prime the local module cache once so that the concurrent
"go build" processes won't race to populate it redundantly. This means
that we'll be producing final builds from the module cache rather than
from vendor as with everything else -- there's currently no way to tell
gox to use -mod=vendor -- but that should be fine in practice since
our go.sum file will ensure that we get the exact sources we expect in
the module cache before building.
2018-11-19 09:02:35 -08:00
Martin Atkins 5255e85238 "go fmt" fixups
Apparently my editor is still not reliably formatting on save, so I missed
a few formatting quirks in these files.
2018-11-19 09:02:35 -08:00
Martin Atkins 73c9521a04 command/e2etest: Temporarily disable tests that access network
Several of these tests rely on external services (e.g. Terraform Registry)
that have not yet been updated to support the needs of Terraform v0.12.0,
so for now we'll skip all of these tests and wait until those systems have
been updated.

This should be removed before Terraform v0.12.0 final to enable these
tests to be used as part of pre-release smoke testing.
2018-11-19 09:02:35 -08:00
Martin Atkins 300eceeb25 plans/planfile: fix TestRoundtrip
This was broken by an earlier change to verify the Terraform version
number when reading a state file. To fix it, we'll use our current version
in our constructed file which should then match when it's read back in.
2018-11-19 09:02:35 -08:00
Martin Atkins 6cb3b0f4cf states/statemgr: Local locks meta is near output path, not input path
This was a mistake while adapting this code from the old state.LocalState.
Since the lock is held on the output file (s.path) the metadata should
live adjacent to that rather than being built from the read path
(s.readPath) that is used only as the initial snapshot on first
instantiation.

This also includes more logging, continuing the trend of other recent
commits in these files. The local state behavior is sufficiently complex
that these trace logs are a great help in debugging issues such as this
one with the wrong files being used or actions being taken in the wrong
order.
2018-11-19 09:02:35 -08:00