Commit Graph

20486 Commits

Author SHA1 Message Date
James Bardin bcb11f6d89 have the consul client manage the lock session
When a consul lock is lost, there is a possibility that the associated
session is still active. Most commonly, the long request to watch the
lock key may error out, while the session is continually refreshed at a
rate of TTL/2.

First have the lock monitor retry the lock internally for at least 10
seconds (5 attempts with the default 2 second wait time). In most cases
this will reconnect on the first try, keeping the lock channel open.

If the consul lock can't recover itself, then cancel the session as soon
as possible (terminating the PreiodicRenew will call Session.Destroy),
and start over. In the worse case, the consul agents were split, and the
session still exists on the leader so we may need to wait for the old
session TTL, plus the LockWait time to renew the lock.

We use a Context for the cancellation channels here, because that
removes the need to worry about double-closes and nil channels. It
requires an awkward adapter goroutine for now to convert the Done()
`<-chan` to a `chan` for PeriodicRenew, but makes the rest of the code
safer in the long run.
2017-07-14 14:42:42 -04:00
James Bardin 193d4b868c backend state tests must honor lineage
Remote state implementations may initialize a lineage when creating a
new named state (i.e. "workspace"). The tests were ignoring that initial
lineage to write a new state to the backend.
2017-07-14 13:50:26 -04:00
Nandor Kracser f6c77339f2 Allow non-AWS S3 backends
This commit makes sts, metadata and other AWS related API calls optional, thus the backend initialization will not send non-AWS API tokens to AWS APIs
2017-07-14 13:08:47 +02:00
James Nugent aa1e5e996a README: Add link to provider repos to "Developing" (#15546)
This commit adds a link to the GitHub organization which contains the
formerly-built-in providers, and modifies the `plugin-dev` target
documentation to use a provider which is unlikely to be moved out of the
core repository.
2017-07-13 20:59:20 +03:00
Jake Champlin dd056ccf38 Merge pull request #15547 from hashicorp/b-fix-travis
fix travis for unit tests
2017-07-13 10:38:11 -07:00
Jake Champlin 3d396e108a
fix travis for unit tests 2017-07-13 13:19:24 -04:00
Lars Lehtonen 8501d83e6c
Fix swallowed errors in command package. 2017-07-11 08:01:02 -07:00
Radek Simko a12daf5aba Update CHANGELOG.md 2017-07-10 21:52:49 -07:00
Radek Simko 07cbd54fbc Actively disallow reserved field names in schema (#15522) 2017-07-10 21:51:55 -07:00
James Bardin a4ee13b8c2 Merge pull request #15506 from hashicorp/jbardin/helper-testing
Automatically insert providers for import acceptance tests
2017-07-07 17:06:46 -04:00
James Bardin 60ea42cd1c Add testProviderConfig to import tests
Provider import tests previously didn't have to supply a config, but
terraform now requires the provider to be declared for discovery.

testProviderConfig returns a stub config with provider blocks based
on the TestCase Providers. This allows basic import tests in providers
to remain unchanged.
2017-07-07 16:16:35 -04:00
James Bardin 8e2ee53ed3 Merge pull request #15501 from hashicorp/jbardin/race
Make sure shadow.closeWalker doesn't copy Mutexes
2017-07-07 12:39:46 -04:00
James Bardin 657932261b Make sure shadow.closeWalker doesn't copy Mutexes
The Close methods on shadow.Values require pointer receivers because
they contain a sync.Mutex, but that value was being copied through
Value.Interface by the closeWalker.  Because reflectwalk passes the
struct fields to the StructField method as they are defined in the
struct, and they may have been read as a value, we can't immediately
call Interface() to check the method set without possibly copying the
internal mutex values. Use the Implements method to first check if we
need to call Interface, and if it's not, then we can check if the value
is addressable.

Because of this use of reflection, we can't vet for the copying of these
locks. The minimal amount of code in the Close method left us only with
a race detected within the mutex itself, which leads to a stacktrace
pointing to the runtime rather than our code.
2017-07-07 11:20:54 -04:00
James Bardin ac9abf579f Merge pull request #15497 from hashicorp/jbardin/race
Fix a couple races
2017-07-07 08:53:29 -04:00
James Bardin c66dd48b6e make shadow.Value a Locker
This way it's correctly handled by CopyStructure
2017-07-06 16:58:29 -04:00
James Bardin 583cc350a9 fix minor races in workspace tests
The improved err scanner loop in meta causes these to race. There's no
need to write back to the same commands struct, so just use a new
instance in each iteration.
2017-07-06 13:49:32 -04:00
Jake Champlin b7166905fa
v0.10.0-beta2 2017-07-06 12:43:01 -04:00
James Bardin 6ba049470e Merge pull request #15490 from hashicorp/jbardin/meta-process-order
meta process order
2017-07-06 11:41:49 -04:00
James Bardin f7f1e8e406 Sort arguments in Meta.process
Meta.process was relying on the system readdir to order the arguments,
but readdir doesn't guarantee any ordering. Read the directory contents
as a whole and sort them in place before adding the tfvars files.
2017-07-06 11:34:47 -04:00
James Bardin 0fdcf2c01e properly cleanup and print correct error messages
Some Meta tests were not cleaning up their temp directories.

The process test wasn't printing the correct arguments in the error
messages.
2017-07-06 11:33:32 -04:00
Paul Stack ccf27a631f Update CHANGELOG.md 2017-07-06 16:53:17 +03:00
Martin Atkins 0543574383 website: update upgrade-guide for plugin filename convention
This changed close to the release of beta1 to use underscores as the
separator and to use a lower-case "v" to avoid any issues on
case-insensitive filesystems.
2017-07-05 17:36:32 -07:00
Martin Atkins b16f94611d Update CHANGELOG.md 2017-07-05 17:34:23 -07:00
Martin Atkins 1f1f898695 Merge #13306: automatically load variables from .auto.tfvars files 2017-07-05 17:32:12 -07:00
Robert Liebowitz 4bf2269784 Add test case for autoloading tfvars 2017-07-05 17:24:17 -07:00
Robert Liebowitz e67ecb5ce2 Restructure Meta.process to logically group code 2017-07-05 17:24:17 -07:00
Robert Liebowitz 8d98fdecac Autoload only .auto.tfvars files 2017-07-05 17:24:17 -07:00
Robert Liebowitz 006744bfe0 Use all tfvars files in working directory
As a side effect, several commands that previously did not have a failure
state can now fail during meta-parameter processing.
2017-07-05 17:24:17 -07:00
Martin Atkins 29b2368fa0 Update CHANGELOG.md 2017-07-05 16:30:08 -07:00
Martin Atkins 39c4d6ab1f Merge #15424: Improve robustness of state persistence handling
Previously the APIs for state persistence and management had some problematic cases where we depended on hidden mutations of the state structure as side-effects of otherwise-innocent-looking operations, which was a frequent cause of accidental regressions due to faulty assumptions.

This new model attempts to isolate certain state mutations to just within the state managers, and makes the state managers work on separated snapshots of the state rather than on the "live" object to reduce the risk of race conditions.
2017-07-05 16:27:08 -07:00
Martin Atkins dfbbc89c66 Update CHANGELOG.md 2017-07-05 16:22:17 -07:00
Martin Atkins fee1197cf9 command: terraform state rm to require at least one argument
Due to how the state filter machinery works, passing no arguments is valid
and matches _all_ resources.

It is very unlikely that someone wants to remove everything from state, so
this ends up being a very dangerous default for the "terraform state rm"
command, and surprising for someone who perhaps runs it looking for the
usage information.

So we'll be pragmatic here and reject the no-arguments case for this
command, accepting that it makes the unlikely case of intentionally
deleting all resources harder in order to make it less likely that it
will happen _unintentionally_.

If someone does really want to remove all resources from the state, they
can provide an explicit empty string argument, but this isn't documented
because it's a weird case that doesn't seem worth mentioning.

This fixes #15283.
2017-07-05 16:19:32 -07:00
James Bardin fb397060eb add some Serial checks to apply and refresh tests 2017-07-05 18:18:28 -04:00
James Bardin 054716c397 use testStateRead helper in apply tests
We have a helper function that we hardly ever use.
TODO: convert the rest of the manual ReadState calls eventually.
2017-07-05 18:18:16 -04:00
James Bardin 501cbeaffe testState shouldn't rely on mods from WriteState
The state returned from the testState helper shouldn't rely on any
mutations caused by WriteState. The Init function (which is analogous to
NewState) shoudl set any required fields.
2017-07-05 17:47:05 -04:00
Martin Atkins d5ebad33a4 Update CHANGELOG.md 2017-07-05 14:37:04 -07:00
Mike Helmick 9d7fce2f69 command: "terraform workspace show" to print current workspace name
This command serves as an alternative to the human-oriented list of workspaces for scripting use-cases where it's useful to know the _current_ workspace name.
2017-07-05 14:35:46 -07:00
James Bardin fba5decae5 update TestState helper
In practice, States must all implement the full interface, so checking
for each method set only leaves gaps where tests could be skipped.
Change the helper to only accept a full state.State implementation.

Add some Lineage, Version, and TFVersion checks to TestState to avoid
regressions.

Compare the copy test against the immediate State returnedm rather than
our previous "current" state.

Check that the states round-trip and still marhsal identically via
MarshalEqual.
2017-07-05 17:18:12 -04:00
Martin Atkins 4d53eaa6df state: more robust handling of state Serial
Previously we relied on a constellation of coincidences for everything to
work out correctly with state serials. In particular, callers needed to
be very careful about mutating states (or not) because many different bits
of code shared pointers to the same objects.

Here we move to a model where all of the state managers always use
distinct instances of state, copied when WriteState is called. This means
that they are truly a snapshot of the state as it was at that call, even
if the caller goes on mutating the state that was passed.

We also adjust the handling of serials so that the state managers ignore
any serials in incoming states and instead just treat each Persist as
the next version after what was most recently Refreshed.

(An exception exists for when nothing has been refreshed, e.g. because
we are writing a state to a location for the first time. In that case
we _do_ trust the caller, since the given state is either a new state
or it's a copy of something we're migrating from elsewhere with its
state and lineage intact.)

The intent here is to allow the rest of Terraform to not worry about
serials and state identity, and instead just treat the state as a mutable
structure. We'll just snapshot it occasionally, when WriteState is called,
and deal with serials _only_ at persist time.

This is intended as a more robust version of #15423, which was a quick
hotfix to an issue that resulted from our previous slopping handling
of state serials but arguably makes the problem worse by depending on
an additional coincidental behavior of the local backend's apply
implementation.
2017-07-05 12:34:30 -07:00
Martin Atkins 909989acfa terraform-bundle tool for bundling Terraform with providers
Normally "terraform init" will download and install the plugins necessary
to work with a particular configuration, but sometimes Terraform is
deployed in a network that, for one reason or another, cannot access the
official plugin repository for automatic download.

terraform-bundle provides an alternative method, allowing the
auto-download process to be run out-of-band on a separate machine that
_does_ have access to the repository. The result is a zip file that can
be extracted onto the target system to install both the desired
Terraform version and a selection of providers, thus avoiding the need
for on-the-fly plugin installation.

This is provided as a separate tool from Terraform because it is not
something that most users will need. In the rare case where this is
needed, we will for the moment assume that users are able to build this
tool themselves. We may later release it in a pre-built form, if it proves
to be generally useful.

It uses the same API from the plugin/discovery package is is used by the
auto-install behavior in "terraform init", so plugin versions are resolved
in the same way. However, it's expected that several different Terraform
configurations will run from the same bundle, so this tool allows the
bundle to include potentially many versions of the same provider and thus
allows each Terraform configuration to select from the available versions
in the bundle, avoiding the need to upgrade all configurations to new
provider versions in lockstep.
2017-07-05 10:02:05 -07:00
Martin Atkins 610fcb605e plugin/discovery: allow customizing the OS/arch for auto-install
Previously we forced only installing for the current GOOS and GOARCH. Now
we allow this to be optionally overridden, which allows building tools
that can, for example, populate a directory with plugins to run on a Linux
server while working on a Mac.
2017-07-05 10:02:05 -07:00
Radek Simko 9ee2fbaa2b Update CHANGELOG.md 2017-07-05 17:34:41 +01:00
Radek Simko 14614a5423 command/validate: Add flag to check that all variables are specified (#13872)
* command/validate: Add flag to check that all variables are specified

* Rename config-only to check-variables
2017-07-05 17:32:29 +01:00
Paul Stack 86a73701e9 Update CHANGELOG.md 2017-07-04 22:41:01 +03:00
Jake Champlin 41fc408ed0 Merge pull request #15462 from hashicorp/f-add-skip-verify-env-var
core: Skip provider checksum validation based on env var
2017-07-03 17:44:14 -04:00
Martin Atkins 6369d669a9 Update CHANGELOG.md 2017-07-03 11:09:54 -07:00
Martin Atkins 194bfe292b vendor: govendor fetch github.com/hashicorp/hil/...
This includes a fix to the scanner to correctly parse quoted strings that
end with escaped backslashes.
2017-07-03 11:08:09 -07:00
Jake Champlin 9944ea6886
core: Skip provider checksum validation based on env var
Skips checksum validation if the `TF_SKIP_PROVIDER_VERIFY` environment variable is set. Undocumented variable, as the primary goal is to significantly improve the local provider development workflow.
2017-07-03 13:59:13 -04:00
James Bardin 124b80398e make sure marshaled Meta fields are still equal
When the InstanceState.Meta fields are marshaled, numeric values may
change types. The timeout system currently inserts integer values, which
will be unmarshal as float64s.

To ensure that a state which has round-tripped through json is equal to
itself, compare the json representation of the Meta values.
2017-06-30 18:29:42 -04:00
Tom Fitzherbert e6ddd62583 Ensures that subnets are created in the desired AZ (#15443) 2017-06-30 16:00:47 +03:00