Commit Graph

417 Commits

Author SHA1 Message Date
Paul Hinze 4bd4e18def core: use same logging setup for acctests
We weren't doing any log setup for acceptance tests, which made it
difficult to wrangle log output in CI.

This moves the log setup functions we use in `main` over into a helper
package so we can use them for acceptance tests as well.

This means that acceptance tests will by default be a _lot_ quieter,
only printing out actual test output. Setting `TF_LOG=trace` will
restore the full prior noise level.

Only minor behavior change is to make `ioutil.Discard` the default
return value rather than a `nil` that needs to be checked for.
2015-12-08 17:50:36 -06:00
Paul Hinze 99244c5597 helper/schema: skip provider input for deprecated fields
There's no reason that a field that's been deprecated should ever
prompt.

fixes #4033
2015-12-07 11:28:45 -06:00
Paul Hinze f1e7cec566 Merge pull request #3992 from svanharmelen/f-change-sets
core: change set internals and make (extreme) performance improvements
2015-12-04 09:03:43 -06:00
Sander van Harmelen ef4726bd50 Change Set internals and make (extreme) performance improvements
Changing the Set internals makes a lot of sense as it saves doing
conversions in multiple places and gives a central place to alter
the key when a item is computed.

This will have no side effects other then that the ordering is now
based on strings instead on integers, so the order will be different.
This will however have no effect on existing configs as these will
use the individual codes/keys and not the ordering to determine if
there is a diff or not.

Lastly (but I think also most importantly) there is a fix in this PR
that makes diffing sets extremely more performand. Before a full diff
required reading the complete Set for every single parameter/attribute
you wanted to diff, while now it only gets that specific parameter.

We have a use case where we have a Set that has 18 parameters and the
set consist of about 600 items (don't ask 😉). So when doing a diff
it would take 100% CPU of all cores and stay that way for almost an
hour before being able to complete the diff.

Debugging this we learned that for retrieving every single parameter
it made over 52.000 calls to `func (c *ResourceConfig) get(..)`. In
this function a slice is created and used only for the duration of the
call, so the time needed to create all needed slices and on the other
hand the time the garbage collector needed to clean them up again caused
the system to cripple itself. Next to that there are also some expensive
reflect calls in this function which also claimed a fair amount of CPU
time.

After this fix the number of calls needed to get a single parameter
dropped from 52.000+ to only 2! 😃
2015-11-22 14:21:28 +01:00
Paul Hinze c7dc1c10a3 helper/schema: skip StateFunc when value is nil
This takes the nil checking burden off of StateFunc.

fixes #3586, see that issue for further discussion
2015-11-20 14:07:18 -06:00
Paul Hinze 938281024f helper/schema: name test cases w/ strings
I promised myself that next time I jumped in this file I'd fix this up.
Now we don't have to manually index the file with comments, we can just
add descriptive names to the test cases!
2015-11-20 13:51:34 -06:00
Paul Hinze 6b6b5a43c3 provider/aws: serialize SG rule access to fix race condition
Because `aws_security_group_rule` resources are an abstraction on top of
Security Groups, they must interact with the AWS Security Group APIs in
a pattern that often results in lots of parallel requests interacting
with the same security group.

We've found that this pattern can trigger race conditions resulting in
inconsistent behavior, including:

 * Rules that report as created but don't actually exist on AWS's side
 * Rules that show up in AWS but don't register as being created
   locally, resulting in follow up attempts to authorize the rule
   failing w/ Duplicate errors

Here, we introduce a per-SG mutex that must be held by any security
group before it is allowed to interact with AWS APIs. This protects the
space between `DescribeSecurityGroup` and `Authorize*` / `Revoke*`
calls, ensuring that no other rules interact with the SG during that
span.

The included test exposes the race by applying a security group with
lots of rules, which based on the dependency graph can all be handled in
parallel. This fails most of the time without the new locking behavior.

I've omitted the mutex from `Read`, since it is only called during the
Refresh walk when no changes are being made, meaning a bunch of parallel
`DescribeSecurityGroup` API calls should be consistent in that case.
2015-11-18 12:39:59 -06:00
Radek Simko 1e3cc7b33f helper: Remove url helper (moved to go-getter) 2015-11-14 08:21:18 +00:00
Paul Hinze 7ffa66d1a5 ssh: accept private key contents instead of path
We've been moving away from config fields expecting file paths that
Terraform will load, instead prefering fields that expect file contents,
leaning on `file()` to do loading from a path.

This helps with consistency and also flexibility - since this makes it
easier to shift sensitive files into environment variables.

Here we add a little helper package to manage the transitional period
for these fields where we support both behaviors.

Also included is the first of several fields being shifted over - SSH
private keys in provisioner connection config.

We're moving to new field names so the behavior is more intuitive, so
instead of `key_file` it's `private_key` now.

Additional field shifts will be included in follow up PRs so they can be
reviewed and discussed individually.
2015-11-12 14:59:14 -06:00
James Nugent f4c03ec2a6 Reflect new comment format in stringer.go
As of November 8th 2015, (4b07c5ce8a), the word "Code" is prepended to
the comments in Go source files generated by the stringer utility.
2015-11-09 11:38:51 -05:00
Martin Atkins a67182543c Nicer error when list/map assigned to string argument.
Previous this would return the following sort of error:
expected type 'string', got unconvertible type '[]interface {}'

This is the raw error returned by the underlying mapstructure library.
This is not a helpful error message for anyone who doesn't know Go's
type system, and it exposes Terraform's internals to the UI.

Instead we'll catch these cases before we try to use mapstructure and
return a more straightforward message.

By checking the type before the IsComputed exception this also avoids
a crash caused when the assigned value is a computed list. Otherwise
the list of interpolations is allowed through here and then crashes later
during Diff when the value is not a primitive as expected.
2015-10-22 21:16:02 -07:00
Mitchell Hashimoto 344e7c26b5 fix a bunch of tests from go-getter import 2015-10-15 13:48:58 -07:00
Paul Hinze 2a179d1065 helper/schema: ValidateFunc support for maps 2015-10-14 15:10:22 -05:00
Panagiotis Moustafellos e4845f75cc removed extra parentheses 2015-10-08 15:48:04 +03:00
Martin Atkins cc8e8a55de helper/schema: Default hashing function for sets
A common issue with new resource implementations is not considering parts
of a complex structure that's used inside a set, which causes quirky
behavior.

The schema helper has enough information to provide a default reasonable
implementation of a set function that includes all non-computed attributes
in a deterministic way. Here we implement such a function and use it
when no explicit hashing function is provided.

In order to achieve this we encapsulate the construction of the zero
value for a schema in a new method schema.ZeroValue, which allows us to
put the fallback logic to the new default function in a single spot.
It is no longer valid to use &Set{F: schema.Set} and all uses of that
construct should be replaced with schema.ZeroValue().(*Set) .
2015-10-03 18:10:47 -07:00
Martin Atkins 3fde993978 Merge #3336: Remove local multierror package.
Instead, use ``github.com/hashicorp/go-multierror``.
2015-10-03 17:53:36 -07:00
Radek Simko 641b701830 schema: Make validation more strict 2015-10-03 14:29:19 -07:00
Sander van Harmelen 2ba8dc38fa Switch to go-multierror
It seems there are 4 locations left that use the `helper/multierror`
package, where the rest is TF settled on the `hashicorp/go-multierror`
package.

Functionally this doesn’t change anything, so I suggest to delete the
builtin version as it can only cause confusion (both packages have the
same name, but are still different types according to Go’s type system.
2015-09-27 18:58:48 -07:00
Anthony Scalisi 198e1a5186 remove various typos 2015-09-11 11:56:20 -07:00
Paul Hinze 7eb72e7a12 helper/schema: record schema version when destroy fails
This was just a missed exit from the resource.Apply function -
subsequent refreshes would add the SchemaVersion back into the state,
but having the state recorded once without the meta information can
cause problems with Atlas's remote state checksumming.
2015-08-03 15:53:15 -05:00
Clint f979fd7dee Merge pull request #2571 from TimeIncOSS/f-aws-autogenerated-elb-name
provider/aws: Allow ELB name to be generated
2015-07-21 15:52:36 -05:00
Radek Simko 9882cc59d8 aws: Add regression test for renaming ecs_cluster 2015-07-12 14:37:39 +01:00
Sander van Harmelen 4a8ef78d33 Fixes #2676 by prefixing all Windows commands
By prefixing them with `cmd /c` it will work with both `winner` and
`ssh` connection types.

This PR also reverts some bad stringer changes made in PR #2673
2015-07-10 12:56:27 +02:00
Sander van Harmelen 97fd4f5b7d Tweaking the tests 2015-07-09 21:29:27 +02:00
Paul Hinze 5c38456b05 core: don't prompt for variables with defaults
In `helper/schema` we already makes a distinction between `Default`
which is always applied and `InputDefault` which is displayed to the
user for an empty field.

But for variables we just have `Default` which is treated like
`InputDefault`. This changes it to _not_ prompt the user for a value
when the variable declaration includes a default.

Treating this as a UX bugfix and the "don't prompt for variables w/
defaults set" behavior as the originally expected behavior we were
failing to honor.

Added an already-passing test to verify and cover the `helper/schema`
behavior.

Perhaps down the road we can add a `input_default` attribute to
variables to allow similar behavior to `helper/schema` in variables, but
for now just sticking with the fix.

Fixes #2592
2015-07-02 10:40:30 -05:00
Radek Simko 70b7243dd6 helper: Add resource.PrefixedUniqueId 2015-06-30 12:54:54 +01:00
Mitchell Hashimoto 2f08a2bb15 Merge pull request #2507 from hashicorp/b-set-remove
helper/schema: diff should include removed set items [GH-1823]
2015-06-26 08:18:28 -07:00
Mitchell Hashimoto 0100d4139b helper/schema: clean up style 2015-06-25 22:01:54 -07:00
Mitchell Hashimoto 6e509aedcb helper/schema: diff should include removed set items [GH-1823] 2015-06-25 21:52:49 -07:00
Paul Hinze 7238b3b4af core: encapsulate representation of StringList
Now the only code that cares about how StringLists are represented lives
inside string_list.go

...which gives us the ability to change it! :)
2015-06-25 17:55:57 -05:00
Paul Hinze 10b3abf405 config: introduce StringList to abstract over list hack
This is the initial pure "all tests passing without a diff" stage. The
plan is to change the internal representation of StringList to include a
suffix delimiter, which will allow us to recognize empty and
single-element lists.
2015-06-25 17:55:56 -05:00
Radek Simko 6fdbca8e58 Merge pull request #2466 from TimeIncOSS/f-schema-field-name-validate
schema: Add field name to ValidateFunc
2015-06-24 18:52:53 +01:00
Mitchell Hashimoto 6b7c2bcb35 Merge pull request #2450 from hashicorp/b-schema-validate-type
helper/schema: validate objects are objects [GH-2166]
2015-06-24 10:35:26 -07:00
Mitchell Hashimoto 630646335f Merge pull request #2451 from hashicorp/b-provider-validate
helper/schema: internal validate as part of provider validation
2015-06-24 10:28:49 -07:00
Radek Simko 92db4802b6 schema: Add field name to ValidateFunc 2015-06-24 18:22:12 +01:00
Mitchell Hashimoto 4e7fcd4f42 helper/schema: test that validatefunc is not called with computed vals 2015-06-23 22:10:46 -07:00
Mitchell Hashimoto e36597cad7 helper/schema: internal validate as part of provider validation
[GH-1291]
2015-06-23 16:52:04 -07:00
Mitchell Hashimoto 4f391902a0 helper/schema: validate objects are objects [GH-2166] 2015-06-23 16:39:02 -07:00
Paul Hinze aa8cf572a8 Merge pull request #2265 from hashicorp/f-schema-validate-field
Support arbitrary per-field schema validation
2015-06-22 18:01:54 -05:00
Paul Hinze 385b17d679 provider/template: don't error when rendering fails in Exists
The Exists function can run in a context where the contents of the
template have changed, but it uses the old set of variables from the
state. This means that when the set of variables changes, rendering will
fail in Exists. This was returning an error, but really it just needs to
be treated as a scenario where the template needs re-rendering.

fixes #2344 and possibly a few other template issues floating around
2015-06-17 15:33:07 -05:00
Paul Hinze a4912cc51f helper/schema: limit ValidateFunc to primitives for now
I couldn't see a simple path get this working for Maps, Sets,
and Lists, so lets land it as a primitive-only schema feature.

I think validation on primitives comprises 80% of the use cases anyways.
2015-06-11 07:06:30 -05:00
Paul Hinze 49352db26f helper/schema: skip ValidateFunc on other errors
Guarantees that the `interface{}` arg to ValidateFunc is the proper
type, allowing implementations to be simpler.

Finish the docstring on `ValidateFunc` to call this out.

/cc @mitchellh
2015-06-08 08:55:45 -05:00
Paul Hinze 61fee6735d helper/schema: ValidateFunc
Allows provider authors to implement arbitrary per-field validation
warnings or errors.
2015-06-08 08:47:41 -05:00
Svend Sorensen 943bf3c00a Use name of function in comment string
Name of function is Difference, not Differences.
2015-06-04 13:03:01 -07:00
Clint Shryock 78e7519efa Updates from go generate 2015-06-03 08:37:57 -05:00
Sam Boyer b82bd0c280 Condense switch fallthroughs into expr lists 2015-05-26 21:52:36 -04:00
Mitchell Hashimoto dd24ed4b76 helper/schema: blank ID refresh doesn't exist [GH-1905] 2015-05-13 20:15:13 -07:00
Justin Campbell bb14bfa657 helper/schema: call InternalValidate w/ schemaMap{} 2015-05-12 11:01:08 -04:00
Paul Hinze 1e3d1b07e6 helper/schema: validate ConflictsWith against top-level
The runtime impl of ConfictsWith uses Resource.Get(), which makes it
work with any other attribute of the resource - the InternalValidate was
only checking against the local schemaMap though, preventing subResource
from using ConflictsWith properly.

It's a lot of wiring and it's a bit ugly, but it's not runtime code, so
I'm a bit less concerned about that aspect.

This should take care of the problem mentioned in #1909
2015-05-12 09:45:15 -05:00
Paul Hinze a96a3372c6 provider/template: don't diff when there's no diff
This reworks the template lifecycle a bit such that we get nicer diff
behavior.

First, we tick ForceNew on for both filename and vars, so that the diff
indicates that the template will be "replaced" on change. This is mostly
cosmetic, but it also tracks conceptually with the fact that the
identifier we use is a hash of the contents, so any change essentially
makes a "new resource".

Second, we change the Exists implementation to only return `false` when
there has been a change in the rendered template. This lets descendent
resources see the computed value changing so that they'll properly
trigger in the plan.

Fixes #1898
Refs #1866 (but does not fix, there's another deeper issue there)
2015-05-11 10:38:19 -05:00