Commit Graph

137 Commits

Author SHA1 Message Date
Mitchell Hashimoto 39542898b0
helper/schema: mark diff as forcenew if element is computed
Fixes #10125

If the elements are computed and the field is ForceNew, then we should
mark the computed count as potentially forcing a new operation.

Example, assuming `groups` forces new...

**Step 1:**

    groups = ["1", "2", "3"]

At this point, the resource isn't create, so this should result in a
diff like:

    CREATE resource:
      groups: "" => ["1", "2", "3"]

**Step 2:**

    groups = ["${computedvar}"]

The OLD behavior was:

    UPDATE resource
      groups.#: "3" => "computed"

This would cause a diff mismatch because if `${computedvar}` was
different then it should force new. The NEW behavior is:

    DESTROY/CREATE resource:
      groups.#: "3" => "computed" (forces new)
2016-11-15 11:02:14 -08:00
Mitchell Hashimoto e45debe0e5
helper/schema: only mark "ForceNew" on resources that cause the ForceNew
Fixes #2748

This changes the diff to only mark "forces new resource" on the fields
that actually caused the new resource, not every field that changed.
This makes diffs much more accurate.

I'd like to request a review but I'm going to defer merging until
Terraform 0.8. Changes like this are very possible to cause "diffs
didn't match" errors and I want some real world testing in a beta before
we hit prod with this.
2016-11-08 15:49:28 -08:00
Mitchell Hashimoto 65b17ccd06
helper/schema: allow ConflictsWith and Computed Optional fields 2016-11-02 22:24:34 -07:00
Mitchell Hashimoto 2d84582881 Merge pull request #9699 from hashicorp/b-removed-forcenew
helper/schema: removed optional items force new
2016-10-31 13:24:36 -07:00
Mitchell Hashimoto 5489d8c549
helper/schema: removed optional items force new
Fixes #5138

If an item is optional and is removed completely from the configuration,
it should still trigger a destroy/create if the field itself was marked
as "ForceNew".

See the example in #5138.
2016-10-28 18:45:12 -04:00
Mitchell Hashimoto 95d37ea79c
helper/schema,terraform: handle computed primtives in diffs
Fixes #3309

There are two primary changes, one to how helper/schema creates diffs
and one to how Terraform compares diffs. Both require careful
understanding.

== 1. helper/schema Changes

helper/schema, given any primitive field (string, int, bool, etc.)
_used to_ create a basic diff when given a computed new value (i.e. from
an unkown interpolation). This would put in the plan that the old value
is whatever the old value was, and the new value was the actual
interpolation. For example, from #3309, the diff showed the following:

```
~ module.test.aws_eip.test-instance.0
    instance: "<INSTANCE ID>" => "${element(aws_instance.test-instance.*.id, count.index)}"
```

Then, when running `apply`, the diff would be realized and you would get
a diff mismatch error because it would realize the final value is the
same and remove it from the diff.

**The change:** `helper/schema` now marks unknown primitive values with
`NewComputed` set to true. Semantically this is correct for the diff to
have this information.

== 2. Terraform Diff.Same Changes

Next, the way Terraform compares diffs needed to be updated

Specifically, the case where the diff from the plan had a NewComputed
primitive and the diff from the apply _no longer has that value_. This
is possible if the computed value ended up being the same as the old
value. This is allowed to pass through.

Together, these fix #3309.
2016-10-25 22:36:59 -04:00
stack72 5a537cdbf9
helper/schema: Adding of MinItems as a validation to Lists and Maps
This is required for the times when the configuration cannot have an
empty configuration. An example would be in AzureRM, when you create a
LoadBalancer with a configuration, you can delete *all* but 1 of these
configurations
2016-10-04 18:57:58 +01:00
James Nugent e0226c9039 core: Check for attrV being nil before dereference
This can be an issue with unset computed fields.

Fixes #8815.
2016-09-14 09:51:15 +01:00
James Nugent 85ec09111b helper/schema: Add diff suppression callback
This commit adds a new callback, DiffSuppressFunc, to  the schema.Schema
structure. If set for a given schema, a callback to the user-supplied
function will be made for each attribute for which the default
type-based diff mechanism produces an attribute diff. Returning `true`
from the callback will suppress the diff (i.e. pretend there was no
diff), and returning false will retain it as part of the plan.

There are a number of motivating examples for this - one of which is
included as an example:

1. On SSH public keys, trailing whitespace does not matter in many
   cases - and in some cases it is added by provider APIs. For
   digitalocean_ssh_key resources we previously had a StateFunc that
   trimmed the whitespace - we now have a DiffSuppressFunc which
   verifies whether the trimmed strings are equivalent.

2. IAM policy equivalence for AWS. A good proportion of AWS issues
   relate to IAM policies which have been "normalized" (used loosely)
   by the IAM API endpoints. This can make the JSON strings differ
   from those generated by iam_policy_document resources or template
   files, even though the semantics are the same (for example,
   reordering of `bucket-prefix/` and `bucket-prefix/*` in an S3
   bucket policy. DiffSupressFunc can be used to test for semantic
   equivalence rather than pure text equivalence, but without having to
   deal with the complexity associated with a full "provider-land" diff
   implementation without helper/schema.
2016-08-31 19:13:53 -05:00
James Nugent 074545e536 core: Use .% instead of .# for maps in state
The flatmapped representation of state prior to this commit encoded maps
and lists (and therefore by extension, sets) with a key corresponding to
the number of elements, or the unknown variable indicator under a .# key
and then individual items. For example, the list ["a", "b", "c"] would
have been encoded as:

    listname.# = 3
    listname.0 = "a"
    listname.1 = "b"
    listname.2 = "c"

And the map {"key1": "value1", "key2", "value2"} would have been encoded
as:

    mapname.# = 2
    mapname.key1 = "value1"
    mapname.key2 = "value2"

Sets use the hash code as the key - for example a set with a (fictional)
hashcode calculation may look like:

    setname.# = 2
    setname.12312512 = "value1"
    setname.56345233 = "value2"

Prior to the work done to extend the type system, this was sufficient
since the internal representation of these was effectively the same.
However, following the separation of maps and lists into distinct
first-class types, this encoding presents a problem: given a state file,
it is impossible to tell the encoding of an empty list and an empty map
apart. This presents problems for the type checker during interpolation,
as many interpolation functions will operate on only one of these two
structures.

This commit therefore changes the representation in state of maps to use
a "%" as the key for the number of elements. Consequently the map above
will now be encoded as:

    mapname.% = 2
    mapname.key1 = "value1"
    mapname.key2 = "value2"

This has the effect of an empty list (or set) now being encoded as:

    listname.# = 0

And an empty map now being encoded as:

    mapname.% = 0

Therefore we can eliminate some nasty guessing logic from the resource
variable supplier for interpolation, at the cost of having to migrate
state up front (to follow in a subsequent commit).

In order to reduce the number of potential situations in which resources
would be "forced new", we continue to accept "#" as the count key when
reading maps via helper/schema. There is no situation under which we can
allow "#" as an actual map key in any case, as it would not be
distinguishable from a list or set in state.
2016-06-09 10:49:42 +01:00
James Nugent 91587a49f3 core: Remove unnecessary debug logging
Some unnecessary debug logging was introduced in 7b6df27e4, this commit
removes it so as not to clutter logs.
2016-06-08 18:38:41 +01:00
Chris Marchesi 9d7fb89114 core: Adding Sensitive attribute to resource schema
This an effort to address hashicorp/terraform#516.

Adding the Sensitive attribute to the resource schema, opening up the
ability for resource maintainers to mark some fields as sensitive.
Sensitive fields are hidden in the output, and, possibly in the future,
could be encrypted.
2016-05-29 22:18:44 -07:00
Sander van Harmelen 8560f50cbc
Change taint behaviour to act as a normal resource
This means it’s shown correctly in a plan and takes into account any
actions that are dependant on the tainted resource and, vice verse, any
actions that the tainted resource depends on.

So this changes the behaviour from saying this resource is tainted so
just forget about it and make sure it gets deleted in the background,
to saying I want that resource to be recreated (taking into account the
existing resource and it’s place in the graph).
2016-05-26 19:55:26 -05:00
Martin Atkins 6a468dcd83 helper/schema: Resource can be writable or not
In the "schema" layer a Resource is just any "thing" that has a schema
and supports some or all of the CRUD operations. Data sources introduce
a new use of Resource to represent read-only resources, which require
some different InternalValidate logic.
2016-05-14 08:26:36 -07:00
James Nugent 7b6df27e4a helper/schema: Read native maps from configuration
This adds a test and the support necessary to read from native maps
passed as variables via interpolation - for example:

```
resource ...... {
     mapValue = "${var.map}"
}
```

We also add support for interpolating maps from the flat-mapped resource
config, which is necessary to support assignment of computed maps, which
is now valid.

Unfortunately there is no good way to distinguish between a list and a
map in the flatmap. In lieu of changing that representation (which is
risky), we assume that if all the keys are numeric, this is intended to
be a list, and if not it is intended to be a map. This does preclude
maps which have purely numeric keys, which should be noted as a
backwards compatibility concern.
2016-05-10 14:49:14 -04:00
Paul Hinze b4df304b47
helper/schema: Normalize bools to "true"/"false" in diffs
For a long time now, the diff logic has relied on the behavior of
`mapstructure.WeakDecode` to determine how various primitives are
converted into strings.  The `schema.DiffString` function is used for
all primitive field types: TypeBool, TypeInt, TypeFloat, and TypeString.

The `mapstructure` library's string representation of booleans is "0"
and "1", which differs from `strconv.FormatBool`'s "false" and "true"
(which is used in writing out boolean fields to the state).

Because of this difference, diffs have long had the potential for
cosmetically odd but semantically neutral output like:

    "true" => "1"
    "false" => "0"

So long as `mapstructure.Decode` or `strconv.ParseBool` are used to
interpret these strings, there's no functional problem.

We had our first clear functional problem with #6005 and friends, where
users noticed diffs like the above showing up unexpectedly and causing
troubles when `ignore_changes` was in play.

This particular bug occurs down in Terraform core's EvalIgnoreChanges.
There, the diff is modified to account for ignored attributes, and
special logic attempts to handle properly the situation where the
ignored attribute was going to trigger a resource replacement. That
logic relies on the string representations of the Old and New fields in
the diff to be the same so that it filters properly.

So therefore, we now get a bug when a diff includes `Old: "0", New:
"false"` since the strings do not match, and `ignore_changes` is not
properly handled.

Here, we introduce `TypeBool`-specific normalizing into `finalizeDiff`.
I spiked out a full `diffBool` function, but figuring out which pieces
of `diffString` to duplicate there got hairy. This seemed like a simpler
and more direct solution.

Fixes #6005 (and potentially others!)
2016-05-05 09:00:58 -05:00
Chris Marchesi 8c5354b7dc Add MaxItems attribute to Schema
* MaxItems defines a maximum amount of items that can exist within a
   TypeSet or TypeList. Specific use cases would be if a TypeSet is being
   used to wrap a complex structure, however more than one instance would
   cause instability.
2016-02-23 16:41:32 -08:00
Paul Hinze 99244c5597 helper/schema: skip provider input for deprecated fields
There's no reason that a field that's been deprecated should ever
prompt.

fixes #4033
2015-12-07 11:28:45 -06:00
Paul Hinze f1e7cec566 Merge pull request #3992 from svanharmelen/f-change-sets
core: change set internals and make (extreme) performance improvements
2015-12-04 09:03:43 -06:00
Sander van Harmelen ef4726bd50 Change Set internals and make (extreme) performance improvements
Changing the Set internals makes a lot of sense as it saves doing
conversions in multiple places and gives a central place to alter
the key when a item is computed.

This will have no side effects other then that the ordering is now
based on strings instead on integers, so the order will be different.
This will however have no effect on existing configs as these will
use the individual codes/keys and not the ordering to determine if
there is a diff or not.

Lastly (but I think also most importantly) there is a fix in this PR
that makes diffing sets extremely more performand. Before a full diff
required reading the complete Set for every single parameter/attribute
you wanted to diff, while now it only gets that specific parameter.

We have a use case where we have a Set that has 18 parameters and the
set consist of about 600 items (don't ask 😉). So when doing a diff
it would take 100% CPU of all cores and stay that way for almost an
hour before being able to complete the diff.

Debugging this we learned that for retrieving every single parameter
it made over 52.000 calls to `func (c *ResourceConfig) get(..)`. In
this function a slice is created and used only for the duration of the
call, so the time needed to create all needed slices and on the other
hand the time the garbage collector needed to clean them up again caused
the system to cripple itself. Next to that there are also some expensive
reflect calls in this function which also claimed a fair amount of CPU
time.

After this fix the number of calls needed to get a single parameter
dropped from 52.000+ to only 2! 😃
2015-11-22 14:21:28 +01:00
Paul Hinze c7dc1c10a3 helper/schema: skip StateFunc when value is nil
This takes the nil checking burden off of StateFunc.

fixes #3586, see that issue for further discussion
2015-11-20 14:07:18 -06:00
Martin Atkins a67182543c Nicer error when list/map assigned to string argument.
Previous this would return the following sort of error:
expected type 'string', got unconvertible type '[]interface {}'

This is the raw error returned by the underlying mapstructure library.
This is not a helpful error message for anyone who doesn't know Go's
type system, and it exposes Terraform's internals to the UI.

Instead we'll catch these cases before we try to use mapstructure and
return a more straightforward message.

By checking the type before the IsComputed exception this also avoids
a crash caused when the assigned value is a computed list. Otherwise
the list of interpolations is allowed through here and then crashes later
during Diff when the value is not a primitive as expected.
2015-10-22 21:16:02 -07:00
Paul Hinze 2a179d1065 helper/schema: ValidateFunc support for maps 2015-10-14 15:10:22 -05:00
Panagiotis Moustafellos e4845f75cc removed extra parentheses 2015-10-08 15:48:04 +03:00
Martin Atkins cc8e8a55de helper/schema: Default hashing function for sets
A common issue with new resource implementations is not considering parts
of a complex structure that's used inside a set, which causes quirky
behavior.

The schema helper has enough information to provide a default reasonable
implementation of a set function that includes all non-computed attributes
in a deterministic way. Here we implement such a function and use it
when no explicit hashing function is provided.

In order to achieve this we encapsulate the construction of the zero
value for a schema in a new method schema.ZeroValue, which allows us to
put the fallback logic to the new default function in a single spot.
It is no longer valid to use &Set{F: schema.Set} and all uses of that
construct should be replaced with schema.ZeroValue().(*Set) .
2015-10-03 18:10:47 -07:00
Anthony Scalisi 198e1a5186 remove various typos 2015-09-11 11:56:20 -07:00
Mitchell Hashimoto 0100d4139b helper/schema: clean up style 2015-06-25 22:01:54 -07:00
Mitchell Hashimoto 6e509aedcb helper/schema: diff should include removed set items [GH-1823] 2015-06-25 21:52:49 -07:00
Radek Simko 6fdbca8e58 Merge pull request #2466 from TimeIncOSS/f-schema-field-name-validate
schema: Add field name to ValidateFunc
2015-06-24 18:52:53 +01:00
Radek Simko 92db4802b6 schema: Add field name to ValidateFunc 2015-06-24 18:22:12 +01:00
Mitchell Hashimoto 4f391902a0 helper/schema: validate objects are objects [GH-2166] 2015-06-23 16:39:02 -07:00
Paul Hinze a4912cc51f helper/schema: limit ValidateFunc to primitives for now
I couldn't see a simple path get this working for Maps, Sets,
and Lists, so lets land it as a primitive-only schema feature.

I think validation on primitives comprises 80% of the use cases anyways.
2015-06-11 07:06:30 -05:00
Paul Hinze 49352db26f helper/schema: skip ValidateFunc on other errors
Guarantees that the `interface{}` arg to ValidateFunc is the proper
type, allowing implementations to be simpler.

Finish the docstring on `ValidateFunc` to call this out.

/cc @mitchellh
2015-06-08 08:55:45 -05:00
Paul Hinze 61fee6735d helper/schema: ValidateFunc
Allows provider authors to implement arbitrary per-field validation
warnings or errors.
2015-06-08 08:47:41 -05:00
Sam Boyer b82bd0c280 Condense switch fallthroughs into expr lists 2015-05-26 21:52:36 -04:00
Paul Hinze 1e3d1b07e6 helper/schema: validate ConflictsWith against top-level
The runtime impl of ConfictsWith uses Resource.Get(), which makes it
work with any other attribute of the resource - the InternalValidate was
only checking against the local schemaMap though, preventing subResource
from using ConflictsWith properly.

It's a lot of wiring and it's a bit ugly, but it's not runtime code, so
I'm a bit less concerned about that aspect.

This should take care of the problem mentioned in #1909
2015-05-12 09:45:15 -05:00
Mitchell Hashimoto 707aa36aec helper/schema: only use ~ on first char of code 2015-04-23 17:20:54 +02:00
Mitchell Hashimoto 1f46bc1926 helper/schema: validate unknown fields with computed values [GH-1507] 2015-04-22 12:52:26 +02:00
Mitchell Hashimoto 3176e5b44a Merge pull request #1595 from TimeIncOSS/aws-account-protection
aws: Allow defining blacklist/whitelist of account IDs
2015-04-22 08:08:01 +02:00
Mitchell Hashimoto 54e3e6104e Merge pull request #1594 from TimeIncOSS/schema-conflicts-with
helper/schema: add schema.ConflictsWith[]
2015-04-22 08:04:49 +02:00
Mitchell Hashimoto 51951d68f4 helper/schema: change diff logic around maps to fix case #57 and #44 2015-04-21 22:13:03 +02:00
Radek Simko 150fd00932 AWS account ID protection added 2015-04-20 12:07:39 +01:00
Radek Simko 34f48b3e06 Add schema.ConflictsWith[]
- this will allow defining logically conflicting attributes
2015-04-20 12:07:34 +01:00
Radek Simko e0df74c863 Add schema.ConflictsWith[]
- this will allow defining logically conflicting attributes
2015-04-20 12:07:00 +01:00
Paul Hinze ef70c8cae5 helper/schema: allow Schema attrs to be Removed
Removed fields show a customizable error message to the user when they
are used in a Terraform config. This is a tool that provider authors can
use for user feedback as they evolve their Schemas.

refs #957
2015-03-05 15:33:56 -06:00
Paul Hinze 888f16d2d3 helper/schema: allow Schema attrs to be Deprecated
Deprecated fields show a customizable warning message to the user when
they are used in a Terraform config. This is a tool that provider
authors can use for user feedback as they evolve their Schemas.

fixes #957
2015-03-05 15:16:50 -06:00
Mitchell Hashimoto 17680bb7ff helper/schema: some more test cases, revert some weird behavior from
dbfb95fcd5

I don't know why that behavior was in there, but it was breaking a lot
of existing Terraform states. Let's circle back on it.
2015-02-18 12:54:46 -08:00
Mitchell Hashimoto 659a77c6ae helper/schema: validate subresources more effectively 2015-02-18 09:41:55 -08:00
Mitchell Hashimoto fa7f496bef helper/schema: zero value of a set should be empty 2015-02-17 16:58:47 -08:00
Mitchell Hashimoto 5c06cc386a helper/schema: empty map values should show up in diff [GH-968] 2015-02-17 15:22:45 -08:00