Commit Graph

507 Commits

Author SHA1 Message Date
clint shryock 7d71b8cc3c helper and terraform interpolate test update 2016-06-10 10:07:17 -05:00
James Bardin 2c7b702d1f Merge pull request #7091 from hashicorp/jbardin/serialize
Serialization for hash panics on TypeMap
2016-06-09 16:16:41 -04:00
James Bardin bab031aac5 Add test for TypeMap in a Schema 2016-06-09 16:00:33 -04:00
James Bardin d8fbaa7924 Serialization for hash panics on TypeMap
The serializeCollectionMemberForHash helper can't be called for the
MapType values, because MapType doesn't have a schema.Elem. Instead, we
can write the key/value pairs directly to the buffer. This still doesn't
allow for nested maps or lists, but we need to define that use case
before committing to it here.
2016-06-09 13:37:58 -04:00
James Nugent 074545e536 core: Use .% instead of .# for maps in state
The flatmapped representation of state prior to this commit encoded maps
and lists (and therefore by extension, sets) with a key corresponding to
the number of elements, or the unknown variable indicator under a .# key
and then individual items. For example, the list ["a", "b", "c"] would
have been encoded as:

    listname.# = 3
    listname.0 = "a"
    listname.1 = "b"
    listname.2 = "c"

And the map {"key1": "value1", "key2", "value2"} would have been encoded
as:

    mapname.# = 2
    mapname.key1 = "value1"
    mapname.key2 = "value2"

Sets use the hash code as the key - for example a set with a (fictional)
hashcode calculation may look like:

    setname.# = 2
    setname.12312512 = "value1"
    setname.56345233 = "value2"

Prior to the work done to extend the type system, this was sufficient
since the internal representation of these was effectively the same.
However, following the separation of maps and lists into distinct
first-class types, this encoding presents a problem: given a state file,
it is impossible to tell the encoding of an empty list and an empty map
apart. This presents problems for the type checker during interpolation,
as many interpolation functions will operate on only one of these two
structures.

This commit therefore changes the representation in state of maps to use
a "%" as the key for the number of elements. Consequently the map above
will now be encoded as:

    mapname.% = 2
    mapname.key1 = "value1"
    mapname.key2 = "value2"

This has the effect of an empty list (or set) now being encoded as:

    listname.# = 0

And an empty map now being encoded as:

    mapname.% = 0

Therefore we can eliminate some nasty guessing logic from the resource
variable supplier for interpolation, at the cost of having to migrate
state up front (to follow in a subsequent commit).

In order to reduce the number of potential situations in which resources
would be "forced new", we continue to accept "#" as the count key when
reading maps via helper/schema. There is no situation under which we can
allow "#" as an actual map key in any case, as it would not be
distinguishable from a list or set in state.
2016-06-09 10:49:42 +01:00
James Nugent cb9ef298f3 core: Defeat backward compatibilty in mapstructure
The mapstructure library has a regrettable backward compatibility
concern whereby a WeakDecode of []interface{}{} into a target of
map[string]interface{} yields an empty map rather than an error. One
possibility is to switch to using Decode instead of WeakDecode, but this
loses the nice handling of type conversion, requiring a large volume of
code to be added to Terraform or HIL in order to retain that behaviour.

Instead we add a DecodeHook to our usage of the mapstructure library
which checks for decoding []interface{}{} or []string{} into a map and
returns an error instead.

This has the effect of defeating the code added to retain backwards
compatibility in mapstructure, giving us the correct (for our
circumstances) behaviour of Decode for empty structures and the type
conversion of WeakDecode.

The code is identical to that in the HIL library, and packaged into a
helper.
2016-06-08 18:38:41 +01:00
James Nugent 91587a49f3 core: Remove unnecessary debug logging
Some unnecessary debug logging was introduced in 7b6df27e4, this commit
removes it so as not to clutter logs.
2016-06-08 18:38:41 +01:00
Radek Simko 1ea5cff9b3 Merge pull request #6946 from TimeIncOSS/f-aws-s3-object-data-source
provider/aws: Add aws_s3_bucket_object data source
2016-06-02 07:04:46 +01:00
Radek Simko bf8931b1ab provider/aws: Mark Lambda function as gone when it's gone (#6924)
* helper/error: Introduce TimeoutError & UnexpectedStateError

* provider/aws: Mark Lambda function as gone when it's gone
2016-06-02 00:50:43 +01:00
Radek Simko 84ab00d92e helper/resource: Add TestStep.PreventPostDestroyRefresh
- This is to allow easier testing of data sources which read data from resources created in the same scope
2016-06-01 19:13:00 +01:00
Chris Marchesi 9d7fb89114 core: Adding Sensitive attribute to resource schema
This an effort to address hashicorp/terraform#516.

Adding the Sensitive attribute to the resource schema, opening up the
ability for resource maintainers to mark some fields as sensitive.
Sensitive fields are hidden in the output, and, possibly in the future,
could be encrypted.
2016-05-29 22:18:44 -07:00
Sander van Harmelen 8560f50cbc
Change taint behaviour to act as a normal resource
This means it’s shown correctly in a plan and takes into account any
actions that are dependant on the tainted resource and, vice verse, any
actions that the tainted resource depends on.

So this changes the behaviour from saying this resource is tainted so
just forget about it and make sure it gets deleted in the background,
to saying I want that resource to be recreated (taking into account the
existing resource and it’s place in the graph).
2016-05-26 19:55:26 -05:00
Martin Atkins 031b561ef0 helper/resource: ignore data resource diffs during destroy
When testing destroy, the test harness calls Refresh followed by Plan,
with the expectation that the resulting diff will be empty.

Data resources challenge this expectation, because they will always be
instantiated during refresh if their configuration isn't computed, and so
the subsequent diff will want to destroy what was instantiated.

To work around this, we make an exception that data resource destroy
diffs may appear in the plan but nothing else.

This fixes #6713.
2016-05-21 14:16:02 -07:00
Mitchell Hashimoto 55583baa7e
Merge branch 'f-aws-import' 2016-05-18 15:28:12 -06:00
James Nugent 3ea3c657b5 core: Use OutputState in JSON instead of map
This commit forward ports the changes made for 0.6.17, in order to store
the type and sensitive flag against outputs.

It also refactors the logic of the import for V0 to V1 state, and
fixes up the call sites of the new format for outputs in V2 state.

Finally we fix up tests which did not previously set a state version
where one is required.
2016-05-18 13:25:20 -05:00
Mitchell Hashimoto 884980da1a
providers/aws: instance, nat, internet gateway 2016-05-16 10:03:57 -07:00
Mitchell Hashimoto b7d4767dd6
helper/schema: pass through import state func 2016-05-16 10:03:57 -07:00
Martin Atkins 6cd22a4c9a helper/schema: emit warning when using data source resource shim
For backward compatibility we will continue to support using the data
sources that were formerly logical resources as resources for the moment,
but we want to warn the user about it since this support is likely to
be removed in future.

This is done by adding a new "deprecation message" feature to
schema.Resource, but for the moment this is done as an internal feature
(not usable directly by plugins) so that we can collect additional
use-cases and design a more general interface before creating a
compatibility constraint.
2016-05-14 08:26:36 -07:00
Martin Atkins fb262d0dbe helper/schema: shim for making data sources act like resources
Historically we've had some "read-only" and "logical" resources. With the
addition of the data source concept these will gradually become data
sources, but we need to retain backward compatibility with existing
configurations that use the now-deprecated resources.

This shim is intended to allow us to easily create a resource from a
data source implementation. It adjusts the schema as needed and adds
stub Create and Delete implementations.

This would ideally also produce a deprecation warning whenever such a
shimmed resource is used, but the schema system doesn't currently have
a mechanism for resource-specific validation, so that remains just a TODO
for the moment.
2016-05-14 08:26:36 -07:00
Martin Atkins 6a468dcd83 helper/schema: Resource can be writable or not
In the "schema" layer a Resource is just any "thing" that has a schema
and supports some or all of the CRUD operations. Data sources introduce
a new use of Resource to represent read-only resources, which require
some different InternalValidate logic.
2016-05-14 08:26:36 -07:00
Martin Atkins 0e0e3d73af core: New ResourceProvider methods for data resources
This is a breaking change to the ResourceProvider interface that adds the
new operations relating to data sources.

DataSources, ValidateDataSource, ReadDataDiff and ReadDataApply are the
data source equivalents of Resources, Validate, Diff and Apply (respectively)
for managed resources.

The diff/apply model seems at first glance a rather strange workflow for
read-only resources, but implementing data resources in this way allows them
to fit cleanly into the standard plan/apply lifecycle in cases where the
configuration contains computed arguments and thus the read must be deferred
until apply time.

Along with breaking the interface, we also fix up the plugin client/server
and helper/schema implementations of it, which are all of the callers
used when provider plugins use helper/schema. This would be a breaking
change for any provider plugin that directly implements the provider
interface, but no known plugins do this and it is not recommended.

At the helper/schema layer the implementer sees ReadDataApply as a "Read",
as opposed to "Create" or "Update" as in the managed resource Apply
implementation. The planning mechanics are handled entirely within
helper/schema, so that complexity is hidden from the provider implementation
itself.
2016-05-14 08:26:36 -07:00
Mitchell Hashimoto 6a675b4a15
helper/resource: ImportState test can verify states 2016-05-11 13:02:37 -07:00
Mitchell Hashimoto ec02c8c0e2
helper/resource: testing of almost all aspects of ImportState tests 2016-05-11 13:02:37 -07:00
Mitchell Hashimoto 2d99c451fb
helper/resource: basic ImportState acceptance testing
Still some TODOs, and more test cases to write, but the basics are all
here.
2016-05-11 13:02:37 -07:00
Mitchell Hashimoto 9bd1c9e7ca
helper/resource: reshuffling to prepare for importstate testing 2016-05-11 13:02:37 -07:00
Mitchell Hashimoto b728e55861
helper/schema: Resource.Data should set latest schema version 2016-05-11 13:02:36 -07:00
Mitchell Hashimoto c02c6c3f9c
helper/schema: default state func for import 2016-05-11 13:02:35 -07:00
Mitchell Hashimoto 2bb814e3de
helper/schema: adapt to ID being arg to ImportState 2016-05-11 13:02:35 -07:00
Mitchell Hashimoto 03931bfda9
helper/schema: ImportState must set ID on the resource data 2016-05-11 13:02:34 -07:00
Mitchell Hashimoto 445194ebdf
helper/schema: test ImportState 2016-05-11 13:02:33 -07:00
Mitchell Hashimoto 1685054a9a
helper/schema: cleaner way to store Ephemeral 2016-05-11 13:02:31 -07:00
Mitchell Hashimoto 84531a3fd5
helper/schema: sets Importable to true for resources that have importer 2016-05-11 13:02:30 -07:00
Mitchell Hashimoto 19609bde0e
helper/schema: can specify Importer on Resource, InternalValidate 2016-05-11 13:02:30 -07:00
Mitchell Hashimoto b8121ea63e
helper/schema: Resource.Data to return a ResourceData for a Resource 2016-05-11 13:02:30 -07:00
Mitchell Hashimoto 4e9877179c
helper/schema: start the resource importer 2016-05-11 13:02:29 -07:00
James Nugent 7b6df27e4a helper/schema: Read native maps from configuration
This adds a test and the support necessary to read from native maps
passed as variables via interpolation - for example:

```
resource ...... {
     mapValue = "${var.map}"
}
```

We also add support for interpolating maps from the flat-mapped resource
config, which is necessary to support assignment of computed maps, which
is now valid.

Unfortunately there is no good way to distinguish between a list and a
map in the flatmap. In lieu of changing that representation (which is
risky), we assume that if all the keys are numeric, this is intended to
be a list, and if not it is intended to be a map. This does preclude
maps which have purely numeric keys, which should be noted as a
backwards compatibility concern.
2016-05-10 14:49:14 -04:00
James Nugent f49583d25a core: support native list variables in config
This commit adds support for native list variables and outputs, building
up on the previous change to state. Interpolation functions now return
native lists in preference to StringList.

List variables are defined like this:

variable "test" {
    # This can also be inferred
    type = "list"
    default = ["Hello", "World"]
}

output "test_out" {
    value = "${var.a_list}"
}
This results in the following state:

```
...
            "outputs": {
                "test_out": [
                    "hello",
                    "world"
                ]
            },
...
```

And the result of terraform output is as follows:

```
$ terraform output
test_out = [
  hello
  world
]
```

Using the output name, an xargs-friendly representation is output:

```
$ terraform output test_out
hello
world
```

The output command also supports indexing into the list (with
appropriate range checking and no wrapping):

```
$ terraform output test_out 1
world
```

Along with maps, list outputs from one module may be passed as variables
into another, removing the need for the `join(",", var.list_as_string)`
and `split(",", var.list_as_string)` which was previously necessary in
Terraform configuration.

This commit also updates the tests and implementations of built-in
interpolation functions to take and return native lists where
appropriate.

A backwards compatibility note: previously the concat interpolation
function was capable of concatenating either strings or lists. The
strings use case was deprectated a long time ago but still remained.
Because we cannot return `ast.TypeAny` from an interpolation function,
this use case is no longer supported for strings - `concat` is only
capable of concatenating lists. This should not be a huge issue - the
type checker picks up incorrect parameters, and the native HIL string
concatenation - or the `join` function - can be used to replicate the
missing behaviour.
2016-05-10 14:49:14 -04:00
Mitchell Hashimoto 35c87836b4 core: Add terraform_version to state
This adds a field terraform_version to the state that represents the
Terraform version that wrote that state. If Terraform encounters a state
written by a future version, it will error. You must use at least the
version that wrote that state.

Internally we have fields to override this behavior (StateFutureAllowed),
but I chose not to expose them as CLI flags, since the user can just
modify the state directly. This is tricky, but should be tricky to
represent the horrible disaster that can happen by enabling it.

We didn't have to bump the state format version since the absense of the
field means it was written by version "0.0.0" which will always be
older. In effect though this change will always apply to version 2 of
the state since it appears in 0.7 which bumped the version for other
purposes.
2016-05-10 14:40:11 -04:00
Paul Hinze b4df304b47
helper/schema: Normalize bools to "true"/"false" in diffs
For a long time now, the diff logic has relied on the behavior of
`mapstructure.WeakDecode` to determine how various primitives are
converted into strings.  The `schema.DiffString` function is used for
all primitive field types: TypeBool, TypeInt, TypeFloat, and TypeString.

The `mapstructure` library's string representation of booleans is "0"
and "1", which differs from `strconv.FormatBool`'s "false" and "true"
(which is used in writing out boolean fields to the state).

Because of this difference, diffs have long had the potential for
cosmetically odd but semantically neutral output like:

    "true" => "1"
    "false" => "0"

So long as `mapstructure.Decode` or `strconv.ParseBool` are used to
interpret these strings, there's no functional problem.

We had our first clear functional problem with #6005 and friends, where
users noticed diffs like the above showing up unexpectedly and causing
troubles when `ignore_changes` was in play.

This particular bug occurs down in Terraform core's EvalIgnoreChanges.
There, the diff is modified to account for ignored attributes, and
special logic attempts to handle properly the situation where the
ignored attribute was going to trigger a resource replacement. That
logic relies on the string representations of the Old and New fields in
the diff to be the same so that it filters properly.

So therefore, we now get a bug when a diff includes `Old: "0", New:
"false"` since the strings do not match, and `ignore_changes` is not
properly handled.

Here, we introduce `TypeBool`-specific normalizing into `finalizeDiff`.
I spiked out a full `diffBool` function, but figuring out which pieces
of `diffString` to duplicate there got hairy. This seemed like a simpler
and more direct solution.

Fixes #6005 (and potentially others!)
2016-05-05 09:00:58 -05:00
Mitchell Hashimoto d85df63526
providers/aws: aws_instance id-only 2016-04-22 09:37:41 -07:00
Mitchell Hashimoto e0da21d381
helper/resource: make id-only check opt-in
As I've been working through the resources, I'm finding that a lot are
going to need some serious work. Given we have hundreds, I think it
might be prudent to make this opt-in for now and we can revisit
automatic/opt-out at some future point.

Importability will likely be opt-in it appears so this will match up
with that.
2016-04-21 08:37:08 -07:00
Mitchell Hashimoto ff7b58f032
providers/aws: peering connection id-only test settings 2016-04-20 12:19:21 -07:00
Mitchell Hashimoto 0ef1b3b84a
providers/aws: response value for DescribeVpcAttribute needs to be
.Value
2016-04-20 11:35:43 -07:00
Mitchell Hashimoto 1db1bf6639
helper/resource: use the full config for id-only checks
Originally I used an empty config module. This caused problems since
important provider configurations weren't available. Instead, I now set
it to use the full config. This isn't an issue since the attributes
themselves aren't available to Refresh anyways.
2016-04-20 11:18:13 -07:00
Mitchell Hashimoto 0c8b0bff2c
helper/resource: can specify specific name to id refresh test 2016-04-20 11:12:30 -07:00
Mitchell Hashimoto 1a9fae6b2e
helper/resource: can disable ID refresh check 2016-04-20 11:09:54 -07:00
Mitchell Hashimoto 35f4201b9e
providers/aws: instance_tenancy is computed, set 2016-04-20 10:48:22 -07:00
Mitchell Hashimoto f2c4f8e9ba
helper/resource: fix tests 2016-04-20 10:08:34 -07:00
Mitchell Hashimoto 5c4f78796b
helper/resource: don't need to id-only check if no test steps 2016-04-20 09:52:53 -07:00
Mitchell Hashimoto a285c04dc9
helper/resource: only verify id-only run if no error 2016-04-20 09:50:59 -07:00