Commit Graph

19 Commits

Author SHA1 Message Date
Pam Selle 02c48f8071 Comment fixing 2020-10-18 13:00:09 -04:00
Brian Flad a8e3787afc
helper/schema: Prevent setSet() panic with typed nil
References:

* https://github.com/hashicorp/terraform/issues/14418
* v0.9.5 (original bug report): a59ee0b30e/helper/schema/field_writer_map.go (L311)
* v0.11.12 (Terraform AWS Provider discovery): 057286e522/helper/schema/field_writer_map.go (L343)

When creating flatten functions in Terraform Providers that return *schema.Set, its possible to return a typed `nil`, e.g.

```go
func flattenHeaders(h *cloudfront.Headers) *schema.Set {
	if h.Items != nil {
		return schema.NewSet(schema.HashString, flattenStringList(h.Items))
	}
	return nil
}
```

This previously could cause a panic, e.g.

```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1881911]

goroutine 1325 [running]:
github.com/hashicorp/terraform/helper/schema.(*MapFieldWriter).setSet(0xc00054bf00, 0xc00073efa0, 0x5, 0x5, 0x5828140, 0x0, 0xc0002cea50, 0xc000e996a8, 0xc001026e40)
	/Users/bflad/go/pkg/mod/github.com/hashicorp/terraform@v0.11.12/helper/schema/field_writer_map.go:343 +0x211
```

Here we catch the typed `nil` and return an empty list flatmap result instead. Unit testing result prior to code update:

```
--- FAIL: TestMapFieldWriter (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
  panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1777cdc]

goroutine 913 [running]:
testing.tRunner.func1(0xc00045b800)
  /usr/local/Cellar/go/1.12.1/libexec/src/testing/testing.go:830 +0x392
panic(0x192cf20, 0x2267ca0)
  /usr/local/Cellar/go/1.12.1/libexec/src/runtime/panic.go:522 +0x1b5
github.com/hashicorp/terraform/helper/schema.(*MapFieldWriter).setSet(0xc0004648a0, 0xc0004408d0, 0x1, 0x1, 0x19e3de0, 0x0, 0xc00045c600, 0x30, 0x19e0080)
  /Users/bflad/src/github.com/hashicorp/terraform/helper/schema/field_writer_map.go:344 +0x68c
github.com/hashicorp/terraform/helper/schema.(*MapFieldWriter).set(0xc0004648a0, 0xc0004408d0, 0x1, 0x1, 0x19e3de0, 0x0, 0x1, 0x18)
  /Users/bflad/src/github.com/hashicorp/terraform/helper/schema/field_writer_map.go:107 +0x28b
github.com/hashicorp/terraform/helper/schema.(*MapFieldWriter).WriteField(0xc0004648a0, 0xc0004408d0, 0x1, 0x1, 0x19e3de0, 0x0, 0x0, 0x0)
  /Users/bflad/src/github.com/hashicorp/terraform/helper/schema/field_writer_map.go:89 +0x504
github.com/hashicorp/terraform/helper/schema.TestMapFieldWriter(0xc00045b800)
  /Users/bflad/src/github.com/hashicorp/terraform/helper/schema/field_writer_map_test.go:337 +0x2ddd
testing.tRunner(0xc00045b800, 0x1a44f90)
  /usr/local/Cellar/go/1.12.1/libexec/src/testing/testing.go:865 +0xc0
created by testing.(*T).Run
  /usr/local/Cellar/go/1.12.1/libexec/src/testing/testing.go:916 +0x35a
```
2019-04-01 20:10:32 -04:00
Farid Neshat 44a45b7332 helper/schema: Fix setting a set in a list
The added test in this commit, without the fix, will make d.Set return
the following error:

`Invalid address to set: []string{"ports", "0", "set"}`

This was due to the fact that setSet in feild_writer_map tried to
convert a slice into a set by creating a temp set schema and calling
writeField on that with the address(`[]string{"ports", "0", "set"}"` in
this case). However the temp schema was only for the set and not the
whole schema as seen in the address so, it should have been `[]string{"set"}"`
so it would align with the schema.

This commits adds another variable there(tempAddr) which will only
contain the last entry of the address that would be the set key, which
would match the created schema

This commit potentially fixes the problem described in #16331
2018-12-05 10:09:54 +01:00
Chris Marchesi 3fa11e456b
helper/schema: Clear existing map/set/list contents before overwriting
There are situations where one may need to write to a set, list, or map
more than once per single TF operation (apply/refresh/etc). In these
cases, further writes using Set (example: d.Set("some_set", newSet))
currently create unstable results in the set writer (the name of the
writer layer that holds the data set by these calls) because old keys
are not being cleared out first.

This bug is most visible when using sets. Example: First write to set
writes elements that have been hashed at 10 and 20, and the second write
writes elements that have been hashed at 30 and 40. While the set length
has been correctly set at 2, since a set is basically a map (as is the
entire map writer) and map results are non-deterministic, reads to this
set will now deliver unstable results in a random but predictable
fashion as the map results are delivered to the caller non-deterministic
- sometimes you may correctly get 30 and 40, but sometimes you may get
10 and 20, or even 10 and 30, etc.

This problem propagates to state which is even more damaging as unstable
results are set to state where they become part of the permanent data
set going forward.

The problem also applies to lists and maps. This is probably more of an
issue with maps as a map can contain any key/value combination and hence
there is no predictable pattern where keys would be overwritten with
default or zero values. This is contrary to complex lists, which has
this problem as well, but since lists are deterministic and the length
of a list properly gets updated during the overwrite, the problem is
masked by the fact that a read will only read to the boundary of the
list, skipping any bad data that may still be available due past the
list boundary.

This update clears the child contents of any set, list, or map before
beginning a new write to address this issue. Tests are included for all
three data types.
2017-11-05 12:04:23 -08:00
James Nugent dbf725bd68 core: Allow dynamic attributes in helper/schema
The helper/schema framework for building providers previously validated
in all cases that each field being set in state was in the schema.
However, in order to support remote state in a usable fashion, the need
has arisen for the top level attributes of the resource to be created
dynamically. In order to still be able to use helper/schema, this commit
adds the capability to assign additional fields.

Though I do not forsee this being used by providers other than remote
state (and that eventually may move into Terraform Core rather than
being a provider), the usage and semantics are:

To opt into dynamic attributes, add a schema attribute named
"__has_dynamic_attributes", and make it an optional string with no
default value, in order that it does not appear in diffs:

        "__has_dynamic_attributes": {
            Type: schema.TypeString
            Optional: true
        }

In the read callback, use the d.UnsafeSetFieldRaw(key, value) function
to set the dynamic attributes.

Note that other fields in the schema _are_ copied into state, and that
the names of the schema fields cannot currently be used as dynamic
attribute names, as we check to ensure a value is not already set for a
given key.
2016-06-11 13:29:05 +01:00
James Nugent 074545e536 core: Use .% instead of .# for maps in state
The flatmapped representation of state prior to this commit encoded maps
and lists (and therefore by extension, sets) with a key corresponding to
the number of elements, or the unknown variable indicator under a .# key
and then individual items. For example, the list ["a", "b", "c"] would
have been encoded as:

    listname.# = 3
    listname.0 = "a"
    listname.1 = "b"
    listname.2 = "c"

And the map {"key1": "value1", "key2", "value2"} would have been encoded
as:

    mapname.# = 2
    mapname.key1 = "value1"
    mapname.key2 = "value2"

Sets use the hash code as the key - for example a set with a (fictional)
hashcode calculation may look like:

    setname.# = 2
    setname.12312512 = "value1"
    setname.56345233 = "value2"

Prior to the work done to extend the type system, this was sufficient
since the internal representation of these was effectively the same.
However, following the separation of maps and lists into distinct
first-class types, this encoding presents a problem: given a state file,
it is impossible to tell the encoding of an empty list and an empty map
apart. This presents problems for the type checker during interpolation,
as many interpolation functions will operate on only one of these two
structures.

This commit therefore changes the representation in state of maps to use
a "%" as the key for the number of elements. Consequently the map above
will now be encoded as:

    mapname.% = 2
    mapname.key1 = "value1"
    mapname.key2 = "value2"

This has the effect of an empty list (or set) now being encoded as:

    listname.# = 0

And an empty map now being encoded as:

    mapname.% = 0

Therefore we can eliminate some nasty guessing logic from the resource
variable supplier for interpolation, at the cost of having to migrate
state up front (to follow in a subsequent commit).

In order to reduce the number of potential situations in which resources
would be "forced new", we continue to accept "#" as the count key when
reading maps via helper/schema. There is no situation under which we can
allow "#" as an actual map key in any case, as it would not be
distinguishable from a list or set in state.
2016-06-09 10:49:42 +01:00
Paul Hinze 069425a700 consul: Fix several problems w/ consul_keys update
Implementation notes:

 * The hash implementation was not considering key value, causing "diffs
   did not match" errors when a value was updated. Switching to default
   HashResource implementation fixes this
 * Using HashResource as a default exposed a bug in helper/schema that
   needed to be fixed so the Set function is picked up properly during
   Read
 * Stop writing back values into the `key` attribute; it triggers extra
   diffs when `default` is used. Computed values all just go into `var`.
 * Includes a state migration to prevent unnecessary diffs based on
   "key" attribute hashcodes changing.

In the tests:

 * Switch from leaning on the public demo Consul instance to requiring a
   CONSUL_HTTP_ADDR variable be set pointing to a `consul agent -dev`
   instance to be used only for testing.
 * Add a test that exposes the updating issues and covers the fixes

Fixes #774
Fixes #1866
Fixes #3023
2016-01-26 14:46:26 -06:00
Paul Hinze edaf5795a5 Merge pull request #3257 from fatih/fix-nil-setting-schema
schema: delete non existing values
2015-12-08 20:15:00 -06:00
Sander van Harmelen ef4726bd50 Change Set internals and make (extreme) performance improvements
Changing the Set internals makes a lot of sense as it saves doing
conversions in multiple places and gives a central place to alter
the key when a item is computed.

This will have no side effects other then that the ordering is now
based on strings instead on integers, so the order will be different.
This will however have no effect on existing configs as these will
use the individual codes/keys and not the ordering to determine if
there is a diff or not.

Lastly (but I think also most importantly) there is a fix in this PR
that makes diffing sets extremely more performand. Before a full diff
required reading the complete Set for every single parameter/attribute
you wanted to diff, while now it only gets that specific parameter.

We have a use case where we have a Set that has 18 parameters and the
set consist of about 600 items (don't ask 😉). So when doing a diff
it would take 100% CPU of all cores and stay that way for almost an
hour before being able to complete the diff.

Debugging this we learned that for retrieving every single parameter
it made over 52.000 calls to `func (c *ResourceConfig) get(..)`. In
this function a slice is created and used only for the duration of the
call, so the time needed to create all needed slices and on the other
hand the time the garbage collector needed to clean them up again caused
the system to cripple itself. Next to that there are also some expensive
reflect calls in this function which also claimed a fair amount of CPU
time.

After this fix the number of calls needed to get a single parameter
dropped from 52.000+ to only 2! 😃
2015-11-22 14:21:28 +01:00
Fatih Arslan 8e7fc240f9 schema: delete non existing values
We need to set the value to an empty value so the state file does
indeed change the value. Otherwise the obsolote value is still
intact and doesn't get changed at all. This means `terraform show`
still shows the obsolote value when the particular value is not
existing anymore. This is due the AWS API which is returning a null
instead of an empty string.
2015-09-16 23:26:27 +03:00
Sam Boyer b82bd0c280 Condense switch fallthroughs into expr lists 2015-05-26 21:52:36 -04:00
Paul Hinze f2368428d3 helper/schema: write "attr.#": "0" for empty maps
This fixes some perpetual diffs I saw in Atlas AccTests where an empty
map (`map[string]interface{}{}`) was being `d.Set` for "metadata_full".

Because the MapFieldWriter was not distinguishing between empty and nil,
this trigger the "map delete" logic and no count was written to the
state. This caused subsequent plans to improperly report a diff.

Here we redefine the map delete functionality to explicitly trigger only
on `nil`, so we catch the `.#` field for empty maps.
2015-05-06 10:21:22 -05:00
Dave Cunningham aa2015ccd0 Fix failing tests 2015-01-28 16:20:14 -05:00
Dave Cunningham 18c26cb2eb Add some missing Float cases 2015-01-28 12:53:34 -05:00
Sander van Harmelen 2edfd0e89d Just my OCD playing up 😉 2015-01-16 13:30:11 +01:00
Greg Osuri 2769d7cf9c Fixes #813: Ensuring set count (.#) is written to the state 2015-01-16 03:43:57 -08:00
Greg Osuri f870eff5f9 core: fix for #813 - added a gaurd for interface conversion 2015-01-16 00:16:38 -08:00
Mitchell Hashimoto 448887f3c4 helper/schema: map counts in state 2015-01-15 14:12:24 -08:00
Mitchell Hashimoto 03c6453a72 helper/schema: FieldWriter, replace Set 2015-01-10 11:44:26 -08:00