terraform/states/state_deepcopy.go

233 lines
7.7 KiB
Go
Raw Normal View History

states: New package with modern models for Terraform state Our previous state models in the "terraform" package had a few limitations that are addressed here: - Instance attributes were stored as map[string]string with dot-separated keys representing traversals through a data structure. Now that we have a full type system, it's preferable to store it as a real data structure. - The existing state structures skipped over the "resource" concept and went straight to resource instance, requiring heuristics to decide whether a particular resource should appear as a single object or as a list of objects when used in configuration expressions. - Related to the previous point, the state models also used incorrect terminology where "ResourceState" was really a resource instance state and "InstanceState" was really the state of a particular remote object associated with an instance. These new models use the correct names for each of these, introducing the idea of a "ResourceInstanceObject" as the local record of a remote object associated with an instance. This is a first pass at fleshing out a new model for state. Undoubtedly there will be further iterations of this as we work on integrating these new models into the "terraform" package. These new model types no longer serve double-duty as a description of the JSON state file format, since they are for in-memory use only. A subsequent commit will introduce a separate package that deals with persisting state to files and reloading those files later.
2018-06-08 02:27:57 +02:00
package states
import (
"github.com/hashicorp/terraform/addrs"
"github.com/zclconf/go-cty/cty"
)
// Taking deep copies of states is an important operation because state is
// otherwise a mutable data structure that is challenging to share across
// many separate callers. It is important that the DeepCopy implementations
// in this file comprehensively copy all parts of the state data structure
// that could be mutated via pointers.
// DeepCopy returns a new state that contains equivalent data to the reciever
// but shares no backing memory in common.
//
// As with all methods on State, this method is not safe to use concurrently
// with writing to any portion of the recieving data structure. It is the
// caller's responsibility to ensure mutual exclusion for the duration of the
// operation, but may then freely modify the receiver and the returned copy
// independently once this method returns.
func (s *State) DeepCopy() *State {
if s == nil {
return nil
}
modules := make(map[string]*Module, len(s.Modules))
for k, m := range s.Modules {
modules[k] = m.DeepCopy()
}
return &State{
Modules: modules,
}
}
// DeepCopy returns a new module state that contains equivalent data to the
// receiver but shares no backing memory in common.
//
// As with all methods on Module, this method is not safe to use concurrently
// with writing to any portion of the recieving data structure. It is the
// caller's responsibility to ensure mutual exclusion for the duration of the
// operation, but may then freely modify the receiver and the returned copy
// independently once this method returns.
func (ms *Module) DeepCopy() *Module {
if ms == nil {
return nil
}
resources := make(map[string]*Resource, len(ms.Resources))
for k, r := range ms.Resources {
resources[k] = r.DeepCopy()
}
outputValues := make(map[string]*OutputValue, len(ms.OutputValues))
for k, v := range ms.OutputValues {
outputValues[k] = v.DeepCopy()
}
localValues := make(map[string]cty.Value, len(ms.LocalValues))
for k, v := range ms.LocalValues {
// cty.Value is immutable, so we don't need to copy these.
localValues[k] = v
}
return &Module{
Addr: ms.Addr, // technically mutable, but immutable by convention
Resources: resources,
OutputValues: outputValues,
LocalValues: localValues,
}
}
// DeepCopy returns a new resource state that contains equivalent data to the
// receiver but shares no backing memory in common.
//
// As with all methods on Resource, this method is not safe to use concurrently
// with writing to any portion of the recieving data structure. It is the
// caller's responsibility to ensure mutual exclusion for the duration of the
// operation, but may then freely modify the receiver and the returned copy
// independently once this method returns.
func (rs *Resource) DeepCopy() *Resource {
if rs == nil {
return nil
}
instances := make(map[addrs.InstanceKey]*ResourceInstance, len(rs.Instances))
for k, i := range rs.Instances {
instances[k] = i.DeepCopy()
}
return &Resource{
Addr: rs.Addr,
Instances: instances,
ProviderConfig: rs.ProviderConfig, // technically mutable, but immutable by convention
}
}
// DeepCopy returns a new resource instance state that contains equivalent data
// to the receiver but shares no backing memory in common.
//
// As with all methods on ResourceInstance, this method is not safe to use
// concurrently with writing to any portion of the recieving data structure. It
// is the caller's responsibility to ensure mutual exclusion for the duration
// of the operation, but may then freely modify the receiver and the returned
// copy independently once this method returns.
func (is *ResourceInstance) DeepCopy() *ResourceInstance {
if is == nil {
return nil
}
deposed := make(map[DeposedKey]*ResourceInstanceObjectSrc, len(is.Deposed))
states: New package with modern models for Terraform state Our previous state models in the "terraform" package had a few limitations that are addressed here: - Instance attributes were stored as map[string]string with dot-separated keys representing traversals through a data structure. Now that we have a full type system, it's preferable to store it as a real data structure. - The existing state structures skipped over the "resource" concept and went straight to resource instance, requiring heuristics to decide whether a particular resource should appear as a single object or as a list of objects when used in configuration expressions. - Related to the previous point, the state models also used incorrect terminology where "ResourceState" was really a resource instance state and "InstanceState" was really the state of a particular remote object associated with an instance. These new models use the correct names for each of these, introducing the idea of a "ResourceInstanceObject" as the local record of a remote object associated with an instance. This is a first pass at fleshing out a new model for state. Undoubtedly there will be further iterations of this as we work on integrating these new models into the "terraform" package. These new model types no longer serve double-duty as a description of the JSON state file format, since they are for in-memory use only. A subsequent commit will introduce a separate package that deals with persisting state to files and reloading those files later.
2018-06-08 02:27:57 +02:00
for k, obj := range is.Deposed {
deposed[k] = obj.DeepCopy()
}
return &ResourceInstance{
Current: is.Current.DeepCopy(),
Deposed: deposed,
}
}
// DeepCopy returns a new resource instance object that contains equivalent data
// to the receiver but shares no backing memory in common.
//
// As with all methods on ResourceInstanceObjectSrc, this method is not safe to
// use concurrently with writing to any portion of the recieving data structure.
// It is the caller's responsibility to ensure mutual exclusion for the duration
states: New package with modern models for Terraform state Our previous state models in the "terraform" package had a few limitations that are addressed here: - Instance attributes were stored as map[string]string with dot-separated keys representing traversals through a data structure. Now that we have a full type system, it's preferable to store it as a real data structure. - The existing state structures skipped over the "resource" concept and went straight to resource instance, requiring heuristics to decide whether a particular resource should appear as a single object or as a list of objects when used in configuration expressions. - Related to the previous point, the state models also used incorrect terminology where "ResourceState" was really a resource instance state and "InstanceState" was really the state of a particular remote object associated with an instance. These new models use the correct names for each of these, introducing the idea of a "ResourceInstanceObject" as the local record of a remote object associated with an instance. This is a first pass at fleshing out a new model for state. Undoubtedly there will be further iterations of this as we work on integrating these new models into the "terraform" package. These new model types no longer serve double-duty as a description of the JSON state file format, since they are for in-memory use only. A subsequent commit will introduce a separate package that deals with persisting state to files and reloading those files later.
2018-06-08 02:27:57 +02:00
// of the operation, but may then freely modify the receiver and the returned
// copy independently once this method returns.
func (obj *ResourceInstanceObjectSrc) DeepCopy() *ResourceInstanceObjectSrc {
states: New package with modern models for Terraform state Our previous state models in the "terraform" package had a few limitations that are addressed here: - Instance attributes were stored as map[string]string with dot-separated keys representing traversals through a data structure. Now that we have a full type system, it's preferable to store it as a real data structure. - The existing state structures skipped over the "resource" concept and went straight to resource instance, requiring heuristics to decide whether a particular resource should appear as a single object or as a list of objects when used in configuration expressions. - Related to the previous point, the state models also used incorrect terminology where "ResourceState" was really a resource instance state and "InstanceState" was really the state of a particular remote object associated with an instance. These new models use the correct names for each of these, introducing the idea of a "ResourceInstanceObject" as the local record of a remote object associated with an instance. This is a first pass at fleshing out a new model for state. Undoubtedly there will be further iterations of this as we work on integrating these new models into the "terraform" package. These new model types no longer serve double-duty as a description of the JSON state file format, since they are for in-memory use only. A subsequent commit will introduce a separate package that deals with persisting state to files and reloading those files later.
2018-06-08 02:27:57 +02:00
if obj == nil {
return nil
}
var attrsFlat map[string]string
if obj.AttrsFlat != nil {
attrsFlat = make(map[string]string, len(obj.AttrsFlat))
states: New package with modern models for Terraform state Our previous state models in the "terraform" package had a few limitations that are addressed here: - Instance attributes were stored as map[string]string with dot-separated keys representing traversals through a data structure. Now that we have a full type system, it's preferable to store it as a real data structure. - The existing state structures skipped over the "resource" concept and went straight to resource instance, requiring heuristics to decide whether a particular resource should appear as a single object or as a list of objects when used in configuration expressions. - Related to the previous point, the state models also used incorrect terminology where "ResourceState" was really a resource instance state and "InstanceState" was really the state of a particular remote object associated with an instance. These new models use the correct names for each of these, introducing the idea of a "ResourceInstanceObject" as the local record of a remote object associated with an instance. This is a first pass at fleshing out a new model for state. Undoubtedly there will be further iterations of this as we work on integrating these new models into the "terraform" package. These new model types no longer serve double-duty as a description of the JSON state file format, since they are for in-memory use only. A subsequent commit will introduce a separate package that deals with persisting state to files and reloading those files later.
2018-06-08 02:27:57 +02:00
for k, v := range obj.AttrsFlat {
attrsFlat[k] = v
}
}
var attrsJSON []byte
if obj.AttrsJSON != nil {
attrsJSON = make([]byte, len(obj.AttrsJSON))
states: New package with modern models for Terraform state Our previous state models in the "terraform" package had a few limitations that are addressed here: - Instance attributes were stored as map[string]string with dot-separated keys representing traversals through a data structure. Now that we have a full type system, it's preferable to store it as a real data structure. - The existing state structures skipped over the "resource" concept and went straight to resource instance, requiring heuristics to decide whether a particular resource should appear as a single object or as a list of objects when used in configuration expressions. - Related to the previous point, the state models also used incorrect terminology where "ResourceState" was really a resource instance state and "InstanceState" was really the state of a particular remote object associated with an instance. These new models use the correct names for each of these, introducing the idea of a "ResourceInstanceObject" as the local record of a remote object associated with an instance. This is a first pass at fleshing out a new model for state. Undoubtedly there will be further iterations of this as we work on integrating these new models into the "terraform" package. These new model types no longer serve double-duty as a description of the JSON state file format, since they are for in-memory use only. A subsequent commit will introduce a separate package that deals with persisting state to files and reloading those files later.
2018-06-08 02:27:57 +02:00
copy(attrsJSON, obj.AttrsJSON)
}
Store sensitive attribute paths in state (#26338) * Add creation test and simplify in-place test * Add deletion test * Start adding marking from state Start storing paths that should be marked when pulled out of state. Implements deep copy for attr paths. This commit also includes some comment noise from investigations, and fixing the diff test * Fix apply stripping marks * Expand diff tests * Basic apply test * Update comments on equality checks to clarify current understanding * Add JSON serialization for sensitive paths We need to serialize a slice of cty.Path values to be used to re-mark the sensitive values of a resource instance when loading the state file. Paths consist of a list of steps, each of which may be either getting an attribute value by name, or indexing into a collection by string or number. To serialize these without building a complex parser for a compact string form, we render a nested array of small objects, like so: [ [ { type: "get_attr", value: "foo" }, { type: "index", value: { "type": "number", "value": 2 } } ] ] The above example is equivalent to a path `foo[2]`. * Format diffs with map types Comparisons need unmarked values to operate on, so create unmarked values for those operations. Additionally, change diff to cover map types * Remove debugging printing * Fix bug with marking non-sensitive values When pulling a sensitive value from state, we were previously using those marks to remark the planned new value, but that new value might *not* be sensitive, so let's not do that * Fix apply test Apply was not passing the second state through to the third pass at apply * Consistency in checking for length of paths vs inspecting into value * In apply, don't mark with before paths * AttrPaths test coverage for DeepCopy * Revert format changes Reverts format changes in format/diff for this branch so those changes can be discussed on a separate PR * Refactor name of AttrPaths to AttrSensitivePaths * Rename AttributePaths/attributePaths for naming consistency Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2020-09-24 18:40:17 +02:00
var attrPaths []cty.PathValueMarks
if obj.AttrSensitivePaths != nil {
attrPaths = make([]cty.PathValueMarks, len(obj.AttrSensitivePaths))
copy(attrPaths, obj.AttrSensitivePaths)
}
var private []byte
if obj.Private != nil {
private = make([]byte, len(obj.Private))
copy(private, obj.Private)
}
states: New package with modern models for Terraform state Our previous state models in the "terraform" package had a few limitations that are addressed here: - Instance attributes were stored as map[string]string with dot-separated keys representing traversals through a data structure. Now that we have a full type system, it's preferable to store it as a real data structure. - The existing state structures skipped over the "resource" concept and went straight to resource instance, requiring heuristics to decide whether a particular resource should appear as a single object or as a list of objects when used in configuration expressions. - Related to the previous point, the state models also used incorrect terminology where "ResourceState" was really a resource instance state and "InstanceState" was really the state of a particular remote object associated with an instance. These new models use the correct names for each of these, introducing the idea of a "ResourceInstanceObject" as the local record of a remote object associated with an instance. This is a first pass at fleshing out a new model for state. Undoubtedly there will be further iterations of this as we work on integrating these new models into the "terraform" package. These new model types no longer serve double-duty as a description of the JSON state file format, since they are for in-memory use only. A subsequent commit will introduce a separate package that deals with persisting state to files and reloading those files later.
2018-06-08 02:27:57 +02:00
// Some addrs.Referencable implementations are technically mutable, but
// we treat them as immutable by convention and so we don't deep-copy here.
var dependencies []addrs.ConfigResource
if obj.Dependencies != nil {
dependencies = make([]addrs.ConfigResource, len(obj.Dependencies))
copy(dependencies, obj.Dependencies)
}
return &ResourceInstanceObjectSrc{
Status: obj.Status,
SchemaVersion: obj.SchemaVersion,
Private: private,
AttrsFlat: attrsFlat,
AttrsJSON: attrsJSON,
Store sensitive attribute paths in state (#26338) * Add creation test and simplify in-place test * Add deletion test * Start adding marking from state Start storing paths that should be marked when pulled out of state. Implements deep copy for attr paths. This commit also includes some comment noise from investigations, and fixing the diff test * Fix apply stripping marks * Expand diff tests * Basic apply test * Update comments on equality checks to clarify current understanding * Add JSON serialization for sensitive paths We need to serialize a slice of cty.Path values to be used to re-mark the sensitive values of a resource instance when loading the state file. Paths consist of a list of steps, each of which may be either getting an attribute value by name, or indexing into a collection by string or number. To serialize these without building a complex parser for a compact string form, we render a nested array of small objects, like so: [ [ { type: "get_attr", value: "foo" }, { type: "index", value: { "type": "number", "value": 2 } } ] ] The above example is equivalent to a path `foo[2]`. * Format diffs with map types Comparisons need unmarked values to operate on, so create unmarked values for those operations. Additionally, change diff to cover map types * Remove debugging printing * Fix bug with marking non-sensitive values When pulling a sensitive value from state, we were previously using those marks to remark the planned new value, but that new value might *not* be sensitive, so let's not do that * Fix apply test Apply was not passing the second state through to the third pass at apply * Consistency in checking for length of paths vs inspecting into value * In apply, don't mark with before paths * AttrPaths test coverage for DeepCopy * Revert format changes Reverts format changes in format/diff for this branch so those changes can be discussed on a separate PR * Refactor name of AttrPaths to AttrSensitivePaths * Rename AttributePaths/attributePaths for naming consistency Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2020-09-24 18:40:17 +02:00
AttrSensitivePaths: attrPaths,
Dependencies: dependencies,
CreateBeforeDestroy: obj.CreateBeforeDestroy,
states: New package with modern models for Terraform state Our previous state models in the "terraform" package had a few limitations that are addressed here: - Instance attributes were stored as map[string]string with dot-separated keys representing traversals through a data structure. Now that we have a full type system, it's preferable to store it as a real data structure. - The existing state structures skipped over the "resource" concept and went straight to resource instance, requiring heuristics to decide whether a particular resource should appear as a single object or as a list of objects when used in configuration expressions. - Related to the previous point, the state models also used incorrect terminology where "ResourceState" was really a resource instance state and "InstanceState" was really the state of a particular remote object associated with an instance. These new models use the correct names for each of these, introducing the idea of a "ResourceInstanceObject" as the local record of a remote object associated with an instance. This is a first pass at fleshing out a new model for state. Undoubtedly there will be further iterations of this as we work on integrating these new models into the "terraform" package. These new model types no longer serve double-duty as a description of the JSON state file format, since they are for in-memory use only. A subsequent commit will introduce a separate package that deals with persisting state to files and reloading those files later.
2018-06-08 02:27:57 +02:00
}
}
// DeepCopy returns a new resource instance object that contains equivalent data
// to the receiver but shares no backing memory in common.
//
// As with all methods on ResourceInstanceObject, this method is not safe to use
// concurrently with writing to any portion of the recieving data structure. It
// is the caller's responsibility to ensure mutual exclusion for the duration
// of the operation, but may then freely modify the receiver and the returned
// copy independently once this method returns.
func (obj *ResourceInstanceObject) DeepCopy() *ResourceInstanceObject {
if obj == nil {
return nil
}
var private []byte
if obj.Private != nil {
private = make([]byte, len(obj.Private))
copy(private, obj.Private)
}
// Some addrs.Referenceable implementations are technically mutable, but
// we treat them as immutable by convention and so we don't deep-copy here.
var dependencies []addrs.ConfigResource
if obj.Dependencies != nil {
dependencies = make([]addrs.ConfigResource, len(obj.Dependencies))
copy(dependencies, obj.Dependencies)
}
return &ResourceInstanceObject{
Value: obj.Value,
Status: obj.Status,
Private: private,
Dependencies: dependencies,
}
}
states: New package with modern models for Terraform state Our previous state models in the "terraform" package had a few limitations that are addressed here: - Instance attributes were stored as map[string]string with dot-separated keys representing traversals through a data structure. Now that we have a full type system, it's preferable to store it as a real data structure. - The existing state structures skipped over the "resource" concept and went straight to resource instance, requiring heuristics to decide whether a particular resource should appear as a single object or as a list of objects when used in configuration expressions. - Related to the previous point, the state models also used incorrect terminology where "ResourceState" was really a resource instance state and "InstanceState" was really the state of a particular remote object associated with an instance. These new models use the correct names for each of these, introducing the idea of a "ResourceInstanceObject" as the local record of a remote object associated with an instance. This is a first pass at fleshing out a new model for state. Undoubtedly there will be further iterations of this as we work on integrating these new models into the "terraform" package. These new model types no longer serve double-duty as a description of the JSON state file format, since they are for in-memory use only. A subsequent commit will introduce a separate package that deals with persisting state to files and reloading those files later.
2018-06-08 02:27:57 +02:00
// DeepCopy returns a new output value state that contains equivalent data
// to the receiver but shares no backing memory in common.
//
// As with all methods on OutputValue, this method is not safe to use
// concurrently with writing to any portion of the recieving data structure. It
// is the caller's responsibility to ensure mutual exclusion for the duration
// of the operation, but may then freely modify the receiver and the returned
// copy independently once this method returns.
func (os *OutputValue) DeepCopy() *OutputValue {
if os == nil {
return nil
}
return &OutputValue{
Addr: os.Addr,
states: New package with modern models for Terraform state Our previous state models in the "terraform" package had a few limitations that are addressed here: - Instance attributes were stored as map[string]string with dot-separated keys representing traversals through a data structure. Now that we have a full type system, it's preferable to store it as a real data structure. - The existing state structures skipped over the "resource" concept and went straight to resource instance, requiring heuristics to decide whether a particular resource should appear as a single object or as a list of objects when used in configuration expressions. - Related to the previous point, the state models also used incorrect terminology where "ResourceState" was really a resource instance state and "InstanceState" was really the state of a particular remote object associated with an instance. These new models use the correct names for each of these, introducing the idea of a "ResourceInstanceObject" as the local record of a remote object associated with an instance. This is a first pass at fleshing out a new model for state. Undoubtedly there will be further iterations of this as we work on integrating these new models into the "terraform" package. These new model types no longer serve double-duty as a description of the JSON state file format, since they are for in-memory use only. A subsequent commit will introduce a separate package that deals with persisting state to files and reloading those files later.
2018-06-08 02:27:57 +02:00
Value: os.Value,
Sensitive: os.Sensitive,
}
}