Commit Graph

77 Commits

Author SHA1 Message Date
James Bardin 8e4bd669e8 test Set.Copy, Graph.UpEdges, Graph.DownEdges 2020-06-11 09:53:09 -04:00
James Bardin 268959f4be make UpEdges and DownEdges return a copy
The public functions for the graph UpEdges and DownEdges is returning
the internal Set from the graph, meaning that callers could
inadvertently corrupt the graph structure by editing the returned Sets.

Make UpEdges and DownEdges return a copy of the set, while retaining the
efficient no-copy behavior for internal callers.
2020-06-11 09:53:09 -04:00
James Bardin 43d93b2036 remove excess logging from dag
The dag Update messages were not particularly helpful in the debugging
of terraform, and since we're going to be relying on DynamicExpand to an
even greater extent, this will eliminate a log of extra log output.
2020-04-08 12:30:35 -04:00
James Bardin 37a7d904aa dag benchmark 2020-02-19 14:53:19 -05:00
James Bardin 0c1ab6142b remove debug code
Remove the abandoned graph debugger
2020-02-19 14:53:19 -05:00
James Bardin 6096371068 remove all unneeded list-based iteration
Make the default walks always use the sets directly, which avoids
repeated copying of every set into a new slice.
2020-02-19 14:53:19 -05:00
James Bardin 26a4de803f remove uses of Set.List
Since the set can be iterated over directly, we no longer need to copy
the values into a new slice.
2020-02-19 14:53:19 -05:00
James Bardin 32ae3b5452 make dag.Set a simple map
This allows iteration directly over the set, removing the need to
allocate and copy a new slice for every iteration.
2020-02-19 14:46:35 -05:00
Karsten Jeschkies ddfda430d4
Correct simple typo in `walk.go`. 2019-10-22 11:40:50 +02:00
Martin Atkins c937c06a03 terraform: ugly huge change to weave in new HCL2-oriented types
Due to how deeply the configuration types go into Terraform Core, there
isn't a great way to switch out to HCL2 gradually. As a consequence, this
huge commit gets us from the old state to a _compilable_ new state, but
does not yet attempt to fix any tests and has a number of known missing
parts and bugs. We will continue to iterate on this in forthcoming
commits, heading back towards passing tests and making Terraform
fully-functional again.

The three main goals here are:
- Use the configuration models from the "configs" package instead of the
  older models in the "config" package, which is now deprecated and
  preserved only to help us write our migration tool.
- Do expression inspection and evaluation using the functionality of the
  new "lang" package, instead of the Interpolator type and related
  functionality in the main "terraform" package.
- Represent addresses of various objects using types in the addrs package,
  rather than hand-constructed strings. This is not critical to support
  the above, but was a big help during the implementation of these other
  points since it made it much more explicit what kind of address is
  expected in each context.

Since our new packages are built to accommodate some future planned
features that are not yet implemented (e.g. the "for_each" argument on
resources, "count"/"for_each" on modules), and since there's still a fair
amount of functionality still using old-style APIs, there is a moderate
amount of shimming here to connect new assumptions with old, hopefully in
a way that makes it easier to find and eliminate these shims later.

I apologize in advance to the person who inevitably just found this huge
commit while spelunking through the commit history.
2018-10-16 18:46:46 -07:00
Martin Atkins 2c70d884d6 dag: node visits return diagnostics rather than errors
This allows node visits to also return warnings.
2018-10-16 18:44:26 -07:00
James Bardin 8cf0a8ca9c faster DAG transitive reduction
In the case of highly-connected graphs, the TransitiveReduction process
was far too computationally intensive. Since no operations are applied
to the nodes, and the walk order is not even user visible, we don't need
to sort them n^2 times.
2017-10-03 11:24:59 -04:00
Radek Simko 93613ee526
terraform+dag: Set lower log levels 2017-08-14 11:43:45 +02:00
Martin Atkins b28fc1cd20 dag: method for filtering a set on arbitrary criteria 2017-05-11 11:57:46 -07:00
Edward Betts be265479a9 correct spelling mistakes (#13979) 2017-04-27 02:10:04 +12:00
James Bardin bfa6ab4617 Fix removeEdge test failures
The removeEdge test could fail intermittently with the wrong order.

The precondition of a 1->2->3 order wasn't met, because there was no
edge from 1->3, so 3->1->2 was also a valid ordering.

The other failure was a bookkeeping error, were the recorded order may
not match the visited order. What happened in this case was the gateCh
was closed by V2, allowing V3 to run which could beat V2 to recording
its visit. Now the visit is recorded as part of the vertex walk, and the
gate is released as the final operation.

The order is deterministic now, so remove the brute-force test loop.
2017-02-20 19:54:34 -05:00
James Bardin d01b0b0647 Remove intermittent failure from newEdge test
Because the vertex visit was record after the Update call, Updated
vertices may have been visited before the visit was recorded, causing
occasional test failures.

The order is now deterministic, and we can remove the brute-force loop.
2017-02-20 19:54:34 -05:00
James Bardin 7bf33c2a7f Remove loop from TestWalker_removeVertex
There's no timing dependent behavior here, since V1 must be visited
before V2, Remove and Update must be called before V2 is visited.
2017-02-20 19:54:34 -05:00
James Bardin 0fb24c1a7a Remove sleep-based concurrency from newVertex test
Add a synchronization channel for the TestWalker_newVertex test, rather
than using a sleep and running it multiple times.
2017-02-20 19:54:13 -05:00
Mitchell Hashimoto 6702a22074
dag: require acyclic graph 2017-02-03 21:18:34 +01:00
Mitchell Hashimoto 07ce5a7624
dag: cleanup some code that had no effect anymore 2017-02-03 21:18:34 +01:00
Mitchell Hashimoto 6366488809
dag: Update can be called with a nil graph 2017-02-03 21:18:33 +01:00
Mitchell Hashimoto 65752cd51a
dag: improve docs and read access during a lock 2017-02-03 21:18:33 +01:00
Mitchell Hashimoto 72a717f2de
dag: change the type sig of Update to Graph so its external friendly 2017-02-03 21:18:33 +01:00
Mitchell Hashimoto 28fff99ea8
dag: replace dag.Walk with our walker 2017-02-03 21:18:33 +01:00
Mitchell Hashimoto b1aa6fd598
dag: improved comments 2017-02-03 21:18:33 +01:00
Mitchell Hashimoto 7f61f11723
dag: support removing edges 2017-02-03 21:18:33 +01:00
Mitchell Hashimoto 5d49e7e6b6
dag: tests for adding edges/vertices during walk-time 2017-02-03 21:18:33 +01:00
Mitchell Hashimoto cbc71d9508
dag: basic working update-able walker 2017-02-03 21:18:33 +01:00
Mitchell Hashimoto 77c445a838
dag: Set difference 2017-02-03 21:18:33 +01:00
Mitchell Hashimoto c8df3e5995
dag: string DotOpts through to vertex
Fixes #11052

It appears that historically nodes did not expect DotOpts to ever be
nil. To avoid nil panics in general I'm in agreement with this behavior
so this modifies dag to always pass in a non-nil DotOpts. Tests
included.
2017-01-08 11:59:18 -08:00
Mitchell Hashimoto 8a9e1c1527
dag: call into DotNode to get attributes 2016-12-03 15:17:14 -08:00
James Bardin ace06f9759 Merge pull request #10149 from hashicorp/jbardin/debug-cmd
Add debug command with json2dot
2016-11-21 17:35:19 -05:00
James Bardin b8adf10236 Add debug command with json2dot
Add `terraform debug json2dot` to convert debug log graphs to dot
format. This is not meant to be in place of more advanced debug
visualization, but may continue to be a useful way to work with the
debug output.
2016-11-21 11:59:20 -05:00
James Bardin 6d30b60144 Add DebugVisitInfo
This encodes vertex debug information into the graph log when a vertex
is visited during a walk operation. These can ordered to show how the
Graph was walked.

Add a mutex to the encoder so it can be used during a parallel walk.

Moved string literal constants used for marshaling to pre-defined constants.
Did some renaming to make the marshal* structures more consistent.
2016-11-15 10:39:08 -05:00
James Bardin 1be8e8c5a0 Add Graph.DebugOperation test 2016-11-14 14:36:06 -05:00
James Bardin de0cb17a39 Add Graph.DebugOperation
The method marks the start of a set of operations on the Graph, with
extra information optionally provided in the second paramter. This
returns a function with a single End method to mark the end of the set
in the logs.

Refactor the existing graph Begin/End Operation calls to use this single
method. Remove the *string types in the marshal structs, these are
strictly informational and don't need to differentiate empty vs unset
strings.

Add calls to DebugOperation for each step while building the graph.
2016-11-14 12:47:51 -05:00
James Bardin 7e66df3290 Rename annotation methods
Change AnnotateVertex and AnnotateEdge to VertexDebugInfo EdgeDebugInfo
to avoid confusion between debug output and future graph annotations.
2016-11-14 11:06:52 -05:00
James Bardin f37b2fafed Remove sync.Once from dag.Graph
dag.Graph is used as a value, but contains a sync data structure. This
prevents copying the value, and is needed if one wants to upgrade a Graph
to an AcyclicGraph.

The sync.Once only guards the init() method, which can be guarded
internally with nil checks on the fields. The init method could be
removed entirely with a proper constructor later on of we so choose.
2016-11-14 08:52:18 -05:00
James Bardin 7bbd6bfbd7 Add methods to annotate Graph vertices and edges
The AnnotateVertex and AnnotateEdge Graph methods will allow terraform
to insert arbitray information into the encoded graph stream for later
processing.
2016-11-14 08:52:18 -05:00
James Bardin 82b1a2abc2 Add graph transformation recording
The external api provided here is simply
dag.Graph.SetDebugWriter(io.Writer). When a writer is provided to a
Graph, it will immediately encode itself to the stream, and subsequently
encode any additional transformations to the graph. This will allow
easier logging of graph transformations without writing complete graphs
to the logs at every step. Since the marshalGraph can also be dot
encoded, this will allow translation from the JSON logs to dot graphs.
2016-11-14 08:50:34 -05:00
James Bardin 6f347ebb3a Remove dot package
Unify all dot functionality in the dag package
2016-11-14 08:50:34 -05:00
James Bardin 8a5d71b0ac Implement dag.GraphNodeDotter (temporarily)
To maintain the same output, the Graph.Dot implementation needs to be
aware of GraphNodeDotter. Copy the interface into the dag package, and
make the Dot marshaler aware of which nodes implemented the interface.
This way we can remove most of the remaining dot code from terraform.
2016-11-14 08:50:34 -05:00
James Bardin 7b774f771b implement dag.Subgrapher interface
This allows the dag package to detect subgraphs, even when impelemnted
by types from other packages
2016-11-14 08:50:34 -05:00
James Bardin 28d406c040 Provider a marshaler for dag.Graph
The dot format generation was done with a mix of code from the terraform
package and the dot package. Unify the dot generation code, and it into
the dag package.

Use an intermediate structure to allow a dag.Graph to marshal itself
directly. This structure will be ablt to marshal directly to JSON, or be
translated to dot format. This was we can record more information about
the graph in the debug logs, and provide a way to translate those logged
structures to dot, which is convenient for viewing the graphs.
2016-11-14 08:50:33 -05:00
Mitchell Hashimoto 83b1c82e93
terraform: put quotes before vertex name to avoid false panicwrap
Fixes #9395
2016-10-23 14:23:36 -07:00
Mitchell Hashimoto 6e632ec2ba
dag: test for EdgesFrom, EdgesTo 2016-10-19 13:38:53 -07:00
Mitchell Hashimoto 4e8e6cd661
dag: add EdgesFrom, EdgesTo, needs tests 2016-10-19 13:38:53 -07:00
James Bardin 1af7ee87a2 Silence log output when not verbose
Set the default log package output to iotuil.Discard during tests if the
`-v` flag isn't set. If we are verbose, then apply the filter according
to the TF_LOG env variable.
2016-08-01 17:19:14 -04:00
Paul Hinze b45f53eef4
dag: fix ReverseDepthFirstWalk when nodes remove themselves
The report in #7378 led us into a deep rabbit hole that turned out to
expose a bug in the graph walk implementation being used by the
`NoopTransformer`. The problem ended up being when two nodes in a single
dependency chain both reported `Noop() -> true` and needed to be
removed. This was breaking the walk and preventing the second node from
ever being visited.

Fixes #7378
2016-07-15 13:43:28 -06:00