Commit Graph

73 Commits

Author SHA1 Message Date
James Bardin 3225d9ac11 record all plugin panics, and print on main exit
Create a logger that will record any apparent crash output for later
processing.

If the cli command returns with a non-zero exit status, check for any
recorded crashes and add those to the output.
2020-10-26 09:34:03 -04:00
James Bardin f8893785f0 separate core and provider loggers
Now that hclog can independently set levels on related loggers, we can
separate the log levels for different subsystems in terraform.

This adds the new environment variables, `TF_LOG_CORE` and
`TF_LOG_PROVIDER`, which each take the same set of log level arguments,
and only applies to logs from that subsystem. This means that setting
`TF_LOG_CORE=level` will not show logs from providers, and
`TF_LOG_PROVIDER=level` will not show logs from core. The behavior of
`TF_LOG` alone does not change.

While it is not necessarily needed since the default is to disable logs,
there is also a new level argument of `off`, which reflects the
associated level in hclog.
2020-10-23 12:46:32 -04:00
James Bardin c2af5333e8 use a log sink to capture logs for panicwrap
Use a separate log sink to always capture trace logs for the panicwrap
handler to write out in a crash log.

This requires creating a log file in the outer process and passing that
path to the child process to log to.
2020-10-21 17:29:07 -04:00
James Bardin fd4f7eb0b9 remove prefixed io
The main process is now handling what output to print, so it doesn't do
any good to try and run it through prefixedio, which is only adding
extra coordination to echo the same data.
2020-10-21 17:29:07 -04:00
James Bardin 1d9d82973b move panicwrap handler to logging package 2020-10-21 13:47:16 -04:00
James Bardin 211edf5d75 use hclog as the default logger
Inject hclog as the default logger in the main binary.
2020-10-19 14:29:54 -04:00
James Bardin 6ca477f042 move helper/logging to internal
remove a dead code file too
2020-10-19 14:27:53 -04:00
Martin Atkins 30204ecded command/cliconfig: Allow development overrides for providers
For normal provider installation we want to associate each provider with
a selected version number and find a suitable package for that version
that conforms to the official hashes for that release.

Those requirements are very onerous for a provider developer currently
testing a not-yet-released build, though. To allow for that case this new
CLI configuration feature allows overriding specific providers to refer
to give local filesystem directories.

Any provider overridden in this way is not subject to the usual
restrictions about selected versions or checksum conformance, and
activating an override won't cause any changes to the selections recorded
in the lock file because it's intended to be a temporary setting for one
developer only.

This is, in a sense, a spiritual successor of an old capability we had to
override specific plugins in the CLI configuration file. There were
some vestiges of that left in the main package and CLI config package
but nothing has actually been honoring them for several versions now and
so this commit removes them to avoid confusion with the new mechanism.
2020-10-16 14:31:15 -07:00
Pam Selle 328baaad84 Fix un-saved error on chdir 2020-10-13 14:22:25 -04:00
Martin Atkins efe78b2910 main: new global option -chdir
This new option is intended to address the previous inconsistencies where
some older subcommands supported partially changing the target directory
(where Terraform would use the new directory inconsistently) where newer
commands did not support that override at all.

Instead, now Terraform will accept a -chdir command at the start of the
command line (before the subcommand) and will interpret it as a request
to direct all actions that would normally be taken in the current working
directory into the target directory instead. This is similar to options
offered by some other similar tools, such as the -C option in "make".

The new option is only accepted at the start of the command line (before
the subcommand) as a way to reflect that it is a global command (not
specific to a particular subcommand) and that it takes effect _before_
executing the subcommand. This also means it'll be forced to appear before
any other command-specific arguments that take file paths, which hopefully
communicates that those other arguments are interpreted relative to the
overridden path.

As a measure of pragmatism for existing uses, the path.cwd object in
the Terraform language will continue to return the _original_ working
directory (ignoring -chdir), in case that is important in some exceptional
workflows. The path.root object gives the root module directory, which
will always match the overriden working directory unless the user
simultaneously uses one of the legacy directory override arguments, which
is not a pattern we intend to support in the long run.

As a first step down the deprecation path, this commit adjusts the
documentation to de-emphasize the inconsistent old command line arguments,
including specific guidance on what to use instead for the main three
workflow commands, but all of those options remain supported in the same
way as they were before. In a later commit we'll make those arguments
produce a visible deprecation warning in Terraform's output, and then
in an even later commit we'll remove them entirely so that -chdir is the
single supported way to run Terraform from a directory other than the
one containing the root module configuration.
2020-09-04 15:31:08 -07:00
Alisdair McDiarmid f1f24df7ff main: Pass untyped nil for missing creds source
If we are unable to create a credentials source for some reason, we can
rely on the disco object to nil-check it before calling any of its
methods. However to do this we must ensure that we pass untyped nil.
This commit rearranges the initialization to ensure that this happens.

The user-facing bug that triggered this work is that running init when
the HOME environment variable is unset would result in a panic on macOS.
2020-06-03 09:46:53 -04:00
Paddy 5127f1ef8b
command: Unmanaged providers
This adds supports for "unmanaged" providers, or providers with process
lifecycles not controlled by Terraform. These providers are assumed to
be started before Terraform is launched, and are assumed to shut
themselves down after Terraform has finished running.

To do this, we must update the go-plugin dependency to v1.3.0, which
added support for the "test mode" plugin serving that powers all this.

As a side-effect of not needing to manage the process lifecycle anymore,
Terraform also no longer needs to worry about the provider's binary, as
it won't be used for anything anymore. Because of this, we can disable
the init behavior that concerns itself with downloading that provider's
binary, checking its version, and otherwise managing the binary.

This is all managed on a per-provider basis, so managed providers that
Terraform downloads, starts, and stops can be used in the same commands
as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable
is added, and is a JSON encoding of the provider's address to the
information we need to connect to it.

This change enables two benefits: first, delve and other debuggers can
now be attached to provider server processes, and Terraform can connect.
This allows for attaching debuggers to provider processes, which before
was difficult to impossible. Second, it allows the SDK test framework to
host the provider in the same process as the test driver, while running
a production Terraform binary against the provider. This allows for Go's
built-in race detector and test coverage tooling to work as expected in
provider tests.

Unmanaged providers are expected to work in the exact same way as
managed providers, with one caveat: Terraform kills provider processes
and restarts them once per graph walk, meaning multiple times during
most Terraform CLI commands. As unmanaged providers can't be killed by
Terraform, and have no visibility into graph walks, unmanaged providers
are likely to have differences in how their global mutable state behaves
when compared to managed providers. Namely, unmanaged providers are
likely to retain global state when managed providers would have reset
it. Developers relying on global state should be aware of this.
2020-05-26 17:48:57 -07:00
Martin Atkins 94b87e056b fixup main.go comment about providersource 2020-04-23 10:52:01 -07:00
Martin Atkins 5af1e6234a main: Honor explicit provider_installation CLI config when present
If the CLI configuration contains a provider_installation block then we'll
use the source configuration it describes instead of the implied one we'd
build otherwise.
2020-04-21 16:28:59 -07:00
Martin Atkins 8c928e8358 main: Consult local directories as potential mirrors of providers
This restores some of the local search directories we used to include when
searching for provider plugins in Terraform 0.12 and earlier. The
directory structures we are expecting in these are different than before,
so existing directory contents will not be compatible without
restructuring, but we need to retain support for these local directories
so that users can continue to sideload third-party provider plugins until
the explicit, first-class provider mirrors configuration (in CLI config)
is implemented, at which point users will be able to override these to
whatever directories they want.

This also includes some new search directories that are specific to the
operating system where Terraform is running, following the documented
layout conventions of that platform. In particular, this follows the
XDG Base Directory specification on Unix systems, which has been a
somewhat-common request to better support "sideloading" of packages via
standard Linux distribution package managers and other similar mechanisms.
While it isn't strictly necessary to add that now, it seems ideal to do
all of the changes to our search directory layout at once so that our
documentation about this can cleanly distinguish "0.12 and earlier" vs.
"0.13 and later", rather than having to document a complex sequence of
smaller changes.

Because this behavior is a result of the integration of package main with
package command, this behavior is verified using an e2etest rather than
a unit test. That test, TestInitProvidersVendored, is also fixed here to
create a suitable directory structure for the platform where the test is
being run. This fixes TestInitProvidersVendored.
2020-04-06 09:24:23 -07:00
Martin Atkins e9d0822b2a command: Accept a "provider source" from the main package
Following the same approach we use for other CLI-Config-able objects like
the service discovery system, the main package is responsible for
producing a suitable implementation of this interface which the command
package can then use.

When unit testing in the command package we can then substitute mocks as
necessary, following the dependency inversion principle.
2020-01-24 13:45:37 -08:00
Pam Selle 78b1220558 Remove config.go and update things using its aliases 2020-01-13 16:50:05 -05:00
Radek Simko 32f9722d9d
Replace import paths & set UA string where necessary 2019-10-11 22:40:54 +01:00
Martin Atkins cfc1c4900d command/login: Use Cli.Ask to request confirmation
This is more straightforward than using readline because it already works
properly with panicwrap.
2019-09-09 11:15:24 -07:00
Martin Atkins 22a2580e93 main: Use the new cliconfig package credentials source
This should not cause any change in behavior yet, but using this new
implementation will allow the "terraform login" and "terraform logout"
commands to store and forget credentials when they are implemented in
subsequent commits.
2019-08-23 11:57:11 -07:00
Martin Atkins e1590d0a70 command/cliconfig: Factor out CLI config handling
This is just a wholesale move of the CLI configuration types and functions
from the main package into its own package, leaving behind some type
aliases and wrappers for now to keep existing callers working.

This commit alone doesn't really achieve anything, but in future commits
we'll expand the functionality in this package.
2019-08-01 10:56:41 -07:00
Matthew Sanabria be04e70a0f
Adding documentation for TF_CLI_CONFIG_FILE environment variable (#20834)
Fixes #15849
2019-04-05 14:21:40 -04:00
Sander van Harmelen 52a1b22f7a Implement the remote enhanced backend
This is a refactored version of the `remote` backend that was initially added to Terraform v0.11.8 which should now be compatible with v0.12.0.
2018-11-06 16:29:46 +01:00
Martin Atkins a3403f2766 terraform: Ugly huge change to weave in new State and Plan types
Due to how often the state and plan types are referenced throughout
Terraform, there isn't a great way to switch them out gradually. As a
consequence, this huge commit gets us from the old world to a _compilable_
new world, but still has a large number of known test failures due to
key functionality being stubbed out.

The stubs here are for anything that interacts with providers, since we
now need to do the follow-up work to similarly replace the old
terraform.ResourceProvider interface with its replacement in the new
"providers" package. That work, along with work to fix the remaining
failing tests, will follow in subsequent commits.

The aim here was to replace all references to terraform.State and its
downstream types with states.State, terraform.Plan with plans.Plan,
state.State with statemgr.State, and switch to the new implementations of
the state and plan file formats. However, due to the number of times those
types are used, this also ended up affecting numerous other parts of core
such as terraform.Hook, the backend.Backend interface, and most of the CLI
commands.

Just as with 5861dbf3fc49b19587a31816eb06f511ab861bb4 before, I apologize
in advance to the person who inevitably just found this huge commit while
spelunking through the commit history.
2018-10-16 19:11:09 -07:00
Martin Atkins bd10b84a8e command/format: include source snippets in diagnostics
If we get a diagnostic message that references a source range, and if the
source code for the referenced file is available, we'll show a snippet of
the source code with the source range highlighted.

At the moment we have no cache of source code, so in practice this
codepath can never be visited. Callers to format.Diagnostic will be
gradually updated in subsequent commits.
2018-10-16 18:20:32 -07:00
Sander van Harmelen 7fb2d1b8de Implement the Enterprise enhanced remote backend 2018-08-03 22:22:55 +02:00
Sander van Harmelen 179b32d426 Add a `CredentialsForHost` method to disco.Disco
By adding this method you now only have to pass a `*disco.Disco` object around in order to do discovery and use any configured credentials for the discovered hosts.

Of course you can also still pass around both a `*disco.Disco` and a `auth.CredentialsSource` object if there is a need or a reason for that!
2018-08-03 11:29:11 +02:00
Martin Atkins 275ab4a74e main: don't print the CLI config into the logs
Now that we're expecting "credentials" blocks in the config (with auth
tokens for private module registries, etc) we should not print out the
config contents into the log, or else people will probably end up
accidentally disclosing their credentials when sharing debug output with
us, or will be reluctant to share debug output.
2017-11-14 15:49:26 -08:00
Martin Atkins 11ba1d2a4c main: factor out CLI config loading into its own function
Previously we handled all of the config sources directly within the main
function. We're going to make CLI config loading more complex shortly, so
having this encapsulated in its own function will avoid creating even more
clutter inside the main function.

Along the way here we also switch from using native Go "error" to using
tfdiags.Diagnostics, so that we can potentially issue warnings here too
in future, and so that we can return multiple errors.
2017-10-21 09:37:05 -07:00
Martin Atkins a2c59c6ecd main: validate credentials blocks in CLI config
We require that each "credentials" block has a valid hostname and that
there be no more than one "credentials_helper" block.

There are some more sophisticated validations we could do here, such as
checking if the same host is declared more than once, but since this
config handling will be rewritten to use HCL2 in the near future, and this
sort of check is easier to do in the HCL2 API, we just check the basic
stuff for now and plan to revisit later.
2017-10-21 09:37:05 -07:00
Martin Atkins cb17a9a607 main: allow enabling plugin caching via config file or environment
Either the environment variable TF_PLUGIN_CACHE_DIR or a setting in the
CLI config file (~/.terraformrc on Unix) allow opting in to the plugin
caching behavior.

This is opt-in because for new users we don't want to pollute their system
with extra directories they don't know about. By opting in to caching, the
user is assuming the responsibility to occasionally prune the cache over
time as older plugins become stale and unused.
2017-09-29 14:03:09 -07:00
Martin Atkins 3f401f0cd4 main: make configuration available when initializing commands
This, in principle, allows us to make use of configuration information
when we populate the Meta structure, though we won't actually make use
of that until a subsequent commit.
2017-09-29 14:03:09 -07:00
Martin Atkins 9b5ae9143a main: enable basic subcommand autocomplete
The CLI package has automatic support for shell autocomplete (bash and
zsh, at time of writing) for subcommands, so all we need to do here is
just opt into it.

Users can install this into their shells by running:
    terraform -install-autocomplete
2017-09-26 14:01:13 -07:00
Martin Atkins 8364383c35 Push plugin discovery down into command package
Previously we did plugin discovery in the main package, but as we move
towards versioned plugins we need more information available in order to
resolve plugins, so we move this responsibility into the command package
itself.

For the moment this is just preserving the existing behavior as long as
there are only internal and unversioned plugins present. This is the
final state for provisioners in 0.10, since we don't want to support
versioned provisioners yet. For providers this is just a checkpoint along
the way, since further work is required to apply version constraints from
configuration and support additional plugin search directories.

The automatic plugin discovery behavior is not desirable for tests because
we want to mock the plugins there, so we add a new backdoor for the tests
to use to skip the plugin discovery and just provide their own mock
implementations. Most of this diff is thus noisy rework of the tests to
use this new mechanism.
2017-06-09 14:03:59 -07:00
Martin Atkins 5ac311e2a9 main: synchronize writes to VT100-faker on Windows
We use a third-party library "colorable" to translate VT100 color
sequences into Windows console attribute-setting calls when Terraform is
running on Windows.

colorable is not concurrency-safe for multiple writes to the same console,
because it writes to the console one character at a time and so two
concurrent writers get their characters interleaved, creating unreadable
garble.

Here we wrap around it a synchronization mechanism to ensure that there
can be only one Write call outstanding across both stderr and stdout,
mimicking the usual behavior we expect (when stderr/stdout are a normal
file handle) of each Write being completed atomically.
2017-05-04 15:36:51 -07:00
Mitchell Hashimoto 3dfc46abd3
log the Go runtime version at TF startup
This will help us debug issues which have been caused by Go bugs.
2017-02-15 16:11:42 -08:00
Mitchell Hashimoto 518ae5ef02
support nested subcommands with TF_CLI_ARGS 2017-02-13 15:18:50 -08:00
Mitchell Hashimoto df93e5120c
allow targeted TF_CLI_ARGS_x 2017-02-13 15:12:29 -08:00
Mitchell Hashimoto 53796fcdb0
test for blank args for TF_CLI_ARGS 2017-02-13 14:53:50 -08:00
Mitchell Hashimoto 7f67b32169
main: add TF_CLI_ARGS to specify additional CLI args 2017-02-13 14:05:37 -08:00
Mitchell Hashimoto e9c35eae32
Forward SIGTERM and handle that as an interrupt 2016-12-08 12:20:25 -05:00
Mitchell Hashimoto 557375f2a2
Update panicwrap to pass through all interrupt signals 2016-12-07 20:51:45 -05:00
Justin Nauman bf48d0132c Checkpoint signature fixes
- Currently the disable_checkpoint setting from $HOME/.terraformrc is never
respsected due to:
-- The runCheckpoint go routine being fired off prior to loading configuration
-- The config.Merge method not actually merging in the c2s settings
2016-11-17 17:54:14 -06:00
James Bardin 797a1b339d DebugInfo and DebugGraph
Implement debugInfo and the DebugGraph

DebugInfo will be a global variable through which graph debug
information can we written to a compressed archive. The DebugInfo
methods are all safe for concurrent use, and noop with a nil receiver.
The API outside of the terraform package will be to call SetDebugInfo
to create the archive, and CloseDebugInfo() to properly close the file.
Each write to the archive will be flushed and sync'ed individually, so
in the event of a crash or a missing call to Close, the archive can
still be recovered.

The DebugGraph is a representation of a terraform Graph to be written to
the debug archive, currently in dot format. The DebugGraph also contains
an internal buffer with Printf and Write methods to add to this buffer.
The buffer will be written to an accompanying file in the debug archive
along with the graph.

This also adds a GraphNodeDebugger interface. Any node implementing
`NodeDebug() string` can output information to annotate the debug graph
node, and add the data to the log. This interface may change or be
removed to provide richer options for debugging graph nodes.

The new graph builders all delegate the build to the BasicGraphBuilder.
Having a Name field lets us differentiate the actual builder
implementation in the debug graphs.
2016-11-04 11:30:51 -04:00
James Bardin 6b7a079bff Add a way to avoid panicwrap when debugging
When trying to use a debugger, you don't want the terraform to run in a
new child process. Setting TF_FORK=0 will skip panicwrap and continue
running the program in the current process. This will also give direct
access to log output, and stdout.
2016-09-21 14:13:05 -04:00
Mitchell Hashimoto b8036095d4
Logs contain CLI args 2016-08-17 07:50:18 -07:00
Chris Bednarski 3c774af9c2 Warn when an internal plugin is overridden
Also added documentation explaining what happened and how to fix it
2016-05-10 14:49:13 -04:00
Chris Bednarski e942a74def Set a log prefix for each plugin and remove go-dynect global log prefix (#6336) 2016-05-10 14:49:13 -04:00
Mitchell Hashimoto d1b46e99bd Add `terraform state list` command
This introduces the terraform state list command to list the resources
within a state. This is the first of many state management commands to
come into 0.7.

This is the first command of many to come that is considered a
"plumbing" command within Terraform (see "plumbing vs porcelain":
http://git.661346.n2.nabble.com/what-are-plumbing-and-porcelain-td2190639.html).
As such, this PR also introduces a bunch of groundwork to support
plumbing commands.

The main changes:

- Main command output is changed to split "common" and "uncommon"
  commands.

- mitchellh/cli is updated to support nested subcommands, since
  terraform state list is a nested subcommand.

- terraform.StateFilter is introduced as a way in core to filter/search
  the state files. This is very basic currently but I expect to make it
  more advanced as time goes on.

- terraform state list command is introduced to list resources in a
  state. This can take a series of arguments to filter this down.

Known issues, or things that aren't done in this PR on purpose:

- Unit tests for terraform state list are on the way. Unit tests for the
  core changes are all there.
2016-05-10 14:14:47 -04:00
James Nugent 8e4da4e2a1 deps: Vendor github.com/hashicorp/go-plugin 2016-05-10 14:14:47 -04:00