diff --git a/tools/terraform-bundle/README.md b/tools/terraform-bundle/README.md new file mode 100644 index 000000000..5e697f528 --- /dev/null +++ b/tools/terraform-bundle/README.md @@ -0,0 +1,156 @@ +# terraform-bundle + +`terraform-bundle` is a helper program to create "bundle archives", which are +zip files that contain both a particular version of Terraform and a number +of provider plugins. + +Normally `terraform init` will download and install the plugins necessary to +work with a particular configuration, but sometimes Terraform is deployed in +a network that, for one reason or another, cannot access the official +plugin repository for automatic download. + +`terraform-bundle` provides an alternative, by allowing the auto-download +process to be run out-of-band on a separate machine that _does_ have access +to the repository. The result is a zip file that can be extracted onto the +target system to install both the desired Terraform version and a selection +of providers, thus avoiding the need for on-the-fly plugin installation. + +## Building + +To build `terraform-bundle` from source, set up a Terraform development +environment per [Terraform's own README](../../README.md) and then install +this tool from within it: + +``` +$ go install ./tools/terraform-bundle +``` + +This will install `terraform-bundle` in `$GOPATH/bin`, which is assumed by +the rest of this README to be in `PATH`. + +## Usage + +`terraform-bundle` uses a simple configuration file to define what should +be included in a bundle. This is designed so that it can be checked into +version control and used by an automated build and deploy process. + +The configuration file format works as follows: + +```hcl +terraform { + # Version of Terraform to include in the bundle. An exact version number + # is required. + version = "0.10.0" +} + +# Define which provider plugins are to be included +providers { + # Include the newest "aws" provider version in the 1.0 series. + aws = ["~> 1.0"] + + # Include both the newest 1.0 and 2.0 versions of the "google" provider. + # Each item in these lists allows a distinct version to be added. If the + # two expressions match different versions then _both_ are included in + # the bundle archive. + google = ["~> 1.0", "~> 2.0"] +} + +``` + +The `terraform` block defines which version of Terraform will be included +in the bundle. An exact version is required here. + +The `providers` block defines zero or more providers to include in the bundle +along with core Terraform. Each attribute in this block is a provider name, +and its value is a list of version constraints. For each given constraint, +`terraform-bundle` will find the newest available version matching the +constraint and include it in the bundle. + +It is allowed to specify multiple constraints for the same provider, in which +case multiple versions can be included in the resulting bundle. Each constraint +string given results in a separate plugin in the bundle, unless two constraints +resolve to the same concrete plugin. + +Including multiple versions of the same provider allows several configurations +running on the same system to share an installation of the bundle and to +choose a version using version constraints within the main Terraform +configuration. This avoids the need to upgrade all configurations to newer +versions in lockstep. + +After creating the configuration file, e.g. `terraform-bundle.hcl`, a bundle +zip file can be produced as follows: + +``` +$ terraform-bundle package terraform-bundle.hcl +``` + +By default the bundle package will target the operating system and CPU +architecture where the tool is being run. To override this, use the `-os` and +`-arch` options. For example, to build a bundle for on-premises Terraform +Enterprise: + +``` +$ terraform-bundle package -os=linux -arch=amd64 terraform-bundle.hcl +``` + +The bundle file is assigned a name that includes the core Terraform version +number, a timestamp to the nearest hour of when the bundle was built, and the +target OS and CPU architecture. It is recommended to refer to a bundle using +this composite version number so that bundle archives can be easily +distinguished from official release archives and from each other when multiple +bundles contain the same core Terraform version. + +## Provider Resolution Behavior + +Terraform's provider resolution behavior is such that if a given constraint +can be resolved by any plugin already installed on the system it will use +the newest matching plugin and not attempt automatic installation. + +Therefore if automatic installation is not desired, it is important to ensure +that version constraints within Terraform configurations do not exclude all +of the versions available from the bundle. If a suitable version cannot be +found in the bundle, Terraform _will_ attempt to satisfy that dependency by +automatic installation from the official repository. + +To disable automatic installation altogether -- and thus cause a hard failure +if no local plugins match -- the `-plugin-dir` option can be passed to +`terraform init`, giving the directory into which the bundle was extracted. +The presence of this option overrides all of the normal automatic discovery +and installation behavior, and thus forces the use of only the plugins that +can be found in the directory indicated. + +The downloaded provider archives are verified using the same signature check +that is used for auto-installed plugins, using Hashicorp's release key. At +this time, the core Terraform archive itself is _not_ verified in this way; +that may change in a future version of this tool. + +## Installing a Bundle in On-premises Terraform Enterprise + +If using a private install of Terraform Enterprise in an "air-gapped" +environment, this tool can produce a custom _tool package_ for Terraform, which +includes a set of provider plugins along with core Terraform. + +To create a suitable bundle, use the `-os` and `-arch` options as described +above to produce a bundle targeting `linux_amd64`. You can then place this +archive on an HTTP server reachable by the Terraform Enterprise hosts and +install it as per +[Managing Tool Versions](https://github.com/hashicorp/terraform-enterprise-modules/blob/master/docs/managing-tool-versions.md). + +After choosing the "Add Tool Version" button, be sure to set the Tool to +"terraform" and then enter as the Version the generated bundle version from +the bundle filename, which will be of the form `N.N.N-bundleYYYYMMDDHH`. +Enter the URL at which the generated bundle archive can be found, and the +SHA256 hash of the file which can be determined by running the tool +`sha256sum` with the given file. + +The new bundle version can then be selected as the Terraform version for +any workspace. When selected, configurations that require only plugins +included in the bundle will run without trying to auto-install. + +Note that the above does _not_ apply to Terraform Pro, or to Terraform Premium +when not running a private install. In these packages, Terraform versions +are managed centrally across _all_ organizations and so custom bundles are not +supported. + +For more information on the available Terraform Enterprise packages, see +[the Terraform product site](https://www.hashicorp.com/products/terraform/). diff --git a/tools/terraform-bundle/config.go b/tools/terraform-bundle/config.go new file mode 100644 index 000000000..8f493e0c5 --- /dev/null +++ b/tools/terraform-bundle/config.go @@ -0,0 +1,62 @@ +package main + +import ( + "fmt" + "io/ioutil" + + "github.com/hashicorp/hcl" + "github.com/hashicorp/terraform/plugin/discovery" +) + +type Config struct { + Terraform TerraformConfig `hcl:"terraform"` + Providers map[string][]discovery.ConstraintStr `hcl:"providers"` +} + +type TerraformConfig struct { + Version discovery.VersionStr `hcl:"version"` +} + +func LoadConfig(src []byte, filename string) (*Config, error) { + config := &Config{} + err := hcl.Decode(config, string(src)) + if err != nil { + return config, err + } + + err = config.validate() + return config, err +} + +func LoadConfigFile(filename string) (*Config, error) { + src, err := ioutil.ReadFile(filename) + if err != nil { + return nil, err + } + + return LoadConfig(src, filename) +} + +func (c *Config) validate() error { + if c.Terraform.Version == "" { + return fmt.Errorf("terraform.version is required") + } + + if _, err := c.Terraform.Version.Parse(); err != nil { + return fmt.Errorf("terraform.version: %s", err) + } + + if c.Providers == nil { + c.Providers = map[string][]discovery.ConstraintStr{} + } + + for k, cs := range c.Providers { + for _, c := range cs { + if _, err := c.Parse(); err != nil { + return fmt.Errorf("providers.%s: %s", k, err) + } + } + } + + return nil +} diff --git a/tools/terraform-bundle/main.go b/tools/terraform-bundle/main.go new file mode 100644 index 000000000..6556c5a75 --- /dev/null +++ b/tools/terraform-bundle/main.go @@ -0,0 +1,83 @@ +// terraform-bundle is a tool to create "bundle archives" that contain both +// a particular version of Terraform and a set of providers for use with it. +// +// Such bundles are useful for distributing a Terraform version and a set +// of providers to a system out-of-band, in situations where Terraform's +// auto-installer cannot be used due to firewall rules, "air-gapped" systems, +// etc. +// +// When using bundle archives, it's suggested to use a version numbering +// scheme that adds a suffix that identifies the archive as being a bundle, +// to make it easier to distinguish bundle archives from the normal separated +// release archives. This tool by default produces files with the following +// naming scheme: +// +// terraform_0.10.0-bundle2017070302_linux_amd64.zip +// +// The user is free to rename these files, since the archive filename has +// no significance to Terraform itself and the generated pseudo-version number +// is not referenced within the archive contents. +// +// If using such a bundle with an on-premises Terraform Enterprise installation, +// it's recommended to use the generated version number (or a modification +// thereof) as the tool version within Terraform Enterprise, so that +// bundle archives can be distinguished from official releases and from +// each other even if the same core Terraform version is used. +// +// Terraform providers in general release more often than core, so it is +// intended that this tool can be used to periodically upgrade providers +// within certain constraints and produce a new bundle containing these +// upgraded provider versions. A bundle archive can include multiple versions +// of the same provider, allowing configurations containing provider version +// constrants to be gradually migrated to newer versions. +package main + +import ( + "log" + "os" + + "io/ioutil" + + "github.com/mitchellh/cli" +) + +const Version = "0.0.1" + +func main() { + ui := &cli.ColoredUi{ + OutputColor: cli.UiColorNone, + InfoColor: cli.UiColorNone, + ErrorColor: cli.UiColorRed, + WarnColor: cli.UiColorYellow, + + Ui: &cli.BasicUi{ + Reader: os.Stdin, + Writer: os.Stdout, + ErrorWriter: os.Stderr, + }, + } + + // Terraform's code tends to produce noisy logs, since Terraform itself + // suppresses them by default. To avoid polluting our console, we'll do + // the same. + if os.Getenv("TF_LOG") == "" { + log.SetOutput(ioutil.Discard) + } + + c := cli.NewCLI("terraform-bundle", Version) + c.Args = os.Args[1:] + c.Commands = map[string]cli.CommandFactory{ + "package": func() (cli.Command, error) { + return &PackageCommand{ + ui: ui, + }, nil + }, + } + + exitStatus, err := c.Run() + if err != nil { + ui.Error(err.Error()) + } + + os.Exit(exitStatus) +} diff --git a/tools/terraform-bundle/package.go b/tools/terraform-bundle/package.go new file mode 100644 index 000000000..2f0f33a89 --- /dev/null +++ b/tools/terraform-bundle/package.go @@ -0,0 +1,237 @@ +package main + +import ( + "archive/zip" + "fmt" + "io/ioutil" + "os" + "path/filepath" + "runtime" + "time" + + "flag" + + "io" + + getter "github.com/hashicorp/go-getter" + "github.com/hashicorp/terraform/plugin" + "github.com/hashicorp/terraform/plugin/discovery" + "github.com/mitchellh/cli" +) + +const releasesBaseURL = "https://releases.hashicorp.com" + +type PackageCommand struct { + ui cli.Ui +} + +func (c *PackageCommand) Run(args []string) int { + flags := flag.NewFlagSet("package", flag.ExitOnError) + osPtr := flags.String("os", "", "Target operating system") + archPtr := flags.String("arch", "", "Target CPU architecture") + err := flags.Parse(args) + if err != nil { + c.ui.Error(err.Error()) + return 1 + } + + osName := runtime.GOOS + archName := runtime.GOARCH + if *osPtr != "" { + osName = *osPtr + } + if *archPtr != "" { + archName = *archPtr + } + + if flags.NArg() != 1 { + c.ui.Error("Configuration filename is required") + return 1 + } + configFn := flags.Arg(0) + + config, err := LoadConfigFile(configFn) + if err != nil { + c.ui.Error(fmt.Sprintf("Failed to read config: %s", err)) + return 1 + } + + if discovery.ConstraintStr("< 0.10.0-beta1").MustParse().Allows(config.Terraform.Version.MustParse()) { + c.ui.Error("Bundles can be created only for Terraform 0.10 or newer") + return 1 + } + + workDir, err := ioutil.TempDir("", "terraform-bundle") + if err != nil { + c.ui.Error(fmt.Sprintf("Could not create temporary dir: %s", err)) + return 1 + } + defer os.RemoveAll(workDir) + + c.ui.Info(fmt.Sprintf("Fetching Terraform %s core package...", config.Terraform.Version)) + + coreZipURL := c.coreURL(config.Terraform.Version, osName, archName) + err = getter.Get(workDir, coreZipURL) + if err != nil { + c.ui.Error(fmt.Sprintf("Failed to fetch core package from %s: %s", coreZipURL, err)) + } + + installer := &discovery.ProviderInstaller{ + Dir: workDir, + + // FIXME: This is incorrect because it uses the protocol version of + // this tool, rather than of the Terraform binary we just downloaded. + // But we can't get this information from a Terraform binary, so + // we'll just ignore this for now as we only have one protocol version + // in play anyway. If a new protocol version shows up later we will + // probably deal with this by just matching version ranges and + // hard-coding the knowledge of which Terraform version uses which + // protocol version. + PluginProtocolVersion: plugin.Handshake.ProtocolVersion, + + OS: osName, + Arch: archName, + } + + for name, constraints := range config.Providers { + c.ui.Info(fmt.Sprintf("Fetching provider %q...", name)) + for _, constraint := range constraints { + meta, err := installer.Get(name, constraint.MustParse()) + if err != nil { + c.ui.Error(fmt.Sprintf("Failed to resolve %s provider %s: %s", name, constraint, err)) + return 1 + } + + c.ui.Info(fmt.Sprintf("- %q resolved to %s", constraint, meta.Version)) + } + } + + files, err := ioutil.ReadDir(workDir) + if err != nil { + c.ui.Error(fmt.Sprintf("Failed to read work directory %s: %s", workDir, err)) + return 1 + } + + // If we get this far then our workDir now contains the union of the + // contents of all the zip files we downloaded above. We can now create + // our output file. + outFn := c.bundleFilename(config.Terraform.Version, time.Now(), osName, archName) + c.ui.Info(fmt.Sprintf("Creating %s ...", outFn)) + outF, err := os.OpenFile(outFn, os.O_TRUNC|os.O_CREATE|os.O_WRONLY, os.ModePerm) + if err != nil { + c.ui.Error(fmt.Sprintf("Failed to create %s: %s", outFn, err)) + return 1 + } + outZ := zip.NewWriter(outF) + defer func() { + err := outZ.Close() + if err != nil { + c.ui.Error(fmt.Sprintf("Failed to close %s: %s", outFn, err)) + os.Exit(1) + } + err = outF.Close() + if err != nil { + c.ui.Error(fmt.Sprintf("Failed to close %s: %s", outFn, err)) + os.Exit(1) + } + }() + + for _, file := range files { + if file.IsDir() { + // should never happen unless something tampers with our tmpdir + continue + } + + fn := filepath.Join(workDir, file.Name()) + r, err := os.Open(fn) + if err != nil { + c.ui.Error(fmt.Sprintf("Failed to open %s: %s", fn, err)) + return 1 + } + hdr, err := zip.FileInfoHeader(file) + if err != nil { + c.ui.Error(fmt.Sprintf("Failed to add zip entry for %s: %s", fn, err)) + return 1 + } + w, err := outZ.CreateHeader(hdr) + if err != nil { + c.ui.Error(fmt.Sprintf("Failed to add zip entry for %s: %s", fn, err)) + return 1 + } + _, err = io.Copy(w, r) + if err != nil { + c.ui.Error(fmt.Sprintf("Failed to write %s to bundle: %s", fn, err)) + return 1 + } + } + + c.ui.Info("All done!") + + return 0 +} + +func (c *PackageCommand) bundleFilename(version discovery.VersionStr, time time.Time, osName, archName string) string { + time = time.UTC() + return fmt.Sprintf( + "terraform_%s-bundle%04d%02d%02d%02d_%s_%s.zip", + version, + time.Year(), time.Month(), time.Day(), time.Hour(), + osName, archName, + ) +} + +func (c *PackageCommand) coreURL(version discovery.VersionStr, osName, archName string) string { + return fmt.Sprintf( + "%s/terraform/%s/terraform_%s_%s_%s.zip", + releasesBaseURL, version, version, osName, archName, + ) +} + +func (c *PackageCommand) Synopsis() string { + return "Produces a bundle archive" +} + +func (c *PackageCommand) Help() string { + return `Usage: terraform-bundle package [options] + +Uses the given bundle configuration file to produce a zip file in the +current working directory containing a Terraform binary along with zero or +more provider plugin binaries. + +Options: + -os=name Target operating system the archive will be built for. Defaults + to that of the system where the command is being run. + + -arch=name Target CPU architecture the archive will be built for. Defaults + to that of the system where the command is being run. + +The resulting zip file can be used to more easily install Terraform and +a fixed set of providers together on a server, so that Terraform's provider +auto-installation mechanism can be avoided. + +To build an archive for Terraform Enterprise, use: + -os=linux -arch=amd64 + +Note that the given configuration file is a format specific to this command, +not a normal Terraform configuration file. The file format looks like this: + + terraform { + # Version of Terraform to include in the bundle. An exact version number + # is required. + version = "0.10.0" + } + + # Define which provider plugins are to be included + providers { + # Include the newest "aws" provider version in the 1.0 series. + aws = ["~> 1.0"] + + # Include both the newest 1.0 and 2.0 versions of the "google" provider. + # Each item in these lists allows a distinct version to be added. If the + # two expressions match different versions then _both_ are included in + # the bundle archive. + google = ["~> 1.0", "~> 2.0"] + } + +` +}