terraform-bundle tool for bundling Terraform with providers
Normally "terraform init" will download and install the plugins necessary to work with a particular configuration, but sometimes Terraform is deployed in a network that, for one reason or another, cannot access the official plugin repository for automatic download. terraform-bundle provides an alternative method, allowing the auto-download process to be run out-of-band on a separate machine that _does_ have access to the repository. The result is a zip file that can be extracted onto the target system to install both the desired Terraform version and a selection of providers, thus avoiding the need for on-the-fly plugin installation. This is provided as a separate tool from Terraform because it is not something that most users will need. In the rare case where this is needed, we will for the moment assume that users are able to build this tool themselves. We may later release it in a pre-built form, if it proves to be generally useful. It uses the same API from the plugin/discovery package is is used by the auto-install behavior in "terraform init", so plugin versions are resolved in the same way. However, it's expected that several different Terraform configurations will run from the same bundle, so this tool allows the bundle to include potentially many versions of the same provider and thus allows each Terraform configuration to select from the available versions in the bundle, avoiding the need to upgrade all configurations to new provider versions in lockstep.
This commit is contained in:
Martin Atkins
parent
610fcb605e
commit
909989acfa
|
|
@ -0,0 +1,156 @@
|
|||
# terraform-bundle
|
||||
|
||||
`terraform-bundle` is a helper program to create "bundle archives", which are
|
||||
zip files that contain both a particular version of Terraform and a number
|
||||
of provider plugins.
|
||||
|
||||
Normally `terraform init` will download and install the plugins necessary to
|
||||
work with a particular configuration, but sometimes Terraform is deployed in
|
||||
a network that, for one reason or another, cannot access the official
|
||||
plugin repository for automatic download.
|
||||
|
||||
`terraform-bundle` provides an alternative, by allowing the auto-download
|
||||
process to be run out-of-band on a separate machine that _does_ have access
|
||||
to the repository. The result is a zip file that can be extracted onto the
|
||||
target system to install both the desired Terraform version and a selection
|
||||
of providers, thus avoiding the need for on-the-fly plugin installation.
|
||||
|
||||
## Building
|
||||
|
||||
To build `terraform-bundle` from source, set up a Terraform development
|
||||
environment per [Terraform's own README](../../README.md) and then install
|
||||
this tool from within it:
|
||||
|
||||
```
|
||||
$ go install ./tools/terraform-bundle
|
||||
```
|
||||
|
||||
This will install `terraform-bundle` in `$GOPATH/bin`, which is assumed by
|
||||
the rest of this README to be in `PATH`.
|
||||
|
||||
## Usage
|
||||
|
||||
`terraform-bundle` uses a simple configuration file to define what should
|
||||
be included in a bundle. This is designed so that it can be checked into
|
||||
version control and used by an automated build and deploy process.
|
||||
|
||||
The configuration file format works as follows:
|
||||
|
||||
```hcl
|
||||
terraform {
|
||||
# Version of Terraform to include in the bundle. An exact version number
|
||||
# is required.
|
||||
version = "0.10.0"
|
||||
}
|
||||
|
||||
# Define which provider plugins are to be included
|
||||
providers {
|
||||
# Include the newest "aws" provider version in the 1.0 series.
|
||||
aws = ["~> 1.0"]
|
||||
|
||||
# Include both the newest 1.0 and 2.0 versions of the "google" provider.
|
||||
# Each item in these lists allows a distinct version to be added. If the
|
||||
# two expressions match different versions then _both_ are included in
|
||||
# the bundle archive.
|
||||
google = ["~> 1.0", "~> 2.0"]
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
The `terraform` block defines which version of Terraform will be included
|
||||
in the bundle. An exact version is required here.
|
||||
|
||||
The `providers` block defines zero or more providers to include in the bundle
|
||||
along with core Terraform. Each attribute in this block is a provider name,
|
||||
and its value is a list of version constraints. For each given constraint,
|
||||
`terraform-bundle` will find the newest available version matching the
|
||||
constraint and include it in the bundle.
|
||||
|
||||
It is allowed to specify multiple constraints for the same provider, in which
|
||||
case multiple versions can be included in the resulting bundle. Each constraint
|
||||
string given results in a separate plugin in the bundle, unless two constraints
|
||||
resolve to the same concrete plugin.
|
||||
|
||||
Including multiple versions of the same provider allows several configurations
|
||||
running on the same system to share an installation of the bundle and to
|
||||
choose a version using version constraints within the main Terraform
|
||||
configuration. This avoids the need to upgrade all configurations to newer
|
||||
versions in lockstep.
|
||||
|
||||
After creating the configuration file, e.g. `terraform-bundle.hcl`, a bundle
|
||||
zip file can be produced as follows:
|
||||
|
||||
```
|
||||
$ terraform-bundle package terraform-bundle.hcl
|
||||
```
|
||||
|
||||
By default the bundle package will target the operating system and CPU
|
||||
architecture where the tool is being run. To override this, use the `-os` and
|
||||
`-arch` options. For example, to build a bundle for on-premises Terraform
|
||||
Enterprise:
|
||||
|
||||
```
|
||||
$ terraform-bundle package -os=linux -arch=amd64 terraform-bundle.hcl
|
||||
```
|
||||
|
||||
The bundle file is assigned a name that includes the core Terraform version
|
||||
number, a timestamp to the nearest hour of when the bundle was built, and the
|
||||
target OS and CPU architecture. It is recommended to refer to a bundle using
|
||||
this composite version number so that bundle archives can be easily
|
||||
distinguished from official release archives and from each other when multiple
|
||||
bundles contain the same core Terraform version.
|
||||
|
||||
## Provider Resolution Behavior
|
||||
|
||||
Terraform's provider resolution behavior is such that if a given constraint
|
||||
can be resolved by any plugin already installed on the system it will use
|
||||
the newest matching plugin and not attempt automatic installation.
|
||||
|
||||
Therefore if automatic installation is not desired, it is important to ensure
|
||||
that version constraints within Terraform configurations do not exclude all
|
||||
of the versions available from the bundle. If a suitable version cannot be
|
||||
found in the bundle, Terraform _will_ attempt to satisfy that dependency by
|
||||
automatic installation from the official repository.
|
||||
|
||||
To disable automatic installation altogether -- and thus cause a hard failure
|
||||
if no local plugins match -- the `-plugin-dir` option can be passed to
|
||||
`terraform init`, giving the directory into which the bundle was extracted.
|
||||
The presence of this option overrides all of the normal automatic discovery
|
||||
and installation behavior, and thus forces the use of only the plugins that
|
||||
can be found in the directory indicated.
|
||||
|
||||
The downloaded provider archives are verified using the same signature check
|
||||
that is used for auto-installed plugins, using Hashicorp's release key. At
|
||||
this time, the core Terraform archive itself is _not_ verified in this way;
|
||||
that may change in a future version of this tool.
|
||||
|
||||
## Installing a Bundle in On-premises Terraform Enterprise
|
||||
|
||||
If using a private install of Terraform Enterprise in an "air-gapped"
|
||||
environment, this tool can produce a custom _tool package_ for Terraform, which
|
||||
includes a set of provider plugins along with core Terraform.
|
||||
|
||||
To create a suitable bundle, use the `-os` and `-arch` options as described
|
||||
above to produce a bundle targeting `linux_amd64`. You can then place this
|
||||
archive on an HTTP server reachable by the Terraform Enterprise hosts and
|
||||
install it as per
|
||||
[Managing Tool Versions](https://github.com/hashicorp/terraform-enterprise-modules/blob/master/docs/managing-tool-versions.md).
|
||||
|
||||
After choosing the "Add Tool Version" button, be sure to set the Tool to
|
||||
"terraform" and then enter as the Version the generated bundle version from
|
||||
the bundle filename, which will be of the form `N.N.N-bundleYYYYMMDDHH`.
|
||||
Enter the URL at which the generated bundle archive can be found, and the
|
||||
SHA256 hash of the file which can be determined by running the tool
|
||||
`sha256sum` with the given file.
|
||||
|
||||
The new bundle version can then be selected as the Terraform version for
|
||||
any workspace. When selected, configurations that require only plugins
|
||||
included in the bundle will run without trying to auto-install.
|
||||
|
||||
Note that the above does _not_ apply to Terraform Pro, or to Terraform Premium
|
||||
when not running a private install. In these packages, Terraform versions
|
||||
are managed centrally across _all_ organizations and so custom bundles are not
|
||||
supported.
|
||||
|
||||
For more information on the available Terraform Enterprise packages, see
|
||||
[the Terraform product site](https://www.hashicorp.com/products/terraform/).
|
||||
|
|
@ -0,0 +1,62 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io/ioutil"
|
||||
|
||||
"github.com/hashicorp/hcl"
|
||||
"github.com/hashicorp/terraform/plugin/discovery"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
Terraform TerraformConfig `hcl:"terraform"`
|
||||
Providers map[string][]discovery.ConstraintStr `hcl:"providers"`
|
||||
}
|
||||
|
||||
type TerraformConfig struct {
|
||||
Version discovery.VersionStr `hcl:"version"`
|
||||
}
|
||||
|
||||
func LoadConfig(src []byte, filename string) (*Config, error) {
|
||||
config := &Config{}
|
||||
err := hcl.Decode(config, string(src))
|
||||
if err != nil {
|
||||
return config, err
|
||||
}
|
||||
|
||||
err = config.validate()
|
||||
return config, err
|
||||
}
|
||||
|
||||
func LoadConfigFile(filename string) (*Config, error) {
|
||||
src, err := ioutil.ReadFile(filename)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return LoadConfig(src, filename)
|
||||
}
|
||||
|
||||
func (c *Config) validate() error {
|
||||
if c.Terraform.Version == "" {
|
||||
return fmt.Errorf("terraform.version is required")
|
||||
}
|
||||
|
||||
if _, err := c.Terraform.Version.Parse(); err != nil {
|
||||
return fmt.Errorf("terraform.version: %s", err)
|
||||
}
|
||||
|
||||
if c.Providers == nil {
|
||||
c.Providers = map[string][]discovery.ConstraintStr{}
|
||||
}
|
||||
|
||||
for k, cs := range c.Providers {
|
||||
for _, c := range cs {
|
||||
if _, err := c.Parse(); err != nil {
|
||||
return fmt.Errorf("providers.%s: %s", k, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
|
@ -0,0 +1,83 @@
|
|||
// terraform-bundle is a tool to create "bundle archives" that contain both
|
||||
// a particular version of Terraform and a set of providers for use with it.
|
||||
//
|
||||
// Such bundles are useful for distributing a Terraform version and a set
|
||||
// of providers to a system out-of-band, in situations where Terraform's
|
||||
// auto-installer cannot be used due to firewall rules, "air-gapped" systems,
|
||||
// etc.
|
||||
//
|
||||
// When using bundle archives, it's suggested to use a version numbering
|
||||
// scheme that adds a suffix that identifies the archive as being a bundle,
|
||||
// to make it easier to distinguish bundle archives from the normal separated
|
||||
// release archives. This tool by default produces files with the following
|
||||
// naming scheme:
|
||||
//
|
||||
// terraform_0.10.0-bundle2017070302_linux_amd64.zip
|
||||
//
|
||||
// The user is free to rename these files, since the archive filename has
|
||||
// no significance to Terraform itself and the generated pseudo-version number
|
||||
// is not referenced within the archive contents.
|
||||
//
|
||||
// If using such a bundle with an on-premises Terraform Enterprise installation,
|
||||
// it's recommended to use the generated version number (or a modification
|
||||
// thereof) as the tool version within Terraform Enterprise, so that
|
||||
// bundle archives can be distinguished from official releases and from
|
||||
// each other even if the same core Terraform version is used.
|
||||
//
|
||||
// Terraform providers in general release more often than core, so it is
|
||||
// intended that this tool can be used to periodically upgrade providers
|
||||
// within certain constraints and produce a new bundle containing these
|
||||
// upgraded provider versions. A bundle archive can include multiple versions
|
||||
// of the same provider, allowing configurations containing provider version
|
||||
// constrants to be gradually migrated to newer versions.
|
||||
package main
|
||||
|
||||
import (
|
||||
"log"
|
||||
"os"
|
||||
|
||||
"io/ioutil"
|
||||
|
||||
"github.com/mitchellh/cli"
|
||||
)
|
||||
|
||||
const Version = "0.0.1"
|
||||
|
||||
func main() {
|
||||
ui := &cli.ColoredUi{
|
||||
OutputColor: cli.UiColorNone,
|
||||
InfoColor: cli.UiColorNone,
|
||||
ErrorColor: cli.UiColorRed,
|
||||
WarnColor: cli.UiColorYellow,
|
||||
|
||||
Ui: &cli.BasicUi{
|
||||
Reader: os.Stdin,
|
||||
Writer: os.Stdout,
|
||||
ErrorWriter: os.Stderr,
|
||||
},
|
||||
}
|
||||
|
||||
// Terraform's code tends to produce noisy logs, since Terraform itself
|
||||
// suppresses them by default. To avoid polluting our console, we'll do
|
||||
// the same.
|
||||
if os.Getenv("TF_LOG") == "" {
|
||||
log.SetOutput(ioutil.Discard)
|
||||
}
|
||||
|
||||
c := cli.NewCLI("terraform-bundle", Version)
|
||||
c.Args = os.Args[1:]
|
||||
c.Commands = map[string]cli.CommandFactory{
|
||||
"package": func() (cli.Command, error) {
|
||||
return &PackageCommand{
|
||||
ui: ui,
|
||||
}, nil
|
||||
},
|
||||
}
|
||||
|
||||
exitStatus, err := c.Run()
|
||||
if err != nil {
|
||||
ui.Error(err.Error())
|
||||
}
|
||||
|
||||
os.Exit(exitStatus)
|
||||
}
|
||||
|
|
@ -0,0 +1,237 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"fmt"
|
||||
"io/ioutil"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"time"
|
||||
|
||||
"flag"
|
||||
|
||||
"io"
|
||||
|
||||
getter "github.com/hashicorp/go-getter"
|
||||
"github.com/hashicorp/terraform/plugin"
|
||||
"github.com/hashicorp/terraform/plugin/discovery"
|
||||
"github.com/mitchellh/cli"
|
||||
)
|
||||
|
||||
const releasesBaseURL = "https://releases.hashicorp.com"
|
||||
|
||||
type PackageCommand struct {
|
||||
ui cli.Ui
|
||||
}
|
||||
|
||||
func (c *PackageCommand) Run(args []string) int {
|
||||
flags := flag.NewFlagSet("package", flag.ExitOnError)
|
||||
osPtr := flags.String("os", "", "Target operating system")
|
||||
archPtr := flags.String("arch", "", "Target CPU architecture")
|
||||
err := flags.Parse(args)
|
||||
if err != nil {
|
||||
c.ui.Error(err.Error())
|
||||
return 1
|
||||
}
|
||||
|
||||
osName := runtime.GOOS
|
||||
archName := runtime.GOARCH
|
||||
if *osPtr != "" {
|
||||
osName = *osPtr
|
||||
}
|
||||
if *archPtr != "" {
|
||||
archName = *archPtr
|
||||
}
|
||||
|
||||
if flags.NArg() != 1 {
|
||||
c.ui.Error("Configuration filename is required")
|
||||
return 1
|
||||
}
|
||||
configFn := flags.Arg(0)
|
||||
|
||||
config, err := LoadConfigFile(configFn)
|
||||
if err != nil {
|
||||
c.ui.Error(fmt.Sprintf("Failed to read config: %s", err))
|
||||
return 1
|
||||
}
|
||||
|
||||
if discovery.ConstraintStr("< 0.10.0-beta1").MustParse().Allows(config.Terraform.Version.MustParse()) {
|
||||
c.ui.Error("Bundles can be created only for Terraform 0.10 or newer")
|
||||
return 1
|
||||
}
|
||||
|
||||
workDir, err := ioutil.TempDir("", "terraform-bundle")
|
||||
if err != nil {
|
||||
c.ui.Error(fmt.Sprintf("Could not create temporary dir: %s", err))
|
||||
return 1
|
||||
}
|
||||
defer os.RemoveAll(workDir)
|
||||
|
||||
c.ui.Info(fmt.Sprintf("Fetching Terraform %s core package...", config.Terraform.Version))
|
||||
|
||||
coreZipURL := c.coreURL(config.Terraform.Version, osName, archName)
|
||||
err = getter.Get(workDir, coreZipURL)
|
||||
if err != nil {
|
||||
c.ui.Error(fmt.Sprintf("Failed to fetch core package from %s: %s", coreZipURL, err))
|
||||
}
|
||||
|
||||
installer := &discovery.ProviderInstaller{
|
||||
Dir: workDir,
|
||||
|
||||
// FIXME: This is incorrect because it uses the protocol version of
|
||||
// this tool, rather than of the Terraform binary we just downloaded.
|
||||
// But we can't get this information from a Terraform binary, so
|
||||
// we'll just ignore this for now as we only have one protocol version
|
||||
// in play anyway. If a new protocol version shows up later we will
|
||||
// probably deal with this by just matching version ranges and
|
||||
// hard-coding the knowledge of which Terraform version uses which
|
||||
// protocol version.
|
||||
PluginProtocolVersion: plugin.Handshake.ProtocolVersion,
|
||||
|
||||
OS: osName,
|
||||
Arch: archName,
|
||||
}
|
||||
|
||||
for name, constraints := range config.Providers {
|
||||
c.ui.Info(fmt.Sprintf("Fetching provider %q...", name))
|
||||
for _, constraint := range constraints {
|
||||
meta, err := installer.Get(name, constraint.MustParse())
|
||||
if err != nil {
|
||||
c.ui.Error(fmt.Sprintf("Failed to resolve %s provider %s: %s", name, constraint, err))
|
||||
return 1
|
||||
}
|
||||
|
||||
c.ui.Info(fmt.Sprintf("- %q resolved to %s", constraint, meta.Version))
|
||||
}
|
||||
}
|
||||
|
||||
files, err := ioutil.ReadDir(workDir)
|
||||
if err != nil {
|
||||
c.ui.Error(fmt.Sprintf("Failed to read work directory %s: %s", workDir, err))
|
||||
return 1
|
||||
}
|
||||
|
||||
// If we get this far then our workDir now contains the union of the
|
||||
// contents of all the zip files we downloaded above. We can now create
|
||||
// our output file.
|
||||
outFn := c.bundleFilename(config.Terraform.Version, time.Now(), osName, archName)
|
||||
c.ui.Info(fmt.Sprintf("Creating %s ...", outFn))
|
||||
outF, err := os.OpenFile(outFn, os.O_TRUNC|os.O_CREATE|os.O_WRONLY, os.ModePerm)
|
||||
if err != nil {
|
||||
c.ui.Error(fmt.Sprintf("Failed to create %s: %s", outFn, err))
|
||||
return 1
|
||||
}
|
||||
outZ := zip.NewWriter(outF)
|
||||
defer func() {
|
||||
err := outZ.Close()
|
||||
if err != nil {
|
||||
c.ui.Error(fmt.Sprintf("Failed to close %s: %s", outFn, err))
|
||||
os.Exit(1)
|
||||
}
|
||||
err = outF.Close()
|
||||
if err != nil {
|
||||
c.ui.Error(fmt.Sprintf("Failed to close %s: %s", outFn, err))
|
||||
os.Exit(1)
|
||||
}
|
||||
}()
|
||||
|
||||
for _, file := range files {
|
||||
if file.IsDir() {
|
||||
// should never happen unless something tampers with our tmpdir
|
||||
continue
|
||||
}
|
||||
|
||||
fn := filepath.Join(workDir, file.Name())
|
||||
r, err := os.Open(fn)
|
||||
if err != nil {
|
||||
c.ui.Error(fmt.Sprintf("Failed to open %s: %s", fn, err))
|
||||
return 1
|
||||
}
|
||||
hdr, err := zip.FileInfoHeader(file)
|
||||
if err != nil {
|
||||
c.ui.Error(fmt.Sprintf("Failed to add zip entry for %s: %s", fn, err))
|
||||
return 1
|
||||
}
|
||||
w, err := outZ.CreateHeader(hdr)
|
||||
if err != nil {
|
||||
c.ui.Error(fmt.Sprintf("Failed to add zip entry for %s: %s", fn, err))
|
||||
return 1
|
||||
}
|
||||
_, err = io.Copy(w, r)
|
||||
if err != nil {
|
||||
c.ui.Error(fmt.Sprintf("Failed to write %s to bundle: %s", fn, err))
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
c.ui.Info("All done!")
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
func (c *PackageCommand) bundleFilename(version discovery.VersionStr, time time.Time, osName, archName string) string {
|
||||
time = time.UTC()
|
||||
return fmt.Sprintf(
|
||||
"terraform_%s-bundle%04d%02d%02d%02d_%s_%s.zip",
|
||||
version,
|
||||
time.Year(), time.Month(), time.Day(), time.Hour(),
|
||||
osName, archName,
|
||||
)
|
||||
}
|
||||
|
||||
func (c *PackageCommand) coreURL(version discovery.VersionStr, osName, archName string) string {
|
||||
return fmt.Sprintf(
|
||||
"%s/terraform/%s/terraform_%s_%s_%s.zip",
|
||||
releasesBaseURL, version, version, osName, archName,
|
||||
)
|
||||
}
|
||||
|
||||
func (c *PackageCommand) Synopsis() string {
|
||||
return "Produces a bundle archive"
|
||||
}
|
||||
|
||||
func (c *PackageCommand) Help() string {
|
||||
return `Usage: terraform-bundle package [options] <config-file>
|
||||
|
||||
Uses the given bundle configuration file to produce a zip file in the
|
||||
current working directory containing a Terraform binary along with zero or
|
||||
more provider plugin binaries.
|
||||
|
||||
Options:
|
||||
-os=name Target operating system the archive will be built for. Defaults
|
||||
to that of the system where the command is being run.
|
||||
|
||||
-arch=name Target CPU architecture the archive will be built for. Defaults
|
||||
to that of the system where the command is being run.
|
||||
|
||||
The resulting zip file can be used to more easily install Terraform and
|
||||
a fixed set of providers together on a server, so that Terraform's provider
|
||||
auto-installation mechanism can be avoided.
|
||||
|
||||
To build an archive for Terraform Enterprise, use:
|
||||
-os=linux -arch=amd64
|
||||
|
||||
Note that the given configuration file is a format specific to this command,
|
||||
not a normal Terraform configuration file. The file format looks like this:
|
||||
|
||||
terraform {
|
||||
# Version of Terraform to include in the bundle. An exact version number
|
||||
# is required.
|
||||
version = "0.10.0"
|
||||
}
|
||||
|
||||
# Define which provider plugins are to be included
|
||||
providers {
|
||||
# Include the newest "aws" provider version in the 1.0 series.
|
||||
aws = ["~> 1.0"]
|
||||
|
||||
# Include both the newest 1.0 and 2.0 versions of the "google" provider.
|
||||
# Each item in these lists allows a distinct version to be added. If the
|
||||
# two expressions match different versions then _both_ are included in
|
||||
# the bundle archive.
|
||||
google = ["~> 1.0", "~> 2.0"]
|
||||
}
|
||||
|
||||
`
|
||||
}
|
||||
Loading…
Reference in New Issue