resilien
/
terraform
2
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

website: Revise the "Module Sources" page 

It's been a while since we made any significant updates to this page.
The main theme of this revamp is to ensure that we highlight how to
provide "ambient credentials" for each of the module sources using the
standard mechanisms for each source type.

While there, I also attempted a general cleanup to highlight the main
cases and make this less like a detailed description of all of
go-getter's esoteric features, and did some general copy-editing to write
it in our usual documentation "voice".
This commit is contained in:
Martin Atkins 2018-07-31 09:30:02 -07:00
parent da1a57df74
commit 15ad2e4f9e
1 changed files with 257 additions and 176 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

433
website/docs/modules/sources.html.markdown
View File

@ -2,19 +2,23 @@
layout: "docs"
page_title: "Module Sources"
sidebar_current: "docs-modules-sources"
description: Explains the use of the source parameter, which tells Terraform where modules can be found.
description: The source argument within a module block specifies the location of the source code of a child module.
---
# Module Sources
As documented in the [Usage section](/docs/modules/usage.html), the only required parameter when using a module is `source`.
As introduced in [the _Usage_ section](/docs/modules/usage.html), the `source`
argument in a `module` block tells Terraform where to find the source code
for the desired child module.
The `source` parameter tells Terraform where the module can be found.
Terraform manages modules for you: it downloads them, organizes them on disk, checks for updates, etc. Terraform uses this `source` parameter to determine where it should retrieve and update modules from.
Terraform uses this during the module installation step of `terraform init`
to download the source code to a directory on local disk so that it can be
used by other Terraform commands.
Terraform supports the following sources:
The module installer supports installation from a number of different source
types, as listed below.
* [Local file paths](#local-file-paths)
* [Local paths](#local-paths)
* [Terraform Registry](#terraform-registry)
@ -28,11 +32,29 @@ Terraform supports the following sources:
* [S3 buckets](#s3-bucket)
Each is documented further below.
Each of these is described in the following sections. Module source addresses
use a _URL-like_ syntax, but with extensions to support unambiguous selection
of sources and additional features.
## Local File Paths
We recommend using local file paths for closely-related modules used primarily
for the purpose of factoring out repeated code elements, and using a native
Terraform module registry for modules intended to be shared by multiple calling
configurations. We support other sources so that you can potentially distribute
Terraform modules internally with existing infrastructure.
The easiest source is the local file path. For maximum portability, this should be a relative file path into a subdirectory. This allows you to organize your Terraform configuration into modules within one repository, for example:
Many of the source types will make use of "ambient" credentials available
when Terraform is run, such as from environment variables or credentials files
in your home directory. This is covered in more detail in each of the following
sections.
We recommend placing each module that is intended to be re-usable in the root
of its own repository or archive file, but it is also possible to
[reference modules from subdirectories](#modules-in-package-sub-directories).
## Local Paths
Local path references allow for factoring out portions of a configuration
within a single source repository.
```hcl
module "consul" {
@ -40,18 +62,36 @@ module "consul" {
}
```
Updates for file paths are automatic: when "downloading" the module using the [get command](/docs/commands/get.html), Terraform will create a symbolic link to the original directory. Therefore, any changes are automatically available.
A local path must begin with either `./` or `../` to indicate that a local
path is intended, to distinguish from
[a module registry address](#terraform-registry).
Local paths are special in that they are not "installed" in the same sense
that other sources are: the files are already present on local disk (possibly
as a result of installing a parent module) and so can just be used directly.
Their source code is automatically updated if the parent module is upgraded.
## Terraform Registry
The [Terraform Registry](https://registry.terraform.io) is an index of modules
written by the Terraform community.
The Terraform Registry is the easiest
way to get started with Terraform and to find modules.
A module registry is the native way of distributing Terraform modules for use
across multiple configurations, using a Terraform-specific protocol that
has full support for module versioning.
The registry is integrated directly into Terraform. You can reference any
registry module with a source string of `<NAMESPACE>/<NAME>/<PROVIDER>`. Each
module's information page on the registry includes its source string.
[Terraform Registry](https://registry.terraform.io/) is an index of modules
shared publicly using this protocol. This public registry is the easiest way
to get started with Terraform and find modules created by others in the
community.
You can also use a
[private registry](/docs/registry/private.html), either
via the built-in feature from Terraform Enterprise, or by running a custom
service that implements
[the module registry protocol](/docs/registry/api.html).
Modules on the public Terraform Registry can be referenced using a registry
source address of the form `<NAMESPACE>/<NAME>/<PROVIDER>`, with each
module's information page on the registry site including the exact address
to use.
```hcl
module "consul" {
@ -60,40 +100,42 @@ module "consul" {
}
```
The above example would use the
The above example will use the
[Consul module for AWS](https://registry.terraform.io/modules/hashicorp/consul/aws)
from the public registry.
Registry modules support versioning. You can provide a specific version, or use
flexible [version constraints](/docs/modules/usage.html#module-versions).
For modules hosted in other registries, prefix the source address with an
additional `<HOSTNAME>/` portion, giving the hostname of the private registry:
```hcl
module "consul" {
source = "app.terraform.io/example-corp/k8s-cluster/azurerm"
version = "1.1.0"
}
```
If you are using the managed version of Terraform Enterprise, its private
registry hostname is `app.terraform.io`. If you are using Terraform Enterprise
on-premises, its private registry hostname is the same hostname you use to
access your Terraform Enterprise instance.
Registry modules support versioning. You can provide a specific version as shown
in the above examples, or use flexible
[version constraints](/docs/modules/usage.html#module-versions).
You can learn more about the registry at the
[Terraform Registry documentation](/docs/registry/modules/use.html#using-modules).
## Private Registries
[Terraform Enterprise](https://www.hashicorp.com/products/terraform) provides a
[private module registry](/docs/enterprise/registry/index.html), to help
you share code within your organization. Other services can also provide
private registries by implementing [Terraform's registry API](/docs/registry/api.html).
Source strings for private registry modules are similar to public modules, but
also include a hostname. They should follow the format
`<HOSTNAME>/<NAMESPACE>/<NAME>/<PROVIDER>`.
```hcl
module "vpc" {
source = "app.terraform.io/example_corp/vpc/aws"
version = "0.9.3"
}
```
Modules from private registries support versioning, just like modules from the
public Terraform Registry.
To access modules from a private registry, you may need to configure an access
token [in the CLI config](/docs/commands/cli-config.html#credentials). Use the
same hostname as used in the module source string. For a private registry
within Terraform Enterprise, use the same authentication token as you would
use with the Enterprise API or command-line clients.
## GitHub
Terraform will automatically recognize GitHub URLs and turn them into a link to the specific Git repository. The syntax is simple:
Terraform will recognize unprefixed `github.com` URLs and interpret them
automatically as Git repository sources.
```hcl
module "consul" {
@ -101,202 +143,241 @@ module "consul" {
}
```
Subdirectories within the repository can also be referenced:
The above address scheme will clone over HTTPS. To clone over SSH, use the
following form:
```hcl
module "consul" {
source = "github.com/hashicorp/example//subdir"
source = "git@github.com:hashicorp/example.git"
}
```
These will fetch the modules using HTTPS. If you want to use SSH instead:
```hcl
module "consul" {
source = "git@github.com:hashicorp/example.git//subdir"
}
```
**Note:** The double-slash, `//`, is important. It is what tells Terraform that that is the separator for a subdirectory, and not part of the repository itself.
GitHub source URLs require that Git is installed on your system and that you have access to the repository.
You can use the same parameters to GitHub repositories as you can generic Git repositories (such as tags or branches). See [the documentation for generic Git repositories](#parameters) for more information.
### Private GitHub Repos
If you need Terraform to fetch modules from private GitHub repos, you must provide Terraform with credentials to authenticate as a user with read access to those repos.
- If you run Terraform only on your local machine, you can specify the module source as an SSH URI (like `git@github.com:hashicorp/example.git`) and Terraform will use your default SSH key to authenticate.
- If you use Terraform Enterprise, consider using the private module registry. It makes handling credentials easier, and provides full versioning support. (See [Private Registries](#private-registries) above for more info.)
If you need to use modules directly from Git, you can use SSH URIs with Terraform Enterprise. You'll need to add an SSH private key to your organization and assign it to any workspace that fetches modules from private repos. [See the Terraform Enterprise docs about SSH keys for cloning modules.](/docs/enterprise/workspaces/ssh-keys.html)
- If you need to run Terraform on a remote machine like a CI worker, you either need to write an SSH key to disk and set the `GIT_SSH_COMMAND` environment variable appropriately during the worker's provisioning process, or create a [GitHub machine user](https://developer.github.com/guides/managing-deploy-keys/#machine-users) with read access to the repos in question and embed its credentials into the modules' `source` parameters:
```hcl
module "private-infra" {
source = "git::https://MACHINE-USER:MACHINE-PASS@github.com/org/privatemodules//modules/foo"
}
```
Note that Terraform does not support interpolations in the `source` parameter of a module, so you must hardcode the machine username and password if using this method.
These GitHub schemes are treated as convenient aliases for
[the general Git repository address scheme](#generic-git-repository), and so
they obtain credentials in the same way and support the `ref` argument for
selecting a specific revision. You will need to configure credentials in
particular to access private repositories.
## Bitbucket
Terraform will automatically recognize public Bitbucket URLs and turn them into a link to the specific Git or Mercurial repository, for example:
Terraform will recognize unprefixed `bitbucket.org` URLs and interpret them
automatically as BitBucket repositories:
```hcl
module "consul" {
source = "bitbucket.org/hashicorp/consul"
source = "bitbucket.org/hashicorp/terraform-consul-aws"
}
```
Subdirectories within the repository can also be referenced:
This shorthand works only for public repositories, because Terraform must
access the BitBucket API to learn if the given repository uses Git or Mercurial.
```hcl
module "consul" {
source = "bitbucket.org/hashicorp/consul//subdir"
}
```
**Note:** The double-slash, `//`, is important. It is what tells Terraform that this is the separator for a subdirectory, and not part of the repository itself.
Bitbucket URLs will require that Git or Mercurial is installed on your system, depending on the type of repository.
## Private Bitbucket Repos
Private bitbucket repositories must be specified similar to the [Generic Git Repository](#generic-git-repository) section below.
```hcl
module "consul" {
source = "git::https://bitbucket.org/foocompany/module_name.git"
}
```
You can also specify branches and version with the `?ref` query, and use HTTPS or SSH:
```hcl
module "consul" {
source = "git::https://bitbucket.org/foocompany/module_name.git?ref=hotfix"
}
```
```hcl
module "consul" {
source = "git::ssh://git@bitbucket.org/foocompany/module_name.git"
}
```
You will need to run a `terraform get -update=true` if you want to pull the latest versions. This can be handy when you are rapidly iterating on a module in development.
Terraform treats the result either as [a Git source](#generic-git-repository)
or [a Mercurial source](#generic-mercurial-repository) depending on the
repository type. See the sections on each version control type for information
on how to configure credentials for private repositories and how to specify
a specific revision to install.
## Generic Git Repository
Generic Git repositories are also supported. The value of `source` in this case should be a complete Git-compatible URL. Using generic Git repositories requires that Git is installed on your system.
Arbitrary Git repositories can be used by prefixing the address with the
special `git::` prefix. After this prefix, any valid
[Git URL](https://git-scm.com/docs/git-clone#_git_urls_a_id_urls_a)
can be specified to select one of the protocols supported by Git.
For example, to use HTTPS or SSH:
```hcl
module "consul" {
source = "git://hashicorp.com/consul.git"
module "vpc" {
source = "git::https://example.com/vpc.git"
}
module "storage" {
source = "git::ssh://username@example.com/storage.git"
}
```
You can also use protocols such as HTTP or SSH to reference a module, but you'll have specify to Terraform that it is a Git module, by prefixing the URL with `git::` like so:
Terraform installs modules from Git repositories by running `git clone`, and
so it will respect any local Git configuration set on your system, including
credentials. To access a non-public Git repository, configure Git with
suitable credentials for that repository.
If you use the SSH protocol then any configured SSH keys will be used
automatically. This is the most common way to access non-public Git
repositories from automated systems beacuse it is easy to configure
and allows access to private repositories without interactive prompts.
If using the HTTP/HTTPS protocol, or any other protocol that uses
username/password credentials, configure
[Git Credentials Storage](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage)
to select a suitable source of credentials for your environment.
If your Terraform configuration will be used within [Terraform Enterprise](https://www.hashicorp.com/products/terraform),
only SSH key authentication is supported, and
[keys can be configured on a per-workspace basis](/docs/enterprise/workspaces/ssh-keys.html).
### Selecting a Revision
By default, Terraform will clone and use the default branch (referenced by
`HEAD`) in the selected repository. You can override this using the
`ref` argument:
```hcl
module "consul" {
source = "git::https://hashicorp.com/consul.git"
}
module "ami" {
source = "git::ssh://git@github.com/owner/repo.git"
module "vpc" {
source = "git::https://example.com/vpc.git?ref=v1.2.0"
}
```
If you do not specify the type of `source` then Terraform will attempt to use the closest match, for example assuming `https://hashicorp.com/consul.git` is a HTTP URL.
Terraform will cache the module locally by default `terraform get` is run, so successive updates to master or a specified branch will not be factored into future plans. Run `terraform get -update=true` to get the latest version of the branch. This is handy in development, but potentially bothersome in production if you don't have control of the repository.
### Parameters
The URLs for Git repositories support the following query parameters:
* `ref` - The ref to checkout. This can be a branch, tag, commit, etc.
```hcl
module "consul" {
source = "git::https://hashicorp.com/consul.git?ref=master"
}
```
The value of the `ref` argument can be any reference that would be accepted
by the `git checkout` command, including branch and tag names.
## Generic Mercurial Repository
Generic Mercurial repositories are supported. The value of `source` in this case should be a complete Mercurial-compatible URL. Using generic Mercurial repositories requires that Mercurial is installed on your system. You must tell Terraform that your `source` is a Mercurial repository by prefixing it with `hg::`.
You can use arbitrary Mercurial repositories by prefixing the address with the
special `hg::` prefix. After this prefix, any valid
[Mercurial URL](https://www.mercurial-scm.org/repo/hg/help/urls)
can be specified to select one of the protocols supported by Mercurial.
```hcl
module "consul" {
source = "hg::http://hashicorp.com/consul.hg"
module "vpc" {
source = "hg::http://example.com/vpc.hg"
}
```
URLs for Mercurial repositories support the following query parameters:
Terraform installs modules from Mercurial repositories by running `hg clone`, and
so it will respect any local Mercurial configuration set on your system,
including credentials. To access a non-public repository, configure Mercurial
with suitable credentials for that repository.
* `rev` - The rev to checkout. This can be a branch, tag, commit, etc.
If you use the SSH protocol then any configured SSH keys will be used
automatically. This is the most common way to access non-public Mercurial
repositories from automated systems beacuse it is easy to configure
and allows access to private repositories without interactive prompts.
If your Terraform configuration will be used within [Terraform Enterprise](https://www.hashicorp.com/products/terraform),
only SSH key authentication is supported, and
[keys can be configured on a per-workspace basis](/docs/enterprise/workspaces/ssh-keys.html).
### Selecting a Revision
You can select a non-default branch or tag using the optional `ref` argument:
```hcl
module "consul" {
source = "hg::http://hashicorp.com/consul.hg?rev=default"
module "vpc" {
source = "hg::http://example.com/vpc.hg?ref=v1.2.0"
}
```
## HTTP URLs
An HTTP or HTTPS URL can be used to redirect Terraform to get the module source from one of the other sources. For HTTP URLs, Terraform will make a `GET` request to the given URL. An additional `GET` parameter, `terraform-get=1`, will be appended, allowing
you to optionally render the page differently when Terraform is requesting it.
When you use an HTTP or HTTPS URL, Terraform will make a `GET` request to
the given URL, which can return _another_ source address. This indirection
allows using HTTP URLs as a sort of "vanity redirect" over a more complicated
module source address.
Terraform then looks for the resulting module URL in the following order:
Terraform will append an additional query string argument `terraform-get=1` to
the given URL before sending the `GET` request, allowing the server to
optionally return a different result when Terraform is requesting it.
1. Terraform will look to see if the header `X-Terraform-Get` is present. The header should contain the source URL of the actual module.
If the response is successful (`200`-range status code), Terraform looks in
the following locations in order for the next address to access:
2. Terraform will look for a `<meta>` tag with the name of `terraform-get`, for example:
* The value of a response header field named `X-Terraform-Get`.
```html
<meta name="terraform-get" content="github.com/hashicorp/example" />
* If the response is an HTML page, a `meta` element with the name `terraform-get`:
```html
<meta name="terraform-get"
content="github.com/hashicorp/example" />
```
In either case, the result is interpreted as another module source address
using one of the forms documented elsewhere on this page.
If an HTTP/HTTPS URL requires authentication credentials, use a `.netrc`
file in your home directory to configure these. For information on this format,
see [the documentation for using it in `curl`](https://ec.haxx.se/usingcurl-netrc.html).
### Fetching archives over HTTP
As a special case, if Terraform detects that the URL has a common file
extension associated with an archive file format then it will bypass the
special `terraform-get=1` redirection described above and instead just use
the contents of the referenced archive as the module source code:
```hcl
module "vpc" {
source = "https://example.com/vpc-module.zip"
}
```
The extensions that Terraform recognizes for this special behavior are:
* `zip`
* `tar.bz2` and `tbz2`
* `tar.gz` and `tgz`
* `tar.xz` and `txz`
If your URL _doesn't_ have one of these extensions but refers to an archive
anyway, use the `archive` argument to force this interpretation:
```hcl
module "vpc" {
source = "https://example.com/vpc-module?archive=zip"
}
```
## S3 Bucket
Terraform can also store modules in an S3 bucket. To access the bucket
you must have appropriate AWS credentials in your configuration or
available via shared credentials or environment variables.
There are a variety of S3 bucket addressing schemes, most are
[documented in the S3
configuration](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro).
Here are a couple of examples.
Using the `s3` protocol.
You can use archives stored in S3 as module sources using the special `s3::`
prefix, followed by
[an S3 bucket object URL](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro).
```hcl
module "consul" {
source = "s3::https://s3-eu-west-1.amazonaws.com/consulbucket/consul.zip"
source = "s3::https://s3-eu-west-1.amazonaws.com/examplecorp-terraform-modules/vpc.zip"
}
```
Or directly using the bucket's URL.
The `s3::` prefix causes Terraform to use AWS-style authentication when
accessing the given URL. As a result, this scheme may also work for other
services that mimic the S3 API, as long as they handle authentication in the
same way as AWS.
```hcl
module "consul" {
source = "consulbucket.s3-eu-west-1.amazonaws.com/consul.zip"
}
```
The resulting object must be an archive with one of the same file
extensions as for [archives over standard HTTP](#fetching-archives-over-http).
Terraform will extract the archive to obtain the module source tree.
The module installer looks for AWS credentials in the following locations,
preferring those earlier in the list when multiple are available:
## Unarchiving
* The `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables.
* The default profile in the `.aws/credentials` file in your home directory.
* If running on an EC2 instance, temporary credentials associated with the
instance's IAM Instance Profile.
Terraform will automatically unarchive files based on the extension of
the file being requested (over any protocol). It supports the following
archive formats:
## Modules in Package Sub-directories
* tar.gz and tgz
* tar.bz2 and tbz2
* zip
* gz
* bz2
When the source of a module is a version control repository or archive file
(generically, a "package"), the module itself may be in a sub-directory relative
to the root of the package.
A special double-slash syntax is interpreted by Terraform to indicate that
the remaining path after that point is a sub-directory within the package.
For example:
* `hashicorp/consul/aws//modules/consul-cluster`
* `git::https://example.com/network.git//modules/vpc`
* `https://example.com/network-module.zip//modules/vpc`
* `s3::https://s3-eu-west-1.amazonaws.com/examplecorp-terraform-modules/network.zip//modules/vpc`
If the source address has arguments, such as the `ref` argument supported for
the version control sources, the sub-directory portion must be _before_ those
arguments:
* `git::https://example.com/network.git//modules/vpc?ref=v1.2.0`
Terraform will still extract the entire package to local disk, but will read
the module from the subdirectory. As a result, it is safe for a module in
a sub-directory of a package to use [a local path](#local-paths) to another
module as long as it is in the _same_ package.