From 4fa2b8c519606e514527d6041ee3ed530922c66b Mon Sep 17 00:00:00 2001 From: Martin Atkins Date: Tue, 31 Oct 2017 16:05:22 -0700 Subject: [PATCH] website: updated docs for the module registry protocol Now includes more complete information on usage of private registries and updates the module registry API documentation to include the new version-enumeration endpoint along with describing Terraform's discovery protocol. --- .../remote-service-discovery.html.md | 112 +++++++++ website/docs/registry/api.html.md | 235 ++++++++++++++---- website/docs/registry/index.html.md | 8 +- website/docs/registry/private.html.md | 37 ++- website/layouts/docs.erb | 4 + 5 files changed, 329 insertions(+), 67 deletions(-) create mode 100644 website/docs/internals/remote-service-discovery.html.md diff --git a/website/docs/internals/remote-service-discovery.html.md b/website/docs/internals/remote-service-discovery.html.md new file mode 100644 index 000000000..5cdf47bf8 --- /dev/null +++ b/website/docs/internals/remote-service-discovery.html.md @@ -0,0 +1,112 @@ +--- +layout: "docs" +page_title: "Internals: Remote Service Discovery" +sidebar_current: "docs-remote-service-discovery" +description: |- + Remote service discovery is a protocol used to locate Terraform-native + services provided at a user-friendly hostname. +--- + +# Remote Service Discovery + +Terraform implements much of its functionality in terms of remote services. +While in many cases these are generic third-party services that are useful +to many applications, some of these services are tailored specifically to +Terraform's needs. We call these _Terraform-native services_, and Terraform +interacts with them via the remote service discovery protocol described below. + +## User-facing Hostname + +Terraform-native services are provided, from a user's perspective, at a +user-facing "friendly hostname" which serves as the key for configuration and +for any authentication credentials required. + +The discovery protocol's purpose is to map from a user-provided hostname to +the base URL of a particular service. Each host can provide different +combinations of services -- or no services at all! -- and so the discovery +protocol has a secondary purpose of allowing Terraform to identify _which_ +services are valid for a given hostname. + +For example, module source strings can include a module registry hostname +as their first segment, like `example.com/namespace/name/provider`, and +Terraform uses service discovery to determine whether `example.com` _has_ +a module registry, and if so where its API is available. + +A user-facing hostname is a fully-specified +[internationalized domain name](https://en.wikipedia.org/wiki/Internationalized_domain_name) +expressed in its Unicode form (the corresponding "punycode" form is not allowed) +which must be resolvable in DNS to an address that has an HTTPS server running +on port 443. + +User-facing hostnames are normalized for internal comparison using the +standard Unicode [Nameprep](https://en.wikipedia.org/wiki/Nameprep) algorithm, +which includes converting all letters to lowercase, normalizing combining +diacritics to precomposed form where possible, and various other normalization +steps. + +## Discovery Process + +Given a hostname, discovery begins by forming an initial discovery URL +using that hostname with the `https:` scheme and the fixed path +`/.well-known/terraform.json`. + +For example, given the hostname `example.com` the initial discovery URL +would be `https://example.com/.well-known/terraform.json`. + +Terraform then sends a `GET` request to this discovery URL and expects a +JSON response. If the response does not have status 200, does not have a media +type of `application/json` or, if the body cannot be parsed as a JSON object, +then discovery fails and Terraform considers the host to not support _any_ +Terraform-native services. + +If the response is an HTTP redirect then Terraform repeats this step with the +new location as its discovery URL. Terraform is guaranteed to follow at least +one redirect, but nested redirects are not guaranteed nor recommended. + +If the response is a valid JSON object then its keys are Terraform native +service identifiers, consisting of a service type name and a version string +separated by a period. For example, the service identifier for version 1 of +the module registry protocol is `modules.v1`. + +The value of each object element is the base URL for the service in question. +This URL may be either absolute or relative, and if relative it is resolved +against the final discovery URL (_after_ following redirects). + +The following is an example discovery document declaring support for +version 1 of the module registry protocol: + +```json +{ + "modules.v1": "https://modules.example.com/v1/" +} +``` + +## Supported Services + +At present, only one service identifier is in use: + +* `modules.v1`: [module registry API version 1](/docs/registry/api.html) + +## Authentication + +If credentials for the given hostname are available in +[the CLI config](/docs/commands/cli-config.html) then they will be included +in the request for the discovery document. + +The credentials may also be provided to endpoints declared in the discovery +document, depending on the requirements of the service in question. + +## Non-standard Ports in User-facing Hostnames + +It is strongly recommended to provide the discovery document for a hostname +on the standard HTTPS port 443. However, in development environments this is +not always possible or convenient, so Terraform allows a hostname to end +with a port specification consisting of a colon followed by one or more +decimal digits. + +When a custom port number is present, the service on that port is expected to +implement HTTPS and respond to the same fixed discovery path. + +For day-to-day use it is strongly recommended _not_ to rely on this mechanism +and to instead provide the discovery document on the standard port, since this +allows use of the most user-friendly hostname form. diff --git a/website/docs/registry/api.html.md b/website/docs/registry/api.html.md index 6f228f53c..a0e85fc95 100644 --- a/website/docs/registry/api.html.md +++ b/website/docs/registry/api.html.md @@ -8,13 +8,183 @@ description: |- # HTTP API -The [Terraform Registry](https://registry.terraform.io) has an HTTP API for -reading and downloading registry modules. +When downloading modules from registry sources such as the public +[Terraform Registry](https://registry.terraform.io), Terraform expects +the given hostname to support the following module registry protocol. -Terraform interacts with the registry as read-only. Therefore, the documented -API is read-only. Any endpoints that aren't documented on this page can and will -likely change over time. This allows differing methods for getting modules into -the registry while keeping a consistent API for reading modules in the registry. +A registry module source is of the form `hostname/namespace/name/provider`, +where the initial hostname portion is implied to be `registry.terraform.io/` +if not specified. The public Terraform Registry is therefore the default +module source. + +[Terraform Registry](https://registry.terraform.io) implements a superset +of this API to allow for importing new modules, etc, but any endpoints not +documented on this page are subject to change over time. + +## Service Discovery + +The hostname portion of a module source string is first passed to +[the service discovery protocol](/docs/internals/remote-service-discovery.html) +to determine if the given host has a module registry and, if so, the base +URL for its module registry endpoints. + +The service identifier for this protocol is `modules.v1`, and the declared +URL should always end with a slash such that the paths shown in the following +sections can be appended to it. + +For example, if discovery produces the URL `https://modules.example.com/v1/` +then this API would use full endpoint URLs like +`https://modules.example.com/v1/{namespace}/{name}/{provider}/versions`. + +The example request URLs shown in this document are for the public +[Terraform Registry](https://registry.terraform.io), and use its API base +URL of `https://registry.terraform.io/v1/modules/` . + +## List Available Versions for a Specific Module + +This is the primary endpoint for resolving module sources, returning the +available versions for a given fully-qualified module. + +| Method | Path | Produces | +| ------ | ------------------------------------- | -------------------------- | +| `GET` | `:namespace/:name/:provider/versions` | `application/json` | + +### Parameters + +- `namespace` `(string: )` - The user or organization the module is + owned by. This is required and is specified as part of the URL path. + +- `name` `(string: )` - The name of the module. + This is required and is specified as part of the URL path. + +- `provider` `(string: )` - The name of the provider. + This is required and is specified as part of the URL path. + +### Sample Request + +```text +$ curl https://registry.terraform.io/v1/modules/hashicorp/consul/aws/versions +``` + +### Sample Response + +The `modules` array in the response always includes the requested module as the +first element. Other elements of this list, if present, are dependencies of the +requested module that are provided to potentially avoid additional requests to +resolve these modules. + +Additional modules are not required to be provided but, when present, can be +used by Terraform to optimize the module installation process. + +Each returned module has an array of available versions, which Terraform +matches against any version constraints given in configuration. + +```json +{ + "modules": [ + { + "source": "hashicorp/consul/aws", + "versions": [ + { + "version": "0.0.1", + "submodules" : [ + { + "path": "modules/consul-cluster", + "providers": [ + { + "name": "aws", + "version": "" + } + ], + "dependencies": [] + }, + { + "path": "modules/consul-security-group-rules", + "providers": [ + { + "name": "aws", + "version": "" + } + ], + "dependencies": [] + }, + { + "providers": [ + { + "name": "aws", + "version": "" + } + ], + "dependencies": [], + "path": "modules/consul-iam-policies" + } + ], + "root": { + "dependencies": [], + "providers": [ + { + "name": "template", + "version": "" + }, + { + "name": "aws", + "version": "" + } + ] + } + } + ] + } + ] +} +``` + +## Download Source Code for a Specific Module Version + +This endpoint downloads the specified version of a module for a single provider. + +A successful response has no body, and includes the location from which the module +version's source can be downloaded in the `X-Terraform-Get` header. Note that +this string may contain special syntax interpreted by Terraform via +[`go-getter`](https://github.com/hashicorp/go-getter). See the [`go-getter` +documentation](https://github.com/hashicorp/go-getter#url-format) for details. + +The value of `X-Terraform-Get` may instead be a relative URL, indicated by +beginning with `/`, `./` or `../`, in which case it is resolved relative to +the full URL of the download endpoint. + +| Method | Path | Produces | +| ------ | ---------------------------- | -------------------------- | +| `GET` | `:namespace/:name/:provider/:version/download` | `application/json` | + +### Parameters + +- `namespace` `(string: )` - The user the module is owned by. + This is required and is specified as part of the URL path. + +- `name` `(string: )` - The name of the module. + This is required and is specified as part of the URL path. + +- `provider` `(string: )` - The name of the provider. + This is required and is specified as part of the URL path. + +- `version` `(string: )` - The version of the module. + This is required and is specified as part of the URL path. + +### Sample Request + +```text +$ curl -i \ + https://registry.terraform.io/v1/modules/hashicorp/consul/aws/0.0.1/download +``` + +### Sample Response + +```text +HTTP/1.1 204 No Content +Content-Length: 0 +X-Terraform-Get: https://api.github.com/repos/hashicorp/terraform-aws-consul/tarball/v0.0.1//*?archive=tar.gz +``` ## List Latest Version of Module for All Providers @@ -22,7 +192,7 @@ This endpoint returns the latest version of each provider for a module. | Method | Path | Produces | | ------ | ---------------------------- | -------------------------- | -| `GET` | `/v1/modules/:namespace/:name` | `application/json` | +| `GET` | `:namespace/:name` | `application/json` | ### Parameters @@ -82,13 +252,13 @@ $ curl \ } ``` -## Latest Module for a Single Provider +## Latest Version for a Specific Module Provider This endpoint returns the latest version of a module for a single provider. | Method | Path | Produces | | ------ | ---------------------------- | -------------------------- | -| `GET` | `/v1/modules/:namespace/:name/:provider` | `application/json` | +| `GET` | `:namespace/:name/:provider` | `application/json` | ### Parameters @@ -210,7 +380,7 @@ This endpoint returns the specified version of a module for a single provider. | Method | Path | Produces | | ------ | ---------------------------- | -------------------------- | -| `GET` | `/v1/modules/:namespace/:name/:provider/:version` | `application/json` | +| `GET` | `:namespace/:name/:provider/:version` | `application/json` | ### Parameters @@ -330,49 +500,6 @@ Note this response has has some fields trimmed for clarity. } ``` -## Download a Specific Module - -This endpoint downloads the specified version of a module for a single provider. - -A successful response has no body, and includes the URL from which the module -version's source can be downloaded in the `X-Terraform-Get` header. Note that -this URL may contain special syntax interpreted by Terraform via -[`go-getter`](https://github.com/hashicorp/go-getter). See the [`go-getter` -documentation](https://github.com/hashicorp/go-getter#url-format) for details. - -| Method | Path | Produces | -| ------ | ---------------------------- | -------------------------- | -| `GET` | `/v1/modules/:namespace/:name/:provider/:version/download` | `application/json` | - -### Parameters - -- `namespace` `(string: )` - The user the module is owned by. - This is required and is specified as part of the URL path. - -- `name` `(string: )` - The name of the module. - This is required and is specified as part of the URL path. - -- `provider` `(string: )` - The name of the provider. - This is required and is specified as part of the URL path. - -- `version` `(string: )` - The version of the module. - This is required and is specified as part of the URL path. - -### Sample Request - -```text -$ curl -i \ - https://registry.terraform.io/v1/modules/hashicorp/consul/aws/0.0.1/download -``` - -### Sample Response - -```text -HTTP/1.1 204 No Content -Content-Length: 0 -X-Terraform-Get: https://api.github.com/repos/hashicorp/terraform-aws-consul/tarball/v0.0.1//*?archive=tar.gz -``` - ## Download the Latest Version of a Module This endpoint downloads the latest version of a module for a single provider. @@ -382,7 +509,7 @@ download endpoint (above) for the latest version. | Method | Path | Produces | | ------ | ---------------------------- | -------------------------- | -| `GET` | `/v1/modules/:namespace/:name/:provider/download` | `application/json` | +| `GET` | `:namespace/:name/:provider/download` | `application/json` | ### Parameters diff --git a/website/docs/registry/index.html.md b/website/docs/registry/index.html.md index db28ae4e9..1010e8fb3 100644 --- a/website/docs/registry/index.html.md +++ b/website/docs/registry/index.html.md @@ -29,9 +29,9 @@ module "consul" { You can also publish your own modules on the Terraform Registry. You may use the [public registry](https://registry.terraform.io) for public modules. -For private modules, you must use [Terraform Enterprise](https://www.hashicorp.com/products/terraform). -You can use modules without a registry by -[sourcing modules directly](/docs/modules/sources.html), however non-registry -modules do not support versioning, documentation generation, and more. +For private modules, you can use a [Private Registry](/docs/registry/private.html), +or [reference repositories and other sources directly](/docs/modules/sources.html). +Some features are available only for registry modules, such as versioning +and documentation generation. Use the navigation to the left to learn more about using the registry. diff --git a/website/docs/registry/private.html.md b/website/docs/registry/private.html.md index d867a0379..75c122832 100644 --- a/website/docs/registry/private.html.md +++ b/website/docs/registry/private.html.md @@ -12,17 +12,36 @@ The registry at [registry.terraform.io](https://registry.terraform.io) may only host public modules. Terraform is capable of loading modules from private registries for private modules. -Official private registries are available via [Terraform Enterprise](#). +Official private registries are available via [Terraform Enterprise](https://www.hashicorp.com/products/terraform). There are two tiers: Pro and Enterprise. The Pro version is only available as a SaaS service whereas the Enterprise version is available for private install. Both versions fully support private registries. -The Terraform project does not provide any free or open source solution to have -a private registry. Terraform only requires the [read API](/docs/registry/api.html) -to be available to load modules from a registry. -Support for specifying an alternative to the public registry will be available -in Terraform 0.11. We welcome the community to create their own private -registries by recreating this API. +Terraform interacts with module registries using [the registry API](/docs/registry/api.html). +The Terraform open source project does not provide a server implementation, but +we welcome community members to create their own private registries by following +the published protocol. + +Modules can alternatively be referenced +[directly from version control and other sources](/docs/modules/sources.html), +but only registry modules support certain features such as +[version constraints](/docs/modules/usage.html#module-versions). + +## Private Registry Module Sources + +Public Terraform Registry modules have source strings of the form +`namespace/name/provider`. Private registries -- whether integrated into +Terraform Enterprise or via a third-party implementation -- require an +additional hostname prefix: + +```hcl +module "example" { + source = "example.com/namespace/name/provider" +} +``` + +Private registry module sources are supported in Terraform v0.11.0 and +newer. ## Coming Soon @@ -33,5 +52,5 @@ by the end of 2017. In the mean time, if you're interested in private registries and being part of the beta, please contact us at [hello@hashicorp.com](mailto:hello@hashicorp.com). -When Terraform Enterprise is publicly available, the documentation will -be available here. +When Terraform Enterprise is publicly available, Private Registry documentation +will be available here. diff --git a/website/layouts/docs.erb b/website/layouts/docs.erb index bbfc0ce89..4b1549189 100644 --- a/website/layouts/docs.erb +++ b/website/layouts/docs.erb @@ -589,6 +589,10 @@ Resource Addressing + > + Remote Service Discovery + + > Internal Plugins