From ef071f3d0e49ba421ae931c65b263827a8af1adb Mon Sep 17 00:00:00 2001 From: Martin Atkins Date: Fri, 24 Jul 2020 12:25:34 -0700 Subject: [PATCH] website: Generalized advice on modules with provider configs As part of documenting the new module for_each capabilities we added a section noting that shared modules using the legacy pattern of declaring their own provider configurations would not be compatible with them. However, that also applies to the new module depends_on and several folks participating in the beta pointed out that the documentation wasn't discussing that at all. In order to generalize the advice, I've moved the old content we had (since v0.11) recommending against provider configurations in shared modules out into its own section, now being more explicit that it is a legacy pattern and not recommended, and then folded the content about for_each and count, now also including depends_on, into that expanded section. As is often the case, that had some knock-on effects on the content on the rest of this page, so there's some general editing and reorganization here. In particular, I moved the "Multiple Instances of a Module" section much further up the page because it's content relevant to users of shared modules, while the later content on this page is more aimed at authors of shared modules, including the new section about the legacy pattern. --- website/docs/configuration/modules.html.md | 303 +++++++++++++-------- 1 file changed, 186 insertions(+), 117 deletions(-) diff --git a/website/docs/configuration/modules.html.md b/website/docs/configuration/modules.html.md index ce1f5771d..7952a8fb5 100644 --- a/website/docs/configuration/modules.html.md +++ b/website/docs/configuration/modules.html.md @@ -128,6 +128,72 @@ modules sourced from local file paths do not support `version`; since they're loaded from the same source repository, they always share the same version as their caller. +## Multiple Instances of a Module + +Use the `for_each` or the `count` argument to create multiple instances of a +module from a single `module` block. These arguments have the same syntax and +type constraints as +[`for_each`](./resources.html#for_each-multiple-resource-instances-defined-by-a-map-or-set-of-strings) +and +[`count`](./resources.html#count-multiple-resource-instances-by-count) +when used with resources. + +```hcl +# my_buckets.tf +module "bucket" { + for_each = toset(["assets", "media"]) + source = "./publish_bucket" + name = "${each.key}_bucket" +} +``` + +```hcl +# publish_bucket/bucket-and-cloudfront.tf +variable "name" {} # this is the input parameter of the module + +resource "aws_s3_bucket" "example" { + # Because var.name includes each.key in the calling + # module block, its value will be different for + # each instance of this module. + bucket = var.name + + # ... +} + +resource "aws_iam_user" "deploy_user" { + # ... +} +``` + +This example defines a local child module in the `./publish_bucket` +subdirectory. That module has configuration to create an S3 bucket. The module +wraps the bucket and all the other implementation details required to configure +a bucket. + +We declare multiple module instances by using the `for_each` attribute, +which accepts a map (with string keys) or a set of strings as its value. Additionally, +we use the `each.key` in our module block, because the +[`each`](/docs/configuration/resources.html#the-each-object) object is available when +we have declared `for_each` on the module block. When using the `count` argument, the +[`count`](/docs/configuration/resources.html#the-count-object) object is available. + +Resources from child modules are prefixed with `module.module_name[module index]` +when displayed in plan output and elsewhere in the UI. For a module with without +`count` or `for_each`, the address will not contain the module index as the module's +name suffices to reference the module. + +In our example, the `./publish_bucket` module contains `aws_s3_bucket.example`, and so the two +instances of this module produce S3 bucket resources with [resource addresses](/docs/internals/resource-addressing.html) of `module.bucket["assets"].aws_s3_bucket.example` +and `module.bucket["media"].aws_s3_bucket.example` respectively. These full addresses +are used within the UI and on the command line, but only [outputs](docs/configuration/outputs.html) +from a module can be referenced from elsewhere in your configuration. + +When refactoring an existing configuration to introduce modules, moving +resource blocks between modules causes Terraform to see the new location +as an entirely separate resource to the old. Always check the execution plan +after performing such actions to ensure that no resources are surprisingly +deleted. + ## Other Meta-arguments Along with the `source` meta-argument described above, module blocks have @@ -161,56 +227,83 @@ about how modules can be used, created, and published is included in In a configuration with multiple modules, there are some special considerations for how resources are associated with provider configurations. -While in principle `provider` blocks can appear in any module, it is recommended -that they be placed only in the _root_ module of a configuration, since this -approach allows users to configure providers just once and re-use them across -all descendent modules. - Each resource in the configuration must be associated with one provider -configuration, which may either be within the same module as the resource -or be passed from the parent module. Providers can be passed down to descendent -modules in two ways: either _implicitly_ through inheritance, or _explicitly_ -via the `providers` argument within a `module` block. These two options are -discussed in more detail in the following sections. +configuration. Provider configurations, unlike most other concepts in +Terraform, are global to an entire Terraform configuration and can be shared +across module boundaries. Provider configurations can be defined only in a +root Terraform module. -In all cases it is recommended to keep explicit provider configurations only in -the root module and pass them (whether implicitly or explicitly) down to -descendent modules. This avoids the provider configurations from being "lost" -when descendent modules are removed from the configuration. It also allows -the user of a configuration to determine which providers require credentials -by inspecting only the root module. +Providers can be passed down to descendent modules in two ways: either +_implicitly_ through inheritance, or _explicitly_ via the `providers` argument +within a `module` block. These two options are discussed in more detail in the +following sections. + +A module intended to be called by one or more other modules must not contain +any `provider` blocks, with the exception of the special +"proxy provider blocks" discussed under +_[Passing Providers Explicitly](#passing-providers-explicitly)_ +below. + +For backward compatibility with configurations targeting Terraform v0.10 and +earlier Terraform does not produce an error for a `provider` block in a shared +module if the `module` block only uses features available in Terraform v0.10, +but that is a legacy usage pattern that is no longer recommended and a legacy +module containing its own provider configurations is not compatible with the +`for_each`, `count`, and `depends_on` arguments that were introduced in +Terraform v0.13. For more information, see +[Legacy Shared Modules with Provider Configurations](#legacy-shared-modules-with-provider-configurations). Provider configurations are used for all operations on associated resources, including destroying remote objects and refreshing state. Terraform retains, as part of its state, a reference to the provider configuration that was most recently used to apply changes to each resource. When a `resource` block is -removed from the configuration, this record in the state is used to locate the -appropriate configuration because the resource's `provider` argument (if any) -is no longer present in the configuration. +removed from the configuration, this record in the state will be used to locate +the appropriate configuration because the resource's `provider` argument +(if any) will no longer be present in the configuration. -As a consequence, it is required that all resources created for a particular -provider configuration must be destroyed before that provider configuration is -removed, unless the related resources are re-configured to use a different -provider configuration first. +As a consequence, you must ensure that all resources that belong to a +particular provider configuration are destroyed before you can remove that +provider configuration's block from your configuration. If Terraform finds +a resource instance tracked in the state whose provider configuration block is +no longer available then it will return an error during planning, prompting you +to reintroduce the provider configuration. ### Provider Version Constraints in Modules +Although provider _configurations_ are shared between modules, each module must +declare its own [provider requirements](provider-requirements.html), so that +Terraform can ensure that there is a single version of the provider that is +compatible with all modules in the configuration and to specify the +[source address](provider-requirements.html#source-addresses) that serves as +the global (module-agnostic) identifier for a provider. + To declare that a module requires particular versions of a specific provider, -use a [`required_providers`](terraform.html#specifying-required-provider-versions) -block inside a `terraform` block: +use a `required_providers` block inside a `terraform` block: ```hcl terraform { required_providers { - aws = ">= 2.7.0" + aws = { + source = "hashicorp/aws" + version = ">= 2.7.0" + } } } ``` -Shared modules should constrain only the minimum allowed version, using a `>=` -constraint. This specifies the minimum version the provider is compatible -with while allowing users to upgrade to newer provider versions without -altering the module source code. +A provider requirement says, for example, "This module requires version v2.7.0 +of the provider `hashicorp/aws` and will refer to it as `aws`." It doesn't, +however, specify any of the configuration settings that determine what remote +endpoints the provider will access, such as an AWS region; configuration +settings come from provider _configurations_, and a particular overall Terraform +configuration can potentially have +[several different configurations for the same provider](providers.html#alias-multiple-provider-instances). + +If you are writing a shared Terraform module, constrain only the minimum +required provider version using a `>=` constraint. This should specify the +minimum version containing the features your module relies on, and thus allow a +user of your module to potentially select a newer provider version if other +features are needed by other parts of their overall configuration. ### Implicit Provider Inheritance @@ -349,78 +442,47 @@ configuration block is valid, it is not necessary: proxy configuration blocks are needed only to establish which _alias_ provider configurations a child module is expecting. -A proxy configuration block must not include the `version` argument. To specify -version constraints for a particular child module without creating a local -module configuration, use the [`required_providers`](/docs/configuration/terraform.html#specifying-required-provider-versions) -setting inside a `terraform` block. +A proxy configuration block declares that a module is expecting to be +explicitly passed an additional (aliased) provider configuration. Don't use a +proxy configuration block if a module only needs a single default provider +configuration, and don't use proxy configuration blocks only to imply +[provider requirements](provider-requirements.html). -## Multiple Instances of a Module +## Legacy Shared Modules with Provider Configurations -Use the `count` or `for_each` arguments to create multiple instances of a module. -These arguments have the same syntax and type constraints as -[`count`](./resources.html#count-multiple-resource-instances-by-count) and -[`for_each`](./resources.html#for_each-multiple-resource-instances-defined-by-a-map-or-set-of-strings) -as defined for managed resources. +In Terraform v0.10 and earlier there was no explicit way to use different +configurations of a provider in different modules in the same configuration, +and so module authors commonly worked around this by writing `provider` blocks +directly inside their modules, making the module have its own separate +provider configurations separate from those declared in the root module. -```hcl -# my_buckets.tf -module "bucket" { - for_each = toset(["assets", "media"]) - source = "./publish_bucket" - name = "${each.key}_bucket" -} -``` +However, that pattern had a significant drawback: because a provider +configuration is required to destroy the remote object associated with a +resource instance as well as to create or update it, a provider configuration +must always stay present in the overall Terraform configuration for longer +than all of the resources it manages. If a particular module includes +both resources and the provider configurations for those resources then +removing the module from its caller would violate that constraint: both the +resources and their associated providers would, in effect, be removed +simultaneously. -```hcl -# publish_bucket/bucket-and-cloudfront.tf -variable "name" {} # this is the input parameter of the module +Terraform v0.11 introduced the mechanisms described in earlier sections to +allow passing provider configurations between modules in a structured way, and +thus we explicitly recommended against writing a child module with its own +provider configuration blocks. However, that legacy pattern continued to work +for compatibility purposes -- though with the same drawback -- until Terraform +v0.13. -resource "aws_s3_bucket" "example" { - # ... -} +Terraform v0.13 introduced the possibility for a module itself to use the +`for_each`, `count`, and `depends_on` arguments, but the implementation of +those unfortunately conflicted with the support for the legacy pattern. -resource "aws_iam_user" "deploy_user" { - # ... -} -``` - -This example defines a local child module in the `./publish_bucket` -subdirectory. That module has configuration to create an S3 bucket. The module -wraps the bucket and all the other implementation details required to configure -a bucket. - -We declare multiple module instances by using the `for_each` attribute, -which accepts a map (with string keys) or a set of strings as its value. Additionally, -we use the `each.key` in our module block, because the -[`each`](/docs/configuration/resources.html#the-each-object) object is available when -we have declared `for_each` on the module block. When using the `count` argument, the -[`count`](/docs/configuration/resources.html#the-count-object) object is available. - -Resources from child modules are prefixed with `module.module_name[module index]` -when displayed in plan output and elsewhere in the UI. For a module with without -`count` or `for_each`, the address will not contain the module index as the module's -name suffices to reference the module. - -In our example, the `./publish_bucket` module contains `aws_s3_bucket.example`, and so the two -instances of this module produce S3 bucket resources with [resource addresses](/docs/internals/resource-addressing.html) of `module.bucket["assets"].aws_s3_bucket.example` -and `module.bucket["media"].aws_s3_bucket.example` respectively. These full addresses -are used within the UI and on the command line, but only [outputs](docs/configuration/outputs.html) -from a module can be referenced from elsewhere in your configuration. - -When refactoring an existing configuration to introduce modules, moving -resource blocks between modules causes Terraform to see the new location -as an entirely separate resource to the old. Always check the execution plan -after performing such actions to ensure that no resources are surprisingly -deleted. - -### Limitations when using module expansion - -Modules using `count` or `for_each` cannot include configured `provider` blocks within the module. -Only [proxy configuration blocks](#proxy-configuration-blocks) are allowed. - -If a module contains proxy configuration blocks, the calling module block must be have the -corresponding providers passed to the `providers` argument. If you attempt to use `count` or -`for_each` with a module that does not satisfy this requirement, you will see an error: +To retain the backward compatibility as much as possible, Terraform v0.13 +continues to support the legacy pattern for module blocks that do not use these +new features, but a module with its own provider configurations is not +compatible with `for_each`, `count`, or `depends_on` and so Terraform will +produce an error announcing that if you attempt to combine these features. For +example: ``` Error: Module does not support count @@ -436,10 +498,23 @@ its provider configurations from the calling module, by using the "providers" argument in the calling module block. ``` -Assuming the child module only has proxy configuration blocks, the calling -module block could be adjusted like so to remove this error: +To make a module compatible with the new features, you must either remove all +of the `provider` blocks from its definition or, if you need multiple +configurations for the same provider, replace them with +_proxy configuration blocks_ as described in +[Passing Providers Explicitly](#passing-providers-explicitly). + +If the new version of the module uses proxy configuration blocks, or if the +calling module needs the child module to use different provider configurations +than its own default provider configurations, the calling module must then +include an explicit `providers` argument to describe which provider +configurations the child module will use: + +```hcl +provider "aws" { + region = "us-west-1" +} -``` provider "aws" { region = "us-east-1" alias = "east" @@ -448,24 +523,24 @@ provider "aws" { module "child" { count = 2 providers = { + # By default, the child module would use the + # default (unaliased) AWS provider configuration + # using us-west-1, but this will override it + # to use the additional "east" configuration + # for its resources instead. aws = aws.east } } ``` -Note how we are now [passing the providers](#passing-providers-explicitly) to the child module. - -In addition, modules using `count` or `for_each` cannot pass different sets of providers -to different instances. For example, you cannot interpolate variables in the `providers` -block on a module. - -This is because when a module instance is destroyed (such as a key-value being removed from the -`for_each` map), the appropriate provider must be available in order to perform the destroy. -You can pass different sets of providers to different module instances by using multiple `module` blocks: - -``` -# my_buckets.tf +Due to the association between resources and provider configurations being +static, module calls using `for_each` or `count` cannot pass different +provider configurations to different instances. If you need different +instances of your module to use different provider configurations then you +must use a separate `module` block for each distinct set of provider +configurations: +```hcl provider "aws" { alias = "usw1" region = "us-west-1" @@ -509,12 +584,6 @@ module "bucket_w2" { } ``` -Each module block may optionally have different providers passed to it -using the [`providers`](/docs/configuration/modules.html#passing-providers-explicitly) -argument. This can be useful in situations where, for example, a duplicated set of -resources must be created across several regions or datacenters. - - ## Tainting resources within a module The [taint command](/docs/commands/taint.html) can be used to _taint_ specific