terraform/internal/initwd/module_install_test.go

698 lines
21 KiB
Go
Raw Normal View History

package initwd
import (
initwd: Error message for local paths escaping module packages Our module installer has a somewhat-informal idea of a "module package", which is some external thing we can go fetch in order to add one or more modules to the current configuration. Our documentation doesn't talk much about it because most users seem to have found the distinction between external and local modules pretty intuitive without us throwing a lot of funny terminology at them, but there are some situations where the distinction between a module and a module package are material to the end-user. One such situation is when using an absolute rather than relative filesystem path: we treat that as an external package in order to make the resulting working directory theoretically "portable" (although users can do various other things to defeat that), and so Terraform will copy the directory into .terraform/modules in the same way as it would download and extract a remote archive package or clone a git repository. A consequence of this, though, is that any relative paths called from inside a module loaded from an absolute path will fail if they try to traverse upward into the parent directory, because at runtime we're actually running from a copy of the directory that's been taking out of its original context. A similar sort of situation can occur in a truly remote module package if the author accidentally writes a "../" source path that traverses up out of the package root, and so this commit introduces a special error message for both situations that tries to be a bit clearer about there being a package boundary and use that to explain why installation failed. We would ideally have made escaping local references like that illegal in the first place, but sadly we did not and so when we rebuilt the module installer for Terraform v0.12 we ended up keeping the previous behavior of just trying it and letting it succeed if there happened to somehow be a matching directory at the given path, in order to remain compatible with situations that had worked by coincidence rather than intention. For that same reason, I've implemented this as a replacement error message we will return only if local module installation was going to fail anyway, and thus it only modifies the error message for some existing error situations rather than introducing new error situations. This also includes some light updates to the documentation to say a little more about how Terraform treats absolute paths, though aiming not to get too much into the weeds about module packages since it's something that most users can get away with never knowing.
2021-05-22 00:28:20 +02:00
"bytes"
"flag"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"strings"
"testing"
Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.
2021-05-28 04:24:59 +02:00
"github.com/davecgh/go-spew/spew"
"github.com/go-test/deep"
Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.
2021-05-28 04:24:59 +02:00
"github.com/google/go-cmp/cmp"
version "github.com/hashicorp/go-version"
"github.com/hashicorp/terraform/internal/configs"
"github.com/hashicorp/terraform/internal/configs/configload"
"github.com/hashicorp/terraform/internal/copy"
"github.com/hashicorp/terraform/internal/registry"
"github.com/hashicorp/terraform/internal/tfdiags"
_ "github.com/hashicorp/terraform/internal/logging"
)
func TestMain(m *testing.M) {
flag.Parse()
os.Exit(m.Run())
}
func TestModuleInstaller(t *testing.T) {
fixtureDir := filepath.Clean("testdata/local-modules")
dir, done := tempChdir(t, fixtureDir)
defer done()
hooks := &testInstallHooks{}
modulesDir := filepath.Join(dir, ".terraform/modules")
inst := NewModuleInstaller(modulesDir, nil)
command: "terraform init" can partially initialize for 0.12upgrade There are a few constructs from 0.11 and prior that cause 0.12 parsing to fail altogether, which previously created a chicken/egg problem because we need to install the providers in order to run "terraform 0.12upgrade" and thus fix the problem. This changes "terraform init" to use the new "early configuration" loader for module and provider installation. This is built on the more permissive parser in the terraform-config-inspect package, and so it allows us to read out the top-level blocks from the configuration while accepting legacy HCL syntax. In the long run this will let us do version compatibility detection before attempting a "real" config load, giving us better error messages for any future syntax additions, but in the short term the key thing is that it allows us to install the dependencies even if the configuration isn't fully valid. Because backend init still requires full configuration, this introduces a new mode of terraform init where it detects heuristically if it seems like we need to do a configuration upgrade and does a partial init if so, before finally directing the user to run "terraform 0.12upgrade" before running any other commands. The heuristic here is based on two assumptions: - If the "early" loader finds no errors but the normal loader does, the configuration is likely to be valid for Terraform 0.11 but not 0.12. - If there's already a version constraint in the configuration that excludes Terraform versions prior to v0.12 then the configuration is probably _already_ upgraded and so it's just a normal syntax error, even if the early loader didn't detect it. Once the upgrade process is removed in 0.13.0 (users will be required to go stepwise 0.11 -> 0.12 -> 0.13 to upgrade after that), some of this can be simplified to remove that special mode, but the idea of doing the dependency version checks against the liberal parser will remain valuable to increase our chances of reporting version-based incompatibilities rather than syntax errors as we add new features in future.
2019-01-14 20:11:00 +01:00
_, diags := inst.InstallModules(".", false, hooks)
assertNoDiagnostics(t, diags)
wantCalls := []testInstallHookCall{
{
Name: "Install",
ModuleAddr: "child_a",
PackageAddr: "",
LocalPath: "child_a",
},
{
Name: "Install",
ModuleAddr: "child_a.child_b",
PackageAddr: "",
LocalPath: "child_a/child_b",
},
}
if assertResultDeepEqual(t, hooks.Calls, wantCalls) {
return
}
loader, err := configload.NewLoader(&configload.Config{
ModulesDir: modulesDir,
})
if err != nil {
t.Fatal(err)
}
// Make sure the configuration is loadable now.
// (This ensures that correct information is recorded in the manifest.)
config, loadDiags := loader.LoadConfig(".")
assertNoDiagnostics(t, tfdiags.Diagnostics{}.Append(loadDiags))
wantTraces := map[string]string{
"": "in root module",
"child_a": "in child_a module",
"child_a.child_b": "in child_b module",
}
gotTraces := map[string]string{}
config.DeepEach(func(c *configs.Config) {
path := strings.Join(c.Path, ".")
if c.Module.Variables["v"] == nil {
gotTraces[path] = "<missing>"
return
}
varDesc := c.Module.Variables["v"].Description
gotTraces[path] = varDesc
})
assertResultDeepEqual(t, gotTraces, wantTraces)
}
func TestModuleInstaller_error(t *testing.T) {
fixtureDir := filepath.Clean("testdata/local-module-error")
dir, done := tempChdir(t, fixtureDir)
defer done()
hooks := &testInstallHooks{}
modulesDir := filepath.Join(dir, ".terraform/modules")
inst := NewModuleInstaller(modulesDir, nil)
_, diags := inst.InstallModules(".", false, hooks)
if !diags.HasErrors() {
t.Fatal("expected error")
} else {
Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.
2021-05-28 04:24:59 +02:00
assertDiagnosticSummary(t, diags, "Invalid module source address")
}
}
initwd: Error message for local paths escaping module packages Our module installer has a somewhat-informal idea of a "module package", which is some external thing we can go fetch in order to add one or more modules to the current configuration. Our documentation doesn't talk much about it because most users seem to have found the distinction between external and local modules pretty intuitive without us throwing a lot of funny terminology at them, but there are some situations where the distinction between a module and a module package are material to the end-user. One such situation is when using an absolute rather than relative filesystem path: we treat that as an external package in order to make the resulting working directory theoretically "portable" (although users can do various other things to defeat that), and so Terraform will copy the directory into .terraform/modules in the same way as it would download and extract a remote archive package or clone a git repository. A consequence of this, though, is that any relative paths called from inside a module loaded from an absolute path will fail if they try to traverse upward into the parent directory, because at runtime we're actually running from a copy of the directory that's been taking out of its original context. A similar sort of situation can occur in a truly remote module package if the author accidentally writes a "../" source path that traverses up out of the package root, and so this commit introduces a special error message for both situations that tries to be a bit clearer about there being a package boundary and use that to explain why installation failed. We would ideally have made escaping local references like that illegal in the first place, but sadly we did not and so when we rebuilt the module installer for Terraform v0.12 we ended up keeping the previous behavior of just trying it and letting it succeed if there happened to somehow be a matching directory at the given path, in order to remain compatible with situations that had worked by coincidence rather than intention. For that same reason, I've implemented this as a replacement error message we will return only if local module installation was going to fail anyway, and thus it only modifies the error message for some existing error situations rather than introducing new error situations. This also includes some light updates to the documentation to say a little more about how Terraform treats absolute paths, though aiming not to get too much into the weeds about module packages since it's something that most users can get away with never knowing.
2021-05-22 00:28:20 +02:00
func TestModuleInstaller_packageEscapeError(t *testing.T) {
fixtureDir := filepath.Clean("testdata/load-module-package-escape")
dir, done := tempChdir(t, fixtureDir)
defer done()
// For this particular test we need an absolute path in the root module
// that must actually resolve to our temporary directory in "dir", so
// we need to do a little rewriting. We replace the arbitrary placeholder
// %%BASE%% with the temporary directory path.
{
rootFilename := filepath.Join(dir, "package-escape.tf")
template, err := ioutil.ReadFile(rootFilename)
if err != nil {
t.Fatal(err)
}
final := bytes.ReplaceAll(template, []byte("%%BASE%%"), []byte(filepath.ToSlash(dir)))
err = ioutil.WriteFile(rootFilename, final, 0644)
if err != nil {
t.Fatal(err)
}
}
hooks := &testInstallHooks{}
modulesDir := filepath.Join(dir, ".terraform/modules")
inst := NewModuleInstaller(modulesDir, nil)
_, diags := inst.InstallModules(".", false, hooks)
if !diags.HasErrors() {
t.Fatal("expected error")
} else {
assertDiagnosticSummary(t, diags, "Local module path escapes module package")
}
}
func TestModuleInstaller_explicitPackageBoundary(t *testing.T) {
fixtureDir := filepath.Clean("testdata/load-module-package-prefix")
dir, done := tempChdir(t, fixtureDir)
defer done()
// For this particular test we need an absolute path in the root module
// that must actually resolve to our temporary directory in "dir", so
// we need to do a little rewriting. We replace the arbitrary placeholder
// %%BASE%% with the temporary directory path.
{
rootFilename := filepath.Join(dir, "package-prefix.tf")
template, err := ioutil.ReadFile(rootFilename)
if err != nil {
t.Fatal(err)
}
final := bytes.ReplaceAll(template, []byte("%%BASE%%"), []byte(filepath.ToSlash(dir)))
err = ioutil.WriteFile(rootFilename, final, 0644)
if err != nil {
t.Fatal(err)
}
}
hooks := &testInstallHooks{}
modulesDir := filepath.Join(dir, ".terraform/modules")
inst := NewModuleInstaller(modulesDir, nil)
_, diags := inst.InstallModules(".", false, hooks)
if diags.HasErrors() {
t.Fatalf("unexpected errors\n%s", diags.Err().Error())
}
}
func TestModuleInstaller_invalid_version_constraint_error(t *testing.T) {
fixtureDir := filepath.Clean("testdata/invalid-version-constraint")
dir, done := tempChdir(t, fixtureDir)
defer done()
hooks := &testInstallHooks{}
modulesDir := filepath.Join(dir, ".terraform/modules")
inst := NewModuleInstaller(modulesDir, nil)
_, diags := inst.InstallModules(".", false, hooks)
if !diags.HasErrors() {
t.Fatal("expected error")
} else {
assertDiagnosticSummary(t, diags, "Invalid version constraint")
}
}
func TestModuleInstaller_invalidVersionConstraintGetter(t *testing.T) {
fixtureDir := filepath.Clean("testdata/invalid-version-constraint")
dir, done := tempChdir(t, fixtureDir)
defer done()
hooks := &testInstallHooks{}
modulesDir := filepath.Join(dir, ".terraform/modules")
inst := NewModuleInstaller(modulesDir, nil)
_, diags := inst.InstallModules(".", false, hooks)
if !diags.HasErrors() {
t.Fatal("expected error")
} else {
assertDiagnosticSummary(t, diags, "Invalid version constraint")
}
}
func TestModuleInstaller_invalidVersionConstraintLocal(t *testing.T) {
fixtureDir := filepath.Clean("testdata/invalid-version-constraint-local")
dir, done := tempChdir(t, fixtureDir)
defer done()
hooks := &testInstallHooks{}
modulesDir := filepath.Join(dir, ".terraform/modules")
inst := NewModuleInstaller(modulesDir, nil)
_, diags := inst.InstallModules(".", false, hooks)
if !diags.HasErrors() {
t.Fatal("expected error")
} else {
assertDiagnosticSummary(t, diags, "Invalid version constraint")
}
}
func TestModuleInstaller_symlink(t *testing.T) {
fixtureDir := filepath.Clean("testdata/local-module-symlink")
dir, done := tempChdir(t, fixtureDir)
defer done()
hooks := &testInstallHooks{}
modulesDir := filepath.Join(dir, ".terraform/modules")
inst := NewModuleInstaller(modulesDir, nil)
_, diags := inst.InstallModules(".", false, hooks)
assertNoDiagnostics(t, diags)
wantCalls := []testInstallHookCall{
{
Name: "Install",
ModuleAddr: "child_a",
PackageAddr: "",
LocalPath: "child_a",
},
{
Name: "Install",
ModuleAddr: "child_a.child_b",
PackageAddr: "",
LocalPath: "child_a/child_b",
},
}
if assertResultDeepEqual(t, hooks.Calls, wantCalls) {
return
}
loader, err := configload.NewLoader(&configload.Config{
ModulesDir: modulesDir,
})
if err != nil {
t.Fatal(err)
}
// Make sure the configuration is loadable now.
// (This ensures that correct information is recorded in the manifest.)
config, loadDiags := loader.LoadConfig(".")
assertNoDiagnostics(t, tfdiags.Diagnostics{}.Append(loadDiags))
wantTraces := map[string]string{
"": "in root module",
"child_a": "in child_a module",
"child_a.child_b": "in child_b module",
}
gotTraces := map[string]string{}
config.DeepEach(func(c *configs.Config) {
path := strings.Join(c.Path, ".")
if c.Module.Variables["v"] == nil {
gotTraces[path] = "<missing>"
return
}
varDesc := c.Module.Variables["v"].Description
gotTraces[path] = varDesc
})
assertResultDeepEqual(t, gotTraces, wantTraces)
}
func TestLoaderInstallModules_registry(t *testing.T) {
if os.Getenv("TF_ACC") == "" {
t.Skip("this test accesses registry.terraform.io and github.com; set TF_ACC=1 to run it")
}
fixtureDir := filepath.Clean("testdata/registry-modules")
tmpDir, done := tempChdir(t, fixtureDir)
// the module installer runs filepath.EvalSymlinks() on the destination
// directory before copying files, and the resultant directory is what is
// returned by the install hooks. Without this, tests could fail on machines
// where the default temp dir was a symlink.
dir, err := filepath.EvalSymlinks(tmpDir)
if err != nil {
t.Error(err)
}
defer done()
hooks := &testInstallHooks{}
modulesDir := filepath.Join(dir, ".terraform/modules")
inst := NewModuleInstaller(modulesDir, registry.NewClient(nil, nil))
command: "terraform init" can partially initialize for 0.12upgrade There are a few constructs from 0.11 and prior that cause 0.12 parsing to fail altogether, which previously created a chicken/egg problem because we need to install the providers in order to run "terraform 0.12upgrade" and thus fix the problem. This changes "terraform init" to use the new "early configuration" loader for module and provider installation. This is built on the more permissive parser in the terraform-config-inspect package, and so it allows us to read out the top-level blocks from the configuration while accepting legacy HCL syntax. In the long run this will let us do version compatibility detection before attempting a "real" config load, giving us better error messages for any future syntax additions, but in the short term the key thing is that it allows us to install the dependencies even if the configuration isn't fully valid. Because backend init still requires full configuration, this introduces a new mode of terraform init where it detects heuristically if it seems like we need to do a configuration upgrade and does a partial init if so, before finally directing the user to run "terraform 0.12upgrade" before running any other commands. The heuristic here is based on two assumptions: - If the "early" loader finds no errors but the normal loader does, the configuration is likely to be valid for Terraform 0.11 but not 0.12. - If there's already a version constraint in the configuration that excludes Terraform versions prior to v0.12 then the configuration is probably _already_ upgraded and so it's just a normal syntax error, even if the early loader didn't detect it. Once the upgrade process is removed in 0.13.0 (users will be required to go stepwise 0.11 -> 0.12 -> 0.13 to upgrade after that), some of this can be simplified to remove that special mode, but the idea of doing the dependency version checks against the liberal parser will remain valuable to increase our chances of reporting version-based incompatibilities rather than syntax errors as we add new features in future.
2019-01-14 20:11:00 +01:00
_, diags := inst.InstallModules(dir, false, hooks)
assertNoDiagnostics(t, diags)
v := version.Must(version.NewVersion("0.0.1"))
wantCalls := []testInstallHookCall{
// the configuration builder visits each level of calls in lexicographical
// order by name, so the following list is kept in the same order.
// acctest_child_a accesses //modules/child_a directly
{
Name: "Download",
ModuleAddr: "acctest_child_a",
Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.
2021-05-28 04:24:59 +02:00
PackageAddr: "registry.terraform.io/hashicorp/module-installer-acctest/aws", // intentionally excludes the subdir because we're downloading the whole package here
Version: v,
},
{
Name: "Install",
ModuleAddr: "acctest_child_a",
Version: v,
// NOTE: This local path and the other paths derived from it below
// can vary depending on how the registry is implemented. At the
// time of writing this test, registry.terraform.io returns
// git repository source addresses and so this path refers to the
// root of the git clone, but historically the registry referred
// to GitHub-provided tar archives which meant that there was an
// extra level of subdirectory here for the typical directory
// nesting in tar archives, which would've been reflected as
// an extra segment on this path. If this test fails due to an
// additional path segment in future, then a change to the upstream
// registry might be the root cause.
LocalPath: filepath.Join(dir, ".terraform/modules/acctest_child_a/modules/child_a"),
},
// acctest_child_a.child_b
// (no download because it's a relative path inside acctest_child_a)
{
Name: "Install",
ModuleAddr: "acctest_child_a.child_b",
LocalPath: filepath.Join(dir, ".terraform/modules/acctest_child_a/modules/child_b"),
},
// acctest_child_b accesses //modules/child_b directly
{
Name: "Download",
ModuleAddr: "acctest_child_b",
Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.
2021-05-28 04:24:59 +02:00
PackageAddr: "registry.terraform.io/hashicorp/module-installer-acctest/aws", // intentionally excludes the subdir because we're downloading the whole package here
Version: v,
},
{
Name: "Install",
ModuleAddr: "acctest_child_b",
Version: v,
LocalPath: filepath.Join(dir, ".terraform/modules/acctest_child_b/modules/child_b"),
},
// acctest_root
{
Name: "Download",
ModuleAddr: "acctest_root",
Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.
2021-05-28 04:24:59 +02:00
PackageAddr: "registry.terraform.io/hashicorp/module-installer-acctest/aws",
Version: v,
},
{
Name: "Install",
ModuleAddr: "acctest_root",
Version: v,
LocalPath: filepath.Join(dir, ".terraform/modules/acctest_root"),
},
// acctest_root.child_a
// (no download because it's a relative path inside acctest_root)
{
Name: "Install",
ModuleAddr: "acctest_root.child_a",
LocalPath: filepath.Join(dir, ".terraform/modules/acctest_root/modules/child_a"),
},
// acctest_root.child_a.child_b
// (no download because it's a relative path inside acctest_root, via acctest_root.child_a)
{
Name: "Install",
ModuleAddr: "acctest_root.child_a.child_b",
LocalPath: filepath.Join(dir, ".terraform/modules/acctest_root/modules/child_b"),
},
}
Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.
2021-05-28 04:24:59 +02:00
if diff := cmp.Diff(wantCalls, hooks.Calls); diff != "" {
t.Fatalf("wrong installer calls\n%s", diff)
}
//check that the registry reponses were cached
Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.
2021-05-28 04:24:59 +02:00
if _, ok := inst.moduleVersions["registry.terraform.io/hashicorp/module-installer-acctest/aws"]; !ok {
t.Errorf("module versions cache was not populated\ngot: %s\nwant: key hashicorp/module-installer-acctest/aws", spew.Sdump(inst.moduleVersions))
}
Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.
2021-05-28 04:24:59 +02:00
if _, ok := inst.moduleVersionsUrl[moduleVersion{module: "registry.terraform.io/hashicorp/module-installer-acctest/aws", version: "0.0.1"}]; !ok {
t.Errorf("module download url cache was not populated\ngot: %s", spew.Sdump(inst.moduleVersionsUrl))
}
loader, err := configload.NewLoader(&configload.Config{
ModulesDir: modulesDir,
})
if err != nil {
t.Fatal(err)
}
// Make sure the configuration is loadable now.
// (This ensures that correct information is recorded in the manifest.)
config, loadDiags := loader.LoadConfig(".")
assertNoDiagnostics(t, tfdiags.Diagnostics{}.Append(loadDiags))
wantTraces := map[string]string{
"": "in local caller for registry-modules",
"acctest_root": "in root module",
"acctest_root.child_a": "in child_a module",
"acctest_root.child_a.child_b": "in child_b module",
"acctest_child_a": "in child_a module",
"acctest_child_a.child_b": "in child_b module",
"acctest_child_b": "in child_b module",
}
gotTraces := map[string]string{}
config.DeepEach(func(c *configs.Config) {
path := strings.Join(c.Path, ".")
if c.Module.Variables["v"] == nil {
gotTraces[path] = "<missing>"
return
}
varDesc := c.Module.Variables["v"].Description
gotTraces[path] = varDesc
})
assertResultDeepEqual(t, gotTraces, wantTraces)
}
func TestLoaderInstallModules_goGetter(t *testing.T) {
if os.Getenv("TF_ACC") == "" {
t.Skip("this test accesses github.com; set TF_ACC=1 to run it")
}
fixtureDir := filepath.Clean("testdata/go-getter-modules")
tmpDir, done := tempChdir(t, fixtureDir)
// the module installer runs filepath.EvalSymlinks() on the destination
// directory before copying files, and the resultant directory is what is
// returned by the install hooks. Without this, tests could fail on machines
// where the default temp dir was a symlink.
dir, err := filepath.EvalSymlinks(tmpDir)
if err != nil {
t.Error(err)
}
defer done()
hooks := &testInstallHooks{}
modulesDir := filepath.Join(dir, ".terraform/modules")
inst := NewModuleInstaller(modulesDir, registry.NewClient(nil, nil))
command: "terraform init" can partially initialize for 0.12upgrade There are a few constructs from 0.11 and prior that cause 0.12 parsing to fail altogether, which previously created a chicken/egg problem because we need to install the providers in order to run "terraform 0.12upgrade" and thus fix the problem. This changes "terraform init" to use the new "early configuration" loader for module and provider installation. This is built on the more permissive parser in the terraform-config-inspect package, and so it allows us to read out the top-level blocks from the configuration while accepting legacy HCL syntax. In the long run this will let us do version compatibility detection before attempting a "real" config load, giving us better error messages for any future syntax additions, but in the short term the key thing is that it allows us to install the dependencies even if the configuration isn't fully valid. Because backend init still requires full configuration, this introduces a new mode of terraform init where it detects heuristically if it seems like we need to do a configuration upgrade and does a partial init if so, before finally directing the user to run "terraform 0.12upgrade" before running any other commands. The heuristic here is based on two assumptions: - If the "early" loader finds no errors but the normal loader does, the configuration is likely to be valid for Terraform 0.11 but not 0.12. - If there's already a version constraint in the configuration that excludes Terraform versions prior to v0.12 then the configuration is probably _already_ upgraded and so it's just a normal syntax error, even if the early loader didn't detect it. Once the upgrade process is removed in 0.13.0 (users will be required to go stepwise 0.11 -> 0.12 -> 0.13 to upgrade after that), some of this can be simplified to remove that special mode, but the idea of doing the dependency version checks against the liberal parser will remain valuable to increase our chances of reporting version-based incompatibilities rather than syntax errors as we add new features in future.
2019-01-14 20:11:00 +01:00
_, diags := inst.InstallModules(dir, false, hooks)
assertNoDiagnostics(t, diags)
wantCalls := []testInstallHookCall{
// the configuration builder visits each level of calls in lexicographical
// order by name, so the following list is kept in the same order.
// acctest_child_a accesses //modules/child_a directly
{
Name: "Download",
ModuleAddr: "acctest_child_a",
Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.
2021-05-28 04:24:59 +02:00
PackageAddr: "git::https://github.com/hashicorp/terraform-aws-module-installer-acctest.git?ref=v0.0.1", // intentionally excludes the subdir because we're downloading the whole repo here
},
{
Name: "Install",
ModuleAddr: "acctest_child_a",
LocalPath: filepath.Join(dir, ".terraform/modules/acctest_child_a/modules/child_a"),
},
// acctest_child_a.child_b
// (no download because it's a relative path inside acctest_child_a)
{
Name: "Install",
ModuleAddr: "acctest_child_a.child_b",
LocalPath: filepath.Join(dir, ".terraform/modules/acctest_child_a/modules/child_b"),
},
// acctest_child_b accesses //modules/child_b directly
{
Name: "Download",
ModuleAddr: "acctest_child_b",
Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.
2021-05-28 04:24:59 +02:00
PackageAddr: "git::https://github.com/hashicorp/terraform-aws-module-installer-acctest.git?ref=v0.0.1", // intentionally excludes the subdir because we're downloading the whole package here
},
{
Name: "Install",
ModuleAddr: "acctest_child_b",
LocalPath: filepath.Join(dir, ".terraform/modules/acctest_child_b/modules/child_b"),
},
// acctest_root
{
Name: "Download",
ModuleAddr: "acctest_root",
Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.
2021-05-28 04:24:59 +02:00
PackageAddr: "git::https://github.com/hashicorp/terraform-aws-module-installer-acctest.git?ref=v0.0.1",
},
{
Name: "Install",
ModuleAddr: "acctest_root",
LocalPath: filepath.Join(dir, ".terraform/modules/acctest_root"),
},
// acctest_root.child_a
// (no download because it's a relative path inside acctest_root)
{
Name: "Install",
ModuleAddr: "acctest_root.child_a",
LocalPath: filepath.Join(dir, ".terraform/modules/acctest_root/modules/child_a"),
},
// acctest_root.child_a.child_b
// (no download because it's a relative path inside acctest_root, via acctest_root.child_a)
{
Name: "Install",
ModuleAddr: "acctest_root.child_a.child_b",
LocalPath: filepath.Join(dir, ".terraform/modules/acctest_root/modules/child_b"),
},
}
Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.
2021-05-28 04:24:59 +02:00
if diff := cmp.Diff(wantCalls, hooks.Calls); diff != "" {
t.Fatalf("wrong installer calls\n%s", diff)
}
loader, err := configload.NewLoader(&configload.Config{
ModulesDir: modulesDir,
})
if err != nil {
t.Fatal(err)
}
// Make sure the configuration is loadable now.
// (This ensures that correct information is recorded in the manifest.)
config, loadDiags := loader.LoadConfig(".")
assertNoDiagnostics(t, tfdiags.Diagnostics{}.Append(loadDiags))
wantTraces := map[string]string{
"": "in local caller for go-getter-modules",
"acctest_root": "in root module",
"acctest_root.child_a": "in child_a module",
"acctest_root.child_a.child_b": "in child_b module",
"acctest_child_a": "in child_a module",
"acctest_child_a.child_b": "in child_b module",
"acctest_child_b": "in child_b module",
}
gotTraces := map[string]string{}
config.DeepEach(func(c *configs.Config) {
path := strings.Join(c.Path, ".")
if c.Module.Variables["v"] == nil {
gotTraces[path] = "<missing>"
return
}
varDesc := c.Module.Variables["v"].Description
gotTraces[path] = varDesc
})
assertResultDeepEqual(t, gotTraces, wantTraces)
}
type testInstallHooks struct {
Calls []testInstallHookCall
}
type testInstallHookCall struct {
Name string
ModuleAddr string
PackageAddr string
Version *version.Version
LocalPath string
}
func (h *testInstallHooks) Download(moduleAddr, packageAddr string, version *version.Version) {
h.Calls = append(h.Calls, testInstallHookCall{
Name: "Download",
ModuleAddr: moduleAddr,
PackageAddr: packageAddr,
Version: version,
})
}
func (h *testInstallHooks) Install(moduleAddr string, version *version.Version, localPath string) {
h.Calls = append(h.Calls, testInstallHookCall{
Name: "Install",
ModuleAddr: moduleAddr,
Version: version,
LocalPath: localPath,
})
}
// tempChdir copies the contents of the given directory to a temporary
// directory and changes the test process's current working directory to
// point to that directory. Also returned is a function that should be
// called at the end of the test (e.g. via "defer") to restore the previous
// working directory.
//
// Tests using this helper cannot safely be run in parallel with other tests.
func tempChdir(t *testing.T, sourceDir string) (string, func()) {
t.Helper()
tmpDir, err := ioutil.TempDir("", "terraform-configload")
if err != nil {
t.Fatalf("failed to create temporary directory: %s", err)
return "", nil
}
if err := copy.CopyDir(tmpDir, sourceDir); err != nil {
t.Fatalf("failed to copy fixture to temporary directory: %s", err)
return "", nil
}
oldDir, err := os.Getwd()
if err != nil {
t.Fatalf("failed to determine current working directory: %s", err)
return "", nil
}
err = os.Chdir(tmpDir)
if err != nil {
t.Fatalf("failed to switch to temp dir %s: %s", tmpDir, err)
return "", nil
}
// Most of the tests need this, so we'll make it just in case.
os.MkdirAll(filepath.Join(tmpDir, ".terraform/modules"), os.ModePerm)
t.Logf("tempChdir switched to %s after copying from %s", tmpDir, sourceDir)
return tmpDir, func() {
err := os.Chdir(oldDir)
if err != nil {
panic(fmt.Errorf("failed to restore previous working directory %s: %s", oldDir, err))
}
if os.Getenv("TF_CONFIGLOAD_TEST_KEEP_TMP") == "" {
os.RemoveAll(tmpDir)
}
}
}
func assertNoDiagnostics(t *testing.T, diags tfdiags.Diagnostics) bool {
t.Helper()
return assertDiagnosticCount(t, diags, 0)
}
func assertDiagnosticCount(t *testing.T, diags tfdiags.Diagnostics, want int) bool {
t.Helper()
if len(diags) != 0 {
t.Errorf("wrong number of diagnostics %d; want %d", len(diags), want)
for _, diag := range diags {
Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.
2021-05-28 04:24:59 +02:00
t.Logf("- %#v", diag)
}
return true
}
return false
}
func assertDiagnosticSummary(t *testing.T, diags tfdiags.Diagnostics, want string) bool {
t.Helper()
for _, diag := range diags {
if diag.Description().Summary == want {
return false
}
}
t.Errorf("missing diagnostic summary %q", want)
for _, diag := range diags {
Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.
2021-05-28 04:24:59 +02:00
t.Logf("- %#v", diag)
}
return true
}
func assertResultDeepEqual(t *testing.T, got, want interface{}) bool {
t.Helper()
if diff := deep.Equal(got, want); diff != nil {
for _, problem := range diff {
t.Errorf("%s", problem)
}
return true
}
return false
}