Commit Graph

152 Commits

Author SHA1 Message Date
James Bardin 0e7dab09e6 Use the new registry.Client
The registry code has been moved into the new registry package. Remove
the duplicated code and use the new registry and registry/test packages.
2017-12-05 14:59:21 -05:00
Paul Banks 06f067b4e6
Update registry API docs with browse and search (#16846)
* Verify discovery works without trailing slash on discovery URL

* Update registry API docs with browse and search endpoints

* Add sample request/responses

* Add comment to test to indicate expecations

* Fix typo

* Remove trailing slash weirdness
2017-12-05 19:38:16 +00:00
Martin Atkins ba0514106a return tfdiags.Diagnostics from validation methods
Validation is the best time to return detailed diagnostics
to the user since we're much more likely to have source
location information, etc than we are in later operations.

This change doesn't actually add any detail to the messages
yet, but it changes the interface so that we can gradually
introduce more detailed diagnostics over time.

While here there are some minor adjustments to some of the
messages to improve their consistency with terminology we
use elsewhere.
2017-11-28 11:15:29 -08:00
James Bardin 8091bd627d move Svchost method to regsrc.Module
The level of abstraction that needs the "svchost" is the Module, not the
FriendlyHost. Us the new method in the module package for registry
interaction.
2017-11-20 18:29:09 -05:00
James Bardin bd576d780a failing test for module not found error
The "not found" error should use the raw string directly from the config
source, but the existing method was adding the default registry if there
was no host indicated.
2017-11-20 17:42:35 -05:00
James Bardin b15258dfec remove provider inheritance from Tree.Load
Now that resources can be connected to providers with different paths in
the core graph, handling the inheritance in config makes less sense.

Removing this to make room for core to walk the Tree and connect
resources directly to the proper provider instance.
2017-11-06 17:27:45 -05:00
James Bardin 2ed529f81a update from review
Reference modules by terraform address
Unquote versions
2017-11-03 10:50:19 -04:00
James Bardin c510473fd2 remove code added by stale vim buffer 2017-11-03 10:33:18 -04:00
James Bardin 1bf64ec788 add "Updating" output and fix output tests 2017-11-03 10:28:08 -04:00
James Bardin 9c334fe012 update init output
Change "Downloading" to 'Initializing" to match the provider loading
dialog.

List each module being loaded.

If a regisry module is being downloaded, list the registry host, and the
version discovered.

Show the source string from the config that is being fetched, rather
than the go-getter url. The full source can be found in the logs for
debugging.

Add much more extensive logging
2017-11-03 10:28:07 -04:00
James Bardin b4e9240679 remove implicit provier tests from config/module
No longer needed
2017-11-02 15:43:45 -04:00
James Bardin 838ffafc18 stop adding implicit provider configs
Now that providers in the graph can adopt resources without an explicit
provider, there's no need to add the implicit configs to the module.Tree
when loading.
2017-11-02 15:29:51 -04:00
Martin Atkins 505683dcd3 config/module: produce explicit error for non-registry hosts
If registry API discovery fails for a particular host then it's better to
generate an explicit error message for that early -- so we can tell the
user exactly what happened -- rather than assuming a default path and
then failing downstream when we get a 404 from that request.
2017-11-01 09:13:08 -07:00
Martin Atkins 288f7c00e4 config/module: allow registry download sources to be relative paths
It's not always easy or convenient for a web application to determine its
own absolute URL to return, so here we pragmatically allow the download
source string from a registry to be a path relative to the download
endpoint.

Since X-Terraform-Get is a go-getter string, not all valid values are
valid URLs and so we sniff for certain relative-path-looking prefixes
in order to decide whether to apply the relative lookup transform.
2017-10-30 10:37:25 -07:00
James Bardin eef2129f1a Storage.GetModule
Add GetModule for the cli to initialize from a regisry module source.

Storage.GetModule fetches a module using the same detection and
discovery as used by the normal module loading. The final copy is still
done by module.GetCopy to remove vcs files.
2017-10-27 19:17:26 -04:00
James Bardin ac68f9a16b make testCredentials token obviously fake 2017-10-27 17:36:30 -04:00
James Bardin ee56e3226b test that credentials are added to registry reqs
Add the missing set in lookupModuleLocation
2017-10-27 16:16:35 -04:00
James Bardin 4e8fe97556 add credentials to module.Storage
Provide a way to pass in credentials to be used by the module.Storage
when contacting registries.

Remove the mockTLSServer and use a static discovery map pointing to the
http url for tests.
2017-10-27 16:16:35 -04:00
James Bardin 3a495ffe56 rename ModuleStorage to Storage
get rid of stutter and use module.Storage
2017-10-27 13:11:21 -04:00
James Bardin f2a7b94692 use the new ModuleStorage in the command package
Update the command package to use the new module storage. Move the old
command output strings into the module storage itself. This could be
moved back later either by using ui callbacks, or designing a module
storage interface once we know what the final requirements will look
like.
2017-10-27 12:58:24 -04:00
James Bardin 36eb40a432 export ModuleStorage and use it for Tree.Load
Exporting ModuleStorage allows us to explicitly pass in the storgae
location rather than extracting it out of the getter.Storage interface,
set a UI for communiating actions back to the user, and accepting a
services Disco for discovery.
2017-10-27 11:29:29 -04:00
James Bardin a7df650f01 fix vet error 2017-10-27 09:08:15 -04:00
James Bardin 9bda82ccf1 add example for the reason behind versionedPathKey 2017-10-27 09:08:15 -04:00
James Bardin d6f75cc022 use URLs rather than strings is registry functions
Parse all the registry strings as urls, and compine with path.Join to
for better validation.
2017-10-27 09:08:15 -04:00
James Bardin 4a0de691f5 update test for error condition 2017-10-27 09:08:15 -04:00
James Bardin 6c808347d7 flag off registry ACC test
Was missing the TF_ACC check
2017-10-27 09:08:15 -04:00
James Bardin 1756b3d09f ProviderConfig needs an Inherited flag
If a provider configuration is inherited from another module, any
interpolations in that config won't have variables declared locally. Let
the config only be validated in it's original location.
2017-10-27 09:08:15 -04:00
James Bardin 0afd4a9097 Lookup registry module versions during Tree.Load.
Registry modules can't be handled directly by the getter.Storage
implementation, which doesn't know how to handle versions. First see if
we have a matching module stored that satisfies our constraints. If
not, and we're getting or updating, we can look it up in the registry.

This essentially takes the place of a "registry detector" for go-getter,
but required the intermediate step of resolving the version dependency.

This also starts breaking up the huge Tree.Load method into more
manageable parts. It was sorely needed, as indicated by the difficulty
encountered in this refactor. There's still a lot that can be done to
improve this, but at least there are now a few easier to read methods
when we come back to it.
2017-10-27 09:08:15 -04:00
James Bardin 0d10564a74 remove the registryDetector
The detection of registry modules will have to happen in mutliple
phases. The go-getter interface requires that the detector return the
final URL, while we won't know that until we verify which version we
need. This leaves the regisry sources broken, to be re-integrated in a
following commit.
2017-10-27 09:08:15 -04:00
James Bardin ee36cf28e0 a series of test commits
wire up HTTP so we can test the mock discovery service

test lookupModuleVersions

Add a versions endpoint to the mock registry, and use that to verify the
lookupModuleVersions behavior.

lookupModuleVersions takes a Disco as the argument so a custom Transport
can be injected, and uses that transport for its own client if it set.

test looking up modules with default registry

Add registry.terrform.io to the hostname that the mock registry resolves
to localhost.

ACC test looking up module versions

Lookup a basic module for the available version in the default registry.
2017-10-27 09:08:15 -04:00
James Bardin ae2903b810 missing current module source in versionedPathKey
The reason versionedPathKey didn't solve the previous test case was a
missing source string for the current module.
2017-10-27 09:08:15 -04:00
James Bardin c01dfa7757 failing test for changing intermediate modules
This test highlights how changing an intermediate source path prevents
reloading of submodules. While this is somewhat of an edge case now, it
becomes quite common in the cacse where module versions are updated.
2017-10-27 09:08:15 -04:00
James Bardin a5c86aeff6 Use the new regsrc and response packages
Adds basic detector for registry module source strings. While this isn't
a thorough validation, this will eliminate anything that is definitely
not a registry module, and split out our host and module id strings.

lookupModuleVersions interrogates the registry for the available
versions of a particular module and the tree of dependencies.
2017-10-27 09:08:15 -04:00
James Bardin 0bc279557e add more metadata to the Tree and the storage path
Submodules were located by using their module path as the storage key.
Now that modules may have versions, a submodule needs to know how to
locate the corect source depending on the versions of its ancestors in
the tree.

Add a version field to each Tree, and a pointer back to the parent Tree
to step back through the ancestors. The new versionedPathKey method uses
this information to build a unique key for each module, dependent on the
ancestor versions.

Not only do stored modules need to know their version if it exists, but
any relative source needs to know all the ancestor versions in order to
resolve correctly.
2017-10-27 09:08:15 -04:00
James Bardin 1f44fd8eb2 move getStorage to moduleStorage too
Add a test to ensure we don't store different modules with the same
relative path in the same location.
2017-10-27 09:08:15 -04:00
James Bardin 3d163917c7 move the getter.Storage into our moduleStorge type
Wrap the Storage interfaces together.
2017-10-27 09:08:15 -04:00
James Bardin 7416e7a6e0 update the module manifest implementation
The getter.Storage abstraction is proving entirely inadequate here, but
we can't replace it wholesale at the moment.

The Tree loader needs to know the location of the manifest before it can
start loading any modules. Since the version will have to be part of the
hashed storage key, there is no way to know what version of each module
are stored. The storageDir function will extract the StorageDir field
from the underlying FolderStorage instance for the tree to locate the
manifest.
2017-10-27 09:08:15 -04:00
James Bardin 283d88551a Add a new module storage manifest
To add registry support, a workaround in the local module storage was
added to record the subdirectory containing the module source from
within the archive file. Here we replace that temporary implementation
with the full manifest needed to record the necessary module metadata
for module loading.

In order to support versioned modules, the actual stored version needs
to be recorded. This can't be derived from the configuration, because
the configuration only contains the constraints, and at load time we need
to be able to enumerate the stored modules and all versions in order to
resolve them.

While the local storage key will be derived from the source and version,
that information is lost once it's hashed. While the entire storage
layer could be replaced to encode the needed data in the path itself,
this provides a minimal change to work with the existing storage code.
2017-10-27 09:08:15 -04:00
James Bardin 0962792257 surface registry errors to the user
Now that we can enforce local modules being relative or absolute paths,
we can be assured that any module source matching a registry pattern
must be found in the registry. This allows us to surface more useful
errors to the user, rather than simply stating that a source string
isn't valid.
2017-10-27 09:08:15 -04:00
James Bardin 40c36b48dc missing test-fixture files 2017-10-27 09:08:15 -04:00
James Bardin 8933ef8f70 make //subdir notation work with registry modules
Append the original source subdir to the discovered subdir when locating
the files in the module tree.
2017-10-27 09:08:15 -04:00
James Bardin a359c4ee2f check the registry detector before local files
Breaking change for 0.11.

Local files were checked first to avoid the possibility of breaking a
module with a local source that looked like a registry ID. Now we can
enfore that any source iwth the pattern "namespace/identifier/provider"
must be a registry module.
2017-10-27 09:08:15 -04:00
James Bardin 6088270c66 skip lookup for paths starting with ../
Prevent an extra registry lookup for sources starting with ../
2017-10-27 09:08:15 -04:00
James Bardin b0163e69d1 update comments 2017-10-27 09:08:15 -04:00
James Bardin 4ab70ba09d rename confusing test module name 2017-10-27 09:08:15 -04:00
James Bardin 2e505083cd change ProviderConfig.Scope to Path
Though it's intended for "interpolation scope", Path is generally used
for this elsewhere.
2017-10-27 09:08:15 -04:00
James Bardin cb0e37a870 implement provider inheritence during loading
This implements provider inheritance during config loading, rather than
during graph evaluation. At this point it's much simpler to find the
desired configuration, and once all providers are declared, all the
inheritance code in the graph can be removed.

The inheritance is dome by simply copying the RawConfig from the parent
ProviderConfig into the module. Since this happens before any
evaluation, we record the original interpolation scope in the
ProviderConfig so that it can be properly resolved later on.
2017-10-27 09:08:15 -04:00
James Bardin 29e5a355b9 add "version" and "provider" to module config
Add the Version and Providers fields to the module config.

Add ProviderConfig.Scope, which will be used to record the original
path of a ProviderConfig for interpolation.
2017-10-27 09:08:15 -04:00
Stefan Staudenmeyer 14792035cf Aliasing the module to omit errors thrown at build time
Signed-off-by: Stefan Staudenmeyer <stefan.staudenmeyer@instana.com>
2017-10-09 16:44:58 +02:00
James Bardin 1fbb2d611a remove extra slash from registry urls
A refactor introduced an extra `/` in the download url, which causes an
extra redirect during discovery.

Improve a registry test to verify that detection doesn't require the
registry after the modules have been fetched.
2017-09-28 14:45:02 -04:00