Commit Graph

736 Commits

Author SHA1 Message Date
Rob Campbell 5daeee5f6d Update various files for new version of "stringer"
The latest version of stringer now uses strconv instead of fmt.
2017-12-11 13:26:29 -08:00
Martin Atkins 87e1fb4d66 config: a nicer error message for invalid provider constraints
Previously our error message here was confusing and redundant:

    Error starting operation: provider.null: invalid version constraint "not valid": Malformed constraint: not valid

Instead, we'll generate a full HCL2 diagnostic here, which results in
something (subjectively) nicer:

    Error: Invalid provider version constraint

    The value "@ 1.0.0" given for provider.null is not a valid version
    constraint.

At the moment this message is an outlier in that the other validation
errors are all still just plain Go errors, but over time we'll want to
adjust all of these to be full diagnostics so that we can embed source
range information in them to help the user find the offending
configuration.
2017-12-07 14:28:43 -08:00
James Bardin e0b2c64645 accessing count directly in an output should is OK
There should be no warning when accessing a resource's count value
directly in an output.
2017-12-06 18:59:53 -05:00
James Bardin 0e7dab09e6 Use the new registry.Client
The registry code has been moved into the new registry package. Remove
the duplicated code and use the new registry and registry/test packages.
2017-12-05 14:59:21 -05:00
Paul Banks 06f067b4e6
Update registry API docs with browse and search (#16846)
* Verify discovery works without trailing slash on discovery URL

* Update registry API docs with browse and search endpoints

* Add sample request/responses

* Add comment to test to indicate expecations

* Fix typo

* Remove trailing slash weirdness
2017-12-05 19:38:16 +00:00
David Meyer fe0cc3b0db config: new "rsadecrypt" interpolation function
The primary use-case is to decrypt windows passwords obtained from AWS, but this function is generic and may find other purposes too.
2017-12-05 11:06:32 -08:00
mizuno 01c1142589 config: new "timeadd" interpolation function 2017-12-05 10:46:12 -08:00
James Bardin 048af6e237 Add test fixture to check for warnings 2017-11-28 14:18:54 -05:00
Martin Atkins cc5e480378 config: detect and warn about access to "counted" attributes
A common pattern is to conditionally assign to "count"
in a resource in order to decide dynamically whether it
should be created. In that situation it's necessary to
refer to attributes of the resource using the splat
syntax, but historically we didn't show errors in output
expressions and so people "got away with" incorrect usage
in that context.

The intent of this warning is to catch
potentially-problematic usage of attributes on such
resources even if the count happens to be currently
set dynamically to 1, which would not generate the
error. Then the user can quickly locate and fix the
incorrect usage regardless of the current value.
2017-11-28 11:15:29 -08:00
Martin Atkins ba0514106a return tfdiags.Diagnostics from validation methods
Validation is the best time to return detailed diagnostics
to the user since we're much more likely to have source
location information, etc than we are in later operations.

This change doesn't actually add any detail to the messages
yet, but it changes the interface so that we can gradually
introduce more detailed diagnostics over time.

While here there are some minor adjustments to some of the
messages to improve their consistency with terminology we
use elsewhere.
2017-11-28 11:15:29 -08:00
James Bardin 8091bd627d move Svchost method to regsrc.Module
The level of abstraction that needs the "svchost" is the Module, not the
FriendlyHost. Us the new method in the module package for registry
interaction.
2017-11-20 18:29:09 -05:00
James Bardin bd576d780a failing test for module not found error
The "not found" error should use the raw string directly from the config
source, but the existing method was adding the default registry if there
was no host indicated.
2017-11-20 17:42:35 -05:00
James Bardin 8619f566bb validate providers passed to a module exist
This validation was lost in the migration of inheritance back to core.
Make sure a module configuraton doesn't specifiy a provider that doesn't
exist.
2017-11-10 11:01:32 -05:00
James Bardin b79adeae02 save resolved providers for resources to state
Use the ResourceState.Provider field to store the full name of the
provider used during apply. This field is only used when a resource is
removed from the config, and will allow that resource to be removed by
the exact same provider with which it was created.

Modify the locations which might accept the alue of the
ResourceState.Provider field to detect that the name is resolved.
2017-11-07 13:09:36 -05:00
James Bardin b15258dfec remove provider inheritance from Tree.Load
Now that resources can be connected to providers with different paths in
the core graph, handling the inheritance in config makes less sense.

Removing this to make room for core to walk the Tree and connect
resources directly to the proper provider instance.
2017-11-06 17:27:45 -05:00
James Bardin 57470f6bf5 remove ProviderConfig Path and Inherited fields
maving all inheritance into core
2017-11-06 17:11:07 -05:00
James Bardin 3297f60d03 remove raw print statements 2017-11-06 17:10:33 -05:00
James Bardin 2ed529f81a update from review
Reference modules by terraform address
Unquote versions
2017-11-03 10:50:19 -04:00
James Bardin c510473fd2 remove code added by stale vim buffer 2017-11-03 10:33:18 -04:00
James Bardin 1bf64ec788 add "Updating" output and fix output tests 2017-11-03 10:28:08 -04:00
James Bardin 9c334fe012 update init output
Change "Downloading" to 'Initializing" to match the provider loading
dialog.

List each module being loaded.

If a regisry module is being downloaded, list the registry host, and the
version discovered.

Show the source string from the config that is being fetched, rather
than the go-getter url. The full source can be found in the logs for
debugging.

Add much more extensive logging
2017-11-03 10:28:07 -04:00
James Bardin b4e9240679 remove implicit provier tests from config/module
No longer needed
2017-11-02 15:43:45 -04:00
James Bardin 838ffafc18 stop adding implicit provider configs
Now that providers in the graph can adopt resources without an explicit
provider, there's no need to add the implicit configs to the module.Tree
when loading.
2017-11-02 15:29:51 -04:00
Martin Atkins 505683dcd3 config/module: produce explicit error for non-registry hosts
If registry API discovery fails for a particular host then it's better to
generate an explicit error message for that early -- so we can tell the
user exactly what happened -- rather than assuming a default path and
then failing downstream when we get a 404 from that request.
2017-11-01 09:13:08 -07:00
Martin Atkins 288f7c00e4 config/module: allow registry download sources to be relative paths
It's not always easy or convenient for a web application to determine its
own absolute URL to return, so here we pragmatically allow the download
source string from a registry to be a path relative to the download
endpoint.

Since X-Terraform-Get is a go-getter string, not all valid values are
valid URLs and so we sniff for certain relative-path-looking prefixes
in order to decide whether to apply the relative lookup transform.
2017-10-30 10:37:25 -07:00
James Bardin eef2129f1a Storage.GetModule
Add GetModule for the cli to initialize from a regisry module source.

Storage.GetModule fetches a module using the same detection and
discovery as used by the normal module loading. The final copy is still
done by module.GetCopy to remove vcs files.
2017-10-27 19:17:26 -04:00
James Bardin ac68f9a16b make testCredentials token obviously fake 2017-10-27 17:36:30 -04:00
James Bardin ee56e3226b test that credentials are added to registry reqs
Add the missing set in lookupModuleLocation
2017-10-27 16:16:35 -04:00
James Bardin 4e8fe97556 add credentials to module.Storage
Provide a way to pass in credentials to be used by the module.Storage
when contacting registries.

Remove the mockTLSServer and use a static discovery map pointing to the
http url for tests.
2017-10-27 16:16:35 -04:00
James Bardin 3a495ffe56 rename ModuleStorage to Storage
get rid of stutter and use module.Storage
2017-10-27 13:11:21 -04:00
James Bardin f2a7b94692 use the new ModuleStorage in the command package
Update the command package to use the new module storage. Move the old
command output strings into the module storage itself. This could be
moved back later either by using ui callbacks, or designing a module
storage interface once we know what the final requirements will look
like.
2017-10-27 12:58:24 -04:00
James Bardin 36eb40a432 export ModuleStorage and use it for Tree.Load
Exporting ModuleStorage allows us to explicitly pass in the storgae
location rather than extracting it out of the getter.Storage interface,
set a UI for communiating actions back to the user, and accepting a
services Disco for discovery.
2017-10-27 11:29:29 -04:00
James Bardin a7df650f01 fix vet error 2017-10-27 09:08:15 -04:00
James Bardin 9bda82ccf1 add example for the reason behind versionedPathKey 2017-10-27 09:08:15 -04:00
James Bardin d6f75cc022 use URLs rather than strings is registry functions
Parse all the registry strings as urls, and compine with path.Join to
for better validation.
2017-10-27 09:08:15 -04:00
James Bardin 4a0de691f5 update test for error condition 2017-10-27 09:08:15 -04:00
James Bardin 6c808347d7 flag off registry ACC test
Was missing the TF_ACC check
2017-10-27 09:08:15 -04:00
James Bardin 1756b3d09f ProviderConfig needs an Inherited flag
If a provider configuration is inherited from another module, any
interpolations in that config won't have variables declared locally. Let
the config only be validated in it's original location.
2017-10-27 09:08:15 -04:00
James Bardin 0afd4a9097 Lookup registry module versions during Tree.Load.
Registry modules can't be handled directly by the getter.Storage
implementation, which doesn't know how to handle versions. First see if
we have a matching module stored that satisfies our constraints. If
not, and we're getting or updating, we can look it up in the registry.

This essentially takes the place of a "registry detector" for go-getter,
but required the intermediate step of resolving the version dependency.

This also starts breaking up the huge Tree.Load method into more
manageable parts. It was sorely needed, as indicated by the difficulty
encountered in this refactor. There's still a lot that can be done to
improve this, but at least there are now a few easier to read methods
when we come back to it.
2017-10-27 09:08:15 -04:00
James Bardin 0d10564a74 remove the registryDetector
The detection of registry modules will have to happen in mutliple
phases. The go-getter interface requires that the detector return the
final URL, while we won't know that until we verify which version we
need. This leaves the regisry sources broken, to be re-integrated in a
following commit.
2017-10-27 09:08:15 -04:00
James Bardin ee36cf28e0 a series of test commits
wire up HTTP so we can test the mock discovery service

test lookupModuleVersions

Add a versions endpoint to the mock registry, and use that to verify the
lookupModuleVersions behavior.

lookupModuleVersions takes a Disco as the argument so a custom Transport
can be injected, and uses that transport for its own client if it set.

test looking up modules with default registry

Add registry.terrform.io to the hostname that the mock registry resolves
to localhost.

ACC test looking up module versions

Lookup a basic module for the available version in the default registry.
2017-10-27 09:08:15 -04:00
James Bardin ae2903b810 missing current module source in versionedPathKey
The reason versionedPathKey didn't solve the previous test case was a
missing source string for the current module.
2017-10-27 09:08:15 -04:00
James Bardin c01dfa7757 failing test for changing intermediate modules
This test highlights how changing an intermediate source path prevents
reloading of submodules. While this is somewhat of an edge case now, it
becomes quite common in the cacse where module versions are updated.
2017-10-27 09:08:15 -04:00
James Bardin a5c86aeff6 Use the new regsrc and response packages
Adds basic detector for registry module source strings. While this isn't
a thorough validation, this will eliminate anything that is definitely
not a registry module, and split out our host and module id strings.

lookupModuleVersions interrogates the registry for the available
versions of a particular module and the tree of dependencies.
2017-10-27 09:08:15 -04:00
James Bardin 0bc279557e add more metadata to the Tree and the storage path
Submodules were located by using their module path as the storage key.
Now that modules may have versions, a submodule needs to know how to
locate the corect source depending on the versions of its ancestors in
the tree.

Add a version field to each Tree, and a pointer back to the parent Tree
to step back through the ancestors. The new versionedPathKey method uses
this information to build a unique key for each module, dependent on the
ancestor versions.

Not only do stored modules need to know their version if it exists, but
any relative source needs to know all the ancestor versions in order to
resolve correctly.
2017-10-27 09:08:15 -04:00
James Bardin 1f44fd8eb2 move getStorage to moduleStorage too
Add a test to ensure we don't store different modules with the same
relative path in the same location.
2017-10-27 09:08:15 -04:00
James Bardin 3d163917c7 move the getter.Storage into our moduleStorge type
Wrap the Storage interfaces together.
2017-10-27 09:08:15 -04:00
James Bardin 7416e7a6e0 update the module manifest implementation
The getter.Storage abstraction is proving entirely inadequate here, but
we can't replace it wholesale at the moment.

The Tree loader needs to know the location of the manifest before it can
start loading any modules. Since the version will have to be part of the
hashed storage key, there is no way to know what version of each module
are stored. The storageDir function will extract the StorageDir field
from the underlying FolderStorage instance for the tree to locate the
manifest.
2017-10-27 09:08:15 -04:00
James Bardin 283d88551a Add a new module storage manifest
To add registry support, a workaround in the local module storage was
added to record the subdirectory containing the module source from
within the archive file. Here we replace that temporary implementation
with the full manifest needed to record the necessary module metadata
for module loading.

In order to support versioned modules, the actual stored version needs
to be recorded. This can't be derived from the configuration, because
the configuration only contains the constraints, and at load time we need
to be able to enumerate the stored modules and all versions in order to
resolve them.

While the local storage key will be derived from the source and version,
that information is lost once it's hashed. While the entire storage
layer could be replaced to encode the needed data in the path itself,
this provides a minimal change to work with the existing storage code.
2017-10-27 09:08:15 -04:00
James Bardin 0962792257 surface registry errors to the user
Now that we can enforce local modules being relative or absolute paths,
we can be assured that any module source matching a registry pattern
must be found in the registry. This allows us to surface more useful
errors to the user, rather than simply stating that a source string
isn't valid.
2017-10-27 09:08:15 -04:00