Commit Graph

216 Commits

Author SHA1 Message Date
Chris Marchesi 3f8dad30c9
helper/resource: Fail tests with no error that have ExpectError set
Looks like while we were checking errors correctly when ExpectError was
set, we weren't checking for the *absence* of an error, which is should
be checked as well (no error is still not the error we are looking for).

Added a few more tests for ExpectError as well.
2017-12-11 14:05:40 -08:00
Martin Atkins ba0514106a return tfdiags.Diagnostics from validation methods
Validation is the best time to return detailed diagnostics
to the user since we're much more likely to have source
location information, etc than we are in later operations.

This change doesn't actually add any detail to the messages
yet, but it changes the interface so that we can gradually
introduce more detailed diagnostics over time.

While here there are some minor adjustments to some of the
messages to improve their consistency with terminology we
use elsewhere.
2017-11-28 11:15:29 -08:00
Nándor István Krácser e5cc8af7f3 helper/resource: fix ungrammatical doc comment in StateChangeConf 2017-11-08 09:43:36 -08:00
Radek Simko d39025b9d8
helper: Escape test name in TF_LOG_PATH_MASK 2017-10-31 06:48:57 +00:00
Radek Simko 48fbd60e33
helper: Fall back to TF_LOG_PATH instead of os.Stderr 2017-10-30 15:58:13 +00:00
James Bardin 3a495ffe56 rename ModuleStorage to Storage
get rid of stutter and use module.Storage
2017-10-27 13:11:21 -04:00
James Bardin f2a7b94692 use the new ModuleStorage in the command package
Update the command package to use the new module storage. Move the old
command output strings into the module storage itself. This could be
moved back later either by using ui callbacks, or designing a module
storage interface once we know what the final requirements will look
like.
2017-10-27 12:58:24 -04:00
Radek Simko 4052a8f956
helper: Allow logs isolation per acceptance test 2017-10-14 16:39:11 +03:00
Chris Marchesi dc1f155728
helper/resource: Failure case for ImportStateIdFunc
Just to make sure returning an error was working.
2017-09-06 19:51:01 -07:00
Chris Marchesi 7bba1d0c95
helper/resource: Add ImportStateIdFunc
Add an ImportStateIdFunc field to the ImportState testing functionality.
This will allow for more powerful generation of complex import state IDs
that can't be accomplished by ImportStateId or ImportStateIdPrefix
themselves.
2017-09-06 19:44:43 -07:00
Radek Simko db97555e3e
helper/resource: Add TestStep.SkipFunc 2017-08-30 10:24:02 +02:00
James Bardin 60ea42cd1c Add testProviderConfig to import tests
Provider import tests previously didn't have to supply a config, but
terraform now requires the provider to be declared for discovery.

testProviderConfig returns a stub config with provider blocks based
on the TestCase Providers. This allows basic import tests in providers
to remain unchanged.
2017-07-07 16:16:35 -04:00
David Glasser 0a1f9156dc Fix resource.UniqueId to be properly ordered over multiple runs
The timestamp prefix added in #8249 was removed in #10152 to ensure that
returned IDs really are properly ordered.  However, this meant that IDs were no
longer ordered over multiple invocations of terraform, which was the main
motivation for adding the timestamp in the first place.  This commit does a
hybrid: timestamp-plus-incrementing-counter instead of just incrementing counter
or timestamp-plus-random.
2017-06-15 08:09:23 -07:00
Martin Atkins c835ef8ff3 Update tests for the new ProviderResolver interface
Rather than providing an already-resolved map of plugins to core, we now
provide a "provider resolver" which knows how to resolve a set of provider
dependencies, to be determined later, and produce that map.

This requires the context to be instantiated in a different way, so this
very noisy diff is a mostly-mechanical update of all of the existing
places where contexts get created for testing, using some adapted versions
of the pre-existing utilities for passing in mock providers.
2017-06-09 14:03:59 -07:00
Martin Atkins 0573ff6793 helper/resource: pass config when testing import
Previously having a config was mutually exclusive with running an import,
but we need to provide a config so that the provider is declared, or else
we can't actually complete the import in the future world where providers
are installed dynamically based on their declarations.
2017-06-09 14:03:59 -07:00
Clint 372a80bc42 provider/aws: Add Sweeper setup, Sweepers for DB Option Group, Key Pair (#14773)
* provider/aws: Add Sweeper setup, Sweepers for DB Option Group, Key Pair

* provider/google: Add sweeper for any leaked databases
* more recursion and added LC sweeper, to test out the Dependency path

* implement a dependency example

* implement sweep-run flag to filter runs

* stub a test for TestMain

* test for multiple -sweep-run list
2017-06-06 10:34:17 -05:00
James Bardin f5cda342f7 Merge pull request #13778 from hashicorp/jbardin/GH-13617
improve resource.WaitForState and add refreshGracePeriod
2017-04-19 18:23:00 -04:00
James Bardin 4c3a053f0c lint errors 2017-04-19 18:19:48 -04:00
James Bardin 14bea66f4e add test for proper cancelation 2017-04-19 18:07:00 -04:00
James Bardin eb4b45941c fix tests affected by refreshGracePeriod
A couple tests require lowering the grace period to keep the test from
taking the full 30s timeout.

The Retry_hang test also needed to be removed from the Parallel group,
becuase it modifies the global refreshGracePeriod variable.
2017-04-19 18:07:00 -04:00
James Bardin af1628eaa4 add refreshGracePeriod
Refresh calls may have side effects that need to be recorded if it
succeeds, especially common when when WaitForState is called from
resource.Retry.

If the WaitForState timeout is reached and there is a Refresh call
in-flight, wait up to refreshGracePeriod (set to 30s) for it to
complete.
2017-04-19 18:07:00 -04:00
= 1608f5544f Added error check 2017-04-19 14:22:35 -06:00
= 6bc3610c09 removing extraneous code 2017-04-19 09:18:30 -06:00
James Bardin 6601b9b8dd adjust the inconsistent_negative test to match
This test unfortunately relies on the timing of the loops in
WaitForState, and the text of the error message. Adjust the timing so
the timeout isn't an even multiple of the poll interval, and make sure
we reach a minimum number of retries.
2017-04-19 10:53:06 -04:00
James Bardin af5e22cf94 don't leave WaitForState goroutine running
Make sure that we can cancel the WaitForState refresh loop when reaching
a timeout, otherwise it may run indefinitely. There's no need to try and
store and read the Result concurrently, just pass the value over a
channel.
2017-04-19 10:41:51 -04:00
Jake Champlin f24087ee54
core: Add ImportStateIdPrefix field for testing
Adds the `ImportStateIdPrefix` field for import acceptance tests. There are (albeit fairly rare) import cases where a resource needs to be imported with a combination of the resource's ID and a known string prefix. This allows the developer to specify the known prefix, and omit the `ImportStateId` field.

```
$ make test TEST=./helper/resource TESTARGS="-run=TestTest_importStateIdPrefix"
==> Checking that code complies with gofmt requirements...
==> Checking AWS provider for unchecked errors...
==> NOTE: at this time we only look for uncheck errors in the AWS package
go generate $(go list ./... | grep -v /terraform/vendor/)
2017/03/30 18:08:36 Generated command/internal_plugin_list.go
go test -i ./helper/resource || exit 1
echo ./helper/resource | \
                xargs -t -n4 go test -run=TestTest_importStateIdPrefix -timeout=60s -parallel=4
go test -run=TestTest_importStateIdPrefix -timeout=60s -parallel=4 ./helper/resource
ok      github.com/hashicorp/terraform/helper/resource  0.025s
```
2017-03-30 18:11:10 -04:00
James Nugent 0e3a7e6d0d helper/resource: Allow unknown pending states (#13099)
Sometimes when waiting on a target state, the set of valid states
through which a value will transition is unknown. This commit adds
support for an empty Pending slice and will treat any states that are not
the target as valid provided the timeouts are not exceeded.
2017-03-28 22:38:42 +03:00
clint shryock 61355c33c5 testing: Add option to run only a plan on a TestStep configuration 2017-03-22 15:38:21 -05:00
James Bardin 5238f51dc7 move TestReset mock from terraform to helper
the terraform package doesn't know about TestProvider, so don't put the
hooks in terraform.MockResourceProvider. Wrap the mock in the test where
we need to check the TestProvider functionality.
2017-03-08 17:48:11 -05:00
James Bardin 6fcb55d09e reword TestProvider doc 2017-03-08 17:41:35 -05:00
James Bardin 4b2e96b2e2 test for TestReset and fixed resource factories 2017-03-07 11:39:12 -05:00
James Bardin 1eb9a2d083 Reset ResourceProviders
Call all ResourceProviderFactories and reset the providers before tests.
Store the provider and/or the error in a fixed factory function to be
returned again later.
2017-03-07 11:39:12 -05:00
Kit Ewbank 64fda44b00 Add 'aws_vpn_gateway' data source. (#11886) 2017-02-13 16:24:55 +00:00
Kit Ewbank 87d98b1ff1 Correct aws_s3_bucket_object data source acceptance tests. (#11346) 2017-01-23 21:53:02 +00:00
Radek Simko 6222b3e91d
helper/resource: Add retries to err message 2017-01-20 11:09:04 -08:00
Radek Simko be4e5e53a9
Add more details (Retries) to resource.NotFoundError 2017-01-20 07:08:30 -08:00
Sean Chittenden 7862e5b499
Differentiate between lookup failure and value inequality.
Before this patch it was not possible to test for a key in a map where
the value is an empty string.  With this patch, however, it is now
possible to write a check like:

```
resource.TestCheckResourceAttr("res.name", "mymap.KeyWithEmptyValue", ""),
```

To test that `KeyWithEmptyValue` is a valid key in `mymap`.
2017-01-11 23:28:48 -08:00
James Bardin e28e11d44c Fix resource.UniqueId to be properly ordered
UniqueId attempted to provide an ordered unique id by using a nanosecond
timestamp, but doesn't take into account that time is not monotonic
increasing. This provides an implementation that will always be
increasing.
2016-11-15 21:58:23 -05:00
Radek Simko 86acdccaf2 helper/resource: Add timeout to TimeoutError msg (#8773) 2016-09-15 10:53:25 +01:00
James Nugent 6c23181686 Merge pull request #8383 from kjmkznr/import-aws-s3-b-notification
provider/aws: Support import `aws_s3_bucket_notification`
2016-09-03 15:50:25 -07:00
James Bardin 481f12cf2c Remove `first` variable an extra if block
clean up the code slighly by moving the Sleep in WaitForState to the end
of the loop.
2016-08-30 17:56:26 -04:00
James Bardin 82be35a797 Fix races in WaitForState
The WaitForState method can't read the result values in a timeout
because they are still owned by the running goroutine. Keep all values
scoped inside the goroutine, and save them into an atomic.Value to be
returned.

Fixes race introduced in #8510
2016-08-30 16:22:21 -04:00
Radek Simko 506c118383
helper/resource: Show last state in timeout err message 2016-08-27 21:25:14 +01:00
Radek Simko aef2513b44
helper/resource: Fix WaitForState tests 2016-08-27 15:11:09 +01:00
James Nugent df06d5623d Merge pull request #8249 from meteor/glasser/name-prefix-timestamp
core: name_prefix names now start with a timestamp
2016-08-23 14:37:56 +02:00
Kazunori Kojima ed05161fd0
provider/aws: Support import `aws_s3_bucket_notification` 2016-08-23 08:19:06 +09:00
James Nugent 6ae7140251 Merge pull request #8370 from hashicorp/dtolnoy-wait
Do not sleep between first attempt or between successful attempts
2016-08-22 14:30:54 +02:00
James Nugent 0a5acaa0d8 Change error message return structure 2016-08-22 10:34:32 +02:00
David Tolnay 9d3f40a513 Fix bad message from TimeoutError
Before:

    Error creating IAM Role my-role: timeout while waiting for state
    to become 'success'. last error: %!s(<nil>)
2016-08-22 10:23:53 +02:00
David Tolnay e27a735da2 Do not sleep between first attempt or between successful attempts 2016-08-22 10:17:23 +02:00
David Glasser 4ad825fe08 core: name_prefix names now start with a timestamp
This means that two resources created by the same rule will get names
 which sort in the order they are created.

The rest of the ID is still random base32 characters; we no longer set
the bit values that denote UUIDv4.

The length of the ID returned by PrefixedUniqueId is not changed by this
commit; that way we don't break any resources where the underlying
resource has a name length limit.

Fixes #8143.
2016-08-17 11:06:28 -07:00
James Nugent b551981cc7 testing: Add ComposeAggregateTestFunc
This commit adds a function which composes a series of TestFuncs, but
will run all tests before returning an error, unlike ComposeTestFunc.
This is useful when verifying contents of state in acceptance tests and
it is desirable to see all the failing cases in one run for slow
resources.
2016-08-16 19:56:18 +01:00
James Nugent b681ab23d3 testing: Add TestCheckResourceAttrSet helper
This commit adds a TestCheckFunc which ensures that a value is set for a
given name/key combination. It is primarily useful for ensuring that
computed values are set where it is not possible to know the expected
value ahead of time.
2016-08-15 15:42:25 -05:00
James Bardin 074be9ae56 Another race in resource.Retry 2016-07-29 18:35:54 -04:00
Paul Hinze 261043fd1a Merge pull request #7523 from hashicorp/f-aws-beanstalk-env-poll-timing
provider/aws: Beanstalk environments, bump the minimum timeout between API calls
2016-07-21 16:43:21 -05:00
Paul Hinze 614806d59f
helper/resource: Fix import test harness, which was modifying state
Maps are reference types, it turns out :D
2016-07-15 13:15:47 -06:00
clint shryock de60481428 provider/aws: Rework Beanstalk optional polling
expose a poll_interval for users to configure polling for updates
2016-07-13 15:38:23 -06:00
Paul Hinze 3b732131d2 Merge pull request #7446 from hashicorp/b-jit-resource-validate
core: rerun resource validation before plan and apply
2016-07-01 15:00:40 -05:00
Paul Hinze ad27190852 Merge pull request #7448 from dtolnay/timeout
Timing out is not success
2016-07-01 14:28:11 -05:00
Paul Hinze 4a1b36ac0d
core: rerun resource validation before plan and apply
In #7170 we found two scenarios where the type checking done during the
`context.Validate()` graph walk was circumvented, and the subsequent
assumption of type safety in the provider's `Diff()` implementation
caused panics.

Both scenarios have to do with interpolations that reference Computed
values. The sentinel we use to indicate that a value is Computed does
not carry any type information with it yet.

That means that an incorrect reference to a list or a map in a string
attribute can "sneak through" validation only to crop up...

 1. ...during Plan for Data Source References
 2. ...during Apply for Resource references

In order to address this, we:

 * add high-level tests for each of these two scenarios in `provider/test`
 * add context-level tests for the same two scenarios in `terraform`
   (these tests proved _really_ tricky to write!)
 * place an `EvalValidateResource` just before `EvalDiff` and `EvalApply` to
   catch these errors
 * add some plumbing to `Plan()` and `Apply()` to return validation
   errors, which were previously only generated during `Validate()`
 * wrap unit-tests around `EvalValidateResource`
 * add an `IgnoreWarnings` option to `EvalValidateResource` to prevent
   active warnings from halting execution on the second-pass validation

Eventually, we might be able to attach type information to Computed
values, which would allow for these errors to be caught earlier. For
now, this solution keeps us safe from panics and raises the proper
errors to the user.

Fixes #7170
2016-07-01 13:12:57 -05:00
Paul Hinze 5656d7388c
helper/resource: Consolidate unit test override
I noticed we had two mechanisms for unit test override. One that dropped
a sentinel into the env var, and another with a struct member on
TestCase. This consolidates the two, using the cleaner struct member
internal mechanism and the nicer `resource.UnitTest()` entry point.
2016-07-01 13:08:43 -05:00
David Tolnay 7e1cd34819 Timing out is not success 2016-06-30 18:07:56 -07:00
Patrick Sodré 80936e3562 provider/triton: add machine domain names (#7149)
* vendor joyent/gosdc/cloudapi

* provider/triton: Add machine domain names

 - Includes acceptance test.
2016-06-26 22:18:17 +01:00
James Nugent 9c7cf639b3 testing: Allow acceptance test to run as unit test
This commit adds a flag to acceptance tests in order to make
appropriately named tests work during `make test` irrespective of the
TF_ACC environment variable. This should only be used on tests which are
known to be fast.
2016-06-11 12:55:14 +01:00
Radek Simko 1ea5cff9b3 Merge pull request #6946 from TimeIncOSS/f-aws-s3-object-data-source
provider/aws: Add aws_s3_bucket_object data source
2016-06-02 07:04:46 +01:00
Radek Simko bf8931b1ab provider/aws: Mark Lambda function as gone when it's gone (#6924)
* helper/error: Introduce TimeoutError & UnexpectedStateError

* provider/aws: Mark Lambda function as gone when it's gone
2016-06-02 00:50:43 +01:00
Radek Simko 84ab00d92e helper/resource: Add TestStep.PreventPostDestroyRefresh
- This is to allow easier testing of data sources which read data from resources created in the same scope
2016-06-01 19:13:00 +01:00
Martin Atkins 031b561ef0 helper/resource: ignore data resource diffs during destroy
When testing destroy, the test harness calls Refresh followed by Plan,
with the expectation that the resulting diff will be empty.

Data resources challenge this expectation, because they will always be
instantiated during refresh if their configuration isn't computed, and so
the subsequent diff will want to destroy what was instantiated.

To work around this, we make an exception that data resource destroy
diffs may appear in the plan but nothing else.

This fixes #6713.
2016-05-21 14:16:02 -07:00
Mitchell Hashimoto 55583baa7e
Merge branch 'f-aws-import' 2016-05-18 15:28:12 -06:00
James Nugent 3ea3c657b5 core: Use OutputState in JSON instead of map
This commit forward ports the changes made for 0.6.17, in order to store
the type and sensitive flag against outputs.

It also refactors the logic of the import for V0 to V1 state, and
fixes up the call sites of the new format for outputs in V2 state.

Finally we fix up tests which did not previously set a state version
where one is required.
2016-05-18 13:25:20 -05:00
Mitchell Hashimoto 884980da1a
providers/aws: instance, nat, internet gateway 2016-05-16 10:03:57 -07:00
Mitchell Hashimoto 6a675b4a15
helper/resource: ImportState test can verify states 2016-05-11 13:02:37 -07:00
Mitchell Hashimoto ec02c8c0e2
helper/resource: testing of almost all aspects of ImportState tests 2016-05-11 13:02:37 -07:00
Mitchell Hashimoto 2d99c451fb
helper/resource: basic ImportState acceptance testing
Still some TODOs, and more test cases to write, but the basics are all
here.
2016-05-11 13:02:37 -07:00
Mitchell Hashimoto 9bd1c9e7ca
helper/resource: reshuffling to prepare for importstate testing 2016-05-11 13:02:37 -07:00
Mitchell Hashimoto 35c87836b4 core: Add terraform_version to state
This adds a field terraform_version to the state that represents the
Terraform version that wrote that state. If Terraform encounters a state
written by a future version, it will error. You must use at least the
version that wrote that state.

Internally we have fields to override this behavior (StateFutureAllowed),
but I chose not to expose them as CLI flags, since the user can just
modify the state directly. This is tricky, but should be tricky to
represent the horrible disaster that can happen by enabling it.

We didn't have to bump the state format version since the absense of the
field means it was written by version "0.0.0" which will always be
older. In effect though this change will always apply to version 2 of
the state since it appears in 0.7 which bumped the version for other
purposes.
2016-05-10 14:40:11 -04:00
Mitchell Hashimoto d85df63526
providers/aws: aws_instance id-only 2016-04-22 09:37:41 -07:00
Mitchell Hashimoto e0da21d381
helper/resource: make id-only check opt-in
As I've been working through the resources, I'm finding that a lot are
going to need some serious work. Given we have hundreds, I think it
might be prudent to make this opt-in for now and we can revisit
automatic/opt-out at some future point.

Importability will likely be opt-in it appears so this will match up
with that.
2016-04-21 08:37:08 -07:00
Mitchell Hashimoto ff7b58f032
providers/aws: peering connection id-only test settings 2016-04-20 12:19:21 -07:00
Mitchell Hashimoto 0ef1b3b84a
providers/aws: response value for DescribeVpcAttribute needs to be
.Value
2016-04-20 11:35:43 -07:00
Mitchell Hashimoto 1db1bf6639
helper/resource: use the full config for id-only checks
Originally I used an empty config module. This caused problems since
important provider configurations weren't available. Instead, I now set
it to use the full config. This isn't an issue since the attributes
themselves aren't available to Refresh anyways.
2016-04-20 11:18:13 -07:00
Mitchell Hashimoto 0c8b0bff2c
helper/resource: can specify specific name to id refresh test 2016-04-20 11:12:30 -07:00
Mitchell Hashimoto 1a9fae6b2e
helper/resource: can disable ID refresh check 2016-04-20 11:09:54 -07:00
Mitchell Hashimoto 35f4201b9e
providers/aws: instance_tenancy is computed, set 2016-04-20 10:48:22 -07:00
Mitchell Hashimoto f2c4f8e9ba
helper/resource: fix tests 2016-04-20 10:08:34 -07:00
Mitchell Hashimoto 5c4f78796b
helper/resource: don't need to id-only check if no test steps 2016-04-20 09:52:53 -07:00
Mitchell Hashimoto a285c04dc9
helper/resource: only verify id-only run if no error 2016-04-20 09:50:59 -07:00
Mitchell Hashimoto baac14aaeb
helper/resource: guard id-only by acc var 2016-04-20 09:34:54 -07:00
Mitchell Hashimoto cb32cb8947
helper/resource: error if id-only check didn't run 2016-04-20 09:25:23 -07:00
Mitchell Hashimoto 86e0c853db
helper/resource: test for failing id-only refresh check 2016-04-20 09:18:25 -07:00
Mitchell Hashimoto 060b43fbd9
helper/resource: remove debug 2016-04-20 09:17:04 -07:00
Mitchell Hashimoto 4f6edf4fe4
helper/resource: id-only refresh testing 2016-04-20 09:16:48 -07:00
James Nugent cb8a0549e1 Merge pull request #5757 from hashicorp/f-output-testing
Add TestCheckOutput helper to resource testing
2016-03-21 17:19:08 +00:00
James Nugent 35f9d2e081 Add TestCheckOutput helper to resource testing
This allows outputs in test configuration to have test functions written
conveniently. Useful for azurerm_template_deployment.
2016-03-21 16:54:02 +00:00
Radek Simko bf9f5879ca helper/resource: Remove NotFoundError function 2016-03-21 16:47:50 +00:00
James Nugent f946695187 Merge pull request #5444 from TimeIncOSS/f-aws-logs-metric-filter
provider/aws: Add support for CloudWatch Log Metric Filter
2016-03-21 16:43:38 +00:00
Paul Hinze c3e27b3e0a provider/test: a test provider
Here we also introduce a `test` provider meant as an aid to exposing
via automated tests issues involving interactions between
`helper/schema` and Terraform core.

This has been helpful so far in diagnosing `ignore_changes` problems,
and I imagine it will be helpful in other contexts as well.

We'll have to be careful to prevent the `test` provider from becoming a
dumping ground for poorly specified tests that have a clear home
elsewhere. But for bug exposure I think it's useful to have.
2016-03-21 08:59:54 -05:00
Radek Simko fd2fcaf1d5 helper/resource: Implement resource-wide NotFoundError 2016-03-15 13:58:25 +00:00
Radek Simko 034287fdc2 helper/resource: Error shouldn't be returned in case of success 2016-03-10 14:14:14 +00:00
Radek Simko 8582f4df9f helper/resource: Fix TestRetry 2016-03-10 14:13:23 +00:00