The Consul KV store limits the size of the values in the KV store to 524288
bytes. Once the state reaches this limit Consul will refuse to save it. It is
currently possible to try to bypass this limitation by enable Gzip but the issue
will manifest itself later. This is particularly inconvenient as it is possible
for the state to reach this limit without changing the Terraform configuration
as datasources or computed attributes can suddenly return more data than they
used to. Several users already had issues with this.
To fix the problem once and for all we now split the payload in chunks of 524288
bytes when they are to large and store them separatly in the KV store. A small
JSON payload that references all the chunks so we can retrieve them later and
concatenate them to reconstruct the payload.
While this has the caveat of requiring multiple calls to Consul that cannot be
done as a single transaction as those have the same size limit, we use unique
paths for the chunks and CAS when setting the last payload so possible issues
during calls to Put() should not result in unreadable states.
Closes https://github.com/hashicorp/terraform/issues/19182
When the path ends with / (e.g. `path = "tfstate/"), the lock
path used will contain two consecutive slashes (e.g. `tfstate//.lock`) which
Consul does not accept.
This change the lock path so it is sanitized to `tfstate/.lock`.
If the user has two different Terraform project, one with `path = "tfstate"` and
the other with `path = "tfstate/"`, the paths for the locks will be the same
which will be confusing as locking one project will lock both. I wish it were
possible to forbid ending slashes altogether but doing so would require all
users currently having an ending slash in the path to manually move their
Terraform state and would be a poor user experience.
Closes https://github.com/hashicorp/terraform/issues/15747
When locking was enabled with the Consul backend and the lock not properly
released, the `terraform force-unlock <lock_id>` command would do nothing as
its implementation would exit early in that case.
It now destroys the session that created the lock and clean both the lock and
the lock-info keys.
A regression test is added to TestConsul_destroyLock() to catch the issue if it
happends again.
Closes https://github.com/hashicorp/terraform/issues/22174
Most of the state package has been deprecated by the states package.
This PR replaces all the references to the old state package that
can be done simply - the low-hanging fruit.
* states: move state.Locker to statemgr
The state.Locker interface was a wrapper around a statemgr.Full, so
moving this was relatively straightforward.
* command: remove unnecessary use of state package for writing local terraform state files
* move state.LocalState into terraform package
state.LocalState is responsible for managing terraform.States, so it
made sense (to me) to move it into the terraform package.
* slight change of heart: move state.LocalState into clistate instead of
terraform
* Azure backend: support snapshots/versioning
Co-authored-by: Reda Ahdjoudj <reda.ahdjoudj@gmail.com>
Co-authored-by: Patrick F. Marques <patrickfmarques@gmail.com>
* Azure backend: Versioning -> Snapshot
Co-authored-by: Reda Ahdjoudj <reda.ahdjoudj@gmail.com>
Co-authored-by: Patrick F. Marques <patrickfmarques@gmail.com>
* update vendored azure sdk
* vendor giovanni storage sdk
* Add giovanni clients
* go mod vendor
* Swap to new storage sdk
* workable tests
* update .go-version to 1.14.2
* Tests working minus SAS
* Add SAS Token support
* Update vendor
* Passing tests
* Add date randomizer
* Captalize RG
* Remove random bits
* Update client var name
Co-authored-by: kt <kt@katbyte.me>
To allow using the same Tablestore table with multiple OSS buckets.
e.g. instead of env:/some/path/terraform.tfstate
the LockID now becomes some-bucket/env:/some/path/terraform.tfstate
* add TencentCloud COS backend for remote state
* add vendor of dependence
* fixed error not handle and remove default value for prefix argument
* get appid from TF_COS_APPID environment variables
Right now, the only environment variable available is the same
environment variable that will be picked up by the GCP provider. Users
would like to be able to store state in separate projects or accounts or
otherwise authenticate to the provider with a service account that
doesn't have access to the state. This seems like a reasonable enough
practice to me, and the solution seems straightforward--offer an
environment variable that doesn't mean anything to the provider to
configure the backend credentials. I've added GOOGLE_BACKEND_CREDENTIALS
to manage just the backend credentials, and documented it appropriately.
In order to make this work reasonably we can't avoid using some funny
heuristics, which are somewhat reasonable to apply within the context of
Terraform itself but would not be good to add to the general "logutils".
Specifically, this is adding the additional heuristic that lines starting
with spaces are continuation lines and so should inherit the log level
of the most recent non-continuation line.
Previously we were using the experimental HCL 2 repository, but now we'll
shift over to the v2 import path within the main HCL repository as part of
actually releasing HCL 2.0 as stable.
This is a mechanical search/replace to the new import paths. It also
switches to the v2.0.0 release of HCL, which includes some new code that
Terraform didn't previously have but should not change any behavior that
matters for Terraform's purposes.
For the moment the experimental HCL2 repository is still an indirect
dependency via terraform-config-inspect, so it remains in our go.sum and
vendor directories for the moment. Because terraform-config-inspect uses
a much smaller subset of the HCL2 functionality, this does still manage
to prune the vendor directory a little. A subsequent release of
terraform-config-inspect should allow us to completely remove that old
repository in a future commit.
* backend/remote-state/s3/backend_state.go: Prior to this commit, the terraform s3 backend did
not paginate calls to s3 when finding workspaces, which resulted in workspaces 'disappearing'
once they are switched away from, even though the state file still exists. This is due to the
ListBucket operation defaulting MaxItems to 1000, so terraform s3 backends that contained
more then 1000 workspaces did not function as expected. This rectifies this situation by
paginating calls to s3 when finding workspaces.
Signed-off-by: Collin J. Doering <collin@rekahsoft.ca>
faster
The acceptance tests for etcdv3, oss and manta were not validating
required env variablea, chosing to assume that if one was running
acceptance tests they had already configured the credentials.
It was not always clear if this was a bug in the tests or the provider,
so I opted to make the tests fail faster when required attributes were
unset (or "").
Support for cross-domain authentication has been added and mapping
environment variables to the correct domain settings has been
fixed.
In addition, support for clouds.yaml files has been added.
This mirrors the change made for providers, so that default values can
be inserted into the config by the backend implementation. This is only
the interface and method name changes, it does not yet add any default
values.
The handling of slashes was broken around listing workspaces in
workspace_key_prefix. While it worked in most places by splitting an
extra time around the spurious slashes, it failed in the case that the
prefix ended with a slash of its own.
A test was temporarily added to verify that the backend works with the
unusual keys, but rather than risking silent breakage around prefixes
with trailing slashes, we also add validation to prevent users from
entering keys with trailing slashes at all.
The API surface area is much smaller when we use the remote backend for remote state only.
So in order to try and prevent any backwards incompatibilities when TF runs inside of TFE, we’ve split up the discovery services into `state.v2` (which can be used for remote state only configurations, so when running in TFE) and `tfe.v2.1` (which can be used for all remote configurations).
James Bardin