Context

When running Terraform/OpenTofu operations in Spacelift, you may experience slower execution times due to various factors such as module downloads, provider installations, and workspace preparation. This article covers different methods to optimize and speed up your Terraform/OpenTofu runs.

Answer

There are several ways to improve the performance of your runs in Spacelift:

1. Provider Caching

You can cache providers to avoid downloading them on every run:

For instance-based workers:

1. Create a cache directory in your startup script:

   mkdir -p /var/cache/terraform/
   chown 1983:1983 /var/cache/terraform/

2. Configure the mount points (note: do not include trailing slashes):

   export SPACELIFT_WORKER_WO_EXTRA_MOUNTS="/var/cache/terraform:/var/cache/terraform"
   export SPACELIFT_WORKER_RO_EXTRA_MOUNTS="/var/cache/terraform:/var/cache/terraform"

3. Set the TF_PLUGIN_CACHE_DIR environment variable to point to this directory

2. Increase the parallelism

Increase the -parallelism flag by adding the following environment variable in your stack environment settings TF_CLI_ARGS_plan="-parallelism=X"

To apply the same setting to applies, also set TF_CLI_ARGS_apply="-parallelism=X".

Note: If your stack manages Spacelift resources with the Spacelift provider, raising parallelism alone may not help. See section 7.

3. Skip the Refresh

If it’s acceptable for plans to miss changes made outside the stack, you can skip the state refresh during planning: TF_CLI_ARGS_plan="-refresh=false"

If you also set parallelism, combine both flags in the same variable: TF_CLI_ARGS_plan="-parallelism=X -refresh=false".

4. Custom Runner Images

Create a custom Docker image that includes:

• Pre-installed Terraform/OpenTofu

• Commonly used providers

• Pre-downloaded module archives

5. For Kubernetes Workers

If using Kubernetes workers, you can set up a binaries cache volume to cache Terraform modules and providers. Refer to the binaries cache volume documentation.

6. Git Checkout Optimization

Enable Git checkout support to potentially speed up the workspace preparation phase, especially for monorepos. See the Git checkout documentation for setup instructions.

7. Stacks Managing Spacelift Resources (Spacelift Provider)

Administrative stacks that manage many Spacelift resources with the spacelift-io/spacelift provider can spend a long time in the refresh during planning. Examples of such resources are stacks, environment variables, policy attachments, mounted files and stack dependencies.

Spacelift injects these variables into runs:

SPACELIFT_MAX_REQUESTS_PER_SECOND=20
SPACELIFT_MAX_REQUESTS_BURST=10

The Spacelift provider uses them as a client-side rate limiter. As a result, refresh speed stays capped at roughly 20 resources per second, no matter how high you set -parallelism.

Options:

1. Reduce the number of managed resources. Use auto-attach labels for policies and auto-attached contexts for shared environment variables, instead of one attachment or variable resource per stack.

2. Raise the client-side limit for that stack. Export higher values in the stack’s before_plan and before_apply hooks, for example:

   export SPACELIFT_MAX_REQUESTS_PER_SECOND=100
   export SPACELIFT_MAX_REQUESTS_BURST=50

These variables are not a documented, supported setting, so treat this as an advanced workaround and apply it only to the stacks that need it. If you start seeing rate-limit errors from the Spacelift API, lower the values or contact support with a run link.

8. Monitoring Performance

To identify bottlenecks in your runs, enable timestamps in your logs by adding the feature flag:

feature:enable_log_timestamps

You can view detailed timing information by disabling the simplified view in the run logs: