Improving packing and unpacking performance for stacks with many modules

If your stacks contain a large number of modules, the packing and unpacking stages of runs can take a significant amount of time. The duration of these stages is primarily determined by the number of files in the workspace. The following recommendations can help reduce this time.

1. Use registry module sources instead of git sources

A git:: source pulls the entire repository, even when using a //subdir path. When many modules use git sources, the overhead adds up quickly. Switching to Spacelift's module registry or other registry module sources typically results in much smaller downloads.

2. Optimize git sources if you must keep them

If switching away from git sources is not currently feasible, you can reduce the amount of data fetched with these techniques:

• Add depth=1 to the source URL to perform a shallow clone. Note that this works with a branch or tag ref, but not with a commit SHA.

• Remove .git folders using an after_init hook to reduce the workspace size. It is recommended to test this on a non-critical stack first.

3. Configure sparse checkout on the stack

Using sparse checkout paths on your stack ensures that only the paths the stack actually needs are cloned and packed. Be sure to include any local module paths that fall outside the project root.