Resolving KMS Decrypt errors during self-hosted reinstallation

Last updated: August 6, 2025

Resolving KMS Decrypt errors during self-hosted reinstallation

If you encounter KMS decrypt errors during a self-hosted Spacelift reinstallation, this is typically caused by encrypted data from a previous installation that references an old KMS key.

Understanding the Issue

When you see an error like:

AccessDeniedException: User: arn:aws:sts::account:assumed-role/spacelift-server-region/id is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:region:account:key/old-key-id

This occurs because Spacelift is trying to decrypt data that was originally encrypted with an old KMS key from a previous installation. Even though your environment variables point to the new key, the encrypted data (ciphertext) in your database still contains references to the old key.

Why Standard Uninstall Doesn't Resolve This

By default, the uninstall.sh script retains your S3 buckets, RDS database, and KMS keys. This means any encrypted data in your database remains intact and still references the old key's metadata.

When using your own database (rather than letting Spacelift manage it), the uninstall script doesn't drop your database tables, so encrypted secrets remain even after uninstallation.

Resolution Steps

Option 1: Complete Clean Uninstall

Use the uninstall.sh script with the -n flag for complete uninstallation, which should remove all resources including the database.

Option 2: Manual Database Cleanup (For Custom Databases)

If you're providing your own database and continue to see errors after reinstallation:

  1. Identify the problematic encrypted data in your database. The most common location is the oidc_settings table, which contains encrypted OIDC secrets.

  2. Manually delete the encrypted entries that reference the old KMS key:

    DELETE FROM oidc_settings WHERE [condition];

  3. If you have OIDC/SSO configured, you may also need to delete the corresponding role from the accounts table to force recreation of the OIDC settings.

  4. Restart your Spacelift services to allow them to recreate the necessary encrypted entries with the new KMS key.

Prevention

To avoid this issue in future reinstallations:

  • Ensure complete cleanup of encrypted data when uninstalling

  • If using a custom database, manually verify that all Spacelift-related tables are cleared before reinstalling

  • Consider using Spacelift-managed databases for easier cleanup during reinstallations

For more information about encryption and uninstallation, refer to the encryption documentation and uninstallation guide.