Context

When Spacelift jobs fail or terminate with errors, the associated pods may remain in the cluster for extended periods, particularly those in Init:Error or Terminated states. This can lead to resource waste and cluster clutter, especially in managed Kubernetes environments like EKS where default garbage collection settings cannot be modified.

Answer

While Kubernetes' native garbage collection handles pods that reach a "failed" status, pods that are stuck in other states (like Init:Error or Terminated) require alternative cleanup approaches. Here are the current recommended solutions:

Option 1: Implement a Cleanup Cronjob

1. Create a Kubernetes cronjob that identifies and removes stale pods based on:

   • Pod status (Init:Error, Terminated)

   • Age of the pods

2. Schedule the cronjob to run at regular intervals to maintain cluster cleanliness

Option 2: Manual Cleanup

Until an automated solution is implemented, you can manually delete stale pods using kubectl commands.