Resolving Ghost Workers in Kubernetes Worker Pools

Last updated: September 9, 2025

If Spacelift shows more workers as busy than the number of pods actually running in your Kubernetes cluster, you likely have "ghost workers" - workers that crashed or terminated without properly deregistering from Spacelift.

Identifying Ghost Workers

To diagnose this issue:

  1. Check your WorkerPool status: kubectl get workerpools

  2. Compare the desired vs. actual worker count

  3. List all workers: kubectl get workers

  4. If Spacelift shows workers as busy but no corresponding pod exists in Kubernetes, these are ghost workers

Resolution Steps

Try these solutions in order:

1. Scale Workers to Zero and Back

This is the most effective method:

  1. Scale your worker pool to 0: kubectl scale workerpool <workerpool-name> --replicas=0

  2. Wait for all workers to disappear from the Spacelift UI

  3. Scale back to your desired number: kubectl scale workerpool <workerpool-name> --replicas=<desired-count>

For more details on scaling, see the scaling documentation.

2. Restart the Controller

If scaling doesn't resolve the issue:

kubectl rollout restart deployment <controller-deployment-name>

This refreshes worker registration with Spacelift.

3. Cycle Workers

As a last resort, you can cycle the workers through the Spacelift UI. Note that the cycle option may be disabled if workers are showing as busy due to the ghost worker issue.

See the worker cycling documentation for more information.

Prevention

Ghost workers typically occur when:

  • Workers crash or are terminated abruptly without proper deregistration

  • Network connectivity issues prevent workers from connecting to Spacelift

  • Authentication problems prevent proper worker registration

If you continue experiencing this issue, check your worker logs and network connectivity to Spacelift.