Troubleshooting MQTT broker connection errors in self-hosted workers

Last updated: September 9, 2025

If your self-hosted workers are unable to connect to the MQTT broker and you're seeing errors like "could not connect to the MQTT broker" or "network Error: EOF", this is typically related to configuration issues with the worker pool credentials.

Common Causes

The most common cause of MQTT connection failures is when the worker pool token contains an outdated IoT endpoint address. This can happen when:

  • You update the IoT DNS address in your server configuration after creating the worker pool

  • The SPACELIFT_TOKEN in your secrets manager contains configuration for the old endpoint

Resolution Steps

1. Verify Secret Configuration

First, check that your secret values in AWS Secrets Manager are correctly configured:

  • Ensure the secret contains the correct token

  • Verify the private key is base64 encoded as required

2. Check the Worker Pool Token

The SPACELIFT_TOKEN is a base64 encoded configuration that contains the IoT endpoint information. To debug connection issues:

  1. Retrieve the SPACELIFT_TOKEN from your secrets manager

  2. Base64 decode the token to inspect its contents

  3. Verify that the IoT endpoint address matches your current server configuration

3. Recreate the Worker Pool

If you've updated the IoT DNS address in your server after creating the worker pool, you'll need to:

  1. Delete the existing worker pool in Spacelift

  2. Create a new worker pool

  3. Update your secrets manager with the new worker pool credentials

  4. Redeploy your workers using the CloudFormation template

This ensures the worker pool token contains the correct IoT endpoint configuration for your updated server setup.

Additional Resources

For detailed information about creating secrets for worker pools, refer to the worker pool secrets documentation.