Our service runs in multiple auto-scaling clusters to verify changes and ensure high availability as they are rolled out to production.
The process of taking a server out of the cluster during an auto-scale event means it can affect long-running connections to our servers.
Our clusters are built to wait for connections to finish their work before shutting down, like so:
- Remove server from DNS
- Wait for DNS propagation and DNS TTL expiry
- No longer accept new connections
- Wait for connections to close with a timeout
- Shutdown the server
This process ensures any FTP uploads and downloads are completed before any scaling event is finalised.