How to Move a Large Site When the Transfer Keeps Timing Out

A large site transfer that keeps timing out is almost never one problem. It is a single long lived connection being asked to survive every transient failure between two machines for hours, and something in that path gives up first. The reliable fix is to stop treating the move as one connection and break it into resumable pieces: a bulk copy that can restart, a database dump that is streamed and restarted, and a final catch up pass that is small enough to finish.

Timing out during a migrate large website transfer timeout is a symptom, not a cause. Find the cause before changing tools, because each cause has a different fix.

Find out what is actually dying

There are four usual suspects. The remote shell drops because of idle or unresponsive session limits on the server or a firewall in between. The TCP connection stalls because a middlebox has a state table with an idle timer. The transfer tool exits because a single file is enormous and the tool holds too much state. Or the receiving disk fills or the process gets killed by an out of memory killer, and the "timeout" is really a crash on the far end.

Start by watching the connection itself. Open a second terminal and run a keepalive probe while the transfer is in progress:

while true; do date +%T; ping -c 1 -W 2 target.example.net | tail -1; sleep 5; done

14:02:11  64 bytes from target.example.net: icmp_seq=1 ttl=54 time=18.2 ms
14:02:16  64 bytes from target.example.net: icmp_seq=1 ttl=54 time=19.0 ms
14:02:21  Request timeout for icmp_seq 0

If the pings stop at the same moment the copy stalls, the path is dropping idle or long lived flows and you need keepalives on every layer. If pings continue but the copy stalls, the problem is on one of the endpoints.

For the shell layer, put keepalives in ~/.ssh/config on the machine you run commands from. ServerAliveInterval sends a probe over the encrypted channel, and ServerAliveCountMax controls how many misses are tolerated before the client exits. TCPKeepAlive works at the socket level and is weaker because the probes are not encrypted, but it is still worth having.

Host target.example.net
    ServerAliveInterval 15
    ServerAliveCountMax 8
    TCPKeepAlive yes
    Compression no

Turn compression off for large already compressed files. It costs CPU and buys nothing on images, video or archives, and it makes the remote side work harder while it is also writing to disk. If the far end is a server you control, also raise ClientAliveInterval in sshd_config so the server does not close quiet sessions.

Make the bulk copy resumable

This is the core of the method. A copy that can restart where it stopped turns a fatal timeout into a minor delay. rsync has a partial transfer mode that keeps the incomplete file and picks it up on the next run. The --partial-dir option stores the fragment in a hidden directory so the next run resumes from it rather than starting the file again.

rsync -aHAX --numeric-ids --partial-dir=.rsync-partial \
  --timeout=300 --bwlimit=20000 \
  -e "ssh -o ServerAliveInterval=15 -o ServerAliveCountMax=8" \
  /var/www/site/ [email protected]:/var/www/site/

The --timeout value is an I/O timeout, not a total runtime limit: if no data moves for that many seconds, that run exits and you run it again. Wrap the whole thing in a loop that retries until it succeeds, and log each attempt so you can see progress shrinking. -aHAX preserves permissions, hard links, ACLs and extended attributes, which matters if the site has uploaded files with odd ownership. --numeric-ids stops user and group names being mapped through the receiving system's passwd file, which is usually wrong across two servers.

If a single file is too large to finish in one window, split it before transfer and reassemble on the far side. Use split with a byte size, copy the parts, and cat them back together. Verify with a checksum on both ends before you delete the original. For very large archives, zstd or gzip piped over the connection is often slower than a plain copy because it cannot resume; prefer a resumable copy of the uncompressed data.

Move the database without one long connection

A database dump streamed into a single connection is the most fragile part of any migration. Dump to a file first, then copy the file with the resumable method above. If the database is large, dump in chunks by primary key range so each file is a manageable size and a failed chunk costs you minutes, not hours.

mysqldump --single-transaction --quick --skip-lock-tables \
  --where="id >= 0 AND id < 1000000" site_db big_table \
  > big_table_0001.sql

Repeat with the next range, then load the chunks in order on the target. The --single-transaction flag gives you a consistent snapshot on transactional storage engines without locking writers out. For a whole database, --single-transaction on the full dump is fine as long as the dump file itself is copied resumably, which it now is. If the dump process is the thing being killed, check memory on the source and add --quick so rows are not buffered in memory.

Before the final cutover, do a catch up pass. Run the bulk copy again, which now moves only changed files, then take a fresh dump of the tables that changed and load it. Keep the old site read only during that window so nothing new arrives.

Check the cheap things before blaming the network

Confirm the receiving filesystem has room and the right inode count, because a full disk presents as a stalled transfer. Check that no firewall or load balancer in the path has a short idle timer, and if it does, either shorten the keepalive interval or route the transfer around it. Confirm the receiving process is not being killed by the out of memory killer by watching dmesg on the target during a run. If you control both ends, a direct connection that skips the public path is faster and far more stable than anything tunnelled through a proxy.

What to do next

Run a dry pass first with rsync -n to see the file list and total size, then run the real copy inside a retry loop and watch the logs. Test the database chunking on one table before you commit to it for the whole schema. Once the bulk data is across and verified with checksums, the final catch up is small enough that a timeout no longer matters, and you can schedule the cutover with confidence.