[jochen-testingbot]

Stacked on #71 (it builds on its reconnect timings); GitHub retargets it to main when #71 merges.

Why

Every browser connection through the tunnel shares one SSH connection, i.e. one TCP flow. On a lossy link that caps the whole tunnel at one flow's loss-limited rate, and one lost packet stalls every request until it is resent. Measured in the tunnel-harness perf lab (50 ms, 200 Mbit/s, 0.5% loss, client on CUBIC):

	1 download	4 parallel
through the tunnel today (one SSH connection)	0.55 MB/s	0.63 MB/s total
plain TCP, same link	0.70 MB/s	2.36 MB/s

A small request during a download took 1.2–1.7 s instead of 91 ms.

What

--ssh-connections N (1–8, default 4). The first connection works as before and keeps the hub forward (-L, a local port only one connection can hold), readiness and the self-test. The others only carry browser traffic and reconnect on their own; a dropped one never stops the local proxy or flips readiness while another connection still forwards.

Each connection's reverse forward goes to its own port in 2020–2035 instead of 2010. On tunnel servers that have it, tunnel-mux (testingbot/ansible, tunnel02) sees those forwards, checks each with the probe request ForwardProbeHandler already answers, and only then serves 2010 itself and spreads connections over the healthy forwards. Squid's parent, no-cache tunnels and the device lab keep using 2010 unchanged.

Compatibility

• Servers without tunnel-mux (the current production image, older tunnel02 builds): after the local proxy starts, the client probes 2010 for up to 3 s (the probe from Keep port 2010 working through reconnects: notice dead sessions, retry the forward, probe it #69). When 2010 is not served it drops the 2020 forward and binds 2010 itself, as every earlier client did, on one connection. This costs those tunnels up to 3 s once at startup; reconnects on the same server do not try again.
• Older clients on new servers: they bind 2010 themselves and never 2020–2035, so tunnel-mux never wakes and nothing changes for them.
• --ssh-connections 1 never touches 2020–2035. --noproxy keeps one connection, because nothing on this side answers the probe.

Tests

• SSHTunnelMultiConnectionTest: against an in-JVM SSH server, with a stand-in for tunnel-mux. Covers:

  • all connections forward on distinct ports and the hub forward stays on the first;
  • fallback to one connection on 2010 without tunnel-mux;
  • --ssh-connections 1 never binding 2020–2035;
  • traffic continuing while the first connection reconnects;
  • an extra connection reconnecting on its own.

  Removing the extra-connection start, or the fallback, fails 4 of the tests. theForwardsStayPutAcrossForwardingChecks covers a bug the lab found: each extra connection's monitor looked for the hub forward, called it lost every 5 s and re-registered its reverse forward, often on a new port. The test fails without the fix (2 re-registrations per connection).

• SshConnectionsOptionTest: covers the option's range.

• The existing embedded-server tests pin --ssh-connections 1, which is what they assert. Without the pin they passed too, but each waited out the fallback (17 s → 58 s).

• Full suite: 1051 tests, 0 failures.

• End to end in tunnel-harness (testingbot/ansible, suite-mux.sh) against real tunnel images:

  • tunnel02 with tunnel-mux (image tunnel02-20261025g): 56/56.
    • Released 5.0 and --ssh-connections 1 leave tunnel-mux idle, with sshd holding 2010 on one session.
    • The default gives 4 sessions and 4 healthy forwards, and the whole device suite passes through tunnel-mux: HTTP methods, 5 MB uploads, 50 MB downloads, caching, CONNECT, SSL bump, direct 2010 and websockets on every path.
    • With one connection blackholed, 1 of 30 requests failed in the ~8 s before tunnel-mux dropped it; the client replaced it on a new port.
    • After a NAT change that blackholed every connection, 2010 was back in about 10 s.
  • Production image (22.04, no tunnel-mux): 14/14. The client falls back to one session on 2010. Startup is 18 s against 15 s, the 3 s wait. The default suite gives the same results as 5.0 (two websocket checks fail on that image's squid, as they do with 5.0), and 2010 was back 8 s after a NAT change.

• Perf lab (50 ms, 200 Mbit/s, 0.5% loss, client on CUBIC), --ssh-connections 1 against 4:

  	1	4
  small request during a download	1209 ms (p90 1667)	62 ms (p90 62)
  300-request burst	6.5 s	1.9 s
  60-object page over 6 connections	1144 ms	734 ms
  single download	0.6–0.7 MB/s	0.6–1.3 MB/s

  A single download still travels on one connection; what changes is that it no longer holds up everything else.