[jochen-testingbot]

What was wrong

A customer reported that port 2010 "stopped working suddenly". Reproduced end to end with the released 5.0 jar against clones of the real tunnel VM images (22.04 production image and the new 26.04 one), with a fake API, the customer's own sites behind the client, and devices going through the VM:

1. A dead session was not noticed for ~15 minutes. After a NAT/IP change (VPN reconnect, Wi-Fi switch, NAT reset), the old SSH connection gets no ACK and no RST. The 30-second KeepAliveTask only writes; nothing read-side times out (no ServerAliveInterval), so the session looked connected until the OS gave up retransmitting (~15 min on Linux). Meanwhile every device request through 2010 failed.
2. The reconnect's -R 2010 is refused, because the tunnel server still holds 2010 for the session being replaced (sshd's ClientAlive never fires on it while devices keep hitting its listener). finishReconnect() answered that by tearing down the healthy new session and re-dialling every 5 s for 30 rounds (each also waiting up to 10 s for the tcpip-forward reply), then rebuilding the tunnel.
3. After that rebuild, 2010 was never requested again. tunnelReady() → forward refused → scheduleSelfTestRetry(), whose task returns early while !tunnelIsForwarding() — forever, without re-requesting the forward; and PortForwardingMonitorTask only repairs the local forward. SSH up, Selenium relay working, /readyz 503, 2010 dead until the customer restarted the client. On the production image (no server-side changes) this was the observed end state: still broken after 25 minutes.
4. Nothing checked the server end of the reverse forward. verifyReverseForwardDelivery() connects to the local jetty port only.
5. Readiness waited 30 s whenever the first self-test failed, which happens on every fresh tunnel VM (squid on the VM marks its parent 127.0.0.1:2010 dead before the client connects and answers the first request 502).

What changed

• SSHTunnel.connect(): setServerAliveInterval(5000) + setServerAliveCountMax(3) — a dead session is dropped after ~15 s. Connection check every 2 s (was 10), forwarding check every 5 s (was 15).
• CustomConnectionMonitor: first reconnect after 1 s (was 5). When the session is up but the forward failed, it re-requests the forwards on the same session every 2 s (ReconnectableTunnel.restartPortForwarding()), and rebuilds only after ~2 min (MAX_FORWARD_RETRIES). A session that drops meanwhile goes back to a normal reconnect.
• PortForwardingMonitorTask → checkRemoteForward(): re-requests a reverse forward that is not established (startup / after a rebuild), and probes the server end: probeReverseForward() opens a direct-tcpip channel to the server's own 127.0.0.1:2010 and sends a request that only this client's proxy answers with the probe's nonce (new ForwardProbeHandler, ahead of the HTTP log so probes are not logged). Two consecutive failures re-register the forward. A server whose sshd does not permit the probe is detected (the probe never succeeds on that session) and ignored, so older tunnel images behave as before. On success after a repair, App.reverseForwardRestored() re-runs the self-test so readiness follows.
• App: first self-test retry after 2 s, then every 30 s as before.

Server side (testingbot/ansible, tunnel02 image): a PAM session hook ends a customer's older sessions at login, so 2010 is free when the reconnect asks for it; PermitOpen allows the probe; tcp_retries2=8.

Tests

• mvn test: 1011 tests, 0 failures (main: 1005). New: SSHTunnelEmbeddedServerTest (probe round trip through a real reverse forward on the embedded MINA server; probe fails without a forward; a forward refused because the port is held is re-requested by the monitor and the app is told), CustomConnectionMonitorTest (refused forward retried on the same session without re-dialling; session drop during that goes back to reconnect; quick first reconnect; rebuild after MAX_FORWARD_RETRIES).
• End to end (harness in testingbot/ansible tunnel-harness/), NAT change with the old flow blackholed on both ends, measured until 2010 answers again:

client	tunnel image	2010 back after
5.0 release	production 22.04	not within 25 min (rebuilt, never re-requested 2010)
5.0 release (OS tuned to drop the flow in seconds)	26.04 + server fixes	46 s
this branch	26.04 + server fixes	24 s, first reconnect attempt
this branch	production 22.04 + login hook only	24 s, first reconnect attempt

• Regression with this branch's jar against the 26.04 image: proxying/bump/websocket suite, every CLI flag (91 checks), resilience (blips, reconnects, 1500 concurrent connections), SOCKS5/Kerberos/PAC — 250 pass; the 2 failures are the PAC bugs fixed in Honour the PAC failover list when a proxy cannot be reached #64 and Resolve a host name passed to isInNet, as the PAC specification does #65.