We build Colitu, a VPN with open-source apps for Android, iOS, Windows, Linux and a browser extension. The apps carry traffic over several transports. Hysteria2 is a QUIC-based one that our apps prefer on many networks, because it holds up well on lossy mobile links.

The apps also check their own connection. Every 5 seconds they send a small request through the tunnel. If three checks in a row get no answer within 4 seconds, the transport is declared dead, and traffic moves to a TCP spare that is already connected. Users stay online. But a transport that keeps getting declared dead is a bug, even if nobody notices.

How we noticed

We were testing port hopping for Hysteria2 on an Android phone on home Wi-Fi. With port hopping, the client moves to a new UDP port every ~30 seconds, so networks that slow down long-running UDP flows have a harder time doing it.

Hopping itself worked. We saw five hops, and the server's replies arrived on every new port.

Then we ran a speed test. Hysteria2 carried traffic for about 50–60 seconds and then stopped for good. The health check gave up and traffic went to the spare. It happened every time.

We repeated the test on a server without port hopping and got the same result. So hopping was not the cause; it was just the test that showed the bug.

What the packets said

We captured traffic on the phone. After the stall:

• the QUIC connection was alive: ACKs and small control packets went both ways;
• UDP datagrams still flowed: DNS and the apps' own QUIC traffic, which Hysteria carries as datagrams;
• the client sent no new TCP stream data at all, not even for brand-new requests;
• the server logged nothing unusual.

A connection that is alive but will not send stream data points at flow control, not at the network.

What we ruled out first

We tested each of these instead of assuming:

One more thing we tried: a bigger flow-control window (64 MB per connection, 4 MB per stream). It did not fix anything. It only delayed the stall: with 6 parallel streams it failed in round 3, with 20 streams immediately. That was the strongest hint about where to look.

Reproducing it on one PC

To get rid of the network entirely, we built the official Hysteria server v2.11.0 from source and ran it on loopback with an Xray 26.10.10 client. A small Python harness opened parallel uploads through the client, aborted them with a TCP reset, and then checked whether new requests still worked.

The only thing we changed between runs was how the upload's destination behaved:

An upload speed test behaves exactly like stall. The test server stops reading once it has measured enough, while the browser's upload request is still open.

The mechanism

QUIC limits how much unread data a peer may have in flight, per stream and for the whole connection. Hysteria's defaults are 8 MB per stream and 20 MB per connection. Received bytes count against both windows until the application actually reads them.

Here is what happens:

client (Xray)                 Hysteria server                 target
     |  upload on stream A  ->  |  io.Copy(target, stream A) -> |
     |                          |                               |  stops reading,
     |                          |  target's TCP buffer full     |  keeps conn open
     |                          |  write to target BLOCKS       |
     |                          |  -> server stops reading A    |
     |  A's unread bytes keep counting against the connection window
     |  (a few streams like A fill all 20 MB)
     |
     |  new request on stream B: blocked, no connection credit left
     |  writer for A: blocked on the same window, so the client
     |  never gets to cancel A either
     |
     |  UDP datagrams: not flow-controlled -> still flowing

1. The client uploads through a Hysteria2 stream. The server dials the target over TCP and copies the stream into it.
2. The target stops reading but does not close. Its receive buffer fills, the server's Write into the target blocks, and the server stops reading from the stream.
3. The stream's unread upload data keeps occupying the connection-level receive window. A few such streams use up the whole window. From then on the client cannot send on any stream of the connection, including brand-new ones.
4. The client never gives the stream up, because its own writer is blocked on the same window. On an instrumented server, the "stream given up" signal only arrived when we killed the client process.
5. UDP datagrams are not covered by stream flow control, which is why DNS kept working and the connection looked healthy.

Interesting detail: we first suspected the server never calls CancelRead on dead streams. That was wrong. The stream's Close already does that. The real problem is that nothing ever reaches that Close, because the copy into the target is stuck in a blocking Write.

The fix

The server cannot tell a stalled target from a slow one by looking at the stream. It can tell by progress: a single write that makes no progress for a long time.

So the patch does two things:

1. Every write into a TCP target gets a fresh 10-second budget. If one write makes no progress for 10 s, the target is closed. The normal cleanup then closes the stream, which releases its share of the connection window.
2. When the client gives the stream up, the target is closed immediately, so both copy directions end even when the target is silent.

A slow target that keeps moving is never cut off, because every write starts a new timer. Only a target that has stopped accepting data is closed.

This is the version we sent upstream in PR #1703:

if h.config.TCPStallTimeout > 0 {
    guard := newStallGuardConn(tConn, h.config.TCPStallTimeout)
    tConn = guard
    stop := context.AfterFunc(stream.Context(), func() { _ = guard.Close() })
    defer stop()
}

// stallGuardConn closes the wrapped connection when a single Write makes no
// progress for timeout, and makes Close idempotent. Every Write gets a fresh
// timeout, so a slow target that keeps accepting data is never closed.
type stallGuardConn struct {
    net.Conn
    timeout   time.Duration
    timer     *time.Timer
    closeOnce sync.Once
    closeErr  error
}

func newStallGuardConn(conn net.Conn, timeout time.Duration) *stallGuardConn {
    c := &stallGuardConn{Conn: conn, timeout: timeout}
    c.timer = time.AfterFunc(timeout, func() { _ = c.Close() })
    c.timer.Stop()
    return c
}

func (c *stallGuardConn) Write(p []byte) (int, error) {
    c.timer.Reset(c.timeout)
    n, err := c.Conn.Write(p)
    c.timer.Stop()
    return n, err
}

func (c *stallGuardConn) Close() error {
    c.closeOnce.Do(func() { c.closeErr = c.Conn.Close() })
    return c.closeErr
}

Our production build (v2.11.0-colitu1) does the same thing with SetWriteDeadline before each write. For the upstream version we made three changes:

• the check closes the connection from a timer, so it only needs Close and works with any net.Conn an outbound returns;
• the timeout is configurable as server.Config.TCPStallTimeout (default 10 s, negative disables it);
• Close is idempotent, because two paths can now close the target.

The PR also adds an integration test. Four streams upload into targets that never read (one end of a net.Pipe()), then a new request to an echo server must complete on the same connection. Without the fix it times out after 10 s. With it, it passes in about 3.5 s, with the stall timeout set to 2 s for the test.

One gotcha while writing that test: with one stalled stream it passed even without the fix. One stream can only fill its own 8 MB window, not the 20 MB connection window. It takes three or four.

Results

In the loopback lab:

On the phone, we ran four speed tests in a row over 160 seconds on a patched server in Berlin. Hysteria2 was never declared dead. Before the patch it died at ~60 s every time, both in Berlin and on a server in Vilnius.

Rollout without an app update

Because the bug was on the server, the fix did not need a new app release. Our node image now builds Hysteria from the official source tag with the patch applied. We rolled it out the same night, one server at a time. Each server's VPN service restarted for about 2 seconds. Android, iOS, Windows and Linux users all got the fix at once.

Upstream

The bug is in the open-source Hysteria server, and its client-side half is in Xray, so every service that runs them is affected, not only ours. We reported both:

• HyNetworks/hysteria#1702: the bug, how to reproduce it, and everything we ruled out.
• HyNetworks/hysteria#1703: the fix on the current development branch, with the configurable timeout and the integration test.
• XTLS/Xray-core#7169: the client side. A client that cancels the write side of a stream it no longer needs (CancelWrite, i.e. RESET_STREAM) would release the window even on servers without the fix. A contributor has since traced it on Xray's main branch, reproduced the stall on loopback, and plans a CancelWrite-based fix.

Limitations

• The lab runs on loopback: no packet loss, no added latency. It shows the mechanism and the effect of the patch, not real-world timings.
• The device result is one phone on one network, four speed tests over 160 seconds. Encouraging, not a long-term measurement.
• For up to 10 seconds before the timer fires, TCP over that connection can still pause. The patch makes the stall short; it does not make it zero. A client-side fix would complement it.

Takeaways

• "Connected but silent" usually means flow control. If ACKs and datagrams flow but stream data doesn't, look at windows before you blame the network.
• A bigger window that only delays a failure is a clue, not a fix.
• Make the destination's behaviour a test variable. Four target modes (fast, slow, upclose, stall) narrowed this down faster than any amount of packet reading.
• Write the test that fails first. Ours passed until we used enough streams to exhaust the connection window, which taught us something about the bug itself.

If you run Hysteria2 and your users see TCP die after big uploads while DNS keeps working, this is probably it. The PR is small and easy to apply to v2.11.0.

Full report with setup details: lab.colitu.com/research/hysteria2-stall-after-upload. Our apps are open source at github.com/colitu. Questions and corrections are welcome in the comments.