OpenSSL HollowByte Flaw Lets 11-Byte TLS Request Freeze Server Memory

A recently disclosed OpenSSL vulnerability dubbed “HollowByte” allows attackers to permanently fragment server memory using nothing more than an 11-byte TLS handshake message, forcing glibc-based servers to reserve up to 131 KB of heap per connection that is never reclaimed. The flaw, reported by Okta’s Red Team and detailed publicly on Thursday, was patched silently in OpenSSL releases 4.0.1, 3.6.3, 3.5.7, 3.0.21, and 3.4.6—all dated June 9—with no CVE, no advisory, and no changelog entry. Matt Caswell’s pull request confirms the OpenSSL security team classified it as a “bug or hardening” issue rather than a vulnerability, a category that does not exist in OpenSSL’s own four-tier severity policy, meaning no scanner or patch management pipeline can flag it.

The root cause lies in how older OpenSSL versions parse the TLS record header. When a ClientHello arrives, the library immediately grows its receive buffer to the size declared in the 3-byte length field—capped at 131 KB for ClientHello—before any body bytes arrive or handshake validation runs. The worker thread then blocks waiting for data that never comes. On glibc systems, when the attacker drops the connection, the buffer is freed back to the allocator, but glibc retains small and medium chunks in its own pools rather than returning them to the kernel. By varying the declared size on each connection, an attacker prevents the allocator from reusing freed chunks, causing heap fragmentation that steadily inflates the process’s resident set size even after the attack stops.

Okta’s benchmarks are stark: a 1 GB NGINX test instance was OOM-killed with 547 MB locked in fragments, while a 16 GB server saw 25% of total system memory frozen without exceeding standard connection limits. The researchers note that conventional connection-exhaustion mitigations, such as per-IP rate caps, are ineffective because the attack stays well under those thresholds. Organizations running unpatched OpenSSL branches should verify their server memory behavior and audit for anomalies consistent with slow, low-volume TLS traffic. Run an SSL/TLS checker to confirm which OpenSSL build your endpoints expose, and use a port scanner to identify internet-facing services that may still be vulnerable to this silent denial-of-service vector.

OpenSSL has not publicly explained why HollowByte was downgraded to a non-security fix, and the decision has drawn criticism from defenders who rely on CVE identifiers to prioritize patching. Okta published its findings without exploit code, and no public proof-of-concept was available on GitHub as of July 18. With no advisory to alert vulnerability management platforms, the burden falls on security teams to manually correlate the June 9 release dates against their deployed OpenSSL versions. Any server running an OpenSSL release older than 4.0.1, 3.6.3, 3.5.7, 3.4.6, or 3.0.21 should be patched immediately, and monitoring should be tuned to detect gradual memory growth on TLS-terminating processes rather than spikes in connection counts.