HollowByte: how an 11-byte packet bloats OpenSSL server memory — no CVE, but patched in 4.0.1/3.6.3/3.5.7/3.4.6/3.0.21

TL;DR — 90 seconds

Affected?

OpenSSL versions before 4.0.1, 3.6.3, 3.5.7, 3.4.6, or 3.0.21 (depending on the branch in use) that terminate TLS connections server-side.

Risk?

No CVE, no CVSS score — classified by OpenSSL as a hardening fix, not a classic vulnerability. Denial of service via memory exhaustion: an 11-byte packet with an inflated length field makes the server reserve memory for data that never arrives; due to glibc allocator behavior, this memory stays bloated even after the connection ends.

Immediate action?

Update to a patched version (4.0.1 / 3.6.3 / 3.5.7 / 3.4.6 / 3.0.21) and restart the affected worker processes to reclaim already-bloated memory — the patch alone is not sufficient without a restart.

Recommendation?

Patch promptly on internet-reachable TLS termination points (reverse proxies, load balancers, direct OpenSSL usage) — especially on small, resource-constrained instances, which test reports show are easily depleted.

Criticality?

medium — no data exposure and no code execution, but a demonstrably effective, low-cost DoS technique against widely deployed TLS infrastructure.

What is the problem?

During a TLS handshake, every message announces in its header how many bytes of payload will follow. Vulnerable OpenSSL versions allocate the memory needed for this already when reading that announcement — not once the data has actually arrived and its size has been verified. An attacker exploits exactly this ordering: a packet of only 11 bytes total is enough to announce a substantially larger message in the header. The server allocates the corresponding memory block and then waits — potentially for a long time — for data that is never sent.

The real amplification effect comes from the interplay with glibc's memory management: glibc does not immediately return small-to-medium allocations to the operating system after they're freed, but keeps them around for potential reuse. An attacker who opens connections across multiple waves, each announcing a differently sized, randomized message length, specifically prevents these memory blocks from being reused — the result is progressive heap fragmentation. This fragmentation outlives the individual connections; the process stays bloated until it is restarted.

OpenSSL treats HollowByte as a hardening measure rather than a vulnerability in the strict sense, presumably because there is no classic memory-safety bug (buffer overflow, use-after-free), but a resource-management issue. The fix changes the allocation behavior: the buffer now grows only as data actually arrives, not based on the header's announcement.

Who is affected?

Check your own OpenSSL version quickly with openssl version on the host, or openssl version inside the respective container image.

Impact

HollowByte causes no data exposure and no code execution — the impact is purely availability. Per the reports reviewed, Okta's red-team testing against NGINX showed that resource-constrained environments can easily be driven to exhaustion, while larger servers can lose up to 25% of their memory without crossing typical memory-usage alert thresholds — a pattern that specifically evades classic monitoring thresholds.

Particularly unpleasant operationally: the bloated memory persists even once the attacker has long stopped and all connections are closed — simply dropping connections or rate-limiting new ones does not undo damage that has already occurred. Without a process restart, memory usage stays permanently elevated, which favors recurring failures under load, increased swap usage, or kernel OOM kills.

Mitigation / immediate steps

Operational decision block

• Act now if … you run internet-reachable, resource-constrained TLS termination points (small VMs, edge instances, ingress controllers with tight memory limits) on unpatched OpenSSL.
• Check with priority if … you run larger servers/clusters that terminate TLS — the effect is slower here but, per test reports, can cause up to 25% memory loss without triggering alerts.
• Next regular window if … your systems don't terminate TLS themselves (pure TCP passthrough) or are already on a patched version.

Step 1 — check version and update

# check installed version
openssl version

# target versions (patched):
# OpenSSL 4.0.1 / 3.6.3 / 3.5.7 / 3.4.6 / 3.0.21 (branch-dependent)

# Debian/Ubuntu
sudo apt update && sudo apt install --only-upgrade openssl libssl3

# RHEL/Alma/Rocky
sudo dnf update openssl

# for source/custom builds: rebuild and roll out against the official
# OpenSSL tag of the respective patched version

Step 2 — restart affected processes

# the patch alone is not enough: already-bloated memory is only
# released by restarting the process
sudo systemctl restart nginx
# or the respective TLS-terminating service (haproxy, envoy, your own server)

# compare memory footprint before/after the restart
ps -o rss,cmd -C nginx

Step 3 — defense-in-depth, independent of the patch

# set tight handshake timeouts (example nginx):
# ssl_handshake_timeout 10s;

# limit maximum concurrent connections per worker
# and throttle connection rates per source IP (e.g. via fail2ban,
# nginx limit_conn/limit_req, or an upstream L4 load balancer)

Detection / verification

Check version

openssl version
# or per container image:
docker run --rm <image></image> openssl version

Observe memory behavior

No specific indicators of compromise for HollowByte itself were published in the sources we reviewed — the steps below are general observation points derived from the mechanism described:

# watch RSS memory usage of the TLS-terminating process over time,
# independent of the number of currently active connections —
# growth without a corresponding rise in active sessions is the core indicator
watch -n 5 'ps -o rss,cmd -C nginx'

# check for unusually many short-lived TLS connections that
# terminate early during the handshake (access/error logs, connection tracking)
ss -tn state syn-recv

# check memory usage after connections have fully torn down —
# if memory stays high despite no active connections remaining,
# that's a strong signal that fragmentation has already occurred

Operator guidance

Mid-market

Patch in the next regular maintenance window and restart the affected services — no CVE, no emergency rollout needed, but no reason to sit on it either, since the technique demonstrably works.

Enterprise / multi-site

Inventory all TLS-terminating components (reverse proxies, load balancers, ingress controllers, your own OpenSSL integrations) and centrally track their OpenSSL version. Extend memory monitoring with an “RSS without active sessions” metric to catch similar patterns earlier in the future.

Hosting / platform operators with many small instances

Urgency is highest here: small, resource-constrained instances are, per test reports, the easiest to exhaust. Prioritize the patch rollout and add handshake timeouts and connection limits as extra protection before the patch is fully rolled out.

Decision block

Act promptly if: small, internet-reachable TLS termination points are running unpatched OpenSSL. A regular window is enough if: already patched, or OpenSSL isn't used for your own TLS termination.

Conclusion

HollowByte shows that not every relevant security improvement carries a CVE and a CVSS score — OpenSSL deliberately classifies this weakness as a hardening fix, since it's not a classic memory-safety bug but a resource-management issue. The effect is real nonetheless, and cheap to reproduce: an 11-byte packet is enough to make a server pre-allocate memory it never fills, and the interplay with glibc's allocator behavior turns that into persistent fragmentation rather than a transient blip. Anyone running TLS termination with OpenSSL shouldn't defer this patch just because it lacks a formal CVE number — classification and actual risk diverge here.

Sources

• BleepingComputer — HollowByte DDoS Flaw Bloats OpenSSL Server Memory With 11-Byte Payload

Only the source above was available for HollowByte at the time of this post; an official OpenSSL advisory or release announcement with the exact commit/changelog references could not be independently verified in addition. Details such as the CVSS rating, official designation, and commit history should be cross-checked against the OpenSSL release notes of the named versions before making far-reaching decisions.