Benchmarks
What it saves, what it costs,
and where it does nothing.
This page answers three questions: how many tokens Compresso saves, whether compressing changes the model’s answers, and how much time it adds to a request. Every number comes from the open source engine’s published benchmarks, including the results that are not flattering.
01 · Accuracy
Does compressing change the answer?
The question that matters most. If the model answers differently on compressed input, nothing else on this page is worth reading.
Recall · article extraction
98.2%
of the original content survived, at 94.9% compression.
Needle in a haystack
4 / 4
A critical error buried at entry 67 of 100 was still found after 87.6% compression.
QA F1, after extraction
+0.02
Answer quality went up, not down — 0.85 → 0.87.
| Metric | Headroom | What it means |
|---|---|---|
| Recall | 0.982 | 98.2% of the real content survived the compression. |
| Precision | 0.879 | Some boilerplate came through with it. |
| F1 score | 0.919 | The two combined. |
| Compression | 94.9% | At a twentieth of the tokens. |
| Metric | Original HTML | Extracted | Delta |
|---|---|---|---|
| F1 score | 0.85 | 0.87 | +0.02 |
| Exact match | 60% | 62% | +2% |
| Metric | Baseline | Headroom |
|---|---|---|
| Input tokens | 10,144 | 1,260 |
| Correct answers | 4/4 | 4/4 |
| Compression | — | 87.6% |
02 · Compression by content type
Two of these compress by nothing at all
Structured, repetitive machine output is where the slack lives. Source code and grep hits, in this run, came out exactly the size they went in — the router looked at them and passed them through untouched. That is the honest shape of the result.
| Content type | Original | Compressed | Saved | Ratio | Latency |
|---|---|---|---|---|---|
| JSON array (100 items) | 3,163 | 297 | 2,866 | 90.6% | 1ms |
| JSON array (500 items) | 9,526 | 1,614 | 7,912 | 83.1% | 2ms |
| Shell output (200 lines) | 3,238 | 469 | 2,769 | 85.5% | 1ms |
| Build log (200 lines) | 2,412 | 148 | 2,264 | 93.9% | 1ms |
| grep results (150 hits) | 2,624 | 2,624 | 0 | 0.0% | <1ms |
| Python source (~480 lines) | 2,958 | 2,958 | 0 | 0.0% | <1ms |
| Total | 23,921 | 8,110 | 15,811 | 66.1% | 5ms |
Try it on today’s session.
Enjoy fewer tokens from your first request.
7 days free, no card.
03 · Production telemetry
And here is what it does in the wild
Lab numbers measure a payload. This measures real sessions — 50,000+ of them. The median session compresses 4.8%, because a real session is mostly things that don’t compress. The mean is pulled to 11.3% by the sessions full of tool output. Both numbers are on this page on purpose.
Median compression
4.8%
P25 and P50 alike.
P75
6.9%
Three sessions in four are below this.
Mean compression
11.3%
Dragged up by the heavy-tool-output tail — that's where the value is.
Fleet total
1.4B
tokens saved across 249 instances, ≈ $4,000.
04 · Latency
The cost side of the trade
Compression is not free. In production the median request pays 52ms — but the P99 pays four seconds.
Median (P50)
52ms
What almost every request feels.
P90
309ms
Noticeable, still under the model's own latency.
P99
4,172ms
The tail is real. Four seconds of local work before the request leaves.
Mean
161ms
Skewed by that tail.
| Step | Median | P90 | What it is |
|---|---|---|---|
| pipeline_total | 16.9ms | 289ms | Full compression pipeline |
| content_router | 11.7ms | 259ms | Content detection + routing |
| compressor:smart_crusher | 50.1ms | 50ms | JSON array compression |
| compressor:text | 32.0ms | 576ms | Text compression (Kompress ONNX) |
| compressor:mixed | 316ms | 428ms | Mixed content |
| compressor:code_aware | 815ms | 886ms | Tree-sitter AST compression |
| _initial_token_count | 2.9ms | 16ms | Token counting (tiktoken) |
| _deep_copy | 0.1ms | 0.3ms | Message copy overhead |
05 · Break-even
Compressing can make the request faster
Fewer tokens means less prefill, so the model starts answering sooner. Whether that pays for the compression depends entirely on how fast the model is: on a large model the trade is overwhelmingly positive, on a small cheap one it is negative every single time.
| Scenario | Compress | GPT-4o mini | GPT-4o | Sonnet 4.5 | Opus 4 |
|---|---|---|---|---|---|
| Search results · 100 items | 189ms | −102ms | +72ms | +72ms | +507ms |
| Search results · 500 items | 943ms | −456ms | +518ms | +518ms | +2,952ms |
| Search results · 1K items | 2,012ms | −1,022ms | +957ms | +957ms | +5,905ms |
| Search results · 5K items | 12,213ms | −7,201ms | +2,822ms | +2,822ms | +27,881ms |
| API responses · 500 items | 743ms | −365ms | +391ms | +391ms | +2,280ms |
| Database rows · 1K | 961ms | −530ms | +331ms | +331ms | +2,483ms |
| String array · 1K | 146ms | −51.9ms | +136ms | +136ms | +605ms |
| Number array · 1K | 301ms | −243ms | −126ms | −126ms | +165ms |
| Mixed array · 250 items | 38.4ms | −19.5ms | +18.2ms | +18.2ms | +113ms |
Net win in 11 of 12 scenarios against Sonnet 4.5; in 12 of 12 against Opus 4; in 0 of 12 against GPT-4o mini. Average compression across the suite: 93%.
Try it on today’s session.
Enjoy fewer tokens from your first request.
7 days free, no card.
06 · Caveats
What these numbers are not
| The latency suite is stale by the publisher's own admission | v0.5.6 added parallel message compression and removed redundant token counting; the page says to expect lower latency now. |
| Latency was measured only on JSON | All 12 scenarios are JSON. No code or prose latency figures exist in that suite, though the production table shows code_aware at 815ms median. |
| Break-even assumes a prefill rate | 0.03ms/token for Sonnet 4.5, $3.00/MTok. Different model, different answer — as the GPT-4o mini column shows. |
| One machine, one Python | macOS arm64, Python 3.11.11, single run. The publisher notes results vary with hardware and content. |
| Fleet telemetry is four days | 30 March – 2 April 2026, 249 clean instances, mixed OS — 57% Linux, 38% macOS. Compresso is macOS-only, so the fleet is not our user base. |
| Accuracy is four benchmarks, not a guarantee | 100–181 samples each. Strong evidence, not proof that your workload survives. |
References: headroomlabs-ai.github.io/headroom/benchmarks/ · /LATENCY_BENCHMARKS/