parsec
  • measurement
  • engineering

measured, not modeled

Every context-compression tool prints a savings number. Almost all of them are modeled: assume a baseline ("without us, this request would have cost X"), estimate X with a multiplier fit against yesterday's traffic, subtract, print a percentage.

Modeled baselines drift. We know because we shipped one. The reference stack behind parsec once estimated savings with a per-step inflation constant — a single fitted number standing in for "what the agent would have sent." Harness versions changed, prompt shapes changed, and the constant quietly stopped describing anything. Worse, we later found a code path whose output never reached the wire at all, yet was still being credited as savings by the accounting layer. Nothing was lying on purpose. That is the point: a modeled number has no mechanism that forces it back to reality.

the counterfactual

Anthropic's count_tokens endpoint is free. That makes an honest measurement affordable on every single request:

  1. The proxy holds the original request body — the one it would have forwarded if it did nothing.
  2. It probes count_tokens on that original body.
  3. It forwards the curated body, and the response comes back with the actually-billed usage: input, output, cache-read, and cache-write tokens.
  4. Both numbers land in the same event row:
tokens_saved = counterfactual_input
             − (billed_input + billed_cache_read + billed_cache_write)

No model of what a baseline agent does. The baseline is this request, this turn, unedited — measured by the same tokenizer that bills you.

The subtraction is deliberately unflattering. Cache reads and cache writes are billed input, so they count against savings; skip them and you treat cache-served prefix as free. On our own ledgers, doing that overstated savings by more than 2×. And because there is no probe for output — you cannot ask what the model would have said — the number is input-side only, and no saved-dollar figure we report ever includes output tokens.

the ledger you can read

Every row is written to ~/.parsec/ledger.jsonl on your machine, whether or not you have an account. It is plain JSONL: cat it, jq it, diff it against your Anthropic invoice. The /parsec:savings command and the status line read this file, nothing else.

Shipping rows to the dashboard is an additional fire-and-forget POST — token counts, model id, opaque conversation ids, never message text — idempotent on request_id so retries are harmless. The local file is the source of truth; the cloud copy is a convenience.

holes, not zeros

Probes fail. Networks flake, endpoints time out. When that happens, the row stores counterfactual_input_tokens = NULL — a hole, never an imputed zero.

This matters more than it looks. An imputed zero is a lie with good posture: it drags the average down, and a broken measurement pipeline becomes indistinguishable from a modestly-performing product. A hole is honest — summaries derive savings only over rows where the probe succeeded, and report coverage alongside the number. The same rule holds one layer up: a model id we have no pricing for reports cost_usd = null, not a fabricated figure.

one number, three jobs

The same measured row is:

  • the billing basis for metered team plans,
  • the upsell proof when a free-tier user decides whether the paid curator is worth it,
  • the trust artifact an enterprise audits before self-hosting.

Those three uses pull a modeled number in three directions — billing wants it defensible, marketing wants it large, audit wants it reproducible. A measurement taken from the request itself, by the provider's own tokenizer, stored where the user can read it, removes the surface where those pressures could act. We could not inflate it without forging the ledger on your disk.

the honesty complement

There is one failure mode measurement alone cannot catch. Every layer of parsec fails open: on any error, the proxy degrades to passthrough and your request goes through untouched. That is the right behavior — but a proxy silently serving a 0% cut is indistinguishable from a working one unless you count the fail-opens.

So the fail-open counter is a first-class, alertable metric, recorded in the same ledger rows as the savings. The savings number tells you what the proxy did. The fail-open rate tells you how often it did nothing. You need both, or the first one is theater.

Measured, not modeled. It is a slower way to get a marketing number, and the only way to get one that survives contact with your invoice.