Cost control

Beta gaps

Caching

Cached tokens are billed at the cached rate. The provider metrics dashboard remains off by default.

Updated 2026-10-27 · MainAI Compute docs

Two signals exist: per-request token usage from the engine, which drives the bill, and optional Prometheus gauges from vLLM/SGLang, which drive the dashboard. Only the first is live by default.

Status matrixLink to this section

Piece	State	What that means
Per-request cached-token billing	Live	Reported cached tokens are priced at the catalog cached-input rate.
Per-request cache sample adapter	Live	Maps `cached_tokens`, cache writes and exact hits into `cache_samples`.
`cache_samples` table	Exists	Seeded demo rows are illustrative.
Provider metrics collector	Wired	Polls `/metrics`; no targets configured.
`provider_cache_snapshots`	Exists	Empty without a collector run.
Provider metrics flag	Off	`CACHE_METRICS_ENABLED=false` hides the metrics dashboard.
Portal metrics screen	Empty state	Returns `enabled: false`; never invents numbers.
Redis response cache	Separate	Not wired. Semantic cache stays off until hit quality is measured.

Per-request cached-token billing

State: Live
What that means: Reported cached tokens are priced at the catalog cached-input rate.

Per-request cache sample adapter

State: Live
What that means: Maps cached_tokens, cache writes and exact hits into cache_samples.

cache_samples table

State: Exists
What that means: Seeded demo rows are illustrative.

Provider metrics collector

State: Wired
What that means: Polls /metrics; no targets configured.

provider_cache_snapshots

State: Exists
What that means: Empty without a collector run.

Provider metrics flag

State: Off
What that means: CACHE_METRICS_ENABLED=false hides the metrics dashboard.

Portal metrics screen

State: Empty state
What that means: Returns enabled: false; never invents numbers.

Redis response cache

State: Separate
What that means: Not wired. Semantic cache stays off until hit quality is measured.

What you see todayLink to this section

Usage rows and the ledger use cached_tokens at the published cached-input price whenever the engine reports them.
x-mainai-cache is bypass unless the gateway reports a read cost; there is no synthesized hit.
GET /api/portal/cache returns enabled: false with empty models and provider arrays while metrics are off.
The demo seed contains cache samples so the UI has a shape to build against. Do not use demo ratios as live truth.

Disabled cache response

json

{
  "enabled": false,
  "from": "2026-10-01T00:00:00.000Z",
  "to": "2026-10-25T00:00:00.000Z",
  "models": [],
  "provider": []
}

Cached-token semanticsLink to this section

A cached input token is an input token the engine did not need to recompute. It still occupies the prompt budget. Two kinds are measured:

Kind	Source	Meaning
`prefix`	OpenAI-compatible usage / SGLang metadata	A shared prompt prefix was reused. Usually automatic.
`exact`	Response cache	The same request produced a stored response.

prefix

Source: OpenAI-compatible usage / SGLang metadata
Meaning: A shared prompt prefix was reused. Usually automatic.

exact

Source: Response cache
Meaning: The same request produced a stored response.

Usage details can carry prompt_tokens_details.cached_tokens and cache_creation_tokens. cached_tokens is a subset of prompt_tokens, not an extra charge. Cache writes can have their own price; seeded models set it to zero.

What appears if metrics are enabledLink to this section

Surface	Today	When enabled
`x-mainai-cache`	`bypass`	`hit` or `bypass` (`miss` reserved)
Usage tokens	Occasional raw engine value	Reconciled `cachedTokens` and `cacheWriteTokens`
Portal cache page	Disabled state	Per-model ratio, cached tokens, estimated savings
Provider gauges	None	vLLM/SGLang hit ratio and KV utilization
Billing	Reported cached tokens at cached rate	Same billing; savings are now visible

x-mainai-cache

Today: bypass
When enabled: hit or bypass (miss reserved)

Usage tokens

Today: Occasional raw engine value
When enabled: Reconciled cachedTokens and cacheWriteTokens

Portal cache page

Today: Disabled state
When enabled: Per-model ratio, cached tokens, estimated savings

Provider gauges

Today: None
When enabled: vLLM/SGLang hit ratio and KV utilization

Billing

Today: Reported cached tokens at cached rate
When enabled: Same billing; savings are now visible

Estimate conservatively. A repeated system prompt may hit a prefix cache, but only engine-reported tokens qualify for the cached rate.

Prompt patterns that will benefitLink to this section

Stable instructions and tool schemas at the front of the prompt.
Long reference documents reused across questions.
Prefixes identical down to whitespace; a changed token invalidates the suffix.
Batch questions against one document instead of resending it per call.

Put variable text last. Do not pad prompts hoping for a hit; every input token still costs something.

PrivacyLink to this section

Cache metrics carry model and token counts, not prompt bodies. An exact-cache entry can store a response body; scope and TTL are gateway settings, not part of the current contract.

Enabling, in orderLink to this section

01Review the adapters and fixtures with a second operator.
02Set CACHE_METRICS_ENABLED=true for the metrics UI.
03Configure one canary CACHE_TARGETS endpoint; start the collector.
04Send canary traffic and confirm rows only for the canary key.
05Compare sample counts with billed usage before enabling the dashboard.
06Wire response caching separately; keep semantic caching off.

Operator procedure: `docs/runbooks/enable-cache-metrics.md`.