millivolt Run it locally
Documentation menu

Protocol and client integration

The client-facing interface accepts OpenAI Chat Completions, OpenAI Responses and Anthropic Messages. The upstream-facing protocol can differ from any of them. Per-request headers select the destination, credentials and optional native adapter. The compatibility matrix maps the supported combinations; cross-protocol translation documents the canonical model and its loss policy. No endpoint/API-key registry is required. Read Security: routing credentials do not authorize operator access, and the whole dashboard plane is gated by MILLIVOLT_OPERATOR_TOKEN (operator access); only /healthz, /favicon.ico (and the other origin-root brand/PWA files), the /session login handshake and the relay stay open.

Endpoint behavior

The dashboard, embedded assets, /admin/* actions, /healthz, the /session login handshake, the /mcp MCP endpoint and registered /metrics/* routes belong to millivolt. Their methods and meanings are listed in operations. GET /v1/models and GET /models use the discovery path described below. POST /v1/chat/completions, POST /v1/responses and POST /v1/messages select the client wire protocol. Other paths reach the relay catch-all and require valid routing headers; the client chooses the upstream endpoint and model.

A known client path is translated only when X-Proxy-Format selects a different upstream protocol. An absent X-Proxy-Format selects passthrough, as does an explicit format equal to the client protocol. Without a configured request rewrite, a passthrough request's body bytes are relayed unchanged; configured request overrides, a sub-conversation strip or ensured tools can still rewrite the upstream body (see request overrides). X-Proxy-In-Format overrides the protocol named by the path, so an otherwise unknown path is translated when that header names the inbound wire; without the override, paths outside the three known endpoints are never translated. A translated request addresses the outbound protocol's canonical endpoint (/v1/chat/completions, /v1/responses or /v1/messages) instead of forwarding the client's path, unless X-Proxy-Path overrides that endpoint. A same-protocol request on one of those three endpoints normalizes to the same canonical spelling, while one on any other path keeps the path the client chose. The path is matched in its escaped spelling, so an encoded spelling of a known route (for example /v1%2Fchat%2Fcompletions or /v1%2Fmodels) is an unknown path: on a same-protocol passthrough, or when no inbound override is present, it is relayed unchanged and is never normalized, translated or treated as model discovery. When X-Proxy-In-Format selects a wire that differs from X-Proxy-Format, the request is a translation like any other and addresses the outbound protocol's canonical endpoint. A base URL with a path joins by comparing the first client segment's decoded spelling with the base's last segment, so an encoded client spelling of that segment can be dropped from the joined upstream path.

Forwarding a body is not the same as understanding every field in it. The normal relay does not rewrite an unfamiliar JSON shape just to observe it, but usage, first-token timing and quality classification depend on recognized formats. Native translation is explicit and narrower than arbitrary HTTP forwarding. There is no automatic provider failover or model-selection registry.

Routing headers

Only X-Proxy-Base-URL is required by the proxy. The upstream may require a key. Routing validation and request construction live in request.go.

Request header Meaning
X-Proxy-Base-URL Absolute HTTP(S) upstream base; no userinfo, query or fragment, including a bare trailing ? or # (a percent-encoded %23 remains path data).
X-Proxy-Key Upstream key; takes precedence over Authorization. Without it, a Bearer credential or the supplied Authorization value is used. No credential is also allowed.
X-Proxy-Auth-Header Key's upstream header; defaults to Authorization.
X-Proxy-Auth-Prefix Prefix before the key; defaults to Bearer. A present empty header selects no prefix.
X-Proxy-Path HTTP relay: replace the incoming path before joining it to the base URL. Cursor uses its native path rule; model discovery ignores this header.
X-Proxy-Query HTTP relay: nonempty query override; otherwise preserve the incoming query. Cursor and model discovery ignore both.
X-Proxy-Headers HTTP-relay header overrides as a JSON object of string arrays, e.g. {"x-example-version":["2026-01"]}. accept-encoding and range are proxy-owned and rejected with 400. A JSON null member is treated as an omitted header. Not applied to model discovery or the Cursor bridge.
X-Proxy-Timeout-Ms Positive milliseconds. HTTP relay: a fresh headers-and-body deadline per attempt, excluding queue/hold/backoff waits. Cursor applies it only to send header waits; model discovery uses its separate configured budget.
X-Proxy-Format Upstream format: openai (Chat Completions), openai-responses (Responses), anthropic (Messages), or cursor (agent.v1). Absent means passthrough; configured request rewrites can still change the body. See supported combinations.
X-Proxy-In-Format Optional client-protocol override: openai, openai-chat or chat (Chat Completions), openai-responses or responses (Responses), or anthropic (Messages). Aliases normalize to the canonical wire. Normally unnecessary because the request path selects the protocol.
X-Proxy-Client Observability/client label; one that only repeats the user agent's product name records the fuller user-agent identity instead. Otherwise inferred from client SDK/User-Agent metadata.
X-Proxy-Session Explicit conversation ID; otherwise a configured tracked body param or automatic grouping applies.
X-Proxy-Parent-Session Direct declared parent; requires an explicit own session on the same request.
X-Proxy-Max-Concurrency Positive provider-plus-key scheduler-group cap. Each admission updates the shared group's policy; later admissions, including headerless ones using config defaults, can replace it. Not the persistent provider-wide control, and not applied to model discovery, which returns before inference admission.
X-Proxy-Limit-Concurrency Sticky provider-wide in-flight cap.
X-Proxy-Limit-Requests Sticky provider-wide request rate as COUNT/WINDOW.
X-Proxy-Limit-Tokens Sticky provider-wide token rate as COUNT/WINDOW.
X-Proxy-Refresh-Token Opt-in stateless expired-token refresh for supported mechanisms. See refresh.

Every X-Proxy-* routing, limit and format header is read with first-value semantics: when a request carries the header more than once, the first value is authoritative and later values are ignored. The rule is deterministic across the whole control prefix, including X-Proxy-Query and X-Proxy-Headers.

Client identity labels are retention-bounded before the record enters the live ring, analytics or durable store: classified client and language 256 bytes, user agent 1024 bytes, and the first forwarded address 128 bytes. Longer values are cut on a UTF-8 boundary and end with …; normal labels are unchanged.

Finalized and pending record text is retention-bounded at the record boundary before it enters the live ring, pending view or durable store: path 1024 bytes, method 32 bytes; provider label, model, provider model, provider request id, provider server, finish reason, error type, error code, error message, request tool choice, response format, service tier and last turn role 512 bytes each; each tool name 128 bytes with at most 64 retained in order; debug session id 128 bytes; and conversation identifiers 512 bytes before the namespace prefix (retained identifier 514 bytes). Absorbed-attempt provider model, provider request id, provider server, error type, error code and error message use the same 512-byte bound; provider labels and attempt error messages are bounded at the same record boundary. Longer values are cut on a UTF-8 boundary and end with …; normal values are unchanged. These record bounds do not alter debug capture bodies, retained headers, content previews or request-parameter JSON.

One caller setting X-Proxy-Limit-* affects every key/client of that provider. Absent headers preserve limits. A present X-Proxy-Max-Concurrency or X-Proxy-Timeout-Ms value follows Go's HTTP field parsing: leading and trailing optional whitespace is removed before the handler reads it, so a whitespace-only value arrives empty and is treated as absent, while a padded numeric value is read as its trimmed spelling. 0, off, none or unlimited clear a X-Proxy-Limit-* policy. Windows accept Go durations from 1 second through 24 hours, plus 1d. Malformed limit values reject the request without applying any of its limit headers. The Limits menu shows the policy and its source. Rates use continuously refilled budgets, not strict fixed-window quotas; see Pause and Limits for charging and burst behavior.

Supported proxy-control headers never forward upstream. Injected names must be valid, unique case-insensitively, and not reserved control, framing (content-length), hop-by-hop or Host names; values must be valid HTTP field values. Ordinary hop-by-hop and Connection-nominated headers are removed. Precedence depends on the request path:

Path Header precedence, lowest to highest
HTTP relay, including Anthropic translation Forwarded values → extracted authentication → configured provider headers → explicit injection → matching request overrides. Configured/injected authentication can override the extracted key.
HTTP model discovery Constructed authentication → configured provider headers. Explicit injection and request overrides are not applied.
Cursor run/model discovery Configured headers → matching request overrides (runs only) → required protocol/framing headers and extracted authentication when a key is present. Explicit injection is not applied.

The provider label is server-derived from the base host's registrable domain; IPs retain their authority/port, and nonregistrable hostnames remain hostnames. Optional provider aliases apply afterward. There is no supported X-Proxy-Provider override.

URL and auth examples

For the HTTP relay, a base ending in /v1 and request path /v1/chat/completions do not produce /v1/v1/chat/completions: the join removes the shared leading segment. A path override still joins to the selected base. Use X-Proxy-Query rather than embedding a query in the base URL.

For a native Anthropic-compatible endpoint, use its documented base/model/key and set translation plus the appropriate upstream auth/path/version headers:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "X-Proxy-Base-URL: $NATIVE_BASE_URL" \
  -H "X-Proxy-Key: $NATIVE_API_KEY" \
  -H 'X-Proxy-Auth-Header: x-api-key' \
  -H 'X-Proxy-Auth-Prefix;' \
  -H 'X-Proxy-Path: /messages' \
  -H 'X-Proxy-Format: anthropic' \
  -H "X-Proxy-Headers: {\"anthropic-version\":[\"$NATIVE_API_VERSION\"]}" \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"$NATIVE_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"hello\"}]}"

The example assumes JSON-safe model/version values. The semicolon in -H 'X-Proxy-Auth-Prefix;' is important: curl sends a present empty header. -H 'X-Proxy-Auth-Prefix:' instead omits it, leaving the proxy's Bearer default. See the official curl header guide. An unauthenticated local upstream needs no dummy API key.

Relay, retries and model discovery

Ordinary requests relay the original body bytes unless a configured request override, sub-conversation strip or ensured-tools rule rewrites them; the request is buffered up to max_request_bytes before metadata extraction and sending. Larger requests are rejected with HTTP 413 without an upstream send. A decoded output cap above 1,000,000 tokens is rejected with HTTP 400 the same way, before any upstream send and at both trust boundaries - the original body's decode and the re-decode of a translated body - for any spelling (max_tokens, max_completion_tokens or max_output_tokens). A body that combines the Responses max_output_tokens spelling with a Chat max_tokens or max_completion_tokens spelling is rejected with HTTP 400 before an upstream send because the wire families are ambiguous. More than one top-level member that folds to the same cap name is also rejected with HTTP 400 before an upstream send, including case variants and escaped spellings: the relay sends every occurrence, so accepting duplicates would let the validated and recorded value differ from the bytes the upstream reads. A valid request is also rejected with HTTP 400 before an upstream send when it carries more than 10,000 entries in a messages, tools, functions, input, stop, stop_sequences or modalities array, more than 10,000 members in a metadata or stream_options object, more than 1,024 aggregate tool_calls members across every message, more than 10,000 distinct top-level members, or more than 10,000 distinct logit_bias keys merged across duplicate top-level objects. Those preflight bounds apply to native and translated requests alike. A translated request also passes through its protocol codec, which rejects with HTTP 400 more than 10,000 entries in a Responses include array or in a tool allowed_callers array, or more than 4,096 blocks in an Anthropic system list. A native passthrough without a configured body rewrite does not apply those codec-only collection bounds: the body is relayed unchanged, so there is no materialization to bound. A configured request_overrides rule, a sub-conversation strip or an ensured tool can still rewrite that body, but those rewrites are not a codec decode. Content composition has two independent limits. A content array above 4,096 parts is not rejected for that reason on the byte-transparent passthrough path: within the aggregate budget it still relays unchanged, but composition skips it, so it contributes no dashboard composition counts. On a translated request the codec applies its own limit: Chat drops the oversized content with a warning, while Responses and Anthropic reject the request with HTTP 400; translation_loss: reject turns any recorded loss into a 400. Composition reads the Chat messages/system and Anthropic messages/system shapes; a byte-transparent Responses body's input/instructions items contribute no composition counts (their model, stream flag and output cap are still recorded). Separately, the aggregate content-part count over every message content array, a top-level system array and every Responses input item's nested content/output array is capped at 16,384. Each Responses annotation array is capped at 4,096 members, and their aggregate inside those content parts is capped at 10,000. Anthropic citation arrays are capped at 10,000 members per array, and every citation array in one document shares a 10,000-member aggregate. Each nested content array is capped at 4,096 blocks, and the nested content blocks of one document share a separate 10,000-block aggregate; the citation and nested-block aggregates are independent, so their combined total can reach 20,000. A document block's citations configuration object is reported as a dropped field instead of being decoded into text citations. A valid request above any of these limits is rejected with HTTP 400 before an upstream send. Invalid JSON is passed through on the nontranslated path. Syntactically valid JSON whose strings carry raw invalid UTF-8 is bounded the same way: valid UTF-8 and JSON escapes decode to no more bytes than their spelling, while the standard decoder replaces each invalid byte with a three-byte U+FFFD. A body whose decoded strings would exceed its uploaded size by more than a small internal allowance is rejected with HTTP 400 before an upstream send; smaller invalid sequences remain accepted on the passthrough path, where a tracked value that fails the identity rules is still dropped rather than rejected. Anthropic translation requires the whole request body to be valid UTF-8 and rejects a raw invalid byte before decoding it. The proxy can retry transient transport failures, 429 and 5xx before returning the final response, and retryable in-band error events on an open stream before any generation content was relayed. Durable quota/billing 429s follow quota_pause_mode instead of the transient retry ladder: off (default) surfaces them immediately; retry parks the triggering request and every new request to the same OpenAI-wire provider behind a recovery gate that closes after quota_pause_recovery_successes consecutive successful probes (the default 1 closes on the first 2xx); a provider Retry-After on the quota response paces the next probe like any other 429; manual installs an indefinite provider pause that only an operator resumes. Quota re-sends never burn the transient retry budget. Provider retry hints and operator holds can substantially extend total wall time. With suppress_client_retries the proxy declares itself the client's sole retry authority: every error response the relay returns then carries x-should-retry: false, the official OpenAI SDK convention (python, node, go and ruby honor it from source), which overrides the SDKs' 408/409/429/5xx auto-retry defaults and any Retry-After, including a provider hint that would otherwise relay verbatim. No response exists to carry the directive on transport-level failures, clients that ignore the header (the Vercel AI SDK / opencode stack, whose retry policy is status- and body-text driven) keep their own policy, and mid-stream SSE errors never auto-retry in official SDKs and cannot carry new headers. Success responses are untouched, and the setting hot-reloads for new responses.

SSE pacing may insert comment keepalives. A known queue/hold wait may commit HTTP 200 before the final upstream outcome, so a subsequent failure must be signaled in-band. Bounded terminal/quality handling can replace a degenerate completion with an error. Non-streaming quality retries may make another upstream request; the adopted attempt's response is spooled under the fixed 1 MiB quality buffer before the client status line commits, so the retry is delivered as one ordinary response with no post-commit keepalive gap or partial payload. A stream truncated before any generation content was relayed is re-sent on the same committed connection, while a truncation after relayed content surfaces the in-band error for the client to retry. A trailing partial line that a rescue or the degenerate terminal boundary drops was never delivered, so it neither counts as relayed content nor as the record's answer content or first-answer time, and it does not veto the re-send; only a delivering path feeds and writes it, exactly once. Only the truncation class is decided before that feed, and only when the analyzer's non-committing tail classifier says the line carries no completion terminal marker: a final Chat [DONE] or non-null finish_reason without a trailing newline is fed rather than dropped. A fed tail is delivered only when it supplies a complete answer or tool call; an empty or reasoning-only completion resolves through the degenerate boundary, which withholds or replaces the terminal bytes. A terminal tail is classified before the pre-content rescues, so it is never re-sent. A delivered tail is written as its own complete SSE event: a pending event is closed first and a blank boundary follows, on clean EOF and on a read error. Answer content that the source places after its finish marker is therefore relayed as a later event, and a client that stops at the finish reason may ignore it. A terminator-less partial line that carries neither a completion terminal marker nor a completion is not delivered, so the stream reports truncation rather than a completion or provider error the client never saw. A standalone in-band error line is not a delivery terminal and is dropped the same way, and a [DONE] on a Responses feed is not its terminal marker, so it is also dropped as truncation. A translated streaming request whose 200 response body is not SSE is still relabelled text/event-stream: the upstream bytes are never converted into SSE frames, and the relay surfaces the non-SSE body as a truncated-stream error rather than translating it. A provider error event streamed in-band on the 200 is retried the same way while nothing content-bearing was relayed: the error classes the OpenAI API documents as retryable (500 server_error, the 503 overloaded/unavailable family, request timeouts, and the equivalent Anthropic/gateway spellings) are dropped before the wire and the request re-sent, so the client never sees the failure. The retryable_error_classes setting extends that vocabulary with operator-supplied type or code strings for gateway-specific spellings (durable quota/billing classes are rejected at load); rate limits in-band (no Retry-After exists there), durable request/billing classes, unknown types, Responses-API feeds once any of their events was relayed (per-response sequence numbers), and errors after relayed contract bytes relay verbatim for the client to handle. Mid-thinking rescues (thinking_retries) extend the re-send to requests that die or end without an answer during the reasoning phase: a truncation or upstream read error after reasoning-only output, a cleanly finished reasoning-only completion (the withheld terminal region is replaced by the fresh attempt), and a retryable in-band error that arrived after reasoning-only output. The fresh stream appends behind the already-relayed reasoning, which is auxiliary display text; answer content, tool calls, refusals, finish_reason: length (the client's own token cap) and Responses-API feeds (per-response sequence numbers) are never rescued. An exhausted budget surfaces the reasoning_only class as a provider error (upstream_error) on both surfaces: in-band on the streaming connection, and as an HTTP 502 error envelope for non-streaming requests. Stream accounting has two accepted limits. A source that never reports a numeric tool-call index is counted per id-bearing fragment; a provider that repeats the same id across continuation fragments can overcount the calls, and the fallback is unchanged until such a provider shape is observed. Provider reasoning tokens are not cross-checked against output tokens: a reasoning count greater than the reported output is accepted, the derived answer count is max(0, output - reasoning), and the record keeps the provider's own totals. A future observed shape that violates the relation needs a targeted validation rather than a universal rejection.

Neither transparency nor exactly-once upstream execution is unconditional.

Before relay or translation, the upstream request is subject to the configured request_overrides (see request overrides): matching rules set, replace or remove upstream headers as the last header stage, and their body fields set max_tokens / max_completion_tokens on Chat Completions or Anthropic client bodies before translation, whatever the target (a Responses client body is not rewritten), fail closed. A configured sub_conversations strip also removes tracked body fields before relay (tracked body params). Only the upstream request changes; the client-visible contract is unchanged.

Scheduling and timing

The provider-plus-key group controls simultaneous upstream sends and its waiting queue. The separate provider-wide Limits policy combines all keys and clients. Queue capacity and wait limits can reject work locally; the configured queue_retry_after duration (for example 2s, never a bare integer) supplies the caller's retry hint. Operator-hold time is excluded from the ordinary queue-wait limit. A hold's own queue cap still applies.

For the HTTP relay, upstream_timeout limits waiting for response headers, not the streaming body. X-Proxy-Timeout-Ms instead gives each upstream send a fresh deadline covering headers and body. Neither is a total end-to-end deadline across queueing, holds, retries and backoff; client cancellation is the way to stop the whole request. The request-body progress bound applies only while the body is being read. Reaching EOF or closing the body clears the connection read deadline, so a long-lived response stream is never cut by the upload guard. The Cursor bridge instead applies the tighter configured/per-request timeout only until send headers arrive, without a body deadline on its parked run. Model discovery ignores the per-request timeout and uses models_discovery_timeout for the whole discovery operation. Buffered whole-document reads (translated non-streaming responses and discovery pages) use whole_document_read_timeout when the send has no deadline of its own and whole_document_read_max as the ceiling even against a longer X-Proxy-Timeout-Ms. A shorter caller deadline still wins. Configuration validation keeps models_discovery_timeout below the ceiling, so its configured budget is untouched. Provider Retry-After/reset hints are a floor on the wait, not a replacement for adaptive exponential backoff. A long hint is not clamped by the ordinary adaptive cap and can therefore outlast it. A short hint does not reset the doubling, so a 1s "retry shortly" cannot re-hammer a struggling upstream.

A finalized request retains its absorbed attempts and final outcome in one record. Every retried or final attempt stores the same upstream-response information: the provider request id, server, processing time, echoed model, rate-limit state and the redacted response headers. A provider-side request id therefore matches to the exact upstream attempt that produced it, including failures the client never saw. Transport failures have no HTTP response, so their attempt entries leave those fields empty. An in-band 429 is the one exception: when a keepalive already committed the client's success 200, the upstream status cannot replace it, so the record keeps {status_code:429,at,in_band:true} to preserve the distinct rate-limit signal. It carries no retry or error detail, is not counted as a retry, and is excluded from retry disclosures and error enumerations. Recovered upstream failures can count as errors even when the final status succeeds. A final, retried or in-band 429 is a separate affected-request signal. A client that disconnects before a final outcome is recorded as status 499 with client_disconnected; a plain 499 is a flow event, but one carrying a structured error (the provider's in-band stream failure the client aborted around) counts as the upstream error it observed. See metric interpretation for duration, first-token latency and throughput.

Error storm protection

Enable storm_enabled in Settings to add shared provider/model protection to existing admission and retry pacing. Requests wait in the proxy with the caller connected; this is not a client SDK queue. The proxy keeps the existing bounded request bytes and credentials only for the request lifetime. Queues are in memory, cannot survive a restart, and stop waiting when the caller cancels.

Each exact recorded model name within its resolved provider has a rolling storm_window. A model qualifies when its sampled attempts meet storm_min_samples and its selected-failure percentage reaches storm_error_percent. Provider-wide protection requires every active model to qualify, as well as the aggregate sample/rate threshold. Only models receiving requests within that same window are active; an idle model or an unused model in a provider catalog cannot affect that decision. This scope combines credentials and clients. Dashboard model-display rewrites do not merge storm scopes.

The detector samples successful upstream attempts and selected failures, counting absorbed retries separately. Incident details also count distinct logical requests with a selected failure in the window; retries move the same request's last-failure membership instead of adding another request. Unselected statuses, non-retryable HTTP account/auth rejections, recognized durable quota envelopes, caller cancellations and fired per-send deadlines are excluded - quota outcomes never become storm evidence, even while a quota gate is open. A selected 429 is a storm failure sample but remains a separate rate-limit signal in dashboard request health. Configurable response/stream failure detection observes a 2xx attempt once when its relay completes, or before an existing quality re-ask. It never adds a replay after meaningful output or native tool execution.

An open scope holds new sends until its cooldown expires, then admits one shared probe. Failed probes multiply the cooldown, with configurable jitter and a cap; valid larger upstream retry hints remain effective. Recovery needs the configured number of consecutive successful probes. The existing provider-plus-key queues, operator holds and provider Limits still apply. The HTTP relay checks storm readiness before acquiring key concurrency, then claims a probe only at the actual send boundary. Readiness never reserves a probe while waiting for a key slot. Native parked runs remain resumable; fresh Cursor HTTP handshakes use the shared native send owner and can retry before a run starts.

The quota pause (quota_pause_mode: retry) is a separate provider-wide gate in the same scheduler: the first durable quota/billing 429 from an OpenAI-wire provider opens it immediately (no sample thresholds), the triggering request absorbs the 429 and re-sends per the same recovery cooldowns, and every new request to the provider parks at admission. The gate closes after quota_pause_recovery_successes consecutive successful probes (the default 1 closes on the first 2xx) and the queue drains; a provider Retry-After on the quota 429 is a floor on the next probe, bounded by the shared hint ceiling. Parked sends obey storm_max_queue and storm_max_wait; beyond them the proxy answers HTTP 429 locally. The gate appears in the storm banner as a quota incident with an operator Resume now action; a manual-mode hold appears on the pause surface instead. The gate is in-memory: a restart drops it and the provider's next durable quota 429 re-arms it. Manual-mode holds persist like every operator pause.

The ordinary HTTP retry budget remains max_retries. Selected failures during an active storm may use up to storm_max_retries additional attempts across the logical request, including quality re-asks. Cursor retains its single-handshake behavior when protection is off and uses only the additional storm budget for failed fresh handshakes. Exhaustion returns the final upstream failure. Retrying inference can repeat upstream execution or billing even when no output arrived; this feature does not provide exactly-once processing. See the HTTP retry constraints.

storm_max_queue bounds callers waiting at storm gates across the process; ordinary scheduler queues have their own limits. storm_max_wait bounds each storm wait, not total request wall time. Scope capacity and oversized scope identifiers also fail closed. Local storm admission rejection returns HTTP 429 with queue_retry_after; a streaming connection already committed by keepalive comments receives an in-band error instead. Successful retries remain one finalized request record with its absorbed attempts and accumulated queue wait.

Model discovery

GET /v1/models and GET /models are constructed discovery services, not ordinary inference records. The proxy normalizes recognized model lists, translates native formats, completes supported pagination, and optionally enriches missing fields. Unrecognized successful list bodies remain verbatim; primary discovery budget/errors fail explicitly, while optional enrichment can fall back to the complete base list. Discovery can apply limit headers without consuming an inference slot. Its time/byte/page limits are configuration-owned.

Primary and enrichment entries share one 10,000-entry discovery budget; enrichment degrades to the base list when the remaining budget cannot hold its metadata source. The rendered list is bounded by models_discovery_max_bytes plus a fixed per-entry normalization allowance (512 bytes for each of the 10,000 entries) and is encoded without HTML escaping, so literal <, >, and & keep their wire size. A rendered list past that derived bound fails closed before any response bytes are written.

Discovery ignores X-Proxy-Path, X-Proxy-Query and the incoming query. HTTP discovery constructs /v1/models using the shared base join; Anthropic pagination constructs its own query. Cursor uses its native unary model service. A primary upstream non-200 response, including a redirect, becomes a 502 discovery error. Discovery authenticates with the resolved target auth header and prefix, the same construction as a relayed request, and has no protocol-specific fallback: an Anthropic endpoint that requires x-api-key must be called with X-Proxy-Auth-Header: x-api-key and an empty X-Proxy-Auth-Prefix.

The ordinary HTTP relay returns upstream redirects rather than following them. Forwarding also changes transport headers and may decompress an upstream gzip response; do not compare transport framing as a byte-identical contract.

Main and sub-conversations

Automatic grouping

An explicit X-Proxy-Session takes precedence, then a configured tracked body param (see tracked body params). Without either, the tracker partitions requests by client label and credential hash, then compares total user/assistant/tool turn counts against its open conversations. A nonexpired conversation with the greatest previous count no larger than the new count is the nearest match. A reset or idle gap can start another conversation, and the configured open-session cap evicts the oldest tracked conversation in that partition.

This is a heuristic for clients that send conversation history, not a content comparison or a universal agent identity. Parallel tasks with similar turn counts can be ambiguous. Tracker state is process-local, so use explicit session IDs when predictable continuity matters and keep the client/key namespace stable. Automatic grouping never establishes a parent/child relationship.

Tracked body params

The request body is client data, and one field can double as the conversation identity. When sub_conversations names the request's classified client, the tracked field the client already sends (exemplar: opencode's promptCacheKey) supplies that request's identity in the k: namespace: behind an explicit X-Proxy-Session, ahead of automatic grouping. Params match only top-level body fields, checked in order. The first present field decides: a present value that is not a JSON string counts as absent and the next field is checked, while a present string that cannot be decoded or fails the identity rules drops the identity for the request - no later field is consulted. A top-level field that occurs more than once is ambiguous and drops the identity the same way, leaving the request on automatic grouping. The body is relayed unchanged unless the entry sets strip, which removes the entry's configured fields from the top level of the client request body before relay or translation. It applies to any JSON body, including a Responses client body. A request translated to the Anthropic wire gains a top-level prompt_cache_key carrying the tracked value. That field is a proxy tracking extension, not an Anthropic cache control, and only upstreams that tolerate unknown top-level fields accept it. An absent or dropped identity is never a client error. A tracked identity never declares a parent: X-Proxy-Parent-Session still requires its explicit session pair, and the session wins identity. The identity rules, name grammar, caps and the settings surface live in operations.

Declaring a parent

A parent sends X-Proxy-Session: main-task. Its child sends its own X-Proxy-Session: worker-1 plus X-Proxy-Parent-Session: main-task, using the same client label and API key. Send IDs without the stored s: prefix.

When declaring a parent, both headers must occur once, contain nonempty valid UTF-8 without controls, differ from each other, and fit 512 bytes after trimming. Self-parent/invalid declarations return 400. The parent header is proxy-only; it neither edits the body nor instructs an upstream to create a child agent.

The server resolves observed relationships within (client, key_hash, conversation_id), including history outside the selected view. The explorer reports main/sub/unresolved counts and role/parent metadata. A missing parent is not observed, not an invented main. Conflicts, cycles or ambiguous reused IDs remain unresolved. “Main” means no parent declared in observed history, not independently verified orchestration.

Parent links preserve the full representable client/key namespace. Conversation filters remain exact leaves, never descendant filters. A changed credential can change the namespace. Historical parentage cannot be inferred from absent data, message similarity, or ordinary conversation-continuity fields.

search

Search the docs

Search titles, headings, and body text. Press / from anywhere.