Streaming

Request

Requires an account with paid inference access.

{
  "model": "MODEL_ID",
  "messages": [
    {
      "role": "user",
      "content": "Hello"
    }
  ],
  "stream": true,
  "stream_options": {
    "include_usage": true
  },
  "max_tokens": 256
}

Event shape

The stream uses SSE frames separated by blank lines. Read JSON from data: lines and concatenate choices[].delta.content by choice index. A network chunk can contain part of an event or several events; do not parse each network chunk as JSON.

data: {"object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: [DONE]

Tools, usage and completion

Tool argument strings arrive in fragments; assemble them by choice index and tool-call index before parsing JSON. Usage can arrive with an empty choices array. Treat [DONE] as the completion marker; a closed connection without it is incomplete. Do not count usage frames twice.

Interrupted streams

The relay emits error.code: "stream_interrupted" when it cannot complete the stream. Preserve the request ID and check accounting before resubmitting. Disconnecting the client is not a cancellation API and does not prove the request was free.

• Errors and retries
• Request accounting