Realtime video understanding

Connection and authentication

Upgrade an HTTP GET request to WebSocket. Connect to wss://api.mosi.cn/v1/realtime with the model query parameter specifying the model; place the API Key only in the Authorization request header.

FieldTypeRequiredDescription

model: stringRequired

Model ID. Pass a stable or snapshot ID in the WebSocket connection's model query parameter

Authorization: HTTP headerRequired

Bearer <API Key>; query parameters and WebSocket subprotocols are not supported for authentication

Session flow

Stage	Client	Server events and checks
Connect	Complete the WebSocket handshake	Receive session.created; the connection automatically creates a session
Configure	Send session.configure exactly once	Receive session.configured and wait for session.ready
Send an image	Send input.frame JSON followed by image binary data	Wait for input.frame.ready before sending binary data; wait for input.frame.accepted before sending the next input. Frames alone do not automatically generate answers
Ask a question	Send input.prompt separately	No need to attach the image again; receive accepted / processed, and possibly text, silence, or interruption events
Continue or finish	Continue sending frames, set final=true, or send session.abort	response.done ends one answer segment; session.done ends the session

Client events

FieldTypeDescription

session.configure: event

Send exactly once after connecting, within session.created.configure_timeout_s.

input.frame: event

Send image frame metadata; immediately send the complete image binary data after receiving input.frame.ready.

input.prompt: event

Send a question separately. The field is prompt, not text; the question uses the most recently received scene.

session.abort: event

End the entire session. This event does not consume seq_no and accepts no other fields.

Server events

FieldTypeDescription

Session events: server events

Determine connection, configuration, readiness, and the final session state.

Input events: server events

Determine whether frames or questions are ready to receive, submitted, or processed.

Output events: server events

Read incremental text and determine whether an answer segment is silent, interrupted, or complete.

Usage events: server events

Read observational usage and cumulative usage for an answer or session.

Error events: server events

Handle protocol or service failures.

Protocol limits

Item	Current protocol value and description
Control events	Each event defaults to at most 256 KiB; calculated separately from image binary size
Configuration and delivery	Configure within the server deadline; deliver binary data within 10 seconds after input.frame.ready, or the server sends an error event with code=session_timeout and ends the session
Input queue	Defaults to 4, configurable range `[1, 256]`; unprocessed input capacity, not concurrent sessions
Context and duration	Whichever is reached first: the returned context_limit or the session duration limit

Error codes

For authentication, permissions, rate limiting, and general HTTP errors during connection, see Error Codes. The following are protocol errors specific to realtime video understanding.

Error codes	Trigger	Action
invalid_request	Invalid client event/fields/order or oversized message	Correct and resend; reconnect if disconnected
context_exhausted	Session context limit reached	End the session and create a new one
session_timeout	Session duration or idle limit reached, or binary data not delivered within 10 seconds after input.frame.ready	Check session activity and server limits; reconnect if needed
response_failed	Model generation or processing failed	Record the error and decide whether to retry for your use case
session_capacity_exceeded	Realtime session capacity exhausted	Reduce concurrency or retry later

Connection closure

Close code	Meaning	Client action
1000	Closed after the server sent session.done; sessions that ended with an error event also use this code	End the session without retrying; judge success by session.done.reason (such as completed, aborted, error) and any preceding error event
1001	Service shutdown or connection leaving	Reconnect as required by your use case
1002	Protocol error	Check event format and send order before retrying
1009	Binary data exceeds limits or control event is too large; an invalid_request error event may precede closure.	Reduce message size and reconnect
1011	Server failure	Retry later and record the error
1013	Service temporarily unable to process	Reduce concurrency and retry later

Next steps

Scenario guide

View the integration guide

Connect, send image frames, ask questions, and end the session.

Models

View model overview

View model IDs, capabilities, and use cases.

Request example

import asyncio
import json
import os
from pathlib import Path

from websockets.asyncio.client import connect
from websockets.exceptions import ConnectionClosed

URL = "wss://api.mosi.cn/v1/realtime?model=moss-vl-realtime-1.0"


async def receive_event(ws):
    try:
        message = await asyncio.wait_for(ws.recv(), timeout=30)
    except (ConnectionClosed, asyncio.TimeoutError) as error:
        raise RuntimeError("Disconnected or timed out before session.done; final state and usage unknown") from error
    if isinstance(message, bytes):
        raise RuntimeError("Unexpected binary server message")
    event = json.loads(message)
    if event["type"] in ("error", "input.frame.rejected"):
        raise RuntimeError(f"{event.get('code', event['type'])}: {event.get('message') or event.get('reason', '')}")
    if event["type"] == "response.text.delta":
        print(event["delta"], end="", flush=True)
    return event


async def wait_for(ws, event_type, seq_no=None):
    while True:
        event = await receive_event(ws)
        if event["type"] == event_type and (seq_no is None or event.get("seq_no") == seq_no):
            return event
        if event["type"] == "session.done":
            raise RuntimeError(f"Session ended before {event_type}: {event.get('reason')}")


async def main():
    frame = Path(__file__).with_name("frame.jpg").read_bytes()
    headers = {"Authorization": f"Bearer {os.environ['MOSS_API_KEY']}"}
    async with connect(URL, additional_headers=headers) as ws:
        await wait_for(ws, "session.created")
        await ws.send(json.dumps({
            "type": "session.configure",
            "prompt": "Observe the scene and answer my questions.",
        }))
        configured = await wait_for(ws, "session.configured")
        if len(frame) > configured["max_frame_bytes"]:
            raise ValueError("frame.jpg exceeds max_frame_bytes")
        await wait_for(ws, "session.ready")
        await ws.send(json.dumps({
            "type": "input.frame", "seq_no": 0,
            "timestamp": 0.0, "mime_type": "image/jpeg",
        }))
        await wait_for(ws, "input.frame.ready", 0)
        await ws.send(frame)
        await wait_for(ws, "input.frame.accepted", 0)
        await ws.send(json.dumps({
            "type": "input.prompt", "seq_no": 1,
            "prompt": "What is in the scene?", "final": True,
        }))
        while True:
            event = await receive_event(ws)
            if event["type"] == "session.done":
                print("\nFinal usage:", event.get("usage"))
                break


if __name__ == "__main__":
    asyncio.run(main())

import { readFile } from "node:fs/promises";
import WebSocket from "ws";

interface RealtimeEvent {
  type: string;
  seq_no?: number | null;
  max_frame_bytes?: number;
  delta?: string;
  code?: string;
  message?: string;
  reason?: string;
  usage?: unknown;
}

function isEvent(value: unknown): value is RealtimeEvent {
  return typeof value === "object" && value !== null
    && "type" in value && typeof value.type === "string"
    && (!("seq_no" in value) || value.seq_no === null || typeof value.seq_no === "number")
    && (!("max_frame_bytes" in value) || typeof value.max_frame_bytes === "number")
    && (!("delta" in value) || typeof value.delta === "string")
    && (!("code" in value) || typeof value.code === "string")
    && (!("message" in value) || typeof value.message === "string")
    && (!("reason" in value) || typeof value.reason === "string");
}

const keyValue = process.env.MOSS_API_KEY;
if (!keyValue) throw new Error("Set MOSS_API_KEY in your server environment");
const frame = await readFile(new URL("./frame.jpg", import.meta.url));
const ws = new WebSocket(
  "wss://api.mosi.cn/v1/realtime?model=moss-vl-realtime-1.0",
  { headers: { Authorization: `Bearer ${keyValue}` } },
);
let completed = false;
// Client-side idle guard, not a server SLA.
let idleTimer: ReturnType<typeof setTimeout>;
function fail(error: unknown): void {
  console.error(error);
  process.exitCode = 1;
  clearTimeout(idleTimer);
  ws.terminate();
}
function resetIdleTimer(): void {
  clearTimeout(idleTimer);
  idleTimer = setTimeout(() => fail(new Error("No event for 30 seconds; final state and usage unknown")), 30_000);
}
resetIdleTimer();
ws.on("error", fail);
ws.on("close", () => {
  clearTimeout(idleTimer);
  if (!completed) {
    console.error("Closed before session.done; final state and usage unknown");
    process.exitCode = 1;
  }
});
ws.on("message", (data, isBinary) => {
  try {
    if (isBinary) throw new Error("Unexpected binary server message");
    const event: unknown = JSON.parse(data.toString());
    if (!isEvent(event)) throw new Error("Invalid server event");
    resetIdleTimer();
    switch (event.type) {
      case "session.created":
        ws.send(JSON.stringify({ type: "session.configure", prompt: "Observe the scene and answer my questions." }));
        break;
      case "session.configured":
        if (event.max_frame_bytes === undefined || frame.length > event.max_frame_bytes) {
          throw new Error("frame.jpg exceeds max_frame_bytes or limit is missing");
        }
        break;
      case "session.ready":
        ws.send(JSON.stringify({ type: "input.frame", seq_no: 0, timestamp: 0, mime_type: "image/jpeg" }));
        break;
      case "input.frame.ready":
        if (event.seq_no === 0) ws.send(frame);
        break;
      case "input.frame.accepted":
        if (event.seq_no === 0) ws.send(JSON.stringify({ type: "input.prompt", seq_no: 1, prompt: "What is in the scene?", final: true }));
        break;
      case "response.text.delta":
        process.stdout.write(event.delta ?? "");
        break;
      case "session.done":
        completed = true;
        clearTimeout(idleTimer);
        console.log("\nFinal usage:", event.usage);
        ws.close();
        break;
      case "error":
      case "input.frame.rejected":
        throw new Error(`${event.code ?? event.type}: ${event.message ?? event.reason ?? ""}`);
    }
  } catch (error) {
    fail(error);
  }
});

Response example

{"type":"session.created","session_id":"session_xxx","task_id":"task_xxx","model":"moss-vl-realtime-1.0-2026-09-09","turn_id":0,"configure_timeout_s":180,"protocol_version":"vl-api-v2","capabilities":["session.usage","response.done.usage","session.done.usage","error.seq_no","input.frame.rejected","response.done.per_response","response.id","usage.text_input_output"]}
{"type":"session.configured","session_id":"video_sess_xxx","request_id":"video_req_xxx","max_frame_bytes":8388608,"input_queue_capacity":4,"max_tokens_per_turn":86400,"parked_request_timeout_s":3600,"context_limit":131072}
{"type":"session.ready","session_id":"video_sess_xxx","request_id":"video_req_xxx","turn_id":0}
{"type":"response.turn.silence","turn_id":0,"seq_no":null,"timestamp":null,"silence_seq":0}
{"type":"input.frame.ready","seq_no":0}
{"type":"input.frame.accepted","seq_no":0,"timestamp":0.0,"final":false,"pending_events":1,"interrupts_current_turn":false}
{"type":"input.frame.processed","seq_no":0,"timestamp":0.0,"final":false,"pending_events":0}
{"type":"input.prompt.accepted","seq_no":1,"final":true,"pending_events":1,"interrupts_current_turn":true}
{"type":"response.turn.interrupted","turn_id":0,"next_turn_id":1,"seq_no":1}
{"type":"input.prompt.processed","seq_no":1,"timestamp":0.0,"final":true,"pending_events":0,"interrupted_turn_id":0,"turn_id":1}
{"type":"response.text.delta","delta":"There is a car in the scene.","turn_id":1,"response_id":"resp_xxx","response_seq":1}
{"type":"response.done","response_id":"resp_xxx","response_seq":1,"turn_id":1,"finish_reason":"stop","boundary":"silence","usage":{"vision_tokens":120,"text_input_tokens":9,"text_output_tokens":8,"text_tokens":17,"total_tokens":137}}
{"type":"session.done","reason":"completed","usage":{"vision_tokens":120,"text_input_tokens":9,"text_output_tokens":8,"text_tokens":17,"total_tokens":137}}

// Other events match the default response.
{
  "type": "session.usage",
  "decoder_tokens": 77,
  "encoder_tokens": 0,
  "encoder_kv_tokens": 0,
  "token_space_used": 77,
  "context_limit": 131072,
  "context_remaining": 130995
}