实时视频理解

连接与鉴权

使用 HTTP GET 升级为 WebSocket。连接地址为 wss://api.mosi.cn/v1/realtime，必须通过 model 查询参数指定模型；API Key 只能放在 Authorization 请求头中。

字段类型要求说明

model: string必填

模型 ID。将稳定 ID 或快照 ID 放在 WebSocket 连接地址的 model 查询参数中

Authorization: HTTP header必填

Bearer <API Key>；不支持通过查询参数或 WebSocket 子协议传入

会话流程

阶段	客户端	服务端与判断
建立连接	完成 WebSocket 握手	接收 session.created；连接自动创建会话
配置	只发送一次 session.configure	接收 session.configured，并等待 session.ready
发送图像	先发 input.frame JSON，再发图片二进制	等待 input.frame.ready 后交付二进制，收到 input.frame.accepted 后再发送下一条输入；画面本身不会自动生成回答
提问	单独发送 input.prompt	问题不需要重复附带图片；接收 accepted / processed，并可能收到文本、静默或打断事件
继续或结束	继续推帧，或设置 final=true / 发送 session.abort	response.done 只结束一段回答；session.done 才结束会话

客户端事件

字段类型说明

session.configure: event

连接建立后必须发送一次，且须在 session.created.configure_timeout_s 期限内完成。

input.frame: event

发送图像帧元数据；收到 input.frame.ready 后，紧接着发送完整图片二进制。

input.prompt: event

单独发送提问文本。字段名是 prompt，不是 text；问题沿用最近已接收的画面。

session.abort: event

主动结束整场会话，不占用 seq_no，也不接受其他字段。

服务端事件

字段类型说明

会话事件: server events

用于判断连接、配置、就绪与会话终态。

输入事件: server events

用于确认图像帧或提问是否已准备接收、提交和完成处理。

输出事件: server events

用于读取增量文本，并判断单段回答是否静默、被打断或正常结束。

用量事件: server events

用于读取过程用量和回答或会话的累计用量。

错误事件: server events

用于处理协议或服务异常。

协议限制

项目	当前协议值与说明
控制事件	单条默认不超过 256 KiB，与图片二进制大小分开计算
配置与交付	在服务端规定的配置期限内完成配置；收到 input.frame.ready 后 10 秒内交付二进制，超时返回 code=session_timeout 的 error 事件并结束会话
输入队列	默认 4，可配置范围 `[1, 256]`；是未处理输入额度，不是会话并发
上下文与时长	以服务端返回的 context_limit 和会话时长限制中先触发者为准

错误码

建连阶段的认证、权限、限流和通用 HTTP 错误，请参见错误码。以下仅列出实时视频理解接口专属的协议错误。

错误码	触发条件	处理建议
invalid_request	客户端事件/字段/顺序不合法或消息过大	修正请求后重发；若连接断开，重新建连
context_exhausted	会话达到上下文上限	结束当前会话并重新建立新会话
session_timeout	会话达到时长或空闲限制，或收到 input.frame.ready 后 10 秒内未交付二进制	检查会话活动和服务端限制，必要时重新连接
response_failed	模型生成或处理失败	记录错误信息并根据业务决定是否重试
session_capacity_exceeded	实时会话容量已满	降低并发或稍后重试

连接关闭

关闭码	含义	客户端处理
1000	服务端已发送 session.done 后关闭，以 error 事件收场的会话也用此关闭码	结束当前会话，不重试；成败以 session.done.reason（如 completed、aborted、error）和此前的 error 事件为准
1001	服务下线或连接离开	根据业务需要重新连接
1002	协议错误	检查事件格式和发送顺序后再重试
1009	二进制超限或控制事件过大；关闭前可能先返回 invalid_request 错误事件。	减小消息后重新建连
1011	服务端故障	稍后重试并记录错误
1013	服务暂时无法处理	降低并发，稍后重试

下一步

场景指南

查看接入指南

从建立连接到发送图像帧、提问和结束会话。

模型

查看模型概览

查看模型 ID、能力和适用场景。

请求示例

import asyncio
import json
import os
from pathlib import Path

from websockets.asyncio.client import connect
from websockets.exceptions import ConnectionClosed

URL = "wss://api.mosi.cn/v1/realtime?model=moss-vl-realtime-1.0"


async def receive_event(ws):
    try:
        message = await asyncio.wait_for(ws.recv(), timeout=30)
    except (ConnectionClosed, asyncio.TimeoutError) as error:
        raise RuntimeError("Disconnected or timed out before session.done; final state and usage unknown") from error
    if isinstance(message, bytes):
        raise RuntimeError("Unexpected binary server message")
    event = json.loads(message)
    if event["type"] in ("error", "input.frame.rejected"):
        raise RuntimeError(f"{event.get('code', event['type'])}: {event.get('message') or event.get('reason', '')}")
    if event["type"] == "response.text.delta":
        print(event["delta"], end="", flush=True)
    return event


async def wait_for(ws, event_type, seq_no=None):
    while True:
        event = await receive_event(ws)
        if event["type"] == event_type and (seq_no is None or event.get("seq_no") == seq_no):
            return event
        if event["type"] == "session.done":
            raise RuntimeError(f"Session ended before {event_type}: {event.get('reason')}")


async def main():
    frame = Path(__file__).with_name("frame.jpg").read_bytes()
    headers = {"Authorization": f"Bearer {os.environ['MOSS_API_KEY']}"}
    async with connect(URL, additional_headers=headers) as ws:
        await wait_for(ws, "session.created")
        await ws.send(json.dumps({
            "type": "session.configure",
            "prompt": "请观察画面并回答我的问题。",
        }))
        configured = await wait_for(ws, "session.configured")
        if len(frame) > configured["max_frame_bytes"]:
            raise ValueError("frame.jpg exceeds max_frame_bytes")
        await wait_for(ws, "session.ready")
        await ws.send(json.dumps({
            "type": "input.frame", "seq_no": 0,
            "timestamp": 0.0, "mime_type": "image/jpeg",
        }))
        await wait_for(ws, "input.frame.ready", 0)
        await ws.send(frame)
        await wait_for(ws, "input.frame.accepted", 0)
        await ws.send(json.dumps({
            "type": "input.prompt", "seq_no": 1,
            "prompt": "画面里有什么？", "final": True,
        }))
        while True:
            event = await receive_event(ws)
            if event["type"] == "session.done":
                print("\nFinal usage:", event.get("usage"))
                break


if __name__ == "__main__":
    asyncio.run(main())

import { readFile } from "node:fs/promises";
import WebSocket from "ws";

interface RealtimeEvent {
  type: string;
  seq_no?: number | null;
  max_frame_bytes?: number;
  delta?: string;
  code?: string;
  message?: string;
  reason?: string;
  usage?: unknown;
}

function isEvent(value: unknown): value is RealtimeEvent {
  return typeof value === "object" && value !== null
    && "type" in value && typeof value.type === "string"
    && (!("seq_no" in value) || value.seq_no === null || typeof value.seq_no === "number")
    && (!("max_frame_bytes" in value) || typeof value.max_frame_bytes === "number")
    && (!("delta" in value) || typeof value.delta === "string")
    && (!("code" in value) || typeof value.code === "string")
    && (!("message" in value) || typeof value.message === "string")
    && (!("reason" in value) || typeof value.reason === "string");
}

const keyValue = process.env.MOSS_API_KEY;
if (!keyValue) throw new Error("Set MOSS_API_KEY in your server environment");
const frame = await readFile(new URL("./frame.jpg", import.meta.url));
const ws = new WebSocket(
  "wss://api.mosi.cn/v1/realtime?model=moss-vl-realtime-1.0",
  { headers: { Authorization: `Bearer ${keyValue}` } },
);
let completed = false;
// Client-side idle guard, not a server SLA.
let idleTimer: ReturnType<typeof setTimeout>;
function fail(error: unknown): void {
  console.error(error);
  process.exitCode = 1;
  clearTimeout(idleTimer);
  ws.terminate();
}
function resetIdleTimer(): void {
  clearTimeout(idleTimer);
  idleTimer = setTimeout(() => fail(new Error("No event for 30 seconds; final state and usage unknown")), 30_000);
}
resetIdleTimer();
ws.on("error", fail);
ws.on("close", () => {
  clearTimeout(idleTimer);
  if (!completed) {
    console.error("Closed before session.done; final state and usage unknown");
    process.exitCode = 1;
  }
});
ws.on("message", (data, isBinary) => {
  try {
    if (isBinary) throw new Error("Unexpected binary server message");
    const event: unknown = JSON.parse(data.toString());
    if (!isEvent(event)) throw new Error("Invalid server event");
    resetIdleTimer();
    switch (event.type) {
      case "session.created":
        ws.send(JSON.stringify({ type: "session.configure", prompt: "请观察画面并回答我的问题。" }));
        break;
      case "session.configured":
        if (event.max_frame_bytes === undefined || frame.length > event.max_frame_bytes) {
          throw new Error("frame.jpg exceeds max_frame_bytes or limit is missing");
        }
        break;
      case "session.ready":
        ws.send(JSON.stringify({ type: "input.frame", seq_no: 0, timestamp: 0, mime_type: "image/jpeg" }));
        break;
      case "input.frame.ready":
        if (event.seq_no === 0) ws.send(frame);
        break;
      case "input.frame.accepted":
        if (event.seq_no === 0) ws.send(JSON.stringify({ type: "input.prompt", seq_no: 1, prompt: "画面里有什么？", final: true }));
        break;
      case "response.text.delta":
        process.stdout.write(event.delta ?? "");
        break;
      case "session.done":
        completed = true;
        clearTimeout(idleTimer);
        console.log("\nFinal usage:", event.usage);
        ws.close();
        break;
      case "error":
      case "input.frame.rejected":
        throw new Error(`${event.code ?? event.type}: ${event.message ?? event.reason ?? ""}`);
    }
  } catch (error) {
    fail(error);
  }
});

响应示例

{"type":"session.created","session_id":"session_xxx","task_id":"task_xxx","model":"moss-vl-realtime-1.0-2026-09-09","turn_id":0,"configure_timeout_s":180,"protocol_version":"vl-api-v2","capabilities":["session.usage","response.done.usage","session.done.usage","error.seq_no","input.frame.rejected","response.done.per_response","response.id","usage.text_input_output"]}
{"type":"session.configured","session_id":"video_sess_xxx","request_id":"video_req_xxx","max_frame_bytes":8388608,"input_queue_capacity":4,"max_tokens_per_turn":86400,"parked_request_timeout_s":3600,"context_limit":131072}
{"type":"session.ready","session_id":"video_sess_xxx","request_id":"video_req_xxx","turn_id":0}
{"type":"response.turn.silence","turn_id":0,"seq_no":null,"timestamp":null,"silence_seq":0}
{"type":"input.frame.ready","seq_no":0}
{"type":"input.frame.accepted","seq_no":0,"timestamp":0.0,"final":false,"pending_events":1,"interrupts_current_turn":false}
{"type":"input.frame.processed","seq_no":0,"timestamp":0.0,"final":false,"pending_events":0}
{"type":"input.prompt.accepted","seq_no":1,"final":true,"pending_events":1,"interrupts_current_turn":true}
{"type":"response.turn.interrupted","turn_id":0,"next_turn_id":1,"seq_no":1}
{"type":"input.prompt.processed","seq_no":1,"timestamp":0.0,"final":true,"pending_events":0,"interrupted_turn_id":0,"turn_id":1}
{"type":"response.text.delta","delta":"画面中有一辆车。","turn_id":1,"response_id":"resp_xxx","response_seq":1}
{"type":"response.done","response_id":"resp_xxx","response_seq":1,"turn_id":1,"finish_reason":"stop","boundary":"silence","usage":{"vision_tokens":120,"text_input_tokens":9,"text_output_tokens":8,"text_tokens":17,"total_tokens":137}}
{"type":"session.done","reason":"completed","usage":{"vision_tokens":120,"text_input_tokens":9,"text_output_tokens":8,"text_tokens":17,"total_tokens":137}}

// 其余事件同默认响应，仅展示新增用量事件。
{
  "type": "session.usage",
  "decoder_tokens": 77,
  "encoder_tokens": 0,
  "encoder_kv_tokens": 0,
  "token_space_used": 77,
  "context_limit": 131072,
  "context_remaining": 130995
}