Image and Video Understanding

Use MOSS-VL to understand images or video and return text in a Responses API object. Every request must include both a text instruction and media.

Scenario information

Use cases

• Understanding the content of a single image
• Comparing and summarizing multiple images
• Image OCR and document content understanding
• Video summarization, event understanding, and question answering

Call steps

1. Create an API Key

   Open the Developer Center, create an API Key, and save it as an environment variable. Do not write it into client-side code or commit it to a repository.

   MOSS_API_KEY="<your_api_key_here>"
   

2. Prepare the request input

   Prepare a text instruction and select the file_id for the image or video. For a local image or video, first use the Upload File endpoint to obtain a file_id. Do not mix images and video in the same request.

   {
     "model": "moss-vl-1.0",
     "input": [
       {
         "role": "user",
         "content": [
           { "type": "input_text", "text": "Describe the main content of this image." },
           { "type": "input_image", "file_id": "<image_file_id>" }
         ]
       }
     ],
     "max_output_tokens": 1024
   }
   

   {
     "model": "moss-vl-1.0",
     "input": [
       {
         "role": "user",
         "content": [
           { "type": "input_text", "text": "Summarize the events in this video in chronological order." },
           { "type": "input_video", "file_id": "<video_file_id>" }
         ]
       }
     ],
     "max_output_tokens": 1024
   }
   

3. Send your first request

   Send a JSON request to /v1/responses. The endpoint synchronously returns a Responses API object with text output.

   If you use the image understanding example, save the image request body from step 2 as request.json before sending the request.

   curl https://api.mosi.cn/v1/responses \
     -H "Authorization: Bearer $MOSS_API_KEY" \
     -H "Content-Type: application/json" \
     -d @request.json
   

   import json
   import os
   import requests
   
   payload = json.load(open("request.json"))
   resp = requests.post(
       "https://api.mosi.cn/v1/responses",
       headers={"Authorization": f"Bearer {os.environ['MOSS_API_KEY']}"},
       json=payload,
   )
   print(resp.status_code)
   print(resp.json())
   

   const payload = {
     model: 'moss-vl-1.0',
     input: [
       {
         role: 'user',
         content: [
           { type: 'input_text', text: 'Describe the main content of this image.' },
           { type: 'input_image', file_id: '<image_file_id>' },
         ],
       },
     ],
     max_output_tokens: 1024,
   }
   
   const resp = await fetch('https://api.mosi.cn/v1/responses', {
     method: 'POST',
     headers: {
       Authorization: `Bearer ${apiKey}`,
       'Content-Type': 'application/json',
     },
     body: JSON.stringify(payload),
   })
   
   console.log(resp.status)
   console.log(await resp.json())
   

4. Read the response

   After the request completes, read output[].content[].text. If the response returns status=incomplete, the output may have been truncated by max_output_tokens.

   {
     "status": "completed",
     "output": [
       {
         "type": "message",
         "role": "assistant",
         "content": [
           {
             "type": "output_text",
             "text": "There are four nuts in the image."
           }
         ]
       }
     ]
   }
   

5. Verify success

   Success means the response status is completed and output[].content[].text contains text generated by the model.

Notes

• A single request currently accepts up to 5 images or 1 video.
• Images and video cannot currently be mixed in the same request.
• The main examples in this scenario guide use file_id because it provides a more stable request flow.
• You can also use image_url / video_url to provide a public or object storage URL, but the URL must be accessible to the service. If you are unsure whether the URL is stable, upload the file first and use its file_id.

Next steps