Skip to content

Multimodal Understanding: Images, Video, and Audio with Gemini

Gemini reads images, video, and audio natively. Here's how to pass each one in — inline, by URL, or through the Files API — and which method fits which file size.

CurrentLast verified

Platforms

  • Gemini API (Python, JavaScript, Java, REST)

What the official documentation says

  • Gemini models are built to be multimodal from the ground up, supporting image processing and computer vision tasks including image captioning, classification, and visual question answering without training specialized ML models.

    Image understanding with the Gemini API

  • Beyond general multimodal capability, Gemini offers enhanced accuracy for specific tasks like object detection and segmentation through additional training.

    Image understanding with the Gemini API

  • Images can be passed to Gemini three ways — by URL (for publicly accessible images), as inline base64-encoded data, or by uploading through the Files API, which is recommended for larger files or reusing an image across multiple requests.

    Image understanding with the Gemini API

  • Gemini can process videos to describe, segment, and extract information from them, answer questions about video content, and refer to specific timestamps within a video.

    Video understanding with the Gemini API

  • Video can be supplied via the Files API (up to 20GB on the paid tier / 2GB on the free tier — recommended for files 100MB+ or videos 10+ minutes), Cloud Storage registration (2GB per file, no storage limit), inline data (under 100MB, under 1 minute), or a public YouTube URL.

    Video understanding with the Gemini API

  • The Files API is recommended for most video use cases, especially files larger than 100MB or when the same file will be reused across multiple requests; an uploaded file must reach an ACTIVE processing state before it can be used in a request.

    Video understanding with the Gemini API

  • Gemini can analyze audio input and generate text responses, including producing a transcript and summary of an audio clip.

    Audio understanding with the Gemini API

  • Audio, like images and video, is uploaded through the Files API and then referenced in an interaction by its returned URI and MIME type.

    Audio understanding with the Gemini API

One model, three kinds of input

Gemini doesn't need a separate vision model or a separate audio model bolted on — it's multimodal from the ground up. The same interactions.create call that handles text also handles images, video, and audio; what changes is what you put in the input array, not which model you call.

Images: three ways in

  • By URL — for images that are already publicly accessible.
  • Inline base64 data — for small, one-off images.
  • The Files API — the recommended path for larger files, or any image you'll reference across more than one request.
from google import genai

client = genai.Client()

uploaded_file = client.files.upload(file="path/to/organ.jpg")

interaction = client.interactions.create(
    model="gemini-3.8-flash",
    input=[
        {"type": "text", "text": "Caption this image."},
        {
            "type": "image",
            "uri": uploaded_file.uri,
            "mime_type": uploaded_file.mime_type
        }
    ]
)
print(interaction.output_text)

Beyond captioning and visual question answering, Gemini has been given additional training specifically for object detection and segmentation — if your use case is "where exactly is this in the image," not just "what's in the image," those are worth reaching for by name.

Video: pick the input method by size, not habit

| Method | Max size | Best for | |---|---|---| | Files API | 20GB (paid) / 2GB (free) | Large files (100MB+), long videos (10min+), reused files | | Cloud Storage registration | 2GB per file, no storage cap | Large, persistent, reused files | | Inline data | Under 100MB | Small, short (under 1 min), one-off inputs | | YouTube URL | N/A | Public YouTube videos |

The Files API is the right default for most real video use cases. One detail that trips people up: an uploaded file isn't immediately ready. Poll until it reports ACTIVE before referencing it in an interaction:

myfile = client.files.upload(file="path/to/sample.mp4")

while not myfile.state or myfile.state.name != "ACTIVE":
    print("Processing video...")
    time.sleep(5)
    myfile = client.files.get(name=myfile.name)

interaction = client.interactions.create(
    model="gemini-3.8-flash",
    input=[
        {"type": "video", "uri": myfile.uri, "mime_type": myfile.mime_type},
        {"type": "text", "text": "Summarize this video. Then create a quiz with an answer key based on the information in this video."}
    ]
)

Gemini can describe, segment, and extract information from a video; answer questions about its content; and refer to specific timestamps — not just summarize it as one undifferentiated blob.

Audio: upload, then ask

Audio follows the same Files API pattern as video — upload, get back a URI and MIME type, reference both in the interaction:

uploaded_file = client.files.upload(file="path/to/sample.mp3")

interaction = client.interactions.create(
    model="gemini-3.8-flash",
    input=[
        {"type": "text", "text": "Describe this audio clip"},
        {
            "type": "audio",
            "uri": uploaded_file.uri,
            "mime_type": uploaded_file.mime_type
        }
    ]
)

A transcript-and-summary request works the same way — the audio doesn't need any special preprocessing beyond the upload.

Choosing the right input method

The three input methods aren't interchangeable defaults — they're sized for different jobs. A quick one-off question about a small screenshot doesn't need the Files API's upload-and-poll dance; a 10-minute video you'll reference in five different prompts shouldn't be inlined as base64. Match the method to the file, and the free-tier vs. paid-tier size limits, before you write the integration, not after a request fails.

For combining multimodal input with function calling or grounding — asking Gemini to look something up because of what it saw in an image, for instance — see Function calling and tools.

How to do it

  1. Pick an input method by file size and reuse — URL or inline base64 for small one-off images, the Files API for anything larger or reused across requests.
  2. For video or audio, upload through client.files.upload and poll until the file's state reaches ACTIVE.
  3. Reference the uploaded file's uri and mime_type in the input array alongside your text prompt.
  4. For images specifically, name object detection or segmentation directly in the prompt if you need a location, not just a general description.
  5. Call client.interactions.create with the combined input and read the answer from interaction.output_text.

On Windows

N/AMultimodal input handling is identical across operating systems — it's a property of the API, not the client OS.

On mobile

N/AThis tutorial covers passing image, video, and audio files to the API from application code, not a mobile app UI.

Use cases

  • Captioning or classifying a batch of images without training a dedicated computer vision model first.
  • Asking Gemini to summarize a long video and generate a quiz with an answer key based on what's actually shown.
  • Getting a transcript and summary of a recorded meeting or interview from an audio file.

Common mistakes

  • Inlining a large image, audio, or video file as base64 data instead of uploading it through the Files API. Inline data is documented for small files only — under 100MB for video, for example — and reused files cost less through the Files API.
  • Assuming a video is immediately usable right after upload. The Files API returns a file resource that must reach an ACTIVE state before it can be referenced in an interaction; using it too early fails.
  • Not checking the free-tier File API size limit (2GB) versus the paid tier (20GB) before assuming a large file will simply work.
  • Forgetting a public YouTube URL is a distinct, separate input method from an uploaded video file, with no size limit documented because nothing is actually uploaded.

FAQ

Do I need a different model or a special setup for images versus text?
No — Gemini is multimodal from the ground up. The same model that handles text also processes images, video, and audio; you just change what you put in the input array of the interaction.
What's the best way to send a small icon or screenshot for a quick one-off question?
Inline base64-encoded data is documented for exactly this — small files, one-off inputs. For anything you'll reuse or anything large, the Files API is the recommended path instead.
Can Gemini answer questions about a specific moment in a video, not just the whole thing?
Yes — the documentation specifically calls out the ability to refer to specific timestamps within a video, in addition to describing, segmenting, and extracting information from it generally.
How large a video can I actually send?
It depends on the method. Inline data tops out under 100MB and under a minute. The Files API allows up to 2GB on the free tier and 20GB on the paid tier. Cloud Storage registration allows 2GB per file with no overall storage limit. A public YouTube URL has no documented size limit since the video itself isn't uploaded.
Can Gemini do more than describe an image — like find where something is?
Yes. Beyond general captioning and visual question answering, Gemini has enhanced accuracy for object detection and segmentation specifically, through additional training beyond its general multimodal capability.

Official sources

These are the pages this tutorial is checked against. Follow them if you need the vendor's exact wording.

Source status