Multimodal

Work with images, video, and audio in your AI applications

Multimodal

Genkit supports multimodal AI capabilities, allowing you to generate and process images, videos, and audio alongside text. Build applications that understand and create visual content.

Image Generation

Generate images from text descriptions:

Go

import (
    "github.com/firebase/genkit/go/ai"
    "github.com/firebase/genkit/go/genkit"
    "github.com/firebase/genkit/go/plugins/googlegenai"
    "google.golang.org/genai"
)

func main() {
    ctx := context.Background()
    g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.VertexAI{}))

    genkit.DefineFlow(g, "image-generation", 
        func(ctx context.Context, input string) ([]string, error) {
            r, err := genkit.Generate(ctx, g,
                ai.WithModelName("vertexai/imagen-3.0-generate-001"),
                ai.WithPrompt("Generate an image of %s", input),
                ai.WithConfig(&genai.GenerateImagesConfig{
                    NumberOfImages:    2,
                    NegativePrompt:    "night",
                    AspectRatio:       "9:16",
                    SafetyFilterLevel: genai.SafetyFilterLevelBlockLowAndAbove,
                    PersonGeneration:  genai.PersonGenerationAllowAll,
                    Language:          genai.ImagePromptLanguageEn,
                    AddWatermark:      true,
                    OutputMIMEType:    "image/jpeg",
                }),
            )
            if err != nil {
                return nil, err
            }

            var images []string
            for _, m := range r.Message.Content {
                images = append(images, m.Text)
            }
            return images, nil
        })
}

Image Understanding

Analyze and describe images:

TypeScript

import { Document } from 'genkit';

const { text } = await ai.generate({
  model: googleAI.model('gemini-2.5-flash'),
  prompt: 'Describe what you see in this image',
  media: [
    {
      url: 'https://example.com/image.jpg',
      contentType: 'image/jpeg',
    },
  ],
});

console.log(text);

Video Understanding

Process and analyze video content:

TypeScript

import { Document } from 'genkit';

// Index video for RAG
const videoDocs = [
  Document.fromMedia(
    'gs://cloud-samples-data/generative-ai/video/pixel8.mp4',
    'video/mp4',
    {
      videoSegmentConfig: {
        startOffsetSec: 0,
        endOffsetSec: 120,
        intervalSec: 15,
      },
    }
  ),
];

await ai.index({
  indexer: videoIndexer,
  documents: videoDocs,
});

// Query video content
const videoQA = ai.defineFlow(
  { name: 'videoQuestions', inputSchema: z.string() },
  async (query) => {
    const docs = await ai.retrieve({
      retriever: videoRetriever,
      query,
      options: { k: 1 },
    });

    const { text } = await ai.generate({
      model: googleAI.model('gemini-2.5-flash'),
      prompt: query,
      media: docs
        .filter(d => d.media[0]?.url)
        .map(d => ({
          url: d.media[0].url,
          contentType: d.media[0].contentType,
          startOffsetSec: d.metadata?.embedMetadata?.startOffsetSec,
          endOffsetSec: d.metadata?.embedMetadata?.endOffsetSec,
        }))[0],
    });
    
    return text;
  }
);

PDF Processing

Extract and analyze content from PDFs:

TypeScript

import { Document } from 'genkit';
import fs from 'fs';

// Read PDF file
const pdfBuffer = fs.readFileSync('document.pdf');
const pdfBase64 = pdfBuffer.toString('base64');

const { text } = await ai.generate({
  model: googleAI.model('gemini-2.5-flash'),
  prompt: 'Summarize the key points from this document',
  media: [
    {
      url: `data:application/pdf;base64,${pdfBase64}`,
      contentType: 'application/pdf',
    },
  ],
});

console.log(text);

Multimodal Embeddings

Create embeddings for images and videos for similarity search:

TypeScript

import { vertexAI } from '@genkit-ai/vertexai';

const ai = genkit({ plugins: [vertexAI()] });

// Embed an image
const embedding = await ai.embed({
  embedder: vertexAI.embedder('multimodalembedding@001'),
  content: Document.fromMedia(
    'gs://my-bucket/image.jpg',
    'image/jpeg'
  ),
});

console.log(embedding.values);

Video Segment Configuration

Control how videos are processed:

TypeScript

const videoDoc = Document.fromMedia(
  'gs://cloud-samples-data/video/google_sustainability.mp4',
  'video/mp4',
  {
    videoSegmentConfig: {
      startOffsetSec: 0,      // Start at beginning
      endOffsetSec: 120,       // Process first 2 minutes
      intervalSec: 15,         // Create embeddings every 15 seconds
    },
  }
);

// For longer videos, process in multiple segments
const segment1 = Document.fromMedia(
  videoUrl,
  'video/mp4',
  {
    videoSegmentConfig: {
      startOffsetSec: 0,
      endOffsetSec: 120,
      intervalSec: 15,
    },
  }
);

const segment2 = Document.fromMedia(
  videoUrl,
  'video/mp4',
  {
    videoSegmentConfig: {
      startOffsetSec: 120,
      endOffsetSec: 240,
      intervalSec: 15,
    },
  }
);

Mixed Media Inputs

Combine multiple types of media in a single request:

TypeScript

const { text } = await ai.generate({
  model: googleAI.model('gemini-2.5-flash'),
  prompt: 'Compare these images and describe the differences',
  media: [
    { url: 'https://example.com/image1.jpg', contentType: 'image/jpeg' },
    { url: 'https://example.com/image2.jpg', contentType: 'image/jpeg' },
  ],
});

Provider Support

Multimodal capabilities vary by provider:

Provider	Image Input	Image Generation	Video Input	PDF Input
Google AI (Gemini)	✅	✅	✅	✅
Vertex AI	✅	✅ (Imagen)	✅	✅
Anthropic (Claude)	✅	❌	❌	✅
OpenAI	✅	✅ (DALL-E)	❌	❌

Supported File Types

Images

• JPEG (.jpg, .jpeg)
• PNG (.png)
• WebP (.webp)
• GIF (.gif)

Video

• MP4 (.mp4)
• MOV (.mov)
• AVI (.avi)
• WebM (.webm)

Documents

• PDF (.pdf)

Media Source Options

HTTP/HTTPS URLs

TypeScript

media: [{
  url: 'https://example.com/image.jpg',
  contentType: 'image/jpeg',
}]

Google Cloud Storage URLs

TypeScript

media: [{
  url: 'gs://my-bucket/video.mp4',
  contentType: 'video/mp4',
}]

Base64 Encoded Data

TypeScript

const base64Data = fs.readFileSync('image.jpg').toString('base64');

media: [{
  url: `data:image/jpeg;base64,${base64Data}`,
  contentType: 'image/jpeg',
}]

Best Practices

Optimize Image Sizes

Resize images before sending to reduce latency and costs:

• Recommended: 1024x1024 or smaller
• Maximum: Check provider limits

Use Cloud Storage for Large Files

For videos and large PDFs, use Google Cloud Storage URLs instead of base64:

TypeScript

// Recommended for large files
url: 'gs://my-bucket/large-video.mp4'

// Avoid for large files (increases request size)
url: `data:video/mp4;base64,${hugeBase64String}`

Process Videos in Segments

For long videos, process in smaller time windows:

TypeScript

const processLongVideo = async (videoUrl: string) => {
  const segmentDuration = 120; // 2 minutes per segment
  const segments = [];
  
  for (let start = 0; start < totalDuration; start += segmentDuration) {
    segments.push(
      Document.fromMedia(videoUrl, 'video/mp4', {
        videoSegmentConfig: {
          startOffsetSec: start,
          endOffsetSec: start + segmentDuration,
          intervalSec: 15,
        },
      })
    );
  }
  
  return segments;
};

Add Specific Instructions

Provide clear context about what to analyze:

TypeScript

const { text } = await ai.generate({
  model: googleAI.model('gemini-2.5-flash'),
  prompt: 'Identify all the text visible in this image, maintaining the original layout and formatting',
  media: [{ url: imageUrl, contentType: 'image/jpeg' }],
});

Handle Multimodal Errors

Some models may reject certain content:

TypeScript

try {
  const { text } = await ai.generate({
    model: googleAI.model('gemini-2.5-flash'),
    prompt: 'Describe this image',
    media: [{ url: imageUrl, contentType: 'image/jpeg' }],
  });
} catch (error) {
  if (error.message.includes('SAFETY')) {
    console.log('Content blocked by safety filters');
  } else if (error.message.includes('UNSUPPORTED')) {
    console.log('File type not supported');
  }
}

Complete Example: Image Analysis Flow

TypeScript

import { genkit, z } from 'genkit';
import { googleAI } from '@genkit-ai/google-genai';

const ai = genkit({ plugins: [googleAI()] });

const ImageAnalysisSchema = z.object({
  description: z.string(),
  objects: z.array(z.string()),
  colors: z.array(z.string()),
  text: z.string().optional(),
  sentiment: z.enum(['positive', 'neutral', 'negative']),
});

const analyzeImage = ai.defineFlow(
  {
    name: 'analyzeImage',
    inputSchema: z.object({ imageUrl: z.string() }),
    outputSchema: ImageAnalysisSchema,
  },
  async ({ imageUrl }) => {
    const { output } = await ai.generate({
      model: googleAI.model('gemini-2.5-flash'),
      prompt: 'Analyze this image and provide a detailed description, list of objects, dominant colors, any visible text, and the overall sentiment',
      media: [{ url: imageUrl, contentType: 'image/jpeg' }],
      output: { schema: ImageAnalysisSchema },
    });
    
    return output;
  }
);

// Use the flow
const analysis = await analyzeImage({ 
  imageUrl: 'https://example.com/photo.jpg' 
});

console.log(analysis.description);
console.log('Objects found:', analysis.objects);

Next Steps

• Learn about RAG for multimodal document retrieval
• Explore Streaming for progressive image generation
• Check out Evaluation for testing multimodal outputs