How to extract keyframes from a video for GPT and Claude vision

GPT, Claude and Gemini models are good at reading images, but most API endpoints do not take a video file. If you want a model to describe a product demo, tag a stock clip, check an ad, or summarise a screen recording, you first have to turn the video into a handful of well-chosen frames with timestamps. This guide shows how to do that in about 20 lines of Python, without installing ffmpeg or OpenCV, and then send the frames to an OpenAI model or Claude.

Disclosure: we (Kanto Labs) built the Apify actor used below. Everything here also works with your own ffmpeg script - the "how many frames, which frames" advice applies either way.

A 10-minute video at 30 fps has 18,000 frames. Vision models bill per image (or per image tile), so sending them all is slow and expensive, and most of them are near-duplicates. What you want is:

8-20 frames at around 1024 px wide is enough for most "what happens in this video" questions.

Video Frame Extractor is a pay-per-frame API on Apify ($0.002 per frame; failed downloads and skipped black frames are free). It takes direct video file links (MP4, MOV, WebM, MKV... from S3, R2, a CDN, Dropbox?dl=1 ), and returns one row per video with a public image URL and a timestamp for every frame. It does not download from YouTube or TikTok pages.

Install the client and set your token (free Apify account -> Settings -> API & Integrations):

pip install "apify-client>=3" openai anthropic export APIFY_TOKEN =... # Windows PowerShell: $env:APIFY_TOKEN="..." Enter fullscreen mode Exit fullscreen mode import os from apify_client import ApifyClient apify = ApifyClient ( os. environ [ " APIFY_TOKEN " ]) run = apify. actor ( " kantolabs/video-frame-extractor " ). call ( run_input = { " videoUrls ": [ " https://media.w3.org/2010/05/sintel/trailer.mp4 " ], " mode ": " auto ", # first frame + strongest scene changes, topped up with even spacing " frameCount ": 12, # frames per video " maxWidth ": 1024, # plenty for vision models, keeps image tokens down }) video = next ( apify. dataset ( run. default_dataset_id ). iterate_items ()) print ( video [ " durationSeconds " ], " s, ", video [ " frameCount " ], " frames " ) for f in video [ " frames " ]: print ( f [ " timecode " ], f [ " sceneScore " ], f [ " url " ]) Enter fullscreen mode Exit fullscreen mode For the 52-second Sintel trailer this returns 12 frames ( 00:00:01.500, 00:00:02.175... 00:00:41.333 ) - the black opening frame is stepped past automatically. sceneScore is set when the frame is a detected cut. The run costs 12 x $0.002 = $0.024.

Cost check for interval: a 10-minute video at one frame every 10 s is 60 frames = $0.12 (raise maxFramesPerVideo, which is 50 by default).

Each row also has a contactSheetUrl: one grid image of all frames labelled with their timecodes. For a quick summary, that single image is often enough and costs one image's worth of tokens.

The frame URLs are public HTTPS links, so the model can fetch them directly. Interleave a short text label with each image so the model knows the order and the time. This uses the Responses API:

from openai import OpenAI content = [{ " type ": " input_text ", " text ": " These are keyframes of one video, in time order, each labelled with its timestamp. " " Describe what happens, scene by scene, and cite the timestamps. " }] for f in video [ " frames " ]: content. append ({ " type ": " input_text ", " text ": f " Frame at { f [ ' timecode ' ] }: " }) content. append ({ " type ": " input_image ", " image_url ": f [ " url " ], " detail ": " low " }) response = OpenAI (). responses. create ( # reads OPENAI_API_KEY model = " gpt-6-astra ", # any current vision model works; cheaper ones are fine for this input = [{ " role ": " user ", " content ": content }], ) print ( response. output_text ) Enter fullscreen mode Exit fullscreen mode "detail": "low" keeps each frame at a small token cost; switch to "high" when the model needs to read small on-screen text. (If you are still on GPT-4o and Chat Completions, the same idea works with image_url parts - but OpenAI has scheduled the GPT-4o snapshots for shutdown in January 2027, so new code should use a current model.)

Claude's Messages API takes images by URL as well. The structure is the same: label, image, label, image.

import anthropic content = [] for f in video [ " frames " ]: content. append ({ " type ": " text ", " text ": f " Frame at { f [ ' timecode ' ] }: " }) content. append ({ " type ": " image ", " source ": { " type ": " url ", " url ": f [ " url " ]}}) content. append ({ " type ": " text ", " text ": " These are keyframes of one video in time order. Describe what happens, " " scene by scene, and cite the timestamps. " }) client = anthropic.

Anthropic () # reads ANTHROPIC_API_KEY response = client. messages. create ( model = " claude-opus-5-5 ", max_tokens = 4000, messages = [{ " role ": " user ", " content ": content }], ) if response. stop_reason == " refusal ": print ( " The model declined this request. " ) else: print ( "". join ( b. text for b in response. content if b. type == " text " )) Enter fullscreen mode Exit fullscreen mode Gemini works too: download each frame's bytes ( requests.get(f["url"]).content ) and pass them as inline image/jpeg parts, or pass the single contact sheet.

videoUrls takes a list, and the actor returns one row per video, so a batch is the same call:

from decimal import Decimal run = apify. actor ( " kantolabs/video-frame-extractor " ). call ( run_input = { " videoUrls ": my_list_of_video_urls, " mode ": " count ", " frameCount ": 8, }, max_total_charge_usd = Decimal ( " 2.00 " )) # hard spending cap for this run for video in apify. dataset ( run. default_dataset_id ). iterate_items (): if video. get ( " error " ): print ( " skipped ", video [ " videoUrl " ], " - ", video [ " error " ]) # free, with a plain-English reason continue... # send video["frames"] to your model as above Enter fullscreen mode Exit fullscreen mode 100 videos x 8 frames = 800 frames = $1.60. With max_total_charge_usd the actor stops extracting once the budget is reached instead of overspending. Failed videos come back as rows with an error message and are not charged.

The same pipeline works in n8n, Make or Zapier: run the actor with the Apify node or an HTTP request to https://api.apify.com/v2/acts/kantolabs~video-frame-extractor/run-sync-get-dataset-items, split the frames array into items, and send each url to an OpenAI "Analyze image" step (or an Anthropic node).

Store page and full input reference: https://apify.com/kantolabs/video-frame-extractor