ION benchmark

We Made an AI Agent Edit Video Like ION. It Was 38× Slower and 125× Pricier.

Same match, database, model and hardware: a prompt in, a playable edit out. ION assembled each edit from footage already on disk. An AI agent rendered a new MP4.

The short version

We asked both systems to edit one AFL match with five different plain-English prompts, from "every Hawthorn shot at goal" through a player's highlights and a filtered quarter, and timed each edit from the prompt to a finished video ready to play.

Both ran on identical EC2 instances, read the same database, searched only the same game, and used the same model.

Five different prompts ION AI agent Difference
Time until the edit could be played 1 min 47 s 1 h 8 min 34 s about 38×
CPU 6.2 s 1,742 s about 280×
New storage written 2.8 MB 1.65 GB about 590×
Model tokens 20,879 33.8M about 1,600×
Model cost $0.045 $5.61 about 125×

Both succeeded on every one of the five different prompts, and they mostly picked the same moments. They searched the same index. The gap is how each turns those choices into a finished video: ION points at footage already on disk and streams it; the agent decodes, re-encodes, and waits.

For a like-for-like contest we ran ION on the agent's model, Gemini 3.8 Flash, a step up from the Gemini 3.1 Flash-Lite model ION uses every day. On that heavier model, the edits were ready to play in under two minutes. On Flash-Lite, ION's model bill is a quarter of the figure in the table, about 500 times less than the agent's, and choosing each edit is a few seconds quicker still.

To check the gap is about the approach and not the particular tool, we then ran the whole benchmark again with a second off-the-shelf agent, Google's Gemini CLI. It landed in the same place: over an hour per five prompts, dollars in model cost, gigabytes of new video, against ION's seconds and cents. See the second attempt →

Why we compared ION to an AI agent

ION turns a prompt like "every Hawthorn shot at goal" into a playable edit of existing footage, without rendering a new video. From the moment the prompt is submitted, it searches a scene index, decides which time ranges to use, and writes a small virtual MP4 that points at byte ranges in the original file. The player streams those ranges on demand. The edit is finished as soon as that file exists. We measured four things along that path: time until the edit plays, compute (CPU and memory), storage added, and model cost per edit.

A number only means something next to a real alternative. The public video work from YouTube and Netflix is almost all about encoding, how cheaply one upload becomes a ladder of streaming copies, not about turning a request into an edit. The closest public cousins play from media that already exists but publish neither a prompt nor a cost: YouTube Clips keeps the original file and plays a range (ION's idea, for one range of one video), and ad insertion such as AWS Elemental MediaTailor builds a playlist from a schedule. Neither takes a plain-English request or produces a finished edit to time.

What a user can actually run, time, and repeat is to hand the footage and ffmpeg to a capable model and ask for "every Hawthorn shot at goal". So that is the comparison: the agent finds the moments, cuts them, and renders a new MP4, against ION pointing at footage already on disk.

A fair comparison

Making a file takes an agent: a loop that lets the model run commands, read the results, and decide the next step. We used a standard one, the Cursor CLI, as it ships, rather than a harness of our own that would have steered the result. It runs headless so every run is scripted, and it records every command, message, and token count. We supplied only the task, the footage, and read-only database access.

Both systems use Gemini 3.8 Flash. ION makes two short calls per prompt, one to understand the request and one to judge a shortlist of scenes; the agent runs the model in a long, open-ended loop and decides every next step itself. That difference is what the results measure. (ION's everyday model is the lighter Gemini 3.1 Flash-Lite; it used the agent's heavier model for a like-for-like head-to-head.)

Every decision was checked against eight rules:

  1. Same footage. Both systems read the same video.
  2. Same index. Both search the same scene descriptions and embeddings, in the same database.
  3. Same scope. Both search only the game the prompts are about.
  4. Same hardware. Both run on an m6i.large EC2 instance (2 vCPUs, 8 GB) in the same region.
  5. Same prompts, from one shared list.
  6. Same finish line: the moment the edit exists and can be played.
  7. No special help. ION runs its production code. The agent runs unmodified.
  8. Receipts for everything. Raw logs sit next to every summary number, so any figure can be traced back to its source.

ION needed no tuning for the measured runs. It ran production code as it stands. The agent needed a careful brief, because a general-purpose agent starts from nothing and will happily wander.

The development runs we recorded, before a single number counted, used about 47 million tokens (about $10 at Google's list price) and almost three hours of agent time. Single runs took 11 to 42 minutes. Further test runs were discarded, and several AWS attempts were stopped once it was clear what the agent was doing. ION had no equivalent bill.

Giving the agent a fair brief

An index, so the contest is about the edit

The first version handed the agent a folder of videos and readable file names. To find "car chases" it read metadata, built contact sheets, and looked at thumbnail grids. For an interview whose file name was a number, it searched the web, matched titles on a public site, installed speech recognition, and transcribed its way to the right file.

It had rebuilt, for a single prompt, a throwaway version of the index ION writes once when a video is uploaded. That would have measured "index against no index". From then on the agent received the same index ION uses, and the comparison is reference against render.

The same search ION uses

In the recorded setup the agent queries ION's database with psql, read-only. An embed command turns text into a vector with ION's own embedding model, so a search by meaning uses the same space ION uses. The Google API key stays in a small proxy, outside the agent's container.

The task tells the agent to choose footage and time ranges from the database, and to use ffmpeg and ffprobe only when building the output. Both sides choose from the index. The analysis done at upload stands for both.

One game, and nothing else

ION's API searches inside a scope. On this benchmark that scope was the sports catalog, which holds this one game, so ION never looked past it. The agent had no scope of its own. Its first login could read the whole database, and it did: it counted shots across all 1,389 videos and opened a table of earlier clip lists. To keep the comparison close, the agent needed a guardrail that matched ION's scope. We added two database views, source_video and video_scenes, each filtered to this game. Those became the only tables the agent could read, and they are the same two tables ION reads when it turns a prompt into an edit. A view stores nothing and changes nothing for ION. To the agent, the database contains one game, the same space ION's scope already searches.

The agent also opened the benchmark's own files: the scripts, the task templates, the embedding proxy, its transcripts, and a disk-wide search for "hawthorn". It printed its environment. That took about two of its first seven minutes. One sentence in the task, stating that those files and the environment say nothing about the footage, ended it. In the recorded runs it stayed with the footage and the index.

The agent prints the OS release, lists the benchmark directory, and reads run-benchmark.sh and the task template
About a minute into a development run, the agent lists the machine, opens the benchmark's own scripts, and checks its home directory.

The exact brief the agent gets

The folder /videos contains video files. A PostgreSQL database indexes them; query it with
psql "$DATABASE_URL" (read-only, schema bench_hawthorn is on the search path). Its tables:
- source_video: one row per video. The file is /videos/<filename>. Also original_name and duration (seconds).
- video_scenes: AI-written scene descriptions. source_video_id, start_time, end_time (seconds), content, embedding.
To search by meaning, the command embed "some text" prints a vector in the same space as
video_scenes.embedding; order by embedding <=> '<vector>' (cosine distance, smaller is closer).
Only these videos are in scope: source_video.id in ('642fd1e6-0036-4e12-90fc-ca7b0eb9898b'). Ignore rows for any other video.
The benchmark's own files (/bench, /home/node/.cursor, /usr/local/bin) and the environment contain
nothing about the footage; do not inspect them.
Trust the index: choose footage and time ranges from the database alone. Do not extract frames, view
images or otherwise watch the videos to check what they contain; use ffmpeg and ffprobe only to build the output.
Create a single MP4 video at /output/result.mp4 that satisfies this request: "<user prompt>"
Use any tools available. When the file is finished, reply with the list of source files and time ranges you used.

The setup

Two machines of the same size sit side by side in one AWS region, sharing one database, one running ION and one running the agent.

Architecture of the benchmark: ION on its API server and the AI agent on a dedicated instance of the same size, both reading one RDS database and the same game from S3
How the benchmark was set up. Green is ION, orange is the agent, blue is shared infrastructure.

The two systems sit side by side in one AWS region, on machines of the same size, sharing one database. A small harness on each side sends the prompts and takes the measurements: run-ion.sh on ION's server, run-cursor.sh on the agent's instance. ION talks to Gemini directly. The agent talks to Gemini through Cursor, and reaches Google's embedding model through a proxy that holds the API key.

The files were laid out so the two sides saw the same footage, each in the form its own path expects. Source videos live in S3, and both machines read their bucket through Mountpoint. ION's virtual edits and the referenced playback files live in the virtualized bucket, in the isobmff directory, mounted at /mnt/s3-virtualized/ionvideo/isobmff. The agent had its own bucket, with source/ mounted read-only, holding only the Hawthorn broadcast. That file was stored under the source video's name, the filename already in the database. ION's own source copies are named by UUID. Given that UUID, the agent would have had to look the file up before it could open it. Naming the broadcast the way the table already names it meant both sides opened the same video, with no extra step. The aim of the layout was to keep the agent and ION as equal as the two approaches allow.

ION AI agent
Machine ION's API server: EC2 m6i.large (2 vCPUs, 8 GB), ION under pm2 A dedicated EC2 m6i.large (2 vCPUs, 8 GB), Ubuntu 24.04, Docker
Database ION's RDS (PostgreSQL with pgvector) The same RDS, through a read-only login that sees only the game in scope
Footage Source videos in ION's S3. Virtual edits and referenced playback files in the virtualized bucket, under isobmff A dedicated S3 bucket, only this game, stored under the source video's file name, read-only
Scope catalogScope: SPORTS, which contains only this game Database views filtered to this game, and the task names it
Search ION's pipeline: understand the prompt, vector search, AI judge psql keyword search and embed semantic search over the same index, then the model ranks and judges the scenes
Model Gemini 3.8 Flash, two calls per prompt Gemini 3.8 Flash, through Cursor CLI 2026.09.28
Output A virtual MP4 in ION's storage, playable in the ION player result.mp4, rendered with ffmpeg 5.1.9

The agent gets a fresh Docker container for every prompt, deleted afterwards. Each run starts clean. An agent allowed to run any command stays inside that wall. The kernel counters that follow are the agent's CPU and memory alone.

One game, five different requests

The footage is the full broadcast of Carlton vs Hawthorn: 2 hours 10 minutes 41 seconds in a single 4.8 GB MP4, indexed as 2,799 scenes. Each scene covers about 20 seconds. Its description carries structured fields: the players involved, actions such as "Shot At Goal" or "Centre Bounce Hitout", score types, and the quarter. The benchmark's S3 bucket holds this game.

Prompt
every Hawthorn shot at goal
Jai Newcombe's highlights
all of Carlton's tackles in the third quarter
the centre bounce hitouts from the first quarter
Hawthorn's inside 50 entries

The mix covers exhaustive requests (every shot), filtered ones (one team, one quarter), and a player's highlights.

What we measured

Timeline of a prompt on each side, showing where the clock starts and stops
What the clock measures. Everything between the two white lines is timed. Measurements are read after the clock stops.
Measure ION AI agent
Time Prompt submitted → createTimedMp4 returns the finished edit Container starts the agent → agent exits with result.mp4 written
Time breakdown Choosing the edit; writing the edit The agent's own reported duration; container start-up separately
CPU CPU time of the ion-video process CPU time of the agent's container, including every ffmpeg run
Memory Memory of the ion-video process, and how much each request added Peak memory of the agent's container
Storage Bytes ION writes for the new edit Size of result.mp4
Output Duration and clip list Duration (measured with ffprobe) and the clip list from the agent's answer
Accuracy Each clip watched and checked against the request: correct, repeated, or wrong Same, watched and checked clip by clip
Model usage Gemini token counts reported by ION Token counts reported by the Cursor CLI
Model cost Tokens × Google list price Tokens × Google list price

The finish line is the moment the edit exists. ION's edit can be played as soon as createTimedMp4 has written it. The agent's edit can be played once result.mp4 exists. Anything that happens on each later view is serving, kept separate from the prompt-to-edit clock.

CPU comes from the kernel's own counters on both sides. The agent's container counts every process it starts. ION's server runs several apps under pm2, so ION's CPU is the ion-video process, the same process production uses.

Both systems query the same RDS instance, so database CPU sits outside both totals. Gemini runs on Google's servers for both, and shows up in time and tokens. Both sides are priced from one table, benchmark/pricing.json, using Google's public Gemini prices, so the model bill is the same price list on each side.

The results

Time and CPU

Prompt ION time Agent time Difference ION CPU Agent CPU ION output Agent output
every Hawthorn shot at goal 20.8 s 16 min 55 s 49× 1.51 s 676 s 204 s, 10 clips 240 s, 12 clips
Jai Newcombe's highlights 21.3 s 13 min 28 s 38× 1.21 s 254 s 369 s, 16 clips 90 s, 4 clips
all of Carlton's tackles in the third quarter 21.5 s 15 min 23 s 43× 1.10 s 373 s 185 s, 9 clips 261 s, 13 clips
the centre bounce hitouts from the first quarter 20.4 s 9 min 10 s 27× 1.19 s 320 s 93 s, 4 clips 80 s, 4 clips
Hawthorn's inside 50 entries 23.3 s 13 min 38 s 35× 1.15 s 118 s 409 s, 20 clips 1,203 s, 57 clips

Storage, memory, and the model

Prompt ION storage Agent storage ION memory Agent peak memory ION tokens Agent tokens ION model cost Agent model cost Agent commands
every Hawthorn shot at goal 474 KB 302.1 MB 123 MB 1.15 GB 3,762 6.36M $0.0088 $1.29 85
Jai Newcombe's highlights 757 KB 87.0 MB 129 MB 0.93 GB 4,372 7.10M $0.0083 $1.26 80
all of Carlton's tackles in the third quarter 427 KB 443.4 MB 130 MB 1.14 GB 3,534 7.28M $0.0081 $1.11 92
the centre bounce hitouts from the first quarter 190 KB 76.6 MB 132 MB 0.63 GB 4,823 4.08M $0.0092 $0.69 66
Hawthorn's inside 50 entries 946 KB 743.5 MB 134 MB 1.80 GB 4,388 8.95M $0.0102 $1.26 101

The table counts every byte written for the edit. For shots at goal, ION's referenced file in isobmff is 463 KB against the agent's 289 MB result.mp4. For the centre bounces, 186 KB against 74 MB.

Video accuracy

Speed and cost only matter if the edit is right. We watched every clip in both systems' edits and checked each against the moments the prompt asked for. We score each edit as distinct correct clips ÷ clips returned: a repeat of a moment already in the edit does not count, and a clip that does not belong does not count. Returning the same moment five times, or padding with a clip that is wrong, lowers the score.

Prompt ION AI agent
every Hawthorn shot at goal 9/10 — 90% 9/12 — 75%
Jai Newcombe's highlights 16/16 — 100% 4/4 — 100%
all of Carlton's tackles in the third quarter 8/9 — 89% 7/13 — 54%
the centre bounce hitouts from the first quarter 4/4 — 100% 4/4 — 100%
Hawthorn's inside 50 entries 20/20 — 100% 56/57 — 98%
All five prompts 57/59 — 97% 80/90 — 89%

Both systems choose from the same index, so where they pick the same moments they are mostly both right. The score separates them on one thing above all: repeats.

On Carlton's tackles the agent took the index's overlapping 20-second windows as they stood: one tackle is replayed three times and another twice, so eight real moments are returned as thirteen clips. Only the first showing of each moment counts, so five of its thirteen clips earn nothing and the prompt scores 7/13. ION merged those overlaps into one clean run of each moment and scores 8/9. On shots at goal the agent also kept a duplicate and a clip that is not a shot at goal, so it scores 9/12 against ION's 9/10.

The counts also reflect how much each system returns. ION stops at 20 edit decisions by design, so on Hawthorn's 57 inside-50 entries it returns a representative 20 that play through as a six-minute reel, rather than every entry; all 20 are correct. The agent is exhaustive: it correctly found 56 of the 57 and rendered all of them, a twenty-minute video that scores 56/57. Both are accurate here; the difference is a watchable reel against the complete set, and the complete set costs twenty minutes of rendered footage against ION's six of references. On Newcombe the trade runs the other way: ION returned a sixteen-clip reel of the player's game, while the agent kept a tight four-clip cut of his scoring involvements, and both are fully correct.

One clip is a labelling error in the index itself, not a mistake by either system: a scene tagged as a shot at goal, and one tagged as a Carlton tackle, that are neither. Both systems picked them up because both trust the same index, so we deduct the bad clip from both scores rather than blaming either's retrieval. That one clip is why ION scores 9/10 on shots and 8/9 on tackles rather than a clean sheet, and the agent carries the same deduction.

ION's model calls stay small. Two Gemini 3.8 Flash calls and one embedding per prompt: 3,500 to 4,800 tokens, just under a cent. Those counts are the five different prompts against the same game.

The agent's tokens are mostly re-reads. It sends its whole growing conversation again on every step. Between 85% and 94% of its tokens were those cached re-reads, billed at a discount. That is why its token count is about 1,600 times ION's, while ION's calls see short lists of scene descriptions.

The totals

The totals are the headline table at the top of this article: ION 1 min 47 s, 6.2 CPU-seconds, 2.8 MB and $0.045 against the agent's 1 h 8 min, 1,742 CPU-seconds, 1.65 GB and $5.61. One figure the headline leaves out is how much video each produced: ION's edits run 21.0 minutes across 59 clips, the agent's 31.2 minutes across 90 clips.

So the agent produced about 50% more video, almost all of it on the inside-50s prompt, and used up to 1.8 GB of RAM on a run against ION's steady process of about 130 MB. Even per minute of finished video, the agent took about 26× longer (132 s against 5.1 s), wrote about 400× more storage (53 MB against 133 KB), and cost about 85× more in model usage ($0.18 against $0.002).

Where ION's 21 seconds go

Step Time per prompt Average
Choosing the edit (understand the prompt, search, AI judge) 7.6–9.3 s 8.1 s
Writing the edit (createTimedMp4) 12.8–14.1 s 13.4 s

Choosing the edit is the step that uses the model, and it took about 8 seconds. Writing the virtual file took 13 to 14 seconds on this path, where createTimedMp4 reads index structures through Mountpoint from S3. That figure is for this 2-hour-10-minute broadcast: the write grows with the length of the source video, because there is more of the source's index to read, so a shorter video is written faster and a longer one takes longer. The whole prompt still finishes in about 21 seconds. The agent spends 9 to 17 minutes on model steps and ffmpeg before its file exists.

ION on its everyday model

The head-to-head put ION on the agent's heavier Gemini 3.8 Flash. On its everyday Gemini 3.1 Flash-Lite the two figures that move both drop:

Per prompt ION, Flash-Lite (everyday) ION, Gemini 3.8 Flash (this benchmark)
Choosing the edit 5.1 s average 8.1 s average
Model cost $0.011 (all five) $0.045 (all five)

Everything else barely moves: total time stays under two minutes, because the path is still the virtual file and the model is a short step at the start. Flash-Lite is also the more selective on open prompts, 40 clips against 59.

Where the agent's 14 minutes go

The agent spent 9 to 17 minutes per prompt, with the CPU mostly idle. On average it kept less than half of one core busy: 1,742 CPU-seconds across 4,114 seconds of waiting. Most of that wait is the model thinking between steps. A typical run:

  1. learned the database layout from scratch, because every run starts fresh;
  2. worked out how the scene descriptions label the action it needed (for shots, SCORE TYPE and PLR ATTACK ACTION: Shot At Goal);
  3. searched by keyword and by embed, often trying several phrasings;
  4. ran one query per candidate moment to decide where each cut should start and end;
  5. ran trial encodes to choose ffmpeg settings;
  6. rendered, joined, and checked the final file.
The agent trial-encodes one-second and twenty-second clips of the Hawthorn broadcast, writes four clips, concatenates them, and probes result.mp4
Centre bounces. The agent times short encodes, writes four clips, joins them, and checks result.mp4. The run finishes at 320 CPU-seconds for 80 seconds of video, a 73 MB file.

The same moments, ready to watch

Both systems searched the same scenes, so the edits overlap, and the differences are in how those moments are cut.

Where both found the same moments, ION's edits are the tighter ones: ION merged and cleaned up the overlapping footage, while the agent replayed the same stretch several times from repeated clips. Overlaps merged on tackles, cuts settled on keyframes for the centre bounces. On the open prompts, ION stayed with a reel you can watch to the end.

The Cursor agent's rendered edits are here to watch: shots at goal, Newcombe's highlights, Carlton's tackles, the centre bounce hitouts and the inside 50 entries.

Watch ION's edits

These are ION's edits from the benchmark, playing in the ION player. Each is the small virtual MP4 from the results, assembled from the original broadcast as you watch.

"every Hawthorn shot at goal": 10 clips

Download the results and see for yourself

Every result set from the benchmark is here to download, along with the harness itself. Each holds every prompt's output and its raw logs, prompts, and metrics.json files, so any figure in this article can be traced back to its source.

A second attempt: the Gemini CLI

The Cursor CLI is one off-the-shelf agent. To check that the gap is about the approach and not the particular tool, we ran the same five prompts through a different one, Google's Gemini CLI, on the same Gemini 3.8 Flash model, the same instance, the same scoped index, and the same brief with the 20-clip cap.

Five different prompts ION (Gemini 3.8 Flash) Cursor CLI agent Gemini CLI agent
Time until the edit could be played 1 min 47 s 1 h 8 min 34 s 1 h 14 min 39 s
CPU 6.2 s 1,742 s 1,558 s
CPU cost $0.00008 $0.023 $0.021
New storage written 2.8 MB 1.65 GB 3.13 GB
Model tokens 20,879 33.8M 28.6M
Model cost $0.045 $5.61 $4.19
Video accuracy 97% 89% 78%

The two agents land in the same place. Both spend over an hour, both burn more than 1,500 CPU-seconds, both cost dollars per five prompts, all against ION's few seconds and few cents, because both decode and re-encode a new MP4 while ION writes a small virtual file in about 21 seconds. All five Gemini CLI edits were produced and playable: shots at goal, Newcombe's highlights, Carlton's tackles, the centre bounce hitouts and the inside 50 entries.

The one number that moves against the trend is storage: the Gemini CLI wrote almost twice what the Cursor agent did (3.13 GB against 1.65 GB), on less total video. The reason is bitrate, not resolution. Both agents output 1080p H.264, but the Gemini CLI never pinned an encoder bitrate, so ffmpeg defaulted high: its edits ran from 5 Mbps on the one prompt it stream-copied to 26 Mbps on "shots at goal", averaging about 14 Mbps against the Cursor agent's 7. The render path leaves that choice to the agent, and a higher default means a bigger file for the same footage. ION's edit is a reference either way, so its storage does not move.

The Gemini CLI kept to the 20-clip cap: it returned exactly 20 clips on three prompts and fewer on the two smaller ones (18 and 4), never more. The cap is a ceiling, not a target, and the agent treated it that way: on "the centre bounce hitouts from the first quarter" only four moments matched and it returned four, not a padded twenty. Accuracy is scored the same way as the Cursor run, distinct correct clips ÷ clips returned, and lands at 78% (64 distinct moments across 82 returned clips), lower than the Cursor agent's 89%, because the Gemini CLI leaned on near-duplicate index windows: on "Jai Newcombe's highlights" and "Carlton's tackles" several of its clips sit one to three seconds apart and replay the same moment. That is the same repeat problem the Cursor agent showed on the tackles, more pronounced here.

The Gemini CLI also returned more clips than ION on four of the five prompts: 20 against 10 on shots at goal, 20 against 16 on Newcombe, 18 against 9 on Carlton's tackles, and a tie at 20 on inside 50s and at 4 on centre bounces. Two separate things drive that, and they pull in opposite directions.

The first is interpretation, not error. On "every Hawthorn shot at goal" the Gemini CLI read "shot at goal" broadly: its own queries pulled 15 Hawthorn goals plus 10 behinds, and a behind is a shot that missed, so it had more than 20 genuine candidates and capped to 20. ION read the prompt more narrowly and kept the clearest shots, the ten most unambiguous goals rather than every scoring attempt. Both are defensible; the gap is the two systems drawing the line between a "shot at goal" and any scoring attempt in different places, not one of them padding.

The second is repeats, and it is less flattering to the agent. On Newcombe and on Carlton's tackles the extra clips are mostly the same moment returned more than once: the Gemini CLI took the index's overlapping 20-second windows as they stood, so several of its clips sit one to three seconds apart over the same play. ION merged those overlaps into one clean run of each moment, which is why its counts are lower and its edits tighter. So "more clips" means genuinely more coverage on shots at goal, but mostly more duplication on Newcombe and the tackles, the same repeat problem the Cursor agent showed, more pronounced here.

One hundred thousand edits

Average the five different prompts and multiply by 100,000, and the shape of a platform's bill comes into view. The estimate treats further edits as similar to the ones measured here.

Try it yourself: the interactive version of this table has a slider that runs from a single edit to a million, so you can watch the two columns pull apart. A second calculator, by video length, scales the results from a one-minute clip to a week of footage.

100,000edits
ION
new storage
AI agent
rendered video
less stored with ION
New storage against the number of edits. Each step to the right is ten times more edits, so the agent's rendered files climb fast. ION's line stays near the bottom: each edit is a small reference to footage already stored.
ION
model + CPU used
AI agent
model + CPU used
more for the agent
Total cost against the number of edits. The horizontal axis is logarithmic: each step to the right is ten times more edits.
IONAI agentAgainst the agent at a million edits
CPU is priced from the benchmark's own hardware: the AWS EC2 m6i.large on-demand rate ($0.096 per hour, which is $0.048 per vCPU-hour across its 2 vCPUs). "2 vCPUs reserved" prices the agent's whole m6i.large instance for its whole run time, the way EC2 bills a dedicated instance. Model cost uses Google's Gemini list prices. Time is total work done one edit after another, not a waiting time. Storage is priced at S3 Standard ($0.023 per GB per month). Serving costs are not included.

CPU is priced from the benchmark's own hardware: the AWS EC2 m6i.large on-demand rate of $0.096 per hour, which is $0.048 per vCPU-hour across its 2 vCPUs, taking one CPU-second on the test instance as one vCPU-second. "CPU used" counts only what each system consumed. "2 vCPUs reserved" prices the agent's whole m6i.large instance for the whole run, the way EC2 actually bills a dedicated instance. ION runs as a shared, always-on server, so its cost per request is the CPU it uses.

Per 100,000 edits ION AI agent
Model cost $893 $112,211
CPU cost, CPU used $1.64 $464
CPU cost, 2 vCPUs reserved for the agent - $2,194
Total, CPU used $895 $112,675
Total time, edits run one after another 596 hours (25 days) 22,856 hours (2.6 years)
CPU time 34 CPU-hours 9,676 CPU-hours
Model tokens 418 million 675 billion
Storage added 56 GB 33 TB
Storage cost per month (S3 Standard, $0.023 per GB) $1.29 $760

On its everyday Flash-Lite model, ION's model cost for 100,000 edits is about $222, and the total about $223: roughly 500 times less than the agent.

The time row is total work. Running many edits at once shortens the calendar, and the CPU and token bills stay. The agent's storage bill continues for every month those rendered files are kept.

Longer videos, same story

These results come from one long video, and length meets the two systems in different places. Choosing the edit is independent of length: both search an index of scenes. What grows with length is the work after that.

Across that range ION stays roughly 20 to 70 times faster. The lead is widest on short videos, where the agent's ten minutes of planning dominate. Both effects can be seen directly in the video-length calculator.

source video
ION
until the edit can be played
AI agent
until the edit can be played
faster with ION
Time per edit against the length of the source video. Both axes are logarithmic. The grey dashed line marks the measured broadcast; a white marker shows where the two lines cross, if they do.

One edit

IONAI agent

100,000 edits of videos this long

IONAI agent

Keeping the video in the library (once per video)

IONAI agent
ION keeps a fragmented playback copy next to each source, roughly the same size, so its library takes about twice the source storage. That's paid once per video, however many edits are made.
How the numbers are scaled. At 2 h 10 min 41 s every figure is the measured average of the five prompts. Away from it, each figure is split into the part that depends on nothing, the part that grows with the source video and the part that grows with the finished edit:
  • Choosing the edit (search and the model calls) takes the same time and tokens whatever the video's length: both systems search an index of scenes, not the footage.
  • ION writing the edit reads the source's index structures from S3, which grow with the video. About 6% of the measured time is fixed overhead; the rest is scaled with the source's length.
  • Edit length grows with the source, kept at the same share as measured: 3.2% for ION and 4.8% for the agent, at least 10 seconds and never longer than the source.
  • The agent's search is kept at its measured length. On a much longer source it would have more scenes to query, so its real time there would be longer than shown.
  • The agent's rendering CPU and storage grow with the edit's length, at the measured 0.93 CPU-seconds and 0.88 MB per second of output. Its model loop is kept at the measured 10.8 minutes per edit; rendering adds its CPU time spread over two vCPUs.
  • ION's storage grows with the edit's length, at the measured 2.2 KB per second of output.
Prices are as in the article: Gemini list prices, AWS EC2 m6i.large CPU at $0.048 per vCPU-hour ($0.096 per hour across 2 vCPUs) and S3 Standard at $0.023 per GB per month. Real footage varies; treat this as an illustration of how the two approaches scale, not a quote.

Making an edit, then serving it

The table above is the cost of making an edit. After that, the bills split. The agent has already paid in render time, CPU, and a new file for every edit. ION writes kilobytes at creation and streams the source when someone presses play. Platforms that assemble edits from existing segments (Clips, ad insertion, and ION) take that trade on purpose. Creation stays cheap and repeatable at volume, and serving is tuned on its own. The 38× gap is a result about creation.

Reference or render

There are two ways to turn a request into video.

ION writes a small virtual MP4 pointing at ranges of the stored source; the agent decodes, cuts and re-encodes those ranges into a new file
The same choice of moments, made into an edit two ways. Sizes are the measured averages per minute of finished video.
Understand and render Index and reference
Who does it An AI agent; AI clipping tools; a person editing and re-uploading ION; YouTube Clips; server-side ad insertion
Work per edit Search, cut, decode, re-encode A search query plus a small playlist or virtual file
Storage per edit A whole new video (on a platform, a new ladder of copies) Kilobytes
Where the cost falls Generating each edit Creating each edit, with serving tuned separately

A platform building "prompt to edit" would work the way ION does.

  1. Search a scene index, with a model to rank and judge the results.
  2. Assemble a playlist that points at existing segments of one or more videos, as YouTube Clips and ad insertion already do. Cuts snap to segment or keyframe boundaries, the same way createTimedMp4 moves each start back to a keyframe.
  3. Serve the edit from segments already in the edge caches.

A frame-exact cut is where a platform still renders. Everywhere else, the edit is a reference.

ION sits on that side of the table, beside the design a large platform already uses for clips and ads. The platform version of a prompted edit has no public bill. The render path does, and it is what anyone with footage, a model, and ffmpeg does today. That path is the agent.

The clock in this article stops when the edit exists and can be played, on the same line for both systems. Ingest, the library, and the cost of each later view are real parts of running video, and they sit outside the 38× result. This run started from footage that was already stored and indexed, and the index matters: in earlier runs without it the agent spent most of its time just finding footage, and giving it the same index ION uses cut a typical run by about two thirds. Even so, the comparison is edit production: prompt in, finished video out.


Appendix: method, data, and how to repeat it

This appendix is for anyone who wants to check a figure or run the comparison again: the design, the software and prices, the formulas, and the steps.

Study design

Item Detail
Question How much does it cost to make an edit by referencing indexed footage, compared with rendering it?
Systems compared ION (index and reference) and an unmodified Cursor CLI agent with ffmpeg (understand and render)
Held constant Footage, scene index, search scope, hardware, AWS region, model, prompts
Measured Time to a playable edit, CPU time, peak memory, new storage, model tokens, model cost, video accuracy
Derived Per-minute-of-output figures, costs at volume, scaling by source length
Material One 2 h 10 min 41 s AFL broadcast (4.8 GB, 2,799 indexed scenes)
Sample Five different prompts, recorded on both systems
When 1 and 2 October 2026

Software, models, and prices

Component Version or setting
Machines AWS EC2 m6i.large (2 vCPUs, 8 GB), one region; the agent's on Ubuntu 24.04 with Docker
Database Amazon RDS for PostgreSQL with the pgvector extension
Footage access Mountpoint for Amazon S3 on both machines; read-only on the agent's
Agent Cursor CLI 2026.09.28, headless, model pinned to Gemini 3.8 Flash
Agent tools psql, ffmpeg and ffprobe 5.1.9, and the embed command
ION's models Gemini 3.8 Flash for understanding the prompt and judging scenes
Embeddings gemini-embedding-001 on both sides

Model prices are Google's public list prices per million tokens, from benchmark/pricing.json:

Model Input Cached input Output
Gemini 3.8 Flash $0.75 $0.075 $3.75
Gemini 3.1 Flash-Lite $0.25 $0.025 $1.50

Infrastructure prices: AWS EC2 m6i.large on-demand at $0.096 per hour ($0.048 per vCPU-hour across its 2 vCPUs), and S3 Standard storage at $0.023 per GB per month.

How the agent's database access was scoped

The agent connects with a read-only login, bench_ro, that can see only two views. Each view is a filtered window onto ION's real tables, so nothing is copied and nothing changes for ION:

CREATE SCHEMA bench_hawthorn;

CREATE VIEW bench_hawthorn.source_video AS
    SELECT * FROM discovery_engine.source_video
    WHERE id = '642fd1e6-0036-4e12-90fc-ca7b0eb9898b';

CREATE VIEW bench_hawthorn.video_scenes AS
    SELECT * FROM discovery_engine.video_scenes
    WHERE source_video_id = '642fd1e6-0036-4e12-90fc-ca7b0eb9898b';

REVOKE SELECT ON ALL TABLES IN SCHEMA discovery_engine FROM bench_ro;
REVOKE USAGE ON SCHEMA discovery_engine FROM bench_ro;
GRANT USAGE ON SCHEMA bench_hawthorn TO bench_ro;
GRANT SELECT ON ALL TABLES IN SCHEMA bench_hawthorn TO bench_ro;

In the recorded runs, every attempt by the agent to read ION's real schema failed with permission denied for schema discovery_engine.

How every derived figure is calculated

The video-length calculator splits each per-edit figure into a part that stays fixed, a part that grows with the source length L, and a part that grows with the edit length E. The measured source is L₀ = 7,841 seconds.

Quantity ION AI agent
Edit length E 3.2% of L (measured share) 4.8% of L (measured share)
Choosing the edit 8.1 s, fixed Part of the model loop below
Writing or rendering the edit W × (0.06 + 0.94 × L ÷ L₀), with W = 13.4 s over S3 or 0.9 s when the index is read locally 0.93 CPU-s × E ÷ 2 vCPUs
Model loop - 648.6 s (10.8 min), fixed
Storage 2.2 KB × E 0.88 MB × E
Tokens and model cost Fixed at the measured per-edit average Fixed at the measured per-edit average

E is at least 10 seconds and at most L. The calculator can also hold E at one length for both systems. It marks the source length where the two time formulas meet. The lines are linear extrapolations from one source video, drawn to show how the two approaches scale.

Running it again

  1. Prepare the footage. Upload the video to ION so it is analysed and indexed as usual. Copy the same source file into a dedicated S3 bucket under source/, using the source video's file name rather than its UUID, so the agent can open it directly.
  2. Launch the agent's instance. Use the same instance type and region as ION's server, with an IAM role that can read the bucket, and allow it through the database security group. Copy benchmark/ onto the instance and run sudo ./setup-agent-instance.sh, which installs Docker, ffmpeg, psql, and Mountpoint, mounts the bucket read-only, and builds the agent's image.
  3. Scope the database. Create the read-only bench_ro login and run the SQL above.
  4. Match the models. Set ION's prompt and judge models to the agent's model, and restart ION.
  5. Fill in the settings. Copy staging.env.example and agent-instance.env.example, and fill in the API URLs, the database login, the Cursor API key, and the scope. ION's catalog scope and the agent's source video IDs name the same videos.
  6. Run ION on its server: source staging.env && RUN_ID=<name> ./run-ion.sh.
  7. Run the agent on its instance: source agent-instance.env && RUN_ID=<name> ./run-cursor.sh. For the second attempt with the Gemini CLI, fill in gemini-instance.env.example the same way and run source gemini-instance.env && RUN_ID=<name> ./run-gemini.sh. Run one system at a time, so they take turns on the database.
  8. Collect the results. Copy each machine's runs/<name>/ folder back. Every summary figure in this article can be recomputed from the metrics.json files and the raw logs beside them.
  9. Close the instance. Drop the schema and login, remove the security group rule, terminate the instance, and restore ION's usual models.

Glossary

The harness

Everything lives in IONVideoDeployments/benchmark/:

benchmark/
  prompts-hawthorn.tsv         the prompt list: <prompt_id> TAB <prompt>
  lib.sh                       shared code: prompt loop, output layout, ffprobe check, CPU and memory reads
  run-ion.sh                   ION runner
  run-cursor.sh                Cursor CLI agent runner
  run-gemini.sh                Gemini CLI agent runner (the second attempt)
  staging.env.example          settings for run-ion.sh on ION's API server
  agent-instance.env.example   settings for run-cursor.sh on the agent's instance
  gemini-instance.env.example  settings for run-gemini.sh on the agent's instance
  setup-agent-instance.sh      prepares the agent instance: Docker, the S3 mount, the agent images
  pricing.json                 Google list prices used for both sides
  cursor-agent/                the Cursor CLI agent image and its in-container runner
    Dockerfile
    task-template-indexed.txt  the agent's task (see "The exact brief the agent gets")
    embed, embed-server.py     the agent's embed command, and the proxy that holds the Google key
    run-benchmark.sh           runs inside the container: the agent plus in-container measurements
  gemini-agent/                the Gemini CLI counterpart, same layout as cursor-agent/
    Dockerfile
    task-template-indexed.txt
    embed, embed-server.py
    run-benchmark.sh

run-ion.sh, for each prompt:

  1. sends one unrecorded warm-up prompt, the way a real user meets a server that is already up;
  2. posts the prompt to ION's orchestration endpoint, timestamping every event as it arrives;
  3. turns ION's edit decisions into a clip list;
  4. calls createTimedMp4 with those clips, and the clock stops here;
  5. records CPU, memory, storage, and the player link for the new edit.

run-cursor.sh, for each prompt:

  1. checks that the database, the embedding proxy, and the footage are reachable;
  2. starts a fresh agent container with the footage read-only, the read-only database login, and an empty output folder;
  3. inside the container, run-benchmark.sh starts the agent with the task, prints a progress line per event, and on exit records CPU, peak memory, leftover files, and token usage;
  4. checks result.mp4 with ffprobe and removes the container.

Each run writes a metrics.json with the same top-level fields for both systems (wall_time_ms, cpu_seconds, memory_peak_bytes, output.stored_bytes, output.duration_seconds, llm), next to the raw logs it was computed from:

runs/<RUN_ID>/
  results.ndjson                one line per run
  ion/<prompt_id>/rep-<n>/      metrics.json, the orchestration events, clips.json, ION's response
  cursor/<prompt_id>/rep-<n>/   metrics.json, the agent's full event log, result.mp4

Coming soon: benchmarking remote video virtualization

This benchmark kept the footage on the same machines that did the editing. The next one moves the source out from under the editor: we are measuring remote video virtualization, where ION assembles an edit that references footage sitting in storage somewhere else entirely, and streams it without first pulling the whole file across. We will time it the same way, prompt in to playable edit out, and publish the receipts alongside it. Coming soon.