I gave an AI agent one rule for a world model

A world model built a 3D world from a painting and invented bridges around it. I gave an AI agent seven small tools and one written rule. It removed three bridges by changing 4.99% of the pixels of the 360-degree picture and nothing inside the painting's frame. In the new draft 3D worlds, a thin black line still shows where one bridge stood.

ExperimentPart two
After: the same 360-degree picture with no bridge on the right, no arch at the right edge and no bridge above the waterfall on the left. The painting's moon, ruins and lake in the middle are the same.Before: the world model's 360-degree picture of the painting, flattened. A large stone bridge with many arches is on the right, a small arch is at the right edge, and a bridge joins two crags above the big waterfall on the left.
The world model's 360-degree picture, flattened. Before is on the left of the divider and after is on the right. Drag the divider, or use the left and right arrow keys, to compare them.

How I direct agents

The agent wrote the shell commands and ran the tools. The hard part wasn't that. It was five decisions that stayed mine, and this page is the record of me making them.

  • What problem is worth solving. A world model invents things the director didn't ask for, and every wrong invention costs money to make again. That's the problem I picked.
  • What to ask the agents to investigate. I gave the agent one written rule and seven tools, not a list of steps. Where to look, what to mask and how to fill it were the agent's calls.
  • How to recognise when their solution is wrong. The rule did hold. The same record also shows a thin black line still joining the two crags in 3D, and a dark slope where open water ran. That comes out of reading the 220 steps, not the summary.
  • How to design experiments that expose those mistakes. The before and after sweeps, the pixel difference against the original picture and the walks through the 3D world are there to catch a fill that looks right flat and wrong in a world.
  • When to change direction instead of throwing more compute at a failing approach. Both fill methods failed on the slope, so the third generation went unspent. A third try with the same fill would have made the same slope.

The rule

No bridges anywhere outside the painting's own frame. Do not change anything inside the painting's frame. Change as little else as possible.

The rule is needed because the world model decides almost everything in this world. The painting covers 3.85% of the full sphere around the viewer, measured by solid angle. The world model invented the other 96.15%, and that part included bridges.

In Into the painting, I told the world model in words not to invent a bridge, and that didn't work. Painting the bridge out of the 360-degree picture by hand worked, but my rough fill showed as a smudge in 3D. Here an AI agent does that job, with tools, under one written rule.

The flattened 360-degree picture tinted violet everywhere except a small rectangle in the middle, which shows the painting in its own colours
The clear rectangle in the middle is the painting. Everything tinted violet was invented by the world model.

What the agent could do

The agent had seven small command-line tools, which it ran from a shell. Only one of the seven, generate, costs money.

ToolWhat it does
noteWrites down the agent's reasoning.
lookRenders a view of the 360-degree picture at any angle, or an overview with a grid.
maskMarks a region by angles.
fillPaints a marked region out.
generateAsks the world model for a new 3D world from a 360-degree picture. It's the only tool that costs money.
compareMeasures the difference between two pictures.
walkRenders a view from inside a generated 3D world.

Each tool appended its own line to a log file every time it was called, so the agent didn't write the record of its tool calls itself.

The agent also opened the saved pictures with a picture viewer, and it read some files directly. Those two kinds of action aren't in the record.

The generate tool has a budget rule built in. It allows at most 3 paid generations and refuses after that. The agent used 2 and left the third unspent.

What it removed

The agent removed three invented bridges outside the frame. The instructions for the tools already gave the positions of two of them, the large stone bridge and the small arch. The agent found the third one, above the big waterfall, in its own first look at the picture. It left the bridge inside the painting's own frame alone. Each pair below is the same part of the 360-degree picture, before and after.

The large stone bridge

The large stone bridge with many arches is gone, and so are the reflections of its piers in the water.

A large stone bridge with many arches runs from a hillside on the left, over the water, to a grassy hillside on the right. One pier stands on a rock in the water
Before
The same view with no bridge: a hillside, a rock shore and open water, with a flat dark slope on the left
After

The small masonry arch

The small masonry arch between two crags is gone, and so is the whole cloud that was behind it.

A small masonry arch joins two crags above a waterfall, with a pale cloud behind it
Before
The same two crags with no arch and no cloud: red sky above, and rock joining the crags lower down
After

The bridge above the big waterfall

The bridge between the two crags above the big waterfall is gone.

A masonry bridge with a row of small arches along its top joins two crags above a wide waterfall
Before
The same two crags and waterfall with no bridge, and red sky between the crags
After

How it did it

The agent drew a mask whose top edge follows the parapet of the large bridge. The mask also covers the reflections of the bridge piers in the water, and the cloud behind the small arch. Then it ran three fill passes: a texture-copy fill of the whole mask, a refill of the left part of the large bridge with an inpainting model, and a texture-copy refill of the slope band.

It generated world 1 and saw a thin black line across the lip of the waterfall in 3D. It cleaned a leftover dark strip at the lip in the picture, and then generated world 2.

What I measured afterwards

After the session I checked the final 360-degree picture against the original with a separate script.

  • 4.99%This share of the picture's pixels differs from the original. It's 7.34% of the sphere by solid angle.
  • 0Inside the painting's frame, the difference is exactly 0.
  • 0Outside the agent's final mask, the difference is exactly 0.
  • IdenticalThe picture the world model returned is pixel-identical to the picture the agent sent.

All three bridges are gone in the picture and in the 3D renders of both new draft worlds. The agent's sweeps found no new bridge.

The original flattened 360-degree picture with pink patches over the large bridge on the right and its far end at the left edge, over the small arch at both edges, and over the bridge above the waterfall
The agent's final mask, in pink, over the original picture. No pixel outside the pink areas changed.

The whole session

This is the real record: all 220 steps in order, with the 23 error lines kept in.

Steps 1 to 52 and 97 to 220 are the agent. At step 52 it stopped without spending anything. It had looked at two of its three fills, and it wrote that in them “the large bridge area is soft and smeared”. The same note gives a second reason: “Then my image viewer stopped responding for more than an hour, so I could not see the full-size fill or improve it.”

Steps 53 to 60 are a second AI agent. I asked it to rewrite the fill tool.

Steps 61 to 96 are a third AI agent, which checked the rewritten tool independently. When the first agent read the log again at step 97, it wrote that the checker “says that fill is sharp but shows patch edges in several places and recommends a corrected mask”.

The agent called generate 5 times. 3 calls were dry runs, which cost nothing, and 2 were real.

22 of the 23 error lines are calls with a wrong argument. 16 are by the checking agent, at steps 62 to 77. 6 are by the first agent, at steps 164 to 169, and it wrote at step 170 that they were “caused by my own shell loop”. The other error line, step 96, is a call by the checking agent with a file that does not exist.

In this public copy, every piece of text that I replaced is in square brackets. I replaced the names of models, services, code libraries and one tool, the world IDs and the credit balance. I cut each file path down to its file name. I left out the extra data fields of each log line: timing, lists of output files, camera positions and raw measurements. I removed no step.

220 steps, 23 error lines, 2 paid generations, 149 pictures

The record has 220 steps. 23 of them are error lines and 2 are paid generations. The list appears here when the page has loaded it. Open the record as a JSON file.

In 3D

Each pair below uses the same camera in two worlds. The first picture in each pair is the draft world from my earlier rough fill. The second picture is world 2. Both are draft quality, which is blurry everywhere, so this is a fair comparison of one draft with another.

A blurry 3D view of the lake shore in the rough-fill world: a flat blue slab and a flat orange slab stand in the water where the large bridge was
This is the earlier rough-fill world, at draft quality. Flat blue and orange slabs stand where the large bridge was.
The same blurry 3D view in world 2: a dark slope and a channel of water where the large bridge was, with no slabs
This is world 2 from the same camera, at draft quality. It has a slope and a water channel, and no slabs.
A second blurry 3D view of the rough-fill world from another angle: a pale column of fog and a dark blue slab stand in the water
This is the earlier rough-fill world from another angle, at draft quality. A column of fog and a blue slab stand in the water.
The same second view in world 2: a flat dark slope runs down to the water, with no fog column and no large slab. A small blue-green patch is at the bottom edge, and a few small black specks are in the air on the right
This is world 2 from the same camera, at draft quality. The fog column and the large slab are gone, and a flat dark slope is in their place. A small blue-green patch is still at the bottom edge.

Below is world 2 to move around in. It's blurry because it's a draft. The file is 6.1 MB, so it loads only after you press play. Then drag to look around. To move, drag the dot in the circle at the bottom left, scroll, or use the W, A, S and D keys or the arrow keys.

World 2 in 3D, looking at the painting: the moon, the rock arch, the ruins and the lake with the sun's reflection, all blurry
This is world 2, looking towards the painting. It's draft quality, so all of it is blurry.

What's still wrong

  • The two new worlds are draft quality, which is blurry everywhere. So a 3D comparison is only fair against the earlier rough-fill world, which is also a draft. It isn't fair against the sharp original world.
  • Both new worlds used the same seed and almost the same picture, so they aren't independent tests.
  • Where open water ran under the left arches of the large bridge, there's now a flat dark slope. The fill is soft there. The fill tool reported its detail, measured as the mean absolute Laplacian, as 1.63 inside the fill and 3.72 around it.
  • In 3D, a thin black line still joins the two crags across the top of the big waterfall, where the bridge stood. It's in both worlds. It isn't in the final 360-degree picture. At step 220 the agent wrote that the line probably comes from the way the draft 3D world is built, and that this isn't settled, because the line also got weaker after the agent removed a dark strip from the picture.
  • The agent removed the whole cloud behind the small arch, and rock now joins the two crags where the arch opening was. Near the large bridge, the fill also has a rock outcrop that's copied from nearby rocks.
  • The agent didn't spend the third generation, because both fill methods had already failed on the slope.
  • This was one painting, one rule and one session.
A blurry 3D view of two crags in world 2 with a thin black line running between them across the top of the waterfall
This is world 2 in 3D, at draft quality. A thin black line joins the two crags across the top of the waterfall.
A blurry 3D view in world 2 of a flat dark slope running down to the water where the large bridge's left arches were
This is world 2 in 3D, at draft quality. The flat dark slope is where open water ran under the bridge's left arches.

Questions the agent left for me

The agent wrote its notes to “the director”, which is me. From its notes I have five things to decide, and I haven't decided any of them yet. I wrote the five questions below in my own words, and under each one is what the agent wrote. It named the first and the third as open questions. The other three are a decision it made and two changes it listed under “Things my fill invented or removed, which the director should know”. The agent calls the small masonry arch and the bridge above the big waterfall “the two small masonry arches”.

  1. Do the small masonry arch and the bridge above the big waterfall count as bridges?The agent, step 219: “Open questions for the director stay open: whether the two small masonry arches count as bridges (I treated them as bridges), and whether the thin dark line counts.”
  2. Does the bridge inside the painting's frame stay?The agent, step 97: “the two small masonry arches count as bridges, the bridge inside the frame stays.”
  3. Does the thin black line across the waterfall count as a bridge?The agent, step 219: “I do not call it a bridge (no masonry, deck or arch), but the director should decide.”
  4. Is removing the whole cloud acceptable under “change as little else as possible”?The agent, step 220: “the whole cloud behind the small arch is gone and replaced by sky”
  5. Is a dark slope in place of open water acceptable?The agent, step 220: “a dark slope where open water ran under the left arches of the large bridge”

What I'd do next

I'd decide those five things first. I'd spend the third generation only after the slope fill is fixed. When a draft world is clean, I'd make one full-quality world from the same picture.

More experiments

Pose the picture

I posed Lola like a teapot

I built a small app where you drag Lola's elbows, wrists and hands into a new pose, then press Regenerate. An image model redraws her in that pose, and the recording shows her posed like a teapot.

Lola Lens

A tool an AI agent calls to check the shot before I pay for it

I built a small MCP server that works out what a planned camera move will show in the 3D layout of Lola's bedroom. It finds the objects the prompt names that the camera never shows, or loses partway, before any video is made.

Lola's house

Building Lola's house as real 3D rooms

Lola is the lead in my AI video series, and her house only existed as a few pictures that didn't agree. This is the whole attempt to turn them into rooms a camera can move through. There's a kitchen grown from one picture, a floor plan that took 48 versions, and a bedroom built in 3D. It covers what went wrong and what fixed it.