lolastudio.

Did the checks predict the problems?

This is the research log behind Lola Studio. I ran the planner's checks on every saved prompt from episode 1 of The Life of Lola, and compared them with the problems labelled in each take.

Back to the Lola Studio case study

The result

None of the 4 failure types in the problem labels is one the planner's risk check looks for.

  • I ran the planner's preflight and risk checks on the 25 saved prompts after the takes were made. Then I compared what they flagged with the 22 problem labels on those takes.
  • The risk check flagged 18 takes, and 1 of them has a problem label. It didn't flag 7 takes, and 6 of those have one.
  • 20 of the 22 problem labels are in two scenes, Finishing work and heading to the gym and Driving to the gym. So these counts mostly show which takes were labelled. They can't say whether the risk check works, only that its patterns don't cover the failure types seen so far.
  • I've checked 1 of the 32 labels, and it's a label on what worked, so none of the problem labels is checked yet. Every failure type name was proposed by an AI agent. This is one episode, so it's a small sample.
25
takes generated for episode 1
328 s
of video in those takes
$26.24
at the list price, not invoices
1 of 32
labels checked by me

Take log

All 25 takes in the order they were started, with what each one cost at the list price.

Scene

Select a take to watch it, see its planned shots, labels and checks, and read its prompt. Scene names with a dotted underline were proposed by an AI agent and I haven't checked them yet.

Episode 1 takes in the order they were started
TakeSceneStarted (UTC)Time to generateLengthCostRunning totalProcess Log labelsAI agent labels
v1Waking up1 Nov, 12:475 min 45 s15 s$1.20$1.2000
v1Getting dressed1 Nov, 12:476 min 04 s15 s$1.20$2.4000
v1Starting work at the desk1 Nov, 12:475 min 52 s15 s$1.20$3.6010
v1The boys head out1 Nov, 12:478 min 03 s15 s$1.20$4.8000
v2Waking up1 Nov, 13:049 min 42 s15 s$1.20$6.0000
v2Getting dressed1 Nov, 13:073 min 23 s6 s$0.48$6.4800
v2The boys head out1 Nov, 13:075 min 36 s15 s$1.20$7.6800
v3The boys head out1 Nov, 13:224 min 41 s10 s$0.80$8.4801
v3Getting dressed1 Nov, 13:223 min 08 s6 s$0.48$8.9600
v4The boys head out1 Nov, 13:423 min 07 s9 s$0.72$9.6801
v1Finishing work and heading to the gym2 Nov, 04:165 min 32 s15 s$1.20$10.8872
v2Finishing work and heading to the gym2 Nov, 04:519 min 09 s15 s$1.20$12.0830
v3Finishing work and heading to the gym2 Nov, 04:536 min 43 s15 s$1.20$13.2853
v1Driving to the gym2 Nov, 05:209 min 45 s15 s$1.20$14.4810
v2Driving to the gym2 Nov, 05:205 min 54 s15 s$1.20$15.6811
v2Gym montage2 Nov, 06:455 min 52 s15 s$1.20$16.8810
v1Gym montage2 Nov, 06:456 min 15 s15 s$1.20$18.0810
v1Coming home and cooking dinner2 Nov, 07:2110 min 43 s15 s$1.20$19.2810
v2Coming home and cooking dinner2 Nov, 07:216 min 04 s15 s$1.20$20.4810
v1Dinner with her sons, then bed2 Nov, 07:217 min 10 s15 s$1.20$21.6810
v2Dinner with her sons, then bed2 Nov, 07:2110 min 12 s15 s$1.20$22.8810
v4Finishing work and heading to the gym2 Nov, 07:567 min 14 s15 s$1.20$24.0800
v5Finishing work and heading to the gym2 Nov, 07:567 min 05 s15 s$1.20$25.2800
v1Going to sleep2 Nov, 08:5812 min 13 s6 s$0.48$25.7600
v2Going to sleep2 Nov, 08:5811 min 14 s6 s$0.48$26.2400
All takes25 takes, 328 seconds of video, $26.24 at the list price and 176 min 26 s of generating, added up take by take. Some takes ran at the same time, so this isn't time on the clock.

Cost is output seconds times the app's list price of $0.08 per second, not an invoice. Start times are in UTC, the same clock as the Process Log.

Evidence

What the planner's checks flag on each saved prompt, next to the problems the labels record. Then every label, grouped by take.

Predicted against what happened

The planner's checks

  • 19 of 19 Fits in one 15 second take. 6 takes were planned shorter than 15 s, so it doesn't apply to them
  • 25 of 25 Every shot is long enough to read
  • 2 of 25 Every shot says where Lola is looking (word match)

The risk check, which isn't calibrated

  • 9 of 25 Text on screen, 2 only in a negated line
  • 25 of 25 Hands handling a small object, 22 only in a negated line
  • 2 of 25 Two characters touching
  • 1 of 25 A transformation on screen
  • 1 of 25 Face turn or reveal
  • 5 of 25 Mirror or reflection

Next to the labels

  • 1 of 18 takes the risk check flagged have a problem label
  • 6 of 7 takes it didn't flag have a problem label

None of the 4 failure types in the problem labels is one the risk check looks for, so it couldn't have named those problems. 20 of the 22 problem labels are in two scenes, so this comparison can't say whether the risk check works.

The planner's checks are preflight() from the Lola Studio planner, run on each prompt's timed shots after the takes were made. Its outfit and prompt length checks need the planner's own fields, so they aren't run on these free-text prompts. The risk check uses text patterns from the planner without their weights, which have never been measured. A match in a line that says NO, never or nobody is marked as negated and shown struck through.

Whether a shot says where Lola is looking is a word match written for this page. 23 of 25 prompts don't pass it, and in 23 of those the first shot with no words about where she is looking is shot 1.

Hands handling a small object matched all 25 prompts, 22 of them only in a negated line. The table shows it only where it matched outside a negated line.

For each take, the checks run on its saved prompt next to its problem labels
TakeChecks passedChecks not passedRisk check matchedProblem labelsTypes in those labels
The boys head out2 of 2
  • Fits in one 15 second take doesn't apply, planned at 10 s
  • Face turn or reveal
1 AI agent
  • She faced the wrong way
The boys head out1 of 2
  • Every shot says where Lola is looking (word match)
  • Fits in one 15 second take doesn't apply, planned at 9 s
Nothing1 AI agent
  • Wrong exit or hands too full
Finishing work and heading to the gym2 of 3
  • Every shot says where Lola is looking (word match)
Nothing6 Process Log, 2 AI agent
  • Wrong light in the windows
  • Wrong object condition or an extra person
Finishing work and heading to the gym2 of 3
  • Every shot says where Lola is looking (word match)
Nothing2 Process Log
  • Wrong light in the windows
Finishing work and heading to the gym2 of 3
  • Every shot says where Lola is looking (word match)
Nothing4 Process Log, 3 AI agent
  • Wrong light in the windows
  • Wrong object condition or an extra person
Driving to the gym2 of 3
  • Every shot says where Lola is looking (word match)
  • Text on screen (negated line)
1 Process LogNo type set
Driving to the gym2 of 3
  • Every shot says where Lola is looking (word match)
  • Text on screen (negated line)
1 Process Log, 1 AI agent
  • Wrong object condition or an extra person

The table shows the 7 takes with a problem label. The other 18 have no problem label yet.

The most common failure types

22 problem labels, 14 from Process Log notes and 8 found by an AI agent. They come from 17 separate notes, because one Process Log sentence can cover two takes and some AI agent labels repeat a Process Log note. So a type's label count can be higher than the number of things that were seen. Select a type to see its labels in the failure library below.

Note from the Process Log, matched to a take by an AI agentFound by an AI agent in the videoChecked by me0 of the 22 problem labels

Every type name was proposed by an AI agent, and I haven't checked them.

Failure library

Every labelled moment in episode 1, grouped by take. Filter by failure type, by where the label came from, or by whether it was a problem or something that worked.

Failure type
Came from
Show

22 labels shown, from 7 of 25 takes.

  • take2c v3 of The boys head out

    Found by an AI agent in the video, not yet checked by Dana

    • ProblemLola looks level into the camera at 8.5 s and 9.5 s, not downward. The prompt asked her to look down the stairs at her sons.

      The prompt line it doesn't match is “looking down at her sons with a warm smile”

      Type She faced the wrong way

      How the AI agent found it

      Evidence. Seen in the web clip frame at 8.5 s (274 x 480). Start and end are the shot's cut times found by scene detection. The quote is the prompt line the frame does not match.

  • take2d v4 of The boys head out

    Found by an AI agent in the video, not yet checked by Dana

    • ProblemLola holds a cream slipper in one hand as well as the toothbrush in the other. The prompt asked for the toothbrush only, with the slippers on her feet.

      The prompt line it doesn't match is “holding a toothbrush”

      Type Wrong exit or hands too full

      How the AI agent found it

      Evidence. Seen in the web clip frame at 7.5 s (274 x 480). Start and end are the shot's cut times found by scene detection. The quote is the prompt line the frame does not match.

  • take5 v1 of Finishing work and heading to the gym

    Process Log note2 Nov 04:26 Review

    “Flags found: daylight in the work room window right after the dusk time-lapse; the racing-top stripes vanish when she raises her arms; the laptop never visibly shuts; the gym bag isn't on the counter, only at the exit; the clip's own sound peaks near 0 dB.”

    • Problem“daylight in the work room window right after the dusk time-lapse”

      Type Wrong light in the windows

    • Problem“the racing-top stripes vanish when she raises her arms”

      No failure type set

    • Problem“the laptop never visibly shuts”

      No failure type set

    • Problem“the gym bag isn't on the counter, only at the exit”

      No failure type set

    • Problem“the clip's own sound peaks near 0 dB”

      No failure type set

    How this note was matched to take5

    The log's 2 Nov 04:17 entry gives this take's task id, which matches the take5 log, and 5.5 minutes matches its generation time.

    An AI agent matched this note to the take and chose problem or worked. I haven't checked that.

    Process Log note2 Nov 04:50 Rejected

    • ProblemRun 1's banana came out of the fridge half peeled.

      Type Wrong object condition or an extra person

    How this note was matched to take5

    Run 1 of Take 5. The next entries in the log name run 2 as take5b and run 3 as take5c, so run 1 is take5.

    An AI agent matched this note to the take and chose problem or worked. I haven't checked that.

    Found by an AI agent in the video, not yet checked by Dana

    • ProblemThe work room window shows bright daylight and green trees straight after the dusk time-lapse in shot 1.

      The prompt line it doesn't match is “the sky turns from afternoon gold to dusky pink and purple”

      Type Wrong light in the windows

      How the AI agent found it

      Evidence. Seen in the web clip frame at 3.5 s (274 x 480). Start and end are the shot's cut times found by scene detection. The quote is the prompt line the frame does not match.

      Agrees with. Process Log label pl-02. The AI agent had read the Process Log before looking at the frames, so this is a confirmation, not an independent match.

    • ProblemThe banana is already half peeled in her hand while the fridge door is still open. Peeling was planned for the next shot.

      The prompt line it doesn't match is “takes out one banana”

      Type Wrong object condition or an extra person

      How the AI agent found it

      Evidence. Seen in the web clip frame at 11.5 s (274 x 480). Start and end are the shot's cut times found by scene detection. The quote is the prompt line the frame does not match.

      Agrees with. Process Log label pl-07. The AI agent had read the Process Log before looking at the frames, so this is a confirmation, not an independent match.

  • take5b v2 of Finishing work and heading to the gym

    Process Log note2 Nov 05:01 Step

    “Run 2: the whole banana is peeled at the flyer and the bag is on the counter, but the laptop is still there.”

    • Problem“but the laptop is still there”

      No failure type set

    How this note was matched to take5b

    The log's 2 Nov 04:52 entry names run 2 as take5b and run 3 as take5c.

    An AI agent matched this note to the take and chose problem or worked. I haven't checked that.

    Process Log note2 Nov 05:01 Step

    • ProblemBoth show daylight in the kitchen windows even though "dusk" was in the prompt.

      Type Wrong light in the windows

    How this note was matched to take5b

    The log's 2 Nov 04:52 entry names run 2 as take5b and run 3 as take5c.

    An AI agent matched this note to the take and chose problem or worked. I haven't checked that.

  • take5c v3 of Finishing work and heading to the gym

    Process Log note2 Nov 05:01 Step

    “Run 3: the desk is right (monitor switched off, no laptop), but she sips the shake rather than making it, the flyer is cropped, and she leaves without the shake.”

    • Problem“she sips the shake rather than making it”

      No failure type set

    • Problem“the flyer is cropped”

      No failure type set

    • Problem“she leaves without the shake”

      No failure type set

    How this note was matched to take5c

    The log's 2 Nov 04:52 entry names run 2 as take5b and run 3 as take5c.

    An AI agent matched this note to the take and chose problem or worked. I haven't checked that.

    Process Log note2 Nov 05:01 Step

    • ProblemBoth show daylight in the kitchen windows even though "dusk" was in the prompt.

      Type Wrong light in the windows

    How this note was matched to take5c

    The log's 2 Nov 04:52 entry names run 2 as take5b and run 3 as take5c.

    An AI agent matched this note to the take and chose problem or worked. I haven't checked that.

    Found by an AI agent in the video, not yet checked by Dana

    • ProblemAt 4.5 s and 5.5 s the work room window shows bright daylight. The prompt asked for a pink and purple dusk sky.

      The prompt line it doesn't match is “pink-purple dusk sky in the window”

      Type Wrong light in the windows

      How the AI agent found it

      Evidence. Seen in the web clip frame at 4.5 s (274 x 480). Start and end are the shot's cut times found by scene detection. The quote is the prompt line the frame does not match.

    • ProblemThe kitchen window on the right shows bright daylight and green plants. The prompt asked for dusk.

      The prompt line it doesn't match is “the kitchen at dusk”

      Type Wrong light in the windows

      How the AI agent found it

      Evidence. Seen in the web clip frame at 7.5 s (274 x 480). Start and end are the shot's cut times found by scene detection. The quote is the prompt line the frame does not match.

      Agrees with. Process Log label pl-15. The AI agent had read the Process Log before looking at the frames, so this is a confirmation, not an independent match.

    • ProblemThe banana is already peeled open in her hand while the fridge door is still open. The prompt asked for a whole, unpeeled banana in this shot.

      The prompt line it doesn't match is “takes out one whole, unpeeled yellow banana, still closed in its skin”

      Type Wrong object condition or an extra person

      How the AI agent found it

      Evidence. Seen in the web clip frame at 11.5 s (274 x 480). Start and end are the shot's cut times found by scene detection. The quote is the prompt line the frame does not match.

  • take6 v1 of Driving to the gym

    Process Log note2 Nov 05:33 Step

    “Neither shows the bag on the passenger seat (a small background detail [the video model] skipped).”

    • Problem“Neither shows the bag on the passenger seat”

      No failure type set

    How this note was matched to take6

    Take 6 has two folders (take6 and take6b), and the sentence says neither run shows it, so it applies to both.

    An AI agent matched this note to the take and chose problem or worked. I haven't checked that.

  • take6b v2 of Driving to the gym

    Process Log note2 Nov 05:33 Step

    “Neither shows the bag on the passenger seat (a small background detail [the video model] skipped).”

    • Problem“Neither shows the bag on the passenger seat”

      No failure type set

    How this note was matched to take6b

    Take 6 has two folders (take6 and take6b), and the sentence says neither run shows it, so it applies to both.

    An AI agent matched this note to the take and chose problem or worked. I haven't checked that.

    Found by an AI agent in the video, not yet checked by Dana

    • ProblemA second dark-haired woman in black stands in the open front doorway while Lola walks down the path towards the car.

      The prompt line it doesn't match is “Lola steps out”

      Type Wrong object condition or an extra person

      How the AI agent found it

      Evidence. Seen in the web clip frame at 0.5 s (274 x 480). Start and end are the shot's cut times found by scene detection. The quote is the prompt line the frame does not match.

The rule behind each failure type used here

An AI agent wrote each type's name. The rule under it is quoted from the Process Log.

Wrong light in the windows wrong_window_light
Rule from the Process Log. The window light follows the picture. Draw a dusk or night version of the room first.
Wrong object condition or an extra person wrong_object_or_extra_person
Rule from the Process Log. State each object's exact condition ("one whole, unpeeled banana"). For doorway wides, say "only one person in the shot".
She faced the wrong way wrong_facing
Rule from the Process Log. For every shot, name what is on the left and right of frame and which way she faces, matched to [Dana]'s reference clip (fridge on the left, window on the right, facing right).
Wrong exit or hands too full wrong_exit_or_full_hands
Rule from the Process Log. Name the door she leaves by. Check hands, outfit changes and hygiene (when she showered) before posting a plan.

Prompt change, output change

Pick a scene and two of its versions. Both clips play together, and the prompt change between them is shown word by word.

Scene
Versions
Left
Right
Left, v1take5, 15 s, $1.20, 6 reference picturesLabels 7 Process Log, 2 AI agent
Right, v2take5b, 15 s, $1.20, 6 reference picturesLabels 3 Process Log

The two clips play muted and stay on the same second.

What changed in the prompt

From v1 to v2, 132 words added and 13 words taken out.

Added Taken out

[animation studio] 3D animated feature look (stylised CG, soft rounded shapes, warm bounce light), matching the reference sheets exactly: same faces, outfits, rooms and colours. Vertical 9:16. NaturalSoft, quiet natural room sound only.only, no loud sounds. NO music. NO voices. NO on-screen text, subtitles or writing. Hard cuts between shots. NO fade out and NO fade to black at the end: the last frame is a normal bright picture and the clip ends on a hard cut. Lola is a woman in her forties with long jet black wavy hair and a wispy side fringe, full makeup with a berry lip. Her work room has navy walls; the kitchen is sage green with butcher-block counters (reference). It is DUSK in every shot: pink and purple evening sky outside every window, warm lamps on inside. SHOT 1 (0-3s): time-lapse, camera locked off across the street from her house in the quiet made-up neighbourhood (reference). The sun sweeps across the front of the house, shadows stretch long, the sky turns from afternoon gold to dusky pink and purple, and the windows glow warm. CUT TO SHOT 2 (3-6s): her work room at dusk (reference), pink-purple dusk sky in the window, desk lamp on, medium side shot: Lola, hair down, in her black long-sleeve racing top with thin magenta sleevestripes down both sleeves (reference; the stripes (reference),stay closesvisible the whole time). She pushes her open laptop andlid all the way down until it is fully closed, then stretches both arms up over her head, eyes closed, smiling. CUT TO SHOT 3 (6-10s): the kitchen at dusk (reference), warm pendant lights on, slow push in from three-quarter front. Lola is now in her gym outfit (reference): black zip-up jacket half zipped over a blue crop top, black tights, white canvas high-tops, hair up in a high ponytail with a black scrunchie. Her black gym duffel bag sits on the island counter.counter right beside her the whole shot. She shakes a plain white protein shaker bottle with one hand, looking at it. CUT TO SHOT 4 (10-12s): side shot at the stainless fridge: she opens the fridge door and takes out one banana.whole, unpeeled yellow banana, still closed in its skin. CUT TO SHOT 5 (12-13.5s): close up on the fridge door as it swings shut, revealing the Solera travel flyer held by two magnets (reference close-up). LolaLola, standing beside it, peels the banana open for the first time and takes a bite, looking at themthe flyer with a small determined smile. CUT TO SHOT 6 (13.5-15s): from behind, she picks uplifts the gym bag from the counter onto her shoulder, shaker in her other hand, and walks out of the kitchen toward the front door. The shot stays fully lit to the last frame.

How to run it again

The data files

  1. The page reads four files, takes.json, labels.json, failures.json and label-run.json. Every number on it is worked out from those files when the page renders, so a change to a file changes the page.
  2. takes.json holds the 25 take logs saved while I made episode 1 of The Life of Lola, with each prompt, length, generation time and cost. The frame for each shot is taken from the middle of its planned time, so a frame can show a neighbouring shot if the clip cut early or late.
  3. labels.json holds 24 labels that quote the Process Log and 8 labels an AI agent found in the video. An AI agent matched each Process Log note to its take, chose problem or worked, and proposed the failure types. I've added my own note to 1 of the Process Log labels.
  4. failures.json holds the 59 failure types an AI agent proposed. 4 of them are used by a problem label.
  5. The checks aren't stored. The page runs preflight() from the Lola Studio planner and the risk check's text patterns on each prompt in takes.json each time it renders.

The AI agent's labelling run, step by step

This is the run recorded in label-run.json on 19 Nov. The agent had read the Process Log first. It looked at 25 takes and 377 distinct frames, and checked 42 frames at full web size. To run it again, follow the same steps on takes.json, then I approve, edit or reject each label it proposes.

  1. Read each take's prompt and parsed shot list from takes.json.
  2. Found the real cut times in each web clip with scene detection, because the per-shot frames sit at the planned midpoints and some clips cut early or late.
  3. Looked at one frame per second from each web clip as contact sheets (240 px wide), and two frames per second for the two one-shot bedtime takes.
  4. Checked every candidate problem on single frames at the clip's full web size (274 x 480) before writing a label.
  5. Compared the bedroom, work room and Lola outfit reference pictures with the frames where a set or outfit question came up.
  6. Used only codes from failures.json. Where a visible difference had no fitting code, it was written down in label-run.json instead of being labelled.

The run proposed 9 labels. I rejected ag-03 on 20 Nov, so it isn't on this page. My reason was “The work room walls are meant to be navy. The reference picture is out of date.” The other 8 are waiting for me to approve, edit or reject them. It also wrote down 8 differences that no failure type covers, without labelling them. No problem matching a code was seen in the frames looked at, at 274 px or smaller. It does not mean the take has no problems.

Limits

  • Costs are the app's list price per second multiplied by each take's seconds. They aren't invoices.
  • Anything an AI agent added is marked as such, and I haven't checked it unless a label says it's checked. That includes its labels, every failure type name, which take a Process Log note belongs to, and the scene names with a dotted underline.
  • 20 of the 22 problem labels are in two scenes, and 18 of the 25 takes have no problem label. A take with no label hasn't been labelled yet, which isn't the same as having no problems. So the comparison with the risk check shows which takes were labelled more than it shows what the check can do.
  • One director, one episode and 25 takes is a small sample. The counts describe this episode only.
  • The checks were run on the saved prompts after the takes were made. They test the checks, not the takes.
  • F1 and E4 on the Lola Studio page measure the act 1 montage take, which isn't one of these 25 takes, so they aren't used here.
  • Names of other studios and tools, and the other name the Process Log uses for me, are replaced with a bracketed description.

Related findings on the Lola Studio page

These come from the case study, not from the labels on this page.

  1. F5

    My notes were about staging, not about her.

    All 6 notes across 3 scenes were about the camera, the action, continuity or the location. These are my notes on the Lola Studio page, not the labels on this page. It tests H4 in E5.

    Compare Waking upCompare The boys head outCompare Getting dressed

  2. F6

    A full 15 second take cost $1.20 at the list price.

    The take log adds up to 25 takes, 328 seconds and $26.24 at the list price. The case study reads this as “The cost per take is low enough to keep testing changes.”

    The take log, with every take's cost