Looping into a Game About AI Evals
Agents play-testing a game about AI Evals
Loop engineering is core to how Jetty’s runbooks work to improve AI workflows. It works really well: the agent has existing inputs, logs and telemetry and the outputs from a previous run. With all of this context, it becomes possible to improve the workflow, sometimes automatically. We decided to take this principle and build a game about AI evaluation. But would it be fun?
It started as a back-and-forth with Claude. By turning a pile of ideas into a product requirements document, we were able to make something you could actually play. Part of what I’m after is letting people feel what it’s like to watch an AI workflow get better with a human in the loop, instead of reading about it.
We started with GEPA: a reflective prompt optimizer by people at Berkeley and Databricks. This technique reads back its own trajectories in plain language, proposes fixes, and tries them. Sometimes, it can even beat reinforcement learning at a fraction of the compute cost.
We used these techniques to make Pelly’s Clinic, a browser game that teaches agent evaluation without ever using the word “eval.” To ground it in an example we’ve already worked on, we decided to frame the game around a doctor-patient visit.
Doctors increasingly hand their patient conversations to an AI “scribe” that listens to the visit and writes it up as a SOAP note: Subjective (what the patient reports), Objective (what the exam shows), Assessment (the diagnosis), Plan (what to do next). It’s the format every clinician’s chart already follows. The danger is that a scribe can hallucinate: write down something the patient never said, or promote a vague complaint into a confident diagnosis. In a medical record, an invented symptom can steer a real treatment decision.
In the game you coach Pelly, pelican working at a harbor wildlife clinic. A doctor sees patients (a gull, a seal, a very difficult octopus), and Pelly’s job is interview each patient and convert what they say into a structured output. But since Pelly is still learning, he can sometimes embellish stories into a “fish story”. For instance a seagull swears three eagles jumped him, and then Pelly charts the eagles as fact.
The first version was functional and completely un-fun: lots of screens, textboxes and back-and-forth dialogue. So we decided to use AI evals to make a game about AI evals better: we let agents play it.
We started with an runbook (a plain-markdown set of instructions) and pointed Jetty at it. The runbook tells an agent to open a real Chromium browser in a sandbox and play the game while journaling every moment of confusion with timestamps and screenshots.
Then it scores its own report against a five-part rubric. The runbook loop judges the insights from the run and decides whether the proposed fixes make sense. From the outermost loop we can ask claude code with the Jetty plugin to review the latest playtest, feed the five insights into a round of changes, redeploy, and set the agent loose again.
The best part was watching the loop argue with itself. Round three’s top complaint: “your title screen says the same thing twice. Replace the tagline with a visual of the actual mechanic.” Round four’s number-one finding, on the very next build: “I spent the first 90 seconds clicking the cards on the title screen because they look exactly like the live game. I thought I was already playing.” Every fix repositioned the next complaint.
After 5 rounds, we’re on our way to balancing a fun interaction while touching on the role of an LLM judge in AI evals. How should we guide the next loop of development?