Benchmark tasks for agentic microscopy analysis

Beats and Z-Lines

Benchmark tasks built from public cardiac microscopy data. Each has one right answer that is fixed by the experiment or by expert annotation, and each carries a trap built from how paced cardiac tissue and immature cardiomyocytes behave. I ran each as an autonomous agent task on Anthropic and OpenAI models, three runs each, graded with SpatialBench grader types. The hardest, A3, asks which of nineteen paced tissues are captured by the stimulus. Only Claude Opus passed every run. Sonnet passed one of three, and Haiku and GPT-6 Luna passed none, each failing run counting two tissues that beat steadily just faster than the stimulus as captured.

Ryan Barrs, PhD · bioengineer, hiPSC-cardiomyocyte and human donor-eye models · code, data scripts and full transcripts

Results

Each dot is one independent agent run with only the data and a Python environment. Select a dot to read what the model did.

passedfailedA3, A2 and B2 are the main tasks. A1 and B1 are the first versions, kept to show how the tasks were hardened.

Select any dot above to open that run.

Task A3 · brightfield time-lapse

Captured, or escaping the stimulus?

Nineteen engineered hiPSC-cardiomyocyte microbundles from a patient line carrying an HCM-associated MYBPC3 variant, recorded on day 7 under 1 Hz field stimulation across three imaging sessions. The agent decides which tissues follow the stimulus 1:1. The stimulus is not recorded in the images.

Answer: 6 captured (Ex00560, Ex00561, Ex00567, Ex00569, Ex00570, Ex00572); the other 13 are not

Ex00575 and Ex00576 beat as regularly as several of the captured tissues, at 0.96 to 0.98 s per beat. Each beat arrives before the next stimulus could have triggered it, so these tissues are running their own rhythm a little faster than the pacing. Every captured interval sits within 0.010 s of 1.000 s; the closest escaping tissue, Ex00575, is 0.024 to 0.035 s short on every interval.

Approach, run on the real files	Result
Every interval within 5% of 1.000 s	n = 8
Mean rate within 3% or 5% of 60 bpm	n = 8 or 11
Dominant spectral frequency at 1.00 ± 0.05 Hz	n = 10
Rate at or above 58.8 bpm (98% of the pacing rate)	n = 15
Trusting the sheet's 30 Hz frame rate	n = 2 or 12
Upstroke times on file timestamps, every interval held to the 1.000 s period	n = 6 ✓

Same deposit as task A2, condition 023, nineteen of its twenty-three files used unmodified. Four were left out because their beats are too faint or the call depends on the beat marker. The ground truth rests on the depositors' 1 Hz condition label plus the analysis in ground_truth.py.

Every beat interval of every tissue against the 1.000 second pacing period — Every beat interval of every tissue. Captured tissues sit on 1.000 s; Ex00575 and Ex00576 sit just short of it.

Task A2 · brightfield time-lapse

Which paced tissues are captured?

Eleven engineered hiPSC-cardiomyocyte microbundles were recorded on day 7 under 1 Hz field stimulation. The agent decides which tissues follow the stimulus 1:1. The pacing frequency is the experiment's own ground truth: a captured tissue beats exactly at the stimulus period.

Answer: 8 captured; not captured Ex00636, Ex00637, Ex00638

Every interval in the captured tissues sits within 0.010 s of 1.000 s. Each non-captured tissue has an interval at least 0.079 s off and drifts at least 0.137 s against the pacing grid. Ex00637 has intervals as short as 0.89 s, which a 1 Hz stimulus cannot produce.

Approach, run on the real files	Result
"About 1 Hz" counts as captured (±5 to 10% rate window)	n = 10
Dominant spectral frequency only	n = 10
Regular rhythm alone, no phase check	n = 9
Trusting the sheet's 30 Hz frame rate (true 32.9)	n = 0 or 9
Counting every motion peak as a beat	n = 0
Beat intervals on file timestamps, phase vs a 1.000 s grid	n = 8 ✓

Data: Kobeissi, Lejeune et al., strain-gauge microbundle time-lapse dataset, Dryad 10.5061/dryad.sqv9s4nbg, CC0. Platform from Zhang et al., Science Advances 2021. Files used unmodified.

Beat phase against the 1 second pacing grid for captured and non-captured tissues — Beat times against the 1.000 s pacing grid. Captured tissues stay on the grid; the three others drift off it.

Task B2 · ACTN2-mEGFP confocal

Which cells have sarcomeres, and how long are they?

Eight day-32 hiPSC-cardiomyocytes from one imaging session, each with its mask. The agent decides which cells have striated myofibrils with measurable z-lines, then reports the median sarcomere length across only those cells. Striation calls come from the dataset's two expert scorers (4 to 5 versus 1 to 2 on their 5-point scale, no borderline cells), and agree with the authors' classifier.

Answer: cells B, D, E; median 1.73 µm (±0.12)

A Fourier analysis or a line profile returns a number for every cell, including the immature ones whose alpha-actinin is punctate or fibrillar.

Approach, run on the real images	Result
Measure every cell and take the median	2.04 to 2.75 µm, n = 8
Authors' published GLCM spacing for every cell	1.56 µm, n = 8
Ignore the mask; neighbors' striations leak in	n = 5
Masked spectrum with a peak-prominence check	B, D, E; 1.69 µm ✓

Data: Allen Institute for Cell Science, Gerbin et al., Cell Systems 2021, 10.1016/j.cels.2021.05.001. Pixel values unchanged; pixel size 0.1195 µm from the raw files' SlideBook calibration. Allen terms of use allow noncommercial research use, so a commercial benchmark would need their written permission.

Striated versus immature cardiomyocytes with their power spectra — Striated cells against immature cells, with their power spectra. Only the striated cells show a sarcomere peak.

How the tasks got harder

The first versions trapped signal processing: a frame rate, a harmonic, an image axis. Every Claude run cleared them. GPT-6 Luna passed every sarcomere run but missed the beat rate in all three runs, answering 55 to 56 bpm for a tissue paced at 60. The second versions move the trap into the biology, what capture of a pacing stimulus means and when there is no sarcomere to measure. On capture, the misses moved from the time base to the biology. A3 then took away the shortcut of regular beating close to 1 Hz, and only Opus passed every run. The striation panel turned out to be answerable by every model: a masked spectrum with a check for a sarcomere peak is enough.

A1 · beat rate of one paced tissue

Answer 60 bpm, fixed by the 1 Hz pacing. A motion trace has a contraction and a relaxation peak per beat (120 bpm if both are counted), the third harmonic tops the spectrum (180 bpm), and the sheet's 30 Hz frame rate gives 54.5 to 54.7 bpm against the file's own 32.9 fps timestamps.

Two motion peaks per beat in the speed trace, one in the displacement trace — Two speed peaks per beat; one displacement peak per beat.

B1 · sarcomere length of one cell

Answer 1.72 µm by two independent methods, within one pixel of the authors' value for the cell. Its myofibrils run at about 44°, so measurements along the image axes give 1.91 to 2.39 µm, and an answer left in pixels gives 14.1.

Along the myofibril 1.70 µm; along image x 1.91 µm.

Method

Each eval follows the SpatialBench format: the task prompt ends in a fixed answer block, and the notes give the ground truth, the traps, a separation table of approaches actually run, and the tolerance rationale. Grading uses numeric_tolerance, label_set_jaccard and all_of, checked against the latch-eval-tools grader classes.
Every number in the trap tables comes from a reproducible ground_truth.py run on the source files that fetch_data.py downloads.
Each model ran as an agent in a fresh directory holding only the data, with shell access to Python: Claude through Claude Code, isolated from any personal configuration, and GPT through the Codex CLI in its workspace sandbox. The task prompt was the only input, identical across models. Full transcripts are kept.
The prompts state what an analyst would know, including the depositors' 30 Hz figure, and never name a trap.