agentmash lab · research
can an agent tell that an idea was already done, under another field’s name?
tools that check whether a scientific idea is new are public now. we found no published measurement of how often they find earlier work written in another field’s words, so we built the benchmark that takes it.
Coelacanth: a benchmark of cross-field rediscoveries for evaluating novelty and precedent checkers
published by the agentmash lab. labelled by five paid annotators with graduate stem training: three msc, two phd.
the data ships as an encrypted archive with a canary string, to keep it out of training sets.
the kind of case it holds
1994 a paper in diabetes care proposed a model for the area under glucose tolerance curves. a letter the same year was titled “tai’s formula is the trapezoidal rule.”
the paper’s words share no term with numerical analysis, so a search in its own field’s vocabulary has nothing to match. coelacanth holds ten documented rediscoveries like it, plus controls, ai-generated methods and fifteen negatives. a system sees only the claim, in its own field’s words, and its date.
a coelacanth is a fish known from fossils and thought gone for 66 million years, until one was caught off south africa in 1938.
what v0 found
- 1 of 9
near-negatives one agent labelled “nothing close”, the right answer. the other got 0 of 9: both call almost everything related prior work.
- 9 of 10
documented rediscoveries a claude model names with no search at all. on documented cases, a retrieval score can’t be told apart from memory.
- 716
labelling decisions, every one made by a person. models drafted claims and options; people chose.
how it was made
people chose every label
each of 136 claim and candidate pairs was read by two different annotators, 272 answers in all, and the author settled the 51 disagreements with a written reason each.
the rules were frozen before any system was scored
the protocol was tagged protocol-v1.0 on 11 october 2026 and its sha256 published,
06331185…f58ab3867a88. every agent configuration was hashed before its first scored run, and every later change is logged as a deviation.memory is measured, not assumed
every positive went to opus 5.5, sonnet 5.5 and haiku 4.5 with no tools, under two fixed prompts, and results report what the models recalled apart from what they did not.
nothing dated after the claim counts
a retrieved work from on or after the claim’s date, a web page read in 2026 included, is a leak and never a find.
what it doesn’t show
ten documented cases is under the twenty a test needs, so every comparison is descriptive. annotator agreement missed the thresholds set before labelling, and the freeze came after the labelling, not before it as planned. the paper reports both.
where the lab comes in
hunch, the lab’s first agent, makes exactly this check before a study starts: was the idea done already, in this field or under another field’s name. hunch was not run in v0. it goes into v1, scored the same way as everything above.
the paper runs on the rules the lab enforces. seal locks a plan by its sha256 before the data, foil runs the controls a result has to beat, assay recomputes every number, and only a person adds a result to the record.
the lab
ten agents for running a study and sending it out
8 research groups run it for their studies.
point at a figure to see what it does, click it to open its page.tap a figure to read more, tap it again to open its page.
hunch finds what to test, seal writes the plan down before anyone looks at the data, atlas checks the data, bench runs the plan, foil tries to explain the result away, and assay judges it. mash, the crew's conductor, runs them in that order. you approve the plan before it runs and the result before it goes into the study's record. then volt, ribbon, tensor or proof takes the paper where it is going: ieee, regeneron isef, an ml conference or a journal. each reads that venue's rules on the day and leaves the submit button to you.
why a lab
-
a wrong result looks right
a failing test catches wrong code. a wrong result passes review and gets cited. ask a model for a reference list and it writes one that looks right whether or not the papers exist.
-
one agent runs it, another attacks it
bench runs the analysis. foil, a different agent, builds the boring explanations the result has to beat, and assay recomputes every number before it rules.
-
the plan comes before the data
seal writes the pre-registration and you approve it. the approval records the file's sha256, and bench won't run if the file changed by one byte. only you can add a result to the study's record.
where the rules came from
the lab grew out of one study, may to september 2026, on drug-tolerant persister cells, the few cancer cells that survive therapy. a team of research agents ran it, and each rule below was written after something in its logs.
fourteen citations were wrong
agents wrote the reference entries themselves. two had invented venues, one took its page range from a different paper, and one replaced a paper on signalling entropy with a paper on polymer physics. the right entry for that one sat in a file next to the draft. now no agent writes a reference list. the text carries a marker like [REF: Surname Year], and a person copies the entry from a verified source.
a 32% forecast that added up to 50%
the red-team agent gave the paper a 32% chance of acceptance and split it into five outcomes. the five summed to 100, and the three accept outcomes alone summed to 50. asked about it, the agent said it had fitted the buckets to the headline. now an agent gives a probability only with a reference class, a base rate and its reasoning, and says likely or unlikely otherwise.
“three of four” was four of four
a dossier said three of four interval bounds fell below 0.20. all four did. the judge copied the sentence into its verdict, and it reached the scientist after three rounds of review with nobody recounting. now every number and every count is recomputed from the results file.
one quantity, two values
a dossier reported the same quantity with different values in two sections, supporting opposite conclusions, and was marked as passing all six checks. now the judge reconciles every number with the results file before it judges anything else.
caught on h1, missed on h2 and h4
the analysis used a different baseline than the pre-registration named. the judge caught it on the first hypothesis and missed it on the second and fourth, which carried the paper's new claims. now one deviation means every hypothesis gets checked for the same kind.
the control that worked
the study's headline positive didn't beat the target's own simple baseline. the control caught it before anyone submitted the paper, and the paper reports the negative. foil runs that control on every result.
how a study runs
steps 3, 8 and 10 are yours. no agent can do them for you.
hunch and atlas
a direction with its prior art, and data checked against its source. nobody looks at outcomes yet.
seal registers
the hypotheses, the exact estimators, what counts as support, refutation or inconclusive, what gets reported if the prediction fails, and every control foil must run.
you approve the plan
the approval records the file's sha256. if the file changes after that, bench stops.
bench runs
the registered analysis, failed predictions included.
foil attacks
baselines, nulls and confounds on every claim the study makes.
seal checks
one row per registered quantity, comparing what the plan named with what the code computed.
assay rules
merge, revise, close or escalate, with every number recomputed.
you decide
what goes into the study's record and what gets published. assay's merge is a recommendation.
the paper goes out
the venue's agent formats it, checks it against rules fetched that day and proposes every cut as a diff.
you submit
cuts, accounts, fees, signatures and the submit button stay with you.
set it up
-
add the agentmash marketplace
once per machine: it tells claude code where the crew and the lab come from.
claude plugin marketplace add agentmash-dev/agentmash-crew -
install the crew and the lab
both are claude code plugins in the agentmash marketplace. the lab needs the crew, because mash runs it.
claude plugin install agentmash-crew@agentmash claude plugin install agentmash-lab@agentmash -
give it a question
the lab asks for the study's source of truth first, then works through the steps above.
/agentmash-lab:run-lab does pre-treatment cell state predict which clones survive therapy? -
approve the plan
it stops and shows you the whole pre-registration. nothing confirmatory runs until you approve it.