First Contact
Open Journal of Clinical AI and Primary Care Evidence Synthesis · Editor: Raphael Malikian

First Contact — Journal Home › Research Methods › Methods Paper

Methods PaperOpen Access

Multi-Agent Peer Review System for Clinical AI, Medical Research and Health Technology Papers: persona-based AI reviewers and looped editorial gates before preprint submission

Persona-based AI peer reviewers and looped editorial gates before preprint submission

Raphael Malikian1

1 Independent Researcher, Palmdale, CA, USA

Corresponding author: Raphael Malikian, rtmalikian@gmail.com · ORCID 0009-0008-4326-4205

Published: 5 November 2026 · Version 1.0.0 · First Contact 2026;1(2):318–325

Malikian R. Multi-Agent Peer Review System for Clinical AI, Medical Research and Health Technology Papers: persona-based AI reviewers and looped editorial gates before preprint submission. First Contact. 2026;1(2):318–325. Available at: https://rtmalikian.github.io/multi-agent-peer-review-system-clinical-ai-medical-research/

View on GitHub Cite Share

Abstract

Before any article is submitted for preprint posting or sent to a journal, a panel of AI sub-agents reviews it. Each agent carries a different critical persona and experience bucket, chosen for that specific article: domain clinician, methodologist, biostatistician, technology specialist, patient reviewer, ethics reviewer, reporting editor, and a sceptic briefed to argue against the headline claim. The paper loops through editorial synthesis, revision and re-review until no Blocker and no Major comment remains open, then passes a final integrity gate that verifies citations and traces every headline number to the dataset. This repository publishes the full system: the looped-gates protocol (Gates 0–6), the persona-selection method, and an evaluation framework. It is pre-submission peer review, never journal peer review.

Persona-based AI peer reviewers. Looped editorial gates. No paper reaches a preprint server until it has survived review.

Licence: Attribution, Non-Commercial Process: Looped gates, not a single pass Companion: Cochrane-level review process

By Raphael Malikian, independent researcher, healthcare founder and operator who bootstrapped and ran a direct primary care clinic as CEO, now building clinical AI. ORCID 0009-0008-4326-4205 · rtmalikian@gmail.com · GitHub @rtmalikian · X @rtmalikian


What this is

A complete, reusable multi-agent peer review system for research papers, systematic reviews, meta-analyses, clinical AI papers, digital health studies, health technology assessments and scholarly commentaries.

Before any article is submitted for preprint posting (Zenodo, arXiv or any other server) or sent to a journal, a panel of AI sub-agents reviews it. Each agent carries a different critical persona and a different experience bucket, chosen for that specific article. A clinician does not review a statistics problem the way a biostatistician does, and a patient reviewer does not read a workflow claim the way an implementation scientist does. This system builds that difference in on purpose, then forces the paper through looped editorial gates until every major criticism has been answered in the text, fixed in the data, or recorded as an honest limitation.

It is the pre-submission peer review layer for a weekly research publication series in clinical AI, and a companion to the open Multi-Agent Cochrane-Level Review Process, which covers how a review is produced. This repository covers how a finished draft is challenged before anyone else sees it.

Honesty note, built into the protocol itself: this is pre-submission peer review. It improves a paper before submission. It is never described, in any manuscript, post or cover letter, as journal peer review, and a paper that passes these gates is "internally peer reviewed before submission", nothing more. See Rule 1 in PROTOCOL.md.

Why a panel, and why personas

A single reviewer, human or AI, has a single blind spot pattern. Real journal peer review works, when it works, because editors send a paper to people who disagree about different things:

  • The domain expert knows the literature cold and spots a missing citation, an outdated comparator or a claim the field settled two years ago.
  • The methodologist reads the design before the results and asks whether the study could answer its own question at all.
  • The biostatistician checks the numbers, the denominators, the model choice and the intervals, and does not care how good the story sounds.
  • The clinician or operator asks whether the workflow described exists in a real clinic, at a real staffing level, on a real Tuesday.
  • The patient and public reviewer asks who this helps, who it might harm, and whether the language respects the people in the data.
  • The ethics and regulatory reviewer checks consent, privacy, data governance, conflicts and claims that go past the evidence.
  • The reporting and reproducibility reviewer checks the checklist (PRISMA 2020, CONSORT, STROBE, STARD, TRIPOD+AI, CARE, as the study type requires), the data and code links, and whether a stranger could rebuild the result.
  • The sceptic has one job: argue the strongest case against the paper's own headline claim, using only the paper's own evidence.

Give all of those jobs to one generic "peer reviewer" prompt and you get one mild, averaged review. Give each job to a separate agent, in a separate session, with no sight of the other reviews, and you get the friction that makes peer review useful. That friction is the product.

Persona selection is per article, never fixed

There is no standing panel. For every paper, a Persona Selector agent reads the title, abstract, study type, methods, population, technology and target venue, then assigns 5 to 7 reviewers from the experience buckets in PERSONA_SELECTION.md, with a written reason for each pick and a written reason for any bucket it leaves out.

Examples the protocol works through in full:

Article type Panel it gets
Systematic review and meta-analysis Evidence synthesis methodologist, biostatistician, domain clinician, information specialist, GRADE and reporting reviewer, sceptic
Clinical AI model or imaging paper ML engineer, biostatistician, practising clinician in the specialty, patient safety reviewer, TRIPOD+AI reporting reviewer, sceptic
Open-source contribution analysis Software engineer in the stack, health informatics researcher, reproducibility reviewer, domain clinician, sceptic
Digital health intervention design Implementation scientist, clinician or operator, patient and public reviewer, ethics and privacy reviewer, health economist or service evaluator, sceptic
Commentary or opinion Domain expert who is likely to disagree, policy or ethics reviewer, literature expert, patient and public reviewer, handling editor
Education or community study Educator, community representative reviewer, methodologist, ethics reviewer, reporting reviewer

Full matrix, persona cards and the selector prompt: PERSONA_SELECTION.md and prompts/01_persona_selector.md.

The looped gates

A paper cannot be submitted until it clears every gate, in order. A failed gate loops back. Nothing is waived by deadline pressure.

Gate Name What happens Fail means
0 Editorial triage Scope, venue fit, ethics flags, article type confirmed, materials complete (manuscript, dataset, figures, references) Paper returned before review starts
1 Persona assignment Selector builds the panel for this article and records its reasoning Panel rebuilt, no review starts on a generic panel
2 Independent review Each reviewer works alone, in a separate session, no sight of other reviews, structured report with severity-graded comments Any Blocker or unanswered Major stops the paper here
3 Editorial synthesis A handling-editor agent merges the reviews, preserves disagreements, issues a decision: Accept, Minor revision, Major revision, or Reject and resubmit Decision is Major revision or worse: loop to Gate 4
4 Revision Author agent (or human author) answers every comment in a response matrix, changes the manuscript, the tables or the analysis, and logs every change A comment answered with words alone, where the text did not change, counts as unanswered
5 Re-review The original reviewers who raised Major comments re-check only their own points against the revised draft Any Major still open loops back to Gate 4
6 Final integrity gate Citations verified against sources, every number traced to the dataset snapshot, figures inspected page by page, declarations, conflicts and AI-assistance note checked, peer review statement worded honestly Any failed check loops back to Gate 4, however late it is found

Loop rule: Gates 3 to 5 repeat until no Blocker and no Major remains open. If a paper reaches a third loop with a Major still unresolved, the system stops and escalates to the human author with the unresolved point stated plainly. It never passes a paper silently, and it never averages a Blocker away because the other reviewers were happy.

Full gate definitions, severity taxonomy, decision rules and stop conditions: PROTOCOL.md.

What a comprehensive peer review has to cover

The protocol's review checklist, applied in full by the panel between them, covers:

  1. Question and originality. Is the question worth asking, is it stated plainly, and what does this paper add to what is already published?
  2. Literature position. Are the key prior papers cited, represented fairly, and engaged with, including papers that disagree?
  3. Design and methods. Could this design answer this question? Are eligibility, sampling, comparators, outcomes and denominators fixed and reported?
  4. Statistics and numbers. Model choice, intervals, heterogeneity, multiplicity, missing data, sensitivity analyses, and whether every number in the text matches the tables and the dataset.
  5. Results and claims. Does every claim sit in the same paragraph as the evidence for it? Do the abstract and conclusions stay inside what the results support?
  6. Bias and limitations. Selection, measurement, confounding, publication and author bias named specifically, with the direction each bias would push the result.
  7. Ethics, privacy and safety. Consent and governance where people or patient data are involved, no individual clinical recommendations in a research paper, no claim of a licence or authority the author does not hold.
  8. Reporting standards. The correct checklist for the study type, completed and checked against the manuscript, item by item.
  9. Reproducibility. Data snapshot, code or extraction files, search strings, prompts and versions recorded well enough for a stranger to check the work.
  10. Clarity and structure. A reader outside the specialty can follow the argument. Jargon explained. Tables and figures readable on their own.
  11. Declarations. Funding, conflicts, contributions and a factual AI-assistance note present and accurate.
  12. Venue fit. The paper matches the target journal or server in scope, format and audience, and the cover letter says nothing the paper does not support.

Start here

  1. Read PROTOCOL.md, the full system.
  2. Run prompts/00_orchestrator.md on a finished draft. It walks the gates in order and refuses to skip one.
  3. Let prompts/01_persona_selector.md build the panel for that paper.
  4. Give each reviewer prompts/02_independent_reviewer.md with their persona card filled in.
  5. Synthesise with prompts/03_handling_editor.md, revise, re-review with prompts/04_rereview.md, and close with prompts/05_integrity_gate.md.
  6. Keep the paperwork: blank forms are in templates/, and a filled-in worked example is in examples/WORKED_EXAMPLE.md.

How this is used in practice

In the weekly research publication series this system was built for, the gates sit between the finished draft and every external step. No Zenodo deposit, no journal submission, no arXiv posting and no conference abstract leaves until the Gate 6 sign-off is logged in the paper's own folder, alongside the reviews, the response matrix and the panel assignment sheet. The peer review record for each paper is kept with the paper, and the manuscript's declarations carry a short, honest statement that the draft received multi-agent pre-submission peer review under this protocol, with the human author accountable for the final text.

Is persona diversity actually better? The test

Honest answer: not yet tested. The claim that varied personas beat repeated reviews from one persona is this system's central hypothesis, and a public comment on launch day put the right challenge: papers with seeded errors, the same model, the same budget, a persona panel against one persona repeated. EVALUATION.md is the pre-registered protocol for that study: 12 papers, 120 seeded errors across the twelve review domains, three arms (persona panel, repeated generic reviewer, repeated strongest specialist), blind scoring, shared blind spot rate and per-domain recall as named outcomes. It is written so the diversity claim can fail, and a null result will amend this README where the claim is made.

Repository map

README.md This file PROTOCOL.md The full multi-agent peer review protocol, gates 0 to 6 EVALUATION.md Seeded-error study protocol: persona panel vs repeated single persona, matched model and budget PERSONA_SELECTION.md Experience buckets, persona cards, per-article panel matrix prompts/ Orchestrator, selector, reviewer, editor, re-review, integrity prompts templates/ Reviewer report, editorial decision, response matrix, gate log, panel sheet examples/WORKED_EXAMPLE.md One paper taken through every gate, with its panel and its loops docs/ GitHub Pages landing page CITATION.cff Cite this system LICENSE.md Malikian Attribution, Non-Commercial Licence v1.0

Licence

Free for non-profit, academic, personal and other non-commercial use, with prominent acknowledgement and a link back to this repository. Commercial use requires a licence by email. Full terms: LICENSE.md.

Suggested acknowledgement:

Pre-submission peer review process: Malikian R. Multi-Agent Peer Review System for Clinical AI, Medical Research and Health Technology Papers. 2026. https://github.com/rtmalikian/multi-agent-peer-review-system-clinical-ai-medical-research

How to cite

Malikian R. Multi-Agent Peer Review System for Clinical AI, Medical Research and Health Technology Papers: persona-based AI reviewers and looped editorial gates before preprint submission. 2026. GitHub: https://github.com/rtmalikian/multi-agent-peer-review-system-clinical-ai-medical-research · ORCID: 0009-0008-4326-4205

Author and collaboration

Raphael Malikian · rtmalikian@gmail.com · GitHub · LinkedIn · X

Open to collaboration on clinical AI research, evidence synthesis and research publishing infrastructure. Open an issue to discuss a paper type the panel matrix does not cover yet, or a reporting checklist that should be added to Gate 6.

Keywords

multi-agent peer review, AI peer review, automated peer review, LLM peer reviewers, pre-submission peer review, manuscript peer review system, clinical AI research, medical research publishing, systematic review peer review, meta-analysis quality, persona-based AI agents, editorial gates, research integrity, reproducibility, PRISMA 2020, TRIPOD+AI, GRADE, preprint submission, Zenodo, scholarly publishing