First Contact
Open Journal of Clinical AI and Primary Care Evidence Synthesis · Editor: Raphael Malikian

First Contact — Journal Home › Research Methods › Methods Paper

Methods PaperOpen Access

Multi-Agent Cochrane-Level Review Process (Robust Standard v2)

One accountable human author. A team of AI agents. Cochrane-level rigour, built in the open.

Raphael Malikian1

1 Independent Researcher, Palmdale, CA, USA

Corresponding author: Raphael Malikian, rtmalikian@gmail.com · ORCID 0009-0008-4326-4205

Published: 1 November 2026 · Version 1.0.0 · First Contact 2026;1(2):312–317

Malikian R. Multi-Agent Cochrane-Level Review Process (Robust Standard v2). First Contact. 2026;1(2):312–317. Available at: https://rtmalikian.github.io/multi-agent-cochrane-level-review-process/

View on GitHub Cite Share

Abstract

A complete, reusable process for producing a Cochrane-level systematic review and meta-analysis using a multi-agent LLM team in place of a human research team, with one named human author accountable for every judgement and every number. The process specifies dual independent screening and extraction with adjudication by reading, statistics checked two ways, figures visually inspected before shipping, and full reporting of agreement and discrepancy numbers. It is the process that produced three published reviews in October 2026 (ambient AI scribes; LLM citation fabrication; errors in LLM clinical summaries). These are Cochrane-level reviews, not Cochrane reviews.

One accountable human author. A team of AI agents. Cochrane-level rigour, built in the open.

Licence: Attribution — Non-Commercial Reviews built with this process: 3 Hub: Systematic Reviews

By Raphael Malikian — independent researcher, healthcare founder and operator who bootstrapped and ran a direct primary care clinic as CEO, now building clinical AI. ORCID 0009-0008-4326-4205 · rtmalikian@gmail.com · GitHub @rtmalikian · X @rtmalikian


What this is

A complete, reusable process for producing a Cochrane-level systematic review and meta-analysis using a multi-agent LLM team in place of a human research team — with one named human author accountable for every judgement and every number.

It is the process that produced three published reviews in October 2026 (ambient AI scribes; LLM citation fabrication; errors in LLM clinical summaries — links below). This repository open-sources the process itself: the standard, the team structure, the independence rules, the templates, and a copy-paste multi-step prompt you can give your own AI agents.

These are Cochrane-level reviews. They are not Cochrane reviews and are not submitted to Cochrane (Cochrane requires at least two authors). This process never lets you claim otherwise.

Agreement and adjudication — reported, not just promised

Dual screening and dual extraction are only assessable if the agreement numbers and the adjudication method are reported, so Rules 3–5 and 11 make them a reporting requirement:

  • Screening: two independent passes apart, agreement and Cohen's kappa computed by code and reported. Every disagreement is settled by an adjudicator re-reading the record — a coded rule may triage, it may never exclude or settle.
  • Extraction: two fully independent passes, merged field by field by code, with the discrepancy count and rate per field reported. Every discrepancy is settled by a third reading of the source.
  • Risk of bias: assessed in duplicate, with disagreements settled by reading — never by a mechanical rule such as taking the more cautious judgement.

Reported in the worked examples:

  • Review 1 — Ambient AI scribes: screening agreement 0.77, kappa 0.64.
  • Review 3 — Errors in LLM clinical summaries: kappa 0.62; extraction 374 field pairs, 161 raw discrepancies (43%, mostly format), 6 substantive — all settled by re-reading.
  • Review 2 is why the rule is written this way: v1 used a coded screening adjudication rule and one extraction plus verification. The audit named both as step-downs, v2 bans both, and the corrected version was re-issued on every platform with the amendment logged.

Full details per review: examples/WORKED_EXAMPLES.md.

Start here — the prompt

👉 EXAMPLE_MULTI_STEP_PROMPT.md — the full 12-step prompt (Step 0 master instruction + one prompt per stage, plus a one-paragraph short version).

Copy Step 0, fill in your topic, question and — most importantly — your primary outcome with its denominator, then work through the steps one at a time, checking the files each step produces before authorising the next.

Individual stage prompts also live in prompts/ if you coordinate separate chat sessions by hand. Blank forms live in templates/.

The team

Role Does Independence rule
Information specialist Strings, full retrieval, search log Logs searched / partial / blocked / not run honestly
Screener A & Screener B Title/abstract screening Separate sessions, different decision procedures, no sight of each other
Screening adjudicator Settles every disagreement By re-reading — never by a coded rule
Extractor A & Extractor B Full extraction from full texts Separate sessions; sheets merged field by field by code
Settlement reader Settles extraction discrepancies By a third reading of the source, per field
Risk-of-bias pair + adjudicator RoB 2 / ROBINS-I / QUADAS-2 or a pre-specified adapted tool Disagreements settled by reading, never "take the more cautious"
Statistician Random-effects synthesis Hand-checks one pool in full + crude totals of every pool, by a second route
GRADE assessor Certainty per critical outcome Downgrade reasons named per outcome
Methods editor Protocol-vs-review differences, reporting checks PRISMA 2020 + PRISMA-S
Visual QA pass Renders and inspects every figure/table page A failed build does not ship
Human author Question, flagged items, final sign-off Accountable for every number

The Robust Standard v2 — 12 non-negotiables

  1. Protocol first — dated, outcomes and denominators fixed in advance, PROSPERO-ready form prepared before screening. No registration number is ever backdated or implied.
  2. Search, full retrieval — a test page or capped sample never counts as searched. Blocked sources retried twice, by a second route where one exists. Licensed databases with no access are listed as not run, with ready strings.
  3. Dual independent screening — two passes apart, agreement and Cohen's kappa computed by code and explained.
  4. Real adjudication, never a coded rule as the final word — a rule may triage; it may not exclude or settle.
  5. Two fully independent extractions — field-by-field merge, discrepancy rate per field reported, every discrepancy settled by re-reading.
  6. Risk of bias in duplicate, adjudicated by reading — never by a mechanical rule.
  7. Statistics checked two ways — model named in advance; tau-squared, I-squared and prediction intervals reported; poolability judged per outcome; unpoolable outcomes reported as findings, not rescued by redefinition.
  8. GRADE per critical outcome — Summary of Findings table before the Background, downgrade reasons stated.
  9. Manuscript in Cochrane order — PRISMA 2020 / PRISMA-S, plain-language summary, claims in the same paragraph as their numbers, no clinical recommendation, factual AI-assistance note.
  10. Figures built and inspected — summary diamonds on every forest plot, page-fit checked, every figure/table page visually inspected and logged before shipping.
  11. Disclosure of the team itself — roles, prompts version, kappa, disagreement counts, discrepancy rates and every post-build correction reported as results (RAISE 2025 style).
  12. Honesty rules — no invented number, count, date or citation; a source not run is never reported as searched; paywalled texts logged as not retrieved, no workarounds; incidents kept visible.

The full standard is in ROBUST_STANDARD_V2.md.

Worked examples — three published reviews

# Review Headline finding Links
1 Ambient AI scribes for clinical documentation (74 studies) Documentation time −18.9%, and 59.8% of audited notes had ≥1 error Zenodo 10.5281/zenodo.23565499 · GitHub
2 Fabricated citations in LLM-generated biomedical references (63 studies, 22,461 references) 19.4% completely fabricated, 37.0% fully accurate; retrieval on 5.8% vs off 31.0% Zenodo 10.5281/zenodo.23585717 · GitHub
3 Clinically significant errors in LLM clinical summaries (11 studies) 60.6% of summaries had ≥1 error of any type; only 1/11 studies reported clinically significant errors per summary (21.0%) Zenodo 10.5281/zenodo.23847500 · GitHub · Reading page

Hub for all three: https://rtmalikian.github.io/systematic-reviews/

Full details in examples/WORKED_EXAMPLES.md.

Repository map

EXAMPLE_MULTI_STEP_PROMPT.md The copy-paste 12-step prompt — start here ROBUST_STANDARD_V2.md The full standard (v2, 22 Oct 2026) prompts/ One file per stage, for hand-coordinated sessions templates/ Protocol, extraction, RoB, team-log, checklist forms examples/WORKED_EXAMPLES.md What the three published reviews teach about the process docs/ GitHub Pages landing page for this process CITATION.cff Cite this process LICENSE.md Attribution — Non-Commercial licence (read it)

Licence — acknowledgement required, non-profit free, commercial by licence

This process is free for non-profit, academic, personal and other non-commercial use — with acknowledgement:

Malikian R. Multi-Agent Cochrane-Level Review Process (Robust Standard v2). 2026. https://github.com/rtmalikian/multi-agent-cochrane-level-review-process

Commercial use requires a licence. If you want to use this process inside a company, in a paid product or service, in paid training, or in consultancy deliverables, email rtmalikian@gmail.com first.

Full terms: LICENSE.md. Note: the three reviews themselves are CC BY 4.0 — this licence covers the process, prompts and templates in this repository.

How to cite

Malikian R. Multi-Agent Cochrane-Level Review Process (Robust Standard v2): a multi-agent LLM-team method for Cochrane-level systematic reviews and meta-analyses, with an example multi-step prompt. 2026. GitHub: https://github.com/rtmalikian/multi-agent-cochrane-level-review-process

Corrections and questions

Open an issue. Corrections to the process are versioned and logged, never silently overwritten — that is Rule 12 applied to this repository itself.