Abstract
Background. Large language models (LLMs) now draft clinical summaries inside electronic health record workflows: discharge summaries, hospital-course narratives, emergency department encounter summaries, progress-note summaries, referral letters and clinic-visit summaries. A clinician reads and signs each summary, and an error in it can travel with the patient. Single studies report error counts in very different ways, and no review has pooled, across summary types, the proportion of summaries that contain at least one clinically significant error, using the individual summary as the denominator throughout.
Objectives. To estimate the proportion of LLM-generated clinical summaries that contain at least one clinically significant error (study-defined), and the proportions with at least one omission, at least one fabrication or addition, and at least one inaccuracy, together with errors per summary, the proportion rated as carrying potential harm of at least moderate level, and any matched comparison with human-written summaries.
Search methods. We searched PubMed (378 records), Europe PMC (467), preprints through Europe PMC (140), Semantic Scholar (5) and ClinicalTrials.gov (1) on 31 October 2026, for studies published from 2022 to 2026, and we chased the reference lists of the included studies. OpenAlex was blocked by a provider rate limit and was not searched. Embase, Scopus, Web of Science, CINAHL, CENTRAL, IEEE Xplore and the ACM Digital Library were not run because no access route was available. This review has no PROSPERO registration number. A PROSPERO-ready form was prepared on 31 October 2026, before screening closed.
Selection criteria. Empirical evaluations in which an LLM generated a clinical summary from existing electronic health record source data (real or synthetic) and the study reported the number of summaries assessed and the number with errors, or data from which both could be derived. Patient-facing simplification or translation, radiology or pathology report generation from images, ambient scribe encounter notes, reviews and frameworks with no primary error counts were excluded.
Data collection and analysis. Screening, extraction and risk of bias assessment were each done twice, independently, by an LLM team working under one accountable human author, with every disagreement settled by reading the source. Title and abstract screening agreement gave a Cohen kappa of 0.62 for three categories and 0.67 for the binary pass or exclude split. Risk of bias used an adapted QUADAS-2 tool for LLM clinical summarisation evaluations (adapted from QUADAS-2 with QUADAS-AI signalling questions), named as adapted at every mention. Proportions were pooled with a random-effects generalised linear mixed model with a logit link, fitted in Python (scipy) with adaptive Gauss-Hermite quadrature because R was not available in the build environment. Certainty was rated with GRADE.
Main results. The search found 991 records. After 508 duplicates were removed, 483 records were screened, 48 reports were assessed in full, and 10 studies were included. Citation chasing added 1 study, giving 11 studies and 22 extraction arms (21 LLM arms and 1 human arm). The primary outcome could not be pooled: only 1 of the 11 studies reported the number of summaries with at least one clinically significant error, a prospective pilot in which 13 of 62 summaries (21.0 percent, exact 95 percent CI 11.7 to 33.2) carried a critical, treatment-impacting error, and even that count is ambiguous in the source text. A supplementary pool of summaries with at least one error of any audited type, specified post hoc in Amendment 01 and labelled as such throughout, gave 60.6 percent (95 percent CI 29.8 to 84.7; 5 units, 4 studies, 323 summaries; prediction interval 6.9 to 97.0). Omissions were pooled at 20.7 percent (95 percent CI 11.5 to 34.6; 16 units, 7 studies, 1011 summaries), rising to 40.6 percent (95 percent CI 22.0 to 62.4) when a study that counted omission of a single risk fact was removed. Fabrication or addition was pooled at 17.6 percent (95 percent CI 2.9 to 60.0; 8 units, 6 studies, 1226 summaries), a figure dominated by one outlying unit and close to uninterpretable as a single frequency. Inaccuracy was pooled at 27.9 percent (95 percent CI 14.6 to 46.7; 5 units, 3 studies, 500 summaries). One study rated 1 of 100 summaries as likely to cause moderate harm (1.0 percent, exact 95 percent CI 0.03 to 5.4). Errors per summary could not be pooled, and no study reported matched LLM and human error counts, so the LLM versus human contrast could not be fitted. Heterogeneity was very high in every pool (I squared above 93 percent). Nine of 11 studies were at high overall risk of bias on the adapted tool, 2 were at some concerns and none was at low risk. Certainty was very low for every rateable outcome, and the LLM versus human contrast was not rateable because there was no evidence.
Authors' conclusions. Errors are common in LLM-generated clinical summaries in the studies that counted them per summary, and the graded, clinically significant end of the error spectrum is the part the literature reports least. The finding of this review is as much about reporting as about frequency: a field that rarely counts clinically significant errors per summary cannot yet say how often its summaries are wrong in a way that matters. Studies should report, for every summary type and model, the number of summaries assessed, the number with at least one error of each type, and the number with at least one error graded clinically significant, so that the next review can pool the primary outcome this one could not.
Plain language summary
The problem. Computer programs called large language models can write clinical summaries, the short documents that describe a hospital stay, an emergency visit or a clinic appointment. Clinicians use these summaries to hand care from one person to another. If a summary leaves something out, changes a fact, or adds something that never happened, the next clinician may act on wrong information. We wanted to know how often that happens, counting one summary at a time.
What we did. We searched medical databases for studies from 2022 to 2026 in which a large language model wrote clinical summaries and researchers checked each summary against the patient record and counted the mistakes. We found 11 studies. Every study was checked twice, disagreements were settled by reading the papers again, and we combined the counts with statistical models that allow the true rate to differ between studies.
What we found. Most studies counted any mistake, but only one study counted the mistakes it classed as serious, meaning errors it judged could change treatment. In that study, 13 of 62 summaries (21.0 percent) had such an error. Across the studies that counted mistakes of any kind, about 3 in 5 summaries had at least one. About 1 in 5 summaries missed something from the record, about 1 in 4 contained an inaccurate statement, and counts of invented content ranged from none at all to nearly every summary, depending on how the study counted. One study rated 1 summary in 100 as likely to cause moderate harm. No study counted errors in model-written and human-written summaries of the same records in the same way, so we cannot say which is safer.
What this means. These systems write summaries with errors often enough that each summary needs a careful check against the record before it is used. The studies disagree with each other a great deal, and our confidence in every combined figure is very low. Until studies count serious errors per summary as a routine step, nobody can give a firm overall rate.
Key findings
- 11 studies (22 arms: 21 LLM, 1 human) included, from 991 records found.
- The primary outcome could not be pooled: only 1 of the 11 studies reported the proportion of summaries with at least one clinically significant error (13 of 62 summaries, 21.0%).
- By error type: omissions pooled at 20.7% of summaries, fabrication or addition at 17.6%, inaccuracy at 27.9%.
- Of 48 reports assessed in full, 31 (65%) reported no extractable per-summary error count at all.
- Certainty was rated very low for every rateable outcome (GRADE).
Summary of findings
Population and setting: LLM-generated clinical summaries (discharge, hospital-course, emergency department, progress, referral and clinic-visit types) generated from electronic health record source data, any specialty and setting, 2022 to 2026. Certainty wording describes confidence in the proportion estimate only. No clinical or medication recommendation is drawn from any rating in this table.
Table 1. Summary of findings
| Outcome | Units, studies, summaries | Result (95 percent CI; prediction interval for pools) | Certainty (GRADE) | What this row means |
|---|---|---|---|---|
| Summaries with at least one clinically significant error (primary, study-defined threshold) | 1 unit, 1 study, 62 summaries | Not pooled. Single unit: 21.0 percent (11.7 to 33.2), 13 of 62. No prediction interval | Very low | In the one study that reported this outcome per summary, about 1 in 5 summaries contained an error the study classed as critical and treatment-impacting. Only 1 of 11 studies reported the outcome this way, so this is a single pilot result, not an overall rate. |
| Summaries with at least one error of any audited type (supplementary, post hoc, Amendment 01) | 5 units, 4 studies, 323 summaries | Pooled 60.6 percent (29.8 to 84.7). Prediction interval 6.9 to 97.0 | Very low | Across the studies that reported any error at all, roughly 3 in 5 summaries had at least one error of some audited kind. The studies audited different things in different ways, so this figure describes the spread of the literature rather than the error rate of any one system. |
| Summaries with at least one omission | 16 units, 7 studies, 1011 summaries | Pooled 20.7 percent (11.5 to 34.6). Prediction interval 1.6 to 80.4. Excluding the risk-fact-specific study R0354: 40.6 percent (22.0 to 62.4) | Very low | About 1 in 5 summaries missed something the source record contained on the main pooling. Half the units counted only whether one specific safety fact was dropped, which pulls the figure down. Without those units the estimate is about 2 in 5. The pair of figures, not either one alone, is the finding. |
| Summaries with at least one fabrication or addition | 8 units, 6 studies, 1226 summaries | Pooled 17.6 percent (2.9 to 60.0). Prediction interval 0.1 to 98.3 | Very low | The pooled figure suggests about 1 in 6 summaries contained invented or added content, but the studies disagree so completely, from 0.0 percent to 97.5 percent, and counted fabrication in such different ways, that no single rate can be trusted. The range across studies is the informative part of this row. |
| Summaries with at least one inaccuracy | 5 units, 3 studies, 500 summaries | Pooled 27.9 percent (14.6 to 46.7). Prediction interval 5.2 to 73.0 | Very low | About 1 in 4 summaries contained at least one inaccurate statement, in the three studies that reported this outcome. Their results varied widely, and the rate in a new setting could be much lower or much higher. |
| Errors per summary | 2 units, 2 studies, 36 summaries | Not pooled. Unit means: 2.84 errors per summary (SD 1.71, n 25) and 1.82 errors per summary (20 errors over 11 summaries, no SD reported) | Very low | The two studies that counted errors per summary found roughly 2 to 3 errors in an average summary. One reported no measure of spread, so no combined average can be calculated. These two figures are illustrations, not an overall average. |
| Summaries rated potential harm at least moderate | 1 unit, 1 study, 100 summaries | Not pooled. Single unit: 1.0 percent (0.03 to 5.4), 1 of 100. No prediction interval | Very low | In the one study that graded harm per summary, 1 summary in 100 was rated likely to cause moderate harm, and none was rated at a higher harm level. This rests on a single rated summary in a single study and should not be read as reassurance. |
| LLM versus human error contrast | 0 units, 0 studies, 0 summaries in any matched contrast | Not fittable. No study reported matched LLM and human per-summary error counts for the same source records under the same outcome definition | Not rateable: no evidence | No included study counted errors in LLM summaries and human summaries of the same records in the same way, so this review cannot say whether LLM summaries contain more or fewer clinically significant errors than human summaries. That missing comparison is a gap in the literature, not a finding of no difference. |
Footnotes to Table 1. (a) Primary outcome ambiguity: the single contributing unit is R0301-2 (Park 2026, prospective pilot). The source text alternates between calling the 13 items errors and cases, and its Table 2 footnote says error counts do not add up to total cases with errors. The count is carried as printed under the abstract survey-case framing. If some cases carried more than one critical error, the true per-summary count would be lower. One further study (R0391) defined a major, clinically significant threshold but reported major counts at error level only, never as summaries with at least one major error, so it could not contribute a unit. (b) The any-error pool is supplementary and post hoc under Amendment 01 wherever it appears. It pools any error of any audited type across different audit definitions and does not replace the primary outcome. (c) All proportion outcomes start at low certainty under the GRADE approach for prevalence and proportion evidence as applied in this series, because the body of evidence consists of observational evaluation studies without a comparator. Ratings below low reflect downgrades for risk of bias, inconsistency, indirectness and imprecision, each named per outcome in the GRADE assessment. (d) Of the 11 included studies, 9 are at overall high risk of bias, 2 are at some concerns and none is at low risk, on the adapted QUADAS-2 tool for LLM clinical summarisation evaluations (adapted from QUADAS-2 with QUADAS-AI signalling questions). (e) Every pooled outcome has I squared above 93 percent and a prediction interval spanning most of the scale. Pre-specified subgroup analyses did not credibly explain the spread.
1 1. Background
Clinical summaries carry care across boundaries. A discharge summary tells the next team what happened in hospital, what changed, and what needs to happen next. A hospital-course narrative, an emergency department encounter summary, a referral letter and a clinic-visit summary do the same work at other boundaries. Each one is a compression of a longer record, and compression is where information can be lost, changed or invented.
Large language models are now used to draft these documents from the record itself. The draft reaches a clinician who reads it, edits it and signs it. That workflow has a clear attraction: drafting takes time, clinician time is short, and a draft that is mostly right may still save work. It also has a clear risk. A summary that reads smoothly can carry an error that a reader does not spot, because the text gives no sign of where it departs from the record. Error in a generated summary is therefore a different problem from error in a human draft written slowly. The reader's check is the last line of defence, and the check is only as good as the errors are visible.
Three error types run through this review, and they do different things to care. An omission removes information the source record contained: a diagnosis, a medication change, a follow-up plan. An inaccuracy changes information: a wrong dose, a wrong date, a finding attributed to the wrong side or the wrong patient encounter. A fabrication or addition inserts information the source record does not contain: a symptom never recorded, a test never done, a history never taken. Studies label these types with their own words, including hallucination and confabulation for fabrication, and content gaps for omission. The labels differ. The underlying events are what this review counts.
Frequency and severity are separate questions, and the literature answers them unevenly. A study can count every departure from the source, down to minor wording slips, or it can count only departures that could change management. The first count is easier to produce and is the one most studies report. The second count is the one a clinician or a service lead wants when deciding how much checking a generated summary needs. This review set its primary outcome at the second count: the proportion of summaries containing at least one error the study itself graded as clinically significant. The results section shows what happened to that choice. The literature, as it stands, almost never reports it.
Related syntheses leave this question open. A PROSPERO registration, CRD420261349028 (Kharat and Agrawal, registered 23 March 2026, planned end 31 October 2026), covers hospital discharge summaries only and explicitly excludes progress notes, referral letters and operative notes. Its scope is the hospital-course narrative and the discharge summary. This review covers all clinical summary types, with discharge as a pre-specified subgroup, one primary outcome, and the individual summary as the denominator throughout. A re-check on 31 October 2026 found that record still listed as planned or ongoing, and found no Cochrane Database of Systematic Reviews review or protocol on this question. Three widely cited primary studies show why per-summary counting matters and why it is rare: the discharge summary study by Williams and colleagues (JAMA Internal Medicine 2025) reported mean errors per summary rather than the proportion of summaries with at least one error; Van Veen and colleagues (Nature Medicine 2024) compared generated and expert summaries on rating scales; and Asgari and colleagues (npj Digital Medicine 2025) set out a framework for assessing clinical safety and hallucination rates. Each is discussed in the results, because each was excluded at full text for want of an extractable per-summary numerator and denominator, and that pattern is part of what this review found.
2 2. Objectives
The primary objective was to estimate the proportion of LLM-generated clinical summaries that contain at least one clinically significant error, where clinical significance is defined by the contributing study. Secondary objectives were to estimate the proportions of summaries with at least one omission, at least one fabrication or addition, and at least one inaccuracy; the mean number of errors per summary; the proportion of summaries rated as carrying potential harm of at least moderate level; and, where a study reported both, the contrast between LLM-generated and human-written summaries.
3 3. Methods
This review follows the structure of the Cochrane Handbook for Systematic Reviews of Interventions, version 6.5, and is reported against PRISMA 2020 and PRISMA-S. It is a Cochrane-style review. It is not a Cochrane review. The protocol (concept-and-protocol.md) was written on 31 October 2026, before screening began. One amendment was logged before any analysis was run (Section 3.3). The review team was an LLM team working in place of a human research team, with one accountable human author. Roles, agreement statistics and the settlement process are reported in Sections 3.6 and 3.7 and in Appendix B, and the arrangement is disclosed in the declarations, in line with the RAISE 2025 guidance on responsible AI use in evidence synthesis.
3.1. Criteria for considering studies for this review
Types of studies. Empirical evaluations of any design that reported the number of summaries assessed and the number with errors, or data from which both could be derived, published from 2022 to 2026. Peer-reviewed papers and preprints were eligible; preprints were flagged and removed in a sensitivity analysis. Reviews, editorials and frameworks with no primary error counts were excluded.
Types of summaries and data. The unit of analysis was an individual clinical summary generated from existing electronic health record source data, real or synthetic. Eligible summary types were hospital discharge summaries and hospital-course narratives, emergency department encounter summaries, progress-note summaries, referral letters and clinic-visit summaries, in any specialty, setting or country.
Exposure. The summary was generated by a large language model of any family, including fine-tuned, retrieval-augmented and agentic pipelines. Model and version were extracted where reported.
Comparators. The source record or structured ground truth, a clinician-authored summary, or blinded expert rating, where present. Single-arm proportions were the primary analysis, so a comparator was not required for inclusion.
Exclusions. Patient-facing simplification or translation of existing documents; radiology or pathology report generation from images; ambient scribe encounter notes generated from conversation; citation or reference generation; and studies with no extractable numerator and denominator per summary.
3.2. Outcomes
The primary outcome, fixed in the protocol before screening, was the proportion of LLM-generated summaries containing at least one clinically significant error, with clinical significance defined by the contributing study and the study severity or harm grade extracted alongside the count.
Secondary outcomes, in protocol order, were: the proportion of summaries with at least one omission; the proportion with at least one fabrication or addition; the proportion with at least one inaccuracy; errors per summary; the proportion of summaries rated as carrying potential harm of at least moderate level; and the LLM versus human error contrast where both were reported for the same source records under the same outcome definition.
A pool needed at least two units under the analysis plan. A single-unit outcome is reported as that unit alone, with an exact confidence interval and no pooled estimate.
3.3. Protocol amendments
Amendment 01, dated 31 October 2026, logged after extraction settlement and before any analysis was run. Settlement showed that the primary outcome was reported by exactly one arm in the included set (R0301, prospective pilot, 13 of 62), and that one further study (R0391) defined a major threshold but reported major counts at error level only. The primary outcome therefore could not be pooled under the analysis plan. Amendment 01 kept the primary outcome in its place with its not pooled verdict, and authorised one supplementary pool: the proportion of summaries with at least one error of any audited type. That pool is labelled supplementary and post hoc at every appearance in this manuscript, in the abstract, in the summary of findings table and in the conclusions. It does not replace the primary outcome. Sensitivities for the supplementary pool were set in the amendment as S1 (units from studies with any explicit severity or harm grading), S2 (excluding preprints) and S3 (excluding high risk of bias).
No other amendment was made. No extraction form change resulted from the two-pass comparison, and no separate extraction pilot phase was run; the two independent passes served as the check. That gap against the ideal team shape is recorded here rather than left silent.
3.4. Search methods
The search was run on 31 October 2026. The concept combined three blocks: the model (large language model, LLM, ChatGPT, GPT-4, generative AI, foundation model), the task (summarization, discharge summary, hospital course, encounter summary, progress note summary, referral letter, clinical summary) and the outcome (error, omission, hallucination, fabrication, inaccuracy, accuracy, completeness, safety), limited to 2022 to 2026. Appendix A gives the string as run for each source.
Sources searched, with records retrieved: PubMed, 378; Europe PMC (title and abstract strategy), 467; preprints through Europe PMC, 140; Semantic Scholar, 5, reached through its bulk search endpoint after the graph API returned rate limit errors on three attempts; ClinicalTrials.gov, 1. The combined total was 991 records.
Two retrieval notes belong in the open. First, a first Europe PMC run on default fields returned 6,569 hits dominated by incidental full-text mentions. That run was superseded by the title and abstract strategy before screening, and its raw output is retained in the search log. Second, OpenAlex was blocked: four search strings returned HTTP 429 with a response stating that the shared daily budget for the network address was exhausted. Under the provider rate limit rule in force for this series, that was treated as a hard stop and no retry was made in this run. OpenAlex is recorded as not searched, and the block is carried into the limitations.
Sources not run: Embase, Scopus, Web of Science, CINAHL, CENTRAL, IEEE Xplore and the ACM Digital Library were not searched on 31 October 2026 because no access route was available. Ready-to-run strings for each are filed in the search log. Google Scholar was not run; it has no full programmatic interface. None of these sources is described as searched anywhere in this review.
Citation chasing: backward citation chasing was carried out on the reference lists of the 10 studies included from database searching. It produced 15 candidate records verified as absent from the 483-record search set. Candidates were screened in duplicate. Fourteen were assessed, 1 was not retrieved, 13 were excluded and 1 was included (Song 2025). No separate forward citation-chasing pass was performed. The search itself covered 2022 to 2026 across the sources above, and the missing forward pass is stated in the limitations.
Registration position: this review has no PROSPERO registration number, and none is claimed or implied. A PROSPERO-ready registration form for this question was prepared on 31 October 2026, before screening closed (stage1-search/prospero-form.md). The related registration CRD420261349028 covers discharge summaries only, as described in the Background. A registry re-check on 31 October 2026 found no new registration on this review question and no Cochrane Database of Systematic Reviews review or protocol on LLM clinical summarization. The re-check used direct retrieval of the known record and web-index searches; PROSPERO own search interface could not be enumerated through the text routes available, and that limit is recorded in the re-check file.
3.5. Selection of studies
Deduplication was by DOI first, then PubMed identifier, then exact normalised title, with a second pass merging records whose normalised titles matched across keys. Of 991 records, 508 duplicates were removed, leaving 483 records for screening.
Title and abstract screening was dual and independent. Screener A used an inclusion-first checklist and Screener B an exclusion-first decision tree, each specified in writing before screening (stage2-screening/screening-rules-A.md and screening-rules-B.md). Screener A passed 58 records (11 include, 47 maybe) and excluded 425. Screener B passed 100 records (8 include, 92 maybe) and excluded 383. Agreement was 431 of 483 records (89.2 percent) across the three categories, Cohen kappa 0.62; for the binary pass against exclude split, agreement was 439 of 483 (90.9 percent), kappa 0.67. The kappa values sit in the substantial range on the Landis and Koch scale, and they are pulled down by prevalence: 382 of 483 records were excluded by both screeners, which raises expected agreement to 0.72 and lowers kappa at a given level of observed agreement. The binary kappa is the decision-relevant figure, because pass against exclude is the split that built the full-text list. All 52 disagreements were read by a screening adjudicator, who read the title, the abstract and both screeners reasons for each record and decided 9 passes to full text and 43 exclusions, with none flagged for the accountable author. The full-text list held 58 reports: 57 passed by both screeners and 1 added by adjudication.
Full-text screening applied a fixed hierarchy of exclusion reasons, coded FT-R1 to FT-R9, with the first applicable reason in the hierarchy recorded as the primary reason. Reports not retrieved and reports excluded are counted in the results.
Citation-chasing candidates were screened by two independent screeners under the same eligibility rules. One candidate, CC2-03, drew a final-decision disagreement and was settled by an independent adjudicator who read the full text and its supplement and included it.
3.6. Data extraction
Two extractors, working apart from each other and from the saved full texts only, extracted every included study onto a form specified before extraction (stage3-extraction/extraction-form.md). The form captured, per arm: citation, DOI, year, journal, country, specialty, summary type, source data (real or synthetic), model family and name, fine-tuning, retrieval and agentic flags, rater blinding, the number of summaries assessed, and the per-summary counts for each outcome, with derivation rules that allowed a count to be derived only from a printed percentage and a printed denominator whose product is a whole number. Each extractor produced 22 arm rows (21 LLM arms and 1 human comparator arm).
The two passes were compared field by field by code over 17 mapped fields across 22 arms: 374 field pairs, of which 161 differed as raw strings, a raw discrepancy rate of 43.0 percent. That headline rate needs its explanation beside it. Most raw differences were format artefacts: a blank cell against the literal code for not reported, parenthetical wording, and one pass carrying any-error values in mapping fields rather than in a dedicated column. Every numeric secondary count that both extractors reported as a number agreed exactly before settlement. Six substantive items went to a settlement reader, who re-read the source for each field and produced the settled sheet (stage3-extraction/extraction-settled.csv); the comparison file (extraction-comparison.csv) and the settlement log (settlement-log.md) are named here so the trail can be followed. Settlements changed pooled inputs in three places: the Park 2026 arms were re-keyed so that the critical-error count of 13 and the any-error count of 25 sit in the prospective pilot arm of 62 summaries; the Dhanda 2026 arms were re-keyed model by model with risk-fact omission counts of 13, 4, 5, 4, 6, 4, 2 and 4; and a derived total of 71 errors for Ganzinger 2025 was removed because it rested on multiplying a rounded mean, which the form rule does not allow. One item could not be fully resolved and is carried with its caveat: in the Park 2026 pilot, the text alternates between calling the 13 items errors and cases, and the Table 2 footnote warns that error counts do not sum to cases. The statistician worked from the settled sheet only.
3.7. Risk of bias assessment
Risk of bias was assessed in duplicate with the adapted QUADAS-2 tool for LLM clinical summarisation evaluations (adapted from QUADAS-2 with QUADAS-AI signalling questions), specified in writing before extraction (stage3-extraction/rob-form.md) and named as adapted at every mention in this manuscript. The adapted tool keeps the four-domain structure of QUADAS-2 and re-aims it at generation evaluations: Domain 1, source-data selection; Domain 2, index generation process; Domain 3, reference standard and error adjudication; Domain 4, flow and timing. Each domain is rated low, some concerns, high or no information, with a support line from the paper for every rating. Judgements from the adapted tool are not merged with judgements from unmodified QUADAS-2, RoB 2 or ROBINS-I anywhere in this review.
Two assessors rated all 11 studies on all four domains, working apart. Both returned the same overall distribution: high 9, some concerns 2, low 0. Overall judgements disagreed on 0 of 11 studies. Domain cells disagreed on 12 of 44 (Domain 1: 5; Domain 2: 1; Domain 3: 2; Domain 4: 4). An adjudicator read the papers and both rationales for all 12 cells and settled 2 cells at the first assessor judgement and 10 at the second, none at a third value (stage3-extraction/rob-settled.csv). Two overall-rule applications were checked in full: two studies (R0167 and R0370) carried three or more domains at some concerns, and the adjudicator confirmed overall some concerns for both, because the form route from that pattern to high applies only where the combination leaves selection, generation and rating unchecked, which neither study meets.
3.8. Statistical analysis
Proportions were pooled with a random-effects generalised linear mixed model with a logit link, a binomial likelihood at unit level and a normal random effect on the logit scale, estimated by maximum likelihood. The unit of analysis was the study arm (a model, condition or summary set within a study), so a study could contribute more than one unit to a pool; unit and study counts are reported together for every pool for that reason. No continuity correction and no transformation to another scale were applied. Boundary counts (0 events, or events equal to the denominator) were retained; the binomial likelihood handles them.
For each pool the report gives the pooled proportion with its 95 percent confidence interval, tau squared (between-unit variance on the logit scale), I squared (the GLMM analogue computed as tau squared divided by tau squared plus the Higgins-Thompson typical within-unit sampling variance on the logit scale), and, for pools with at least 3 units, a 95 percent prediction interval computed on the logit scale and back-transformed. Unit-level intervals in the forest plots are Clopper-Pearson exact intervals. Single-unit outcomes carry exact (Clopper-Pearson) confidence intervals. Hartung-Knapp adjustment was specified in the analysis plan for the primary pool only; the primary pool does not exist, so no Hartung-Knapp fit was fittable and none is reported. No funnel plot test, Egger test or trim-and-fill estimate was run, per the analysis plan; for single-arm proportions with this level of heterogeneity, such tests would not separate reporting bias from genuine spread.
Software disclosure. The analysis plan named R with the metafor package as the primary route. That route could not be run in the build environment: R was not installed, the package installation route timed out, and the fallback statistics library could not be installed because the package proxy returned an HTTP 403. The identical GLMM-logit model was therefore fitted with a hand-written Python implementation (Python 3.12.3, numpy 1.26.4, scipy 1.11.4), integrating the random effect with 21-point adaptive Gauss-Hermite quadrature. The R entry file in the analysis folder is an unexecuted stub that documents this; the Python file is the code that produced every number. A first, non-adaptive quadrature implementation produced a wrong likelihood surface at large tau squared. It was caught by validation against per-unit numerical integration before any number was reported, and all of its outputs were discarded. The adaptive implementation agrees with the integration route to 6 decimal places on the headline pool.
Checks. The statistician hand-checked the supplementary any-error pool end to end (the pre-specified primary pool check was transferred to it, because the primary pool does not exist): an independent recomputation by a separate route (per-unit adaptive numerical integration with a different optimiser) gave a pooled proportion of 0.6055 (95 percent CI 0.2986 to 0.8470) with tau squared 1.9649, matching the analysis script to rounding. Crude totals (the summed numerators and denominators) were re-summed from the settled sheet for all 15 fitted pools and matched the analysis inputs exactly. Eight unit proportions were recomputed by hand, including the primary unit (13 of 62) and the outlying fabrication unit (654 of 671).
3.9. Subgroup and sensitivity analyses
Subgroups were pre-specified in the protocol: summary type (discharge against non-discharge), source data (real against synthetic or benchmark records), rater blinding (blinded against unblinded or not reported), and model family. Subgroup analyses were run for the any-error and omission pools only, where unit counts allowed. A subgroup level with fewer than 2 units was not fitted; where any level had fewer than 2 units, or the pool had fewer than 5 units, results are descriptive only. Interaction tests, where run, were likelihood-ratio tests from a moderator GLMM with a common tau squared across levels. P values are unadjusted across the pre-specified families. Model family was tabulated descriptively only, because several family levels rest on a single study.
Sensitivity analyses were pre-specified: S1, restricted to studies with explicit severity or harm grading (applied to the secondary pools as the severity-graded analogue set in the analysis plan and Amendment 01); S2, excluding preprints; S3, excluding studies at high overall risk of bias on the adapted tool. One further sensitivity was required by the settled data and is reported beside the omission pool wherever it appears: exclusion of R0354, whose omission counts are specific to a single embedded risk fact per synthetic vignette rather than general omission audits.
3.10. Certainty of the evidence
Certainty was rated with GRADE, per outcome, by an assessor working from the settled extraction sheet, the settled risk of bias judgements and the settled analysis outputs, under a framework specified before results (stage4-analysis/grade-framework.md). Proportion evidence in this series starts at low certainty, because the body consists of observational evaluation studies without a comparator; the starting point is a property of the design set, and no domain is downgraded twice for the same concern. No upgrading factor was pre-committed and none was applied. Four pressure points were named in the framework before results: definition heterogeneity across studies (P1), the synthetic data share (P2), unblinded or unreported rating in Domain 3 of the adapted tool (P3), and inconsistency that pre-specified subgroups do not credibly explain (P4). Full domain-by-domain reasons for every rating are in stage4-analysis/grade-assessment.md, and the summary of findings table stands before the Background, as the series standard requires.
3.11. Deviations from the protocol
Recorded plainly: (1) Amendment 01 added the supplementary any-error pool, for the reasons in Section 3.3; (2) the statistics were fitted with the disclosed Python fallback in place of R and metafor (Section 3.8); (3) no separate extraction pilot phase was run (Section 3.3); (4) no separate forward citation-chasing pass was performed (Section 3.4); (5) OpenAlex was blocked and not searched, and seven licensed or specialist databases were not run (Section 3.4). No deviation changed an eligibility rule, an outcome definition or a pooled input after analysis began.
4 4. Results
4.1. Results of the search
Databases and registers returned 991 records (PubMed 378, Europe PMC 467, preprints through Europe PMC 140, Semantic Scholar 5, ClinicalTrials.gov 1). After 508 duplicates were removed, 483 records were screened and 425 were excluded at title and abstract. Fifty-eight reports were sought for full text, 10 were not retrieved, and 48 were assessed. Thirty-eight reports were excluded at full text: 31 with no extractable per-summary numerator and denominator, 3 with an ineligible summary type only, 2 duplicate reports of included studies, 1 with an excluded task only, and 1 with no LLM generation. Ten studies were included from this route. Citation chasing on those 10 studies produced 15 candidates; 14 were assessed (1 not retrieved), 13 were excluded and 1 was included. The review includes 11 studies, extracted as 22 arms (21 LLM arms and 1 human comparator arm). Figure 1 sets out the flow.

Three excluded reports deserve a sentence each, because readers will look for them. Williams and colleagues (JAMA Internal Medicine 2025) compared physician-written and LLM-written discharge summaries in 100 summaries and reported mean errors per summary (2.91 for LLM summaries and 1.82 for physician summaries) but no count of summaries with at least one error, so the study could not enter any proportion pool, and it could not enter the errors per summary pool either, because that outcome was specified for included studies. Van Veen and colleagues (Nature Medicine 2024) reported rating-scale comparisons rather than per-summary error counts. Asgari and colleagues (npj Digital Medicine 2025) presented a framework for assessing safety and hallucination rates rather than a per-summary count set that met the eligibility rules. The dominant exclusion reason tells the same story at scale: 31 of 38 full-text exclusions were for want of an extractable per-summary numerator and denominator.
4.2. Characteristics of included studies
Table 2 lists the 11 studies. Publication years run from 2024 to 2026. Two studies are preprints posted on Research Square (Dhanda 2026 and Jolin-Rodrigo 2026). Bodanki 2026 was extracted from its medRxiv preprint (posted 15 September 2025); a journal version has since appeared in the Journal of the American Medical Informatics Association, and the reference list cites both, with the citation correction logged in the team log. Summary types cover emergency department encounter summaries (3 studies), clinic-visit summaries (3 studies), hospital discharge summaries (2 studies), hospital-course narratives (2 studies) and referral letters (1 study), with one study (Grolleau 2026) producing hospital-course drafts for use in discharge summaries. Source data were real records in 8 studies and synthetic records in 3. Six studies carry at least one GPT family arm, with Llama, Gemini, Gemma and Qwen arms elsewhere, and one vendor-managed service whose underlying model was not stated (Nakano 2026). Study sizes, counted as summaries assessed across arms, ran from 11 (Nakano 2026) to 671 (Bodanki 2026).
Table 2. Characteristics of included studies
| Study | Year | Country | Summary type | Source data | Model (arms) | Summaries assessed (total across arms) | Error outcomes reported per summary |
|---|---|---|---|---|---|---|---|
| Song | 2025 | South Korea | ED encounter summary | Real | Ko-Llama3-Luxia-8B (Y-KNOT-EDN system) | 50 | Any error; omission; fabrication (label: confabulation) |
| Tung | 2024 | Singapore | Clinic-visit summary | Synthetic | GPT-4; junior clinicians (human arm) | 15 | Fabrication (LLM arm); inaccuracy (human arm) |
| Ganzinger | 2025 | Germany | Hospital discharge summary | Real | LLaMA3 | 25 | Errors per summary (mean and SD) |
| Watcharasarn | 2025 | Thailand | Referral letter | Synthetic | GPT-4o | 50 | Omission (label: content gaps) |
| Williams | 2025 | United States | ED encounter summary | Real | GPT-4; GPT-3.5-turbo | 200 | Any error; omission; fabrication (label: hallucination); inaccuracy |
| Park | 2026 | United States | Clinic-visit summary | Real | GPT-4o retrieval-augmented agent (2 arms: initial evaluation, prospective pilot) | 112 | Clinically significant error and any error (pilot arm only) |
| Dhanda | 2026 | India (author institution only) | Clinic-visit summary | Synthetic | llama3.2:1b; gemma3:4b; qwen2.5; gemma2 (8 arms, 2 instruction conditions each) | 400 | Omission, risk-fact-specific only |
| Jolin-Rodrigo | 2026 | Spain | ED encounter summary | Real | gemini-2.5-flash; gpt-oss-20b | 200 | Omission; fabrication (label: hallucinations); inaccuracy |
| Nakano | 2026 | Japan | Hospital discharge summary | Real | Ubie generative AI (underlying model not stated) | 11 | Any error; omission; errors per summary |
| Bodanki | 2026 | United States | Hospital-course narrative | Real | GPT-4 (Epic-embedded system) | 671 | Fabrication (summaries with at least one unsupported statement) |
| Grolleau | 2026 | United States | Hospital-course narrative | Real | Gemini 2.5 Pro | 100 | Omission; fabrication (label: hallucination); inaccuracy; harm at least moderate |
Study identifiers used in the analysis files: Song CC2-03; Tung R0047; Ganzinger R0123; Watcharasarn R0144; Williams R0167; Park R0301; Dhanda R0354; Jolin-Rodrigo R0370; Nakano R0391; Bodanki R0429; Grolleau R0438. Full citations are in the references. Williams 2025 in this table is the PLOS Digital Health emergency department study; the JAMA Internal Medicine discharge summary study (Williams CYK et al., a distinct study with the same lead author) was excluded at full text (Section 4.1).
4.3. Risk of bias in included studies
Table 3 gives the settled judgements from the adapted QUADAS-2 tool for LLM clinical summarisation evaluations (adapted from QUADAS-2 with QUADAS-AI signalling questions). Nine studies are at high overall risk of bias, 2 (Williams R0167 and Jolin-Rodrigo R0370) are at some concerns, and none is at low risk. Domain 3, reference standard and error adjudication, carries the heaviest pattern: 7 studies at high and 1 at no information, which matters because every numerator in this review is the product of a rating process. Domain 1, source-data selection, is at high for 3 studies. No study is at low risk in Domain 3.
Table 3. Risk of bias summary (adapted tool, named as adapted)
| Study | Domain 1: source-data selection | Domain 2: index generation process | Domain 3: reference standard and error adjudication | Domain 4: flow and timing | Overall |
|---|---|---|---|---|---|
| Song (CC2-03) | Some concerns | Some concerns | No information | Low | High |
| Tung (R0047) | High | Some concerns | Some concerns | Some concerns | High |
| Ganzinger (R0123) | High | Some concerns | High | Low | High |
| Watcharasarn (R0144) | High | Some concerns | High | Some concerns | High |
| Williams (R0167) | Some concerns | Some concerns | Some concerns | Low | Some concerns |
| Park (R0301) | Some concerns | Some concerns | High | High | High |
| Dhanda (R0354) | Some concerns | Some concerns | High | Low | High |
| Jolin-Rodrigo (R0370) | Some concerns | Some concerns | Some concerns | Some concerns | Some concerns |
| Nakano (R0391) | Some concerns | No information | High | Some concerns | High |
| Bodanki (R0429) | Some concerns | Some concerns | High | Some concerns | High |
| Grolleau (R0438) | Low | Some concerns | High | High | High |
Tool name, in full, at every mention: adapted QUADAS-2 tool for LLM clinical summarisation evaluations (adapted from QUADAS-2 with QUADAS-AI signalling questions). Song (CC2-03) reaches overall high through Domain 3 at no information under the form overall rule. Williams (R0167) and Jolin-Rodrigo (R0370) each carry three or more domains at some concerns; the adjudicator confirmed overall some concerns for both, for the reason given in Section 3.7.
4.4. Results summary
Table 4 collects the main result for each outcome. The sections that follow report each outcome in protocol order, with its sensitivities and, where fitted, its subgroups.
Table 4. Results summary per outcome (main rows)
| Outcome | Units | Studies | Summaries | Events (crude) | Pooled proportion, percent (95 percent CI) | Tau squared | I squared, percent | Prediction interval, percent |
|---|---|---|---|---|---|---|---|---|
| Clinically significant error (primary) | 1 | 1 | 62 | 13 | Not pooled. Single unit 21.0 (11.7 to 33.2) | Not applicable | Not applicable | Not applicable |
| Any error of any audited type (supplementary, post hoc, Amendment 01) | 5 | 4 | 323 | 198 | 60.6 (29.8 to 84.7) | 1.96 | 95.0 | 6.9 to 97.0 |
| Omission | 16 | 7 | 1011 | 323 | 20.7 (11.5 to 34.6) | 1.84 | 94.6 | 1.6 to 80.4 |
| Omission, excluding R0354 | 8 | 6 | 611 | 281 | 40.6 (22.0 to 62.4) | 1.49 | 95.5 | 5.1 to 89.8 |
| Fabrication or addition | 8 | 6 | 1226 | 787 | 17.6 (2.9 to 60.0) | 7.10 | 98.6 | 0.1 to 98.3 |
| Inaccuracy | 5 | 3 | 500 | 153 | 27.9 (14.6 to 46.7) | 0.81 | 93.4 | 5.2 to 73.0 |
| Harm at least moderate | 1 | 1 | 100 | 1 | Not pooled. Single unit 1.0 (0.03 to 5.4) | Not applicable | Not applicable | Not applicable |
| Errors per summary | 2 | 2 | 36 | Not applicable | Not pooled (narrative, Section 4.11) | Not applicable | Not applicable | Not applicable |
| LLM versus human contrast | 0 | 0 | 0 | Not applicable | Not fittable (Section 4.12) | Not applicable | Not applicable | Not applicable |
Single-unit confidence intervals are exact (Clopper-Pearson). Pooled estimates are from the random-effects GLMM described in Section 3.8. Events are the crude summed numerators across units and are shown as a check figure; the pooled estimate is not the crude proportion, and the two differ most where unit sizes differ most (Sections 4.7 and 4.8).
4.5. Reporting-gap analysis: the 31 full-text exclusions for missing per-summary counts (FT-R7)
The largest single result of the full-text stage is a reporting gap, and it is reported here as a result, with its own table, under a presentation decision recorded in the team log. It changes no eligibility rule and no pool: every study in this section stayed excluded, under the pre-specified rule F6, which required an extractable per-summary numerator and denominator.
Of the 48 reports assessed at full text, 31 (65 percent) were excluded under FT-R7 because they reported no extractable per-summary error count. Those 31 reports cover 27 studies; four reports are second versions of another report in the set (two preprint and journal pairs, and the Van Veen and Shemtob pairs counted the same way). Set against the included set, the pattern completes itself: of the 11 included studies, only 1 (Park 2026) reported the primary outcome, the proportion of summaries with at least one clinically significant error. The literature that was screened for this review therefore divides into a large majority that evaluated generated summaries without ever counting errors per summary, and a small minority that did.
Table 5 classifies all 31 FT-R7 reports by what they reported instead, taken from the full-text decisions file and the full-text notes. Rating scores and preference judgements were the most common substitute (10 reports), followed by similarity and automatic metrics (6), error-instance counts that are not distributed by summary (6), and item-level or field-level counts (6). Means per summary were the least common (3 reports), and they are the closest miss, because a mean sits one reporting step from the count this review needed.
Table 5. The 31 FT-R7 reports, classified by how errors were reported instead of per-summary counts
| Report (record ID) | Class | What was reported instead |
|---|---|---|
| Williams 2025, JAMA Internal Medicine (R0231) | Means per summary | Mean 2.91 (SD 2.54) errors per LLM summary against 1.82 (1.94) per physician summary; per-type means 0.93 inaccuracies, 1.75 omissions, 0.23 hallucinations. Worked example, below |
| Williams 2024, medRxiv preprint of R0231 (R0082) | Means per summary | The same study and the same results in preprint form |
| Osborne 2026 (R0336) | Means per summary | 138 error annotations across both summary types, reported as means only: LLM 0.42 (SD 0.98) against human 0.73 (1.34) errors per summary |
| Van Veen 2024, Nature Medicine (R0028) | Rating scores and preference | Reader study: human preferred 19 percent, GPT-4 non-inferior 45 percent, GPT-4 preferred 36 percent; comparative correctness ratings |
| Van Veen 2023, preprint of R0028 (R0005) | Rating scores and preference | The same reader study in preprint form |
| Caterson 2024 (R0033) | Rating scores and preference | Accuracy rated 0 to 10 (8.7 against 7.3); omissions and insertions described without counts |
| Clough 2024 (R0097) | Rating scores and preference | All 25 summaries judged acceptable quality; mean compliance 97 percent with a minimum dataset |
| Oliveira 2025 (R0151) | Rating scores and preference | Feedback percentages per evaluation response (for example 29.6 percent incorrect), with response-level denominators |
| Takayama 2025 (R0165) | Rating scores and preference | Expert domain scores, including a Presence of Hallucinations domain, over 48 summaries and 48 referrals |
| Vasilev 2025 (R0169) | Rating scores and preference | Binary per-summary hallucination ratings converted to 1 or 5 point scores; only rounded mean scores published (3.8, SD 1.9) |
| Golchini 2025 (R0232) | Rating scores and preference | Five-point Likert means for accuracy, completeness and utility over 99 encounters; preference 50.5 percent LLM; errors qualitative |
| Han 2026 (R0366) | Rating scores and preference | Four-point Likert ratings by physicians and parents over 50 summaries, omission carried as a rating item |
| Carenzo 2026 (R0455) | Rating scores and preference | Global confidence rating (8 of 9 positive) and section-level completeness and accuracy ratings |
| Chuang 2024 (R0092) | Similarity and automatic metrics | ROUGE, BERTScore and prompt variance; no human error assessment at summary level |
| Dorfner 2025 (R0170) | Similarity and automatic metrics | Benchmark accuracy percentages per task; summarisation scored by task accuracy |
| Kruse 2025 (R0209) | Similarity and automatic metrics | Automatic metrics and a qualitative expert review of 10 output pairs; no counts |
| Kruse 2025, preprint pair of R0209 (R0262) | Similarity and automatic metrics | ROUGE and BERTScore, Likert criteria on a subset; error analysis stated as not intended to yield quantitative results |
| Zhao 2026 (R0349) | Similarity and automatic metrics | ROUGE, BERTScore, BLEU and Jaro-Winkler, authorship classification, preference testing; hallucinations qualitative |
| Akinwumi 2026 (R0399) | Similarity and automatic metrics | ROUGE-L about 0.43 and BERTScore 0.61; error rates quoted only from other studies |
| Urquhart 2024 (R0023) | Error-instance counts | Four hallucination instances in total across 18 test summaries, not distributed by summary; event capture per clinical event, 41.5 percent |
| Osborne 2024 (R0096) | Error-instance counts | Fabrication and omission instances over 28 human and AI summaries combined (omissions 13 against 16); denominator mixes human and AI summaries |
| Shemtob 2025, JAMIA Open (R0142) | Error-instance counts | 11 incidences of hallucination or inaccuracy across 70 AI summaries; incidences are not summaries with errors |
| Shemtob 2025, preprint of R0142 (R0141) | Error-instance counts | The same study in preprint form |
| Kahl 2026 (R0372) | Error-instance counts | Coded feedback comments over 147 summaries (omissions 46, hallucinations 5); multiple comments per summary possible |
| Lyo 2026 (R0414) | Error-instance counts | 25 hallucination instances across 90 cases, alongside Likert ratings and preference |
| Wang 2025 (R0116) | Item or field-level counts | Extraction accuracy 96.5 percent with precision, recall and F1 per extracted data item across 23 records |
| Liang 2025 (R0184) | Item or field-level counts | False positives counted per question item (29 returned, 8 confirmed); summary quality on a five-point scale |
| Indrakanti 2025 (R0202) | Item or field-level counts | True and false positives per data point by section; three incomplete cases and one wrong case mentioned, with no combinable union |
| Singh 2026 (R0356) | Item or field-level counts | Errors per field-dimension pair over 401 admissions (omissions 1010 in LLM summaries); per-patient mean burden 2.484, falling to 1.807 after prompt refinement |
| Deschepper 2026 (R0374) | Item or field-level counts | Percentages of item responses across 61 layout items and 33 content items rated by 31 clinicians; denominators are item responses |
| Grolleau 2026, framework paper (R0422) | Item or field-level counts | Fact-level presence proportions (48 to 65 percent) and LLM-jury agreement with a physician panel (kappa 81 and 75 percent) |
The worked example is Williams 2025 (R0231, with its preprint R0082). That study did the hard part of the work this review values: 100 encounters, LLM and physician narratives, blinded attending reviewers, errors counted and typed. It reported the result as means: 2.91 (SD 2.54) unique errors per LLM narrative against 1.82 (1.94) per physician narrative, with per-type means of 0.93 inaccuracies, 1.75 omissions and 0.23 hallucinations per LLM narrative. What it never reported is the number of narratives that contained at least one error. A mean cannot be turned back into that count: a mean of 2.91 is consistent with every narrative carrying errors and with a minority carrying many. The study was therefore excluded under the pre-specified F6 rule, in both its reports, and it could not enter the errors per summary pool either, because that outcome was specified for included studies. One extra column in its results table would have made it the largest study in this review and the source of the matched LLM against human contrast that Section 4.12 records as missing.
Two of the classes in Table 5 deserve a distinction. Vasilev 2025 (R0169) collected binary per-summary hallucination ratings and then published them only as converted mean scores, so the count existed and was lost at the reporting step. Singh 2026 (R0356) counted errors at field level by design, across 401 admissions, and its per-patient mean burden (2.484 omissions per patient in LLM summaries) answers a field-level question that a per-summary count would not answer better. The reporting gap is therefore not a single failure repeated 31 times. It is a set of reasonable measurement choices, made study by study, whose combined effect is that the per-summary risk of a clinically significant error, the quantity a service lead needs, is almost nowhere on the record.
4.6. Primary outcome: summaries with at least one clinically significant error (not pooled)
One unit in the entire included set reports this outcome: the prospective pilot arm of Park 2026 (R0301-2), a GPT-4o retrieval-augmented agent summarising breast surgical oncology clinic records, in which 13 of 62 summaries (21.0 percent, exact 95 percent CI 11.7 to 33.2) carried an error the study classified as critical and treatment-impacting. No pooled estimate exists, and none is presented.
The single count carries a caveat that settlement could not remove. The paper alternates between calling the 13 items errors and cases, and the footnote to its Table 2 warns that error counts do not sum to cases with errors. The count is carried as printed, under the abstract survey-case framing. If some cases carried more than one critical error, the true number of summaries with at least one critical error would be lower than 13, and the text alone cannot settle the point. The pilot raters were unblinded, and the study is at high overall risk of bias on the adapted tool, with Domain 3 at high. GRADE certainty for this outcome is very low (Section 4.14).
One further study, Nakano 2026 (R0391), defined a major, clinically significant threshold but reported major counts at error level only (11 major instances), never as the number of summaries with at least one major error. It contributes no unit. The remaining 9 studies did not grade clinical significance per summary at all. The not pooled verdict is a result about the literature, and the discussion treats it as one.
4.7. Supplementary pool: summaries with at least one error of any audited type (post hoc, Amendment 01)
This pool is supplementary and post hoc under Amendment 01. It pools a different question from the primary outcome: any error of any audited type, under each study own audit definition.
Five units from 4 studies, covering 323 summaries, gave a pooled proportion of 60.6 percent (95 percent CI 29.8 to 84.7), with tau squared 1.96, I squared 95.0 percent and a prediction interval of 6.9 to 97.0 percent. The units were Song 2025 (7 of 50), Williams 2025 GPT-4 (67 of 100), Williams 2025 GPT-3.5 (90 of 100), Park 2026 pilot (25 of 62) and Nakano 2026 (9 of 11). Unit proportions run from 14.0 percent to 90.0 percent, and the pooled figure should be read as a frequency summary across very different audit definitions, not as the error rate of any single system. The crude total was 198 of 323 (61.3 percent). GRADE certainty is very low.

Sensitivities for this pool: S1, the severity-graded analogue (4 units, 3 studies, 273 summaries), gave 72.5 percent (95 percent CI 48.7 to 88.0), tau squared 0.92, I squared 90.6 percent, prediction interval 23.7 to 95.7 percent. S2, excluding preprints, is identical to the main pool (60.6 percent, 95 percent CI 29.8 to 84.7), because no unit in this pool is a preprint; that identity is stated rather than presented as an independent result. S3, excluding high risk of bias, keeps only the 2 Williams 2025 units (1 study, 200 summaries) and gives 80.7 percent (95 percent CI 59.5 to 92.3); a 2-unit, single-study sensitivity describes that study more than it tests the pool.
Subgroups for this pool were descriptive only. Discharge against non-discharge could not be tested (the discharge level holds 1 unit, Nakano 2026, 9 of 11). Source data could not be tested (all 5 units used real source data). Blinding could not be tested (no unit was blinded; 1 unit was unblinded and 4 did not report blinding). Model family, descriptively: GPT family 3 units, crude 182 of 262 (69.5 percent); Llama family 1 unit, 7 of 50 (14.0 percent); unclear family 1 unit, 9 of 11 (81.8 percent). No claim is made from any of these splits.
4.8. Omission
Sixteen units from 7 studies, covering 1011 summaries, gave a pooled proportion of 20.7 percent (95 percent CI 11.5 to 34.6), with tau squared 1.84, I squared 94.6 percent and a prediction interval of 1.6 to 80.4 percent. The crude total was 323 of 1011 (31.9 percent). The crude figure sits above the pooled estimate because the GLMM weights units toward equality, so the 8 small Dhanda 2026 units at low proportions offset the larger high-proportion units that dominate the crude total. GRADE certainty is very low.
Eight of the 16 units are the Dhanda 2026 arms, and their counts are risk-fact-specific: each synthetic vignette held one embedded safety-critical fact, and the count is the number of summaries that dropped that fact. The union with routine-fact omission is not reported in that study. The main pool therefore mixes fact-specific omission with general omission audits. The required sensitivity that removes those units gives 40.6 percent (95 percent CI 22.0 to 62.4; 8 units, 6 studies, 611 summaries; tau squared 1.49; I squared 95.5 percent; prediction interval 5.1 to 89.8 percent). The near-doubling on removal of one study counting rule is itself the clearest sign of what the main figure mixes, and the two figures are reported together wherever the omission result is used.


Sensitivities: S1, severity-graded analogue (6 units, 4 studies, 511 summaries), gave 41.9 percent (95 percent CI 22.0 to 64.9). S2, excluding preprints (6 units, 5 studies, 411 summaries), gave 34.2 percent (95 percent CI 17.8 to 55.6); this restriction removes all 8 Dhanda 2026 arms and both Jolin-Rodrigo 2026 arms, and its rise against the main pool runs in the same direction as the Dhanda exclusion above. S3, excluding high risk of bias (4 units, 2 studies, 400 summaries), gave 55.0 percent (95 percent CI 32.0 to 76.1).
Subgroups, with interaction tests from the moderator model: discharge (2 units, moderator pooled 15.7 percent, 95 percent CI 2.2 to 60.2) against non-discharge (14 units, 21.5 percent, 95 percent CI 11.5 to 36.5) gave a likelihood-ratio p value of 0.734, and no subgroup difference is claimed. Source data gave real (7 units, 36.1 percent, 95 percent CI 18.5 to 58.4) against synthetic (9 units, 12.7 percent, 95 percent CI 6.0 to 25.0), p = 0.044. Blinding gave blinded (2 units, both Jolin-Rodrigo 2026 arms, 61.7 percent, 95 percent CI 23.6 to 89.4) against unblinded or not reported (14 units, 16.8 percent, 95 percent CI 9.5 to 28.1), p = 0.032. Both of the smaller p values are unadjusted, each rests on a level built from 1 or 2 studies, and each is confounded with study: the synthetic level is Dhanda 2026 plus one further study, and the blinded level is a single study. Both are reported as hypothesis-generating only, and neither is promoted to a headline. Model family was descriptive only: GPT 4 units, crude 219 of 350 (62.6 percent); Gemini 2 units, 55 of 200 (27.5 percent); Llama 3 units, 23 of 150 (15.3 percent); Qwen 2 units, 10 of 100 (10.0 percent); Gemma 4 units, 15 of 200 (7.5 percent); unclear 1 unit, 1 of 11 (9.1 percent).
4.9. Fabrication or addition
Eight units from 6 studies, covering 1226 summaries, gave a pooled proportion of 17.6 percent (95 percent CI 2.9 to 60.0), with tau squared 7.10, I squared 98.6 percent and a prediction interval of 0.1 to 98.3 percent. GRADE certainty is very low.
This pool carries a warning that the brief for the analysis stated and this manuscript repeats in full. One unit, Bodanki 2026 (R0429), an Epic-embedded GPT-4 system reported as a preprint, counts summaries with at least one unsupported statement at the level of atomic content units, and reports 654 of 671 summaries (97.5 percent) at or above that bar. Most other units sit below 25 percent: the range across units runs from 0.0 percent (Tung 2024, 0 of 5, a retained boundary count) to 97.5 percent. The crude total is 787 of 1226 (64.2 percent), dominated by the Bodanki unit, which contributes 671 of the 1226 summaries and 654 of the 787 events. The GLMM, weighting units toward equality, pulls the pooled estimate to 17.6 percent with an interval that spans most of the scale. With tau squared of 7.10 and a prediction interval of 0.1 to 98.3 percent, the single pooled figure is close to uninterpretable as a frequency. The unit-level figures in Figure 5 carry the information in this pool, and the pooled number should not be quoted without them.

Sensitivities: S2, excluding preprints (5 units, 4 studies, 355 summaries), gave 9.7 percent (95 percent CI 1.3 to 46.6), tau squared 4.48, I squared 97.1 percent, prediction interval 0.1 to 91.8 percent. Removing the Bodanki unit halves the crude proportion (30.7 percent against 64.2 percent) while the pooled estimate barely moves, which shows that the spread of the remaining units, not the single outlier alone, drives the interval width. S3, excluding high risk of bias (4 units, 2 studies, 400 summaries), gave 25.9 percent (95 percent CI 8.2 to 57.7). S1 was not run for this pool: the analysis plan runs S1 for the primary outcome, and S1 analogues were run for the any-error and omission pools under Amendment 01 and the analysis specification. The absence is a recorded decision, not a gap in the file.
4.10. Inaccuracy
Five units from 3 studies (Williams 2025, Jolin-Rodrigo 2026 and Grolleau 2026), covering 500 summaries, gave a pooled proportion of 27.9 percent (95 percent CI 14.6 to 46.7), with tau squared 0.81, I squared 93.4 percent and a prediction interval of 5.2 to 73.0 percent. The crude total was 153 of 500 (30.6 percent). This is a small pool, and its tau squared is imprecisely estimated. The one human arm in the dataset (Tung 2024 human arm, inaccuracy 0 of 10) enters no LLM pool and is excluded here by design. GRADE certainty is very low.

Sensitivities: S2, excluding preprints (3 units, 2 studies, 300 summaries), gave 20.3 percent (95 percent CI 10.7 to 35.1). S3, excluding high risk of bias (4 units, 2 studies, 400 summaries), gave 30.1 percent (95 percent CI 13.8 to 53.7). No subgroup analysis was run for this pool.
4.11. Harm at least moderate, and errors per summary
Harm at least moderate (not pooled). One unit reports this outcome per summary: Grolleau 2026 (R0438), Gemini 2.5 Pro hospital-course narratives, in which 1 of 100 summaries (1.0 percent, exact 95 percent CI 0.03 to 5.4) was rated likely to cause moderate harm (AHRQ harm level 3 in the paper numbering). Levels 4 and 5 were explicitly zero in that study. Williams 2025 graded harm per error, not per summary, so no second unit exists. The estimate rests on a single event in a single study at high overall risk of bias, with unblinded rating. One rating decision changed would double the count or reduce it to zero. GRADE certainty is very low, and the low event count is not evidence that harm is rare across systems; it is one rated summary in one study.
Errors per summary (not pooled). Two units report per-summary error means on a comparable counting unit. Ganzinger 2025 (R0123) reports a mean of 2.84 errors per summary (SD 1.71, n 25). Nakano 2026 (R0391) reports 20 total errors over 11 summaries, a mean of 1.82, with a median of 2.0, an interquartile range of 1.0 to 2.0 and a range of 0 to 5, but no SD or standard error. Only one unit carries a dispersion measure, so no pooled mean can be computed without inventing a variance for the second. The outcome is reported narratively, as here. The Bodanki 2026 figure of 7.11 is excluded from this outcome in full: it is a mean per statement (4770 unsupported atomic content units out of 40,452 statements), a different counting unit from errors per summary, and pooling it with per-summary counts would mix two different quantities. GRADE certainty for the narrative body is very low.
4.12. LLM versus human contrast (not fittable)
No study reports matched LLM and human per-summary error counts for the same source records under the same outcome definition, so the contrast is not fittable and no cross-study comparison is presented as a contrast. The detail shows how close the literature comes without reaching the comparison. Tung 2024 reports LLM fabrication (0 of 5) but not human fabrication, and human inaccuracy (0 of 10) but not LLM inaccuracy, so no matched outcome pair exists in that study. In Ganzinger 2025 the physician summaries were used for similarity metrics and were not error rated. In Park 2026 the human summaries were the comparison reference and their errors were not counted. The single human arm in the dataset enters no pool. GRADE: not rateable, because there is no body of evidence to rate. The absence is reported as a gap, not as a finding of no difference.
4.13. Summary of sensitivity and subgroup findings
Across pools, the sensitivity analyses move the estimates without narrowing them. The any-error pool rises under S1 (72.5 percent) and under S3 (80.7 percent, a single study). The omission pool rises under every restriction (S1 41.9 percent, S2 34.2 percent, S3 55.0 percent, Dhanda exclusion 40.6 percent), each time because low-rate units leave the pool. The fabrication pool barely moves when its dominant crude contributor leaves (S2 9.7 percent against a main 17.6 percent, on a much lower crude base). The inaccuracy pool moves least (S2 20.3 percent, S3 30.1 percent against a main 27.9 percent). No sensitivity analysis produces a narrow interval, and none changes a GRADE rating. The subgroup record is in Sections 4.6 and 4.7: one null interaction (discharge, p = 0.734), and two hypothesis-generating interactions for omission (source data, p = 0.044; blinding, p = 0.032), each unadjusted and each confounded with study.
4.14. Certainty of the evidence
Every rateable outcome is rated very low certainty, and the LLM versus human contrast is not rateable. All proportion outcomes start at low under the GRADE approach for prevalence and proportion evidence used in this series. The downgrades that follow are named per outcome in the GRADE assessment (stage4-analysis/grade-assessment.md) and summarised here. Risk of bias pressed on every outcome except inaccuracy: 9 of 11 studies are at high overall risk on the adapted tool, rating was blinded in only 2 of 16 omission units, and the single-unit outcomes rest entirely on unblinded, high Domain 3 rating processes. Inconsistency pressed hardest: I squared is above 93 percent in every pool, tau squared reaches 7.10 in the fabrication pool, and prediction intervals span most of the proportion scale, with no subgroup split that credibly explains the spread. Indirectness pressed through definition heterogeneity (studies counting different events under the same error labels, and, in the omission pool, 8 of 16 units counting a single risk fact) and, for omission, through a synthetic share of 450 of 1011 summaries (44.5 percent). Imprecision pressed on the single-unit outcomes (62 summaries and 13 events for the primary; 1 event in 100 summaries for harm) and on the wide pooled intervals elsewhere. No upgrade was applied to any outcome.
5 5. Discussion
5.1. Principal findings
This review set out to pool one number: the proportion of LLM-generated clinical summaries containing at least one clinically significant error. It could not. Of 11 included studies, 1 reported that outcome per summary (Park 2026, prospective pilot: 13 of 62 summaries, 21.0 percent, exact 95 percent CI 11.7 to 33.2), and that single count is ambiguous in its own source text between errors and cases. Every other pooled figure in this review answers an easier question. The supplementary any-error pool (post hoc, Amendment 01) gave 60.6 percent (95 percent CI 29.8 to 84.7; 5 units, 4 studies, 323 summaries). Omission gave 20.7 percent (95 percent CI 11.5 to 34.6; 16 units, 7 studies, 1011 summaries), or 40.6 percent (95 percent CI 22.0 to 62.4) without the risk-fact-specific units. Fabrication or addition gave 17.6 percent (95 percent CI 2.9 to 60.0; 8 units, 6 studies, 1226 summaries), with a prediction interval of 0.1 to 98.3 percent that makes the single figure close to uninterpretable as a frequency. Inaccuracy gave 27.9 percent (95 percent CI 14.6 to 46.7; 5 units, 3 studies, 500 summaries). Harm of at least moderate level rests on 1 rated summary in 1 study (1 of 100, 1.0 percent, exact 95 percent CI 0.03 to 5.4). No study reported a matched LLM versus human comparison. Certainty is very low for every rateable outcome.
5.2. Frequency and severity: the central tension
The studies in this review counted errors often and graded them rarely, and that imbalance is the central tension in the results. A count of any error, however small, is a frequency measure. In the supplementary pool it reaches 60.6 percent (95 percent CI 29.8 to 84.7), and in the severity-graded sensitivity of the same pool it reaches 72.5 percent (95 percent CI 48.7 to 88.0). A count of errors graded clinically significant is a severity measure, and the literature supplies exactly one per-summary version of it: 21.0 percent (95 percent CI 11.7 to 33.2), from 62 summaries in one pilot. A reader who takes the frequency figures as severity figures will overstate danger. A reader who takes the single severity figure as settled will overstate knowledge, because its interval runs from about 1 summary in 9 to about 1 summary in 3 and its numerator may count error instances rather than summaries. The harm outcome shows the same tension from the other side: 1 of 100 summaries rated likely to cause moderate harm (1.0 percent, exact 95 percent CI 0.03 to 5.4) is a small number built from a single rating decision, and it cannot carry a reassurance. Frequency is what the field measures. Severity is what services need. The distance between the two, in this literature, is the finding.
5.3. Why the primary outcome failed to pool is a finding
The primary outcome failed to pool for a plain reason: study reports do not carry the count. The full-text exclusion record shows the same pattern before inclusion is even reached: of the 48 reports assessed at full text, 31 (65 percent) reported no extractable per-summary error count and were excluded under FT-R7, and Section 4.5 classifies all 31 by what they reported instead. The discharge summary study by Williams and colleagues (JAMA Internal Medicine 2025) is the clearest case: a careful blinded comparison that reports mean errors per summary (2.91 LLM, 1.82 physician) and so can enter no proportion pool in this review. Inside the included set, Nakano 2026 defined a major threshold and then reported it at error level only (11 major instances across 11 summaries), one reporting step away from contributing the second primary unit this review needed. Park 2026 reported the count and left its unit ambiguous. A literature that evaluates safety-relevant systems without routinely reporting how many summaries contained a serious error has a reporting gap, and the gap is measurable: 1 of 11 studies. This review treats that count as a result, stated in the abstract and the summary of findings, rather than as a defect to be worked around with a quieter headline.
5.4. What the pooled figures can and cannot carry
Heterogeneity is very high in every pool: I squared 95.0 percent for any error, 94.6 percent for omission, 98.6 percent for fabrication or addition and 93.4 percent for inaccuracy, with prediction intervals of 6.9 to 97.0, 1.6 to 80.4, 0.1 to 98.3 and 5.2 to 73.0 percent respectively. A prediction interval that covers most of the scale says that a new study, in a new setting with a new system, could find almost any proportion. The pooled means are therefore summaries of a spread, and the spread is the information. The fabrication pool makes the point in its sharpest form: unit proportions run from 0.0 percent to 97.5 percent because the counting units differ, with one Epic-embedded system study (Bodanki 2026, extracted from its preprint) counting summaries with at least one unsupported atomic content unit at 654 of 671, while other studies counted summary-level hallucination or confabulation labels and sat below 25 percent. Pooling across counting units produces a number, 17.6 percent (95 percent CI 2.9 to 60.0), that describes no study in the pool well. The omission pool has the same structure in a milder form: 8 risk-fact-specific units at 2 to 13 events per 50 summaries pull the pooled estimate to 20.7 percent, and removing them gives 40.6 percent, so the honest report is the pair.
5.5. Risk of bias and the rating problem
Every numerator in this review is a judgement made by a rater against a source record, and the quality of that judgement process is where the evidence is weakest. On the adapted QUADAS-2 tool for LLM clinical summarisation evaluations (adapted from QUADAS-2 with QUADAS-AI signalling questions), 9 of 11 studies are at high overall risk of bias, 2 are at some concerns and none is at low risk, with Domain 3 (reference standard and error adjudication) at high for 7 studies and at no information for 1. Blinding is the sharpest gap: in the omission pool, 2 of 16 units were rated blind, 1 was unblinded and 13 do not report blinding. The blinding subgroup interaction for omission (blinded 61.7 percent against unblinded or not reported 16.8 percent, p = 0.032) rests on a single blinded study and is confounded with it, so it cannot show that blinding changes counts; it can only underline that the field rarely reports the safeguard at all. Sensitivity S3, which keeps only the 2 studies at some concerns, moves every pool it touches (any error 80.7 percent, omission 55.0 percent, fabrication 25.9 percent, inaccuracy 30.1 percent) without narrowing any interval, because 2 studies cannot carry a pool.
5.6. Relation to other work
This review differs from the registered discharge-summary review (PROSPERO CRD420261349028) in scope, and the difference shows in the data: the discharge subgroup in the omission pool holds 2 units and its interaction test is null (p = 0.734), while the non-discharge stratum, which the registered review excludes by design, supplies 14 of the 16 omission units and most of the other pools. The two reviews will therefore answer different questions even where their study sets overlap. Against the wider primary literature, the pattern in Section 5.3 holds: the three anchor studies a reader is most likely to expect here (Williams, JAMA Internal Medicine 2025; Van Veen 2024; Asgari 2025) were each excluded at full text because their reports do not yield a per-summary error numerator and denominator. That is a statement about reporting formats, not about the quality of those studies.
5.7. Strengths and limitations
Strengths, stated with their evidence: the protocol and every instrument (screening rules, extraction form, the adapted risk of bias tool, the GRADE framework, the analysis plan) were written before the stage they govern; screening, extraction and risk of bias were each done twice independently, with agreement measured (screening kappa 0.62 three-category and 0.67 binary) and every disagreement settled by reading; the extraction discrepancy rate (43.0 percent raw by field, mostly format artefacts, with all numeric outcome counts in agreement before settlement) is reported rather than hidden; the pooled any-error estimate was recomputed by an independent numerical route and matched to rounding; and the one amendment was logged, dated and labelled before analysis.
Limitations, stated plainly. OpenAlex was blocked by a provider rate limit and was not searched. Embase, Scopus, Web of Science, CINAHL, CENTRAL, IEEE Xplore and the ACM Digital Library were not run, because no access route was available; ready strings are in the search log, and a review that runs them may find studies this one missed. Ten of 58 full texts sought were not retrieved, and any of them could hold an eligible study. No separate forward citation-chasing pass was performed. The primary outcome could not be pooled because the literature almost never reports a study-defined clinically significant count per summary, and the one included count (Park 2026) is ambiguous between errors and cases. The fabrication pool is dominated by one outlying unit. Heterogeneity is very high in every pool (I squared above 93 percent) and prediction intervals span most of the scale, so pooled figures are spread summaries, not system properties. Statistics were fitted with a hand-written Python GLMM (scipy, adaptive Gauss-Hermite quadrature) because R was unavailable in the build environment; the pooled any-error estimate was independently recomputed by a second numerical route and matched, and the analysis code is published with the review package. The review team was an LLM team with one accountable human author; that arrangement is disclosed in the declarations and carries the limits set out there.
6 6. Conclusions
6.1. Implications for practice
This section concerns how summarization systems should be evaluated, audited and reported. It makes no clinical recommendation and no medication recommendation, and nothing in it is advice for the care of an individual patient.
First, evaluation practice: a system should not be described as safe or unsafe for summary drafting on the strength of an any-error count alone. In this review the any-error figure (60.6 percent, 95 percent CI 29.8 to 84.7) and the one clinically significant figure (21.0 percent, 95 percent CI 11.7 to 33.2) measure different things, and a service that tracks only the first cannot see the second. Evaluations should grade errors for clinical significance at the time of rating, per summary, as a routine step.
Second, audit practice: audits should report the rating process with the counts. In the included studies, rating was blinded in a small minority of units (2 of 16 omission units), risk of bias on the adapted tool was high for 9 of 11 studies, and harm grading per summary appeared once (1 of 100 summaries at moderate level, with higher levels at zero). An audit that names its raters, its blinding, its error definitions and its harm scale can be compared with the next audit. An audit that reports a bare percentage cannot.
Third, reporting practice for vendors and study teams: publish, for each model version and summary type, the number of summaries assessed, the number with at least one error of each type, and the number with at least one clinically significant error, with the counting unit stated in words. The fabrication pool in this review (pooled 17.6 percent, prediction interval 0.1 to 98.3 percent) shows what happens to the evidence base when counting units differ in silence: the literature becomes impossible to combine, and no one can tell whether a system improved.
6.2. Implications for research
The first research need is the primary outcome itself. Studies should predefine a clinically significant error threshold, apply it per summary, and report the numerator and denominator. One study in 11 did so here, and a second (Nakano 2026) defined the threshold and stopped one step short. A small number of studies reporting that single count, across summary types, would let the next review pool the outcome this one could not.
The second need is the matched comparison. No included study counted errors in LLM and human summaries of the same records under the same definition, so the LLM versus human contrast is an empty cell in the summary of findings. A study that error-rates both drafts, blind to authorship, would fill the most decision-relevant gap in this literature.
The third need is counting discipline for fabrication. Studies should state whether fabrication is counted per summary, per statement or per content unit, and report the summary-level count even when a finer unit is the primary analysis. The Bodanki 2026 unit (654 of 671 summaries with at least one unsupported statement) and the summary-level units below 25 percent in the same pool cannot both inform a single frequency until that reporting step is standard.
The fourth need is replication across the gaps in this search: the databases that were not run, the 10 reports that were not retrieved, and forward citation chasing. A re-run of this search after the OpenAlex rate limit resets, and with licensed database access, is the cheapest available check on the completeness of the study set.
Declarations
Funding. None. This review received no funding from any source.
Conflicts of interest. None declared. The author is fully independent. No vendor, funder or institution had any role in the design, conduct, analysis or writing of this review.
Author framing. The author is a healthcare founder and operator who bootstrapped and ran a direct primary care clinic as chief executive, and now builds clinical AI. No claim of current licensure or current clinical practice is made anywhere in this review.
Author contributions. Raphael Malikian is the accountable human author: he set the question and the protocol, directed the review, and takes responsibility for the manuscript and for every number in it. The remaining roles were filled by an LLM team working in place of a human research team: information specialist, two screeners and a screening adjudicator, two extractors and a settlement reader, two risk of bias assessors and an adjudicator, a statistician, a GRADE assessor, a methods editor and a visual QA role. Appendix B summarises the team log.
AI-assistance note. This review was produced with large language model assistance at every stage, under the direction and accountability of the named human author. Screening, extraction, risk of bias assessment, statistical fitting, GRADE assessment and manuscript drafting were carried out by LLM agents working from written instruments specified before each stage, with dual independent passes and adjudication by reading at each judgement step, as described in the Methods. The human author is accountable for the review as a whole. This disclosure follows the RAISE 2025 guidance on responsible AI use in evidence synthesis (Thomas et al., cited in the references): evidence synthesists are ultimately responsible for their synthesis, AI use that informs judgements must be reported in full, and human oversight applies throughout. No AI system is an author of this review.
Data and code availability. The settled extraction dataset, the risk of bias judgements, the analysis code and outputs, and the search, screening and settlement files are deposited with the review package. GitHub repository: https://github.com/rtmalikian/llm-clinical-summarization-errors-meta-analysis (verified live on 31 October 2026), with a reading page at https://rtmalikian.github.io/llm-clinical-summarization-errors-meta-analysis/. Zenodo record: a preprint deposit is planned as the series' sole preprint home; no DOI is claimed in this version because the deposit had not been made when this version was finalised.
Registration. This review has no PROSPERO registration number. A PROSPERO-ready form was prepared on 31 October 2026, before screening closed (stage1-search/prospero-form.md). The related registration CRD420261349028 covers discharge summaries only and excludes progress notes, referral letters and operative notes; this review spans all summary types, with discharge as a subgroup.
References
Asgari E, Montaña-Brown N, Dubois M, Khalil S, Balloch J, Au Yeung J, Pimenta D. A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. npj Digital Medicine. 2025;8:274. DOI 10.1038/s41746-025-01670-7. PMID 40360677. (Discussed as excluded at full text.)
Bodanki P, Capone C, Hazare NS, Agaron S, Vijayaraghavan M, Japa S, Zaretsky I, Epstein J, Sawant A, Shaikh A, Leibner E, Sharma A, Gangadharan S, Ghani T, Tai HH, Ngai D, El-Haj S, Parchure P, Itwaru A, Kaplan B, Vakil A, Tamler R, Klein A, Freeman R, Kovatch P, Darrow B, McGreevy J, Stump L, Nadkarni GN, Timsina P, Sakhuja A. Multidimensional Assurance of an Electronic Health Record Embedded Generative Artificial Intelligence Summarization Tool. Journal of the American Medical Informatics Association. 2026. DOI 10.1093/jamia/ocag117. Preprint: medRxiv. 2025. DOI 10.1101/2025.09.15.25335813. The full text extracted in this review was the preprint. The extraction file carried a DOI that matches no record; this citation follows the verified source records, and the correction is logged in the team log and the claims verification file.
Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions. Version 6.5 (updated August 2024). Cochrane; 2024.
Dhanda K. Does a Safety-Priming Instruction Reduce Risk-Fact Omission in Psychiatric Note Summarization? A Synthetic-Vignette Study of Small Open-Weight Language Models. Research Square preprint. 2026. DOI 10.21203/rs.3.rs-11019497/v1. (Version extracted in this review. A second Research Square record for this work, DOI 10.21203/rs.3.rs-9046762/v1, was found at verification; the difference is logged in the claims verification file.)
Ganzinger M, Kunz N, Fuchs P, Lyu CK, Loos M, Dugas M, Pausch TM. Automated generation of discharge summaries: leveraging large language models with clinical data. Scientific Reports. 2025;15:16466. DOI 10.1038/s41598-025-01618-7.
Grolleau F, Liang AS, Keyes T, Ma SP, Lew T, Huynh TR, Steele N, Chung P, Qin P, Chandra G, Wang SF, Mullen E, Carpenter L, Hoppenfeld M, Morrin M, Kyerematen BA, Ambers N, Kotecha N, Alsentzer E, Hom J, Shah NH, Schulman K, Chen JH. Physician-Reported Safety Outcomes of AI-Generated Hospital Course Summaries. JAMA Network Open. 2026;9(5):e2616556. DOI 10.1001/jamanetworkopen.2026.16556.
Iorio A, Spencer FA, Falavigna M, Alba C, Lang E, Burnand B, et al. Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients. Journal of Clinical Epidemiology. 2015;68(3):333-340. DOI 10.1016/j.jclinepi.2014.11.001. (Applied in this series, by extension, to proportion evidence.)
Jolin-Rodrigo I, Carrero-Fernandez A, Cuadros-Gonzalez J, R-Moreno MD. Evaluating LLMs for HIV Epidemiological Surveillance in Spanish: Clinical Summarization, Risk Factor Extraction, and Inference Stability across Models. Research Square preprint. 2026. DOI 10.21203/rs.3.rs-10166213/v1. (Version extracted in this review. A second Research Square record for this work, DOI 10.21203/rs.3.rs-9276141/v1, was found at verification; the difference is logged in the claims verification file.)
Nakano M, Imura T. Implementation of an AI-Assisted Workflow for Rehabilitation Discharge Summary Creation: Evaluation of Documentation Efficiency, Usability, and Hallucinations in a 290-Bed General Hospital in Japan. Cureus. 2026;18(5):e109247. DOI 10.7759/cureus.109247. PMID 42326256. PMCID PMC13282032. Two verification passes disagreed on this citation; it was adjudicated against the PubMed record for this title on 31 October 2026 (19 May 2026; 18(5):e109247), whose abstract matches the extracted study (11 summaries assessed, 19 hallucinations, 1 omission). The adjudication is logged in the team log and the claims verification file.
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. DOI 10.1136/bmj.n71.
Park KU, Sather BK, Shah A, Minami CA, Butler M, Butler L, Love JA, McLean K, Dunn A, Mittendorf EA, King TA. Artificial Intelligence-Generated Electronic Medical Record Summarization in Breast Surgical Oncology. Annals of Surgical Oncology. 2026;33(10):9102-9109. DOI 10.1245/s10434-026-20170-w. PMID 42455485. Volume, issue and pages verified on the publisher record and the PubMed record on 31 October 2026 (published 15 July 2026).
Thomas J, Hair K, Noel-Storr A, et al. Artificial intelligence methods and processes in systematic reviews: an introduction to RAISE 2025. Cochrane Evidence Synthesis and Methods. 2026;3(2):e70047. DOI 10.1002/cesm.70047. Preprint: medRxiv. 2025. DOI 10.1101/2025.07.07.25331007.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-174. PMID 843571.
Kharat M, Agrawal N. Accuracy, Completeness, and Safety of Large Language Model-Generated Discharge Summaries: A Systematic Review and Meta-Analysis. PROSPERO registration CRD420261349028. Registered 23 March 2026. Status on re-check, 31 October 2026: planned or ongoing.
Sounderajah V, Ashrafian H, Golub RM, Shetty S, De Fauw J, Denniston AK, et al.; QUADAS-AI Steering Group. Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol. Nature Medicine. 2025. DOI 10.1038/s41591-024-03471-z. PMID 39820808. QUADAS-AI remains in development; the risk of bias tool used in this review is an adapted tool, as stated in the Methods.
Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB; PRISMA-S Group. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Systematic Reviews. 2021;10:39. DOI 10.1186/s13643-020-01542-z.
Whiting PF, Rutjes AWS, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Annals of Internal Medicine. 2011;155(8):529-536. DOI 10.7326/0003-4819-155-8-201110180-00009.
Song JW, Park JS, Kim JH, You SC. Large Language Model Assistant for Emergency Department Discharge Documentation. JAMA Network Open. 2025;8(10):e2538427. DOI 10.1001/jamanetworkopen.2025.38427.
Tung JYM, Gill SR, Sng GGR, Lim DYZ, Ke Y, Tan TF, Jin L, Elangovan K, Ong JCL, Abdullah HR, Ting DSW, Chong TW. Comparison of the Quality of Discharge Letters Written by Large Language Models and Junior Clinicians: Single-Blinded Study. Journal of Medical Internet Research. 2024;26:e57721. DOI 10.2196/57721.
Van Veen D, Van Uden C, Blankemeier L, Delbrouck JB, Aali A, Bluethgen C, et al. Adapted large language models can outperform medical experts in clinical text summarization. Nature Medicine. 2024;30(4):1134-1142. DOI 10.1038/s41591-024-02855-5. PMID 38413730. (Discussed as excluded at full text.)
Watcharasarn R. Completeness and Quality of Neurology Referral Letters Generated by a Large Language Model for Standardized Scenarios. Medicina. 2025;61(11):1931. DOI 10.3390/medicina61111931.
Williams CYK, Subramanian CR, Ali SS, Apolinario M, Askin E, Barish P, et al. Physician- and Large Language Model-Generated Hospital Discharge Summaries. JAMA Internal Medicine. 2025;185(7):818-825. DOI 10.1001/jamainternmed.2025.0821. PMID 40323616. (Discussed as excluded at full text.)
Williams CYK, Bains J, Tang T, Patel K, Lucas AN, Chen F, Miao BY, Butte AJ, Kornblith AE. Evaluating large language models for drafting emergency department encounter summaries. PLOS Digital Health. 2025;4(6):e0000899. DOI 10.1371/journal.pdig.0000899.
Appendix A. Search strings per source
Searches run 31 October 2026. Strings are condensed here from stage1-search/search-log.md, which holds the full log, the raw response files and the deduplication record.
PubMed (E-utilities; hits 378, retrieved 378). ("large language model"[tiab] OR "large language models"[tiab] OR LLM[tiab] OR ChatGPT[tiab] OR "GPT-4"[tiab] OR "generative AI"[tiab] OR "foundation model"[tiab]) AND (summarization[tiab] OR summarisation[tiab] OR "discharge summary"[tiab] OR "discharge summaries"[tiab] OR "hospital course"[tiab] OR "encounter summary"[tiab] OR "progress note summary"[tiab] OR "referral letter"[tiab] OR "clinical summary"[tiab] OR "clinical summaries"[tiab] OR "clinical summarization"[tiab] OR "clinical summarisation"[tiab]) AND (error[tiab] OR omission[tiab] OR hallucination[tiab] OR fabrication[tiab] OR inaccura*[tiab] OR accuracy[tiab] OR completeness[tiab] OR safety[tiab]) AND 2022:2026[dp].
Europe PMC (REST API, title and abstract strategy; hits 467, retrieved 467). (TITLE_ABS:"large language model" OR TITLE_ABS:"large language models" OR TITLE_ABS:LLM OR TITLE_ABS:ChatGPT OR TITLE_ABS:"GPT-4" OR TITLE_ABS:"generative AI" OR TITLE_ABS:"foundation model") AND (TITLE_ABS:summarization OR TITLE_ABS:summarisation OR TITLE_ABS:"discharge summary" OR TITLE_ABS:"discharge summaries" OR TITLE_ABS:"hospital course" OR TITLE_ABS:"encounter summary" OR TITLE_ABS:"referral letter" OR TITLE_ABS:"clinical summary" OR TITLE_ABS:"clinical summaries" OR TITLE_ABS:"clinical summarization" OR TITLE_ABS:"clinical summarisation") AND (TITLE_ABS:error OR TITLE_ABS:errors OR TITLE_ABS:omission OR TITLE_ABS:omissions OR TITLE_ABS:hallucination OR TITLE_ABS:hallucinations OR TITLE_ABS:fabrication OR TITLE_ABS:inaccuracy OR TITLE_ABS:inaccuracies OR TITLE_ABS:accuracy OR TITLE_ABS:completeness OR TITLE_ABS:safety) AND FIRST_PDATE:[2022-01-01 TO 2026-12-31].
Preprints through Europe PMC (SRC:PPR filter applied to the Europe PMC string above; hits 140, retrieved 140).
Semantic Scholar (bulk search endpoint, after the graph API returned HTTP 429 on 3 attempts; hits 5, retrieved 5). Query: large language model clinical summarization discharge summary error hallucination. Year filter 2022 to 2026.
ClinicalTrials.gov (API v2; hits 1, retrieved 1). Query term: large language model clinical summarization discharge summary. The single record was NCT07866001.
OpenAlex (blocked). Four strings attempted through api.openalex.org (large language model clinical summarization error; large language model discharge summary hallucination; ChatGPT clinical summary accuracy omission; LLM discharge summaries completeness safety evaluation), each filtered to 2022 to 2026. Every call returned HTTP 429. Not searched.
Not run (no access), ready strings filed in the search log: Embase, Scopus, Web of Science, CINAHL, CENTRAL, IEEE Xplore, ACM Digital Library. Google Scholar not run (no full programmatic interface).
Appendix B. LLM team log summary
The full log is review3/llm-team-log.md. This appendix summarises roles, counts and corrections.
Roles. Accountable human author: Raphael Malikian. LLM team roles: information specialist; Screener A (inclusion-first rule set) and Screener B (exclusion-first rule set); screening adjudicator; Extractor A and Extractor B; extraction settlement reader; risk of bias Assessor 1 and Assessor 2 and an adjudicator, using the adapted QUADAS-2 tool for LLM clinical summarisation evaluations (adapted from QUADAS-2 with QUADAS-AI signalling questions); statistician; GRADE assessor; methods editor; visual QA. Prompts version v1, 31 October 2026. Every instrument was specified in writing before the stage it governs.
Search. 991 records identified across 5 searched sources; 508 duplicates removed; 483 records screened. OpenAlex blocked by a provider rate limit (HTTP 429 on 4 strings) and not searched; 7 databases not run for want of access. Backward citation chasing on the 10 database-included studies produced 15 candidates and 1 included study. Forward citation chasing was not performed as a separate pass.
Screening. Screener A: 11 include, 47 maybe, 425 exclude. Screener B: 8 include, 92 maybe, 383 exclude. Agreement 431 of 483 (89.2 percent) across three categories, Cohen kappa 0.62; binary pass against exclude agreement 439 of 483 (90.9 percent), kappa 0.67. Disagreements: 52 records, all read by the adjudicator; 9 passed to full text, 43 excluded. Full text: 58 sought, 10 not retrieved, 48 assessed, 38 excluded (31 no extractable per-summary numerator and denominator, 3 ineligible summary type, 2 duplicate report, 1 excluded task, 1 no LLM generation), 10 included, plus 1 from citation chasing after adjudication of the single chasing disagreement: 11 studies.
Extraction. Two independent passes, 22 arm rows each (21 LLM arms, 1 human arm). Comparison by code over 374 field pairs (17 fields, 22 arms): 161 raw string discrepancies, 43.0 percent, mostly format artefacts (blank against a not reported code, parenthetical wording, field placement). Every numeric secondary count reported as a number by both extractors agreed exactly. Six substantive items went to settlement by re-reading; settlements re-keyed the Park 2026 arms, re-keyed the Dhanda 2026 omission counts (13, 4, 5, 4, 6, 4, 2, 4), removed a rounded-mean derived total for Ganzinger 2025, and verified the Williams 2025 derived counts (67 and 90 of 100). One item stayed unresolved and is carried with its caveat: the Park 2026 pilot count of 13, ambiguous between errors and cases.
Risk of bias. Two assessors rated all 11 studies on 4 domains of the adapted tool. Overall judgements agreed on 11 of 11 (high 9, some concerns 2, low 0). Domain cells disagreed on 12 of 44; the adjudicator settled 2 at the first assessor judgement and 10 at the second. Settled judgements are in stage3-extraction/rob-settled.csv and Table 3 of the manuscript.
Analysis. Pools fitted with a hand-written Python GLMM (scipy, adaptive Gauss-Hermite quadrature) because R was unavailable; an earlier non-adaptive quadrature implementation produced a wrong likelihood at large tau squared, was caught by validation against numerical integration before any number was reported, and was discarded in full. The any-error pool was hand-checked end to end by an independent route and matched to rounding (0.6055 against 0.6055). Crude totals for all 15 fitted pools were re-summed from the settled sheet and matched exactly. One pre-analysis amendment (Amendment 01) authorised the supplementary any-error pool. No pooled number in the manuscript comes from the discarded implementation or from any unsettled sheet.