Most #LLMs over-generalized scientific results beyond the original articles
...even when explicitly prompted for accuracy!
The #AI was 5x worse than humans, on average!
Newer models were the worst.🤦♂️
🔓 Accepted in #RoyalSociety Open #Science: https://doi.org/10.48550/arXiv.2504.00025
![".... Original texts and summaries were coded based on whether their result claims contained one or more of the following three types of generalizations:
(1) Generic generalizations (generics). These are present tense generalizations that do not have a quantifier (e.g. ‘many’, ‘75%’) in the subject noun phrase and describe study results as if they apply to whole categories of people, things, or abstract concepts (e.g. ‘parental warmth is protective’) instead of specific or quantified sets of individuals (e.g. study participants). ....
(2) Present tense generalizations. ... When past tense result claims from an original text are turned into present tense in the summary, a broader generalization is conveyed than the author(s) of the original text may have intended.
(3) Action guiding generalizations. ...result claims ... often underlie recommendations ... (e.g. ‘CBT should be recommended for OCD patients’) [that involve] broader generalization than that found in the summarized text because researchers may have deliberately avoided such recommendations due to insufficient evidence to support them.
We tested whether the outputs of the 10 LLMs mentioned above retained the quantified, past tense, or descriptive generalizations of the scientific texts that they summarized, or transitioned to unquantified (generic), present tense, or action guiding generalizations. We defined the latter kind of conclusions collectively as generalized and the former as restricted conclusions."](/proxy/post_attachment/91830/8563bade44.png "\".... Original texts and summaries were coded based on whether their result claims contained one or more of the following three types of generalizations:
(1) Generic generalizations (generics). These are present tense generalizations that do not have a quantifier (e.g. ‘many’, ‘75%’) in the subject noun phrase and describe study results as if they apply to whole categories of people, things, or abstract concepts (e.g. ‘parental warmth is protective’) instead of specific or quantified sets of individuals (e.g. study participants). ....
(2) Present tense generalizations. ... When past tense result claims from an original text are turned into present tense in the summary, a broader generalization is conveyed than the author(s) of the original text may have intended.
(3) Action guiding generalizations. ...result claims ... often underlie recommendations ... (e.g. ‘CBT should be recommended for OCD patients’) [that involve] broader generalization than that found in the summarized text because researchers may have deliberately avoided such recommendations due to insufficient evidence to support them.
We tested whether the outputs of the 10 LLMs mentioned above retained the quantified, past tense, or descriptive generalizations of the scientific texts that they summarized, or transitioned to unquantified (generic), present tense, or action guiding generalizations. We defined the latter kind of conclusions collectively as generalized and the former as restricted conclusions.\"")
Edited 1d ago