AutoCrit - A Meta-Reasoning Framework for Self-Critique and Iterative Error Correction in LLM Chains-of-Thought: Evaluation Design and Construct Validity

Authors

  • Wade Hayes Author
  • Wyatt Coleman Author
  • Alec Richardson Author

Keywords:

reliable retrieval-augmented generation, evaluation design and construct validity, evidence synthesis, reproducibility, research evaluation

Abstract

Evaluation is persuasive only when the measured outcome corresponds to the construct claimed by the study and the comparison answers the stated research question. This structured evidence review evaluates "AutoCrit: A Meta-Reasoning Framework for Self-Critique and Iterative Error Correction in LLM Chains-of-Thought" alongside nine author-disjoint, topically matched publications in reliable retrieval-augmented generation. It compares construct definitions, evaluation choices, operating assumptions, and reported limitations instead of treating bibliographic similarity as empirical equivalence. Viewed through evaluation design and construct validity, the map separates claims supported by the available record from questions that still require full-text extraction, replication, or new experiments. The synthesis is interpretive rather than meta-analytic and therefore does not present a pooled effect estimate or a new causal result. The resulting agenda aligns claims, outcomes, comparators, sampling, and uncertainty before any performance estimate is interpreted.

References

  1. Sang, Y. (2025). AutoCrit: A Meta-Reasoning Framework for Self-Critique and Iterative Error Correction in LLM Chains-of-Thought. 2025 6th International Conference on Machine Learning and Computer Application (ICMLCA), 1177-1180. https://doi.org/10.1109/icmlca66850.2025.11336788
  2. Hassan, S.-B., Abdullah,, & Abbas, M. (2026). Agentic Self-RAG: Multi-Agent Reasoning for Self-Correcting Retrieval-Augmented Generation. 2026 International Conference on IT and Industrial Technologies (ICIT), 1-6. https://doi.org/10.1109/icit68548.2026.11577708
  3. Fasolino, I. (2026). In RAG We Trust? Measuring Robustness of Retrieval-Augmented Generation Under Document Poisoning. . https://doi.org/10.32388/7hxkqe
  4. Samal, M.-P. (2026). A Theoretical Analysis of Self-Contained Retrieval-Augmented Generation with Small Language Models. . https://doi.org/10.36227/techrxiv.177219989.96070478/v1
  5. Pi, W. (2024). Efficient Information Retrieval and Response Generation with Retrieval-Augmented Generation (RAG). . https://doi.org/10.59350/q2pq3-0fv85
  6. Xue, X., Zhang, G., Jiang, L., & Liu, C. (2025). A Comparative Study of Retrieval-Augmented Generation, Graph Retrieval-Augmented Generation, and Fine-Tuned Large Language Models for Fire Engineering Knowledge Retrieval. . https://doi.org/10.2139/ssrn.5316632
  7. Bose, R. (2025). Introduction to Retrieval-Augmented Generation (RAG). Mastering Retrieval-Augmented Generation, 3-32. https://doi.org/10.1007/979-8-8688-1808-0_1
  8. Zhai, W. (2025). SAM-RAG: An Self-adaptive Framework for Multimodal Retrieval-Augmented Generation. 2025 International Joint Conference on Neural Networks (IJCNN), 1-8. https://doi.org/10.1109/ijcnn64981.2025.11227819
  9. Gudipati, S.-K., Mishra, N.-A., Ankam, G., Akkisetti, S.-R., Mohammed, A., & Ponugoti, S. (2026). CityCopilot-X: A Real-Time Explainability Panel for Retrieval-Augmented Generation (RAG). 2026 IEEE 5th International Conference on AI in Cybersecurity (ICAIC), 1-9. https://doi.org/10.1109/icaic67076.2026.11395791
  10. Kau, A. (2024). Understanding Retrieval Pitfalls: Challenges Faced by Retrieval Augmented Generation (RAG) models. . https://doi.org/10.59350/xcq3s-jvk04

Downloads

Published

2026-08-27

Issue

Vol. 1 (2026): Archive

Section

Review Article

License

Copyright (c) 2026 Journal of Algorithmic Discovery and Applied AI

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.