Detecting RAG Hallucinations with Output Probabilities and Attention

A retrieval-augmented generation (RAG) system can retrieve the correct passage and still produce the wrong answer. The model may ignore the passage, rely on its pretrained knowledge, or combine the supplied facts incorrectly. Retrieval improves access to evidence, but it does not verify how that evidence is used. An answer that is unsupported by, or contradicts, the retrieved material is referred to here as a contextual hallucination.

This project was a 48-hour investigation of whether simple signals available inside the generating model could help identify such failures. I started with output-token probabilities. The idea was that an unsupported answer may contain tokens that appear unusual to the model that produced it. I then measured how much attention the model allocated to the retrieved context relative to its own answer. Finally, I combined the two signals to test whether they captured different types of failure.

No method was strongest across all tasks. Output probabilities provided a modest but more transferable signal than expected. Attention was substantially stronger when training and evaluation used answers from the same model, but transferred poorly to some financial and biomedical tasks. Attention to relevant evidence did not establish that the model had used it correctly.

Building the detector

I started with the four output-probability statistics proposed by Quevedo et al. (2024). Their method paired these statistics with a linear classifier and provided a simple, interpretable, and inexpensive baseline.

The detector evaluates an answer after it has been generated. I passed the retrieved context, question and completed answer through Llama-2-7B-Chat once, allowing the model to assign a probability to each answer token using the preceding text. This teacher-forced pass produced output probabilities and attention maps without generating a second answer.

Four features summarised the output probabilities. The minimum and mean token probabilities measured confidence in the observed answer. The remaining features recorded the largest gap between an observed token and the model’s preferred token, and the smallest spread between the highest- and lowest-probability vocabulary tokens. A standardised logistic regression classifier combined the four features into a hallucination score. Higher values indicated that an answer was more likely to contain a contextual hallucination.

Output probabilities measure surprise, but they do not describe how attention was distributed between the retrieved evidence and the answer prefix. Chuang et al. (2024) proposed Lookback Lens to capture this information. For each of the model’s 32 layers and 32 attention heads, I calculated the ratio of attention assigned to the retrieved context to attention assigned to the context and earlier answer tokens combined. Averaging the ratios across the answer produced 1,024 features. A second logistic regression classifier converted them into an attention-based hallucination score. I adapted the original method to score completed answers after generation.

A final logistic regression classifier combined the probability and attention scores. I used five-fold cross-fitting to produce held-out component scores for training this classifier. The component classifiers were then refitted on the full training set. This prevented the combined model from learning from overly optimistic in-sample scores.

I also tested additional probability statistics, span-level attention features, and the comparison of retrieved context with internal knowledge proposed by Yeh et al. (2026). None improved enough to justify the added complexity or computation, so I kept the two-score design.

The two signals generalised differently

I evaluated the detector in two settings. In the first, the model that generated the answers also provided the internal signals. I trained and evaluated the detectors on separate parts of RAGTruth (Niu et al., 2024). An answer was treated as hallucinated if it contained at least one annotated hallucinated span. In the second setting, I used a broader mix of answers across reading comprehension, finance, general question answering and biomedicine. Most of these answers were produced by other systems. They came from HaluBench (Ravi et al., 2024).

I used AUROC to measure how consistently each hallucination score ranked hallucinated answers above faithful ones. A value of 0.5 corresponds to random ranking and a value of 1.0 corresponds to perfect ranking. I treat the broader set as a transfer check because it also informed method selection.

Attention clearly improved on output probabilities for answers produced by the same model. Combining the signals did not improve the result further. Across the broader task mix, output probabilities transferred better than attention and the combined detector achieved the strongest overall result. Attention therefore captured a stronger but less transferable signal, while output probabilities provided weaker but more transferable information.

Attention did not establish correct use of evidence

Performance varied sharply between tasks. The combined detector performed best on a reading-comprehension task, while output probabilities were strongest on a general question-answering set. On the financial and biomedical tasks, all three approaches were close to random.

The failures on financial and biomedical questions clarified the limitation of the attention approach. The ratio describes how attention was allocated to the retrieved passage, but not whether the model selected the correct value, applied the correct operation, tracked the correct entity, or drew a supported conclusion.

Output probabilities have a corresponding limitation. An uncommon but supported answer may appear surprising, while a fluent unsupported statement may remain highly probable. Probability disagreement and context attention are therefore useful indicators of possible hallucination, but neither is a direct test of whether a claim follows from the supplied evidence.

The two signals are therefore more suitable for prioritising answers for review than for verifying that those answers are correct.

Internal model signals came with practical costs

These are white-box methods because they rely on values inside the generating model. The combined detector cannot be used through an interface that returns only text and requires a second pass over each completed answer.

In a small benchmark on one NVIDIA H100, the additional pass added little latency. The larger cost came from storing attention maps, which required almost 14 GiB of additional GPU memory for the longest inputs tested. Output-probability features avoid this storage. Using attention in a long-context RAG system would require a more efficient way to extract or approximate the signal. White-box detection removes the need to run a separate verifier model, but it does not remove deployment costs.

The next step is detecting correct use of evidence

The next step is to understand when a model not only attends to the correct evidence but also applies it correctly. A stronger detector would need signals for claim-level entailment, numerical consistency, and entity tracking.

References

• Chuang, Y.-S. et al. (2024). Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps.
• Niu, C. et al. (2024). RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models.
• Quevedo, E. et al. (2024). Detecting Hallucinations in Large Language Model Generation: A Token Probability Approach.
• Ravi, S. S. et al. (2024). Lynx: An Open Source Hallucination Evaluation Model.
• Touvron, H. et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models.
• Yeh, S. et al. (2026). LUMINA: Detecting Hallucinations in RAG System with Context-Knowledge Signals.