Detecting RAG Hallucinations with Output Probabilities and Attention

This project focused on investigating simple ways to detect hallucinations in LLMs. Hallucinations can be defined very broadly making the prevalence of the behavior difficult to measure. To achieve tractable scope for a short project I limited the investigation to hallucinations in retrieval-augmented generation (RAG) systems where the information provided by RAG can be treated as ground truth. Despite RAG supplying the relevant information or evidence to the model by adding it to the context, the model is still not guaranteed to correctly incorporate the context into its response sometimes leading to hallucinations.

I wanted to investigate whether signals available inside a model can identify these hallucinations. The results presented here focus on the most successful approach combining simple features based on output-token probabilities and adapted Lookback Lens, which measures how the scoring model’s attention is divided between the retrieved context and the answer generated so far.

I found that the attention-based detector was much stronger when trained and evaluated on answers from the same model, but this advantage did not carry over to the broader hallucination detection. Output probabilities were weaker in the matched-model setting but achieved a higher AUROC than attention when trying to detect hallucinations in responses not generated by the same model. Combining the two produced the best performance among the approaches I tested.

Model, data and evaluations

The model I used throughout this investigation is Llama-2-7B-Chat. Hallucination detection can be framed in two different ways. The first uses the internal states of the same model that generated the response, which was the primary focus of this project. The second resembles the guard-model approach used in safety, where a separate model is trained or fine-tuned specifically to detect hallucinations. Although I did not investigate this second approach in depth, I included HaluBench aiming to gain some insight into whether signals learned from Llama 2’s own outputs could transfer to responses generated by other systems. Identifying a broadly useful signal of disagreement between a context and response could help in developing stronger specialized hallucination detectors.

Two datasets were used for training and evaluation:

• RAGTruth: dataset with responses from different models annotated for presence/absence of hallucinations. I only used answers generated by Llama-2-7B-Chat since that is the model of interest.
• HaluBench: a broader mixture drawn from a number of datasets (DROP, FinanceBench, covidQA, HaluEval and PubMedQA). Most answers in this dataset were generated by models different from Llama-2-7B-Chat.

I used area under the receiver operating characteristic curve (AUROC) as the main evaluation metric. AUROC measures how consistently a detector ranks hallucinated answers above faithful ones and is not tied to a specific threshold the way accuracy would be.

Output probability and attention map signals

The first approach I investigated used four features proposed by Quevedo et al. and derived from output token probability distribution. The four features were the minimum and mean observed-token probabilities, the largest difference between the probability of the model’s preferred token and that of the observed token, and the smallest spread between the highest- and lowest-probability vocabulary tokens. The idea is that hallucinated, and hence unsupported, parts of the response are likely to have probability distributions with features distinct from a well grounded response. These features are used in a logistic regression to provide the output-probability based score.

Output probabilities do not show whether the model is attending to the retrieved context and a fluent unsupported answer can remain highly probable. I was curious to see whether attention maps can provide a stronger complementary signal. More specifically, I adapted the response-level version of Lookback Lens, which relies on comparing the attention to retrieved-context and earlier parts of the answer. For every token after the first output token and every attention head in every layer the approach computes the average attention to the retrieved-context tokens and the average attention to earlier response tokens. Average attention to the retrieved-context divided by the sum of the two averages provides per layer per head features that are then used in a logistic regression to provide attention based score.

I also investigated several extensions and alternatives before settling on the two signals above. Additional probability statistics did not improve the detector consistently. Span-level attention features were promising for locating hallucinated chunks, but aggregating them into a response-level score added complexity without significantly improving the detection performance. I also tested the comparison of retrieved context with internal model knowledge proposed by Yeh et al.. It did not improve performance and required an additional forward pass as well as hidden-state evaluation.

Combining signals

To combine the output probability and attention based scores I trained a logistic regression using scores as input features. I used five-fold cross-fitting to generate held-out component scores for training the combined classifier, then refitted the component models on the full training set.

Results

The table above shows results from using the output-probability and attention-based scores on their own and in combination. On the RAGTruth test set, the attention-based score performs much better than the output-probability score. This suggests that a model’s attention patterns can provide a useful signal for detecting hallucinations in its own outputs. All three approaches perform substantially worse on HaluBench, where most answers were generated by other models. This suggests that directly using one model’s internal states to detect hallucinations in another model’s outputs is unlikely to work well without additional model- or task-specific training. Interestingly, in this setting the output-probability score performs better than the attention-based score. However, the aggregate HaluBench result conceals substantial differences between its sources:

The variation across different data sources is something to investigate further. Drawing genuinely meaningful conclusions from the above would require careful consideration of not just the data but its source as one would expect anything generate by Llama-2 family models to produce a much stronger signal given that we are using a model from that family.

What the current scores are missing is whether the model has correctly incorporated information from the context into its output. The attention score indicates how strongly the model attends to the retrieved context relative to the generated answer, but it does not tell us whether that context is used correctly. The model may still select the wrong information or fail to perform the deductions needed to produce an answer that is properly supported by the context. I would be very curious to explore signals that illuminate this part of the generation process and whether they could improve hallucination detection.

Practical considerations

White-box approaches clearly have a promise for gaining additional insight not just in this problem but more generally however it should be noted that using white-box approaches is not free. In my post-hoc scoring implementation, each completed answer required an additional scoring pass. I benchmarked this scoring pass on one NVIDIA H100. The detector added 0.06 seconds on average, approximately 3% of the mean generation time. Memory was a more significant constraint. Relative to the memory already occupied by the loaded model, the scoring added about 13.9 GiB in the worst cases (long prompts) while generation for the same group of prompts added about 1.7 GiB.

Next steps

The thing I am most interested to investigate next is how to detect whether an answer follows from the retrieved context, rather than only whether the model attends to it. I am also curious whether attention features can be extracted or approximated more efficiently for long-context inputs.

References

• Chuang, Y.-S. et al. (2024). Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps.
• Niu, C. et al. (2024). RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models.
• Quevedo, E. et al. (2024). Detecting Hallucinations in Large Language Model Generation: A Token Probability Approach.
• Ravi, S. S. et al. (2024). Lynx: An Open Source Hallucination Evaluation Model.
• Touvron, H. et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models.
• Yeh, S. et al. (2026). LUMINA: Detecting Hallucinations in RAG System with Context-Knowledge Signals.