Reliability Beyond Accuracy: Factuality, Uncertainty & Bias in LLMs

Overview

A comparative research review synthesizing four foundational LLM-reliability papers — DoLa, KWTDK, BOLD, and HELM — across three dimensions: factuality, uncertainty calibration, and social/political bias.

What I analyzed

• Decoding-time hallucination mitigation strategies (DoLa).
• Fine-tuned uncertainty estimation via calibration on graded data, evaluated with ECE and AUROC.
• Open-ended bias benchmarking on BOLD's 23K-prompt set.
• HELM's multi-metric evaluation framework, spanning 7 metrics across 30 models.

Outcome

Proposed novel benchmark directions for jointly measuring hallucination and ideological framing on shared prompts — extending HELM's multi-metric approach beyond evaluating factuality, calibration, and bias in isolation.