LLMs and Systematic Measurement Errors

Measures derived from Large Language Models (LLMs) are pervasive while researchers commonly treat such measures as observed data rather than generated regressors. For S&P 500 firms from 2005-2024, we show that LLM-derived measures contain non-classical measurement error driven by two distortions: omission and injection. Due to these distortions, LLM measures neglect meaningful terminology, shift meaning and reshape tone. Using the predictive content of omission and injection rates, we develop and apply a measurement error diagnostic to three settings: Item 1A risk disclosures, earnings calls, and direct-prompt measures. Measurement error is most severe for construct-style ratings of long disclosures, milder for the same constructs on shorter earnings-call texts, and substantially attenuated for direct-prompt measures, though the diagnostic reveals measurement-related contamination across all three designs.