I hate when something piques my interest because it feels related to something I’m working on but it turns out it isn’t. Nine times out of ten I can’t just quit the read (especially if it’s under twenty pages) but it never feels like a productive use of time. I saw this paper on measurement using LLMs on Arxiv this morning and it felt like exactly the kind of problem I’ve been thinking about in my day-to-day – how do we get AI to estimate unobservables? In my case this is often a behavioral response to an intervention (or why my code throws an error – unobservable to the lazy developer). This is a bit different, instead keying in on “hard” unobservables – situations where the reason we can’t measure the latent variable has more to do with the nature of corporate data and AI adoption rates than what’s going on in people’s heads. But I do think it’s an interesting look at where you could take this stuff in academic settings where you can’t necessarily know who has three Claude Code instances running at a given point in time or who is blowing through a million tokens every other day (I have deadlines).
Maya is probing a pretty cool space here – probably partially why I couldn’t put this down easier – but I do have some concerns about whether or not we’re actually able to take away what he wants us to here.
My fundamental frame of reference for this brand of work is Gallegos et al. 2023 (Bias and Fairness in LLMs). It’s not super relevant here but I think it gives me a skepticism about whether using LLMs “just works” – and here I’m not totally convinced. The assumption that you’re telling the AI to only do a specific thing implies that only that specific thing is going on, or at least that the LLM is able to key in only on that thing, particularly when we’re maybe moving outside of the corpus, and it feels a bit like “well I said make no mistakes.” Maya’s first validity condition – semantic exogeneity – is essentially the claim that a carefully designed prompt that never mentions wages or employment produces a score that’s orthogonal to labor market outcomes. Which is a reasonable thing to want, but LLMs aren’t neutral semantic processors – they’re compressed representations of a corpus where wages, status, and occupational prestige are thoroughly entangled with task descriptions. You can sanitize the prompt all you want, but the weights don’t especially care.
I also don’t love the idea that convergent validity is a good basis for performance evaluation for LLMs, particularly in a paper where the author is selling this as a method. I’d love to see performance against an evaluated model or a labeled set – standard practice in CS/AI papers. We’ve been constructing models of latent variables for a long time across a bunch of disciplines, and surely there’s some labeled latent variable of hard facts that could have been used here (I should probably look into this before lobbing a molotov). Regardless, it feels like a faulty premise – the idea that consistently held beliefs are useful ground truth feels like it runs contra to core principles of economics, and it’s especially awkward when the raters in question were kind of built to agree with each other. An AI that consistently deviated from what other AIs were saying wouldn’t be especially useful, so the development process itself selects for convergence. It’s a bit like using two students who studied from the same textbook as independent graders, where high agreement doesn’t mean either of them is right.
This actually creates a problem for the ORIV estimation that I think is underappreciated. The whole point of ORIV is that you need two instruments with independent measurement error – you use Sonnet scores as an instrument for Haiku scores and vice versa, and the independence assumption is what makes the attenuation correction valid. But if the models share training-induced biases, the errors aren’t independent, and the correction is wrong in ways that are hard to detect. The author actually hints at this when cross-family reliability between Claude and GPT-4o-mini drops substantially – which is what you’d expect if same-family models are correlated draws from overlapping distributions – but then recommends same-family models for ORIV, which is roughly the opposite of what the independence assumption requires.
There’s also a subtler issue about where the reliability numbers are coming from. LLM judgment is probably quite good on the easy cases – task descriptions that unambiguously signal something about augmentability. The harder cases, where something like reasoning actually matters (“comfortable with Copilot” versus “comfortable with modern technology in their daily workflow”), are where you’d want the instrument to do real work, but aggregate reliability metrics don’t tell you whether agreement is concentrated in the easy cases or distributed across the hard ones. Divergence is the signal, and the cases where the models disagree are probably more informative about the limits of this approach than the cases where they agree – and the paper doesn’t really look there.
None of this means the approach is wrong. Using LLMs to score semantic content at scale is genuinely useful and cheap – Maya points out that scoring 18,796 tasks cost about five dollars, which is a real and non-trivial advantage over expert panels. The ORIV framework is clever and the paper is careful. But the validity conditions need to do more work on the endogeneity problem, and convergent validity across models that were built to agree is a weaker foundation than it looks.