A few weeks ago, Bryan Caplan pointed out this great working paper by David Hedengren and Thomas Stratmann:
The Dog that Didn’t Bark: What Item Non-Response Shows about Cognitive and Non-Cognitive Ability
What survey respondents choose not to answer (item non-response) provides a useful task based measure of cognitive ability (e.g., IQ) and non-cognitive ability (e.g., Conscientiousness). Using the German Socio-Economic Panel (SOEP) and the National Longitudinal Survey of Youth 1997 (NLSY97), we find consistent correlation between item non-response and traditional measures of IQ and Conscientiousness. We also find that item non-response is more strongly correlated with earnings in the SOEP than traditional measures of either IQ or Conscientiousness. We also use the Survey of Income and Program Participation (SIPP) Gold Standard, which has no explicit measure of either cognitive or non-cognitive ability, to show that item non-response predicts earnings from self-reported and administrative sources. Consistent with previous work showing that Conscientiousness and IQ are positively associated with longevity, we document that item non-response is associated with decreased mortality risk. Our findings suggest that item non-response provides an important measure of cognitive and non-cognitive ability that is contained on every survey.
The triumph of this paper is the mass of survey data to which this technique can be applied. As the authors state in the opening paragraph:
Studying the importance of non-cognitive skills, such as conscientiousness, perseverance, and motivation, has been hamstrung by the fact that many popular data sets in economics do not contain information on an individual’s personality traits. However, surveys contain a valuable but neglected piece of data: what respondents do not say. Respondents skip, refuse to answer, or claim ignorance on at least a few questions in virtually all surveys. When a respondent forgets to fill in answers to some questions on the survey form, or refuses to provide an answer to the interviewer, we gain important information about the respondent.
Very nice catch, thanks.
Henry Harpending
hmmm. You are taking a lot at face value here, Jason. Without bothering to read the whole study, let us mull over the issues in the abstract, shall we?
They find that people who dont answer many questions have less earnings. And how does the Gsoep measure earnings? Why, via a whole battery of questions of course! So people who answer less questions turn out to have less total earnings. Surprise, surprise.
Then they find that people who dont answer questions are less likely to die. And how do you think the Gsoep knows that they have died? Well, they often only know this from getting answers from others as to the question whether they are still alive. So people who answer fewer questions are less likely to have others give out information on them? Again, surprise, surprise.
It is the mortality effect that sounds particularly suspicious and should make you much more sceptical of this paper than you seem to be. The authors are right that IQ is related to longevity but in the opposite direction of what they find themselves: the smarter are usually found to live longer, so saying that the non-respondents are the dumbos who live longer doesnt quite fit, now does it?
Apart from all that, thanks for the write-up of the PhD conference. Very kind.
The authors test their results using non-survey income and death data:
On the direction of the IQ-longevity correlation, their abstract is poorly worded, as the last sentence of the above quote suggests.
Fair enough. The abstract indeed reads to say that more item non response would lead to lower morbidity.
Still, the authors are relying on what they call a gold-standard data file. And how do you think social security knows about income? Even in Germany taxes involve a large set of questions people have to fill in, so the potential for a spurious relation remains. They should focus on an item of earnings that is easier to manually verify, such as perhaps job title.
With the reversed longevity result though it does start to sound more plausible!