The Sipp Cognitive Research Evaluation Experiment - Basic Results and Documentation: Working Paper 212

In response to known and suspected problems in the measurement of program participation and related variables, Census Bureau research staff developed new, experimental survey procedures, based on cognitive theory and research, to reduce response errors in the Survey of Income and Program Participation (SIPP). We implemented a “field laboratory” test of the new procedures, using both an experimental design and an administrative record check. A key feature of the new procedures was getting households to use their personal income records as a substitute for faulty, minimum effort memory retrieval. Results of the SIPP Cognitive Research Evaluation Experiment indicate that the new procedures had no important effects on reducing either underreporting or overreporting errors in respondents’ reports of participation in the income programs tested. However, by the second interview wave, the new procedures did produce substantial improvement in the reporting of income amounts. Experimental group households did use personal income records at astonishingly high rates; furthermore, record use correlated with the quality of income amount reporting. This paper describes the basic features of the experimental procedures and their evolution; it presents the available evidence concerning the implementation of those procedures; it summarizes the key substantive findings of the experiment concerning program participation and program income reporting quality; and finally, it offers some possible reasons why record use did not affect reporting of program participation, but did have important effects on income amount reports.