Survey non-response is a very clever way of extracting Conscientiousness personality scores from datasets lacking personality tests, which is good since Conscientiousness is majorly predictive of lifetime incomes and education albeit not as much as IQ (previously http://econlog.econlib.org/archives/2012/11/hedengrens_dog.html ); now there's a followup showing nonresponse is useful as a predictor in some more realworld datasets:
"Don’t Know? Or Don’t Care?: Predicting Educational Attainment Using Survey Item Response Rates and Coding Speed Tests as Measures of Conscientiousness", Hitt & Trivitt 2013; excerpts:
"Leading research shows the importance of non-cognitive skills for educational attainment, but advances in this research have been slowed by a common data limitation: most datasets do not contain explicit measures of non-cognitive skills. We examine a new proxy for non-cognitive skills, survey item response rates. Using a detailed national survey of American adolescents, we find that the percentage of questions left unanswered is a significant predictor of educational attainment. The fewer questions left unanswered, the higher the likelihood overall that respondents will enroll in college. We replicate our analysis using a more rudimentary dataset, of the kind typically used in program evaluations, and again find that item response rates are predictive of educational attainment. We posit that survey item response rates capture conscientiousness, a personality trait that is not explicitly measured in most surveys. Thus item response rates provide a convenient measure of non-cognitive skills. We also examine another proxy for non-cognitive skills, results on a coding speed test. Coding speed is also predictive of educational attainment, independent of cognitive ability. Our results suggest coding speed also captures conscientiousness, albeit different facets of conscientiousness than item response rates. We conclude that coding speed and item response rates can both be used to measure the impact of public policy on important non-cognitive skills.
The research that has established the connection between non-cognitive skills and educational attainment has used rich, but rare, datasets that contain questions about student attitudes and behavior, such as the National Longitudinal Surveys. These datasets are invaluable, but they cannot tell us about the performance of a particular program, such as experimental preschool programs or a private school voucher initiative. Evaluating these programs requires original data collection and such datasets are typically assembled without non-cognitive skills in mind, making it difficult to evaluate policies for their impact on non-cognitive skills. Education program evaluations typically emphasize cognitive abilities (as measured by test scores), attainment levels, and responses taken from surveys asking a narrow range of questions about program goals or other specific factors researchers believe may mediate the effects of the program. Such datasets rarely include explicit measures of non-cognitive ability. We examine a new proxy for non-cognitive ability that is present in most surveys and has historically been overlooked by human capital researchers: item response rates. Building on recent research from Hedengren and Strattman (2012), we examine a pattern in survey data by calculating survey item response rates - the frequency with which a respondent answers survey questions...Surveys are not cognitively challenging. They can be quite tedious and boring and respondents typically have little material incentive to complete them. As such, survey response rates inadvertently measure effort and focus. Hedengren and Strattman find that survey item response rates are positively correlated with income. Moreover, they observe that, after controlling for cognitive ability, survey item response rates are closely associated with a personality trait known as conscientiousness.
We test whether survey item response rates are indeed predictive of attainment, as one would expect of a measure of conscientiousness. We use two datasets to test this hypothesis: the 1997 National Longitudinal Survey of Youth (NLSY97) and smaller survey of private school students in Milwaukee, Wisconsin. Both datasets are surveys of adolescents. The NLSY97 is a particularly useful, nationally representative dataset collected by the US Bureau of Labor Statistics. The Milwaukee survey data form a less comprehensive dataset that resembles the datasets that education researchers typically have available to conduct program evaluations.
The field of research on non-cognitive skills has since matured considerably. For example, Cunha, Heckman and Schennach (2010) measure self-esteem and “locus of control” in parents and behavioral problems in children participating in the survey Children of the 1979 National Longitudinal Survey of Youth. They use these factors to estimate the influence of cognitive and non-cognitive skills in educational attainment. These non-cognitive abilities explain nearly as much of the variation in educational attainment as cognitive ability: “16% is due to adolescent cognitive capabilities; 12% is due to adolescent non-cognitive capabilities.” Cuhna and colleagues arrive at this result despite the fact that their data lacked measures of other important personality traits. Had additional measures been included in their data, it is plausible non-cognitive skills would explain an even greater percentage of the variance in educational attainment. Other recent work shows the importance of anger and personality traits (e.g. Almund et al., 2011).
Research on non-cognitive abilities provides insight to racial and gender disparities in educational attainment, for example higher education attainment. A significant development in higher education over the past three decades has been the increasing prevalence of women in college graduation despite increasing returns to college for both genders. Becker, Hubbard and Murphy develop a theoretical model showing this phenomenon can be explained largely by gender differences in non-cognitive skills and provide quantitative evidence to support their model. Women, on balance, have higher non-cognitive skills than men, and men have a wider variance in the distribution of non-cognitive skills (Becker, Hubbard and Murphy, 2010). A higher percentage of women than men possess the non-cognitive skills needed to succeed in college. Policies that improve non-cognitive skills in boys might well lead to greater gender parity in college enrollments and diplomas.
Income, educational attainment and longevity have all been linked to conscientiousness – that is, when researchers have been able to measure it. (Hill et al. 2011). Our primary concern is datasets that lack items that measure conscientiousness. However, before discussing that problem, we must discuss the development of conscientiousness measures that, when used, have provided key insight to educational attainment.
The study of conscientiousness stems from a larger project to map personality. Similar to the study of cognitive ability and IQ, a factor analytic approach has been adopted to analyze separate components of personality (Goldberg, 1993). Factor models are developed and refined through survey methods, wherein respondents are asked to group certain behaviors with one another. Certain groupings begin to emerge. As these groupings become apparent, personality psychologists have separated related behaviors into different personality factors. The field has now broadly adopted the “Big Five” Factor Model, which identifies the major components of personality as agreeableness, extraversion, neuroticism, openness to experience, and conscientiousness. This model is now increasingly being used in human capital research (e.g. Almlund et al. 2011).
In the “Big Five” model, the factors agreeableness, neuroticism and extroversion have names that are relatively self-explanatory.
Researchers with access to grades and disciplinary records have been able to link conscientiousness to academic success. Conscientiousness is strongly predictive of grades throughout elementary and high school (Poropat, 2009). Conditioning on cognitive ability, students who are more conscientious are more likely to complete homework assignments and show up to class (Lubbers et al. 2010; Conard, 2006). Grades, attendance, and homework completion are predictive of high school graduation (Allensworth & Easton, 2007; Segal 2012a), an unsurprising fact since graduation is often contingent on these factors. It follows that conscientiousness and related non-cognitive skills are predictive of high school graduation (Heckman, Humphries and Mader , 2010; Lleras 2008). Even into college where cognitive ability plays a large role, high school grades and conscientiousness remain strong predictors of persistence and attainment (Bowen, Chingos & McPherson 2009; Noftle and Robbins, 2007).
Other personality factors have been shown to be important to early academic success as well. Agreeableness and openness to experience are correlated with higher grades in elementary school, largely to the same degree as conscientiousness. At the secondary and postsecondary levels, however, conscientious stands alone as a strong predictor of performance (Poropat, 2009).
Program evaluations are beginning to more closely monitor non-cognitive skill development. For example, Chicago Public Schools now issues bi-annual district-wide surveys that contain five question items on conscientiousness and "grit." Such developments are promising, but they do not solve the problem that plagues past datasets. Researchers for decades did not explicitly measure conscientiousness or related noncognitive skills.
As in most surveys, NLSY97 respondents have the opportunity to skip questions or to answer “I don’t know.” Typically, researchers have treated skipped questions and answers of “I don’t know” as missing data. Hedengren and Strattman (2012) contend that many respondents may simply “plead ignorance” to questions to which they actually know the answer, signaling a loss of interest or a lack of effort – in other words, a lack of conscientiousness. A unique feature of the NLSY97 is that respondents are prompted by computer to complete questions that they may have skipped, making non-response less likely to result from an innocent error on the part of the respondent...The average item response rate is 98.8 percent the range is from 45.1 to 100 percent with 3,361 out of 8,984 respondents completing all appropriate questions. As of 2010, the average respondent was 28 years of age and had received 13.3 years of education, 922 respondents never completed high school, 1,918 completed high school but never enrolled in college, 1,295 enrolled in college but had yet to complete a four-year degree, and 237 had graduated with at least a four-year degree.
The results when estimating the linear BHM model are shown in Table 2. The full model is presented in Column 1. Coding speed and survey response rates are statistically significant and positive across all models. In the full OLS model, a one standard deviation increase in coding speed predicts an increase of 0.173 years of education received; a one standard deviation increase in item response rate predicts a 0.135 increase in years of education.
The literature on non-cognitive ability suggests that there should be a differential impact of changes in non-cognitive ability, across different levels of cognitive ability. Cunha and colleagues (2010) find improvements in non-cognitive skills have the greatest impact on high school graduation for students with low cognitive skills and the greatest impact on college completion for students with high cognitive skills. We find this exact pattern for response rates and coding speed. Table 4A shows the marginal effects of a change in item response rates, with cognitive ability set at five different levels and all other variables (including coding speed) set at the means. Table 4B similarly shows marginal effects of a change in coding speed, with all other variables (including item response rates) set at the means. Some interesting findings emerge from these tables. The non-cognitive skill measured by response rates is effective in reducing the probability of failing to complete high school, but does not have a measurable effect on GED completion. The non-cognitive skill captured in coding speed appears to reduce the probability of dropping out and getting a GED. Both measures of non-cognitive skills reduce the probability of obtaining only a high school diploma. The response rate measure increases the probability a student will complete a 4 year college degree but the coding speed measure increases the probability of attending college, completing a four-year degree, and completing a graduate degree. A one standard deviation increase in survey item response rates decreases the likelihood of failing to earn any degree by 1.92 percentage points for females in the 10th percentile overall of cognitive ability, versus a decrease of 0.17 percentage points for those in the 2090th percentile. We see the same pattern for males and with coding speed with the marginal effect of non-cognitive ability on completing high school monotonically decreasing in cognitive ability.
With respect to college completion, the opposite holds true in that increases in non-cognitive skills are more beneficial to students with higher cognitive abilities. For both measures of non-cognitive ability, the marginal effects of non-cognitive skills on completing a 4 year college degree are positive and consistently significant and show an overall trend of increasing with ability and household head education for both males and females. The coding speed measure also shows the same pattern for completion of a graduate degree.
We use data on Milwaukee private school students participating in a local school voucher program. The substance and quality of our Milwaukee dataset is much more akin to the data typically available to education researchers. Collected by the School Choice Demonstration Project at the University of Arkansas, the dataset contains pen-and-paper surveys of more 424 private school ninth and tenth graders participating in a school voucher program that is targeted to low income students. In 2007, students were issued paper surveys containing 42 basic questions about their home and school environments. The survey contained no questions about students’ personalities, self-image or conscientiousness - again, typical of surveys offered in program evaluations. School staff administered the surveys...We use the percentage of answers "Not Sure" to calculate item response rates. This calculation of response rate differs somewhat from that used in the NLSY97, which also included skipped questions. In the Milwaukee surveys, skipped questions and illegible responses were coded together as "Not Ascertained." Our review of the data found that answers "Not Ascertained" were heavily concentrated in a small percentage of student surveys that, for unknown reasons, were thoroughly illegible. In these instances, we cannot determine whether students left questions blank, were called out of the classroom mid-survey, or wrote illegibly...We limit our analysis to 9 th and 10 th graders who took the pen and paper survey in 2007, since this is the only student group with sufficient time during the observation period to have graduated high school and enrolled in college.
Survey item response rate is our proxy for non-cognitive ability. Our controls for cognitive ability are scores on standardized math and reading tests, which were administered by school staff on site. Our controls for human capital are a combination of student and school reported factors: race, gender, English language learner, and qualification for free/reduced lunch program Given that survey administration, testing conditions and data reporting can vary in quality between schools, we cluster our standard errors at the school level.
The patterns are strikingly similar to those found in our analysis of the NLSY97. An increase in the response rate or cognitive ability significantly increases the probability of enrolling in a 4 year college. Once again, the estimated marginal effects vary across cognitive ability. Marginal effects by gender and cognitive ability are shown in Table 7. For both males and females we see the largest reduction in probability in dropping out of high school for those with less cognitive ability and the highest increase in the probability of enrolling in a 4 year college for those with higher cognitive ability. For males at the tenth percentile of cognitive ability, a one standard deviation increase in item response rates increases the likelihood of enrolling in a four year college by 3.49 percent, whereas at the 90 th percentile it increases the likelihood by 5.01 percent."
#personality #psychology #Conscientiousness
"Don’t Know? Or Don’t Care?: Predicting Educational Attainment Using Survey Item Response Rates and Coding Speed Tests as Measures of Conscientiousness", Hitt & Trivitt 2013; excerpts:
"Leading research shows the importance of non-cognitive skills for educational attainment, but advances in this research have been slowed by a common data limitation: most datasets do not contain explicit measures of non-cognitive skills. We examine a new proxy for non-cognitive skills, survey item response rates. Using a detailed national survey of American adolescents, we find that the percentage of questions left unanswered is a significant predictor of educational attainment. The fewer questions left unanswered, the higher the likelihood overall that respondents will enroll in college. We replicate our analysis using a more rudimentary dataset, of the kind typically used in program evaluations, and again find that item response rates are predictive of educational attainment. We posit that survey item response rates capture conscientiousness, a personality trait that is not explicitly measured in most surveys. Thus item response rates provide a convenient measure of non-cognitive skills. We also examine another proxy for non-cognitive skills, results on a coding speed test. Coding speed is also predictive of educational attainment, independent of cognitive ability. Our results suggest coding speed also captures conscientiousness, albeit different facets of conscientiousness than item response rates. We conclude that coding speed and item response rates can both be used to measure the impact of public policy on important non-cognitive skills.
The research that has established the connection between non-cognitive skills and educational attainment has used rich, but rare, datasets that contain questions about student attitudes and behavior, such as the National Longitudinal Surveys. These datasets are invaluable, but they cannot tell us about the performance of a particular program, such as experimental preschool programs or a private school voucher initiative. Evaluating these programs requires original data collection and such datasets are typically assembled without non-cognitive skills in mind, making it difficult to evaluate policies for their impact on non-cognitive skills. Education program evaluations typically emphasize cognitive abilities (as measured by test scores), attainment levels, and responses taken from surveys asking a narrow range of questions about program goals or other specific factors researchers believe may mediate the effects of the program. Such datasets rarely include explicit measures of non-cognitive ability. We examine a new proxy for non-cognitive ability that is present in most surveys and has historically been overlooked by human capital researchers: item response rates. Building on recent research from Hedengren and Strattman (2012), we examine a pattern in survey data by calculating survey item response rates - the frequency with which a respondent answers survey questions...Surveys are not cognitively challenging. They can be quite tedious and boring and respondents typically have little material incentive to complete them. As such, survey response rates inadvertently measure effort and focus. Hedengren and Strattman find that survey item response rates are positively correlated with income. Moreover, they observe that, after controlling for cognitive ability, survey item response rates are closely associated with a personality trait known as conscientiousness.
We test whether survey item response rates are indeed predictive of attainment, as one would expect of a measure of conscientiousness. We use two datasets to test this hypothesis: the 1997 National Longitudinal Survey of Youth (NLSY97) and smaller survey of private school students in Milwaukee, Wisconsin. Both datasets are surveys of adolescents. The NLSY97 is a particularly useful, nationally representative dataset collected by the US Bureau of Labor Statistics. The Milwaukee survey data form a less comprehensive dataset that resembles the datasets that education researchers typically have available to conduct program evaluations.
The field of research on non-cognitive skills has since matured considerably. For example, Cunha, Heckman and Schennach (2010) measure self-esteem and “locus of control” in parents and behavioral problems in children participating in the survey Children of the 1979 National Longitudinal Survey of Youth. They use these factors to estimate the influence of cognitive and non-cognitive skills in educational attainment. These non-cognitive abilities explain nearly as much of the variation in educational attainment as cognitive ability: “16% is due to adolescent cognitive capabilities; 12% is due to adolescent non-cognitive capabilities.” Cuhna and colleagues arrive at this result despite the fact that their data lacked measures of other important personality traits. Had additional measures been included in their data, it is plausible non-cognitive skills would explain an even greater percentage of the variance in educational attainment. Other recent work shows the importance of anger and personality traits (e.g. Almund et al., 2011).
Research on non-cognitive abilities provides insight to racial and gender disparities in educational attainment, for example higher education attainment. A significant development in higher education over the past three decades has been the increasing prevalence of women in college graduation despite increasing returns to college for both genders. Becker, Hubbard and Murphy develop a theoretical model showing this phenomenon can be explained largely by gender differences in non-cognitive skills and provide quantitative evidence to support their model. Women, on balance, have higher non-cognitive skills than men, and men have a wider variance in the distribution of non-cognitive skills (Becker, Hubbard and Murphy, 2010). A higher percentage of women than men possess the non-cognitive skills needed to succeed in college. Policies that improve non-cognitive skills in boys might well lead to greater gender parity in college enrollments and diplomas.
Income, educational attainment and longevity have all been linked to conscientiousness – that is, when researchers have been able to measure it. (Hill et al. 2011). Our primary concern is datasets that lack items that measure conscientiousness. However, before discussing that problem, we must discuss the development of conscientiousness measures that, when used, have provided key insight to educational attainment.
The study of conscientiousness stems from a larger project to map personality. Similar to the study of cognitive ability and IQ, a factor analytic approach has been adopted to analyze separate components of personality (Goldberg, 1993). Factor models are developed and refined through survey methods, wherein respondents are asked to group certain behaviors with one another. Certain groupings begin to emerge. As these groupings become apparent, personality psychologists have separated related behaviors into different personality factors. The field has now broadly adopted the “Big Five” Factor Model, which identifies the major components of personality as agreeableness, extraversion, neuroticism, openness to experience, and conscientiousness. This model is now increasingly being used in human capital research (e.g. Almlund et al. 2011).
In the “Big Five” model, the factors agreeableness, neuroticism and extroversion have names that are relatively self-explanatory.
Researchers with access to grades and disciplinary records have been able to link conscientiousness to academic success. Conscientiousness is strongly predictive of grades throughout elementary and high school (Poropat, 2009). Conditioning on cognitive ability, students who are more conscientious are more likely to complete homework assignments and show up to class (Lubbers et al. 2010; Conard, 2006). Grades, attendance, and homework completion are predictive of high school graduation (Allensworth & Easton, 2007; Segal 2012a), an unsurprising fact since graduation is often contingent on these factors. It follows that conscientiousness and related non-cognitive skills are predictive of high school graduation (Heckman, Humphries and Mader , 2010; Lleras 2008). Even into college where cognitive ability plays a large role, high school grades and conscientiousness remain strong predictors of persistence and attainment (Bowen, Chingos & McPherson 2009; Noftle and Robbins, 2007).
Other personality factors have been shown to be important to early academic success as well. Agreeableness and openness to experience are correlated with higher grades in elementary school, largely to the same degree as conscientiousness. At the secondary and postsecondary levels, however, conscientious stands alone as a strong predictor of performance (Poropat, 2009).
Program evaluations are beginning to more closely monitor non-cognitive skill development. For example, Chicago Public Schools now issues bi-annual district-wide surveys that contain five question items on conscientiousness and "grit." Such developments are promising, but they do not solve the problem that plagues past datasets. Researchers for decades did not explicitly measure conscientiousness or related noncognitive skills.
As in most surveys, NLSY97 respondents have the opportunity to skip questions or to answer “I don’t know.” Typically, researchers have treated skipped questions and answers of “I don’t know” as missing data. Hedengren and Strattman (2012) contend that many respondents may simply “plead ignorance” to questions to which they actually know the answer, signaling a loss of interest or a lack of effort – in other words, a lack of conscientiousness. A unique feature of the NLSY97 is that respondents are prompted by computer to complete questions that they may have skipped, making non-response less likely to result from an innocent error on the part of the respondent...The average item response rate is 98.8 percent the range is from 45.1 to 100 percent with 3,361 out of 8,984 respondents completing all appropriate questions. As of 2010, the average respondent was 28 years of age and had received 13.3 years of education, 922 respondents never completed high school, 1,918 completed high school but never enrolled in college, 1,295 enrolled in college but had yet to complete a four-year degree, and 237 had graduated with at least a four-year degree.
The results when estimating the linear BHM model are shown in Table 2. The full model is presented in Column 1. Coding speed and survey response rates are statistically significant and positive across all models. In the full OLS model, a one standard deviation increase in coding speed predicts an increase of 0.173 years of education received; a one standard deviation increase in item response rate predicts a 0.135 increase in years of education.
The literature on non-cognitive ability suggests that there should be a differential impact of changes in non-cognitive ability, across different levels of cognitive ability. Cunha and colleagues (2010) find improvements in non-cognitive skills have the greatest impact on high school graduation for students with low cognitive skills and the greatest impact on college completion for students with high cognitive skills. We find this exact pattern for response rates and coding speed. Table 4A shows the marginal effects of a change in item response rates, with cognitive ability set at five different levels and all other variables (including coding speed) set at the means. Table 4B similarly shows marginal effects of a change in coding speed, with all other variables (including item response rates) set at the means. Some interesting findings emerge from these tables. The non-cognitive skill measured by response rates is effective in reducing the probability of failing to complete high school, but does not have a measurable effect on GED completion. The non-cognitive skill captured in coding speed appears to reduce the probability of dropping out and getting a GED. Both measures of non-cognitive skills reduce the probability of obtaining only a high school diploma. The response rate measure increases the probability a student will complete a 4 year college degree but the coding speed measure increases the probability of attending college, completing a four-year degree, and completing a graduate degree. A one standard deviation increase in survey item response rates decreases the likelihood of failing to earn any degree by 1.92 percentage points for females in the 10th percentile overall of cognitive ability, versus a decrease of 0.17 percentage points for those in the 2090th percentile. We see the same pattern for males and with coding speed with the marginal effect of non-cognitive ability on completing high school monotonically decreasing in cognitive ability.
With respect to college completion, the opposite holds true in that increases in non-cognitive skills are more beneficial to students with higher cognitive abilities. For both measures of non-cognitive ability, the marginal effects of non-cognitive skills on completing a 4 year college degree are positive and consistently significant and show an overall trend of increasing with ability and household head education for both males and females. The coding speed measure also shows the same pattern for completion of a graduate degree.
We use data on Milwaukee private school students participating in a local school voucher program. The substance and quality of our Milwaukee dataset is much more akin to the data typically available to education researchers. Collected by the School Choice Demonstration Project at the University of Arkansas, the dataset contains pen-and-paper surveys of more 424 private school ninth and tenth graders participating in a school voucher program that is targeted to low income students. In 2007, students were issued paper surveys containing 42 basic questions about their home and school environments. The survey contained no questions about students’ personalities, self-image or conscientiousness - again, typical of surveys offered in program evaluations. School staff administered the surveys...We use the percentage of answers "Not Sure" to calculate item response rates. This calculation of response rate differs somewhat from that used in the NLSY97, which also included skipped questions. In the Milwaukee surveys, skipped questions and illegible responses were coded together as "Not Ascertained." Our review of the data found that answers "Not Ascertained" were heavily concentrated in a small percentage of student surveys that, for unknown reasons, were thoroughly illegible. In these instances, we cannot determine whether students left questions blank, were called out of the classroom mid-survey, or wrote illegibly...We limit our analysis to 9 th and 10 th graders who took the pen and paper survey in 2007, since this is the only student group with sufficient time during the observation period to have graduated high school and enrolled in college.
Survey item response rate is our proxy for non-cognitive ability. Our controls for cognitive ability are scores on standardized math and reading tests, which were administered by school staff on site. Our controls for human capital are a combination of student and school reported factors: race, gender, English language learner, and qualification for free/reduced lunch program Given that survey administration, testing conditions and data reporting can vary in quality between schools, we cluster our standard errors at the school level.
The patterns are strikingly similar to those found in our analysis of the NLSY97. An increase in the response rate or cognitive ability significantly increases the probability of enrolling in a 4 year college. Once again, the estimated marginal effects vary across cognitive ability. Marginal effects by gender and cognitive ability are shown in Table 7. For both males and females we see the largest reduction in probability in dropping out of high school for those with less cognitive ability and the highest increase in the probability of enrolling in a 4 year college for those with higher cognitive ability. For males at the tenth percentile of cognitive ability, a one standard deviation increase in item response rates increases the likelihood of enrolling in a four year college by 3.49 percent, whereas at the 90 th percentile it increases the likelihood by 5.01 percent."
#personality #psychology #Conscientiousness
+Yonatan Zunger How about a "Show less" link at the top of an expanded post?Nov 27, 2014
