Respondent & Interviewer Effects

Results from the General Social Survey’s “important matters” (GSS-IM) name generator question provide an important—and well documented—cautionary note about the potential limitations that can arise from the demanding nature of gathering social network data. Using data from the 2004 GSS, McPherson, Smith-Lovin, and Brashears (2006) report that since the initial inclusion of the GSS-IM in the 1985 GSS, the number of average discussion partners had dropped by one (from essentially 3 in 1985 to 2 in 2004), and that the number of people who reported being isolated (i.e., reported having no GSS-IM partners nominated) basically doubled to nearly 20%.

This finding generated substantial, and quick, reactions from many scholars in the network field, in part because they seemed to diverge so notably from other data sources (Fischer 2009). Fischer commissioned a survey experiment, to attempt to understand what was going on, especially with the dramatic increase in those who reported being isolated on the GSS-IM measure. One key change between the survey waves was that the important matters network module fell near the middle of the survey in 1985, and at the end of the survey in 2004. Additionally, another module immediately preceding this one asked respondents to name organizations they were members of, and then had follow up questions about each identified organizational membership. Perhaps this primed respondents to realize that each response to the name generator was likely to spur additional name interpreter questions. So, in an attempt to estimate the potential fatigue and learning effects the design may have produced, a 2010 survey randomized whether respondents received the 1985 or 2004 design. These data basically replicated the observed differences (the apparent increase in isolation) across the two designs; concluding this particular finding was largely due to design effects (Fischer 2011).99 Moreover, additional re-analyses and subsequent survey experiments suggested interviewer effects—attributable to a very small number of interviewers—were likely the primary source of the difference, concluding the observed increase in isolation from 1985 to 2004 was entirely an artifact (Paik and Sanchagrin 2013). However, in light of these alterations to our understanding of the isolation trend (stability) from the 2004 GSS, it is important to note that “Even accounting for the inflated number of zeros in 2004, there is a major decline in the number of named alters in the data” (McPherson, Smith-Lovin, and Brashears 2009 :680).