Google Alerts really wants me to know about the work of University of Chicago economists Matthew Gentzkow and Jesse Shapiro.  Every few weeks, I get an email touting a Wall Street Journal story about their research suggesting that TV’s arrival in the US had a beneficial effect on students’ cognitive achievement.

The problem?  The article appeared in September 2008.

Maybe Google Alerts knows about this blog, though, and is suggesting that I spotlight the economists’ unique methodologies, not the study.  I spoke with them about the detective work of finding patterns in reams of data.

DK: How would you characterize how an economist would study children and media?

MG: Economists try to find some factor that’s shifting media exposure, that doesn’t directly effect, or correlate with things that affect, the behavior.  Correlations between exposure and behavior can be confounded by what drives the heterogeneity of exposure across families or across time.

JS: This gets put under the label “natural experiments.” Ideally, we would all love to do randomized experiments, to evaluate media the same way we evaluate drugs, but to see the impact of having television versus not having television, you can’t really do that, or at least it’s hard.  So, we look for something as close as possible to experiments.

MG: A lot of the interest is in longer-run effects so even if you could isolate some kids from some media, following those kids for 30 years is extremely difficult.

DK: For the study profiled in the Journal, were you looking at something about children and media, or looking at other economic factors and this jumped out as an effect?

JS: We were looking at data from a huge survey done in the early 1960s, and it occurred to us that the timing was just right to look at effects of television’s introduction in the ’40s and ’50s.  We were thinking about what television might have affected, and this huge trove of data was sitting there – they’d given standardized tests to hundreds of thousands of kids all over the country.  We couldn’t use this strategy with a few thousand kids or ten thousand kids; it required this huge amount of data.

DK: How do you look for potential influencing factors – tease out what’s relevant – in such a massive amount of data?

JS: There are two things you need to put structure to:

What factors are changing the media environment?  We had this nice natural experiment where TV was not available in some places while it was available in other places, as it was gradually rolled out in the 40s and 50s.  We spent quite a bit of time trying to understand why some places got it sooner.

At the same time, you’re looking for variables in the data that allow you to hold other potential confounds in the background – to maximize the “signal to noise ratio” and make it easier to see the effects of media if they’re present.

MG: Even when you have something that seems like a natural experiment, you have to work pretty hard to assess its validity.  Who got TV early and who got TV late wasn’t random.  Predictably, the first places were big cities – with the most advertising capability and so forth – and those places are systematically different.  Just comparing early TV places to late TV places would be really confounded.  My idea was to just compare small, fairly rural places; some happen to be close to a big city and so got TV earlier, while others didn’t.  That seemed much more random, and the characteristics of places that got TV or didn’t look pretty similar.

DK: Is it possible to account for content?

MG: It’s a limitation of natural experiment design.  You introduce television, and you only have TV or no TV…not some places that got educational television and others that got westerns or sports.

DK: Would it be harder, also, to tease out media effects in today’s environment, with so many more media influences on kids?

JS: The introduction of television was a change on a scale vastly bigger than anything since.  Going, in 10 years, from no TV to people watching 4 hours a day was just massive.  Video games or iPads – those are all changes on a much smaller scale.

DK: What would you tell people who create children’s media about how to read research? How can a person without a research background spot flaws?

MG: Think about who is the control group. Why should I believe that they are the same as the treatment group except for the media exposure?

We were studying effects of television using the natural experiment, but we were interested in what would have happened using the correlational method.  If you compared kids who reported watching a lot of TV with kids who didn’t, the answers vary dramatically depending on what you hold constant.  At that time, richer families were more likely to own television; kids from more educated families (that had TV) watched less.  Holding income constant, you generally would find that watching TV was bad for kids; holding family background but not income constant, you’d generally find it was good.  So, ask if the control group is good, or if it’s just – in hard ways to detect – poorer or richer, or more or less educated, or have more or less stability at home.

Unless someone has a really good argument otherwise, my default position is not to believe anything that doesn’t say it was explicitly randomized, that looks at “effects” on pretty distant, big picture outcomes like test scores, grades, school attendance, or whether kids get into college.

As Drs. Gentzkow and Shapiro point out, economic studies differ from other methods in their reliance on population-size analyses, broader social movements, and long view.  It’s a more effective tool for considering what happened and why, than it is for projecting future developments or setting strategy.

Word of the Post:

“Natural Experiment:” One in which the experimental conditions are determined by external forces such as nature, policy or the built environment, rather than a controlled choice or arrangement by the investigator.