Media Bias Beyond a Reasonable Doubt

Last Updated April 2, 2005. The graphic at left [click the thumbnail] is from an innovative study, “A Measure of Media Bias“, by Tim Groseclose/UCLA/Stanford and Jeff Milyo/University of Chicago/University of Missouri. I first read this study when published as a working paper in September 2003. It was released in final form December 2004. I will explain their methodology, but before you jump ship from fear of statistics, have a look at the study conclusion:

Although we expected to find that most media lean left, we were astounded by the degree. A norm among journalists is to present both sides of the issue.Consequently, while we expected members of Congress to cite primarily think tanks that are on the same side of the ideological spectrum as they are, we expected journalists to practice a much more balanced citation practice, even if the journalist’s own ideology opposed the think tanks that he or she is sometimes citing. This was not always the case. Most of the mainstream media outlets that we examined (ie all those besides Drudge Report and Fox News’ Special Report) were closer to the average Democrat in Congress than they were to the median member of the House.

I’ve quoted the original September 2003 version (PDF from the Harvard archives) of their paper because the authors have pulled their verbal punches a bit in the final version. And more from the same 2003 study introduction:

Our results show a very significant liberal bias. All of the news outlets except Fox News’ Special Report received a score to the left of the average member of Congress. Moreover, by one of our measures all but three of these media outlets (Special Report, the Drudge Report, and ABC’s World News Tonight) were closer to the average Democrat in Congress than to the median member of the House of Representatives. One of our measures found that the Drudge Report is the most centrist of all media outlets in our sample. Our other measure found that Fox News’ Special Report is the most centrist. These findings refer strictly to the news stories of the outlets. That is, we omitted editorials, book reviews, and letters to the editor from our sample.

What makes this a unique and compelling study is that it directly measures media output. You’ve no doubt read of the various studies which ask reporters to rate themselves, or examine the newsroom’s voting record, etc.. The problem with all of those studies is that while they do measure political leanings, but they do not measure the actual news output. Consider the Pew Research result that in 2004 reporters voted for Kerry over Bush by 9:1. That doesn’t demonstrate media bias – a valid criticism is “that is meaningless, reporters do not reflect their personal views in their writing“.This is how they did it: Groseclose and Milyo used a creative statistical approach to derive an objective measure of conservative/liberal bias of the straight news coverage (no editorials) of twenty selected “mainstream media” sources, including all the usual suspects. Their insight was to construct a comparison between members of the US Congress and each media outlet – using the ratings of members voting records as published by the liberal Americans for Democratic Action (ADA). These ADA scores distribute the members of both houses on a scale from extremely liberal to extremely conservative.The ground truth that Groseclose/Milyo were after is the score that represents the centrist American voter. To get there they do some small statistical adjustments to arrive at the adjusted ADA score (as displayed in the graphic above). Skipping the messy details, all covered very clearly in the paper, their resulting adjusted ADA scores gives the centrist members of Congress a score of about 50.How do they connect the profile of Congress with the corresponding profile of the media outlets? The next clever step is to derive the adjusted ADA scores for the 200 thinktanks most frequently cited by members. Finally they map the media outlets ADA scores by analyzing the frequency with which each straight-news piece quotes the same set of think tanks. This is a massive analytical task – the researchers used a large team of assistants.

As a simplified example, imagine that there were only two think tanks, one liberal and one conservative. Suppose that the New York Times cited the liberal think tank twice as often as the conservative one. Our method asks: What is the typical ADA score of members of Congress who exhibit the same frequency (2:1) in their speeches? This is the score that we would assign to the New York Times.

If you prefer PDFs for a longish report like this, Harvard has this in their archives. Footnotes:[1] Wall Street Journal left of CBS, New York Times? Some readers have commented “how can this be, everybody knows the Journal is conservative”? Well, yes, the editorial pages of the WSJ are conservative – the Groseclose Milyo study is about straight news pages.It is best to read the study report itself. For a quick overview, here are the study comments on the WSJ result:

One surprise is the Wall Street Journal, which we find as the most liberal of all 20 news outlets. We should first remind readers that this estimate (as well as all other newspaper estimates) refers only to the news of the Wall Street Journal; we omitted all data that came from its editorial page. If we included data from the editorial page, surely it would appear more conservative. Second, some anecdotal evidence agrees with our result. For instance, Reed Irvine and Cliff Kincaid (2001) note that “The Journal has had a long-standing separation between its conservative editorial pages and its liberal news pages.” Paul Sperry, in an article titled the “Myth of the Conservative Wall Street Journal,” notes that the news division of the Journal sometimes calls the editorial division “Nazis.” “Fact is,” Sperry writes, “the Journal’s news and editorial departments are as politically polarized as North and South Korea.”[24]Third, a recent poll from the Pew Research Center indicates that a greater percentage of Democrats, 29%, say they trust the Journal than do Republicans, 23%. Importantly, the question did not say “the news division at the Wall Street Journal.” If it had, Democrats surely would have said they trusted the Journal even more, and Republicans even less.[25]

Finally, and perhaps most important, a scholarly study by Lott and Hasset (2004) gives evidence that is consistent with our result. As far as we are aware this is the only other study that examines the political bias of the news pages of the Wall Street Journal. Of the ten major newspapers that it examines, the study estimates the Wall Street Journal as the second-most liberal.[26] Only Newsday is more liberal, and the Journal is substantially more liberal than the New York Times, Washington Post, L.A. Times, and USA Today.

The Paul Sperry article referenced above can be found here: The Myth of the Conservative Wall Street Journal .

 

15 thoughts on “Media Bias Beyond a Reasonable Doubt

  1. BD,

    except for that one major outlier.

    I was a bit surprised to see WSJ left of NYT, but I think it is correct. The WSJ news pages are significantly left – the editorial content significantly right.

  2. This study found that the WSJ is the most liberal mainstream media outlet? That sort of result will discredit the entire study to any liberal who sees it. The chart looks about like I would expect it to look, except for that one major outlier.

  3. And you sincerely believe a study that beyond a shadow of a doubt claims the Wall Street Journal is by far the most liberal major news organization in the country?

  4. Hi,

    The Groseclose-Milyo paper is fatally flawed and its conclusions are wrong.

    A detailed response/critique is here:
    http://media.eriposte.com/2-9.htm

    Regarding Lott/Hassett, I’m not sure which paper of theirs Sperry refers to, but if it is the paper “Is Newspaper Coverage of Economic Events Politically Biased?”, then that paper is also seriously flawed. Rebuttal here:
    http://media.eriposte.com/2-10.htm

    Regards.

  5. TR writes

    The Groseclose-Milyo paper is fatally flawed and its conclusions are wrong
    .
    Regarding Lott/Hassett, I’m not sure which paper of theirs Sperry refers to, but if it is the paper “Is Newspaper Coverage of Economic Events Politically Biased?”, then that paper is also seriously flawed.

    Fatally flawed, seriously flawed – rather strong declaratives!

    If you’re not sure which Sperry paper is referenced, you could try your text search function – it is discussed several places in the body of the article and in the footnotes.

    As to fatally flawed – I’ve read your post criticizing Groseclose/Milyo (G/M) carefully. As far as I can tell it is a non-contact sport – you posit various straw men and then knock them down. I couldn’t find anything logically connected to the actual content or methodology of the Groseclose/Milyo study. I spent enough time examining material on your site to conclude that your mission is to attempt to rebut any evidence of liberal media bias in the US news. Did I get that right? That’s an exhausting mission – I salute your energy.

    Reading your criticism led me to scrutinize my own attempts to summarize G/M – it became quickly obvious that I had also misrepresented their study. While I attempted to explain the general concept to a non-mathematics reader I introduced my own spurious points. E.g., regarding quantifying the position of a centrist voter, which is interesting but quite peripheral to the logic of the study. That’s the same sort of error that tripped up Nunberg. I’ll fix it soon…

    I’ve been searching on and off for the past year – unsuccessfully – for informed criticism that finds flaws with their work (the null search result doesn’t prove non-existence, possibly just lack of persistence). The response to the Geoff Nunberg critique is footnoted/referenced in the Dec 2004 Groseclose/Milyo – the Nunberg critique itself is linked in their response. While I thought the G/M response was conclusive, the absence of any counter-rebuttal does imply that Nunberg was satisfied that his points were covered in the reply. I should say absence of a counter-rebuttal by Nunberg in my search results.

    There’s a volume of amateur discussion pro and con on the web – surprisingly, conservatives generally think the results are valid, liberals generally don’t like the results. Which tells us little about the strength of the G/M paper.

    The commentary I’ve found by professionals with the statistics competence required to analyze G/M have been positive about the methodology (but I infer these profs agreed with the results, irrespective of the quality of the study). So I’m still looking… comments suggesting serious critiques would be appreciated.

    Latstly, the following argument is similar to what I just criticized in the eriposte.com piece, in that it doesn’t make contact with the G/M content, but I’ll offer it anyway because it is less work:

    The researchers published a working draft of this study 15 months before final publication. That draft was circulated and reviewed at Harvard, Yale and other economics grad schools, as I’ve found it in their archives.

    I’ve not discussed with the researchers, but I do have a Ph.D. in mathematics and thus some research/grad school time behind me. That protocol is normally followed by researchers soliciting criticism and contributions before final publication – i.e., striving for the highest quality result they can manage. Since the final version is subtantively the same as the original as to methodology and central conclusions, that supports the inference that the professional community doesn’t have serious problems with this work.

  6. Hi,

    You say:
    “As far as I can tell it is a non-contact sport – you posit various straw men and then knock them down. I couldn’t find anything logically connected to the actual content or methodology of the Groseclose/Milyo study.”

    I am afraid you must not have read my response carefully. I don’t create any straw men at all. The examples I pick are cases that are directly related to the authors’ assumptions. As I point out repeatedly, the authors’ ACLU example is in itself a vindication of my critique of the first part of their paper. The authors’ example citing the median based approach is in itself a vindication of my critique of their approach. I simply borrowed their example and modified it with more numbers to make the point even more clearer.

    Also, just because a paper is circulated among peers and receives no significant critiques that changes its position does not indicate that the paper is correct. History is replete with papers that were published after peer review, that were later shown to be incorrect. The real question is whether or not most of the people publishing research on media bias truly understand how to define the question of media bias (especially in the American context). It is possible to get carried away with a sophisticated study that produces interesting conclusions based on flawed assumptions.

    Thanks.

  7. TR writes:

    The Groseclose-Milyo paper is fatally flawed and its conclusions are wrong.

    TR decries linking of media newsreporting politics to that of elected congresscritters, and also wants to limit definitions of bias to the “accuracy” of reporting, as if bias only produces inaccuracy in what is said.

    These form the basis for TR’s “there’s no such thing as liberal media bias” articles, and are specious.

    As he admits, selection of what news to report is a huge issue, which he admits is also included in his definition of “accuracy.” Which logically leads to the conclusion that his preference for an “accuracy” measure is really no different from what G&M have measured. They simply measured non-critique citations of certain sources. Their methodology is not biased by the selection of which sources.

    On the media-congress relationship, it’s an obfuscation to play with the idea that congress references “centrist” sources more. Play with the data and outcomes all you like, you still will get the same relative results, which is what counts in this case. Media news reporting is generally liberal — whether a lot or a little — rather than conservative.

    And its no surprise. Rabidly liberal news reporters can’t possibly compensate for their own personal biases. Pew research leaves no doubt of the personal views of national reporters. How can national reporters so one-sided in their societal views (91% don’t think God-belief necessary to be moral, 88% think homosexuality should be accepted by society) think they can possibly remove their bias in reporting to people who think so differently than they? (Public: 40% and 51% on the same questions)

  8. MrPete,

    Thanks for your comment – all excellent points.

    Apology – for unobvious reasons Spam Karma marked your comment for the moderation queue. That is why you did not see your comment immediately posted.

    I’ll do some testing to determine why Spam Karma was suspicious.

  9. The Groseclose and Milyo survey assume that democrat = liberal. This is deeply flawed. 20% of the population is liberal, so only 40% of the democratic party – at most – is liberal.

    So, Groseclose and Milyo are saying that the democratic party, which is at most only 40% liberal, defines liberal-ness.

    Seems to me that there was an agenda – to defining the terms that would result in the desired results.

    Another point worth mentioning is that the media would more than likely seem to have a liberal bias, even if it didn’t, since the duty of the media is to act as a watchdog and to question authority. Both liberals and the press share this characteristic, so people could reasonably infer a liberal bias even if no bias exists.

  10. Homanid,

    I think you missed the essential concept of the study: the liberal/conservative rankings are NOT based on democrat = liberal in any way.

    The ratings are based upon the voting records as published by the liberal Americans for Democratic Action (ADA). These ADA scores distribute the members of both houses on a scale from extremely liberal to extremely conservative.

  11. If we accept that GM are actually measuring bias, this is what their finding indicate:

    1. Most media outlets are more liberal than the average U.S. voter.
    2. With the exception of the WSJ, all of the media outlets included in their study are more conservative than the average democrat.
    3. All media outlets are more liberal than the average Republican voter.
    4. Fox News (ADA score = 40) is 23 points more liberal than the average Republican voter (ADA score = 17). Why don’t Republicans complain about the liberal bias on FOX News?
    5. The “distance” (23 points) between FOX News and the average Republican voter is GREATER than the distance between the average U.S. voter and the NYT (20 points).

  12. Many thanks for your insights. I noted that your #2 implies that the average journalist in the GM sample succeeds to shift her writing closer to the avg. voter than her political preference [typically 90% Democrat per subjective surveys]. I suppose that is an interesting result on its own.

    Your comment also alerted me to the fact that UCLA has rearranged subdomains, breaking two links – which I just repaired.

  13. All this discussion about the accuracy of the methodology is downright funny.

    From 1964 to 1976 people in the media voted for the more liberal candidate from 81 to 94% of the time.

    That, more than anything else could, tells you where their interests lie.

    There, case settled!

  14. @pagpag: At times conservatives have noted liberal bias in one article or another on Fox News. I believe lack of general complaint against them from conservatives is due to Fox being the most conservative mainstream option available. Fox at times makes obvious attempts to bring in liberal viewpoints, and conservatives respect that a “Fair and Balanced” approach would need to include that.

Comments are closed.