Convincing Evidence That the Exit Poll Discrepancies Were Caused by Bush Supporters Being Less Willing Than Kerry Supporters to Fill Out Exit Poll Questionnaires?

January 19, 2005

Edison/Mitofsky today publicly released a 77 page report on what went wrong in their Election 2004 exit polls. 

(CNN story on the Edison/Mitofsky Report available here.)

(PDF file of report currently available here from CNN.)

(Also, the PDF is archived on my Typepad account here.)

[Sidenote: For a full background on what went wrong, please see my post here.  Or, for an even fuller background please see the many exit poll posts of Mark Blumenthal of MysteryPollster.com.   But for the purposes of this post, the key thing that went wrong was that the exit polls—as they appeared on various news websites (e.g., CNN) up to the wee early morning hours after Election Day (i.e., until about 2 or 3 AM on Wednesday, Nov 3)—overwhelmingly predicted better showings by Senator Kerry than the official tallies did in virtually all states: safe Democrat states, safe Republican states, and battleground states.  The mean and median discrepancies between the official (Bush % – Kerry %) vote margin and the exit poll prediction of the (Bush % – Kerry %) vote margin were 3.7% and 3.9% in Kerry’s favor, respectively.  Moreover, in four key battleground states eventually won officially by President Bush—Ohio, New Mexico, Nevada, and Iowa—the exit polls on CNN’s website up until the wee Wednesday early morning hours after Election Day had Senator Kerry ahead of President Bush.]

I’ll likely have much more to write about the Edison/Mitofsky report later.  Their main claim is that the overestimate of Kerry support was predominantly due to Kerry supporters being more willing than Bush supporters to fill out exit poll questionaires, thus skewing the sample.   The overall design of the poll was judged to be sound.  Most pertinently for our post, the precincts that were chosen to be exit polled were won by Bush and Kerry in proportions that statistically were not significantly different from the official tallies.  Thus, the focus of the Edison/Mitofsky report is on "within precinct error (WPE)" and not on error from choice of precinct to be exit polled.

And now here’s the key point of this post.  The key evidence Edison/Mitofsky offer in support of this "differential nonresponse" hypothesis is the following from pages 36 and 37 of the PDF file:

2. Precinct Partisanship:

When the precincts were grouped based on their vote (high Kerry through high Bush), the high Bush precincts have the greatest statistical bias. The average signed WPE increases sharply with the increase in the Bush vote.  A small Bush overstatement exists in the highest Kerry precincts. The analysis is more meaningful if the precincts where Kerry and Bush received more than 80% of the vote are ignored. In the highest Kerry precincts there is little room for overstatement of his vote. Similarly the highest Bush precincts have more freedom to only overstate Kerry rather than Bush. The three middle groups of precincts show a relatively consistent overstatement of Kerry.

Partisanship in Exit Polled Precincts versus Within Precinct Error (WPE) of Exit Poll

[NB: WPE is defined as the Official Bush-Kerry Vote Margin minus the Exit Poll Bush-Kerry Vote Margin.  Positive WPEs mean Kerry did better in official tally than exit poll.  Negative WPEs mean Kerry did worse in official tally than in exit poll.]

Precinct Partisanship Mean WPE Median WPE Mean Absolute Value of WPE Number of Precincts

Highly Democratic

(Official Kerry Vote > 80%)

0.3% -0.4% 8.8% 90

Moderately Democratic

(Official Kerry Vote 60-80%)

-5.9% -5.5% 13.4% 165

Even

(Official Kerry Vote 40-60%)

-8.5% -8.3% 15.2% 540

Moderately Republican

(Official Kerry Vote 20-40%)

-6.1% -6.1% 13.2% 415

Highly Republican

(Official Kerry Vote < 20%)

-10.0% -5.8% 12.4% 40

There was no significant difference between the completion rates and the precinct partisanship:

Partisanship in Exit Polled Precincts versus Completion, Refusal, and Miss Rates of Exit Poll

Precinct Partisanship Completion Rate Refusal Rate Miss Rate

Highly Democratic

(Official Kerry Vote > 80%)

53% 35% 12%

Moderately Democratic

(Official Kerry Vote 60-80%)

55% 33% 12%

Even

(Official Kerry Vote 40-60%)

52% 37% 11%

Moderately Republican

(Official Kerry Vote 20-40%)

55% 35% 10%

Highly Republican

(Official Kerry Vote < 20%)

56% 33% 11%

I have to say that, at first glance, this seems like pretty convincing evidence of a differential nonresponse by party.

UPDATE (Jan 19, 2005 — 10 PM):   But on second thought, I see I made a logical error.   When I first glanced at the data and wrote the post above, I believed the data was "pretty convincing evidence of a differential nonresponse by party" because, as the report emphasized, the precincts with the highest victory margins for Bush had the highest WPEs and those with the highest victory margins for Kerry had the lowest WPEs.

But then I remembered that the fundamental and infuriating aspect of the survey nonresponse problem is that if some group of people refuse to take part in your exit poll, you cannot have any definitive knowledge about them beyond that available at first glance (gender, approximate age, probable ethnicity, and that they–by walking out of the polling place—in all likelihood just voted). Bush versus Kerry voting preference, alas, isn’t one of those things.  [Sidenote: I see nothing in the report—and I must admit I’ve still more skimmed it than read it—that such first glance data on refusers was recorded, let alone analyzed.]

Thus, I now see that the presence of such a trend in the data in and of itself cannot offer any actual corroboration for the hypothesis that differential nonresponse by party caused most of the exit poll discrepancies, though seeing such a trend may suggest a mechanism for the differential response phenomenon, assuming the hypothesis is true.   For example, the trend might suggest that the more Republican that the environs of the Kerry-voter were, the more pissed off he or she was and the more enthusiastic to tell an exit pollster of his or her vote for Kerry. (Though on that note, realize that if there were a constant difference in the nonresponse rate of Bush voters versus the nonresponse rate of Kerry voters across all precincts, it’d be the even precincts and not the strongly Republican ones that should have the greatest WPE due to differential nonresponse.)

In fact, this data cannot do anything more than offer a way to approximate quantitatively the size of the differential nonresponse effect under the assumption that it indeed accounts essentially for the entirety of the exit poll discrepancies.   Again, the fundamental and infuriating aspect of the survey nonresponse problem is that if some group of people refuse to take part in your exit poll, you cannot have any definitive knowledge about them beyond that available at a glance, and Bush versus Kerry voting preference isn’t one of those things.   

So what is the size of the mean differential nonresponse under the assumption that constitutes the entirety of the exit poll discrepancies?   The Edison/Mitofsky report does not provide an explicit estimate.  However, the data it provides that is quoted above suffices to answer this question under the mild additional assumption that third party voting was negligible.   The simple algebra is as follows:

Let K = Kerry’s vote percentage.  Thus, 1-K is assumed to be Bush’s vote percentage.  Let c = the mean completion rate for all voters, and W = the mean within precinct error.  We seek to solve for b = the completion rate for Bush voters and k = the completion rate for Kerry voters.   Thus, we solve this pair of equations

    c  = b(1-K) + kK

    W = [b(1-K) – kK]/c – (1 – 2K)

[The first equation says the mean total completion rate c has to be the average of the mean Bush and Kerry completion rates, b and k, weighted by the respective Bush and Kerry votes, 1-K and K.   The second equation says the mean within precinct error W is the exit poll (Kerry-Bush) vote margin [b(1-K) – kK]/c minus the official (Kerry-Bush) vote margin (1 – 2K).]

which after some simple algebra yields

      b = c + cW / [2(1-K)]

      k = c – cW/(2K)

Thus, using the above quoted data from the Edison/Mitofsky Report and taking K to be the midpoint of the Kerry vote intervals defining the precinct partisanship categories, I thus produce the following estimates:

Estimated Response Rates versus Precinct Partisanship

Precinct Partisanship Mean WPE (W) Measured Total Response Rate (C) Estimated Kerry Response Rate (k) Estimated Bush Response Rate (b)

Highly Democratic

(Assumes K = 90%)

0.3% 53% 53% 54%

Moderately Democratic

(Assumes K = 70%)

-5.9% 55% 57% 50%

Even

(Assumes K = 50%)

-8.5% 52% 56% 48%

Moderately Republican

(Assumes K = 30%)

-6.1% 55% 61% 53%

Highly Republican

(Assumes K = 10%)

-10.0% 56% 84% 53%

The differential response in the moderately Democratic, even, and moderately Republican precincts is 7 or 8 percentage points more for Kerry voters than Bush voters, which prima facie does not seem very large at all, and thus for these precincts the hypothesis of differential response seems like a pretty plausible one to account for the entirety of the exit poll discrepancies. (Honestly, though, what actual data does anyone really have to assess whether a differential response rate is in fact unreasonably large or believeably small?  Again, this data can’t prove the hypothesis beyond reasonable doubt.  It just doesn’t add any further reasonable doubts for this vast majority of the precincts.)  On the other hand, the differential response in highly Republican precincts does seem, prima facie, to be implausibly huge: 31 percentage points if we assume K = 10% or 16 percentage points if we make the most conservative possible assumption that K = 20% (the very edge of the Edison/Mitofsky definition of "highly Republican").   Thus, for these precincts, the hypothesis of differential response seems, prima facie, pretty dubious as a complete explanation of the exit poll discrepancies.

I don’t know what to say about this, but as it’s getting late, whatever I shall say will wait for another day.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: