Sep 23 2008

Those Wild Polls

Published by at 9:05 am under 2008 Elections,All General Discussions

Been contemplating a post on polls and statistics for a few days now. The ebb and flow of the polls has me a bit perplexed. The national pattern is not synchronized with the state level polls.  Many key states are either in the “toss up” or very weak “lean” categories. normally blue states (e.g., PA, MN, WI) are now toss ups, as are normally red states (e.g., VA, FL) (reference RCP map). What we see really is about even on the solid Rep/Dem with the rest of the country able to slide from “lean” to “toss up” on the whim of the next state poll (reference the RCP historical churn of the battleground states). Personally, I would widen my “lean” criteria a bit more than RCP is doing right now. 

National polls should be showing a dead heat (and were for a few days) in a situation like this, but we instead see a small shift to Obama (in some, not all polls). So that doesn’t fit the ‘traditional model’ we have seen for decades. And maybe that is part of the problem. In a 1000 person poll sample around 20 people represent each state. In fact it is less than that for many surveys, but the point is clear. If your models don’t extrapolate from 20 to 1000 to a nation of 300 million right, the results are garbage. Being off by just a small percentage can mean being way off in reality.

And for once I am not sure what value RCP is bringing to the poll watching. There are no error bars (or MoE) around their numbers – which makes them sort of meaningless. And they let a lot of polls in which probably are not accurate. Take Colorado for example.  Today’s CO polls show this: Obama = +4, +10, +1; McCain is +2.  The RCP average is Obama +3.3. But the +10 poll is clearly anoutliar (just looking at the other historical polls). Remove the outlier and the tally is +4 +1 -2 = +3/3 = Obama 1+. That is where CO is and we all know it. The error bars on the first average is fairly large, on the second one I computed much smaller.

The point is the polling models that have beeb quasi reliable for years are just not set up for this kind of environment. No incumbent with the party in office holding high policy negatives (but high personal positives). Countering that is the opposition party holding Congress and even more negatives, along with a weak candidate riding a PR images instead of substance, who took months to admit he was wrong about Iraq. Add in the Palin factor and I would say the poll models are crap.

It get’s even more bizarre if you look at poll internals. I noted one poll showing McCain-Palin moving 36% points with women! That is 1/3rd of all women moving to support McCain-Palin, on top of the support McCain had before. Most of that came from the ‘undecided’ column, but not all. What is interesting about this poll is it doesn’t try to model the support into a state or national level, where these internals get watered down based on the model weighting a combination of factors. The women’s vote will be attenuated by age, income, party affiliation, etc. The problem is there is no way for 1/3 of the women’s vote to shift allegiences and Obama still be marginally ahead. This is why I think the polls may be spewing garbage, their internals and their bottom lines are incoherent.

There have been many polls showing a shift of women to McCain-Palin (see here, here and here).  There are also polls showing a big shift with independents (see here and here) and polls showing McCain taking a sizeable chunk of Democrats (see here). The problem is these movements are not consistent across all polls. There are other internals that can be used to negate these shifts. The classic one is to overload the sample with Democrats to neutralize the GOP and Independent strengths.

This race is all tied up and I would say, even if the pollster claims an MoE of +/- 2% the error bars are really closer to +/- 5%.  That makes sense if you look at the swings in polls in any state. Let me pick on my the state I am watching the closest right now, PA. Here is the RCP list of polls: Obama = +5, tie, +3, +2 for an average of +2.5.  This race is a tie right now.

But which way is the trend going? Well, at the end of July Obama had a +9 lead, so it is trending (with small ups and downs) towards McCain. This is validated by another poll (not in the RCP mix) which today shows a tie, but the reporting also shows the historic trend of a 10+ point Obama lead shrinking to zero over time. The trend selects many of the polls in the RCP average to show the path PA is on. With about a month and a half to go, unless this trend is reversed, McCain could be up significantly by election day.

Another PA poll out shows the same stastical tie (here, with Obama +2), but the internals show something, again, which would make one question whether this is a tie (here):

Support by party affiliation, among Democrats: Obama 72, McCain 17
Support by party affiliation, among Republicans: McCain 80, Obama 11

McCain is taking 17% of Democrats from Obama, who supposedly is grabbing 10% of the GOP. I don’t have the time to research it, but I am certain that no Democrat has come in a close second with more than single digit defections. Again, the internals seem to defy the bottom line (and note this level of defection has been seen in other polls as noted above).

I guess what I am getting to here is the idea that the national polls are completely hosed because the state level polls are in such a state of flux. You can go to the RCP state level polls and watch in each case Obama lose a commanding lead over the past few weeks. All of the battleground polls show the same trend – double digit leads shrunk to statistical ties. Yet we see national polls with a slight Obama lead?

We learned during the primaries how bad the polls were off. It took a few months to correct the models and get the numbers close to right. For the general there are not opportunities to fix the defects.  And statisticians are loathed to admit there cherished (and proprietary) voter models are screwed up.  Without performing a massive rebaselining survey to remap the internals, there is no way to find the defects, let alone take the surgical corrective actions to shift weights by tenths of a percent across various dimensions which can turn a 10 point lead into a 5 point loss in one step.

Obama may actually be building a lead, but to me the internal conflicts tell me all these polls are spewing garbage right now. One thing is for sure, if McCain does well in the debates I think we will see another shift in support (I don’t think Obama doing well will have the same impact given he is ‘in the lead’ now). At some point all these undecideds are going to start moving in one direction or another. And we may not understand it happened until we get the vote tallies back.

Update: Another indicator that can be trusted is crowd size. McCain-Palin have them in spades (no rock bands required to lure the crowds). ABC news is reporting on something I noticed last night in deep blue Wisconsin – Obama could barely fill 2/3rds of a venue which McCain-Palin had filled to capacity (with some wanting to get in) just the week before. That tells me something, Obama is behind by quite a bit in WI right now.

12 responses so far

12 Responses to “Those Wild Polls”

  1. dave m says:

    With 350,000 votes so far this week,

    http://news.aol.com/political-machine/straw-poll

    shows McCain 63 – Obama 37 nationally
    We have lost 2 states, Vermont and DC, though even Dick Morris
    would probably admit that if these numbers are born out, McCain will
    win.

    Of course, it’s an Internet poll, so it deserves no credibility
    whatsoever.

    Remind me again why the mainstream news pollsters deserve any
    credibility. I seem to have forgotten.

  2. perdogg says:

    The McCain camp is openning offices in CA because internals show McCain within 5 points of Obama in California.

  3. AJStrata says:

    perdogg,

    you got a link?

    AJStrata

  4. […] to add fuel to the fire I wrote about this morning, there is a new poll out showing Obama is not gaining support among Clinton supporters – yet […]

  5. perdogg says:

    But Republicans aren’t giving up on California. California Republican Party chief operating officer Bill Christiansen said some private polls put the two candidates within five points of each other, and the McCain campaign is staffing 50 offices throughout the state. Four years ago, the Bush-Cheney campaign barely had a presence in California, and spent little money. “But we are fully funded this time,” Christiansen said, declining to give a figure.

    http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2008/09/21/MNPP130KA7.DTL&feed=rss.news

  6. AJStrata says:

    thanks Perdogg,

    I missed the paragraph the first time I read the article

  7. cochino says:

    AJ,
    I agree that the numbers don’t add up. McCain has been gaining or holding even in virtually every demographic; Obama has been holding even or dropping in virtually every demographic. Yet, this is not reflected in the national numbers at all. If anything, they show Obama widening his margin this past week. It’s hard to make sense of it. The state polls are also crazy. Does anyone really believe Obama would win Colorado by 10 points if the election were held today? I mean, why even release a poll like that?

    The problem is that so much in political polling depends on party affiliation. Its the bedrock of polling. Because Democrats and Republicans have different likelihoods of responding, they need to weight results. They have to try to do this, or the polls would really be worthless. They try to gauge general support for the parties regularly to try to “keep up” with changes in affiliation. That’s how Obama can be dropping in every group, yet gain in the overall national or state numbers. The weighting used (i.e. proportion Dems vs. Repubs) is a variable in the mix. A drop in Obama’s numbers in every demographic in a given week could be balanced by an increase in the estimated proportion of Dems that same week.

  8. jwb says:

    AJ…any statistics about Americans becoming tired of poll upon poll upon…? Paln has commented about us being tired of politics as usual and we know how dependent the Clinton administration was upon government by poll.

    It seems that the main topic of most news casts is always the results of the latest poll. The MSM has brainwashed this country such that too many people think our government is a democracy rather than a republic.

    I have been called by telephone pollsters a couple times in the past month and I just plain out lie to the pollster! In previous elections, I used to just hang up. I’ve heard a number of people who lie like me…is there a movement?

  9. AJStrata says:

    JWB,

    Pollsters dirty little secret is it is getting harder and harder to sample. Most people hang up (I do).

    Yes, there is a backlash and a credibility problem with all these amateur polls out there. That is why you have to deal with them as rough data points in an image and not factual or concrete.

  10. MerlinOS2 says:

    AJ

    RCP is having their averages skewed by the worthless CBS/NYT poll the included in the mix.

    Look at the internals and you see they load dem/rep up 4 to 3.

    Out of the same poll they put the Congressional Generic number on the board at +20 dem when every body else is polling in the single digits

    Add to that the Presidential approval numbers hit 27/68 with a 40+ spread and an outlier compared to all others polling.

    Somebody loaded the poll to put it into the mix just to keep the average tilted toward the Obama camp.

    It should have been thrown out due to the methodology and the bias in the poll.

    Add to that Kos has added a daily poll from an organization with historic dem poll bias and they are all there just to slide the averages a bit to the left.

  11. perdogg says:

    Did you all notice something strange about the Survey USA poll for VA that had Obama by 5? 7% of the respondants are not placed as either Dem, GOP, or Indy.

  12. perdogg says:

    Gallup is down to 3. 9-23-08