Been contemplating a post on polls and statistics for a few days now. The ebb and flow of the polls has me a bit perplexed. The national pattern is not synchronized with the state level polls.  Many key states are either in the “toss up” or very weak “lean” categories. normally blue states (e.g., PA, MN, WI) are now toss ups, as are normally red states (e.g., VA, FL) (reference RCP map). What we see really is about even on the solid Rep/Dem with the rest of the country able to slide from “lean” to “toss up” on the whim of the next state poll (reference the RCP historical churn of the battleground states). Personally, I would widen my “lean” criteria a bit more than RCP is doing right now. 

National polls should be showing a dead heat (and were for a few days) in a situation like this, but we instead see a small shift to Obama (in some, not all polls). So that doesn’t fit the ‘traditional model’ we have seen for decades. And maybe that is part of the problem. In a 1000 person poll sample around 20 people represent each state. In fact it is less than that for many surveys, but the point is clear. If your models don’t extrapolate from 20 to 1000 to a nation of 300 million right, the results are garbage. Being off by just a small percentage can mean being way off in reality.

And for once I am not sure what value RCP is bringing to the poll watching. There are no error bars (or MoE) around their numbers – which makes them sort of meaningless. And they let a lot of polls in which probably are not accurate. Take Colorado for example.  Today’s CO polls show this: Obama = +4, +10, +1; McCain is +2.  The RCP average is Obama +3.3. But the +10 poll is clearly anoutliar (just looking at the other historical polls). Remove the outlier and the tally is +4 +1 -2 = +3/3 = Obama 1+. That is where CO is and we all know it. The error bars on the first average is fairly large, on the second one I computed much smaller.

The point is the polling models that have beeb quasi reliable for years are just not set up for this kind of environment. No incumbent with the party in office holding high policy negatives (but high personal positives). Countering that is the opposition party holding Congress and even more negatives, along with a weak candidate riding a PR images instead of substance, who took months to admit he was wrong about Iraq. Add in the Palin factor and I would say the poll models are crap.

It get’s even more bizarre if you look at poll internals. I noted one poll showing McCain-Palin moving 36% points with women! That is 1/3rd of all women moving to support McCain-Palin, on top of the support McCain had before. Most of that came from the ‘undecided’ column, but not all. What is interesting about this poll is it doesn’t try to model the support into a state or national level, where these internals get watered down based on the model weighting a combination of factors. The women’s vote will be attenuated by age, income, party affiliation, etc. The problem is there is no way for 1/3 of the women’s vote to shift allegiences and Obama still be marginally ahead. This is why I think the polls may be spewing garbage, their internals and their bottom lines are incoherent.

There have been many polls showing a shift of women to McCain-Palin (see here, here and here).  There are also polls showing a big shift with independents (see here and here) and polls showing McCain taking a sizeable chunk of Democrats (see here). The problem is these movements are not consistent across all polls. There are other internals that can be used to negate these shifts. The classic one is to overload the sample with Democrats to neutralize the GOP and Independent strengths.

This race is all tied up and I would say, even if the pollster claims an MoE of +/- 2% the error bars are really closer to +/- 5%.  That makes sense if you look at the swings in polls in any state. Let me pick on my the state I am watching the closest right now, PA. Here is the RCP list of polls: Obama = +5, tie, +3, +2 for an average of +2.5.  This race is a tie right now.

But which way is the trend going? Well, at the end of July Obama had a +9 lead, so it is trending (with small ups and downs) towards McCain. This is validated by another poll (not in the RCP mix) which today shows a tie, but the reporting also shows the historic trend of a 10+ point Obama lead shrinking to zero over time. The trend selects many of the polls in the RCP average to show the path PA is on. With about a month and a half to go, unless this trend is reversed, McCain could be up significantly by election day.

Another PA poll out shows the same stastical tie (here, with Obama +2), but the internals show something, again, which would make one question whether this is a tie (here):

Support by party affiliation, among Democrats: Obama 72, McCain 17
Support by party affiliation, among Republicans: McCain 80, Obama 11

McCain is taking 17% of Democrats from Obama, who supposedly is grabbing 10% of the GOP. I don’t have the time to research it, but I am certain that no Democrat has come in a close second with more than single digit defections. Again, the internals seem to defy the bottom line (and note this level of defection has been seen in other polls as noted above).

I guess what I am getting to here is the idea that the national polls are completely hosed because the state level polls are in such a state of flux. You can go to the RCP state level polls and watch in each case Obama lose a commanding lead over the past few weeks. All of the battleground polls show the same trend – double digit leads shrunk to statistical ties. Yet we see national polls with a slight Obama lead?

We learned during the primaries how bad the polls were off. It took a few months to correct the models and get the numbers close to right. For the general there are not opportunities to fix the defects.  And statisticians are loathed to admit there cherished (and proprietary) voter models are screwed up.  Without performing a massive rebaselining survey to remap the internals, there is no way to find the defects, let alone take the surgical corrective actions to shift weights by tenths of a percent across various dimensions which can turn a 10 point lead into a 5 point loss in one step.

Obama may actually be building a lead, but to me the internal conflicts tell me all these polls are spewing garbage right now. One thing is for sure, if McCain does well in the debates I think we will see another shift in support (I don’t think Obama doing well will have the same impact given he is ‘in the lead’ now). At some point all these undecideds are going to start moving in one direction or another. And we may not understand it happened until we get the vote tallies back.

Update: Another indicator that can be trusted is crowd size. McCain-Palin have them in spades (no rock bands required to lure the crowds). ABC news is reporting on something I noticed last night in deep blue Wisconsin – Obama could barely fill 2/3rds of a venue which McCain-Palin had filled to capacity (with some wanting to get in) just the week before. That tells me something, Obama is behind by quite a bit in WI right now.

Tags: 2008 Election, Polls