Yes Virginia, pollsters really are wrong

The United States presidential election of 2016 is scheduled for Tuesday, November 8. In the meantime, we know that in almost every race for 36 years, the eventual nominees have won either Iowa or New Hampshire.

While Hillary Clinton won by a razor slim margin in Iowa, Bernie Sanders took home New Hampshire by a wide margin. With the Republicans, it seems increasingly likely that Trump could win the nomination.

This blog entry is part of our series on business analytics and big data

CLICK - Outside chance - the primary contest is about to get serious.

Mr Sanders promises free education in public universities. He wants to have government rather than private insurers to pay health care bills.

This could cost US $14 trillion over a decade, and would result in new taxes, costing most workers 8.4 percent of their income.

Workers might approve of such changes. But Mr Sander’s plans would have no chance of making it past Congress, even in one with a majority of Democrats.

Nevertheless, he won 60 percent of the vote in New Hampshire.

This represented one of the biggest victories in a contested Democratic primary.

Of course, polls play an important roll during any election, including US Presidential (see chart). But can we trust these polling data?

As the above graphic shows, polls gave Governor Kasich less than 5 percent. Nonetheless, he raked in 16 percent of votes in the New Hampshire primary of the Republican party. Trump hoovered up 35 percent of the vote as predicted by pollsters (see above graphic from The Economist, January 30, 2016, p.17).

The above illustrates that polling is a tough job, especially if one intends to get it right. In 2014, pollsters struggled with these challenges during the Scottish referendum and Swedish elections. Both times they got it wrong.

The same happened during the UK elections on May 7, 2015. The final polls showed Labour and the Conservatives neck-and-neck at 34 percent.

But when the final numbers where in, David Cameron’s Tories ended up 7 percent ahead of Ed Miliband’s Labour party. As pollsters have pointed out, however, they got the numbers right for the smaller parties. Too bad, their prediction was outright wrong for those fighting for the post of running the country, i.e. becoming Prime Minister…

So if you see a statement like the one below, will you believe it?

Polls indicate, Hillary Clinton is leading Bernie Sanders by 30 points in South Carolina

We discuss this below in more detail.

1. Getting the right sample size is very expensive

In early states such as Iowa and New Hampshire pollsters conducted more polls for 2016 than in 2012, when the last presidential race was happening. But the average sample size has fallen. According to the Financial Times, for New Hampshire the averages look as follows:

2012 – average pool of Republican voters interviewed was 590
2016 – average is 490

Of course, the margin of error of plus or minus 5 percent are common with such polls. Governor Kasich was predicted to get less than 5 percent. With a margin of error of 5 percent he could have gotten 10 percent but he actually got 16 percent.

In this case, the pollsters were vastly off, illustrating that sample size matters. In other words, does polling 490 Republicans justify the conclusions drawn by the authors?

National Statistical Service, Australian Bureau of Statistics – Sample size calculator

Polls are getting ever more costly thus end up with smaller samples
1. 2016 US Presidential Election

Pollsters now make 30-35 calls to complete a single interview.

Ten years ago, a pollster called ten people to get one to participate in the poll or study.

To interview 1,000 voters — only 400 of which may be likely to vote Republican — the pollster now has to dial up to 35,000 numbers.

2. UK polling

In the UK, today it takes about 15,000 phone calls to get 1,000 interviews.

About 20 years ago, 2,000 calls were needed to get 1,000 replies.

The above shows, depending upon the country, we may need to make anywhere from 10,000 to just about 40,000 calls to end up with a sample of 1,000 respondents.

But in the case of an election, we also need to make sure that those who answer are also those who will go cast their vote. This challenge is discussed below.

2. Selecting the wrong sample is a growing risk

Polls are conducted over two or three days, meaning we try to interview those who are more easily contacted. This may happen over the internet or via phone, but such work makes it hard to arrive at a sample surveyed that is representative of all voters.

National Statistical Service of Australia – calculate your sample size required for the relative standard error you desire

For instance, May 2015 British election data indicate that opinion polls failed to reach the harder to find Tory voters. In turn, estimates of the vote share attributed to Labor were skewed.

According to the British Social Attitudes survey, Labour was six points ahead among respondents who answered the door on the first visit. However, looking at those that required three to six home visits to be interviewed, the Tories enjoyed an 11-point advantage. Adjusting for social class and age, first time respondents are less conservative. “Busy” respondents are more likely to be so – but harder to chase for pollsters.

Hence, for pollsters it is not easy to get those “busy” people that might vote for a particular party as outlined below. Another challenge is that more and more people no longer have a landline. They can only be reached by mobile number (see also FT mentioning Martin Boon, director of ICM research). Because of this trend, in the US polls are increasingly conducted using mobile phone numbers to call.

3. Getting likely voters is a challenge for pollsters

Selecting those that are likely to vote for your political poll (see graphic below) is important. Finally, the views of voters and nonvoters are often very different, as was the case in 2014 in the US mid-term elections.

Pew Research (2016-01-07). Can likely voter models be improved? Section 2: Measuring the likelihood to vote

As well, for the 2015 UK elections, under-30s generally lean left, but very often fail to turn out on polling day. The pollsters, however, reached an atypical group of youngsters, who were unusually engaged with politics and committed to voting.

Another factor that can bias polls is that they are often based on internet polls. These tend to use volunteers to sign up to online panels. They may be drawn at random to participate. Nevertheless, the underlying group of people is self-selecting. Thus, data collected using a sampling method known as random digit dialing or “RDD” results in better data sets (i.e. random-probability samples representative of the population).

As this shows, getting those to participate in a poll that are likely to vote continues to be a challenge.

Sign up for our newsletter. Get the next post about big data, business analytics via email

Bottom line

Here are three challenges pollsters will continue to grapple with and we voters must keep in mind when studying poll results.

3 challenges for pollsters
1. Polls are expensive!

Ever fewer people want to participate in consumer surveys or election polls. Hence, it takes thousands of calls to get a useful sample. In turn, inferences can be made from such a sample about the election outcome.

But just the calls may take about 400 hours. In turn you get 1,000 replies and still have to pay for data analyses and a write-up. A costly exercise that is not getting cheaper.

2. Getting a representative data set is nearly impossible

As the 2015 UK election illustrated, it takes a greater effort to get Tory voters than others.

Cultural differences may also limit the applicability of Pew Research’s findings in the US whereby “…better-educated people tend to be more available and willing to do surveys than are those with less education.”

However, the current solution of using online non-probability survey panels makes polling results less accurate.

3. Identifying and polling likely voters is hard

It is tough to get accurate readings to predict election outcomes by using self-selected online panels.

YouGov knows this is a sensitive issue and does not publish any answers to research methodology questions on its blog. For instance, the voters over 70 who broke heavily for the Tories were not reflected in YouGov’s online panels used for predicting the 2015 UK elections.

Those under 30 years of age are less likely to cast their ballot. If your panel or poll includes those overeager millennials, who were unusually engaged with politics, your findings will be skewed.

Polling models will continue to be improved, but surprises will also continue to happen.

InterestingWhile Apps and Web respondents do not differ in their type of responses, the response rate is lower using mobile apps to collect data.

Nevertheless, polls are the best way of trying to figure out what the election result is going to be next time an election, primary or referendum comes around. So go and vote.

Join the 3,000+ organizations using the DrKPI Blog Benchmark to double reader comments in a few months while increasing social shares by 50 percent – register now!

Mr Sander’s foreign policy promise boils down to one thing: he will not start a foreign war. The rest is a replay of 1960s student radicalism. He intends to convert the US into a Scandinavian social democracy. Mr Trump would make America great again. He thinks the US has been screwed by its allies. If he needs expertise, since he has none, he will hire experts. In the meantime his solution is to “bomb the shit out of ISIS.” That is about it for his foreign policy.

Unfortunately, both Sanders and Trump are arch-fantasists with the ability to recruit voters. Both will spell extremism as far as US foreign policy is concerned. The system, dominated by the Democrats and Republicans, has always rejected the political extremes. The nation has benefited from a deep political stability, which has contributed greatly to its economic strength and global power. If America’s immunity to extremism is ending, the whole world will feel the consequences. Not a good prospect for anyone, is it?

This post: Yes Virginia, pollsters really are wrong

Join the conversation

  1. Do you have an example of a great poll / study?
  2. I refused participating in a poll / consumer survey last week. You?
  3. Do you think pollsters will get it right in the 2016 US elections?
  4. US Primaries: Clinton and Trump or Sanders and Trump? What do you think?

Of course, I will answer you in the comments. Guaranteed.

Urs E. Gattiker

Professor Urs E. Gattiker - DrKPI is corporate Europe's leading social media metrics expert (see his books). He continues to work with start-ups. Urs is CEO of CyTRAP Labs GmbH.

12 thoughts on “Yes Virginia, pollsters really are wrong

  • 19. February 2016 at 21:49
    Permalink

    It does not matter who will win, because all that’s left of democracy is a “feel-good” tool, making people believe that their vote counts and matters.

    Be it Sanders, Trump or any other guy, there is no promise they can make that hasn’t been made before and will not be broken. I for once feel that these circus like spectacles should be “R” rated , because children learn that lying, cheating, back stabbing, mud slinging and black mailing are the tools that get you into the White House.

    Let’s also face the fact that the people are generally being “dumbed down”, misinformed, lethargic and at least here, are born with their party DNA. Time to establish a “voter permit” to make sure only those people get to elect a leader who are well educated and informed.

    Let me close with saying that democracy is past its prime and on the way out. Democracy is a temporary occurrence.

    Cheers
    Hans

    PS.: Carson strikes me as the most honest and sincere of the entire bunch.

    Reply
    • 19. February 2016 at 21:52
      Permalink

      Yes Hans

      Democracy is a difficult thing to master and this little plant is having a tough time right now if we look at the US Primaries.
      Nevertheless, unless we change the way primaries are being done in the US, little if anything will change.

      Carson comes across as honest maybe if mot honest of the bunch of people I have watched. However, his program is a stillbirth because if I look at the costs (as outlined in my blog), Americans will not accept to pay on average 10 or more percent federal taxes to make it happen (tuition-free state universities).

      I don’t even want to think about how he is gonna get this accomplished since he needs to intervene in a matter (higher education) that is under a state’s jurisdiction (funding, rules, etc.). Will the states just sit still and take it / accept it?

      That bothers me all greatly that a person with no program and apparently no idea how he is gonna deliver on his ideas and promises can get so far in the primaries.

      What will the Primaries in South Carolina bring us this weekend.

      How do the Democratic Presidential primaries look like

      Reply
      • 20. February 2016 at 9:50
        Permalink

        PS.: Still amazing how a nouveau rich bully like Trump with nothing but 5 or 6 constantly repeated “statements” can make it that far. When I hear this nonsense like,

        the Chinese and Japanese Companies are taking our money from us

        I need an extra dose of Alka Seltzer.
        Someone needs to tell this “dude” that these are the American consumers giving them the money and corporations like Walmart, Apple, Nike and many others are selling out this formerly great nation. Especially funny is Walmart, playing the great patriotic card.

        The man has no substance, no manners and no social skill. Again, I am amazed.

        Reply
        • 20. February 2016 at 9:56
          Permalink

          It is always sad to see how apparently a large group of Republican voters do not seem to want to grasp the correlation between:

          – the cheaper the better,

          In turn, this demand from consumers does in part Besides the store’s interest to maximise profits, consumers demand for great value (lowest price for best quality) results in my store:

          – procurement of supplies from those manufacturers who can satisfy the demand for low price and quality the best.

          Ergo… China comes to mind. Same if we worry that much about work conditions … why do we still purchase an Apple iphone? Of course, we could purchase a Fairphone, but “I want an iPhone”.

          It is so much easier to ignore how our decisions affect things. Our demand for ever better product quality at the lowest price can sometimes mean working conditions are dismal at these factories.

          I love outsourcing because it puts all the risk on the supplier. Apple can even claim to have nothing to do with those working conditions at its suppliers, unless we as consumers say “no longer.”

          Bottom line
          The above in part explains, why Trump can get away with such a stupid one liner. His audience is not willing to think a bit.
          Sanders’ fans are not willing to think either. If they would, college students would have asked him who is supposedly paying for his ideas. Will graduates like to start work paying 10 to 15% more taxes than they do now? Probably not.

          Sanders is right to question the way tuition fees have gone up and the kind of scary debt load this causes for students. But his solution is not workable. For instance, a Republican Congress will never approve his financing ideas since it results in a bit jump in taxes for the average US worker.

          My fear, Hans is that things will not improve much during the USA Presidential Election later this year. Candidates will still skirt the real issues because voters do not want to hear about it… In turn, the election debates will continue with stupid one liners and promises that can often not be put into practice once the person is elected.
          This is a dangerous cocktail. If voters do not demand a discussion of the issues that matter, we are destined for voters being disenchanted shortly after the new President has moved into the White House. The President will be unable to deliver and voters will feel cheated… but in part they are to blame…

          Thanks for sharing Hans.
          Donald Trump - Hairdo undone... from the wind

          Reply
  • 21. February 2016 at 12:40
    Permalink

    People in most, if not all Western societies have been trained well and prefer to camp out in front of an Apple store to be “the first” rather than participating in constructive work for their countries.

    “That’s what politicians are there for”.

    Kennedy’s

    “do not ask what your country can do for you, but what you can do for your country”

    remains ignored for the most part.
    People do not understand their duties within a working democracy and as has been said

    “stupid and misinformed people are easier to govern”.

    Watching closely what is happening around the world, reading various magazines from many geographical regions and tuning into foreign TV I must say, that this is NOT an American, but a problem shared by all “developed” nations to various degrees.

    As I mentioned before, Democracy is in it’s final stages and a mere shadow of what it was intended to be.
    Have a great weekend
    Hans

    Reply
    • 21. February 2016 at 20:45
      Permalink

      Dear Hans

      People are looking for a change. Sanders is saying what many people are thinking. He seems to preach a good sermon.

      Sanders has been angry at the good old USA all his life. He seems to be the best vehicle for the rage of the young.

      Sanders is 74 and one of the oldest candidates ever to run. Nevertheless, he has the youngest supporters. In the Iowa caucuses, he won an estimate 84 % of the vote of those aged 18 to 29 years of age.

      Hillary Clinton got 14%, even though she came out ahead overall.

      It seems young voters are expressing dissatisfaction with a model of social betterment that has produced disappointing results.

      Greetings
      Urs

      Reply
  • Pingback: Meinungsumfrage|Poll |Statistik|Kampagne|Politik|repräsentativ |

  • 22. February 2016 at 15:35
    Permalink

    Urs,

    it appears that always the young people have their dreams of a rosy future world before they are being “absorbed” and reality sets in. There are protests and demonstrations against just about everything and demands are being made.

    But then comes the time when questions like “HOW will you accomplish this ” are being asked and no one has an answer except ” politicians have to figure that out”. Much of it appears like protests against bad weather.

    What I like about Sanders and Trump to some degree is simply the fact, that they sent wake up calls to the “establishment” and have shaken up the political system a bit.. That is what Americans in general do like.

    Will it bring about change ?
    Of course not. We shall see how voters will distribute their votes after more candidates dropped out. Bush finally came to his senses after wasting roughly 55 Mio on trying to rescue his sinking ship.

    Interesting though is, that Hillary is sitting on the biggest pile of money and that should say plenty about who is being seen favorably by special interest and lobbyists.

    And the show must go on.
    All the best
    Hans

    Reply
    • 22. February 2016 at 15:44
      Permalink

      Thanks Hans for stopping by again.
      Donald Trump, Hillary Clinton win big in Nevada; Jeb Bush exits
      I just looked at the results again.

      Yees Hillary Clinton won. Interesting is, however, that she spent much less on TV ads the last week than Sanders did.

      Something that surprises me a bit, because her stuff thought it would be a tight race. Far tighter than the results would suggest.

      Shows us how exact polls are, ever less so making everybody’s job that much tougher.

      In South Carolina, Trump won and can, therefore, secure all 50 delegates the state offer.
      In contrast to the Democratic primaries in Nevada where Sandars does get 15 while Clinton gets 19.

      If Clinton does win the presidential bid, I am curious if she will do better then Obama and definitely Bush… We need a US president that re-unites the bi-partisan system that is cracking in its seems (too much conflict, no progress).

      But I am not holding my breadth on this, too tough a job with too many obstacles against the White House resident.

      Thanks for sharing Hans.

      PS. Incidentally, the Swiss are struggling right now to get things right and not accept a referendum on Feb. 28, that would result in us treating immgrants unfairly: Erfolgreiches Campaigning: DurchsetzungsInitiative

      Reply
  • 22. February 2016 at 17:31
    Permalink

    Western nations are generally hanging on to past glory like the 70 year old movie star with lots of custom jewelry, plastic surgery and layers of makeup. In addition, the attempts of self motivation and refusal to face the inevitable.

    As for Switzerland, as the only last functioning bastion of Democracy, one should face the fact, that there is no room in politics for “feelings”. Politics are reality based and also here, Machiavelli had plenty to say about politics and morals.

    Interesting discussion, even though I am missing other voices and comments.

    Have a great week
    Hans

    Reply
  • Pingback: CAR T-Cell | #csCH16 | #helpilayda | Leukämie | CD19-CAR T cells

Leave a Reply

Your email address will not be published. Required fields are marked *