How to interpret tracking polls - Cover Story

Each election season, pollsters conduct "tracking polls" to measure public opinion over time. Some seek to provide strategic information to political campaigns, while others aim to supply interesting facts to journalists, pundits and scholars. Whatever the objectives, producers and consumers of tracking poll data have an interest in maximizing the accuracy of the results.

The limiting factor, however, is money. Surveys are expensive. Much as one might like to obtain precise results by gathering a large number of interviews, the bigger the sample, the more daunting the price tag. Given this trade-off between cost and accuracy, most efforts to track opinion over time rely on surveys of between 200 and 1,000 respondents.

That's why you should be careful when analyzing day-to-day tracks yourself or when listening to such analyses from so-called experts. A well-conducted survey of 1,000 respondents is nothing to sneeze at, but it's not infallible either. Suppose we were tracking the percentage of the public that holds a favorable impression of the president, and suppose that unbeknownst to us, the true percentage is 55 percent.

Even under the best conditions the sampling error associated with our survey's assessment of the president's favorability rating is 1.6 percent, which means that under the usual assumptions, 95 percent of the surveys we might undertake will produce favorability ratings between 52 percent and 58 percent. So if one day the president's approval rating stands at 57 percent, and the next it falls to 53 percent, it is entirely possible that the apparent change in opinion is due to random sampling variability. Different surveys conducted in an identical manner may produce somewhat different results.

All this is common knowledge among pollsters. Yet the implications of sampling variability for the interpretation of polls are routinely ignored.

Consider, for example, the election year commentary of many political pundits and reporters. When media analysts see a candidate's lead in three successive tracking polls grow from 2 percent to 3 percent to 9 percent, they generally report that the current lead stands at 9 percent. That's a tempting story to tell, but it's unlikely to be true.

The reason is that even a survey of 1,000 respondents is too small to trump the polls that came before. Just a week earlier, journalistic reasoning put the lead at 3 percent. One suspects that the lead has grown since then, but some of the 6 percentage-point surge is potentially attributable to sampling error. The question is: How much of the observed change should we ascribe to a real shift in public opinion and how much to a quirky sample? We need a method for solving what we shall dub the "Pundit Problem," which arises whenever one seeks to formulate a best guess about current political conditions based on a series of fallible poll readings. Our purpose here is to offer a solution.

Extracting Signal From Noise in Survey Data

It turns out that this class of problems comes up a lot in engineering. Planetary probes are sent off into space and periodically transmit information about their location. Because the instruments that gauge location are imperfect, controllers back on earth have to figure out the probe's location after discounting instrument error. Substitute tracking polls for signals and public opinion for location, and you have the Pundit Problem.

Engineers solve this problem by using a mathematical device called filtering. We won't go into the gory details of how this statistical technique works here. The important thing is that filtering dramatically improves the accuracy of polls as forecasting tools.

To filter tracking polls, we need to know two things: 1) the sampling error of each poll and 2) the rate at which public opinion might be expected to change over time. The former is easy to calculate given the results of each poll and the number of respondents in each random sample. A bit more difficult is the task of estimating how much public opinion typically moves from one period to the next. This quantity could be estimated if one had 20 or more tracking polls, but under ordinary circumstances, it requires an educated guess.

A rule of thumb we've found useful is that the typical senatorial or gubernatorial race shifts by about 2 percentage points per month. Particularly volatile races might be in the order of 5 percentage points. These rough-and-ready numbers may seem like guesswork, but bear in mind one important mathematical fact: The standard pundit's approach to poll-watching implicitly assumes that the month-to-month movement in public opinion is infinite. There's guesswork, and then there's guesswork.

With estimates of the sampling error and rate of opinion change in hand, we can massage a set of tracking poll results and come up with much more accurate assessments of where public opinion stands. Often these assessments differ dramatically from the conventional read-just-the-last-poll approach.

Don't Become Apoplectic

Suppose that down the home stretch of an election campaign, one is hired to gather 120 interviews per night for seven consecutive nights. With such small samples, the results are all over the place. In a trial heat, candidate Smith's vote percentages are 48, 54, 51, 61, 57, 59 and 50. Naturally, when the campaign staff sees that last poll come in, they become apoplectic. What media miscue, gaffe or maneuver on the other side could be causing Smith's rating to plummet? Filtering the data, however, casts the situation in proper perspective (see table), even if the race were extremely volatile.

The filter shines when applied to sawtooth patterns in public opinion like this, but it also brings into proper perspective more mundane patterns as well. To return to our earlier example, suppose we saw candidate Smith's lead grow from 2 percent to 3 percent to 9 percent. In other words, we conducted three successive polls of 1,000 respondents apiece and saw Smith's percentages climb from 51 percent to 51.5 percent to 54.5 percent. Again, assume that shifts in public opinion tend to be on the order of 5 percentage points.

The optimal guess of where public opinion stands in week three is 53.67 percent, which puts Smith's lead at 7.3 percent, not 9 percent. A bit of additional statistical analysis shows that there is only about a 1-in-10 chance that the lead is actually as large as 9 percent.

One final advantage of filtering is that it is easy to build background knowledge into the estimation process. Most political campaigns with big media buys and prominent candidates are expected to be close even before the first poll results come in. Conventional poll-reading ignores background information. Filtering does not.

Filtering and the Bottom Line

Filtering offers a way to get much more bang from one's survey buck. In the preceding example in which the observed lead went from 2 percent to 3 percent to 9 percent, the sampling error of the third poll was 1.6 percent, whereas the uncertainty of the third filtered estimate was 1.35 percent. Think of it this way: If you filter this sequence of tracking polls, the gain in precision is tantamount to adding 366 respondents to the last survey. Even the leanest survey outfits have to spend thousands of dollars to interview this many additional respondents. Not a bad trick one can perform in a few seconds on a computer.

Pollsters may prefer to think of filtering in terms of how they can reduce their sample sizes while preserving the same degree of efficiency. Our experience with both actual survey data and computer simulations suggests that filtering allows pollsters to reduce the sample size of their tracking polls by approximately 20-30 percent. Again, in the case of the three polls that march from 2 percent to 3 percent to 9 percent, it turns out that three successive surveys of 1,000 respondents apiece analyzed without filtering are equivalent to three surveys of 700 respondents apiece analyzed with the filtering technique. Both provide the same quality of information in week three. These are not trivial savings in terms of time and money.

A Word About Moving Averages

The technique that we used to derive our estimates works like a weighted average of current and past polls. The most recent polls get the most weight, and older polls get less. The exact weighting scheme, however, depends on sample size, the volatility of public opinion, and the length of time that passes between polls. This weighting algorithm is almost invariably different from the naive moving averages sometimes applied to tracking poll data. It is not uncommon for survey houses to report an average of polls in which each poll is assigned an equal weight.

           Survey         Filtered
Day        Result         Estimate

1           48               48
2           54               51
3           51               51
4           61               55
5           57               56
6           59               57
7           50               54

But equal weighting, by placing too much weight on older polls, makes the read-the-last-poll mistake in reverse. It assumes that public opinion never actually changes during the course of the campaign (if that's the case, why bother polling?). The advantage of filtering is that it creates a weighting scheme customized for the data at hand.

Filtering vs. Read-the-Last-Poll Approach

So much for armchair applications. What happens when this technique is applied to actual survey data? Better yet, what happens when this method is used to forecast election outcomes? How does filtering compare with the read-the-last-poll approach, which bases its projections solely on the most recent survey, and the naive moving average, which lumps together the last few surveys prior to an election?

To test the accuracy of these approaches, we went back to old issues of the Cook Political Report (Oct. 6 and Oct. 26) and culled all of the available tracking polls for 1996, with an eye toward predicting the vote totals reported in congressional and gubernatorial elections of 1996. We selected polls conducted during at least three points during the campaign by the same survey organization using the same survey population (e.g., likely voters). Nineteen campaigns, most of them quite competitive, met these criteria.

We compared the three kinds of forecasts to the actual percentage of the major party vote won by the Democratic candidate. Of the three forecasting approaches, the filter performed best. It gave the closest forecast in 11 of 19 races, tying with the read-the-last-poll method in two others. The read-the-last-poll approach won four outright, and a simple moving average pulled up the rear with two victories. The average mispredictions were 5 percentage points for the moving average method, 4.3 for read-the-last-poll and 3.7 for filtering. In short, filtering is the best way to interpret tracking polls.

But If You're Still Not Convinced...

"This is all very well and good," replies the skeptic, "but if the final results aren't dramatically different from what I would have come up with had I simply relied on the last poll, it's just not worth the effort." Perhaps, but it should be noted that the examples given above tend to understate the value of filtering. The reason is that to this point we have considered only samples analyzed as a whole. In actual practice, survey analysts and campaign managers track subgroups in the population - women, young voters, Reagan Democrats, Latinos and so forth. Once one begins to chop up a sample in this way, the number of observations begins to dwindle. Conventional methods of interpreting poll results become riskier. Filtering, on the other hand, continues to do what it can to make a silk purse out of a sow's ear.

Suppose, for example, that one seeks to track the opinions of blacks during a volatile campaign, but one has only 150 black respondents in each of four surveys. Support for candidate Smith among blacks comes in at 48 percent, 38 percent, 57 percent and 55 percent. Where does candidate Smith actually stand in week four? Probably not 55 percent. The chances that the lead is as high as 55 percent are only about 1 in 10. The best guess is 52 percent.

But what about the fact that candidate Smith's popularity seems to be surging over the last two weeks? There's a bit of a surge, but less than meets the eye. Once we have the fourth poll in hand, we can look back on the earlier polls and revise our estimates of where public opinion stood in that week. That kind of retrospective filtering is called smoothing. This technique suggests that the best guesses of black opinion for weeks one through four are 48 percent, 48 percent, 51 percent and 52 percent. Even if you believed that support were indeed surging a bit, a forecast of 55 percent for the following week would still be unwarranted.

Just Do It

Try filtering some of your own polling data. We've assembled a demonstration website at Yale where users may input their poll numbers and extract filtered estimates (http://statlab.stat.yale.edu/~gogreen/kfform.html). Our filtering program (patent pending, all rights reserved) is easy to use and potentially very useful to pollsters gearing up for this year's election season.

Filtering is easy to implement. Although filtering is unknown to those in the survey world, it should be dusted off and put into practice. After all, it appeals to the two noble aspirations of every survey organization: producing accurate forecasts and saving money.

Donald P. Green is Professor of political science and Director of the Institution for Social and Policy Studies at Yale, where he teaches American politics and statistics.

Alan Gerber is Assistant Professor of political science at Yale where he teaches American politics and political economy. During 1996, he worked with the Mellman Group on several congressional races.

COPYRIGHT 1998 Campaigns & Elections, Inc.
COPYRIGHT 2000 Gale Group