New Hampshire: Graveyard of Pollsters

Byline: Richard Morin and Claudia Deane

In this column: New Hampshire: Graveyard of PollstersCracking on ZogbyA Nation of OptimistsThe K-Factor in Iowa

"Now I know why they call this the Granite State," said candidate Bob Dole eight years ago after losing a Republican primary in the state for the second time. "It's so hard to crack."

Dole's assessment is as true for pollsters as for the candidates in New Hampshire, where bad methodology, bad timing or simply bad luck have produced some of the most memorable miscues in the annals of polling. Consider these flubs:

In 2000, the headline on an AP day-before-the-primary story was "Nearing the N.H. finish line; Polls declare GOP dead heat. . . . " John McCain then went on to beat George W. Bush by 18 percentage points.

The New Hampshire-based American Research Group's tracking poll ended up buried deepest in the snow bank: They had Bush winning by two the day before the primary, merely 20 points off the mark. On the Democratic side, the losing pollster at least got the winner right: The Quinnipiac poll predicted Gore would win by 17 percentage points, but he actually won by four.

It was the second debacle for ARG in as many New Hampshire Republican primaries. The day before the 1996 contest, ARG's Dick Bennett told the Union Leader, "It looks like Dole's going to win," based on the Kansan's seven point advantage in their tracking poll. He didn't, losing to Pat Buchanan by a single percentage point.

Exit pollsters aren't immune. In 1992, Voter Research and Surveys' exit poll showed George H.W. Bush beating Buchanan by a relatively narrow 6 percentage points, only to have Bush finish 16 points ahead on election night.

In 1988, it was the Gallup poll that fell victim. Gallup's final preelection survey had Bob Dole up by 8 percentage points. He ended up losing to Bush senior by 9. We "went on to call the rest of the primaries that season without a hitch . . . haven't had many questions about how we succeeded in so many of those subsequent races, however," former Gallup head Andrew Kohut, now with the Pew Research Center, reminisced the other day. "Moral of the story: on my tombstone it will say 'Here lies Andy Kohut -- got NH wrong in '88.'" [The Post had nothing to boast about here either, with our final product suggesting Dole up by three.]

So is New Hampshire just jinxed, or what?

Not necessarily. The dirty little secret in New Hampshire and elsewhere is that too many of the widely reported pre-election polls cut corners or otherwise use methods that are less than gold standard.

Perhaps the best-known of the bunch, Zogby International, does all kinds of controversial things to produce its headlines-grabbing tracking poll (see next item). Surveys taken by students for Franklin Pierce College, which is reporting a Democratic preference poll today, uses samples based on lists of registered voters that have proven to be incomplete, outdated or both. Suffolk University, which is polling for a Boston television station, asks a curiously convoluted candidate preference question that ends: "toward whom would you vote or lean?" Many professionals consider student interviewers unreliable, especially when unsupervised. Franklin Pierce and Suffolk University also use student interviewers, as does the University of New Hampshire. Polling directors at the schools insist that the kids are alright: "Their quality is tremendous," said Richard Killion, who oversees Franklin Pierce polls, later adding: "It really improved when I started paying them."

Research 2000, which does polls for the Concord Monitor, doesn't randomly select respondents within the households they contact. Instead, they interview the person who answers the telephone if they qualify as a likely Democratic primary voter on the basis of answers to subsequent questions. The data is adjusted so that the proportion of men and women, Democrats and independents as well as other key groups match the proportion who voted in the 2000 New Hampshire primary, reports Del Ali, president of the firm. That could improve the accuracy of the results if Tuesday's electorate is a carbon copy of 2000, but could be a problem if it is not.

Likewise, student interviewers at Franklin Pierce College also don't randomly select people once they reach a household. They talk to whomever answers the telephone unless they're sample is skewing more male or female. In that case, if they need more women, they ask in subsequent calls to speak to women, and vice-versa if their sample appears to be light on men.

The problem with not randomly selecting within households is that the resulting sample is not truly random and more likely to be biased in ways that are not readily apparent.

Despite these questionable practices, the track record of polls in New Hampshire may be no worse than elsewhere -- which may offer slim comfort to pollsters. "I think you probably would find this kind of thing in other primary states, but New Hampshire is much more visible," said UNH pollster Andrew Smith. "But New Hampshire does have some quirks that, if a pollster doesn't pay attention, they're going to be wrong."

Among them: wildly varying turnout; same day registration; rapid immigration of new voters; and a fine and confusing tradition of allowing residents to register as "undeclared."

"When you look at primaries, you'll probably see a lot more mistakes than you'd see in a general election," Smith said. "You're always going to have a more difficult time figuring out who's likely to vote."

What an Unkind Crack

That well-traveled pollster and media demi-star John Zogby is once again criss-crossing the country conducting exit polls in primary states from New Hampshire to Arizona, apparently much to the chagrin of the political mavens at ABC News.

"The svengalis at ABC News and some major papers don't like Zogby's tracking," reported The Note, the widely read ABC News online politics briefing . Then they quote an anonymous "ABC guru" who called Zogby's tracking polls "crack for the weak."

"I saw that," laughed Zogby, adding he was not surprised. ABC's polling department has reviewed his methodology and rated his polls "not airworthy," he said. "But that's all right. We're doing okay without ABC."

Okay, indeed. Last year The American Prospect in an otherwise critical article suggested that his polling firm's reputation ranked second only to Gallup's. This year he's working for a gaggle of media clients, including NBC and Reuters. Zogby will be polling in no fewer than seven primaries and he had the only daily tracking in Iowa, which garnered huge media coverage but ended up a bit off the mark.

ABC isn't the only news organization that holds Zogby polls in some disrepute. Many media pollsters -- including these writers -- question at least some of his methods. (Not that our doubts keep his work from being prominently featured in newspaper and broadcast news stories, at least in part on the theory that good or bad, his polls are news.)

"Zogby is not a reputable pollster," said Warren Mitofsky, who is co-directing the media exit polls this year for the major television networks and the Associated Press. "He is more a salesman and a self-promoter than a pollster. He has made lots of mistakes on election outcomes -- five in 2002. . . . I have heard of volatile campaigns, but he has volatile polls." (Zogby acknowledged on his Web site last February that "this past election cycle was not my finest hour.")

Iowa was not particularly kind to Zogby or the other public polls, all of which underestimated Kerry's big win and Gephardt's dismal showing. Keating Holland, who directs surveys for CNN, noted that Zogby "was off by 10 points on Kerry and overestimated Gephardt by 7, which is not a good track record. We'll see what he does from here on out." (Zogby said things were changing fast in Iowa up through election day and his final poll did have Kerry leading, Edwards surging and Gephardt finishing fourth. Zogby was the only one doing tracking polling in Iowa.)

"We clearly have different points of view about how polling should be done," said Kathleen Frankovic of CBS News who discussed methodology with Zogby after the 1996 election.

Why do so many pollsters go squinty-eyed when Zogby's name is mentioned? Here are some of their biggest complaints about the ubiquitous pollster from Utica:

Zogby draws his samples only from computer files of people with listed telephone numbers, thereby missing 30 percent of the population with unlisted telephone number.

The biggest advantage to what Zogby does is that most of those telephone numbers are good. The disadvantage is that you miss everyone with an unlisted number, and people with unlisted numbers tend to be different than those who are in the telephone book. Most other public pollsters rely on Random Digit Dialing, a technique that captures both listed and unlisted numbers. (Zogby says he plans to release a study later this year that shows no demographic or ideological differences between people with listed numbers and those who are unlisted.)

Zogby also calls people during the day as well as in the evening. About 30 percent of his interviews are collected before 5:30 p.m. Daytime interviews are great if you want to talk to lots of retirees and housewives. But his critics charge that they're not so good if you want to interview working men and women. Zogby counters that daytime interviewing actually produces a more representative sample because he can talk to people who work at night.

Zogby also adjusts his sample based on historic trends and his judgment of "what is happening on the ground" in a particular race, and it is this imposition of his own judgment that disturbs many pollsters.

He will, for example, reduce the proportion of 18- to 24-year-olds in his sample of self-described likely voters if he suspects on the basis of past voting history and the "lay of the land" that a sample contains too many younger people. He also, on occasion, adjusts the religious composition of his sample if he suspects he has over or under-represented one faith.

Most pollsters cringe at such extra-curricular adjustments. "I know I do some things different that others," he said. "I know the so-called 'Poll-ice' would deny it, but there's art as well as science involved in this."

Some of his techniques that were once widely criticized have now been more generally adopted. Zogby adjusts his sample so it matches the proportion of Republicans, Democrats and independents, based on past elections.

Critics say people can change their party identification, so past estimates of partisanship may be outdated. In fact, party identification nationally has changed little over the past two decades, prompting some organizations, including The Washington Post and ABC News, to weight to party identification on some political polls.

Zogby knows he's not popular with many of his professional peers. But he doesn't seem to care. "I'm a humble guy from Utica just plying my trade," he laughed. "I'm just not a member of the club."

Sunny Side Right

Surging Democratic candidate John Edwards says he wants to get the votes of America's optimists.

A recent Post-ABC News poll holds some mixed news for the North Carolina senator: he's got a real opening, but it's with Republicans.

The survey asked Americans whether they were optimistic or pessimistic about the country in general, the national and local economy, their own finances, the situation in Iraq, homeland defense and the nation's moral tone.

Overwhelmingly the most positive respondents: strong Republicans. And the most negative? Yep, strong Democrats. Eight in ten committed Republicans said they were optimistic about all or nearly all these things, compared to only two in ten strong Democrats. (Maybe that's why Dean leads among Democrats in the national polls rather than Sunny Boy Edwards?)

Other sourpusses: atheists and agnostics looked none too sunny. Only three in ten scored high on the optimism scale, compared to close to half of those who claimed a religious affiliation.

The K-Factor in Iowa

On the morning after Iowa, Joe Lenski of Lenski/Mitofsky, called major clients of the Iowa Democratic entrance poll to ask the age-old question, "How was it for you?"

Quite good, actually. And thanks for asking. The National Election Pool entrance poll had performed close to flawlessly for the sponsoring networks and Associated Press, as well as for their media clients. Quite a change from 2002, when the news media exit polls didn't perform at all, due to a technical meltdown that ultimately led to the demise of NEP's predecessor, Voter News Service.

But one lingering question: Why did the final entrance poll estimates differ by so much from the final results. For example, the final estimate released from the National Election Pool put John Edwards support at 26 percent, and he ended up getting 32 percent. The John Kerry estimate was also noticeably different. NEP estimated 32 percent, Kerry got 36 percent.

Could it be that the very first survey by NEP was . . . wrong?

Right question, wrong answer, Lenski said. Yes, the latest Democratic entrance poll had the biggest gap between entrance poll estimate and the final result ever since the networks started doing these surveys in 1984, he said. But it had nothing to do with the quality of the poll and everything to do with the quirky nature of the Iowa caucuses . . . and Rep. Dennis Kucinich (D-Ohio).

Here's what happened. Iowa Democrats caucused several times on Monday. The results of the first caucus, held shortly after 7 p.m. Central Standard Time, became the NEP final estimate; in this case, 26 percent for Edwards and 35 percent for Kerry.

But then Iowa Democrats began a furious half-hour of politicking and candidate-switching that ultimately ended in a final caucus tally that gave Kerry 38 percent and Edwards 32 percent support (technically expressed as a percentage of "delegate equivalents).

Lenski said his analysis of past entrance polls found that leading candidates "tend historically to gain two to four points in the delegate equivalents statewide," he said. Mondale in 1984, Gephardt and Simon in 1988 and Gore in 2000 gained 2 to 4 points in the delegate equivalents compared with the initial preference. That is also what Kerry did in 2004.

But what about that six-point surge for Edwards, a new record? He says that's the payoff Edwards got, thanks to a deal in which Kucinich pledged to shift he supporter to Edwards in precincts where he got less than 15 percent of the total support if Edwards would do the same for him in his low-performing precincts.

Needless to say, Kucinch, who got 1 percent support, had many more low performing precincts than Edwards.

"I think that is evidence that the deal with Kucinich helped to move a couple of points to Edwards in the delegate equivalents," Lenski said. "Also in our entrance poll Edwards was the top second choice of the relatively small number of Kucinich voters," more evidence of the K-Factor in Monday's caucuses.

Share your opinions with the Poll Watchers at polls@washpost.com

COPYRIGHT 2005 Washingtonpost Newsweek Interactive
COPYRIGHT 2004 Gale Group