Polling job performance and favorability: three approaches - Cover Story

CLEAR DISTINCTIONS, SAFE ASSUMPTIONS

One of the most over-repeated phrases we pollsters hear is, "there are three types of liars: liars, damn liars and statisticians." While I obviously believe that to be an unfair characterization, it is easy to understand why many Americans are skeptical of survey research and often disregard the results. There are clear distinctions between the way different pollsters measure public opinion. For instance, media pollsters (Gallup, Harris, Yankelovich) use very different methods and approaches than do most political pollsters. There are several theories for why this is - and I certainly have my own - but that is a subject for a different article.

An area with widespread variation, especially between media and political pollsters, is the ability to accurately ascertain 1) job performance or job approval and 2) favorability ratings. A key element is simply understanding the difference between those above terms, what they mean to voters in general, and then comprehending the techniques available to accurately measure each of them.

While performance and approval are relatively similar, favorability must be measured differently to establish true strengths and weaknesses, not just overall - but in the levels of intensity as well - which is even more important than the aggregate rating.

First, the best way to measure job performance is to forgo terms such as approve/disapprove or positive/negative for a more descriptive scale ranging from excellent to poor. One of my own pet peeves is using the word "fair" to determine job performance. Webster defines fair as "sufficient but not ample." I'm sure most everyone else has his or her own personal definition and it is a safe assumption that each one is different.

It is reasonable to surmise that there is a range in which people view the word "fair" - from status quo to Webster to simply sub-par. You have to look at any survey result using such nebulous terms very critically. Therefore, the job performance scale in my surveys uses "not so good" rather than "fair" or "only fair" as the third of the four categories (the others being "excellent," "good" and "poor"). This alternative wording, using "not so good," does have the effect of elevating overall job performance ratings somewhat.

However, extensive research has shown that the typical wording severely underestimates the level of satisfaction with an incumbent's performance, as a substantial proportion of those who say an incumbent's performance is "fair" remain willing to support his or her re-election.

For instance, in a recent congressional survey - just to illustrate this distinction for a client - the incumbents' job performance was asked both ways (excellent, good, fair, poor vs. excellent, good, not so good, poor). The incumbent was receiving more than one in three votes (35 percent) from those who rated him "fair" or "poor" while he was receiving fewer than one in five votes (19 percent) from those who rated him "not so good" or "poor" - a significant and important difference.

Next, in looking at a candidate's or individual's favorability rating, there is little variance in the way most pollsters ask the initial question. The differences come from the way intensity is determined. The three main terms used are "strongly," "highly" and "very" on the high end and "mostly" vs. "somewhat" on the low end.

I have found the descriptive terms "strongly" and "highly" tend to underestimate intensity of favorability. Similarly "mostly" tends to overestimate the deficiency in intensity of favorability. Again, the difference is in the gradations, not the aggregate. But for correlation to vote performance and other important factors, the level of intensity is far more important than the overall percentage of favorability.

For instance, in asking favorability on a recent survey, I used a split sample with the term "strongly" for one-half of respondents and "very" for the other half. For the same individual (being tested) the intensity ratings were a full 10 points higher among those choosing "strongly" (48 percent) than they were with those choosing "very" (38 percent). Furthermore, the candidate was 15 points more likely to be the vote choice of those who were "very" (85 percent) favorable than he was with those who were "strongly" (70 percent) favorable. Once more, an important and significant distinction.

Finally, I have found that "mostly" tends to underrate that same intensity proportion and, again, miscalculate the individual's dependable strength in relation to voter support. Therefore, I use the term "somewhat" as a counterbalance to "very" to ascertain a candidate's overall favorability rating.

Pollsters - political and media analogously - are likely to have their own opinions regarding these issues. My conclusions are arrived at through the predominant and overriding goal of utilizing both job performance and favorability to determine a candidate's strength with the electorate. If survey questions, and therefore numbers or findings, don't give you a basis from which to form an overall campaign strategy, in my view, they are worthless. Therefore, every question - whether it deals with job ratings, favorability or any other presumably important point - must pass one very important test: Does it help the poll consumer/client accomplish his or her overall goal? (which in my case is usually winning an election). If the answer is no, the question either needs to be reworded or it doesn't need to be asked.

OLD FASHIONED BUT HIGHLY VALID

For years the rule in the polling business has been "sampling is everything." But as the field becomes more and more crowded and sophisticated, there is increasing focus on how questions are asked. While several other major public polls have consistently shown President Clinton's job performance ratings above 65 percent recently, we have generally rated his performance in the '50s. To be sure, one reason is that we poll only likely voters, as opposed to many of our colleagues who poll all adults or registered voters. Using the tighter screen for actual voter likelihood tends to decrease the number of low-income and minority representation in the sample - people with less of a tendency to vote and many with Democratic Party leanings.

But that is not the only difference. We use the old-fashioned, but highly valid, four-choice question on job performance. Essentially, we ask likely voters to rate the performance of public officials as "Excellent, Good, Fair, or Poor." We also do the personal rating using a scale of "Very Favorable, Somewhat Favorable, Somewhat Unfavorable, Very Unfavorable or Not Familiar Enough to make a judgment."

The four-choice performance question offers more choices and presents a clear difference between positive and negative ratings. To those public officials who have questioned the placing of the "Fair" rating in the negative column, we respond that it might be a passable rating but not enough for either bragging rights or re-election certainty. It's like a C- or D, it passes you but doesn't get you into a good graduate school.

The "Approve/Disapprove" rating does not get into subtle differences nor does it relate the intensity factor. Thus, when we report our findings, we are able to offer an overall Positive/Negative as well as the intensity poles - Excellent and Fair. The simple "approve" is like the Pass/Fail system in college. So you made it, but how do you compare with other students?

I can see the rationale of using "Only Fair," but I think it risks leading the respondent for two reasons: first, because it adds a word, thus breaking the symmetry. Second, "only" is not a neutral word, it requires an emphasis which can be leading.

I have the same problem with using "Very Good" in the scale. First, there is a wide difference between "Very Good" and "Fair." Second, the additional word loads the question. As for the use of "Pretty Good" and "Only Fair," what is the difference?

We have never had a problem between job performance and personal ratings. A good case in point is Vice President Al Gore whose job rating seems to go up and down depending on the latest news but whose "Favorability" rating stays pretty high.

ESPECIALLY REVEALING

Over the years, many questionnaire formats in political polls tend to fall into and out of favor as interpretive tools. Often a particular format is especially revealing for a certain season and then fades as the broader political topography shifts.

And then there are those key core questions whose impact may vary from poll to poll, and election cycle to election cycle, but whose analytic value remains considerable. The most prominent of these, easily - at least in races involving an incumbent - is the incumbent's job performance rating.

In our firm, we strongly prefer the following format in assessing job performance: "excellent," "good," "only fair," or "poor." It is critical that a campaign be able to delineate voters who are generally positive to the incumbent's stewardship, and those generally negative, as well as to gain some insight with respect to the intensity of those assessments. This is a more finely calibrated process than it may appear at first glance.

For example, voters have proven consistently less willing to say they "disapprove" of an incumbent's performance (even using a four-point scale) than they are to rate it "only fair." This tendency yields an overall "approval rating" that is misleadingly high.

Also, using "very good" as a substitute for "good" has been shown to depress disproportionately an incumbent's positive ratings, while the failure to modify the rating of "fair" with the word "only" makes that option overly inviting, especially in certain electoral groups.

Generally speaking, and especially as a campaign moves into the final several months, there is an approximate one-to-one correlation between positive job ratings ("excellent" plus "good") and vote share. Typically when there is a disparity between the two in a poll (i.e., 50 percent positive rating and 57 percent for the incumbent in the trial heat) it is the vote share that drifts toward the job rating, rather than vice versa.

It is critical - for both incumbents and challengers - to have as accurate a reading as possible about voters' true assessment of the incumbent's job performance, and this is why we so strongly recommend to our clients the above format.

In addition, it is important to distinguish job ratings from assessments of personal appeal. The former are usually determinative, the latter, tangential, when it comes to the ultimate electoral outcome. (Typically, personal appeal runs higher than job ratings.)

This helps explain, of course, the nature of a challenger's bid. For a challenger to win, he must successfully call into question the incumbent's stewardship, as reflected in his job rating. This is why - despite many challengers' best intentions a challenger who truly intends to win a campaign of "dueling white papers" is kidding himself. It is imperative that voters hand down a political indictment if the incumbent is to be defeated.

Alternatively, an incumbent must make it a priority - between and during campaigns - to reinforce perceptions of his performance, especially among weakly aligned supporters (as identified in a poll).

Since voters do not like to "pre-commit" to a candidate the way they did 10 years ago, other poll questions - such as "re-elect" scores absent a named opponent - have become far less revealing.

The first piece of poll data to look at when the crosstabs start cranking isn't the trial heat, it's the incumbent's job rating. And time after time - especially in today's "show me" era when credit that used to be given by voters now must be claimed by incumbents - it has made the critical difference between winning and losing.

Chris Wilson is vice president of Fabrizio, McLaughlin & Associates, a Republican polling and political consulting firm in Alexandria, VA.

John Zogby is founder of Zogby International, a public opinion, survey research and marketing firm based in Utica, NY and Washington, DC.

Alan Secrest is a president of Cooper, Secrest & Associates Inc., a Democratic polling firm based in Alexandria, VA.

COPYRIGHT 1998 Campaigns & Elections, Inc.
COPYRIGHT 2000 Gale Group