Comments on the extraversion-ESP meta-analysis by Honorton, Ferrari, and Bem - response to article in this issue, p. 255 - extrasensory perception

Among the most important contributions made by the late Charles Honorton were meta-analyses involving psi effects that had been studied in a sufficiently large numbers of experiments to merit such treatment (Honorton, 1985; Honorton & Ferrari, 1989; Honorton, Ferrari, & Bern, 1998). All of them had been published in the Journal of Parapsychology except the one involving ESP and the psychological trait of extraversion, a deficiency we are pleased to remedy in this issue. The principal conclusion from this latter meta-analysis was that there exists a positive relationship between extraversion and ESP scores, for both forced-choice and free-response measures of ESP, that is statistically significant (p = 4 x [10.sup.-6]) although small in magnitude (r = +.09). More refined analyses led Honorton and his co-authors to conclude that for forced-choice ESP tests, the result was due to an artifact, namely, the effect of "subjects' knowledge of their ESP performance upon their responses to the extraversion measure" (Honorton et al., 1998, p. 255). The purpose of this short paper is to challenge this latter conclusion, or, to put it conversely, defend the legitimacy of the extraversion-ESP relationship for forced-choice experiments.

The general problem that underlies our disagreement with Honorton et al. has to do with confounded predictors. When we compare the performance of two groups of subjects in a single experiment that is properly conducted, the two groups differ (whether in terms of their individual traits or the treatments administered to them) on only one factor or independent variable. Therefore, significantly different scores on the dependent variable can be attributed only to this one independent variable or predictor. However, such is not the case when the results of two groups of experiments are compared in a meta-analysis. Because such databases usually include studies designed by many different investigators who generally do not seek to address in a controlled fashion the relationship of interest to the meta-analyst, such comparability on extraneous variables is not to be expected. In other words, the two groups of studies may differ in many other ways besides the one the meta-analyst seeks to assess, and one of these may be the real cause of differences in scores on the dependent variable.

In the extraversion meta-analysis by Honorton et al., one independent variable of interest to the authors was whether subjects completed the extraversion questionnaire before or after they learned their ESP scores. They found that the positive extraversion-ESP correlation was only significant for those subjects who completed the extraversion scale after they took the ESP test, and thus after they knew their ESP scores. The authors offered two explanations for the results of these compromised studies. First, subjects' perception of their success or failure in the ESP task might have influenced how extraverted or introverted they perceived themselves to be. Second, the experimenters' knowledge of the ESP scores might have led to experimenter expectancy effects.

With regard to these explanations, it should be noted that the authors provided no rationale for their plausibility, nor did they provide any evidence that extraversion scales are susceptible to such biases. We were first moved to look into the authors' claim of an artifactual relationship because their explanations did not appear plausible to us. This is not for parapsychological reasons, but because of the nature of the measures of extraversion that the studies employed. Personality inventories have been constructed in such a way as to meet the criterion that the scores they produce are very, nonreactive to a person's ongoing experiences - changes of mood or state of mind, etc. This psychometric aim has generally been achieved (e.g., Cattell, 1957; Gough, 1957; Kelly, 1955). Another way of putting this is to say that these instruments have high test-retest reliability. The proposed interpretations would require us to believe that feedback about an ESP test resulted in sizable mean shifts in scores that are known to be nonreactive to such minor perturbations.

A more direct test of Honorton et al.'s subject bias hypothesis was provided by Krishna and Rao (Experiment 2, 1991), who gave a forced-choice ESP test followed by Cattell's High School Personality Questionnaire (HSPQ) to 80 female Indian students, ranging in age from 15 to 19, in a group setting. Prior to taking the HSPQ, half of the students were falsely told that they had all scored well above chance on the ESP test, whereas the other half were falsely told they had scored well below chance. The means of the two groups on the HSPQ extraversion scale did not differ significantly from each other (p = .53), which means that ESP feedback did not influence how subjects described themselves with respect to extraversion. Similar results were found with the HSPQ neuroticism scale.

Coding Changes

Honorton kindly gave copies of the raw data files used in the meta-analysis to Carpenter. He went through the reports of all 60 studies used in the meta-analysis to check on the accuracy of the coding with respect to order of testing. Of the 45 forced-choice studies, Honorton et al. had coded 16 as having given the extraversion test before the ESP test (EIPRE), and 18 as having given the extraversion test after the ESP test (EIPOST). Insufficient information was provided in the reports of the other 11 studies, which were classified as UNKNOWN.

Carpenter found no coding errors in the free-response studies, but he concluded that three of the forced-choice studies coded as UNKNOWN should have been coded as EIPRE (Sargent & Harley, 1981; Shields, 1962(1)), thereby raising the size of this group from 16 to 19 studies. This adjustment 'also raises the extraversion-ESP correlation in the EIPRE group from r = -.02 to r = +.01. The associated z is raised from -0.78 to +0.43, and the corresponding p-value is reduced from .782 to .334, one-tailed. Cohen's q, which the authors used to evaluate the difference between the correlation in the EIPRE group (+.01) and the corresponding one in the EIPOST group (+.17), is reduced from .25 to .16, z = 2.98, p = .003. The effect of these coding changes is trivial and they do not alter the authors' conclusions.

Confounded Predictors

Honorton et al. reported that the significant correlation between extraversion and ESP in the EIPOST group was entirely attributable to the 9 (of 18) studies where the degree of feedback was specified in the report. The authors noted that the 9 studies in which degree of feedback was not reported were mostly group experiments in which feedback, if given at all, is ordinarily delayed. It therefore seemed best to reclassify these studies as UNKNOWN, leaving the remaining 9 to constitute the EIPOST group (a decision made before the following analyses were undertaken).

Elsewhere in their paper, the authors noted that among forced-choice experiments in their meta-analysis there is a highly significant difference between studies in which subjects were tested individually and those in which they were tested in groups (p = 5.2 x [10.sup.-4]). The significant extraversion-ESP correlation is contributed entirely by the studies using individual testing. Could this be a possible confound with order? To find out, we compared the group-testing status ("test setting") of the EIPRE and EIPOST subgroups. We found that all 9 of the EIPOST studies used individual testing whereas only 8 of the 19 EIPRE studies did. The resulting 2 x 2 contingency table yielded p = .004 by Fisher's Exact Test. Thus, there clearly is a confound between order of testing and test setting status as predictors of the extraversion-ESP relationship. Using the z-transformed correlation coefficients between extraversion and ESP as the dependent variable, we found that among the 19 EIPRE studies the correlation for the individual-testing studies was r = +.21, compared to r = -.05 for the group-testing studies. The difference is significant, t(17) = 2.17, p = .045. Finally, we combined order and test setting as predictors of the transformed extraversion-ESP correlations in a multiple regression analysis. The contribution of test setting is significant, t = 2.22, p = .036, whereas the contribution of order is not, t = 0.93, p = .364.

Disconfounding the Confound

The preceding analyses suggest that group-testing status, rather than order of testing, is the true mediator of the extraversion-ESP correlation. In this section, we report additional evidence in support of this conclusion. Three of the 9 studies in the EIPOST group were actually series in a larger experiment reported by Kanthamani and Rao (1972). All used very similar methodology, as did a fourth series in which the extraversion test was given before the ESP test. This series was appropriately placed in the EIPRE group by the authors. With the exception of different experimenters giving the ESP test, the only methodological difference that seemed to exist between Series 4 and the other three was order of testing (Kanthamani & Rao, 1971). Therefore, these studies, treated collectively, closely approach the kind of unconfounded manipulation of the order of testing that we ordinarily would find only in a single experiment. The results of the four series are presented in Table 1, which is a reproduction of Table 2 in Kanthamani and Rao (1972). Note that the outcome in "Exp. C" (the EIPRE series) is quite comparable to the net result of the other three series. The difference of 0.49 between the ESP means of extraverts and introverts in Exp. C is very similar to the mean difference of 0.56 in [TABULAR DATA FOR TABLE 1 OMITTED] the combined results of the other three series. The published data only allowed a comparison of the two groups of series using the run as the unit of analysis. We compared the difference between the proportion of hits obtained by extraverts and introverts in the combined EIPOST (.023) series to the corresponding difference in the one EIPRE series (.020). The difference between the differences proved to be statistically negligible, z = 0.12, p = .904.(2)

These results from an unconfounded comparison reinforce the conclusion that in the Honorton et al. meta-analysis, the order effect upon the extraversion-ESP relationship is itself an artifact of test setting. That is to say, the reason why the EIPOST studies were more successful than the EIPRE studies is that they uniformly used individual testing, whereas most of the EIPRE studies used group testing. This view is also reinforced by the fact that individual testing was found to be superior to group testing in Honorton's meta-analysis of precognition experiments (Honorton & Ferrari, 1989). It is also possible that other, unrecognized confounding variables contributed to the problem.

The above outcomes mean that the order of testing artifact cannot be used as a basis for discounting the legitimacy of the positive extraversion-ESP correlation for forced-choice studies. The one other basis on which its legitimacy might be challenged is Honorton et al.'s finding that studies using suboptimal methods of preventing sensory cues yielded a significantly higher extraversion-ESP correlation than did studies using optimal methods (p = .042). However, we selected this marginal p-value post hoc from among a total of four methodological flaws evaluated, and it cannot withstand a correction for multiple analyses. Moreover, a correlation between the z-transformed correlation coefficients of the 45 forced-choice studies in the database and their overall quality codings was slightly positive, r = +.04. Nonetheless, we do not think the finding about sensory cues can be completed discounted in assessing the validity of the extraversion-ESP relationship for forced-choice studies.

1 The Shields (1962) report described two experimental series.

2 For the sake of completeness, it should be noted that Exp. B differed from all the other series in that the subjects were males rather than females. Removing Exp. B from the analysis would only serve to make the EIPRE-EIPOST difference even smaller.

REFERENCES

CATTELL, R. B. (1957). Personality and motivation: Structure and measurement. Yonkers-on-Hudson, NY: World Books.

GOUGH, H. G. (1957). Manual for the California Psychological Inventory. Palo Alto, CA: Consulting Psychologists Press.

HONORTON, C. (1985). Meta-analysis of psi ganzfeld research: A reply to Hyman. Journal of Parapsychology, 49, 51-92.

HONORTON, C., & FERRARI, D. C. (1989). "Future Telling": A meta-analysis of forced-choice precognition experiments, 1935-1987. Journal of Parapsychology, 53, 281-308.

HONORTON, C., FERRARI, D. C., & BEM, D. J. (1998). Extraversion and ESP performance: A meta-analysis and a new confirmation. Journal of Parapsychology, 62, 257-278.

KANTHAMANI, B. K., & RAO, K. R. (1971). Personality characteristics of ESP subjects: I. Primary personality characteristics and ESP. Journal of Parapsychology, 35, 189-207.

KANTHAMANI, B. K., & RAO, K. R. (1972). Personality characteristics of ESP subjects: III. Extraversion and ESP. Journal of Parapsychology, 36, 198-212.

KELLY, E. L. (1955). Consistency of the adult personality. American Psychologist, 10, 659-681.

KRISHNA, S. R., & RAO, K. R. (1991). Effect of ESP feedback on subjects' responses to a personality questionnaire. Journal of Parapsychology, 55, 147-158.

SARGENT, C., & HARLEY, T. A. (1981). Three studies using a psi-predictive trait variable questionnaire. Journal of Parapsychology, 45, 199-214.

SHIELDS, E. (1962). Comparison of children's guessing ability (ESP) with personality characteristics. Journal of Parapsychology, 26, 200-210.

Rhine Research Center 402 N. Buchanan Blvd. Durham, NC 27701 email: john@rhine.org

COPYRIGHT 1998 Parapsychology Press
COPYRIGHT 2004 Gale Group