Experiment one of the SAIC remote viewing program: a critical re-evaluation - Science Applications International Corporation

For over 20 years, the US government has funded experiments that examine the possible existence of "remote viewing," the ability to psychically acquire information from a distant location. During the early 1970s, this work was carried out at the Stanford Research Institute (SRI). The program was transferred to SRI International (formerly SRI) in 1973, and then continued between 1992 and 1994 at Science Applications International Corporation (SAIC).

In September 1995, the American Institutes for Research (AIR), contracted by the Central Intelligence Agency, assembled a "blue ribbon" panel to evaluate this research (Mumford, Rose, & Goslin, 1995). This panel included two reviewers chosen for their familiarity with parapsychological research: Dr. Jessica Utts, a professor of statistics at the University of California at Davis and Dr. Ray Hyman, a psychologist at the University of Oregon.

Utts (1995) used eight methodological criteria to assess ten of the SAIC studies, and chose one of the experiments - Experiment One - to demonstrate the use of the criteria; the experiment appeared to satisfy all eight and obtained statistically significant results. Hyman (1995) also stated that he could find no obvious problems with the experiments.

The publication of this report prompted the first author (henceforth referred to as R. W.) to become interested in attempting to replicate some of the SAIC research on remote viewing, and, in February 1996, he contacted Edwin May regarding this plan. May was enthusiastic about the idea and asked R. W. to propose an experimental protocol. Before doing so, R. W. thought it would be useful to examine the protocol used in Experiment One as a potential template for the replication. He chose this study for three reasons. First, it was the one study that had been discussed in detail by Utts (1995) in her AIR report. Second, it was one of the few SAIC studies that had been published in a peer-reviewed journal (Lantz, Luke, & May, 1994). Third, as for all of the SAIC experiments, several pitfalls to be avoided in remote viewing methodology had already been identified in critiques of the SRI work upon which it built,(1) and its design had been examined before the study began by a "Scientific Oversight Committee" that included a Nobel Prize-winning physicist and internationally known professors of statistics and psychology.

Because the information in Lantz et al.'s (1994) paper and the AIR report was not sufficient to fully evaluate the experiment's protocol, R. W. asked May for additional details. As the assessment of the protocol progressed, it became apparent that Experiment One's methodology contained some problems. R. W. therefore decided to examine the protocol in detail, both to reassess the validity of the experiment and to help ensure that future studies, including his own replication attempt, would avoid similar pitfalls. May agreed to provide the necessary unpublished information concerning Experiment One's procedure, in order that R. W. could conduct a detailed critique for publication. All personal communications cited from May are electronic mail messages provided for the purpose of the critique, following this agreement.(2) We are indebted to May for taking the time to address the issues raised. The outcome of this critical reevaluation is presented here.

SUMMARY OF EXPERIMENT ONE

Lantz et al. (1994) describe three experimental roles for SAIC personnel in Experiment One without identifying the holders of those roles by name. May (personal communication to R. W., April 29, 1996) informed R. W. that the Principal Investigator was Nevin Lantz, the Experiment Coordinator was Wanda Luke, and the Analyst (independent judge) was May himself. To clarify later discussion, these experimenters are referred to by name instead of by title in the following account of Experiment One, which is summarized from Lantz et al.

Experiment One was carried out in 1992 and employed a 2 x 2 design to explore whether receivers' psychic performance was affected by (a) the presence or absence of a sender and/or (b) the type of target used during a trial. Half of the trials, therefore, involved a sender while the other half had no sender, and half of the trials involved "static" targets (photographs of outdoor scenes taken from National Geographic magazine) while the other half used "dynamic" targets (a varied selection of video clips from popular movies). Prior to the experiment, the targets had been grouped into sets of five, each set composed of only static targets or only dynamic targets. Targets within any one set were chosen to be as different from one another as possible.

Five receivers were involved in the experiment, each doing 10 trials in each of the 4 conditions. Two of the viewers lived in California and the others resided in Kansas, New York, and Virginia, respectively. Lantz, who acted as sender, was in Lititz, Pennsylvania. May and Luke were at SAIC in Menlo Park, California.

Prior to the study, Luke randomly selected 40 targets (20 static, 20 dynamic) for each participant. A copy of each target was placed into an envelope and a trial number (1 to 40) was written on the outside of the envelope. Half of the static targets and half of the dynamic targets were randomly assigned to the "no-sender" condition and the envelopes containing these targets were sealed. The remaining targets were assigned to the "sender" condition and these envelopes were left unsealed.

Luke then mailed all of these targets to Lantz. For each trial, Lantz selected the appropriate envelope at a prearranged time. For "no-sender" trials, he simply placed the unopened envelope on his desk. For "sender" trials, he looked at the target for approximately 15 minutes. During this time, the receiver, at his or her home, produced a "response" by writing and/or drawing his or her impressions of the target. At the end of the trial the receiver faxed this response to Lantz. Lantz then sent the target to the receiver by return mail, to act as feedback. The receiver subsequently sent the original copy of the response and the copy of the target to Luke at SAIC.

When Luke received the response, she removed the receiver's name and the date and time of the trial. She then gave May, the independent judge, the receiver's response and the appropriate "target set" (i.e., the set containing a copy of the actual target and four decoys). He ranked the target set items in order of their correspondence to the receiver's response, giving a rank of 1 to the target most similar to the response.

Using Solfvin, Kelly, and Burdick's (1978) sum of ranks statistic(3) to assess the outcome, we find that the whole study was statistically significantly above chance (N = 200, effect size, z/[N.sup.1/2] = .12, p [less than] .043, one-tailed). Lantz et al. (1994) also found post hoc that trials involving the static targets were independently significant (N = 100, effect size = .24, p [less than] .0073), whereas those with dynamic targets were at chance (N = 100, effect size = .00, p = .5). There was no significant difference in scoring between those trials that involved a sender and those that did not, nor were there any significant interactions between the sender and target variables.

A CRITICAL RE-EVALUATION

Examination of Lantz et al.'s (1994) description of the study indicated the possibility of four methodological loopholes. Assessment of three of these potential loopholes was complicated by the fact that the SAIC team encountered problems in reconstructing from memory certain unrecorded aspects of Experiment One's procedure. It is necessary to describe these problems in some detail to make clear what kind of procedures are at issue and the extent of the difficulties in reconstructing them. These three potential loopholes and related issues of memory are discussed in the first three sections. A final section outlines the fourth potential loophole.

1. Safeguards Against the Experimenters Inadvertently Altering the Receivers' Responses

In ESP research it is generally accepted that any individual who records, transcribes, or edits a receiver's response should be blind to the identity of the target (Milton & Wiseman, 1997). This safeguard prevents experimenters inadvertently biasing the response to match the target, for example, by unconsciously tending to edit out inaccurate sections of the response more than accurate sections. In Experiment One, R. W. noticed what appeared to be an opportunity for a similar potential flaw, namely, for a nonblind experimenter (Luke) to accidentally leave handling cues on the receivers' responses that might unconsciously bias the independent judge.

According to Lantz et al.'s (1994) account, Luke had two opportunities to discover the identity of each trial's target. First, she randomly selected the targets, packaged them, and shipped them to Lantz. Second, the receivers returned the feedback copy of the target to Luke. Lantz et al. also note that the receivers sent their responses to Luke, who removed the receiver's name, the date, and trial number from the response sheet. In short, Lantz et al.'s account of Experiment One described no safeguards against Luke knowing the target for each trial and accidentally leaving marks (e.g., smudges) on the response sheets that might have cued the independent judge to the target's identity without him having consciously observed the cues.

Given that this procedure outlined by Lantz et al. (1994) seemed to contain a potential flaw, R. W. asked May whether their description of this aspect of the procedure was correct. May (personal communication, April 29, 1996) confirmed that there had only been one person responsible for selecting the targets and removing the trial-identifying information from the responses when they were returned to SAIC, and that person was Luke. He also stated that the receivers never sent the target material back to SAIC (personal communication, April 29, 1996). However, Lantz et al. state, "The target copy and original response were subsequently sent to the Experiment Coordinator [Luke] in Menlo Park, CA [SAIC]" (p. 289). May's memory of this aspect of the protocol conflicted with the account in Lantz et al. (see the first two rows of Table 1). R. W. asked May about this discrepancy and he replied that he could not have remembered it correctly and that either the Lantz paper or his memory was incorrect (personal communication, April 29, 1996).

May then asked Luke and Lantz about this aspect of the protocol and sent the resulting "collective memory" to R. W. (personal communication, August 29, 1996). This collective memory differed in two important ways from the information in Lantz et al. (1994) and from the situation originally recalled by May (see the first three rows of Table 1). Both Lantz et al. and May stated that only Luke selected the targets, whereas the collective memory indicated that three people were involved. Also, Lantz et al. and May stated that Luke received and edited the responses, whereas the collective memory stated that DeGraff, a research assistant not mentioned in Lantz et al., carried out these tasks.

May then checked again with the other SAIC personnel and sent R. W. a second collective memory which differs from the information presented by Lantz et al. (1994), May's initial recollection, and the first collective memory (see the third row of Table 1). The second collective memory stated that Macgowan (another research assistant also not mentioned in Lantz et al.) and DeGraff initially selected and packaged the targets. Lantz et al. and May stated that these tasks were carried out by Luke, while the first collective memory stated that all three individuals were involved. The second collective memory also stated that the targets were returned to Macgowan and DeGraff. In contrast, Lantz et al. stated that they were returned to Luke, whereas May did not think they were returned at all. Finally, according to the second collective memory, the responses were returned to Luke. This is in agreement with Lantz et al. and May, but not the first collective memory, which stated that they were returned to DeGraff.

Table 1

DIFFERENCES BETWEEN SAIC EXPERIMENTERS' ACCOUNTS OF TARGET SELECTION
AND TARGET AND RESPONSE HANDLING IN EXPERIMENT ONE

                                       Handling         Handling
                       Target          returned        and editing
                     selection          targets         responses

Lantz et al.
(1994)                  Luke             Luke             Luke

                                     Targets not
May                     Luke           returned           Luke

First                   Luke,
collective             DeGraff,          Not
memory                Macgowan         mentioned         DeGraff

Second
collective             DeGraff,        DeGraff,
memory                Macgowan          Macgowan           Luke

Third                  DeGraff,
collective            Macgowan,          Not
memory           Student assistant     mentioned           Luke

In response to receiving an earlier version of this paper that was submitted to the 1997 Convention of the Parapsychological Association (Wiseman & Milton, 1997), May sent R. W. a final account of events (personal communication, April 30, 1997). He noted that he believed this description was the most reliable as he had spent a considerable amount of time carefully interviewing the people involved. In this final account of events, the targets were selected by DeGraff, Macgowan, and a student assistant. Luke received the responses and was told from which target pack the target was chosen, but was blind to the target. The final row in Table 1 shows the differences between this third collective memory and the previous recollections.

The experimenters clearly found it difficult to reconstruct from memory a consistent account of this aspect of the experimental protocol. This is perhaps not surprising, given that the experiment took place approximately four years before this reassessment began, and was only one of several remote viewing experiments conducted by the research team. If their final description of Experiment One is correct, then there is no opportunity for the kind of inadvertent sensory cuing that we have discussed. However, the difficulties in reconstruction experienced by the SAIC team makes it difficult to know how much certainty one can attribute to the final account of the experimental protocol.

2. Safeguards Against Receivers Altering their Responses after Finding Out the Identity of the Target

Several commentators on parapsychological methodology have recommended that the target should remain shielded from the receiver until after his or her response has been secured (Akers, 1984; Morris, 1986). This safeguard is important not only to rule out deliberate cheating but also because receivers might unknowingly smudge those parts of the response that most resemble the target, for instance as they trace a line with a finger or point out target-matching areas on the response sheet to another person. The judge's attention might be drawn to such areas because they look somewhat different from the rest of the response, not realizing that the differences are due to handling, or that the response sheet has been handled nonblind.

It is difficult to determine whether Experiment One employed sufficient measures to guard against this problem from the information presented in Lantz et al. (1994). They state that at the end of each trial, (a) the receivers faxed a copy of their responses to Lantz, (b) Lantz then sent the target, as feedback, to the receivers by return mail, and (c) the receivers sent their original responses to SAIC. However, Lantz et al. did not make it clear whether the receivers sent their responses to SAIC before or after they had received feedback from Lantz, and whether the judge used the faxed or mailed response (a point also raised by Utts, 1995).

R. W. asked May to clarify this point. He replied that it was possible that the receivers may have been in possession of their response and the feedback target at the same time (personal communication, April 29, 1996). May also noted that he had carried out the judging using a photocopy of the original version, not the faxed version (personal communication, April 29, 1996). He would therefore have been judging photocopies of some responses that may have been handled by the receivers after they had obtained feedback. However, May noted that Luke had compared the responses that the receivers faxed to Lantz with the originals that they sent to SAIC and found no differences between them (personal communication, April 29, 1996). Utts (1995) reported asking May the same question and obtained the same response.

In a later communication, in which May presented the third and final collective memos, May noted that this memory of events was incorrect, and that the SAIC team's final collective memory of events indicates that he had judged from the faxes sent to Lantz, not the original responses (personal communication, April 30, 1997). There is, however, some inconsistency among the SAIC team concerning this aspect of the final collective memory.

Assuming that the final reconstruction of events is correct, the possibility of receivers inadvertently leaving any cues on their responses is ruled out. It does, however, open up a new potential pathway for sensory leakage, namely, that Lantz (who knew those targets in the "sender" condition) may have left cues on the faxes when he handled them directly after the completion of the trial. If this were the case, Lantz would only have left such cues on responses in the "sender" condition, as he would have been unaware of the targets in the "no-sender" condition. One would therefore predict that the "sender" trials would outscore the "no sender" trials, but, in fact, there was no significant difference in scoring between the two sets of trials. However, the problems in reconstructing this aspect of the procedure again makes it difficult to know how much certainty to attribute to the final account of the protocol.

3. Safeguards Against Sensory Leakage Between the Experimenter(s) and Judge

In ESP research it is generally recognized that anyone who knows the identity of the target should not have any contact with the judge. This is to minimize the possibility of the judge obtaining information about the target without realizing it through nonverbal cues, inadvertent references to target content, etc. (Milton & Wiseman, 1997).

Lantz et al. (1994) did not document how much contact the experimenter(s) who knew the identities of the targets had with the judge (May). However, May (personal communication, February 22, 1997) informed R. W. that the SAIC personnel involved in the project (i.e., Luke, DeGraff, and Macgowan) were working in offices a few meters away from his office and often had contact with him. In addition, although May had apparently told Utts that he had had no contact with Lantz (Utts, 1995a), he informed R. W. that the two of them occasionally met up (approximately three times a year) when Lantz visited SAIC (personal communication, April 30, 1997). However, May also stressed that the experimenters never discussed ongoing experiments with personnel involved with the project (personal communication, April 29, 1996).

It is difficult to assess whether this agreement not to discuss the study with the judge would have been fully effective as a safeguard. In the present case, an experimenter who knew the target identities would have known which particular images or videotapes were acting as targets particularly often. This knowledge may have biased the experimenter toward referring to target-relevant content in his or her everyday conversation with the judge without either of them realizing it. Psychologists know little about the full range of ways in which such inadvertent cuing can arise and affect the behavior of participants in experiments, but there is no doubt that it happens (Rosenthal & Rubin, 1978).

The possible impact of this potential problem is extremely difficult to gauge. Clearly, no accidental, blatant references to target material (such as "by the way, one of the targets was Niagara Falls") could have been made by an experimenter, otherwise May and the other experimenters would have realized that the protocol had been breached and would have taken appropriate steps. Whether or not there were subtle cues of the type discussed, undetectable by the experimenters, is impossible to assess. The evaluation of this problem is further complicated by the difficulties discussed in the first section involved in establishing which experimenters knew the targets' identities and had contact with the judge, either directly or via intermediaries.

4. Safeguards Against Cues to Past Targets from Responses Judged Out of their Original Order

Problems (discussed below) can arise in open-deck studies with trial-by-trial feedback of the target's identity, such as Experiment One, if the trials are judged out of order. Lantz et al. (1994) did not document whether the trials were judged in the order in which they were carried out, so R. W. asked May for more information. May (personal communication to R. W., April 29, 1996) stated that trials were judged in random order rather than in their original order, about ten at any one time.

Kennedy (1979) has noted that such a procedure could introduce an artifact. His argument is best illustrated by the following scenario. Imagine an experiment of only two trials. In the first trial, the receiver draws a picture and then obtains feedback, discovering that the target was a picture of a waterfall. On trial two, the receiver cannot get the waterfall out of his or her head and draws an exact copy of it. The receiver then finds out that the target for this second trial was a picture of a mountain. If these trials were judged in the order in which they were carried out, the receiver would be likely to score only at chance. However, imagine that the trials were presented to the judge in reverse order. The judge would see the second trial first - the receiver's drawing of the waterfall - and be no more likely than chance to assign the actual target - the mountain - a first place rank. However, when the judge examined the target set from the first trial, the fact of having seen the receiver's drawing of a very similar waterfall might bias the judge to pay more attention to the waterfall target, not realizing that this was the result of cues. (Such lack of realization would be more likely in a real experiment than with the glaring cues of our fictional experiment.) The extra attention paid to the target might be enough to make it more likely to be selected by the judge. Although the trials that May judged were in random, rather than exact reverse order, quite a few trials in the study would have been judged before trials that actually took place earlier. Because of this problem, Kennedy (1979) recommended that whenever receivers are given trial-by-trial feedback, trials should be judged in the order in which they were carried out.

DISCUSSION

Despite many strengths in Experiment One's design, there remain two opportunities in its protocol for sensory leakage, assuming that the third collective recollection of the SAIC team is correct. The impact of both of these problems is hard to assess. No meaningful empirical assessment would be possible, given that the details of the procedures crucial to such assessments are either not retrievable or not replicable, consisting as they do of the content and circumstances of casual conversations held years ago, and the mindset and cognitive style of May as a judge of free-response material. However, the judging problem seems particularly severe in its potential effects, given that the cues themselves would not be sensorially weak. Moreover, the difficulties experienced by the SAIC team in producing a consistent account of the study's procedure raise the question of whether, despite their best efforts of recollection, the final account of the study is reliable. If it is not, then it is possible that there were two more procedural problems than the final two identified.

The inconsistencies between May's initial recollections in answer to queries both from R. W. and from Utts and the third collective memory (which May considers to be more certain) of a number of procedural details in Experiment One raises a wider issue. May initially recalled that Luke had selected the targets, but the third collective recollection of the SAIC team was that DeGraaf, Macgowan, and a student assistant had carried out this task. May recalled at first that he had judged from photocopies of the receivers' original responses, but the third collective SAIC recollection indicated that he had done the judging from the faxes. The third collective memory of events also indicates that May had had some contact with Lantz, the sender, although Utts (1995) relates in her AIR report that May had told her that he had had no contact with Lantz. Utts often had to rely upon May for details of aspects of the SAIC experiments not contained in written documentation (May, 1996); her assessment was conducted only a few months before R. W.'s began. The inconsistencies between May's initial recollections in response to questions and the later account that the SAIC team consider more reliable bring into question whether the AIR assessors were in a position to make an accurate evaluation of the SAIC work as a whole.

Remote viewing studies constitute one of parapsychology's major research areas and have consistently attracted strong media interest. Given the problems identified in Experiment One, we recommend that future research in this area avoid the potential problems of design identified here, and provide more detailed documentation that will allow the full assessment that controversial and important work in the public domain deserves.

1 For the debate about these studies, see Targ and Puthoff (1974, 1977), Marks and Kammann (1980), Marks (1981a,b), Puthoff and Targ (1981) and Morris (1980).

2 Although Dr. May did not object to his email messages on this topic being cited verbatim in an earlier version of this paper presented at the 1997 Convention of the Parapsychological Association, he has asked the authors to paraphrase them here.

3 Lantz et al. (1994) do not apply the continuity correction recommended by Solfvin et al., and our probability estimates are therefore slightly more conservative than those that they report.

REFERENCES

AKERS, C. (1984). Methodological criticisms of parapsychology. In S. Krippner (Ed.), Advances in parapsychological research 4 (pp.112-164). New York: Plenum Press.

HYMAN, R. (1995). Evaluation of the program on anomalous mental phenomena. In M.D. Mumford, A. M. Rose, & D. A. Goslin (Eds.), An evaluation of remote viewing: Research and applications (pp. 3.43-3.78). American Institutes for Research.

KENNEDY, J. E. (1979). More on methodological issues in free-response psi experiments. Journal of the American Society for Psychical Research, 73, 395-401.

LANTZ, N. D., LUKE, W. L., & MAY, E. C. (1994). Target and sender dependencies in anomalous cognition experiments. Journal of Parapsychology, 58, 285-302.

MARKS, D. (1981a). On the review of The Psychology of the Psychic: a reply to Dr. Morris. Journal of the American Society for Psychical Research, 75, 197-203.

MARKS, D. (1981b). Sensory cues invalidate remote viewing experiments. Nature, 292, 177.

MARKS, D., & KAMMANN, R. (1980). The psychology of the psychic. Buffalo, NY: Prometheus Books.

MAY, E. C., & LANTZ, N. D. (1991). Target and sender dependencies in anomalous cognition: Report submitted to the Scientific Oversight Committee. Science Applications International Corporation.

MAY, E. C. (1996). The American Institutes for Research review of the Department of Defense's STARGATE program: A commentary. Journal of Scientific Exploration, 10, 89-107.

MILTON, J., & WISEMAN, R. (1997). Guidelines for extrasensory perception research. Hatfield, UK: University of Hertfordshire Press.

MORRIS, R. L. (1980). Some comments on the assessment of parapsychological studies: a review of The Psychology of the Psychic by D. Marks & R. Kammann, Journal of the American Society for Psychical Research, 74, 425-443.

MORRIS, R. L. (1986). What psi is not: The necessity for experiments. In H. L. Edge, R. L. Morris, J. Palmer, & J. H. Rush (Eds.), Foundations of parapsychology: Exploring the boundaries of human capability (pp. 70-110). Boston: Routledge & Kegan Paul.

MUMFORD, M. D., ROSE, A. M., & GOSLIN, D. A. (1995). An evaluation of remote viewing: Research and applications. American Institutes for Research.

PUTHOFF, H., & TARG, R. (1981). Rebuttal of criticisms of remote viewing experiments, Nature, 292, 388.

ROSENTHAL, R., & RUBIN, D. B. (1978). Interpersonal expectancy effects: The first 345 studies. Behavioral and Brain Sciences, 3, 377-386.

SOLFVIN, G. F., KELLY, E. F., & BURDICK, D. S. (1978). Some new methods of analysis for preferential-ranking data. Journal of the Society for Psychical Research, 72, 93-109.

TARG, R., & PUTHOFF, H. (1974). Information transmission under conditions of sensory shielding. Nature, 252, 602-607.

TARG, R., & PUTHOFF, H. (1977). Mind-reach. New York: Delacorte.

UTTS, J. (1995). An assessment of the evidence for psychic functioning. In M. D. Mumford, A. M. Rose, & D. A. Goslin (Eds.), An evaluation of remote viewing: Research and applications (pp. 3.2-3.42). American Institutes for Research.

WISEMAN, R., & MILTON, J. (1997). Experiment One of the SAIC remote viewing program: A critical re-evaluation. Proceedings of Presented Papers: The Parapsychological Association 40th Annual Convention, pp. 493-507.

Department of Psychology (Wiseman) University of Hertfordshire College Lane Hatfield, AL10 9AB England, UK email: R. Wiseman@herts.ac.uk

and

Department of Psychology (Milton) University of Edinburgy 7 George Square Edinburgh EH8 9JZ Scotland, UK email: 101335.2551@compuserve.com

COPYRIGHT 1998 Parapsychology Press
COPYRIGHT 2004 Gale Group