'Experimental Methodology in NLP Research'

Experimental Methodology in NLP Research

Introduction

Einspruch & Forman (1985) criticised 29 research studies on the basis of 6 methodological errors, concluding that a large amount of psychological research into NLP concepts was invalid because of errors and oversights in experimental design which compromised its scientific credibility. Sharpley (1987) rightly questioned the validity of some of these methodological objection on the grounds that they unnecessarily discounted a large number of potentially valuable results. He maintained that even when factoring in methodological errors, significant results had been attained by the research.

However, an analysis of the literature reviews and the studies to which they refer does reveal some consistent oversights of certain distinctions that are vital for scientific enquiry into NLP to succeed. While these distinctions are already available to well trained NLP graduates (given that they are core patterns presupposed in many applications of NLP), they will be explained in detail here because of their frequent occurrence in experimental research into NLP. The importance of integrating this knowledge into future design methodologies in NLP research cannot be overemphasised.

The most important methodological issues raised by Einspruch & Forman can be grouped into three patterns, all based on the associative nature of the human nervous system. This basic pattern of human functioning provides important insights into the significance of context in experimental studies and in theoretical considerations.

Pattern #1: Humans Are Influenced by Associations

Psychological Underpinnings

The basic psychological understanding of association was famously illustrated by Pavlov (1927) in his classic experiments with ringing bells and salivating dogs, where he trained the dogs by consistently preceded their feeding time with auditory stimuli such as bells and tuning forks. After a training period he then reproduced the stimuli in the absence of the food and found that in itself, the auditory stimuli was enough to make the dogs salivate by virtue of the learned association. Associative learning has been explored in detail by thousands of psychological researchers, but the lessons learned from our in-depth explorations have occasionally been overlooked.

NLP Research

Einspruch & Forman draw attention to one example in particular. Dorn (1983) attempted to determine participants’ PRS by using three different methods of assessment. One method involved participants selecting their preferred predicate from each of 18 sets of three words; one visual, one auditory, one kinaesthetic; assuming that selecting one out of three words would be done on the basis of a preferred representational system, as opposed to having its own specific associations which influence choice of one word over others in the triad.

Pattern #2: Humans Function Within, and Are Influenced by, Context

Psychological Underpinnings

This pattern corresponds to basic experimental design considerations. It has been a long-understood concept that similarity between the test context and acquisition context is an important factor in memory experiments. It is also well known that “elements of the training context (i.e., background cues) may act as conditioned stimuli during a test trial” (Miller & Schachtman, 1985). These observations draw on associative processes such as classical conditioning (e.g. Rescorla & Wagner, 1972). Even at the time of Pavlov and his contemporaries, it was widely recognised and understood that, “despite the experimenter’s best efforts to make the subject attend exclusively to the nominal controlling stimuli, the test context [influences] behaviour through direct associations between it and any reinforcers that [have] previously been presented” (Miller & Schachtman, 1985).

What is clear in the psychological literature is that the confounding and influential effects of context significantly affect processing across the spectrum of human cognition; over such wide ranging topics as learning, memory and recall, language interpretation, problem solving and perception (see Balsam & Tomie, 1985, for detailed reviews of the impact of context in these areas of cognition).

The impact of context is illustrated by a classic example of context-dependent memory; that the processing of memories is heavily influenced by the context within which learning and recall take place. Baddeley and Godden (1975) tested the memory of participants in two different environments: underwater and on land. When words were recalled on land, participants recalled correctly 37% of words learned on land, compared with 23% learned underwater. When words were recalled underwater. participants recalled 24% of words learned on land compared with 32% learned underwater. Endel Tulving (considered by many to be the father of learning and memory experimentation in psychology), in 1983, formalised this very idea with his well-known theory of encoding specificity in learning and recall.

NLP Research

With regard to the research into NLP, Einspruch & Forman (1985) noted correctly that “the representational system in which information will be stored or from which it will be retrieved is highly contextualized (i.e., varies with the situation), and this context will directly influence the system used.” These researchers were particularly perceptive in noting that “context plays an important role in determining the meaning as well as the structure of any communication.” This is well-known in linguistics and psycholinguistics (e.g. the involvement of context in the resolution of syntactic sentence ambiguity, Mitchell, 1994).

Einspruch & Forman criticised the experimental results of Gumm, Walker, & Day (1982) for neglecting to control for context. After interviewing experimental participants to determine their PRS, they were moved to a room surrounded by curtains, where their heads were placed in a restraining device so that eye-movements could be filmed. This severe contextual alteration would certainly be sufficient to abolish any effects of from dominant representational system usage.

“The processing of memories is heavily influenced by the context within which learning and recall take place.”

Gumm et. al. attempted to determine the PRS using 3 different techniques: predicate tracking, eye movement monitoring and self-report. Their finding that “each assessment method was shown to be biased toward revealing a particular representational modality,” and that such a bias “may be the result of the counsellor’s primary employment of a particular method of assessing the client’s PRS,” illustrates a recognition for the influence of context and their lack of control for it in their experiment. People inevitably adjust processing strategies according to both explicit and implicit demands of a presenting task, and such experiments only serve to illustrate this point further.

Pattern #3: Language Creates a Context Within Which People Respond

Current Understanding

The current representational system model proposes that the system a person accesses is heavily influenced by:

the current context and the type of question asked (which together create task demands);
the way a person represents the particular context which is being asked about.

NLP Research

Gumm et. al. measured PRS by “recording the position of the initial eye movement following the end of each question.” An understanding of eye-accessing strategies precludes this method of accessing some type of a stable representational system.

Bandler & Grinder (1979), when discussing eye-accessing cues, draw attention to the possibility that when you ask someone what their mother looks like, they may first access the auditory system (there are many reasons this may happen – a person may talk to their mother more often than they see her, etc.), subsequently check their feelings (that may give them a response which indicates that they are indeed listening to their mother), then access an image of her from memory. Thus there may be, on occasion, a sequence of accessing cues (called a strategy) which take place, which will be different depending upon task demands, type of language used to elicit the strategy and habitual responses, among others.

Gumm et. al. also used a self-report measure, asking participants what they thought their primary representational system was. The results of such investigations are likely to depend in large part on the verbal frame presented by the experimenter. For example, Elich, Thompson, & Miller (1985) told subjects that “personality characteristics would be assessed through the use of imagery.” Subsequently, “subjects were asked to describe the image or sequence of images evoked by the question,” and “subjects were asked to imagine and describe their favourite experience in order to assess spoken predicates.”

The unsurprising results of this study were that:

“Imaging did not occur exclusively in the single modalities suggested by Bandler and Grinder but involved the multi-modal experience of a visual image followed by the image intended by the question.”
“With the auditory and kinaesthetic questions, the most common occurrence was a visual image followed by either the auditory or kinesthetic image.”
“The images evoked by the control questions were visual.”
“Most subjects regarded themselves as visualizers.”
“Most predicates were visual.”

Elich et. al. recognised that “the term imagery may have set an expectation to have visual images and use visual predicates. If so, PRS is heavily influenced by situational variables like language.”

Falzett (1981) had participants read and generate an internal response to 6 questions in order to determine the person’s PRS. Unfortunately the content of these questions are leading enough in terms of sensory predicates to be good candidates for eye-accessing cue elicitation questions.

Two questions were kinaesthetically biased:

I’d like you to think about the last time you were really comfortable.
What was the last thing you touched that you really enjoyed?

One question was auditorily biased:

What was the last song you heard before coming here?

And only three of the six were adequately general to examine strategic preferences without leading in any way:

I’d like you to think of a time when you had accomplished an important goal.
What is the last thing you remember before you came in here?
I’d like you to now think of a pleasant childhood experience.

The following elicitation requests were designed to elicit verbal responses about which confederates could feed back predicates to gain rapport. However, they are structured in such a way that makes them likely to elicit belief strategies (to find out how a person knows something to be true) and hence create task demands which differ from those desired in the experiment (i.e. eliciting PRS predicates):

When you knew someone understood you
When someone loved or cared about you
When you knew someone trusted you

Out of 24 participants, Falzett found “only 3 who were not predominantly kinaesthetic” in their responses to these questions. This may simply indicate that most people in the study had a tendency to use a kinaesthetic component as a significant part of their belief strategies.

Examples of more general experiences which would control for context and potential mental strategies would be:

A boring experience in the same context
A common experience in the same context

These would be less likely to have leading or biasing factors which confound results.

Additionally, Falzett’s finding that matching predicates increased trustworthiness may simply be an artefact of confederate’s usage of predominantly kinaesthetic predicates, which may have imparted the confederate with an air of genuine self-expression and thus, trustworthiness. Without control sets, however, such a hypothesis is impossible to verify.

The observation by Falzett that eye-accessing cues yielded the best results for determining PRS was dangerously generalised by Dorn (1983) to mean that eye-accessing cues are “most conducive to research on NLP and should be employed over the predicate usage method.” This consideration was likely made because of the inconsistencies of research findings surrounding the PRS. However, methodological errors like those above creates a sense of chasing one’s own tail in terms of attempting to make conclusions about experimental findings in NLP research.

Summary

While Sharpley (1987) criticised Einspruch & Forman (1985) for dismissing numerous NLP research papers on the basis of their unfamiliarity with and lack of training in NLP, it is important to recognise that good quality training can avoid certain methodological pitfalls which would have been apparent to those who had undergone adequate training.

However, the ability to understand the issues uncovered above is by no means confined to NLP practitioners, given that they are all well known psychological effects and widely accepted in the psychological research community as being important considerations to take into account during the methodological design phase.

Thus, while psychologists and cognitive scientists undergoing research into NLP concepts do not necessarily have to have undergone NLP training, the additional filters and perspectives of comprehensive NLP training can allow a more coherent explanation of experimental results in terms of the patterns of behaviours occurring within experimental contexts

Given that even the research which had found effects supporting NLP concepts suffered from various methodological confounds, it is not easy to make any generalisations about the validity or use of past research into NLP. What is clear, is the importance of careful, well planned research into NLP in the future, to assist the development of the field of NLP as a whole.

“Careful, well planned research into NLP in the future is important to assist the development of the field.”

References

Balsam, P.D. & Tomie, A. (1985) Context and Learning. Hillsdale, NJ: Lawrence Erlbaum Associates.

Bandler, R., & Grinder, J. (1979). Frogs into princes : neuro linguistic programming. Moab, Utah: Real People Press.

Dorn, F. (1983). Assessing primary representational system (PRS) preference for Neuro-Linguistic Programming (NLP) using three methods. Counselor Education and Supervision Vol 23(2) Dec 1983, 149-156, 23, 149-156.

Einspruch, E. L., & Forman, B. D. (1985). Observations Concerning Research Literature on Neuro-Linguistic Programming. Journal of Counseling Psychology, 32(4), 589-596.

Elich, M., Thompson, R. W., & Miller, L. (1985). Mental imagery as revealed by eye movements and spoken predicates: A test of neurolinguistic programming. Journal of Counseling Psychology, 32(4), 622-625.

Falzett, W. (1981). Matched versus unmatched primary representational systems and their relationship to perceived trustworthiness in a counseling analog. Journal of Counseling Psychology, 28, 305-308.

Godden, D. R., & Baddeley, A. D. (1975). Context-dependent memory in two natural environments: On land and under water. British Journal of Psychology, 66, 325 – 331.

Gumm, W., Walker, M., & Day, H. (1982). Neurolinguistic programming: Method or myth? Journal of Counseling Psychology, 29, 327-330.

Miller, R.R. & Schachtman, T.R. (1985): The Several Roles of Context at the Time of Retrieval. In P.D. Balsam & A. Tomie (Eds.), Context and Learning. Hillsdale, NJ: Lawrence Erlbaum Associates.

Pavlov, I. P. (1927). Conditioned reflexes. London: Routledge and Kegan Paul.

Mitchell, D.C. (1994): Sentence parsing, in Morton Ann Gernsbacher (ed.), Handbook of psycholinguistics, Academic Press

About the Author

Richard Thompson, BSc. (Cognitive Science), is a Graduate of Exeter University, and is a freelance writer and web consultant. He holds the Graduate Certificate in NLP, from INSPIRITIVE, and enjoys receiving responses to his work.

Learn more about NLP by reading our Ultimate Compendium of NLP

If you found this article, please share.