LeBaron, C., & Streeck, J. (2000). Gesture, knowledge, and the world. In McNeill, D. (Ed.), Review of language and gesture: Window into thought and action. Cambridge: University Press.
KNOWLEDGE, AND THE WORLD
D. LeBaron and Jürgen Streeck
THE FORMATION OF GESTURES AND THE FABRICATION OF KNOWLEDGE
Among the essential (but almost forgotten) insights
of the Age of Enlightenment was the recognition that “human understanding”
(Locke 1959 )—the formation and accumulation of common knowledge—is
dependent upon the formation and accumulation of material signs: material
entities that reside, however fleetingly, in the public realm where they may be
reused and thereby shared; representations that embody a world that may be
jointly acted and reflected upon; artifacts that, while products of minds, are
nevertheless external to them, providing tools not only for the mind, but also
for labor and human self-creation; socially shared cognitive tools that evolve
over time as humanity’s mind evolves through relying on and refining them.
With this proposition, anti-Cartesian philosophers such as Condillac for
the first time directed attention to the importance of symbol systems (or media)
for human cognition, self-creation, and society. Condillac in particular
recognized the inherently social character of the human mind, and he suggested
that signs and insights originate in social practice. He wrote:
history of communication will show the various circumstances under which signs
have been formed. It will show us the true meanings of signs and ... leave no
doubt about the origin of our ideas (Condillac 1746, p. 61).
called signs “sensations transformées”, transformed sensations, by which he
meant the entire complex of affect, desire, sensory perception, and motor action
that make up what nowadays we might call “embodied experience”. In his view,
the incarnation of shared experiences in communally usable material signs—and
the social distibution of these signs and the knowledge embodied in them—is
the core of human cultural evolution, of the progress of the “connaissances
The formation of a symbol is a defining moment in the fabrication of
shared knowledge because it allows the participants to focus upon and re-invoke
previously shared experiences and to plan and conduct shared activities in their
wake. An important version of this process is “langage d’action”, the
performance of schematic motor actions that are abstracted from actions in the
material world—in a word, gestures. Gestures, in Condillac’s view,
constituted the original, natural language of humankind.
The method by which Condillac and his contemporaries studied the
fabrication of knowledge through the formation of signs was the “thought
experiment.” Today, we are equipped to investigate it through close analyses
of empirical practice. In this chapter, therefore, we want to describe how
gestural signs are formed and how, in this process, communal knowledge is
incorporated, stored, and organized. Our aim is twofold: to ascertain several of
the roles that hand-gestures play in the formation and distribution of knowledge
within specific “communities of practice” (Lave 1991); and to provide
evidence for the foundations of symbolic gestures in practical, instrumental,
non-symbolic action and experience. Our approach is guided by Condillac’s
vision that the fabrication of knowledge and the formation of signs are not
simply dependent upon one another, but are two aspects of the same process.
Instead of solely locating gesture in the “process of utterance” (Kendon
1980) or deriving it from “mental
representations” (McNeill 1985), we seek to establish its experiential
foundations in activities of the hands in the material world and to explicate
its indexical ties to this world. Gestures, in our view, neither originate in
the speaker’s mind nor in the process of speaking even though speech and
gesture are routinely coordinated. Rather, gestures originate in the tactile
contact that mindful human bodies have with the physical world. In other words,
we propose—and claim that our data demonstrate—that conversational
hand-gestures ascend from ordinary, non-symbolic, exploratory and instrumental
manipulations in the world of matter and things, and that the knowledge that the
human hands acquire (both individually and historically) in these manipulations
is realized through and brought to bear upon the symbolic tasks of gestural
representation. Ultimately, it is through these indexical ties of gestures to
the material world that gestural signifiers can be seen and recognized:
onlookers can see the motion of a configured, but empty hand as (for example) a
“turning”, only when they infer an object that can be turned—no matter
whether the gesture refers to the object, to an act of turning it, or rather to
the vehicle of a metaphor (see McNeill 1985). Without the index to the world of
things, the movement of the hand could not be seen as an action, and the
signifier would be lost in a sea of meaningless motions.
Our investigations have focused on gestural practices in activity-rich
and cognitively-complex settings such as do-it-yourself workshops and
architecture classrooms, but also everyday conversational settings where
practical knowledge is shared. Within settings of material practice, the
participants' hands are not only involved in symbolic actions (as they are in
conversations), but also practical actions with things. Hands are entangled in
the world that they reach--touching objects, grasping tools, wielding
instruments, managing matter. Hands are busy in many ways, shifting back and
forth (sometimes rapidly) between doing things, showing things, and showing how
to do things with things. Such
practical and collaborative settings, we contend, are more
"foundational" and paradigmatic for studying the "communicative
hand" (Bates 1974) than are the purely symbolic realms of conversation or
narrative monologue. Conversation
removed from hands-on interaction with things may efface the natural links
between symbolic actions and the exploratory and instrumental doings that hands
perform. To study gesture as only a feature of conversation is to obscure the
embodied knowledge, the lived experience, that hands bring to their symbolic
Moreover, as we have examined gestures within settings of practice and
instruction, we have witnessed the processes whereby gestures become
conventional, shared, jointly used symbolic forms. Gestures may be regarded as
mediating devices that provide a link between interpsychological and
intrapsychological functioning (Vygotsky 1978); in studying them, one may
therefore (1) look for those physical objects and experiences by which they are
referentially anchored; (2) study how they organize social interaction on the
one hand, and shape individual cognition on the other; and (3) explicate their
social, situated, semantic histories.
By positioning our study outside a conversational framework that
privileges speech, we regard gestures within the contexts of the very
experiences they come to formulate and index. In settings of material practice,
gestures are often contiguous to the experiences they symbolize (and comment
upon, qualify, alter, and so on), and index tactile and visual experiences that
the participants jointly possess from recent interaction, if only because they
jointly witnessed them. Gestures also share physical space with things, most
obviously those things within reach of hands.
The first segments we want to examine are taken from a video-recording of
a do-it-yourself workshop.[i]
While teaching a lesson on the uses of sheet-rock, the instructor introduces a
few tools for the job. He stands at the front of the room where tools and
materials are spread along a countertop. He picks up a tool, labels it (a
"scraper"), and moves it around in mid-air so class members can see
the tool—and see how the teacher handles it. Such behavior is rather
commonplace, a seemingly trivial scene within an ordinary classroom where
teaching is done through "showing," "modeling," or
"mimesis." This first stage of the teacher's lesson on scrapers is
a couple of things you need for preparing sheet-rock.
((picks up scraper))
( - - - - )
((puts scraper in other hand, looks at it))
One of them will be the scraper of some sort.
((scrapes in mid-air with both hands; puts s. down))
This is a uh- very heavy-duty scraper.
((picks up other scraper, holds it up))
Uh, you’d also have a scraper looks like this.
((begins to put it down,
( - - )
((scrapes, puts down))
((picks up third scraper,
A little bit bigger, this is a- both a scraper and a and a tool
((points scraper to wall))
((points scraper to wall))
you use for applying compound to sheet-rock.
Although this classroom moment appears commonplace, it is nevertheless a
paradigmatic instance of symbol-formation, for it includes the situated creation
of a form-meaning pair that embodies a node of locally produced, shared
knowledge. The moment exemplifies the "transformation of sensation"
into a sign. The sign that is established here is a gesture—a
hand-configuration and movement—which signifies a class of object,
The formation of this sign begins as a demonstration of a material
object, an instrument. The instrument is picked up from the table and turned
into an exhibit when it is moved from one hand to the other while simultaneously
receiving the instructor’s concentrated gaze (line 3, Figure 6.1).
The tool’s use is then demonstrated through a schematic motion in a
virtual field of action—that is, the instructor “scrapes”, but performs in
the air (line 5, Figure 6.2).
Thus, during this first sequence (lines 2 - 5), one hand is configured to
hold the scraper, as it would be held if actually used to work upon sheet-rock.
The hand takes on the same configuration when the second scraper—i.e., the
putty-knife—is picked up and its use is demonstrated (lines 6 - 8).
And the same hand-configuration occurs again with the third scraper
(lines 9 - 10). Thus, there is a natural contiguity between the members of a
class of objects (in this case, instruments) and a configuration of the hand.
While this configuration has no symbolic value during the moment above when the
different scrapers are actually being held, the hand configuration becomes
symbolic (a “transformed sensation”) subsequently, when the instruments are
implied but not literally hand-held.
During a subsequent moment, transcribed below, the instructor’s hands
move without holding a scraper (or any other instrument), but they remain
configured in a fashion that can now, in this context, be recognized as an index
of scrapers, at least by those who have witnessed the prior scene: the
hand-shape makes sense only vis-a-vis the instruments that this hand has
previously held, and it simultaneously conjures up the image of these
instruments. While the tools are physically absent from the instructor’s
demonstration, their “virtual” or symbolic-cognitive presence can be jointly
inferred by all recipients of this communicative act. The hand-shape turns into
a socially shared symbol.
Having put the scrapers down (line 13, segment 1), the teacher now raises
an empty hand that is shaped "as if" holding a scraper (line 1 below).
The hand moves in mid-air and thereby initiates a more complex
demonstration: an absent scraper is used to distribute invisible mortar on a
non-existing surface so as to make virtual piles of the compound, as described
simultaneously (albeit vaguely) through speech (Figure 6.3). While the hand
moves with precision, the verbal instructions are quite inexplicit. Without
touching the tools that remain visible on the countertop, the instructor’s
hands move to make the scrapers "present" through an indexical
So that’s the idea behind those ( - - - )
(( scrapes ‘compound’
No matter how much you put in
if you scrape it
(( scrapes ‘compound’
and scrape it off
Here, the instructor demonstrates a complex, skilled activity.
In a sequence aimed at demonstrating the importance of applying the right
amount of compound to sheet-rock, he shows how excess compound can be scooped up
with and removed from the scraper. The instructor’s movements are swift and
precise, and for those who have some visual familiarity with this line of
construction work, it is easy to “see” not only the scraper, but also the
compound, the surface (i.e., the sheet-rock), as well as another, unnamed tool
which is used to remove the compound from the scraper.
We “see” how compound is
scraped up to make a pile which is then lifted off the sheet-rock and scraped to
the other tool.
Thus, from the teacher's initial handling of the scraper, a pattern of
movement is abstracted—that is, a gesture. Performed publicly, it constitutes
a communal sign, a convention, in which a shared experiential nexus or bit of
knowledge about the world is embodied. The gesture evolves as a situationally-transparent,
symbolic construction through which practical knowledge may be handed about.
It emerges in two stages of what might be called an "indexicalization
schema": originating in the hands-on manipulation of the material world
within reach, the abstracted gesture retains an indexical link to it, which can
be used in both directions—the gesture presupposes the material world for its
intelligibility, but can also and by the same token evoke it. The sign is now
available to invoke a nexus of practices, things, and relations between them,
and is potentially applicable to infinite communicative purposes, syntactic
contexts, and semantic roles. By simply raising a hand with a recognizable
shape, a complex of actions, objects, instruments, and so on may be denoted by
the teacher or other participants.
The sign is also available to be folded back upon itself, for the
layering-on of further in-formation, for example, to denote manners, possible
mistakes, or more elaborate lines of action. In this fashion, the shared
knowledge of the community can grow via the invention, re-use, and
trans-formation of an ephemeral, but nevertheless material, sign. A pattern of
muscular movement has been abstracted from the hands' entanglements with
concrete worlds and can be used in other contexts, including strictly symbolic
or representational ones such as conversation. But to function in these
contexts, audiences must be able to "fill in" indexical background: in
these episodes, recipients of the instructions initially had to “see” a
counter-top as a symbolic representation of sheet-rock and, subsequently—on
the basis of locally produced knowledge—recognize that a hand was configured
in a certain way because it represented the holding of a scraper. (Compound also
had to be filled in.)
Our focus now shifts from the do-it-yourself workshop where hands engage
in (symbolic) instrumental actions from which gestures are then derived, to a
university classroom where a professor’s hands manipulate a material, spatial
object in an exploratory fashion. The professor’s movements made in this
process are subsequently abstracted as gestures that do not represent actions or
instruments, but rather features of the object explored. While the signifier is
a motion, the signified is a fixed, immutable structure in space.
As part of his undergraduate course on architectural design, the
professor critiques miniature buildings made by students using cardboard and
glue. The students sit arranged in a large circle, all oriented toward the
middle of the classroom where the professor sits on an elevated stool, next to a
large table. One by one, the students step forward and place their cardboard
models on the table, making them available for others to see, and available for
the professor's critique. Here, we
focus on one cardboard model: introduced by the student as a "tunnel
shape," it is subsequently described as a "sewer-like" culvert by
the professor, who first explores it, then interprets it, and thereby critiques
Before talking about the model, the professor silently explores its
three-dimensional features. He leans forward and over the model to look inside;
he moves his body to see the sides; then he touches it, lifts it, and slowly
turns it in mid-air, observing it from various angles (Figure 6.5) and at the
same time feeling with his fingers the architectural shapes created by the
student's hands. The professor's exploration might be called a
"primary" stage of knowledge formation--and hence a
"necessary" stage of symbolic action. As he encounters the cardboard
model for the first time, his hands become entangled with it. He has a series of
visual and tactile experiences, made possible by his practical actions upon the
object that he begins to grasp. He becomes knowledgeable regarding the cardboard
model, through the very experiences that his subsequent gestures will formulate
and index. At the same time, the professor's lived experience is a shared
experience: as he explores, he also shows. By turning the model in mid-air to
observe its various angles, he enables his students to do likewise; and he
directs their attention toward the model through his orientation, gaze, and
extended fingers, which all function like pointing gestures (Scheflen 1976).
Moreover, the professor's exploration may be regarded as a demonstration.
By sitting in the middle of the room, at the center of the group, he makes his
own experience an object of attention, a public performance, a potentially
vicarious experience. His behaviors are both private and self-conscious,
appearing as practical precursors of the group's symbolic tasks.
His solo actions are a form of social practice as he interacts with a
material world—something the instructor and scraper only implied.
In short, his hands mediate between thing and thought.
Eventually, the professor begins talking about the cardboard model as his
hands continue to move in relation to it. His exploring hands are also pointing
fingers that direct the students' attention toward specific spatial features,
thereby highlighting shapes of the model being discussed. For instance, the
professor slowly slides an extended index finger along one edge, finding and
highlighting its curved shape (Figure 6.6), creating a figure-ground separation
that informs students how to see the model. At the same time, the professor may
be teaching students how to see his hand—his behavior may serve to highlight
the movement of his hand, not just the shape of the model.
With increasing frequency over a ten-minute period, the professor's hands
move without actually touching the cardboard model. That is, his hands perform
shapes in the air that are physically separated from the material object that
they index. The following transcription represents such a moment.
... you have (.) uh- uh long bent
sort of uh- uh linear experience
touching the cardboard model, the professor describes it. He refers to its
"long bent" form (line 1) and highlights the same by touching the
model with his extended index finger (Figure 6.6). His speech is organized
(i.e., he says "long" before he says "bent") according to
the shapes that his finger encounters as it slides upon a long, straight edge
before moving around a bend. Immediately
after highlighting the model's "long bent" shape, the professor
reproduces this shape in the air--mere inches above the cardboard, but
nevertheless separated from it (Figure 6.7). This is a defining moment. The
mid-air motion is recognizable as a gesture, because its shape is performed
apart from the tangible influence of its referential anchor. The gesture emerges
as a natural extension and an incipient feature of practical actions upon an
object. The new convention is shared, understood by those participating (perhaps
vicariously) in the hands-on activity. Moreover, the new symbol heralds
knowledge-formation that the speech also marks: as touch moves to gesture,
concrete talk turns abstract; the words "long bent" (line 1) describe
tactile and visible features that are quickly recast as a "linear
experience" (line 2), and the professor begins to discuss the full-body
consequences of the hand-size shape. In sum, the emerging gesture shows a close
connection with a material object, which serves as a springboard into
interaction about imagined experience--altogether a sort of transformed
Continuing for several minutes, the professor's hand gestures evolve as
his critique unfolds. Sometimes,
his movements are relatively small: with a flick of his wrist, he outlines a
"long bent" shape in the air, approximately the size of the
architectural model (hand-size) and located only inches above it (see figure
6.7). Other times his movements are
big: “long bent” shapes
performed at eye level, where his whole arm extends, separated from the model by
feet rather than inches (Figure 6.8). The following transcription represents
such a moment.
((large gesture begins))
can see the wall at the back
it's lit it goes (- - ) way
((large gesture ends))
(.) it disappears around the
professor's descriptions become vivid as he expounds upon the full-body
consequences of the hand-size model. When
he talks about seeing “the wall at the back” (line 1), he extends his right
arm fully so that his flat palm may be regarded as a distant surface (similar to
Figure 6.8). As he describes what
“you can see” (line 1), he draws his gesturing hand toward his eye and
lightly touches the side of his face. Then
his hand moves outward from his eye and straight along his line of vision, which
“goes” (line 2) parallel with the imaginary passage.
As he says "back" (line 2), his arm becomes fully extended
again, and then his hand hooks to the left (Figure 6.8) when he says
"disappears around the corner" (line 3). Altogether, the professor
outlines a "long bent" shape, created in mid-air but referentially
anchored in the model on the table below. At the same time, he performs a
"lived" experience: gazing off "into space," he talks
himself along the hallway that he imagines and motions, using present-tense
language to describe architectural details as they spatially and temporally
unfold. Because his performance is public, the professor provides--at least
potentially--a vicarious experience to seated students, as suggested by the
pronoun "you" (line 1).
Group knowledge forms as "long bent" gestures occur and evolve
to embody the unfolding shape of the professor's critique. Shared understanding is especially evident when students
participate in the class discussion and also perform "long bent"
gestures to articulate their views, which build upon the group's understanding.
For instance, consider the following excerpt. Immediately after the
professor ends his comments and invites other opinions, a student performs a
large gesture while giving a favorable critique of the architectural model:
kno::w ( - - ) where to go:
not- you're not (.) wandering
around ( - - - ) and (.) you ha:ve
(.) u:h dire:ction to it
articulate his view, the student uses behaviors that may be recognized by other
classroom participants. With the
utterance "u:h" (line 4), he lifts his right hand to his right eye;
while describing the "dire:ction" (line 4) of the architectural
structure, his hand moves outward from his eye and straight along his line of
vision; and once his arm is fully extended, his hand hooks to the left (Figure
6.9). Altogether, the student outlines a "long bent" shape in mid-air
that is referentially anchored in the model on the table--largely through the
professor's prior public performance. Moreover,
the student's gesture occurs as he expounds upon the full-body consequences of
the hand-size model. Like the professor, the student uses present-tense language
to describe an imaginary "lived" experience. The student's behavior is
a potentially vicarious experience for others in the classroom, as suggested by
the pronoun "you" (lines 1, 2, and 3).
The architectural model is different than the do-it-yourself scraper, for
reasons that have consequences for the formation and recognition of gestures. On
the one hand, the scraper is lifted and used.
It is treated as a tool, an instrument to be employed, an extension of
the human hand that subjects implied objects to performed actions. On the other
hand, the model is lifted and explored. It
is regarded as an object, an entity with properties to be discovered, a thing
understood through the human actions that its shape implies and guides. Although
material things may embody instructions for how they are to be used or regarded
(Norman 1988), differences between the scraper and model are instantiated
through behaviors during the "primary" stage of knowledge formation.
Wielded again and again, the scraper is never mistaken for a miniature building.
Although cardboard is capable of scraping mortar, the professor initially and
immediately moves his body relative to the architectural model, projecting
fixedness upon it. As a consequence of primary behaviors, gestures eventually
emerge imbued with verb-like and noun-like attributes. Within the workshop, the
teacher's gestures are verb-like, with the implied scraper in the role of
instrument: his hand movements are recognizable as hand actions--a rather
low-level abstraction. Within the classroom, by contrast, the professor's
gestures become large and semantically complex: his hand movements are noun-like
as they outline (in the air) three-dimensional features discovered in the model;
these same hand movements are at the same time verb-like as they represent a
person's movement through the imaginary building being outlined--a rather
high-level abstraction. Moreover, the professor's gestures serve as a heuristic
device, enabling him to translate the hand-sized model into a full-size
architectural experience that becomes grounds for his critique. Furthermore, his
gestures serve as a teaching device, helping students to read the cardboard
shape as a representation of an embodied experience, vicariously lived and
Our analysis of these episodes has demonstrated at least three things.
First, the interactions within the classrooms exemplify the step-by-step
processes by which embodied—manual—action in the world of matter and things
may be transformed into symbolic action. That is, activities within and upon the
material world may be abstracted, schematized, and converted into components of
the shared communicative repertoire of a local “community of practice”. In
short, our analysis has traced the experiential, practical roots of individual
gestures. Second, our analysis shows how the shared knowledge of these
communities grows through the formation of these gestures. Like all signs, the
gestures comprise cognitive and communicative features and functions: they do
not only represent or express, they constitute
socially shared knowledge. The further growth of knowledge in these
communities is dependent upon and made possible by the shared possession of
newly formed signs, which can then be modified and elaborated
so as to represent proper modes of action, possible mistakes, compound
activities, and so on. Finally, the episodes demonstrate that the proper seeing
and understanding of these signs requires the material world as an indexical
background: the configurations and motions of the hands only make sense by
virtue of their contiguity to things, and it is by reference to these things
that the motions of the hands can be recognized as schematic actions (or
manipulations). In short, seeing a gesture requires knowledge of the world.
BACKGROUND KNOWLEDGE AND THE PERCEPTION OF GESTURES
Within the instructional situations we have examined, such knowledge is
available from the course of recent interaction; the publicly visible, material
experiences and activities of the hands from which the gestures have been
derived is part of the local, shared memory of all participants. Some might
object, therefore, that these incidents are uncharacteristic of gestures as they
are used in the exclusively symbolic realm of everyday conversation,
“conversation pure”, which is more often than not predicated upon the
absence of the material events and states-of-affairs that constitute its topics.
Moreover, it might be objected, symbolic communication in the absence of things
constitutes the more astonishing and more important human achievement.
However, it is our contention that gesture—certainly descriptive or
“iconic” gesture—necessarily involves indexical links to the material
world, even though these links are rarely established or explicated in the
communicative situation itself. Rather, in conversational contexts that are
detached from the talked-about world, participants must fill in encyclopedic
knowledge (ranging from universal bodily experiences to highly specific cultural
practices) to see and recognize gestures. Phenomenally, there are only motions
of the hands. What is perceived, however, are typically not motions but actions
and, simultaneously, implied objects acted upon. Thus, we do not see two flat
hands moving apart, but we see them dividing a substance (or a substance being
divided)—which, in the given conversational context, may signify a group of
friends, dough, or the mélange of issues that will have to be discussed. To see
the hands engaged in a schematic act of dividing, and to “see” a something
that is being divided, our eyes must be intelligent, experienced in the ways of
the world of the hands. Without chipping in our “beholder’s share” (Gombrich
1961), we could not see the gesture, that is, the signifier—let alone identify
We briefly illustrate these indexical underpinnings of iconic gestures by
examining two excerpts from a series of conversational stories about car
accidents. In the videotaped recording (which was made in Germany), two Japanese
friends, having discussed the difficulties of obtaining a German driver’s
license, proceed to tell one another about their involvement in various
accidents that resulted from the drivers’ inattention to the road. As is
common in such narratives, the narrators’
bodies variously re-enact the protagonist’s actions while driving the
car, shift to render the behavior
of the cars on the road, or move to represent the setting.
More specifically, their hands are used in semiotically different ways to
represent different “players” in the event, enabling the two interlocutors
to speak from constantly shifting perspectives: at times in “the first
person”, they re-enact the acts of the protagonist’s hands; at other times
in “the third person”, their hands serve as symbolic tokens for the moving
and crashing cars; at yet other times, their hands are extraneous producers of
symbolic constructs—sequentially rendered, ephemeral, three-dimensional
shapes—which represent components of the setting. In the following, these
components, in particular, deserve our attention.
The first of the two fragments is a narrative segment told “in the
first person”. The speaker, Satomi, describes how she absent-mindedly drove
along a straight road when she realized that she had to turn to the right. At
this point in the story, she puts down the tea-cup that she has been holding in
her hands, readies her hands by moving them to her stomach, and says:
Nde ne douiu wakeka sugoi:
And I don’t know why but
((two hands hold and turn
ko kirrisugitan da yo ne h.
turned the steering wheel too much, like this.
with the lexical description of her turning of the steering wheel, Satomi makes
a large, two-handed gesture which international audiences unfailingly recognize
as an enactment of car-driving: holding her two hands in a “grip shape”
parallel to her chest, she moves one diagonally up and and the other one
diagonally down and thereby shows the turning of a steering wheel (Figures
10-11). The gestural portrayal is consistent with the lexical description, and
the gesture appears to add little to the narrative, except perhaps visual
precision (i.e., how far she turned the wheel).
And yet we may ask how it is that we can so easily identify the
gesture—and the action that the gesture enotes as well as the object that is
involved in the action—all of this despite the fact that we do not understand
the language. In fact, it is this gesture (which is performed several times
during this narrative) that enables people who do not understand Japanese to
“see” a story about driving cars and accidents. Here is the simple yet
significant explanation: we recognize the gesture to the extent that we share
the material culture that the speaker is drawing upon. Throughout the world, as
in Japan, there are few objects (if any) that are routinely handled—held and
turned—in the fashion of a steering wheel, other than the steering wheel
itself: the gesture therefore can only be about driving a car. We recognize the
action that is abstractly performed by the configured motions of the configured
hands because we are culturally familiar with the material world where such an
action could “really” be performed. Our perception of the gesture as a
schematic action requires our beholder’s share, that we “fill in” generic
objects to which the motions of the hands may relate.
In the “steering wheel” instance above, our encyclopedic knowledge of
the material world—what kinds of things are handled in the fashion seeable in
the gesture?—enables us to recognize the signified. In other cases, world
knowledge enables us to see the gestural signifier—the seeing of a motion as
an action—even when this action has no correspondence in the event that is
signified. For example, through a version of the method of “documentary
interpretation” (Mannheim 1959) we manage to see that, for example, the motion
pattern of the hands could relate to a specifically shaped object—that they
are virtually holding a bowl (see Scheflen 1974). Thus, we “see” a
bowl—and disregard the “holding” (which has no correspondence in the
event). The schematic act of holding—the gestural signifier—would be an
exclusively descriptive or pictorial, not a referential, device. Nevertheless,
to perceive it, we must know about basic embodied acts and their generic
This applies to the following segment in which Satomi’s friend and
conversational partner, Tomoio, also shows a round object that can be turned;
she shows it by performing a one-handed gesture. But this is an object that
could not be manipulated using the action-pattern from which the gesture is
formed; the motion here serves descriptive purposes, and no action corresponds
to it in the event that is reported. The object shown by the gesture is a
worn-out tire skidding on a slippery road.
h de kou hashitteta no (.) ‘hh (shitara) sa ame ga furi hajimete
And we were driving like this, then it
sono kuruma ga sa: ‘hh akseru funde sa
The driver stepped on the gas and
((one hand “turning a
subetta yo taiya ga bo:zu datta no
The tires were worn out.
During her utterance at line 7.4, Tomoio opens her right hand wide and
rotates it a few times back and forth so that her wrist provides an axis (Figure
6.12). She performs a schematic
manual action to evoke a particular kind of object—an object that is round and
can be rotated around an axis. To see the gesture this way, onlookers must
possess and fill in basic experiential knowledge about relationships between
acts of turning, round shapes, and rotating things. Using this knowledge, they
can “see” in the speaker’s hand a round object and recognize the hand’s
rotation as a description of the object’s behavior, skidding, rather than as a
schematic action. The signified (the car’s tire), of course, could not in
reality be turned in this fashion by a single human hand, because it is so much
bigger. But this is a matter of contextual specification: what matters here is
that the object was round and that it rotated, not what size it had. To use
grammatical terminology: while Satomi’s gesture would be understood as having
the grammatical shape of a transitive verb plus nominal undergoer (steering a
steering wheel), Tomoio’s has that of a nominal subject modified by an
intransitive verb (a tire skidding). This distinction is not inherent in the
gestures’ shapes, but rather is a seeing that is achieved in a context and
informed by knowledge about the material world. We must know that there are
things that can be turned back and forth, and that they may be round and
rotating around an axis. Tires are objects that have these features. Thus, by
moving from the schematic act of the hand to the object and back to the hand,
onlookers know to abstract from the hand’s action: here, it is only a
component of the signifier, not of the signified; there is no hand nor human
agent in the moment described, just a worn-out tire skidding on a slippery road.
In both of these segments, the gestures achieve their pictorial effects
through a presupposed indexical link between the hand and the world of things.
In the first case, we “see” the object that the protagonist’s hands
actually manipulated in the event; the gesture is a virtual action which
represents an instrumental action in the depicted scene. In the second example,
the gestural motion enables us to “see” a kind of object: one that is round
and rotating. This narrative hand has no correspondence in the event; there is
no hand that holds and turns a tire (nor could there be).
Iconic gestures of this kind have been described throughout the
literature on gesture (cf. Calbris 1990; McNeill 1992; Wundt 1975 (1911), among
others), but little attention has been devoted to what they tell us about the
roots of gesture in non-symbolic action nor to the role that these experiential
roots—and, thus, our experiential, embodied knowledge about the material world
and its manipulations —play in our ability to see and understand gestures. Our
suggestion here is that an essential component of the pictorial language of
gesture—of the logic by which “iconic” (or descriptive) gestures achieve
their communicative effects—are the tactile, indexical ties that the hands and
their motions have and retain to the material world that they can reach,
explore, know about, and act upon. Before we learn how to gesture, we learn how
to handle things. This knowledge is incorporated in our hands; we use it and we
appeal to our interlocutor’s possession of it when we represent the world
gesturally and invite our interlocutors to understand our gestures accordingly.
Gestural communication is mediated by shared experiential knowledge about the
Our aim in this chapter has been to examine processes of symbol formation
(Werner & Kaplan 1963) and use within communities of practice and to regard
these processes as native to the performance and recognition of gestural
representations within conversations abstracted from hands-on interaction with
things. Initially, we have investigated the orchestration of linguistic and
embodied communication within “material-rich” learning settings such as
do-it-yourself and architecture-classrooms and suggested that the evolution of
bodies of knowledge is a communal process of forming symbols that embody
experiences that have emerged in situated action. The contiguity of gesture and
experience in these contexts enables us to examine the indexicalization
practices (a term suggested to us by John Gumperz) through which gestures are
made meaningful and gradually become "independent" meaningful symbols
within a (however limited) community of practice. At the same time, we have
displaced gesture, specifically iconic gesture, from the bipolar logical space
in which it is almost exclusively situated and studied when it is examined in
the context of abstract narrative or conversation. Our studies therefore also
reveal something about the ways in which generally representational gestures emerge and are understood. Within
“pure” conversations, gesture is typically assumed to correspond either to a
mental image or to objects, actions, or events “out there” (beyond the
immediate setting) with which it is alleged to share similarities of form.
Neither mental image nor referent are available in the documented situation.
However, in practical settings where gestures share the situation with the
experiences that they formulate, we can begin to see the important indexical
underpinnings of the pairings of form and meaning, of signifier and signified:
here, these underpinnings are the situated histories of practical and symbolic
action in which tactile, practical acts—manipulations—are gradually
abstracted into “free-standing” symbols which can then signify, not only
acts, but also manners and mistakes as well as objects acted upon and their
features. We believe that these gestures' socially-situated emergence may
reflect, generally, the way humans interactively constitute and use gestural
The experiential grounding of conversational gestures is typically not
available to interlocutors in the situation at hand, but it is presupposed.
These gestures have lost the immediate connection to the material experiences
from which they may have derived, and circulate in social life as purely
symbolic, gestural forms. They appear self-contained because their situated
emergence and existence extends beyond the situated humans who use them. But
many of the gestural shapes that people recognize in talk-in-interaction may
have originally been devised in some prior situation of the kind that we have
described above and from which they were abstracted and passed on. And competent
members of human societies are capable of recognizing these links between
gestures and objects because they are familiar with the basic actions of which
gestures are abstract versions, as well as with the world that affords them.
Moreover, the speaker’s hands know how to do things other than gesticulation,
and it seems unlikely that the skills that the hands bring to bear on their
symbolic tasks are entirely separate from those that they have acquired while
handling things. Rather, the patterns that are at hand when there is a need to
gesture appear to be made from the same fabric as those that are required in
instrumental action. And this “producer’s knowledge”, too, is socially and
In the picture that emerges from these studies, then, gesture is not a symptom of mental events, and it does not derive its communicative potential from a hypothetical relationship to images on the speaker’s mind. Rather, it is an embodied and therefore public symbolic practice, kinesthetically known by its makers, visually known by its beholders, and derived from and embedded in an objective world within which mindful hands operate. Gestures do not represent by virtue of any similarity to their denotata. Rather, they are abstracted from—and are interpreted vis-a-vis the backdrop of—the realm of material action. Hands learn how to handle things before they learn how to gesticulate. They are knowing signifiers and bring embodied, schematic knowledge, acquired through physical action, to the diverse symbolic tasks of gesticulation. Gestures, designed as they always are for particular recipients, appeal to those recipients’ knowledge, knowledge that may have been acquired over the course of the current situation or in a cultural and physical world that is in some part the shared property of the members of a single society or cultural group, and in other parts common to all human beings.
J. A. V. 1974. The communicative hand. In Benthall and Polhemus (Eds.), pp.
J. and Polhemus, T.
(Eds.) 1974 The
Body as a Medium of Expression London: Allan Lane.
G. 1990. The Semiotics of French Gestures.
Bloomington: Indiana University Press.
E. 1746. An essay on the origin of human knowledge, being a supplement to Mr.
Locke's essay on the human understanding. London: J. Noursse.
E. H. 1961. Art and Illusion. A Study in the Psychology of Pictorial Representation
(2 ed.). Princeton N.J.: Princeton University Press.
, C. and Duranti, A. (eds.) 1992 Rethinking
Context: Language as an Interactive Phenomenon. Cambridge: Cambridge
W. F. 1992. ‘The indexical ground of deictic reference.’ In Goodwin and
Duranti (eds.), pp. 43-76.
A. 1980. ‘Gesticulation and speech: Two aspects of the process of
utterance.’ In Key, M.R. (ed.),
M.R (ed.) 1980.The
Relationship of Verbal and Nonverbal Communication. The Hague: Mouton.
J. 1991. ‘Situating learning in communities of practice.’
In Resnick, Levine, and Teasley (eds.), pp. 63-83.
J. 1959 (1690). An Essay concerning Human Understanding. New York: Dover.
K. 1959. Essays on the Sociology of Knowledge. London: Routledge and Kegan
D. 1985. ‘So you think gestures are nonverbal?’
Psychological Review 92: 350-371.
D. 1992. Hand and Mind. What Gestures
Reveal about Thought. Chicago: The University of Chicago Press.
D. 1988. The Design of Everyday Things.
New York: Doubleday.
L. B., Levine, J. M. and Teasley, S. D. (eds.) 1991.
Perspectives on Socially Shared Cognition. Washington, D.C.: American
A. 1974. How Behavior Means. Garden City: Anchor Press.
A. 1976. Human
Territories. Englewood Cliffs, NJ: Prentice-Hall.
H. and Kaplan, B. (1963). Symbol Formation.
New York: Wiley.
W. 1975 (1911). Völkerpsychologie. Eine
Untersuchung der Entwicklungsgesetze von Sprache, Mythus und Sitte. Aalen:
We wish to thank Cam Tew for allowing us to make use of this material.
We use the following transcript conventions:
colons indicate a sound-stretch (vowel-elongation)
a dash marks a ‘cut-off’, generally a glottal stop
a dot in parenthesis indicates a ‘micro-pause’ (less than one
- - ) dashes in parenthesis
indicate a pause; each dash represents approximately
one tenth of a second
[ square brackets indicate the simultaneity of two events, either two or more
utterances or an utterance and an
action or gesture
brackets mark the description of an action
‘ ‘ single quotation marks within the description of an action signify a “virtual”
component of the action, for
example, an object that is “implied”.
Go to top of this page
Return to Gerry Stahl's Home Page
Send email to Gerry.Stahl@drexel.edu
This page last modified on August 01, 2003