Gestures, Knowlege and the World

LeBaron, C., & Streeck, J. (2000). Gesture, knowledge, and the world. In McNeill, D. (Ed.), Review of language and gesture: Window into thought and action. Cambridge: University Press.

Among the essential (but almost forgotten) insights of the Age of Enlightenment was the recognition that “human understanding” (Locke 1959 [1690])—the formation and accumulation of common knowledge—is dependent upon the formation and accumulation of material signs: material entities that reside, however fleetingly, in the public realm where they may be reused and thereby shared; representations that embody a world that may be jointly acted and reflected upon; artifacts that, while products of minds, are nevertheless external to them, providing tools not only for the mind, but also for labor and human self-creation; socially shared cognitive tools that evolve over time as humanity’s mind evolves through relying on and refining them.

With this proposition, anti-Cartesian philosophers such as Condillac for the first time directed attention to the importance of symbol systems (or media) for human cognition, self-creation, and society. Condillac in particular recognized the inherently social character of the human mind, and he suggested that signs and insights originate in social practice. He wrote:

(The) history of communication will show the various circumstances under which signs have been formed. It will show us the true meanings of signs and ... leave no doubt about the origin of our ideas (Condillac 1746, p. 61).

Condillac called signs “sensations transformées”, transformed sensations, by which he meant the entire complex of affect, desire, sensory perception, and motor action that make up what nowadays we might call “embodied experience”. In his view, the incarnation of shared experiences in communally usable material signs—and the social distibution of these signs and the knowledge embodied in them—is the core of human cultural evolution, of the progress of the “connaissances humaines”.

The formation of a symbol is a defining moment in the fabrication of shared knowledge because it allows the participants to focus upon and re-invoke previously shared experiences and to plan and conduct shared activities in their wake. An important version of this process is “langage d’action”, the performance of schematic motor actions that are abstracted from actions in the material world—in a word, gestures. Gestures, in Condillac’s view, constituted the original, natural language of humankind.

The method by which Condillac and his contemporaries studied the fabrication of knowledge through the formation of signs was the “thought experiment.” Today, we are equipped to investigate it through close analyses of empirical practice. In this chapter, therefore, we want to describe how gestural signs are formed and how, in this process, communal knowledge is incorporated, stored, and organized. Our aim is twofold: to ascertain several of the roles that hand-gestures play in the formation and distribution of knowledge within specific “communities of practice” (Lave 1991); and to provide evidence for the foundations of symbolic gestures in practical, instrumental, non-symbolic action and experience. Our approach is guided by Condillac’s vision that the fabrication of knowledge and the formation of signs are not simply dependent upon one another, but are two aspects of the same process.

Instead of solely locating gesture in the “process of utterance” (Kendon 1980) or deriving it from “mental representations” (McNeill 1985), we seek to establish its experiential foundations in activities of the hands in the material world and to explicate its indexical ties to this world. Gestures, in our view, neither originate in the speaker’s mind nor in the process of speaking even though speech and gesture are routinely coordinated. Rather, gestures originate in the tactile contact that mindful human bodies have with the physical world. In other words, we propose—and claim that our data demonstrate—that conversational hand-gestures ascend from ordinary, non-symbolic, exploratory and instrumental manipulations in the world of matter and things, and that the knowledge that the human hands acquire (both individually and historically) in these manipulations is realized through and brought to bear upon the symbolic tasks of gestural representation. Ultimately, it is through these indexical ties of gestures to the material world that gestural signifiers can be seen and recognized: onlookers can see the motion of a configured, but empty hand as (for example) a “turning”, only when they infer an object that can be turned—no matter whether the gesture refers to the object, to an act of turning it, or rather to the vehicle of a metaphor (see McNeill 1985). Without the index to the world of things, the movement of the hand could not be seen as an action, and the signifier would be lost in a sea of meaningless motions.

Our investigations have focused on gestural practices in activity-rich and cognitively-complex settings such as do-it-yourself workshops and architecture classrooms, but also everyday conversational settings where practical knowledge is shared. Within settings of material practice, the participants' hands are not only involved in symbolic actions (as they are in conversations), but also practical actions with things. Hands are entangled in the world that they reach--touching objects, grasping tools, wielding instruments, managing matter. Hands are busy in many ways, shifting back and forth (sometimes rapidly) between doing things, showing things, and showing how to do things with things. Such practical and collaborative settings, we contend, are more "foundational" and paradigmatic for studying the "communicative hand" (Bates 1974) than are the purely symbolic realms of conversation or narrative monologue. Conversation removed from hands-on interaction with things may efface the natural links between symbolic actions and the exploratory and instrumental doings that hands perform. To study gesture as only a feature of conversation is to obscure the embodied knowledge, the lived experience, that hands bring to their symbolic tasks.

Moreover, as we have examined gestures within settings of practice and instruction, we have witnessed the processes whereby gestures become conventional, shared, jointly used symbolic forms. Gestures may be regarded as mediating devices that provide a link between interpsychological and intrapsychological functioning (Vygotsky 1978); in studying them, one may therefore (1) look for those physical objects and experiences by which they are referentially anchored; (2) study how they organize social interaction on the one hand, and shape individual cognition on the other; and (3) explicate their social, situated, semantic histories.

By positioning our study outside a conversational framework that privileges speech, we regard gestures within the contexts of the very experiences they come to formulate and index. In settings of material practice, gestures are often contiguous to the experiences they symbolize (and comment upon, qualify, alter, and so on), and index tactile and visual experiences that the participants jointly possess from recent interaction, if only because they jointly witnessed them. Gestures also share physical space with things, most obviously those things within reach of hands.

The first segments we want to examine are taken from a video-recording of a do-it-yourself workshop.[i] While teaching a lesson on the uses of sheet-rock, the instructor introduces a few tools for the job. He stands at the front of the room where tools and materials are spread along a countertop. He picks up a tool, labels it (a "scraper"), and moves it around in mid-air so class members can see the tool—and see how the teacher handles it. Such behavior is rather commonplace, a seemingly trivial scene within an ordinary classroom where teaching is done through "showing," "modeling," or "mimesis." This first stage of the teacher's lesson on scrapers is transcribed[ii] as follows.

Although this classroom moment appears commonplace, it is nevertheless a paradigmatic instance of symbol-formation, for it includes the situated creation of a form-meaning pair that embodies a node of locally produced, shared knowledge. The moment exemplifies the "transformation of sensation" into a sign. The sign that is established here is a gesture—a hand-configuration and movement—which signifies a class of object, “scraper”.

The formation of this sign begins as a demonstration of a material object, an instrument. The instrument is picked up from the table and turned into an exhibit when it is moved from one hand to the other while simultaneously receiving the instructor’s concentrated gaze (line 3, Figure 6.1). The tool’s use is then demonstrated through a schematic motion in a virtual field of action—that is, the instructor “scrapes”, but performs in the air (line 5, Figure 6.2).

Thus, during this first sequence (lines 2 - 5), one hand is configured to hold the scraper, as it would be held if actually used to work upon sheet-rock. The hand takes on the same configuration when the second scraper—i.e., the putty-knife—is picked up and its use is demonstrated (lines 6 - 8). And the same hand-configuration occurs again with the third scraper (lines 9 - 10). Thus, there is a natural contiguity between the members of a class of objects (in this case, instruments) and a configuration of the hand. While this configuration has no symbolic value during the moment above when the different scrapers are actually being held, the hand configuration becomes symbolic (a “transformed sensation”) subsequently, when the instruments are implied but not literally hand-held.

During a subsequent moment, transcribed below, the instructor’s hands move without holding a scraper (or any other instrument), but they remain configured in a fashion that can now, in this context, be recognized as an index of scrapers, at least by those who have witnessed the prior scene: the hand-shape makes sense only vis-a-vis the instruments that this hand has previously held, and it simultaneously conjures up the image of these instruments. While the tools are physically absent from the instructor’s demonstration, their “virtual” or symbolic-cognitive presence can be jointly inferred by all recipients of this communicative act. The hand-shape turns into a socially shared symbol.

Having put the scrapers down (line 13, segment 1), the teacher now raises an empty hand that is shaped "as if" holding a scraper (line 1 below). The hand moves in mid-air and thereby initiates a more complex demonstration: an absent scraper is used to distribute invisible mortar on a non-existing surface so as to make virtual piles of the compound, as described simultaneously (albeit vaguely) through speech (Figure 6.3). While the hand moves with precision, the verbal instructions are quite inexplicit. Without touching the tools that remain visible on the countertop, the instructor’s hands move to make the scrapers "present" through an indexical implication.

Here, the instructor demonstrates a complex, skilled activity. In a sequence aimed at demonstrating the importance of applying the right amount of compound to sheet-rock, he shows how excess compound can be scooped up with and removed from the scraper. The instructor’s movements are swift and precise, and for those who have some visual familiarity with this line of construction work, it is easy to “see” not only the scraper, but also the compound, the surface (i.e., the sheet-rock), as well as another, unnamed tool which is used to remove the compound from the scraper. We “see” how compound is scraped up to make a pile which is then lifted off the sheet-rock and scraped to the other tool.

Thus, from the teacher's initial handling of the scraper, a pattern of movement is abstracted—that is, a gesture. Performed publicly, it constitutes a communal sign, a convention, in which a shared experiential nexus or bit of knowledge about the world is embodied. The gesture evolves as a situationally-transparent, symbolic construction through which practical knowledge may be handed about. It emerges in two stages of what might be called an "indexicalization schema": originating in the hands-on manipulation of the material world within reach, the abstracted gesture retains an indexical link to it, which can be used in both directions—the gesture presupposes the material world for its intelligibility, but can also and by the same token evoke it. The sign is now available to invoke a nexus of practices, things, and relations between them, and is potentially applicable to infinite communicative purposes, syntactic contexts, and semantic roles. By simply raising a hand with a recognizable shape, a complex of actions, objects, instruments, and so on may be denoted by the teacher or other participants.

The sign is also available to be folded back upon itself, for the layering-on of further in-formation, for example, to denote manners, possible mistakes, or more elaborate lines of action. In this fashion, the shared knowledge of the community can grow via the invention, re-use, and trans-formation of an ephemeral, but nevertheless material, sign. A pattern of muscular movement has been abstracted from the hands' entanglements with concrete worlds and can be used in other contexts, including strictly symbolic or representational ones such as conversation. But to function in these contexts, audiences must be able to "fill in" indexical background: in these episodes, recipients of the instructions initially had to “see” a counter-top as a symbolic representation of sheet-rock and, subsequently—on the basis of locally produced knowledge—recognize that a hand was configured in a certain way because it represented the holding of a scraper. (Compound also had to be filled in.)

Our focus now shifts from the do-it-yourself workshop where hands engage in (symbolic) instrumental actions from which gestures are then derived, to a university classroom where a professor’s hands manipulate a material, spatial object in an exploratory fashion. The professor’s movements made in this process are subsequently abstracted as gestures that do not represent actions or instruments, but rather features of the object explored. While the signifier is a motion, the signified is a fixed, immutable structure in space.

As part of his undergraduate course on architectural design, the professor critiques miniature buildings made by students using cardboard and glue. The students sit arranged in a large circle, all oriented toward the middle of the classroom where the professor sits on an elevated stool, next to a large table. One by one, the students step forward and place their cardboard models on the table, making them available for others to see, and available for the professor's critique. Here, we focus on one cardboard model: introduced by the student as a "tunnel shape," it is subsequently described as a "sewer-like" culvert by the professor, who first explores it, then interprets it, and thereby critiques it.

Before talking about the model, the professor silently explores its three-dimensional features. He leans forward and over the model to look inside; he moves his body to see the sides; then he touches it, lifts it, and slowly turns it in mid-air, observing it from various angles (Figure 6.5) and at the same time feeling with his fingers the architectural shapes created by the student's hands. The professor's exploration might be called a "primary" stage of knowledge formation--and hence a "necessary" stage of symbolic action. As he encounters the cardboard model for the first time, his hands become entangled with it. He has a series of visual and tactile experiences, made possible by his practical actions upon the object that he begins to grasp. He becomes knowledgeable regarding the cardboard model, through the very experiences that his subsequent gestures will formulate and index. At the same time, the professor's lived experience is a shared experience: as he explores, he also shows. By turning the model in mid-air to observe its various angles, he enables his students to do likewise; and he directs their attention toward the model through his orientation, gaze, and extended fingers, which all function like pointing gestures (Scheflen 1976). Moreover, the professor's exploration may be regarded as a demonstration. By sitting in the middle of the room, at the center of the group, he makes his own experience an object of attention, a public performance, a potentially vicarious experience. His behaviors are both private and self-conscious, appearing as practical precursors of the group's symbolic tasks. His solo actions are a form of social practice as he interacts with a material world—something the instructor and scraper only implied. In short, his hands mediate between thing and thought.

Eventually, the professor begins talking about the cardboard model as his hands continue to move in relation to it. His exploring hands are also pointing fingers that direct the students' attention toward specific spatial features, thereby highlighting shapes of the model being discussed. For instance, the professor slowly slides an extended index finger along one edge, finding and highlighting its curved shape (Figure 6.6), creating a figure-ground separation that informs students how to see the model. At the same time, the professor may be teaching students how to see his hand—his behavior may serve to highlight the movement of his hand, not just the shape of the model.

With increasing frequency over a ten-minute period, the professor's hands move without actually touching the cardboard model. That is, his hands perform shapes in the air that are physically separated from the material object that they index. The following transcription represents such a moment.

While touching the cardboard model, the professor describes it. He refers to its "long bent" form (line 1) and highlights the same by touching the model with his extended index finger (Figure 6.6). His speech is organized (i.e., he says "long" before he says "bent") according to the shapes that his finger encounters as it slides upon a long, straight edge before moving around a bend. Immediately after highlighting the model's "long bent" shape, the professor reproduces this shape in the air--mere inches above the cardboard, but nevertheless separated from it (Figure 6.7). This is a defining moment. The mid-air motion is recognizable as a gesture, because its shape is performed apart from the tangible influence of its referential anchor. The gesture emerges as a natural extension and an incipient feature of practical actions upon an object. The new convention is shared, understood by those participating (perhaps vicariously) in the hands-on activity. Moreover, the new symbol heralds knowledge-formation that the speech also marks: as touch moves to gesture, concrete talk turns abstract; the words "long bent" (line 1) describe tactile and visible features that are quickly recast as a "linear experience" (line 2), and the professor begins to discuss the full-body consequences of the hand-size shape. In sum, the emerging gesture shows a close connection with a material object, which serves as a springboard into interaction about imagined experience--altogether a sort of transformed sensation.

Continuing for several minutes, the professor's hand gestures evolve as his critique unfolds. Sometimes, his movements are relatively small: with a flick of his wrist, he outlines a "long bent" shape in the air, approximately the size of the architectural model (hand-size) and located only inches above it (see figure 6.7). Other times his movements are big: “long bent” shapes performed at eye level, where his whole arm extends, separated from the model by feet rather than inches (Figure 6.8). The following transcription represents such a moment.

The professor's descriptions become vivid as he expounds upon the full-body consequences of the hand-size model. When he talks about seeing “the wall at the back” (line 1), he extends his right arm fully so that his flat palm may be regarded as a distant surface (similar to Figure 6.8). As he describes what “you can see” (line 1), he draws his gesturing hand toward his eye and lightly touches the side of his face. Then his hand moves outward from his eye and straight along his line of vision, which “goes” (line 2) parallel with the imaginary passage. As he says "back" (line 2), his arm becomes fully extended again, and then his hand hooks to the left (Figure 6.8) when he says "disappears around the corner" (line 3). Altogether, the professor outlines a "long bent" shape, created in mid-air but referentially anchored in the model on the table below. At the same time, he performs a "lived" experience: gazing off "into space," he talks himself along the hallway that he imagines and motions, using present-tense language to describe architectural details as they spatially and temporally unfold. Because his performance is public, the professor provides--at least potentially--a vicarious experience to seated students, as suggested by the pronoun "you" (line 1).

Group knowledge forms as "long bent" gestures occur and evolve to embody the unfolding shape of the professor's critique. Shared understanding is especially evident when students participate in the class discussion and also perform "long bent" gestures to articulate their views, which build upon the group's understanding. For instance, consider the following excerpt. Immediately after the professor ends his comments and invites other opinions, a student performs a large gesture while giving a favorable critique of the architectural model:

To articulate his view, the student uses behaviors that may be recognized by other classroom participants. With the utterance "u:h" (line 4), he lifts his right hand to his right eye; while describing the "dire:ction" (line 4) of the architectural structure, his hand moves outward from his eye and straight along his line of vision; and once his arm is fully extended, his hand hooks to the left (Figure 6.9). Altogether, the student outlines a "long bent" shape in mid-air that is referentially anchored in the model on the table--largely through the professor's prior public performance. Moreover, the student's gesture occurs as he expounds upon the full-body consequences of the hand-size model. Like the professor, the student uses present-tense language to describe an imaginary "lived" experience. The student's behavior is a potentially vicarious experience for others in the classroom, as suggested by the pronoun "you" (lines 1, 2, and 3).

The architectural model is different than the do-it-yourself scraper, for reasons that have consequences for the formation and recognition of gestures. On the one hand, the scraper is lifted and used. It is treated as a tool, an instrument to be employed, an extension of the human hand that subjects implied objects to performed actions. On the other hand, the model is lifted and explored. It is regarded as an object, an entity with properties to be discovered, a thing understood through the human actions that its shape implies and guides. Although material things may embody instructions for how they are to be used or regarded (Norman 1988), differences between the scraper and model are instantiated through behaviors during the "primary" stage of knowledge formation. Wielded again and again, the scraper is never mistaken for a miniature building. Although cardboard is capable of scraping mortar, the professor initially and immediately moves his body relative to the architectural model, projecting fixedness upon it. As a consequence of primary behaviors, gestures eventually emerge imbued with verb-like and noun-like attributes. Within the workshop, the teacher's gestures are verb-like, with the implied scraper in the role of instrument: his hand movements are recognizable as hand actions--a rather low-level abstraction. Within the classroom, by contrast, the professor's gestures become large and semantically complex: his hand movements are noun-like as they outline (in the air) three-dimensional features discovered in the model; these same hand movements are at the same time verb-like as they represent a person's movement through the imaginary building being outlined--a rather high-level abstraction. Moreover, the professor's gestures serve as a heuristic device, enabling him to translate the hand-sized model into a full-size architectural experience that becomes grounds for his critique. Furthermore, his gestures serve as a teaching device, helping students to read the cardboard shape as a representation of an embodied experience, vicariously lived and critiqued.

Our analysis of these episodes has demonstrated at least three things. First, the interactions within the classrooms exemplify the step-by-step processes by which embodied—manual—action in the world of matter and things may be transformed into symbolic action. That is, activities within and upon the material world may be abstracted, schematized, and converted into components of the shared communicative repertoire of a local “community of practice”. In short, our analysis has traced the experiential, practical roots of individual gestures. Second, our analysis shows how the shared knowledge of these communities grows through the formation of these gestures. Like all signs, the gestures comprise cognitive and communicative features and functions: they do not only represent or express, they constitute socially shared knowledge. The further growth of knowledge in these communities is dependent upon and made possible by the shared possession of newly formed signs, which can then be modified and elaborated so as to represent proper modes of action, possible mistakes, compound activities, and so on. Finally, the episodes demonstrate that the proper seeing and understanding of these signs requires the material world as an indexical background: the configurations and motions of the hands only make sense by virtue of their contiguity to things, and it is by reference to these things that the motions of the hands can be recognized as schematic actions (or manipulations). In short, seeing a gesture requires knowledge of the world.

Within the instructional situations we have examined, such knowledge is available from the course of recent interaction; the publicly visible, material experiences and activities of the hands from which the gestures have been derived is part of the local, shared memory of all participants. Some might object, therefore, that these incidents are uncharacteristic of gestures as they are used in the exclusively symbolic realm of everyday conversation, “conversation pure”, which is more often than not predicated upon the absence of the material events and states-of-affairs that constitute its topics. Moreover, it might be objected, symbolic communication in the absence of things constitutes the more astonishing and more important human achievement.

However, it is our contention that gesture—certainly descriptive or “iconic” gesture—necessarily involves indexical links to the material world, even though these links are rarely established or explicated in the communicative situation itself. Rather, in conversational contexts that are detached from the talked-about world, participants must fill in encyclopedic knowledge (ranging from universal bodily experiences to highly specific cultural practices) to see and recognize gestures. Phenomenally, there are only motions of the hands. What is perceived, however, are typically not motions but actions and, simultaneously, implied objects acted upon. Thus, we do not see two flat hands moving apart, but we see them dividing a substance (or a substance being divided)—which, in the given conversational context, may signify a group of friends, dough, or the mélange of issues that will have to be discussed. To see the hands engaged in a schematic act of dividing, and to “see” a something that is being divided, our eyes must be intelligent, experienced in the ways of the world of the hands. Without chipping in our “beholder’s share” (Gombrich 1961), we could not see the gesture, that is, the signifier—let alone identify the signified.

We briefly illustrate these indexical underpinnings of iconic gestures by examining two excerpts from a series of conversational stories about car accidents. In the videotaped recording (which was made in Germany), two Japanese friends, having discussed the difficulties of obtaining a German driver’s license, proceed to tell one another about their involvement in various accidents that resulted from the drivers’ inattention to the road. As is common in such narratives, the narrators’ bodies variously re-enact the protagonist’s actions while driving the car, shift to render the behavior of the cars on the road, or move to represent the setting. More specifically, their hands are used in semiotically different ways to represent different “players” in the event, enabling the two interlocutors to speak from constantly shifting perspectives: at times in “the first person”, they re-enact the acts of the protagonist’s hands; at other times in “the third person”, their hands serve as symbolic tokens for the moving and crashing cars; at yet other times, their hands are extraneous producers of symbolic constructs—sequentially rendered, ephemeral, three-dimensional shapes—which represent components of the setting. In the following, these components, in particular, deserve our attention.

The first of the two fragments is a narrative segment told “in the first person”. The speaker, Satomi, describes how she absent-mindedly drove along a straight road when she realized that she had to turn to the right. At this point in the story, she puts down the tea-cup that she has been holding in her hands, readies her hands by moving them to her stomach, and says:

Along with the lexical description of her turning of the steering wheel, Satomi makes a large, two-handed gesture which international audiences unfailingly recognize as an enactment of car-driving: holding her two hands in a “grip shape” parallel to her chest, she moves one diagonally up and and the other one diagonally down and thereby shows the turning of a steering wheel (Figures 10-11). The gestural portrayal is consistent with the lexical description, and the gesture appears to add little to the narrative, except perhaps visual precision (i.e., how far she turned the wheel).

And yet we may ask how it is that we can so easily identify the gesture—and the action that the gesture enotes as well as the object that is involved in the action—all of this despite the fact that we do not understand the language. In fact, it is this gesture (which is performed several times during this narrative) that enables people who do not understand Japanese to “see” a story about driving cars and accidents. Here is the simple yet significant explanation: we recognize the gesture to the extent that we share the material culture that the speaker is drawing upon. Throughout the world, as in Japan, there are few objects (if any) that are routinely handled—held and turned—in the fashion of a steering wheel, other than the steering wheel itself: the gesture therefore can only be about driving a car. We recognize the action that is abstractly performed by the configured motions of the configured hands because we are culturally familiar with the material world where such an action could “really” be performed. Our perception of the gesture as a schematic action requires our beholder’s share, that we “fill in” generic objects to which the motions of the hands may relate.

In the “steering wheel” instance above, our encyclopedic knowledge of the material world—what kinds of things are handled in the fashion seeable in the gesture?—enables us to recognize the signified. In other cases, world knowledge enables us to see the gestural signifier—the seeing of a motion as an action—even when this action has no correspondence in the event that is signified. For example, through a version of the method of “documentary interpretation” (Mannheim 1959) we manage to see that, for example, the motion pattern of the hands could relate to a specifically shaped object—that they are virtually holding a bowl (see Scheflen 1974). Thus, we “see” a bowl—and disregard the “holding” (which has no correspondence in the event). The schematic act of holding—the gestural signifier—would be an exclusively descriptive or pictorial, not a referential, device. Nevertheless, to perceive it, we must know about basic embodied acts and their generic objects.

This applies to the following segment in which Satomi’s friend and conversational partner, Tomoio, also shows a round object that can be turned; she shows it by performing a one-handed gesture. But this is an object that could not be manipulated using the action-pattern from which the gesture is formed; the motion here serves descriptive purposes, and no action corresponds to it in the event that is reported. The object shown by the gesture is a worn-out tire skidding on a slippery road.

7.1 Tomoio h de kou hashitteta no (.) ‘hh (shitara) sa ame ga furi hajimete And we were driving like this, then it started raining.

During her utterance at line 7.4, Tomoio opens her right hand wide and rotates it a few times back and forth so that her wrist provides an axis (Figure 6.12). She performs a schematic manual action to evoke a particular kind of object—an object that is round and can be rotated around an axis. To see the gesture this way, onlookers must possess and fill in basic experiential knowledge about relationships between acts of turning, round shapes, and rotating things. Using this knowledge, they can “see” in the speaker’s hand a round object and recognize the hand’s rotation as a description of the object’s behavior, skidding, rather than as a schematic action. The signified (the car’s tire), of course, could not in reality be turned in this fashion by a single human hand, because it is so much bigger. But this is a matter of contextual specification: what matters here is that the object was round and that it rotated, not what size it had. To use grammatical terminology: while Satomi’s gesture would be understood as having the grammatical shape of a transitive verb plus nominal undergoer (steering a steering wheel), Tomoio’s has that of a nominal subject modified by an intransitive verb (a tire skidding). This distinction is not inherent in the gestures’ shapes, but rather is a seeing that is achieved in a context and informed by knowledge about the material world. We must know that there are things that can be turned back and forth, and that they may be round and rotating around an axis. Tires are objects that have these features. Thus, by moving from the schematic act of the hand to the object and back to the hand, onlookers know to abstract from the hand’s action: here, it is only a component of the signifier, not of the signified; there is no hand nor human agent in the moment described, just a worn-out tire skidding on a slippery road.

In both of these segments, the gestures achieve their pictorial effects through a presupposed indexical link between the hand and the world of things. In the first case, we “see” the object that the protagonist’s hands actually manipulated in the event; the gesture is a virtual action which represents an instrumental action in the depicted scene. In the second example, the gestural motion enables us to “see” a kind of object: one that is round and rotating. This narrative hand has no correspondence in the event; there is no hand that holds and turns a tire (nor could there be).

Iconic gestures of this kind have been described throughout the literature on gesture (cf. Calbris 1990; McNeill 1992; Wundt 1975 (1911), among others), but little attention has been devoted to what they tell us about the roots of gesture in non-symbolic action nor to the role that these experiential roots—and, thus, our experiential, embodied knowledge about the material world and its manipulations —play in our ability to see and understand gestures. Our suggestion here is that an essential component of the pictorial language of gesture—of the logic by which “iconic” (or descriptive) gestures achieve their communicative effects—are the tactile, indexical ties that the hands and their motions have and retain to the material world that they can reach, explore, know about, and act upon. Before we learn how to gesture, we learn how to handle things. This knowledge is incorporated in our hands; we use it and we appeal to our interlocutor’s possession of it when we represent the world gesturally and invite our interlocutors to understand our gestures accordingly. Gestural communication is mediated by shared experiential knowledge about the material world.

Our aim in this chapter has been to examine processes of symbol formation (Werner & Kaplan 1963) and use within communities of practice and to regard these processes as native to the performance and recognition of gestural representations within conversations abstracted from hands-on interaction with things. Initially, we have investigated the orchestration of linguistic and embodied communication within “material-rich” learning settings such as do-it-yourself and architecture-classrooms and suggested that the evolution of bodies of knowledge is a communal process of forming symbols that embody experiences that have emerged in situated action. The contiguity of gesture and experience in these contexts enables us to examine the indexicalization practices (a term suggested to us by John Gumperz) through which gestures are made meaningful and gradually become "independent" meaningful symbols within a (however limited) community of practice. At the same time, we have displaced gesture, specifically iconic gesture, from the bipolar logical space in which it is almost exclusively situated and studied when it is examined in the context of abstract narrative or conversation. Our studies therefore also reveal something about the ways in which generally representational gestures emerge and are understood. Within “pure” conversations, gesture is typically assumed to correspond either to a mental image or to objects, actions, or events “out there” (beyond the immediate setting) with which it is alleged to share similarities of form. Neither mental image nor referent are available in the documented situation. However, in practical settings where gestures share the situation with the experiences that they formulate, we can begin to see the important indexical underpinnings of the pairings of form and meaning, of signifier and signified: here, these underpinnings are the situated histories of practical and symbolic action in which tactile, practical acts—manipulations—are gradually abstracted into “free-standing” symbols which can then signify, not only acts, but also manners and mistakes as well as objects acted upon and their features. We believe that these gestures' socially-situated emergence may reflect, generally, the way humans interactively constitute and use gestural forms.

The experiential grounding of conversational gestures is typically not available to interlocutors in the situation at hand, but it is presupposed. These gestures have lost the immediate connection to the material experiences from which they may have derived, and circulate in social life as purely symbolic, gestural forms. They appear self-contained because their situated emergence and existence extends beyond the situated humans who use them. But many of the gestural shapes that people recognize in talk-in-interaction may have originally been devised in some prior situation of the kind that we have described above and from which they were abstracted and passed on. And competent members of human societies are capable of recognizing these links between gestures and objects because they are familiar with the basic actions of which gestures are abstract versions, as well as with the world that affords them. Moreover, the speaker’s hands know how to do things other than gesticulation, and it seems unlikely that the skills that the hands bring to bear on their symbolic tasks are entirely separate from those that they have acquired while handling things. Rather, the patterns that are at hand when there is a need to gesture appear to be made from the same fabric as those that are required in instrumental action. And this “producer’s knowledge”, too, is socially and culturally shared.

In the picture that emerges from these studies, then, gesture is not a symptom of mental events, and it does not derive its communicative potential from a hypothetical relationship to images on the speaker’s mind. Rather, it is an embodied and therefore public symbolic practice, kinesthetically known by its makers, visually known by its beholders, and derived from and embedded in an objective world within which mindful hands operate. Gestures do not represent by virtue of any similarity to their denotata. Rather, they are abstracted from—and are interpreted vis-a-vis the backdrop of—the realm of material action. Hands learn how to handle things before they learn how to gesticulate. They are knowing signifiers and bring embodied, schematic knowledge, acquired through physical action, to the diverse symbolic tasks of gesticulation. Gestures, designed as they always are for particular recipients, appeal to those recipients’ knowledge, knowledge that may have been acquired over the course of the current situation or in a cultural and physical world that is in some part the shared property of the members of a single society or cultural group, and in other parts common to all human beings.

Bates, J. A. V. 1974. The communicative hand. In Benthall and Polhemus (Eds.), pp. 175-194.

Benthall, J. and Polhemus, T. (Eds.) 1974 The Body as a Medium of Expression London: Allan Lane.

Calbris, G. 1990. The Semiotics of French Gestures. Bloomington: Indiana University Press.

Condillac, E. 1746. An essay on the origin of human knowledge, being a supplement to Mr. Locke's essay on the human understanding. London: J. Noursse.

Gombrich, E. H. 1961. Art and Illusion. A Study in the Psychology of Pictorial Representation (2 ed.). Princeton N.J.: Princeton University Press.

Goodwin , C. and Duranti, A. (eds.) 1992 Rethinking Context: Language as an Interactive Phenomenon. Cambridge: Cambridge University Press.

Hanks, W. F. 1992. ‘The indexical ground of deictic reference.’ In Goodwin and Duranti (eds.), pp. 43-76.

Kendon, A. 1980. ‘Gesticulation and speech: Two aspects of the process of utterance.’ In Key, M.R. (ed.), pp.207-228.

Key, M.R (ed.) 1980.The Relationship of Verbal and Nonverbal Communication. The Hague: Mouton.

Lave, J. 1991. ‘Situating learning in communities of practice.’ In Resnick, Levine, and Teasley (eds.), pp. 63-83.

Locke, J. 1959 (1690). An Essay concerning Human Understanding. New York: Dover.

Mannheim, K. 1959. Essays on the Sociology of Knowledge. London: Routledge and Kegan Paul.

McNeill, D. 1985. ‘So you think gestures are nonverbal?’ Psychological Review 92: 350-371.

McNeill, D. 1992. Hand and Mind. What Gestures Reveal about Thought. Chicago: The University of Chicago Press.

Resnick, L. B., Levine, J. M. and Teasley, S. D. (eds.) 1991. Perspectives on Socially Shared Cognition. Washington, D.C.: American Psychological Association.

Wundt, W. 1975 (1911). Völkerpsychologie. Eine Untersuchung der Entwicklungsgesetze von Sprache, Mythus und Sitte. Aalen: Scientia Verlag.

[i] We wish to thank Cam Tew for allowing us to make use of this material.

[ii] We use the following transcript conventions:

:: colons indicate a sound-stretch (vowel-elongation)

- a dash marks a ‘cut-off’, generally a glottal stop

(.) a dot in parenthesis indicates a ‘micro-pause’ (less than one ‘beat’)

( - - ) dashes in parenthesis indicate a pause; each dash represents approximately

one tenth of a second

[ square brackets indicate the simultaneity of two events, either two or more

utterances or an utterance and an action or gesture

(( )) double brackets mark the description of an action

‘ ‘ single quotation marks within the description of an action signify a “virtual”

component of the action, for example, an object that is “implied”.