The Role of Computational Cognitive Artifacts

in Collaborative Learning and Education

Project Summary

This project addresses ROLE quadrants 2 and 3: It builds bridges from cognitive and social theories of the role of artifacts to research on learning in educational settings, and it develops a methodology for the principled assessment and research-based design of technological artifacts to mediate learning processes. The goal of the project is to refine both a micro-analytic methodology and an artifact-centered theoretical framework that can aid in the principled design of distance learning environments. The project will not only result in a much-needed methodology for future designers of educational technology, it will also deepen our understanding of the role that such computational cognitive artifacts can play in collaborative learning and formal education.

The project studies small groups of students using prototype versions of learning environments to see what the students go through in learning how to use the computer-based artifacts and what problems interfere with the learning goals. The project brings together educational software developers and experts in human-human and human-computer interaction to conduct the analysis of videotaped student interactions and to iterate the design of the educational environments.

Three software systems developed by project team members are studied, gradually advancing from a relatively simple computer simulation, through a semester-long on-line biology lab curriculum, to a distance education version of the labs:

1. SimRocket simulates the launch of rockets having varying characteristics. Five middle school students used the simulation to predict the effects of the different characteristics on the height attained by the rocket. Their sessions working with the simulation were videotaped.
2. VirtualBiologyLab is a series of 10 freshman college biology lab experiments simulated and conducted on the Web. Groups of 2 or 3 students work together to complete the lab – analysis of their successes and difficulties feeds back into the iterative design of the digital curriculum.
3. WebGuide is a knowledge-building environment for supporting collaboration. Functionality from WebGuide will be integrated into the VirtualBiologyLab in the final project year, and used by geographically distributed high school students for a version of the biology labs redesigned for Advanced Placement study.

From the perspective of the project’s theoretical framework, these learning environments are treated as interacting networks of computational cognitive artifacts. For instance, in analyzing students working with SimRocket, the project team looks at how the students talk about and make use of (a) the rocket simulation, (b) a display of rocket characteristics, and (c) a data collection form. Each of these three artifacts is designed in a way that permits it to be used in certain ways to accomplish certain tasks: (a) some artifacts like the simulation are computational and change on their own in response to inputs; (b) others like the display convey knowledge; while (c) yet others like the form provide cognitive support by organizing and preserving information. In each case, the students must learn how to recognize and take advantage of these artifact affordances. Designers of learning environments must design both the affordances of individual artifacts and the curricular context that will make these meaningful to students within a coherent educational experience.

The project analyzes the collaborative efforts of small groups of students in order to determine (a) the extent to which students can understand the use of educational artifacts, (b) where this is problematic, and (c) how student learning can be scaffolded to overcome problems. In collaborative interactions, students must display to each other their beliefs, their questions, their problems, and the resolution of problems. When this process is videotaped and carefully analyzed, it makes the students’ learning visible to researchers as well. The project adopts a micro-analytic form of communication analysis called micro-ethnography to study what is displayed. This is a rigorous method for analyzing both vocal and visible forms of human interaction recorded on video.

The project team has already begun to adapt micro-ethnography and the theory of artifacts to the analysis of student interactions with on-line educational technologies. Members of the project team have collaborated in various combinations in the past, including a semester-long pilot project investigating SimRocket student interaction data and theories of artifacts. The 4 Principal Investigators are experienced in the design of educational technology and/or the micro-ethnographic analysis of people interacting with educational artifacts. All 9 of the Advisory Board consultants assessed educational technologies in their PhD dissertations and/or in their current work. The 2 Graduate Research Assistants are pursuing dissertations closely linked to this project.

Project Description

Overview of Proposal

1. The Problem of Educational Artifacts
2. Three Educational Artifacts for Study
3. A Staged Research Plan
4. Theoretical Framework for Cognitive Artifacts
5. Research Methodology for Studying Interaction
6. Sample Analysis of an Episode of Collaborative Learning
7. Pilot Studies Conducted
8. Results from Prior Support
9. Contributions to ROLE Goals and Potential Impact
10. Project Team – see Biographical Sketches

1. The Problem of Educational Artifacts

As schools across the nation get wired for computer-based learning, the problem of scarcity of effective on-line curriculum and content becomes increasingly urgent. Current research stresses the importance of carefully designed software artifacts that are “student centered, knowledge centered, assessment centered, and community centered” (Bransford et al., 1999) . Yet, with a few notable exceptions, there is little in the way of constructivist curriculum and content that meets these criteria and is also ready to take advantage of computer- and Internet-based media. In fact, there is little systematic knowledge of how to develop such educational artifacts, grounded in a theoretical understanding of the role that such artifacts might play in learning and in a methodology for software testing.

Distance Learning

The general problems of computer-mediated education multiply substantially under the pressure to rush to distance learning. Universities and dot.com’s around the country have jumped on the distance education bandwagon, without necessarily thinking through the complex educational issues involved. It is true that distance education has the potential to address various pressing educational, social, geographic, and economic issues (Keegan, 1986) . It is also true that the technical infrastructure that will enable this revolution in education to proceed is being quickly set in place. However, the design and development of the necessary curriculum and content lags far behind. In the dash to market, providers of distance education are likely to settle on software technologies that were developed for other uses and are inappropriate for educational applications, curricula that implement outmoded approaches like drill-and-practice, and content that has not been tested for its learning effects. We need to develop more new models of computer support for learning that are effective in distance learning.

Designing Computer Support for Learning

Based on our own experiences with software in classrooms, we have found that computer-supported collaborative learning (CSCL) has a vast – and largely untapped – potential. Access to global sources of information is just one facet. In addition, computer simulations can transform conceptual representations into interactive worlds for inquiry. They can transcend real-world barriers of time, expense, geography, scale, expertise, etc. to allow students to engage with and experience phenomena that have until now been unapproachable. Hypertext systems of information can personalize presentations to meet individual learning needs. Communication media can promote collaboration in ways never before possible, as well as among people who could not hitherto interact. Structured curricular databases and shared knowledge-building environments can support student learning processes. However, we have seen that students always use computer artifacts in ways not envisioned by the designers. So, careful study of the artifacts in naturalistic settings is critical to the development of effective educational technology.

Understanding Computational Cognitive Artifacts

It is possible to ground the design and assessment of educational software applications in an understanding of their role as “artifacts” in learning. Our preliminary understanding views various forms of artifacts as absolutely central to human cognition and learning. People construct their understanding through interaction with artifacts; often artifacts extend, amplify, or transform cognition; eventually the artifacts may be internalized as mental procedures (Cole & Griffin, 1980; Donald, 1991; Engelbart, 1995; Hutchins, 1999; Norman, 1993; Papert, 1980; Pea, 1985; Vygotsky, 1930/1978) . We intend to further develop this theoretical framework, which is inherent in theories of situated action and situated learning, in distributed cognition, in activity theory, and in various philosophies.

In particular, we propose to apply this framework to analyze educational technologies as “computational cognitive artifacts” (CCAs). We use this term to refer to computer-based or Internet-based educational artifacts (simulations, data analysis tools, on-line curricular modules, etc.): They are “computational” if they respond interactively to user interactions by changing their display. They are “cognitive” to the extent that they can become part of the user’s thinking, by, for instance, helping the user to visualize some phenomenon, providing an external memory or workspace for manipulating representations, or aiding in conducting a computation. They may also be “cognitive” in the further sense that they can be internalized in the user’s mind so that he or she can make use of them as a mental metaphor or representation in the future when they are no longer even virtually present on a monitor screen. They are “artifacts” in the sense that they are perceptible objects that were designed to serve as some kind of tool – even though today they may not be physical objects that can literally be grasped.

To conceptualize an educational application as a CCA is not to assume a priori that it functions effectively in this role. Rather, it is to raise a set of critical issues:

·        Does it facilitate human-computer interactions or mediate human-human interactions computationally?

·        Does it support and enhance cognitive functions of its users?

·        Does it function as a useful artifact in accordance with its design?

By conceptualizing certain types of educational technology (e.g., computer simulations) as CCAs, we can ask if they are fulfilling this role effectively in specific situations that we observe. We propose to investigate how people develop the understanding required to use CCAs effectively in CSCL settings, and conversely to study the roles these CCAs then play within the collaborative learning and education taking place.

Assessing Collaborative Learning for Iterative Design

Our goal is to contribute to the design of CCAs that are effective for supporting collaborative learning and education. Our theoretical framework does not directly imply criteria for the design of educational technology. Rather, it suggests that we develop prototypes of software applications and look at how students actually relate to them as computational cognitive artifacts – that is, that we look at how students concretely explore, come to understand, and use the software as an artifact for extending their cognitive powers – and then we iteratively revise the design of the software. For us as researchers to look at this, we need a methodology. We believe that micro-ethnography provides such a methodology. Micro-ethnography was designed to look very closely at social interaction processes. This project will adapt micro-ethnography to look at computer-mediated interactions in situations of collaborative learning.

The proposed project is an application of micro-ethnography’s method to the concerns of human-computer interaction. It brings together a team of people from these areas who are experienced in interdisciplinary research (see Biographical Sketches). Our team includes faculty and students from Communication, Computer Science, Cognitive Science, and Education, as well as developers of educational software – within a broader academic community that is supportive of this project. This project is unique in bringing together educational software developers and specialists in the micro-analysis of interaction to develop and systematically test a rigorous methodology and a grounded theoretical framework for the design of distance learning artifacts.

2. Three Educational Artifacts for Study

In our project we will study the use of three software systems that we have developed: SimRocket, VirtualBiologyLab, and WebGuide (see section on Pilot Studies for more details):

A Middle School Computer Simulation

SimRocket is a simulation of rocket launches. We already conducted and videotaped a three-hour trial of it with 5 middle school students and a teacher. We have begun to analyze the data from this trial. We have observed that the simulation artifact played a central role in the interaction: it opened up and defined the whole educational space, providing the narrative context as well as the source of data for collection and interpretation. In the analysis of a specific episode with SimRocket (see below), we will see the collaborative interaction revolving around three inter-related artifacts: the computational simulation of rocket launches, an external memory display of rocket characteristics, and a paper chart of recorded rocket heights. By closely analyzing the interactions among the teacher and students we see: (a) successes and failures of students to grasp the meaning/use of these artifacts, and (b) the teacher’s attempts as an experienced scientist to guide the group to effective use of the artifacts. Because of problems in the interaction that become apparent to the participants, the teacher must make his analytic skills observable and the students must make their adoptions or misunderstandings apparent. We also see that there is not a single simple artifact here, but a subtle network of artifacts with different functions. Furthermore, the artifacts only exercise their cognitive function or activate their meaning when they are being used appropriately. Our observations of the teacher’s patterns of face-to-face interaction suggest forms of scaffolding that could be introduced in distance learning where a teacher is not physically present.

A College On-line Lab

The VirtualBiologyLab is a much more complex network of interdependent artifacts. It is a complete one-semester curriculum on the Web, intended to replace college freshman biology wet labs for non-majors. Each of 10 planned labs takes an estimated three hours for a student to work through – and enables students to conduct seminal experiments from the history of biology that would not be feasible in traditional physical wet labs (see attached letter of support from the developer). One can distinguish multiple kinds of artifacts composing the software: a guiding narrative, animations of lab equipment, simulations of lab procedures, data collection / analysis / graphing / display tools, background materials (theory, history, remedial text), links to related websites, and interactive assessment exercises. The virtual lab is designed to be used by students independent of any teacher guidance, although it is loosely coordinated with a biology lecture course. The on-line system must work as a whole, motivating and guiding students through a sequence of tasks; each of the distinct component artifacts must work effectively on its own and within the whole pedagogical context.

A High School Distributed Education Lab

WebGuide is a knowledge-building environment to support collaborative learning. It provides a collaboratively constructed and shared external memory medium on the Web. The display is dynamically computed to show a hierarchy of notes arranged as a personal or group “perspective” on the persistent, asynchronous discussion. This perspective mechanism is an artifact that people must learn how to use and navigate to mirror and support the interpersonal relationships of collaboration. WebGuide also incorporates a variety of knowledge management functions that must be learned in order to manipulate the ideas stored in the system and to build effective shared knowledge. Certain components of WebGuide will be integrated with a version of the VirtualBiologyLab toward the end of our project to explore a collaborative distance learning biology curriculum at the high school Advanced Placement level. We will also extend the lab software to incorporate educational scaffolding techniques from other knowledge-building environments like CSILE/KnowledgeForum (Scardamalia & Bereiter, 1996) , KIE/WISE (Cuthbert, 1999) , and CoVis (Pea, 1993) .

3. A Staged Research Plan

Collaborative learning is a complex process. Accordingly, our project will build up gradually from our relatively simple pilot study to a full example of collaborative distance education.

Project Schedule

The project will consist of three main stages:

1.            Analysis of the three hours of video tape already collected of five middle school students and a teacher conducting a study of rocket design using the SimRocket computer simulation.

2.a.         A very brief study of college freshmen in a biology wet lab. This will serve as an informal baseline for the next stage.

2.b.         This is the core study for the project. We will videotape small groups of students working together with the on-line VirtualBiologyLab. This software is currently under development at the University of Colorado. The developers are involved in this project and will be iterating the design of the software in response to the analysis of the usage trials. We will focus our analysis on critical steps in the lab, like learning how to use a particular piece of equipment or a data analysis tool.

3.            A distance education version of VirtualBiologyLab will incorporate a collaboration medium based on WebGuide. This will be offered as an Advanced Placement curriculum to geographically distributed high schools students. The curriculum will be designed to be collaborative, and we will log user interactions and use these to study the learning taking place.

Following is a timeline for these stages:

Data Gathering and Analysis

Our gathering and analysis of data involves the PIs working closely with the graduate and undergraduate team members. In addition, our consultants participate in workshops held monthly. The workshops not only review project progress and plan next steps, but they importantly include group data sessions for the analysis of data. The data gathering and analysis process (for instance for the VirtualBiologyLab sessions) will typically proceed through the following steps:

1.        Videotaping of students. Two or three students are gathered around a computer. Cameras and microphones are set up to capture the facial expressions and body movements of all participants. The monitor image is also captured. Microphones are arranged to capture all speech as clearly as possible and to distinguish the speakers.

2.        The video is combined (picture-in-picture) and time-code is burned in to provide a frame-by-frame reference system.

3.        A minute-by-minute record log is created, describing in a sentence or two what takes place each minute. This is typically done by a graduate student and reviewed by a PI. The log may be revised later.

4.        A list of interesting episodes is created. Episodes are meaningful interactions lasting up to several minutes. The list is discussed by the whole project team at a group workshop.

5.        Selected episodes are digitized and made available electronically. This allows them to be replayed easily, looped, freeze-framed, slowed down, and studied by project consultants at distant locations.

6.        A detailed transcript is created. It transcribes both speech and visible behaviors. Speech of different participants is color-coded. The transcripts are printed and posted on the Web with the digitized clips.

7.        Each episode is assigned to a project team member who “owns” that piece of data. The owner watches the clip many times to understand what is happening there.

8.        A data session is conducted with the whole project team at a group workshop. This is a collaborative analysis of the data’s empirical details. Usually, about two hours are spent on a single episode. The session is led by the owner of the data, who presents the episode and raises issues. The owner may audio-tape this session to preserve ideas and interpretations that come up.

9.        The owner of the episode returns to a study of the video clip. At this point, the transcript may be revised and extended to include more details of interaction. The owner may invite other project team members to view and discuss the clip. The owner may present the clip at another data session. Finally, the owner drafts a micro-ethnographic analysis of the episode. This is distributed for comment. The analysis includes:

a.       A detailed description of the actions of all participants and their interactions.

b.      A discussion of what learning is evidenced in the data.

c.      A discussion of the role of any artifacts.

d.      A discussion of problems with the software, learning problems, etc.

10.     The analyses of the episodes are reviewed by the whole project team and various suggestions are made based on this:

a.       Proposed revisions to the software.

b.      Changes to the list of interesting episodes, such as the inclusion of additional episodes.

c.      Alterations to the research plan, such as scheduling additional usage sessions or changing the way they are conducted.

d.      Revisions to the research methodology and theoretical framework.

Project Assessment and Dissemination

We will engage in formative evaluation of our project throughout. That will be an important function of our larger team, which includes assessment experts, and will form a regular part of the monthly workshops. We will check that we are making progress toward our project goals in accordance with the project timeline and are following our data analysis procedures. Specifically, we will check that we are developing our methodology for making learning visible and for iteratively designing software artifacts, as well as disseminating our findings.

The micro-analytic approach that the project will develop provides a built-in assessment process for the project. By videotaping sessions of students working with artifacts, we will derive a formative evaluation of the learning facilitated by the artifacts. By the end of the project, we will be able to compare in a detailed and documented way how well our revised versions of educational software artifacts perform as compared to how they worked in the pilot studies and in earlier phases of the project. In addition, we will assess how successful we were in the course of the project in developing, formulating, and applying micro-ethnographic methodology for studying the educational role of cognitive artifacts and for assessing the ability of students to adopt the computational artifacts into their collaborative learning.

In addition to the micro-ethnographic analysis which examines both how students learn with computer technologies and their learning processes as revealed through their interactions (computer-mediated and face-to-face), it is important to understand how students relate to the technologies, as well as the degree to which students learn. In order to understand this, a triangulated approach to assessment will be adopted. Some students in the core trials of VirtualBiologyLab will be given a set of pre-assignment questions to gauge their prior knowledge and understanding of the concepts. Once they have completed the trial, they will be asked the same questions so that we can calculate their learning gains. In addition, we will interview these students in order to understand their perceptions of the artifacts as effective learning tools. This information will be gathered with each iteration and use of the software under development, and the comments and perceptions will be fed back into the development of software and the articulation of learning processes that involve computer software and computer-mediated collaboration. Understanding student perceptions of their experiences will also enable us to track our progress toward our research goals and to evaluate the effectiveness of the theory and method under development by answering the critical question of, does it work: have we indeed made learning visible in a way that can contribute to iterative design of effective software artifacts?

The PI will be personally responsible for coordinating activities associated with the project. He will supervise the work of students and consultants and ensure that they are working in accordance with the project plan, including the preceding procedure for the collection and analysis of data. The PI will make certain that the plan is followed and the timetable met (taking into account changes adopted during the life of the project). He will also attempt to mediate any conflicts that arise within the diverse and interdisciplinary project staff. The PI will engage project Advisory Board consultants who are assessment specialists to assist in on-going project evaluation and to conduct a quarterly project review for reporting to the Advisory Board.

Data collection and analysis issues including sampling and confidentiality will conform to rigorous research conventions and University of Colorado Human Subjects standards.

We will establish a website for both internal use and broad dissemination. The website will collect and coordinate materials and findings of the project. It will include logs of our videotapes, digitized clips of selected episodes, detailed transcripts, analyses of interactions, etc. It will also include all papers submitted to journals and conferences.

This project and its findings will be broadly disseminated in the CSCL, CSCW, HCI, education, and communication research communities through conferences and journals. It will be particularly prominent at CSCL 2002 and subsequent meetings of CSCL, AERA, CSCW, Group, ICLS, and WebNet. It will also significantly impact the release of a published VirtualBiologyLab curriculum at the college and the high school level.

4. Theoretical Framework for Cognitive Artifacts

In the current Fall 2000 semester, the PI offered an interdisciplinary seminar on the theory of artifacts. Many of the project co-PIs, graduate research assistants, and consultants participated fully in the seminar. The project’s theoretical framework grow out of this seminar. It will be considerably refined through a grounded theory analysis of the data collected in the project. This theoretical framework provides a bridge from selected findings of various cognitive sciences to research on learning in educational settings. It will guide the questions we pose in looking at our data.

Mediated Cognition

We start from three principles enunciated by Vygotsky (1930/1978; 1934/1986) :

1.        Mediated cognition. Modern human cognition is thoroughly mediated by physical and symbolic artifacts such as tools and words. We extend this to the use of computer-based artifacts like simulations, data analysis tools, and collaboration media.

2.        Social cognition. Meanings and practices are first established interpersonally and may then be internalized in individual minds. We take advantage of this by analyzing the interpersonal interactions, which are largely observable to the trained analyst as well as to the participants.

3.        Zone of proximal development. A student learns most productively when guided somewhat beyond his or her current developmental level by peers or a mentor. We use this principle to design experimental situations in which a small group of students is challenged to engage in a scaffolded scientific task.

Collaborative Knowledge Building

We conceptualize our subject matter as the process of “knowledge-building” (Bereiter, 2000) . This is an active collaborative learning process in which a community constructs conceptual meaning. For instance, in our SimRocket pilot study the students came to understand the effect of different variables upon future rocket launches and learned to isolate variables to measure their independent effects. The process of collaborative knowledge-building is interpersonal and observable – primarily through analysis of the communicative interactions through which it takes place.

Collaborative knowledge-building involves an interplay between individuals and the group, with individuals contributing from their personal perspectives and the group accepting these contributions in its own way (Stahl & Herrmann, 1999) . This perspective-taking and perspective-making unfolds in the observable world of signs and artifacts, such as spoken utterances and external memory devices (Boland & Tenkasi, 1995) . The physical and symbolic artifacts mediate between personal and group understandings.

The Role of Artifacts

It is possible to re-conceptualize learning (both individual and collaborative) through a focus on the artifacts that are involved. Artifacts – including software artifacts – embody intentionality, meaning, and experiences of their creators and preserve these for future users (Donald, 1991; Hall, 1996) . The problem is for users of artifacts to know how to reactivate this stored wisdom. This requires complex skills of interpretation (Gadamer, 1960/1988; Stahl, 1993) . Education can be viewed as largely the effort to socialize children and other new-comers into a practical understanding of the artifacts and practices that constitute a society’s or a community’s culture (Lave & Wenger, 1991) . The written word and the symbols of mathematics, for instance, are cognitive artifacts that take years of schooling to master. While people have been producing and using artifacts forever (Donald, 1991; Geertz, 1973) , we have little experience designing and teaching computational artifacts.

Artifacts play an absolutely central role in learning and understanding according to the philosophic roots that underlie contemporary cognitive theories that are influential for CSCL theories (Koschmann, 1996; Koschmann, 1999; Koschmann, in press) , such as situated action (Suchman, 1987) , situated learning (Lave & Wenger, 1991) , activity theory (Engeström et al., 1999) , distributed cognition (Hutchins, 1996) , dialogicality (Bakhtin, 1986) , and critical inquiry (Dewey & Bentley, 1949/1991) .

According to Hegel (1807/1967) , the very basis of self-consciousness and sociality in mutual recognition is thoroughly mediated by the creation and use of artifacts – which embody human consciousness or meaning in their imposed form or design. Marx (1867/1976) argues that the production, circulation, and consumption of artifacts as commodities is both affected by the prevailing social relations and reproduces those relations – and influences how we understand and learn about contemporary artifacts; these commodities are essentially stored labor – physical and intellectual – that comes alive in use. Marx traces the social history of artifacts from simple tools through machinery to computational automated industry. For Husserl (1936/1989) , meaning is established and historically sedimented in the form of artifacts; Heidegger (1927/1996) expands this analysis to argue that the life-world of our everyday involvements is structured as networks of meaningful artifacts. More recently, software is seen as a new form of stored meaning or intentionality (Keil-Slawik, 1992; Stahl, 1993; Winograd & Flores, 1986) . For instance, effects of “artificial intelligence” are accomplished by embedding human intelligence in software procedures and knowledge-bases.

Engelbart (1995) and Norman (1993) claim that it is artifacts that make us smart, by amplifying our very limited native abilities like short-term memory. Others (e.g., Cole & Griffin, 1980; Pea, 1985) counter that these artifacts change our tasks, rather than simply increasing our powers, but this still places artifacts centrally in our attempts to increase our intellectual capabilities. Donald (1991) argues that the entire enterprise of modern knowing and science only became possible with the development of artifacts like books, which provided external memories that could be circulated and that might outlive their creators. Papert (1980) , reflecting on his own learning history, believes that playing with automobile gears as a young child “did more for my mathematical development than anything I was taught in elementary school. Gears, serving as models, carried many otherwise abstract ideas into my head” (p. vi).

If one looks closely at learning – from infancy to kindergarten, formal schooling, and on-the-job – one sees that artifacts (now including computational artifacts) are pervasive. While it is clear that a primary function of education (and socialization into culture generally) is to teach new-comers how to understand and use the available artifacts of one’s society or of its specialties, we have only narrow studies of how this takes place. For instance, Bruner (1990) discusses how children acquire the ability to follow and generate narratives as verbal cognitive artifacts, and Hall (Hall & Stevens, 1995) investigates how young students use design tools.

How Artifacts are Understood

Even in our very preliminary pilot study of the SimRocket data, it has already become clear that the process of coming to understand a computer simulation that models a scientific phenomenon is a complex process, which strains the cognitive abilities of middle school students. Without strong guidance from a teacher, the students would at best have treated the simulation as a video game, perhaps competing to get the highest rocket flight, but not investigating the scientific factors that might lead to success.

Although students often make statements that sound like they understand how to construct certain kinds of knowledge, when one watches them struggling through the steps that are actually required one gains a much more detailed understanding of what is involved for a novice, what supports are helpful, and where problems typically arise. For instance, while the students in the pilot study were proficient at taking averages of sets of numbers in a traditional math lesson, they ran into many problems when averaging their rocket data. A major problem had to do with the organization of the data and of their averages on a data sheet. The two teams of students became very confused about which rocket heights had been observed by which team, and which averages were associated with them. While an adult experienced with scientific experiments can keep these things straight without thinking about it, the students had to learn this skill. They did this partially by negotiating with the teacher, who alerted them to problems and guided them back on track, and partially by collaboratively applying their own intellectual and communicative skills.

Our work and that of our current and past colleagues explores the use of gesture in understanding artifacts and in constructing shared understanding of artifacts. In his seminal example of micro-ethnographic analysis (which studies the interaction of five young children in a school project, and thereby provides a model for us) and subsequently, Streeck (1983; 1993; 1996) focuses on the roles of gesture in making social understanding visible. LeBaron analyzes different forms of gesture that are successively used to build a shared vocabulary of meaningful gestural artifacts (LeBaron, 1998; LeBaron & Hopper, 1997; LeBaron & Koschmann, 1999; LeBaron & Koschmann, 2001; LeBaron & Streeck, 2000) . Koschmann also highlights the role of gesture in educational settings (Koschmann et al., 1997; Koschmann & LeBaron, submitted; Koschmann et al., 1998; Koschmann & Stahl, 1998) . Our micro-ethnographic method (see below) is explicitly adapted to making learning visible by systematically attending to the sorts of gestures and bodily interactions that people use to co-construct the meaning of artifacts.

According to our theoretical framework, learning through interaction with artifacts is an inherently social process, involving either interaction with other people through the artifact or at least interacting with an artifact that was made by other people and that incorporates their intentions. For our research, collaborative interactions have an important characteristic: in order to collaborate, participants must make their ideas and their relationships visible to each other as part of their communication. That is, they make learning visible. As researchers, we can capture this in video or computer logs and analyze it. That way, we can see how students are relating to computational artifacts and what they are learning in the process. This overcomes the traditional problem of educational assessment, where it is assumed that learning is invisible to researchers and must be inferred from learning outcome measures. Thus, our approach avoids the restriction of educational assessment to the kinds of analyses of pre/post-test statistics and after-the-fact interviews that so often lead to “no significant difference” (Russell, 1999) results, which are of little value for design purposes.

Of course, not all learning is made visible, so other methods to indirectly measure learning outcomes are necessary and complementary. But focusing on the visible displays of learning prevents the common tendency to lose track of the learning in favor of secondary phenomena that seem easier to describe or quantify. For instance, much of the traditional literature on cooperative learning focuses on small group facilitation, rather than on cognitive and group learning processes (for a recent review of this literature, see (Brody & Davidson, 1998) reviewed by the PI (Stahl, 2000a) ). Even recent CSCL studies often miss the interesting learning phenomena (e.g., (Hakkarainen & Lipponen, in prep) and (Jong et al., in prep) , reviewed by the PI (Stahl, in prep) ).

Grounded Practical Theory

While we have encountered many suggestive ways of thinking about artifacts in our readings, the roles and functioning of artifacts are most clearly revealed by close observation of our data. We expect to come to a deep understanding of the role of artifacts in education – and conversely of the role of learning in artifacts – through our study of student interactions with educational artifacts.

Glaser & Strauss (1967) have described techniques for deriving theory from qualitative data in sociology. In philosophy, Gadamer (1960/1988) has proposed that hermeneutic understanding can be derived through reflection on life experience and situated interpretation. Schön (Schön, 1983; Schön, 1987) argues for reflective practice in professional activities like design and teaching.

Project co-PI Craig and his colleagues (Craig & Sanusi, 2000; Craig & Tracy, 1995) – building on Glaser & Strauss, Gadamer, Schön, and others – have developed an approach to grounded practical theory within communication analysis. The general idea of this approach is that practical theory – theory designed to inform praxis – involves conceptually reconstructing practice. This can be done on three levels:

1. A problem level that accounts for difficulties or dilemmas typically encountered in the practice.
2. A technical level that describes a repertoire of practical techniques for addressing problems.
3. A philosophic level that formulates normative principles to govern the use of techniques

For example, collaborative learning is a normative principle that can govern the use of practical techniques such as the SimRocket exercise. A problem noted in the pilot project was that middle school students may not collaborate toward certain desirable learning objectives without some guidance by the teacher (level 1). In the SimRocket data, we see a teacher using various interactional techniques that may display his orientation to this problem. To facilitate reflection on those techniques, the problem might be conceptualized theoretically as an instance of the more general dilemma of any pedagogical practice that attempts to be learner-centered while achieving specific learning objectives. "Scaffolding" names a general sort of technique that teachers can use to address this dilemma (level 2), but scaffolding can be, for example, either too directive (becoming teacher centered) or too nondirective (risking failure to achieve prescribed learning objectives). The collaborative learning principle (level 3) suggests a solution to the dilemma: the use of scaffolding techniques that focus the group's attention on a task that both structurally entails the prescribed learning objectives and requires active student collaboration to be completed. This may provide a principled basis for assessing the teacher's techniques in the SimRocket data, and also a principled basis for design revisions in the computational artifact (to better enable preferred forms of scaffolding). By the same token, the micro-ethnographic analysis provides a basis for assessing the relevance and applicability of this or any other theoretical reconstruction of the practice that might be proposed.

Such a grounded practical theory approach will guide us to:

1. Reflect upon problems that arise in the interactions we observe.
2. Define techniques that are responsive to these problems.
3. Formulate principled ways to move from empirical observations to software recommendations.

5. Research Methodology for Studying Interaction

Iterative Software Design

The core of the project is to develop a methodology for driving the iterative development of software for computer-mediated education. The idea is to start with an initial prototype, videotape small groups of students collaborating with the software, analyze the problems that arise as well as the kinds of learning that take place, formulate revisions to the software based on that analysis, and iterate system design (along with any associated recommendations for classroom presentation) toward improved learning.

Iterative design is a well-established approach in software development, particularly when the effectiveness of the software depends upon the ability of people to use it as intended. The problem is how to analyze the quality of usage in successive trials. This is best done by interpreting in a rigorous way how learning is taking place. During the past 25 years, scientific methodologies for interpreting social interaction have been developed. We focus on one particularly promising school of this science, micro-ethnography.

Micro-ethnography

For this project, we adopt a recent tradition of human interaction analysis (Jordan & Henderson, 1995) that we refer to as “micro-ethnography.” This methodology builds on a convergence of conversation analysis (Sacks, 1992) , ethnomethodology (Garfinkel, 1967) , nonverbal communication (Birdwhistell, 1970) , and context analysis (Kendon, 1990) . An integration of these methods has only recently become feasible with the availability of videotaping and digitization that records human interactions and facilitates their detailed analysis. It involves close attention to the role that various micro-behaviors – such as turn-taking, participation structures, gaze, posture, gestures, and manipulation of artifacts – play in the tacit organization of interpersonal interactions. Utterances made in interaction are analyzed as to how they shape and are shaped by the mutually intelligible encounter as a holistic context – rather than being taken as expressions of individuals’ psychological intentions or of external social rules (Streeck, 1983) . At the same time, micro-ethnography addresses larger social concerns, such as criminal justice (LeBaron & Hopper, 1997) , medical education (LeBaron & Koschmann, 1999) , and problem solving in complex technological settings (Hutchins & Palen, 1998) .

Micro-ethnographic research typically involves the following components:

1.        A specific setting, or research site – such as several students gathered around a computer running specific software.

2.        A detailed analysis of both audible and visible micro-behaviors, which are to be understood in terms of their embeddedness within the particular social and material environment – such as a classroom.

3.        A recognition that culture (which includes the meaning and use of shared artifacts) is a product and a process of naturally-occurring communication, simultaneously co-constructed and experienced by participants – and thereby made available for empirical study and interpretation by researchers.

4.        A use of recent technologies, like digitized video, that allow researchers to look at in detail the orderly performance of social life – such as the negotiation of learning between teacher and student or among collaborating peers.

We will build on this micro-ethnographic approach and on the expertise and methodology which has evolved through the micro-ethnographic data sessions conducted by the project co-PIs and their colleagues for several years. We will collect appropriate data and conduct our own data sessions for project staff, as we have already begun to do with our pilot study data.

Micro-ethnography can be adapted from the study of human-human interaction to that of human-computer interaction or computer-mediated collaboration. Our pilot studies suggest that such an adaptation of the methodology can be accomplished effectively. Our past use of micro-ethnography in collaborative educational settings – particularly in medical problem-based learning – has been very insightful and encouraging.

Micro-ethnography and Human-Computer Interaction

Our research approach brings together educational software designers and micro-analytic researchers. We use micro-ethnography to analyze empirical student interactions with educational software artifacts. Techniques related to micro-ethnography, such as video analysis and conversation analysis, have previously been used to analyze human-computer interaction in limited cases (Bødker, 1989; Bødker, 1996; Frohlich & Luff, 1990; Hollan et al., 2000; McIlvenny, 1990; Nardi, 1996; Suchman, 1987; Suchman & Trigg, 1991)