Collaborative Learning through Computer-Mediated Argumentation

A.L. Veerman, J.E.B. Andriessen & G. Kanselaar

Abstract: This article reports on three studies that involved undergraduate students collaboratively working on authentic discussion tasks in synchronous and asynchronous Computer Mediated Communication systems (Netmeeting, Belvédère, Allaire Forums). The purposes of the assignments were respectively to develop insight and understanding in a theoretical framework, to co-construct meaningful didactics for a computer based training program and to develop insight and understanding in educational theories in relation to technology. To examine whether the use of the CMC systems could meet these ends, students’ dialogues are characterised in terms of their constructive and argumentative contributions and by their focus on the meaning and application of concepts. In addition, we compare different forms of support, from peer coaching synchronous discussions, offering graphical support at the interface to reflective moderation in asynchronous discussions. Although we compare different systems and assignments, the interplay between focused argumentation and constructive contributions allows us to analyse and compare interaction and learning in different CMC systems, with and without forms of pedagogical support.

Keywords: argumentation, computer-mediated communication, discussion forums

Introduction

The aim of the present research is to study constructive discussions in Computer Mediated Communication (CMC) systems in the context of regular academic courses. Constructive discussions in this context involve information exchanges in which information is constructed through addition, explanation, evaluation, transformation or summarising (Veerman, Andriessen & Kanselaar, in press). The main questions of our research are how to characterise such discussions on focused argumentation and constructive contributions and how to provide pedagogical support by humans or CMC systems.

An important issue in learning research is the construction of knowledge through negotiation. It is believed that learning is particularly effective when collaborating students encounter conflicts, engage in argumentation and manage through negotiation to produce a shared solution (e.g. Piaget, 1977; Doise & Mugny, 1984; Baker, 1996; Erkens, 1997; Savery & Duffy, 1996; Petraglia, 1997). In effective collaborative argumentation students share a focus on the same issues and negotiate about the meaning of each other's information. Incomplete, conflicting, doubted or disbelieved information is critically checked, challenged or countered on its strength (is the information true?) and its relevance (is the information appropriate?), until finally a shared answer, solution or concept arises. However, generating effective argumentation in educational situations is not always guaranteed. First of all, focusing is important for the interpretation and understanding of communication. Students have to initiate and maintain a shared focus of the task. They have to agree on the overall goal, descriptions of the current problem-state, and available problem-solving actions (Roschelle & Teasley, 1995). Failure to maintain a shared focus on themes and problems in the discussion results in a decrease of mutual problem solving (Baker & Bielaczyc, 1995; Erkens, 1997). Second, assessing information critically on its meaning, strength or relevance depends on many factors, such as the (peer) student, the role of the tutor, the type of task, the type of instruction and the selected medium (Veerman & Andriessen, in press). Key problems that may inhibit students to engage in critical argumentation are that students tend to believe in one overall correct solution, that students show difficulties with generating and comparing counter-arguments and with using strong, relevant and impersonalised justifications (Treasure-Jones, submitted thesis, p.13; Kuhn, 1991). In addition, students’ exposure of a critical attitude can be inhibited because of socially biased behaviour. For example, students may fear to loose face (e.g. in front of the classmates), to go against dominant persons in status or behaviour (e.g. a tutor), or for what other people think (e.g. that you are not a nice person)

To support and optimise students’ engagement in argumentative dialogues for learning purposes, Computer Mediated Communication systems (CMC) provide new educational opportunities. CMC systems are network-based computer systems offering electronic opportunities for group communication, such as Newsgroups, E-mail conferencing systems, Internet Relay Chat and Virtual Classrooms. Text-based and time-delayed communication can be beneficial to keep track and keep an overview of complex questions or problems under discussion. Text-based discussion is by necessity explicit and articulated. In addition to the chat windows a history of the dialogue can be used to reflect over time on earlier stated information. Contradictions, gaps or conflicts may be revealed through text-based and time-delayed discussion. In addition, in CMC systems students lack physical and psychological cues such as physical appearance, intonation, eye-contact, group identity etc. sometimes leads to democratising effects (Short, Willams & Christie, 1976; Kiesler 1986; Rutter, 1987; Spears & Lea, 1992; Smith, 1994; Steeples, Unsworth, Bryson, Goodyear, Riding, Fowell, Levy & Duffy, 1996). Critical behaviour, therefore, is expected to be less biased towards a tutor or a dominant peer-student than in face-to-face discussion.

However, it is unclear how the use of an electronic system, and which characteristics of such a system, relates to effective collaboration in learning situations. The purpose of this contribution is to analyse and compare the interplay between focusing, argumentation and learning in different CMC systems, with and without forms of pedagogical support. The research questions are:

(1) How can electronic dialogues, in synchronous and asynchronous CMC systems, be characterised in terms of focusing and argumentation and how does that relate to the production of constructive activities?

(2) How does pedagogical support, provided by humans or the CMC system, relate to constructive discussions in learning situations?

Three studies

We compare the results of three studies that involved undergraduate students collaboratively working on authentic discussion tasks in electronic environments as part of a 2 to 3 month course in Educational Technology and Computer Based Learning. In these tasks, students worked in pairs, triplets or small groups (8-12 students). With or without a (peer) coach present, they had to externalise (incomplete) knowledge, beliefs and values and to use each other as a source of knowledge and reflection in order to reach a (shared) solution. In sequence of admission, the tasks were aimed at the following goals:

(1) developing insight and understanding in a theoretical framework

(2) co-constructing meaningful didactics for a computer based training program

(3) developing insight and understanding in educational theories in relation to technology

The tasks were respectively conducted as a synchronous text-based discussion task, a synchronous text-based and graphical discussion task and an asynchronous text-based discussion task. Before we turn to some more detailed descriptions of these three studies, we first sketch our analysing system by explanation and an example.

Data Collection

All electronic sessions were logged automatically on the computer and were screened on the presence of content-related fragments, defined as written exchanges in which thematic information had been expressed in relation to the task goal. These fragments were subsequently analysed on focus, argumentative moves and constructive activities produced.

Focus categories were related to the task goals: the development of meaning of concepts and the use of conceptual knowledge to solve a problem or to support a claim. Two focus categories reflected this: (1) focus on the meaning of concepts, (2) focus on the use of concepts. In addition, the focus could be (3) on the task strategy (planning how to start the task, time management, how to carry out the task etc.). In addition, two categories of focus shifts were distinguished: focus shifts from understanding to the use of concepts and vice versa, and shifts from the meaning and use of concepts to the task strategy and vice versa.

The categories of information exchange indicated how argumentation was triggered. Considering several approaches in the field of analysing Educational Dialogue (including analyses of Dialogue Games, Exchange Structures and Communicative Acts, Argument and Rhetorical Structure; see Pilkington, McKendree, Pain & Brna, 1999; Treasure-Jones, submitted thesis; Erkens, 1997), we embodied six dialogue moves into our analysing system: statements, checks, challenges, counters, acceptances and conclusions. Although all these categories may embody elements of argument, we only consider checks, challenges and counters as argumentative information exchanges.

At the epistemological level, the discussions were analysed on types of constructive activities. We analysed goal-oriented activities in which relevant information was added, explained, or evaluated, summarising information and information transformations. Inter-judge reliability of the coding system showed a Cohen's kappa (Cohen, 1968) of 0.91 for the focus variable, a kappa of 0.89 for information exchange categories and a kappa of 0.74 for constructive activities

Example of analysis

In Figure 1 we present an authentic example from the first study of a content-related discussion fragment analysed with MEPA (Erkens, 1998), a tool developed for Multiple Episode Protocol Analysis. In this study students had to discuss a protocol of a tutoring session by using the Conversational Framework (Laurillard, 1993), a model for categorising teacher-student interaction. For technical reasons, messages longer than 2 lines are truncated to two lines in the screen dump.

Description of the analysis:

Student 1 (S1) starts a content-related discussion phase (F3: d1) and proposes to categorise sentence 4 of the protocol of the tutoring session. The focus is on the task strategy (where to start the discussion = c. strategy), the proposal is coded as a statement.

Student 2 (S2) agrees and states what category of the Conversation Framework (CF) fits sentence 4. S2 focuses on the use of the CF and adds content-related information (‘student tries to define a conception’).

S1 challenges S2 by proposing another category and adds information (‘…student decides what to do’).

S2 then counters S1 and the information is evaluated (‘…the student tries to define the task assignment. The student asks a question but there are no questions in the Conversational framework! So, this is not an adaptation towards an earlier action as a consequence of feedback…).

S1 shifts focus towards the meaning of concepts and checks understanding. New content-related information is added (… to jump from category 8 to 4 or … via adaptation or reflection).

Then, S1 shifts back to the use of concepts and checks mutual agreement. S2 agrees and accepts the choice for category 10.

In Table 1, an overview is given of the main differences considering the entities that affect argumentation in the three studies. The description of the three studies will follow next.

Study 1: Netmeeting

We integrated an experiment in an actual undergraduate course on Educational Technology. One of the learning goals in this course was to reach insight and understanding in the ‘Conversational Framework’ (Laurillard, 1993; p.102), a model that one can use for analysing and categorising teacher-student interaction. The model is considered to be discussible and can be interpreted in many different ways (Bostock, 1996). We designed a task using this framework in which students were first asked to individually analyse and code a dialogue with it. Then, they were randomly paired and confronted with each other’s codings and asked to analyse the same protocol together, using the synchronous electronic communication system NetMeeting. Students were instructed to finally reach shared answers.

Student pairs were assigned to three different conditions: a ‘structure’ peer-coaching condition, a ‘reflective’ peer-coaching condition and a control group (no coaching). The ‘structure’ coach focused on argument building, particularly on generating and comparing alternative and contrasting statements, arguments and elaborations. The ‘reflective’ coach focused on checking information on meaning, strength and relevance and on questioning connections between claims and arguments. We compared coached student pairs to a control group of non-coached student pairs (see for further detail: Veerman, Andriessen & Kanselaar, in press).

Study 2: Belvédère

Belvédère is a synchronous network-tool developed by the Learning Research and Development Centre at the University of Pittsburgh (Learning Research and Development Centre, 1996). Among many other applications, Belvédère can be used for constructing argumentative diagrams online with individuals or groups of students of any number (see: Figure 2). The working screen of the program displays a communication and a diagram construction window. To communicate with a partner the student has chat-box, similar to the Netmeeting system, in which messages can be created, sent and are displayed in the shared chat-history. Adding data into the diagram window is constrained; students must use the predefined set of boxes ('hypothesis', 'data', 'unspecified') and links ('for', 'against', 'and').

Thus, Belvédère provides students with a tool for non-linear argumentative diagram construction in addition to linear chat discussion. Since an argument is not linear by nature (e.g. Adam, 1992), Belvédère may be especially good for mediating argumentative discussions. Research has shown that graphical representations tend to foster comprehension when they support a focus on salient and important features of the task (Ghyselink & Tardieu, 1999; Reimann, 1999). The diagram construction tool, therefore, may help students to organise their argument and to keep track of the main issues under discussion, including unjustified statements, unclear information, gaps or conflicts that can trigger discussion (Veerman & Treasure-Jones, 1999).

We integrated the Belvédère study in an undergraduate course on Computer Based Learning. Small groups of students (of 2 or 3 persons) were instructed to use the Belvédère system for 60 - 90 minute discussions about learning goals and pedagogical aspects concerning their own design of a Computer Based Learning program. They were asked to submit their finally constructed diagrams to the tutor. We analysed (1) students' discussions on focus, argumentation and the production of constructive activities, (2) students' diagrams on organisation and overlap of information with the chat discussion and (3) on the relationship between types of chat discussions and types of diagrams constructed (see for further detail Veerman, Andriessen & Kanselaar, 1999).

Study 3: Forumdiscussions

Allaire Forums is an asynchronous network-tool that can be used for electronic group discussion. In Allaire Forums conferences can be created, which embody single or multiple ‘threads’ and 'branches'. In Figure 3, a thread is shown with several messages sent and ‘branched’.

We integrated a study about asynchronous Forum discussions in a 3-month undergraduate course on Educational Technology. The learning goal was for students to integrate personal experiences, theoretical knowledge and expert insights to reach conceptual understanding and insight and to use conceptual knowledge for application into educational practice. We created two conferences with discussions considering either theoretical or applied aspects of educational technology. Every two weeks, in each conference we created two to three new threads embodying strong statements for discussion, e.g. "simulation based learning fits the principles of a constructivist learning environment" or "human tutors adapt better to students’ learning styles than intelligent tutoring systems". These statements were based upon the literature students were supposed to study.

Students were assigned to different theoretical and practical discussions (8 — 12 students per discussion). They were provided with computers at home and had access to Allaire forums 24 hours a day. Some discussions embodied a ‘reflective’ moderator where others did not. A ‘reflective’ moderator focused on checking information on meaning, strength and relevance and on questioning connections between claims and arguments (see e.g. Veerman, Andriessen & Kanselaar, in press). In addition, the moderator summarised information when a shared thematic focus in regard to the learning goal was lost. In sum, 28 discussions were collected. We compared ‘highly’ moderated student discussions (at maximum 25% of the messages was sent by a moderator) to less moderated and not moderated discussions. Thus, we could analyse (1) the effects of moderation interventions on students’ focused discussion and the production of constructive activities within each discussion, and (2) the effects of faded moderation interventions on students’ behaviour.

It is obvious that we are comparing different systems, and different assignments, and this does not allow strict comparison of results. However, the interplay between focusing, argumentation and constructive activities is studied in all cases, and a comparison may lead to interesting hypotheses for further research.

Results

Conclusions

In this study, we analysed and compared academic discussions in synchronous and asynchronous Computer Mediated Communication systems (Netmeeting, Belvédère, and Allaire Forums) on the interplay of focused argumentation and the production of constructive activities (as a definition for learning in process).

The results indicate that, first of all, all discussions were highly argumentative. From our point of view, this can be an effect of task characteristics that provoke and structure argumentation to a greater extent than to interface characteristics (Veerman & Treasure-Jones, 1999). Second, students' learning from electronic discussion requires analysing focus in relation to argumentation. Argumentative information exchanges are not solely related to the production of constructive activities. Thirdly, considering focusing, argumentation and the production of constructive activities asynchronous and synchronous discussions could be characterised differently. In both the synchronous NetMeeting and Belvédère system, students engaged into short and strong discussions. They checked, challenged and countered each other's information and produced additions and evaluations. The Belvédère system appeared to stimulate students to check and counter each other's information most frequently and to focus strongly on the meaning of concepts. The asynchronous Forum discussions could be characterised as highly constructive, students tend to argue 'mildly' by information checks and they mainly added or explained information. Students focused on and shifted focus between the meaning and use of concepts. Across studies, student groups could be clustered as groups of Conceptualisers (engaged in meaningful discussion), Conceptual Achievers (meaningful and task-oriented) and Achievers (task-oriented; aimed finishing the task). Forum discussions only embodied student groups that were labelled as Conceptualisers or Conceptual Achievers. The NetMeeting and Belvédère discussions embodied all types of student groups, however, the Belvédère discussions were more conceptually oriented. Remarkably, in neither the asynchronous nor the synchronous discussions students were driven to produce summaries or information transformation. This may be due to required cognitive effort but also to an incomplete, intuitive and personalised understanding of information under discussion (Treasure-Jones, submitted thesis; Kuhn, 1991). Students need sufficient understanding of a topic and a mutual framework for interpreting each other's information before they can state firm positions and stick to a position (Coirier, Andriessen & Chanquoy, 1999). To reach new insights, there must be a certain level of (shared) understanding. In our studies, considering task characteristics, students' preparation time, prior knowledge and time available for discussion, reaching (deeper) understanding may have been the highest goal reachable, especially through the use of asynchronous CMC systems. Reaching new insights may be just the next step, for instance, when students are sufficiently prepared to take firm positions to engage in critical, strong and hefty argumentation (triggered best by synchronous CMC systems?).

To conclude with, reflective support offered by human moderation or structured (graphical) interaction at the interface may help students to engage in meaningful discussion. Students can be enhanced to engage in conceptually focused critical argumentation. However, we don't think that educational task- or software designers have to embrace all problems we found. E.g. spending energy on finding support for the so-called Achievers may be a waste of time. Such students can be found across every type of task and CMC system. Of course, software designers have to consider students' attitudes, motivation and self-regulation when developing a system, but we also think that academic students have to take responsibility for their own learning, especially in interactive learning scenarios (Andriessen & Sandberg, 1999).

Bibliography

Baker, M. (1996). Argumentation and cognitive change in collaborative problem-solving dialogues. COAST Research Report Number CR-13/96, France.

Baker, M., & Bielaczyc, K. (1995). Missed opportunities for learning in collaborative problem-solving interactions. In J. Greer (Ed.), Proceedings of AI-ED 95 - 7th World Conference on Artificial Intelligence in Education (pp. 210-218). Charlottesville: Association for the Advancement of Computing in Education (AACE).

Cohen, J. (1968). Weighted kappa: nominal scale agreement with provisions for scales disagreement of partial credit. Psychological Bulletin 70, 213-220.

Coirier, P., Andriessen, J.E.B., & Chanquoy, L. (1999). From planning to translating: The specificity of argumentative writing. In: P. Coirier & J.E.B.Andriessen (Eds.) Foundations of argumentative text processing. Amsterdam: Amsterdam University Press.

Doise, W. & Mugny, G. (1984). The social development of the intellect. Oxford: Pergamon Press.

Erkens, G. (1997). Cooperatief probleemoplossen met computers in het onderwijs: Het modelleren van cooperatieve dialogen voor de ontwikkeling van intelligente onderwijssystemen [Cooperative problem solving with computers in education: Modelling of cooperative dialogues for the design of intelligent educational systems]. Ph.D. thesis, Utrecht University, the Netherlands.

Erkens, G. (1998). Multiple Episode Protocol Analysis (MEPA 3.0). Internal publication. Department of Educational Sciences, Utrecht University, the Netherlands.

Kiesler, S. (1986). The hidden message in computer networks. Harvard Business Review, 64 (1), 46-58.

Kuhn, D. (1991). The Skills of Argument. Cambridge: University Press.

Laurillard, D. (1993). Rethinking university teaching: a framework for the effective use of educational technology. Routledge, London.

Learning Research and Development Center (1996). Advanced Cognitive Tools for Learning. http://advlearn.lrdc.pitt.edu/belvedere/index.html . University of Pittsburgh, Pittsburgh.

Mason, R. (1992). The textuality of computer networking. In R. Mason (Ed.), Computer Conferencing: The Last Word. Victoria, British Columbia: Beach Holme Publishers Limited.

Piaget, J. (1977). The development of thought: Equilibration of cognitive structures. New York: Viking Penguin.

Petraglia, J. (1997). The rhetoric and technology of authenticity in education. Mahwah, NJ: Lawrence Erlbaum.

Pilkington, R., McKendree, J., Pain, H., Brna, P. (1999). Analysing educational dialogue interaction: towards models that support learning. ‘A One day Workshop at AI-Ed '99’, 9th International Conference on Artificial Intelligence in Education, Le Mans, France. July 19-22, 1999.

Roschelle, J. & Teasley, S.D. (1995). Construction of shared knowledge in collaborative problem solving. In C. O'Malley (Ed.), Computer-supported collaborative learning. New York: Springer-Verlag.

Rutter, D. R. (1987). Communicating by Telephone. Oxford: Pergamon Press.

Savery J. & Duffy, T. M. (1996). Problem based learning: An instructional model and its constructivist framework. In B. Wilson (Ed.) Constructivist learning environments: Case studies in instructional design (pp. 135-148). Englewood Cliffs, NJ: Educational Technology Publications.

Short, J., E. Williams, and B. Christie (1976). The social psychology of telecommunications. New York: Wiley.

Smith, J.B. (1994). Collective Intelligence in Computer-Based Collaboration. Mahwah, NJ: Lawrence Erlbaum.

Spears, R. & Lea, M. (1992) Social influence and the influence of the social in computer mediated communication In: Context of computer mediated communication. Ed: Martin Lea, Bodmin, Great Brittain.

Steeples, C., Unsworth, C., Bryson, M., Goodyear, P., Riding, P., Fowell, S., Levy, P., & Duffy, C. (1996). Technological support for teaching and learning: Computer-mediated communications in higher education (CMC in HE). Computers & Education, 26 (1-3), 71-80.

Treasure-Jones, T. (submitted thesis). Supporting students’ critical essay planning: towards a computer based approach. Computer Based Learning Unit, The University of Leeds.

Veerman, A.L. & Andriessen, J.E.B. (in press). Collaborative argumentation in academic education. Learning to Argue in Higher Education, Heinemann-Boynton/Cook book.

Veerman, A.L., Andriessen, J.E.B. & Kanselaar, G. (1999). Computer-mediated discussions through graphically structured interaction. Presented at the 8th European Conference for Research on learning and Instruction. August 24-28, Göteborg, Sweden.

Veerman, A.L. & Treasure-Jones, T. (1999). Software for problem solving through collaborative argumentation. In P. Coirier & J.E.B. Andriessen (Eds.), Foundations of argumentative text processing. Amsterdam University Press, the Netherlands.

Veerman, A.L., Andriessen, J.E.B. & Kanselaar, G. (acceptedin press) Enhancing learning through synchronous discussion. Special Issue of Computers & Education - CAL99 conference-. Elsevier Science Ltd., London.

Authors’ addresses