In Proceedings of the Computer Support for Collaborative Learning (CSCL) 1999 Conference, C. Hoadley & J. Roschelle (Eds.) Dec. 12-15, Stanford University, Palo Alto, California. Mahwah, NJ: Lawrence Erlbaum Associates.

Design and Evaluation of a Collaborative Telelearning Activity aimed at Teacher Training

Barbara Wasson

University of Bergen, Department of Information Science

Abstract: This paper describes VisArt, a collaborative telelearning scenario aimed at teacher training. Students at three educational institutions in Norway collaborated at a distance through TeamWave Workplace to design a learning activity for some subject of their choice. At the University of Bergen, VisArt was part of a graduate course assignment where the students had to participate in VisArt and write a report reflecting on their participation from a theoretical perspective. Salomon’s ideas on genuine interdependence and Gutwin et al.’s notions of awareness formed the foundation for the student’s theoretical reflection. This paper describes the VisArt scenario, the evaluation studies being carried out within the DoCTA project, and summarises the student’s own theoretical reflections.
Keywords: teacher professional development, evaluation, groupware

Introduction

In the Spring 1999 pedagogical information science course at the University of Bergen (see http://www.ifi.uib.no/staff/barbara/courses/pivar99.html), one of the two semester assignments involved participation in VisArt, a collaborative telelearning activity. In VisArt, distributed teams of students collaborate over the internet in a common design activity. VisArt is one scenario in the DoCTA (http://www.ifi.uib.no/projects/docta) research project that is lead by the author. The collaborative telelearning assignment required the students to not only participate in the VisArt activity, but evaluate their own participation from a theoretical perspective.

The paper proceeds by describing the DoCTA project and the theoretical framework within which the semester assignment is situated. Then VisArt and the collaborative telelearning assignment are described and its evaluation discussed. The paper concludes with a summary of some of the student’s reflections on the VisArt scenario.

Project DoCTA

DoCTA (Design and use of Collaborative Telelearning Artefacts) is a multidisciplinary research project administered and co-ordinated by the Department of Information Science at the University of Bergen. The project is funded by KUF's ITU (IT in education) programme, and is a collaboration with HiNT (Nord-Trøndelag College), HSH (Stord / Haugesund College) and Telenor Research & Development, Kjeller. The 16 researchers and graduate students involved have various backgrounds including computer science, psychology, sociology and education.

The project focuses on the design and use of technological artefacts to support collaborative telelearning aimed at teacher training. The research is not limited to only studying these artefacts per se, but includes social, cultural, pedagogical and psychological aspects of the entire process in which these artefacts are an integral part. This means that we both provide and study virtual learning environments that are being deployed to students organised in geographically distributed teams. The main research focus is reflected in both the theoretical and methodological approach chosen in the project. The theoretical, or conceptual approach, is rooted in a sociocultural perspective that emphasises an understanding of language, culture and other aspects of social setting. The methodology is influenced by ethnographic studies, favouring naturalistic and qualitative research methods.

Various scenarios utilising the Internet are used to engage the students in collaborative learning activities. Through participation, teachers gain experience with not only collaborative learning, but with collaborative telelearning, and the design of textual or visual artefacts. Each collaborative learning activity has been designed to place a strong emphasis on active engagement through both "hands-on" practical experience and explicit reflection on the collaboration process. Experiences can be transferred back to their own schools, and ideas about collaboration can be integrated into their everyday teaching.

Four different collaborative telelearning scenarios are studied. The first scenario, PedInfo, is a pilot study for analysing the use of TeamWave Workplace (TW, http://www.teamwave.com) for collaborative activities in a graduate university course at the University of Bergen (UiB). The next two scenarios, IDEELS and Demeter, involve European inter-cultural simulations where the goal is to design a textual artefact (such as a treaty or policy statement). A fourth scenario, VisArt, which has been designed and developed explicitly for use between the three educational partners (UiB, HiNT and Stord), has the goal of designing a visual artefact to be used in teaching a subject of choice. In the remainder of this paper we focus on the VisArt scenario.

Theoretical Foundations

The theoretical perspectives that provide inspiration and guidance for this research come from: computer supported collaborative learning (CSCL), in particular Salomon’s (1992) work on genuine interdependence; computer supported collaborative work (CSCW), in particular Gutwin et al.’s (1995) ideas on awareness; coordination science (Malone & Crowston, 1994); sociocultural perspectives (Wertsch, del Río & Alvarez, 1995) on learning and thinking; and, the emerging notion of distributed learning communities.

Computer supported collaborative learning (CSCL) is an emerging paradigm (Koschmann, 1996) for research in educational technology that focuses on the use of information and communications technology (ICT) as a mediational tool within collaborative methods (e.g., peer learning and tutoring, reciprocal teaching, project- or problem-based learning, simulations, games) of learning. It is an approach to ICT in education that emphasises an understanding of language, culture and other aspects of the social setting (Scott, Cole & Engel, 1992). Its intellectual heritage can be found in social constructivism (Doise, 1990), the Soviet cultural-historical psychology (e.g., Vygotsky (1978), Leontiev (1978), Davydov (1988)) and situated cognition (Suchman, 1987; Lave, 1988).

CSCL research involves naturalistic observations being used in an exploratory fashion to permit a more complete understanding of this instructional mode. The emphasis is on the process and not so much focused on outcome. Evaluations often result in descriptive studies which focus on artefacts that support or are produced by teams of learners and usually contain participant accounts of their own work. The influence of CSCL research on DoCTA is evident in the type of research questions we ask and in the choice of conceptual framework for organising our evaluations.

Guribye & Wasson (1999) describe the underlying conceptual framework adopted in DoCTA as an integration of three different, although closely interrelated approaches: activity theory (Leontev, 1978, Engeström 1987), distributed cognition (Hutchins, 1995), and situated action (Suchman, 1987, Lave, 1988, Mantovani, 1996). All three approaches underscore the need to look at real activities in real situations (Nardi, 1996, our italics), and always, in some way, include the context in studies of human activity. One of the goals of this research is to argue that, together, these approaches make up a rich framework for describing, evaluating and analysing collaborative telelearning scenarios.

Salomon (1992) is concerned with effective collaboration and he argues that collaboration will only be effective if there is genuine interdependence between the collaborating students. Genuine interdependence is described as 1) the necessity to share information, meanings conceptions and conclusion, 2) the necessity for division of labour into complementary roles, and 3) the need for joint thinking in explicit terms. In project DoCTA, Salomon’s ideas influence the design of the collaborative tasks given to the students — effort is placed in designing tasks that create genuine interdependence between the students.

Coordination theory (Malone & Crowston, 1994) provides a means for specifying (inter)dependencies between, and among, actors, goals, activities, and resources by identifying a dependency type (e.g., shared resource) and a coordination process (e.g., group decision-making) for managing the dependency. In their work, coordination is defined as managing dependencies between activities, hence they have focused on dependence between activities. Wasson & Bourdeau (1997) report that viewing collaborative telelearning from a coordination theory perspective offers a means of understanding the inter-relationships between actors and entities and how these relationships can and should be supported. Adopting Salomon’s ideas about genuine interdependence and a coordination science approach, they modelled (inter)dependencies between actors in collaborative telelearning scenarios and they have extended the definition of coordination to be managing dependencies between activities (Malone & Crowston, 1994) and supporting (inter)dependencies between actors. Wasson (1997, 1998) proposes a set of actor (inter)dependencies and related coordination processes for collaborative telelearning. The importance of this work for DoCTA lies in making sure the technological environment within which the students carry out their tasks provides mechanisms that make coordination as effortless as possible. Otherwise coordination issues can become a bottleneck that hinders students from carrying out their individual and collaborative work.

Gutwin et al. (1995) present a framework of awareness for collaborative learning which comprises four types of awareness: social awareness, task awareness, concept awareness and workspace awareness. Social awareness refers to awareness of the social connections within the group or team (e.g., What role will I take in the team, what roles will the other members take). Task awareness concerns awareness about how to complete the common task (e.g., What do we know about the task, how much time do we have, what steps must we take to complete the task). With concept awareness the individual student is aware of how a particular activity or piece of knowledge fits into his or her own existing knowledge. Workspace awareness is an up-to-the-minute knowledge of the other students’ interactions with the workspace (e.g., what are they doing, where are they, what has been done so far, what is left to do). They suggest that social awareness, being inter-personal, is best supported implicitly by providing communication opportunities for team members to negotiate their roles. Task and concept awareness are often supported by providing explicit scaffolds to assist with organisation and helping stay on task. Workplace awareness is maintained by providing "tracking information such as other learners’ location in the shared workspace, their actions, the interaction history and their intentions (Gutwin et al., 1995, p. 147).

The final area providing inspiration is summarised as follows "…a distributed collaborative learning community is a ‘place’ that is created by the individual students through their individual and collective actions, …The designers’ role is to support the students’ work of creating that community, and in such a way that the computer systems become integrated parts of the students’ activity (Fjuk, 1998, p. 70)". Furthermore, Fjuk (1995) concludes that collaborative telelearning applications need to have both a mediating role between the individual learner and the peer-students and between the individual learner and her learning tasks. Thus, collaborative telelearning can be understood as a medium for inter-human interactions and articulation of individual work. As designers of DoCTA learning tasks and the technological support environments, we must keep in mind that our role is to provide a supporting environment that makes coordination, communication and collaboration as transparent as possible (Bourdeau & Wasson, 1997) enabling students to create their own learning community. The environment must also support both individual and collaborative work.

VisArt

The VisArt scenario involved students taking courses at three educational institutions (University of Bergen, Høgskolen I Nord-Trøndelag, Stord/Haugesund Høgskolen) in Norway. Teams comprised of 3 students, 1 student from each institute, collaborated to design a learning activity in TW. There were no opportunities for the teams to meet face-to-face. TeamWaveWorkplace (TW) was used as the main information and communication technology. The VisArt activity took place during February and March 1999 and provided an opportunity for the DoCTA project to study an authentic collaborative telelearning activity. One week of training in using the TW tool and in collaboration proceeded three weeks of design activity. Figures 1 and 2 give examples of the TW working room and designed learning room designed by Team07.

Participation at the University of Bergen

The University of Bergen students participated in VisArt through the pedagogical information science course given in the Spring of 1999. The first semester assignment on collaborative telelearning involved participating in the VisArt scenario. Participation alone, however, was not enough. Since this is a graduate course and two of the topics the students were learning about were collaborative learning and telelearning, the VisArt scenario provided an opportunity for both participation in collaborative telelearning and an opportunity for theoretical reflection. A semester assignment was designed to cover both opportunities.

The students were given literature on CSCL (including Salomon (1992) & Gutwin et al. (1995)) and were asked to organise a team effort thinking about Salomon’s definition of genuine interdependence (sharing information, division of labour, and joint thinking). They were to participate in the VisArt design activity for 1 month and then were to write an individual report on the experience.

Figure 1 Team 7’s working room in TeamWave

Figure 2 Team 7’s room for learning about polar bears

Participant’s Profile

The learner’s participating in the scenario had different backgrounds, ranged in age from 23 to 68 years and many had family responsibilities and full time jobs. The University of Bergen students were taking a graduate course in pedagogical information science and were learning about collaborative telelearning. They were a blend of pedagogical information science graduate students (with a teacher’s background) and information science graduate students (with a social science background). The students at Stord Teacher’s College, were senior undergraduate students training to be teachers who were taking a distance learning course on pedagogical information science and were also learning about collaborative learning. Finally, the Nord-Trøndelag students were taking an undergraduate introduction course to uses of technology in learning and had a choice to participate in the scenario to learn about a telelearning application.

Evaluation of VisArt

The evaluation of the VisArt scenario is being carried out on several levels and from several perspectives. In addition to the student’s own theoretical reflections (described in the next section), VisArt is being evaluated as part of the DoCTA project. Eight Master’s theses are being carried out. These theses include usability studies of TW, looking at the efficiency of TW from a qualitative perspective using the data logs generated by TW, performing a formative evaluation of how to support collaborative design activities, seeing how TW supports coordination, how to design training and assistance in a collaborative telelearning setting, and several activity theory studies of how students, instructors and facilitators organise their work. Table 1 summarises the research questions of the 8 graduate students. Numerous data and data collection techniques have been used including questionnaires, semi-structured interviews, log files produced automatically by TW, email send among team members, chat files saved by the teams, documents and artefacts produced by the teams, participant observations, video taping of activities, self-evaluations and the participants own interpretation of the experience.

The learners’ experiences

A summary of the learners’ experiences as described in their individual reports is given in the following. In particular, attention is paid to how their group met Salomon’s criteria for genuine interdependence and how TW supported Gutwin’s ideas on awareness.

In general, the students were very satisfied with TW. As one student writes

"An important side with TeamWave is that one can work both asynchronously and synchronously. … For example one can use the shared whiteboard synchronously when the users are online at the same time and write on it together, but it is also possible to use the whiteboard asynchoronously when the different users log on at different times and work individually on tasks on the whiteboard. …That it supports both forms of work makes the program package flexible and accessible at all times."

Several students wrote that the successful use of TW was not just tied to the ease of use, rather, that it is used in an activity that meets Salomon’s requirements. As one student succinctly put it

"I think that a requirement for successful use of TW is that the participants are motivated and have mindful engagement and that the tool is used for something meaningful."

Table 1 Research Questions for Master’s theses related to the VisArt scenario

Domain/theoretical framework

Research Question

Usability of TeamWave Workplace

Usability — effectiveness

Usability — efficiency

Usability - satisfaction

Can students working in collaboration reach their team goals with TW?

To what extent do the tools enable students to meet their task?

Are students satisfied of TW tools?

Efficiency from a qualitative perspective

Comparison of anticipated use and actual use of the tools

Research methodology

What are the differences between the anticipated use of the tools and the actual use of the tools? Which are the differences of tool use when alone and when others in the room? How are the rooms used?

What implications do the use of the tools have for electronic data-collection?

Formative evaluation

 

Tailoring for instructional design

User centred design

How to support pedagogical room design in TW? How do students use TW tools to design a room for teaching?

What is to be improved?

Coordination theory

 

Supporting coordination

User centred design

How to support coordination in a collaborative telelearning environment?

What kind of coordination do students do?

Activity theory

 

Organisation of work — student perspective

Organisation of work — Instructor`s perspective

How do the students organise their work?

How instructors and facilitators organise their work

Design of training and assistance

 

Assistance design/ improvement

How effective is the training designed for collaborating and learning to use TW? How to improve help and assistance?

 

Sharing of information

The majority of the groups had a heterogeneous makeup with the group members having different backgrounds. As one group said, this meant that they had different preconceptions and different experiences with collaboration. They said, that

"according to Salomon it is exactly these differences that makes collaboration work…to use each others competence and pull something useful of these competencies through collaboration."

When it came to TW supporting the sharing of information, it was possible to do so both synchronously and asynchronously. One particular group worked mostly synchronously (with respect to sharing and they meant this was time-consuming. Often they would meet at a pre-arranged time and have little to say to one another and this was a disadvantage as it required that they bind themselves to meeting at a particular time. It would have been

"better if we could have worked a little bit more asynchronously, and insead given each other feedback, for example by using a PostIt, when we were logged on to work with one of our own tasks. TW provides several tools to support this asynchoronous communication…but our use of this functioned badly".information)
Division of labor

With regard to division of labor, one of the students said the following:

"TW was a good support for division of labour in those cases where it was not necessary to be in our common work area at the same time. If one person started on a task, then left the work area, the next person could come in an continue working on the task since there was a common work area."
Joint thinking

One of the teams said that the requirement for joint thinking in explicit terms that can be examined, changed, and elaborated upon by peers was supported in TW through the Chat, Postit and Brainstorm tools. For them

"This particular need was satisfied through the many online joint discussions between team members. The fact that team members had the opportunity to express their ideas, conceptions, meanings, etc., in clear and observable terms (by means of Postit and Chat artifacts), enables each one of us to analyze, expand on, or change these as they were being developed."
Awareness

The students were in general agreement that TW supported many of the aspects of awareness as identified by Gutwin et al. (1995). As one student points out, it is not TeamWave Workplace alone that supported social awareness, but the combination with some of the training activities that were designed for learning about TW and about each other that supported social awareness. He says

"The social awareness tied to expected roles is first and foremost tied to personal relationships. I feel that the training activities we carried out in TeamWave were important for this. Through informal chats we were able to each get a feeling of what we could expect of each other and what type of interaction and tone we would have between us. … It was never explicitly said that "I am … and you are …", but over time we picked up aspects of each others personaility and found our own roles."

According to the students TW supported workspace awareness by several techniques. The Information Areas provided awareness about the other users presently connected to the server, and indicated who was in the same room. The coloured cursor on the white board made it easy to follow what each of the other users who are in the same room are doing, and if another user is typing on the Whiteboard, PostIt, or Brainstormer, you see what they are typing immediately. One of the other students meant that TW’s Chat tool

"provides for some of the social aspects of collaboration. We used it for informal discussions around different themes, both scientific and social. Through the common mileu that a room in TeamWave enables, both task and conceptual awareness are fulfilled."

Another student said that

"The opportunities provided by TeamWave Workplace in dealing with the issues in social awareness were generally poor. Support for interactivity in information exchange and negotiation of roles, which is fundamental given the interpersonal nature of social awareness, was limited to artifacts such as Chat and Postit…In the case of concept and task awareness, the requirement within CSCL environments is for explicit artifacts that can be used as scaffolds to help students with task organisation, providde them with concentration aids and step-by-step guidelines for learning tasks. TeamWave’s support for both concept and task awareness is provided by tools such as ToDoList and Concept Map (for task organisation, and Message Board (For structured message capabilities)."

Conclusions

It is early in the evaluation of VisArt to make many conclusions. From a sociocultural research perspective, however, the student’s own reflections are a very important part of evaluation and as illustrated in the previous sections they demonstrated an ability to reflect theoretically on practice. The research reports they submitted in the course contained comments and reflections that were both thoughtful and insightful and will lead to improvements in future versions of the scenario. In general our first impressions of our research data indicate that the students were positive to both the distributed collaboration and to the groupware tool TeamWave Workplace. Results from the Master’s theses evaluations will be available from late 1999 and at that point more concrete conclusions will be made.

Acknowledgments

DoCTA is funded by The Norwegian Ministry of Education, Research and Church Affairs (KUF) under their Information Technology in Education (ITU) programme. The author would like to thank all those involved in the VisArt scenario: Anders Mørch, Arnstein Eidsmo, Glenn Munkevold, Lars Vavik, Eskil Andreassen, Rune Baggetun, Frode Guribye, Hege Higraff, Øyvind Meistad, Trond Pedersen, Kurt Rysjedal, Helge Underhaug, and Jo Wake.

Bibliography

Bourdeau, J. & Wasson, B. (1997). Orchestrating collaboration in collaborative telelearning. In B. du Boulay & R. Mizoguchi (Eds.) Proceedings of the 8th World Conference on Artificial Intelligence in Education, 565-567. Amsterdam: IOS Press.

Davydov, V. (1988). Learning activity: The main problems needing further research. Activity Theory, 1(1-2), 29-36.

Doise, W. 1990. The development of individual competencies through social interaction. In H.C. Foot, M.J. Morgan & R.H. Shute (Eds.) Children helping children. Chichester: J. Wiley and Sons. 43-64.

Engeström, Y (1987). Learning By Expanding: An activity-theoretical approach to developmental research. Helsinki: Orienta-Konsultit Oy

Fjuk, A. (1995).Towards an Analytical Framework for CSCdistanceL. In J.L. Schnase & E.L. Cunnius (Eds.) Proceedings of CSCL'95, 130-134. Mahwah, NJ: Lawrence Erlbaum Associates.

Fjuk, A. (1998). Computer support for distributed collaborative learning. Ph.D thesis, Dept. of Informatics, University of Oslo.

Guribye, F. & Wasson, B. (1999). Evaluating collaborative telelearning scenarios: A sociocultural perspective. To appear at EdMedia’99.

Hutchins, E. (1995).Cognition in the Wild. Cambridge, MA: MIT Press.

Koschmann, T. (1996). Paradigm shifts and instructional technology: An introduction. In T. Koschmann (Ed.) CSCL: Theory and practice of an emerging paradigm, 1-23. Mahwah, NJ: Lawrence Erlbaum Associates.

Lave, J. (1988). Cognition in practice. Cambridge: Cambridge University Press.

Leont’ev, A. N. (1978).Activity, Consciousness, Personality. Englewood Cliffs,NJ: Prentice Hall.

Malone, T. & Crowston, K. (1994). The Interdisciplinary study of coordination. ACM Computing Surveys, 26(1), 87-119.

Mantovani, G (1996).New Communication Environments: From Everyday to Virtual. London:Taylor & Francis Ltd.

Nardi, B. A. (1996).Studying Context: A comparison of activity theory, situated action models and distributed cognition. In Nardi, B. (Ed.) Context and Consciousness: Activity Theory and Human-computer Interaction. Cambridge, MA: MIT Press

Salomon, G. (1992). What does the design of effective CSCL require and how do we study its effects? SIGCUE Outlook, Special Issue on CSCL, 21(3), 62-68.

Suchman, L. (1987). Plans and situated action. The problems of human-machine communication. Cambridge: Cambridge University Press.

Vygotsky, L.S. (1978). Mind in society: The development of higher psychological processes. (M. Cole, V. John-Steiner, S. Scribner & E. Souberman, (Eds. and Translators). Cambridge, MA: Harvard University Press.

Wasson, B. (1997). Developing a model of (inter)dependencies for collaborative telelearning. LICEF Research Report, Télé-université, Montreal, Canada.

Wasson, B. (1998). Identifying Coordination Agents for Collaborative Telelearning, International Journal of Artificial Intelligence in Education, 9, 275-299.

Wasson, B. & Bourdeau, J. (1998). Modelling actor (inter)dependence in collaborative telelearning. Full paper to appear in Proceedings of Ed-Media’98, Freiburg, Germany.

Wertsch, J. V., del Río, P. & Alvarez, A. (1995).Sociocultural studies: history, action and mediation. In Wertsch, J. V., del Río, P. & Alvarez, A. Sociocultural Studies of Mind. Cambridge University Press.

Author’s address

Barbara Wasson (barbara.wasson@ifi.uib.no)
Department of Information Science; University of Bergen; P.O. Box 7800; N-5020 Bergen, Norway. Tel. +47 55 58 41 20. Fax +47 55 58 4107.

Powered by CILTKN