Gay

Using Multimedia Technologies in Evaluation

Geri Gay

Cornell University

Media- and technology-rich environments need equally rich data collection and analysis tools that can capture the system's complexity and provide useful representations of human-computer interactions. There seems to be much promise in the use of technology to evaluate technology. Researchers and evaluators have for many years relied upon computers in evaluations of multimedia environments. Early computer tracking systems allowed inquirers to record system usage, alert developers to problem areas, and save records of interactions for later use. Today's approaches have similar functions, plus the ability to intersect multimedia, network, and participatory environments. They not only allow evaluators to gather data from the electronic systems as they are being used, but also simplify and improve the integration of other data such as observations, interviews, documents produced, video and audio records, and so forth. The technologies support evaluators as they analyze the data and develop their interpretations. When combined, these data paint a rich and textured description of users, the multimedia, and the interactions that offers a holistic picture of the state of the artifact or program being evaluated.

In our research group, we consider the use of technology to evaluate technology, with an emphasis on recording user interactions with systems and gathering user feedback within the context of a multimedia environment. For examples of some of these evaluation tools in use in a KBE see:

Gay, G., & Lentini, M. (1995). Use of communication resources in a networked collaborative design environment. Acker, S.R. (1996). Space, and the credible city: Academic work in the virtual university. Journal of Computer-Mediated Communication [On-line], 1 (1). Available: http://jcmc.huji.ac.il/vol1/issue1/IMG_JCMC/ResourceUse.html

Gay, G., Sturgill, a. Martin, W. & Huttenlocher, D. (1999) Document-centered peer collaborations: An exploration of the educational uses of networked communication technologies. Journal of Computer-Mediated Communication [On-line], 4 (3). Available: http://www.ascusc.org/jcmc/vol4/issue3/Article.Conote.GGay/gay.html

The proliferation of computer and information technologies over the past decade has begun to transform the landscape of evaluation research practices. Data collection can now include web-based surveys, computer-mediated synchronous and asynchronous communication applications and reliance on organizational information systems and intranets. Evaluators have become proficient in the use of diverse computer technologies for quantitative data analysis, and increasingly for qualitative analysis (NUD*IST, QUALPRO, FolioViews, NVivo, etc.). Because of the pervasiveness of computer artifacts and computer-mediated communication in programs and in evaluation activities, we might say, following Ihde (1990, p.1), that our lives as evaluators are "technologically textured." Evaluation activities are embedded in complex technosystems that have significantly shaped how evaluations are designed and conducted. We use computer technologies and their multimedia functionalities for the collection of (multimedia) data, for its organization and analysis, and for our composition, presentation and dissemination of findings. Various groupware applications, can facilitate collaboration among evaluators and stakeholders, new ways of conducting evaluations, the integration of methods as well as organizational learning through formative evaluation inquiry.

For all evaluations of technology (and similar to evaluations of face-to-face programs), technology evaluators can rely on different kinds of data from a variety of sources, such as usage statistics, interviews with computer systems experts, and user focus groups. The claims that can be made based on this multi-method approach can be richer, better-substantiated, and more useful to readers.