The Less Simple Answer to Evaluating Technology's Impact

Recognizing technology's four major uses is the first step to assessing its value

Schools have been spending a good deal of money on technology over the past few years. One estimate puts the amount at about $4.3 billion dollars for the 1996-97 school year alone, and projections are for $5.2 billion to be spent this year.

Computers, networks, printers, scanners, file servers, CD-ROM drives and even interactive television facilities are common sights in most schools--perhaps not in the numbers many students, teachers or parents would like--but certainly in quantities suggesting this investment should be having a significant impact on education.

Why then is it difficult, if not impossible, to find definitive studies that show the positive impact technology has had on learning? And why is it vital that we in education begin to find ways to assess that impact?

The Naysayers Arise

Let's take the second question first. A backlash against technology in the popular press has begun. Critics, such as Todd Oppenheimer, author of "The Computer Delusion" in The Atlantic Monthly last July, rail against technology use for several reasons.

A primary concern is that unproven technologies are being purchased at the expense of proven programs, notably in the arts. After examining historical uses of technology in the classroom, Larry Cuban, a professor at Stanford who formerly worked as a district superintendent, predicts that schools will not effectively use computers even when they are present in sufficient numbers. Social commentators, meanwhile, worry that computers will dehumanize education.

To no small degree, technology has become the highly visible symbol of society's demand that K-12 schools shift from providing a liberal arts education to one that also stresses work skills.

Change always brings the naysayers out of the woodwork, both those with political agendas and those with well-meaning educational concerns. But even without the detractors, I believe educators have an ethical responsibility to continually examine how we use finite funding to make sure we provide the maximum educational bang for the buck.

One reason educators find it difficult to describe and measure technology's impact is that schools do not use it in a single way for a single purpose. In order to assess the impact of technology in elementary and secondary education, one first must recognize and fully appreciate the four major uses of technology in schools. Our approach to evaluating each use needs to be quite different.

Multiple Purposes

* Use No. 1: To improve administrative effectiveness through efficient communication, planning and record keeping.

Like any large organization, schools use technology to improve daily operations. Administrative software packages keep student records, figure payroll, generate state reports and schedule classes. Telephones, voice mail, e-mail, intranets and Web sites use the power of networking to collect, distribute and update information. Web pages, desktop-published documents and video productions inform our communities about school activities.

For example, our school district uses its networked servers for a growing number of purposes. A networked calendar (MeetingMaker) is used to schedule meetings for building and district administrators. Our Web server makes the most current versions of the district's policies, curriculum and job descriptions readily available to the staff and community.

Gauging the impact of technology for administrative uses also involves evaluating the cost effectiveness and reliability of data processing. This means looking at the ease of use, minimization of manual data input and flexibility of generating reports.

For optimal use of technology for administrative support, districts need to plan for:

* powerful, stable voice, video and data networks that reach into every classroom and office;

* shared databases for student data management, finance, payroll and scheduling, accessible from all desktops;

* shared calendars and facilities schedulers; and

* district-produced information accessible electronically through Web sites, email discussion groups, public access television or desktop-published mailed documents.

Any communication audit a school district undertakes certainly needs to include good questions about how information is shared electronically. Are the technologies available in sufficient quantities so digital formats can begin to replace printed ones? Can we keep our curriculum guides on a school intranet? Can we replace the daily paper bulletin with a mailing list? Can the information previously delivered to the office by scan sheet now be sent through the network? No cost savings will be achieved until that happens.

Resource Availability

* Use No. 2: To provide access to current, accurate and extensive information resources for all learners in the district and community in a cost-effective and reliable manner.

Technology connects all members of an organization to the resources they need. Whether it is a superintendent downloading the latest regulation from the state department of education, a curriculum coordinator looking for federal grant information, a teacher participating in a mailing-list discussion about classroom management or a student looking for a magazine article for a research paper, all learners in schools can use electronic information sources.

Schools are effectively using:

* electronic library catalogs for accessing district collections from the media center, classroom and home;

* stand-alone and networked CD-ROM reference materials, including electronic encyclopedias, magazine indexes and full-text databases like SIRS and Newsbank;

* full-text periodical indexes accessed through the Internet like UMI's Pro-Quest or Electronic Library;

* staff and student e-mail and desktop video conferencing to communicate with experts throughout the world; and

* Internet connections to World Wide Web, gopher and File Transfer Protocol sites.

Evaluating technology on this level is difficult. Few quantitative standards exist at the state or local level for determining whether informational materials are available to students and staff in sufficient quantity and whether they are being used effectively.

A district's library/technology committee can develop checklists of desired resources. Through exit interviews, students completing major projects can be asked if the resources they needed were available. Surveys of teachers and administrators can determine what resources are being used and how. School accreditation committees, such as the North Central Association of Colleges and Schools, can include resource availability in their evaluation of programs.

Modifying Instruction

* Use No. 3: To provide teachers the tools and resources needed to ensure students will meet local and state learning objectives and have the means to assess and record student progress.

Teachers nearly always first use technology to enhance their professional productivity. This includes using specially designed teacher tools like computerized grade books, worksheet generators and curriculum templates; generating more effective communications, study guides and tests using a word processor; and delivering more compelling lectures using a presentation program.

Basic teacher computer literacy is easily measured when it is accurately described. The Mankato, Minn., Public Schools have developed 10 rubrics designed to help teachers assess their own level of technological proficiency and help the district evaluate the effectiveness of its staff development efforts. (See related story, page 16.)

But technology use by teachers needs to evolve beyond enhancing traditional methods of instruction. Rather than the computer simply being a tool that allows a common task to be done more efficiently, higher-order technology skills modify how instruction is delivered, how student performance is measured and how teachers view themselves as professionals. The technology is used to restructure the educational process to ensure that:

* all students master the basic skills of writing, reading and computation;

* all students practice information literacy and research skills and the critical thinking skills inherent in them; and

* all teachers have the tools and ability to locate the research findings that will guide their use of technology and collect the data that measures the effectiveness of their practices.

Technology evaluation then becomes an integral part of the evaluation of the total school program, and decision makers must attempt to isolate what technology use has contributed to the total program's strengths and weaknesses.

Student Mastery

* Use No. 4: To allow students to learn and demonstrate their mastery of technology to access, process, organize, communicate and evaluate information to answer questions and solve problems.

Drill-and-practice software, integrated learning systems, videotaped lessons, computer-animated picture books, low-level problem-solving and simulation computer software have long been the mainstays of technology use by students in most schools. This is also where a good deal of effort has gone in assessing the effectiveness of educational computing with poor, or at least mixed, results.

The use of technology to teach basic skills, memorized facts or low-level thinking skills, while at times motivational for very young or at-risk students, is expensive for the results achieved. (My cynical side says this use of educational computing is attractive to administrators who hope that well-designed programmed instruction can overcome the disastrous performance of students with poorly trained or incompetent teachers.)

The assessment of this use of technology really has to be an assessment of total student gain of rather low-level thinking skills and is extremely difficult to do for many reasons--Hawthorne effect, bias of software producers who may be conducting the evaluations, lack of resources for controlled study groups, etc. Unfortunately, this technology use seems to be tainting the current attitudes of decision makers about all uses of technology in schools.

A Challenging Assessment

That leaves us with the best use of technology in schools by students as an information-processing and productivity tool. The use by students at all grade levels of real-world productivity software to complete complex, authentic projects is the proper instructional role of technology. Such real-world technology includes word processors, databases, spreadsheets, presentation programs, multimedia authoring tools, e-mail, video production equipment, digital reference materials, electronic indexes and network search engines. Here students will be asked to complete tasks similar to those they will be asked to do in jobs that require using information to solve problems--the kinds of jobs that are both better paying and give greater job satisfaction.

One example of how technology can be used as an information processing tool can be found at our Web site (www.isd77.k12.mn.us/schools/dakota/war/worldwar.html). These Web pages are the end product of a World War II unit taught at one of Mankato's middle schools. The teacher and media specialist collaborated on the project that brought community members who lived through the war into the school where students interviewed them. Students then created Web pages of the interviews interwoven with scanned photos and memorabilia.

But big challenges present themselves when technology is used on a large scale as an information-processing tool. First, it requires a good deal more investment in time and effort on the part of teachers in learning how to use it. Anybody can learn to operate drill-and-practice software in a few minutes, but learning to use a database to store, categorize and sort information literally can take hours of instruction, weeks of practices, genuine effort and guaranteed episodes of pure frustration. Teachers must spend additional time developing lessons that incorporate the computer productivity skill into their specific subject areas.

Second, the product of such instruction is not a neatly quantifiable score on an objective, nationally normed, quickly scored test. Conducting and assessing such projects require the ability to develop and apply standards, delay for long periods of time the satisfaction of task completion, and acknowledgment and acceptance that conclusions, evaluations and meanings that result from the efforts are often ambiguous. (Darn, just like in the real world!)

Finally, students need more than the 20 to 40 minutes of lab access time per week to learn these uses of technology. That means more equipment and software and making the technology available in more locations (including classrooms and media centers) than if computers are used simply as electronic worksheets or flashcards.

Assessment of this technology use needs to be done less to satisfy a state department, legislature or academic body than to inform the students themselves, their parents and the community in which they live. It means undertaking the difficult task of creating benchmarks that describe student information and technology skills at various grade levels and the assessment tools needed to measure progress toward those benchmarks. It means finding ways to aggregate the assessed benchmark data to determine how well the entire program or school is doing and using that data to modify instructional practices.

It also means using technology to build personal portfolios of thoughtful, creative work that students and teachers can share with parents; to present worthwhile and authoritative reports to classmates; and to make meaningful contributions to efforts aimed at solving school or community problems. Lastly, it means being able to determine if the use of technology is making our children better citizens, better consumers, better communicators, better thinkers and better people.

Doug Johnson is district media supervisor in the Mankato Public Schools, Box 8713, Mankato, Minn. 56002-8713. E-mail: djohnsl@mail.isd77.k12.mn.us. He is the author of The Indispensable Librarian: Surviving and Thriving in School Media Centers in the Electronic Age.

Rubrics to Gauge Your Staff's Computer Literacy

Parents and administrators want computer-literate teachers. Students seek out teachers who know how to use technology meaningfully. And teachers themselves acknowledge that computer skills are growing more important to their professional duties.

Yet what specific computer skills actually comprise computer literacy is rarely articulated. Computer literacy, as it applies to teachers, easily can be ill-defined as a politically correct buzzword without meaning or purpose.

When the Mankato, Minn., Area Public Schools began its formal staff development program to train teachers how to use technology five years ago, I wrote a series of rubrics (graduated performance indicators) that described what the district expected a computer-using teacher to be able to do after 30 hours of formal computer instruction and six to nine months of practice. We call our program the CODE 77 Rubrics. (CODE 77 stands for Computers On Desks Everywhere in District No. 77.)

Four Levels of Competence

These rubrics primarily address professional productivity. They are the foundation on which more complex technology and technology-related professional skills are built. Teachers who have mastered these skills are able to use the computer to improve their traditional instructional tasks, such as writing, record-keeping, designing student materials and presenting lessons. These skills also build the confidence teachers need to use technology to restructure the educational process.

Each of the 10 rubrics has four levels: Level 1, Pre-awareness; Level 2, Awareness; Level 3, Mastery; Level 4, Advanced.

This is the rubric for word processing:

* Level 1: "I do not use a word processor nor can I identify any uses or features it might have that would benefit the way I work."

* Level 2: "I occasionally use the word processor for simple documents that I know I will modify and use again. I generally find it easier to hand write or type most written work I do."

* Level 3: "I use the word processor for nearly all my written professional work: memos, tests, worksheets and home communication. I can edit, spell check and change the format of a document. I can paginate, preview and print my work. I feel my work looks professional."

* Level 4: "I use the word processor not only for my work, but have used it with students to help them improve their own communication skills."

Prior to training, we assumed most teachers would be at level 1 or 2, and our training efforts were designed accordingly. By the end of the training, we anticipated teachers would be at level 3 or 4 in most skill areas and have moved up at least one level in all areas.

Dual Purposes

These rubrics have served two purposes in our district. By asking teachers to complete an anonymous self-assessment using the rubrics before training and again after training, we have been able to judge the effectiveness of our staff development efforts. Simple graphs showing the percentage of training participants at each level before and after training are constructed. These results are shared with the staff development committees and administration.

The rubrics also serve to provide a road map for teachers wanting to improve their computer skills. By examining the specific skills that are described, teachers know in what areas they need to continue to take classes or practice.

I have written a second set of seven rubrics that describe a teacher's use of technology to fundamentally change how instruction is delivered, how student performance is measured and how teachers view themselves as professionals. Now technology is being used to restructure the educational process to allow it to do things. it never was able to do previously.

The rubrics and instructions to teachers for completing a self-assessment can be found in the December and January issues of Technology Connection magazine. A link to a longer article "One Step Forward; Two Steps Back" that describes how the rubrics have been used for staff development evaluation can be found on my home page at www.isd77 k12.mn.us/staffdir/staff2/Johnson_Doug.html. Readers are welcome to modify the rubrics to their own district's specific needs and as technology changes.

These rubrics also can be modified to help benchmark student performance. (See Jamie McKenzie's adaptation of these rubrics for that purpose at www.bhs1.bham.wednet.edul.)

COPYRIGHT 1998 American Association of School Administrators
COPYRIGHT 2004 Gale Group