Technology Audits �� Grappling with Accountability

"TECHNOLOGY AUDITS ... GRAPPLING WITH ACCOUNTABILITY" IS MORE THAN THE TITLE OF THIS ARTICLE. THE PHRASE DESCRIBES AN ADVANCE IN THINKING ABOUT HOW EDUCATORS, PARENTS, AND COMMUNITY LEADERS CAN BEGIN TO UNDERSTAND AND ACT UPON TECHNOLOGY'S IMPACT BY INVESTIGATING, DOCUMENTING, AND COMMUNICATING THE RESULTS, BENEFITS, AND EFFECTIVENESS OF TECHNOLOGY IN PREPARING LEARNERS FOR THE FUTURE.

Countless school districts across America dutifully prepare technology plans because they must do so to receive state or federal dollars. Committees of teachers, technology staff, and administrators pore over other district's plans, state recommendations, and consultants' suggestions in order to arrive at the plan that will "do the job." Once these plans are "completed," they take their place on the shelf with numerous other old documents, failing to become part of any comprehensive effort to make a difference in how teachers teach or how learners learn. And it is usual that relatively few people in the district have any clue as to what might be contained in the plan.

What is usually missing in these plans is any focus on results. What is also missing is any concerted effort by districts to analyze their use of technology over time, in order to determine some degree of accountability--"Why do we use technology?"

IT'S PERSONAL

For me, the audit idea has taken on significance both as a tool that resonates with educators and as a service to which I have been able to contribute. Since 1997, I have been a participant in the development of a new process for helping school districts make sense of technology. This is a remarkable process, which I wish had been on the drawing boards much sooner. I struggled with a planning/managing scenario in the Charlotte-Mecklenburg (North Carolina) Schools as their Coordinating Director for Technology from 1992 to 1996. Had the idea occurred to us that we might audit ourselves, or have an outside party audit us, we would have saved a great deal of time and money, and no small amount of frustration. A congruent statement about measuring the benefit of technology in school districts throughout the nation was just beginning to take shape; however, the question of how to undertake such measurements still loomed. Some of the answers were soon to arrive on the scene.

Four Indiana districts were the pilot sites for the Indiana University/Phi Delta Kappa (PDK) "TechAudits," as they were then known. I was lucky enough to be part of the teams that went into each of those districts. Typically, teams were comprised of a former superintendent of schools, a person with technology coordination background, a university person with broad experience in the observation of educational technology, and a person with perspective on the national scene as both a writer and a consultant to districts. On two of the audits, a state department of education staff member and a library media specialist joined in the week-long efforts.

Over the course of those audits, we grappled with what an "audit" is, how it works, and how it helps clients. I have visited dozens of schools in four districts, talked with hundreds of teachers, parents, students, and administrators. In those first efforts, we shot hours and hours of video followed by days and days of editing. One district's reaction to the audit is recounted later in this article.

Throughout this early experience, team members learned how very much districts have in common. Districts were and are spending large amounts of money on technology--both in terms of technical as well as people infrastructure. All of these districts were struggling with people issues: how to allocate them, train them, and maintain their professional well-being. In most cases, technical issues were being handled with fairly high proficiency, a fact borne out by the CEO Forum reports mentioned below.

GOIN' TO KANSAS CITY

In the Kansas City, Missouri, School District, which is home to me, I am serving as the liaison between the district and PDK in partnership with Region VII in providing for the development of an audit during the 2000-2001 school year. It is a complex process beginning with scrutiny of the district's procurement process. This district has a long history of personnel changes that has contributed to misgivings about fairness and equity in technology purchasing. A report was generated very quickly in order to move the purchasing of critical equipment off dead center. The audit in Kansas City can be thought of as a three-part effort, with this first report on procurement as the first part. For the interim report and the long-range report and action plan, the next several months will include mediation to determine the scope and sequence of the audit. Finally, determinations about site visits, protocols, and budgets will be made. Out of all this activity will come the means to bettering student outcomes as a consequence of organizing and assessing technology in the district.

In both Missouri and Kansas, my focus is on how better to establish linkage with organizations and state agencies. The priority will be to focus on results in districts choosing to participate; in much the same way we are dealing with Kansas City. But for a successful tech audit anywhere in the United States, good communication is necessary between PDK and a variety of organizations and state agencies, as well as with the school districts themselves.

THE CEO FORUM REPORTS

The October 1997 Year 1 report of the CEO Forum on Education and Technology states that "to date, there has been little conclusive research directed toward measuring the full benefits of technology on the processes of teaching and learning." The report added: "More effective means of measuring the nature and extent of these benefits vis-a-vis education objectives is critical to energize widespread technology integration efforts in our nation's schools."

In the June 2000 Year 3 report of the CEO Forum, progress is noted in the areas of hardware acquisition, connectivity, and technology training for teachers. The report continues ... "However, as technology transforms the global digital economy, schools lag behind in the true integration of the vast resources technology makes available. While a majority of teachers now utilize technology to help perform administrative functions, only 33 percent of K-12 teachers say that they feel `very well prepared' or `well prepared' to integrate high-quality digital content into instruction."

Finally, the CEO Forum urges schools to make the necessary commitments and adjustments to transform themselves into digital learning environments. [Editor Note: See Anne Bryant's essay, "Investment in Education and Technology: The Key to Preparing for the Digital Age, " on page 21 of this issue.]

SETTING THE STAGE

Such adjustments undertaken by (some) school districts extend from exerting effort to build the capacity for technology to actually teaching and learning with technology tools are beginning to be examined through an internal or external audit process. This examination sets out to help a district identify where it is currently, to assist it in knowing where it wants to be, and to move the district toward that objective.

What does it mean to understand what a district looks like? Will the district know when it has moved into a different dimension of technology use? What does it look like when all of the pieces come together to form a truly integrated use of technology?

Bernajean Porter, in her book Grappling with Accountability, describes a continuum of technology utilization extending from Literacy Uses to Adapting Uses to Transforming Uses. This article reviews all of the implications for each stage of usage, but within and among each of these stages fall hundreds of variations. Any one school district may find that in some locations one or more of these stages exists--it is not an effort to label a district as being very good or being very bad with technology usage. It is simply where they may be at a given moment.

* The Literacy Scenario. Johnny, a fifth grader, goes to the computer lab every Wednesday. He learns word-processing, spreadsheets, and how to create pictures with a paint program. A computer teacher taught him keyboarding last year, and he expects to take keyboarding again before the end of the year to be able to type even better. Johnny is also putting together a great slide show on UFOs. It is his first-quarter computer project. He would like to use computers more, but the lab is usually full with other classes. However, Johnny's classroom has a computer this year, so when he finishes his math assignments, his teacher lets him play "SimCity" or "Math Blaster" or sometimes make a crossword puzzle with his vocabulary words. He really likes playing the new "Oregon Trail." Even though they studied the westward movement last year, the game is still fun and interesting. Johnny has discovered he really likes computers. Next year he has signed up for the two new classes, robotics and an HTML Web-design course.

* The Adapting Scenario. Ninth grader Akhito uses a computer in her classroom or in the computer lab to help her with her schoolwork. Because she is having trouble with equations, her teacher suggests special drill-and-practice software when she is in the lab. Akhito also uses the SAT software to prepare for her test next month. When their class goes to the library, Akhito and her friend Safia research their earth science report together using online research tools. Her science teacher made a research "template" on disk and expects it to be completed when the class goes to the writing lab tomorrow. In the computer lab, a special software program prompts them through scientific report questions. Their prompted answers import into a word processor, a spreadsheet graph of their collected data is inserted in the document, and finally, a spell checker is used before their assignment is printed out. Akhito is glad to have these tools to make her school work even better.

* The Transforming Scenario. For their high school social studies class, Jose, Ruby, and Gregory have chosen to design a school project that will research a riverfront development expected to be built on a landfill. They plan to present their findings to the mayor's planning commission at the end of the term. After their civics, science, language arts, and math teachers approve their second semester community project connected to learning standards and an assessment process, they begin the collaborative task of researching the environmental and economic development issues that will help to formulate a recommendation. They are able to do their work from school, the community library, home, and their local college by using laptops with modems to connect to each other and to resources as needed. Their research is supported by online digitized text and graphic resources. They also video-teleconference with a network of researchers their teachers joined last year, and use a database tool designed for group resource sharing. With continuous review and guidance by their teachers and some peer technical assistance in preparing their presentation with multimedia tools, they complete their project. When the team finally presents its finding, the planning commission values and utilizes the knowledge created by this student team while deciding the city's zoning issues. The students' findings and successes will be published electronically on an environmental home page on the town's Web site for others to reference in the future.

In these examples, schools have taken advantage of resources available to them. Teachers were prepared to do what needed to be done within each of the suggested scenarios. It meant that (in addition to having the technology that was available and understood by teachers and students alike) teaching was moving from highly directed instruction toward a more project-based, constructivist learning environment.

An audit is about knowing where you are, and where you want to be along this continuum.

AN AUSPICIOUS BEGINNING

Phi Delta Kappa has for years helped districts through Curriculum Management Audits (CMAs). The CMA is fundamentally a discrepancy analysis--it is a process of letting an audit team examine a district's curriculum frameworks with an eye toward discovery of what is missing or what should be expanded. When PDK made the decision to develop Technology Audit Services, it was determined that the outcome should not place blame nor shame upon anyone or any process. Instead, the Technology Audit Service is geared to provide for the means to continuous, specific improvements.

Technology Audits were developed initially by the Center for Excellence in Education at Indiana University (IU) for a partnership with Phi Delta Kappa International. Aided by a grant from the Kellogg Foundation, IU, Howard Mehlinger (then director of the Center), and his staff began in June 1998 to develop procedures for technology audits and to lead the initial effort. In January 1999, a meeting was held in Indianapolis to discuss the tentative procedures and to elicit the interest of individuals who might wish to become auditors. A series of "awareness conferences" was held in spring 1999 to market the service under the name TechAudits; PDK also began to market the new service.

In May 1999, the first audits were undertaken in two Indiana communities, Corydon and Anderson. These two efforts were tests of chosen procedures; both audits were heavily subsidized. In December 1999 and in February 2000, teams of auditors completed audits and produced written reports for school systems in Burlington, Vermont, and Pike Township in Indianapolis. Another audit team traveled to Perry Township in Indianapolis in December 1999, with the intention of completing a multimedia report using an approach somewhat modified from that used in Corydon and Anderson. The Perry Township multimedia report was completed recently.

Pike's assistant superintendent for secondary education, Larry Galyen, reports that months later the district is still living with the impact of the audit. In this case, the audit brought two camps of thinking together on personnel issues that had been going unresolved. This has allowed the district to move forward with its own technology plan implementation as well as the three-year plan required by the state of Indiana.

Beyond the planning and plan implementation, the audit brought the district's focus on reality and, as Galyen says, "crystallized our thinking about the district's direction for technology now and in the future." Among the audit team's recommendations was that the district make a deliberate effort to publicize and draw attention to some of its most innovative practices in the use of technology. Then, it was further suggested, the district should adopt the most effective practices district-wide to ensure conformity of student experience. Further, it was suggested in the audit report that the district consider adopting the International Society for Technology Education standards for students. These, it was said, could provide a baseline, and the school could adopt or modify these as they were deemed important.

Galyen indicated that the audit continues as an essential document that will be used as a map to this district's effective use of technology. A side benefit, he said, was the excitement generated among staff in preparation for and during the weeklong visit by the PDK audit team. Teachers, students, and other staff felt that their opinions counted and that what they said to auditors, as well as what the auditors were able to see for themselves, would begin to make a difference.

LOOKING AHEAD

From all this a pattern becomes clear: Effort has, for years, been the object of our energy and spending. Effort is good and necessary. The outcome of effort is usually quantified as "sufficient" numbers of machines gathered in labs or in classrooms sometimes joined together by networks of fiber backbone and cabling infrastructure. Effort may also be measured in terms of "seat time" in front of a computer. In fact, it can include a teacher's approach to including technology in her or his lesson plan. Effort may also extend to a learner's striving to include a Power Point[R] presentation in a report about the Mississippi River, and it may also be calibrated against a curriculum for teaching technology. But, as we've seen in the CEO Forum reports and in the scenarios describing levels of technology integration, it will be results, not effort, that take center stage in the 21st-Century school.

RELATED ARTICLE: TECHNOLOGY AUDITS TODAY

Currently, Phi Delta Kappa (PDK) is partnered with Education Technology Planners, led by Bernajean Porter. Porter was enlisted to direct the PDK Technology Audit Services under the umbrella of the Phi Delta Kappa International Center for Professional Development and Services. Her Grappling with Accountability--Resource Tools for Organizing and Assessing Technology for Student Results, published in 1999, is a bundle of licensed materials that form the framework for the organization, and the conduct of technology assessments, workshops, and comprehensive audits. The new venture is called Technology Audit Services for Schools. While some degree of "customization" may be called for, the audit services offered by PDK are packaged as four distinct services.

One-Day Technology Accountability Workshop

In a one-day workshop, participants are introduced to a six-step process for collecting, analyzing, and interpreting data designed to help them answer questions about technology use in their school districts, such as: Has our technology program improved teacher and student productivity? Does our district demonstrate equity in its technology program development and implementation? Has the school system established clear and measurable objectives for student use of technology? Answers to questions like these can be used to guide future technology efforts in a school district. The one-day workshop is an awareness opportunity for decision-makers throughout the district, as well as a good starting point for achieving self-awareness within the school and community.

Online or Paper Survey Subscription Service

Schools can choose to use one or a combination of the four field-tested survey instruments by subscribing to this service for one to three years. This service includes a valid sample selection process, data analysis, and the preparation of a comprehensive interpretive report. Survey data for the following four instruments are available online or by using paper survey forms:

* Student survey

* Teacher survey

* Principal survey

* District instructional technology survey

* Parent and community survey (available after January 2001)

Six-Day Technology Audit Training for School Teams

For districts wanting to strengthen their existing plans or create new ones, or who want to apply for technology grants, or who are needing to assess the impact of technology implementation efforts--the Technology Audit Internship Training can speed up and enrich a district's efforts in these areas with six days of intensive, experiential training, conducted at the district level. The consequence is a local team trained to conduct highly successful, "do-it-yourself" technology audits.

The six days are usually formatted as

* a three-day session on action planning for data collection

* a two-day session on drafting reports

* a one-day session on presenting reports and sustaining momentum

Comprehensive PDK Technology Audit

These audits are intended to assess all aspects of technology use in a school district. A customized audit, conducted by an external team of evaluators with national expertise, is an examination of the alignment of educational purposes, technology power, infrastructure, and instructional uses at the district level. The audit team reviews documents, examines survey results, analyzes student artifacts, conducts focus groups and interviews, and makes an onsite visit to assess 20 variables essential to effective technology use for student learning. PDK Auditors are responsible for creating and presenting a comprehensive technology audit report that includes detailed recommendations for organizing the district's technology investments to better serve students and teachers. Audit results can be used to guide future technology efforts in the district.

With this range of services, districts can choose a service or combination of services that best suits their needs, their budget, and their instructional philosophy.

PDK is currently making proposals to a number of districts. Each proposal requires time and attention to detail about what the district wants and when. Any district thinking about this process should make its inquiry as early as possible so that calendars can be coordinated.

To learn more about Tech Audits, contact PDK's Center for Professional Development and Services at PO. Box 789, 408 N. Union Street, Bloomington, IN 47402; 800/766-1156; www.pdkintl.org.

--Hal Gardner

Henry B. "Hal" Gardner and his wife, Jean, are partners in HomoFaber Technologies: Curriculum and Technology Solutions, an educational technology consulting and sales firm located in Kansas City, Missouri. Over the last 25 years, Hal has worked in media and technology with the Missouri Department of Elementary and Secondary Education, Missouri School Boards Association, Charlotte-Mecklenburg (NC) Schools, and Charlotte Education Foundation. He was instrumental in the development of legislation in Missouri that resulted in an earmarked tax on video rentals for distance learning technologies in state schools. He can be reached at kchbg@aol.

COPYRIGHT 2000 Agency for Instructional Technology
COPYRIGHT 2001 Gale Group