Status of Large-Scale Assessment Practices for Students with Disabilities in Rural America, The

Abstract

The status of large-scale assessment practices for students with disabilities in rural states compared to urban and other states is explored in this article. Based on responses to a survey of state special education directors, rural states showed more stable assessment participation rates across years than urban states. Fewer rural states had data available about assessment performance levels, but of those with data, more showed an increase in performance levels than urban states. More rural than urban states had data on accommodations use and more of these states showed stable use across years, whereas more urban states showed an increase in the use of assessment accommodations. Nearly all rural states use a portfolio approach to alternate assessment in contrast to urban states in which a variety of approaches are used. Through the story of one rural education cooperative, an illustration is provided that assessment data can and is being used to make a difference in student outcomes.

Our education cooperative serves 7040 students K-12 and covers 7000 square miles across seven counties. Collaboration is necessary for survival. Our districts work together on things that each of them can't manage on their own. We've found over the last decade that the shift to standards-based reform is full of opportunities for cooperation and collaboration. A good example of how cooperation works to benefit all of our schools was when we made the shift to data-based decision-making. We had test data and other sources of information about how students were doing, but we needed to put it together and use it to continually improve our educational system. We found the level of support needed to help the schools do this is huge. You need to start at the very beginning: what do these scores mean, how do you begin to analyze, how do you look at it, where do we give teachers skills? Ultimately we built capacity in the classroom to systematically gather and use multiple sources of data. We targeted assistance to teachers to figure out strategies-short and long term-to address specific issues we-and they-saw. We were looking for strategies to ensure all kids in those classrooms were making progress, to ensure all kids were successful. We're also learning that the most successful teachers believe it is their responsibility to teach every student who comes through the door-no matter where the student is or what the student does.

Sound familiar? Do all rural states and districts operate this way? Is this different from educational systems in urban states and districts? And, the bottom line is, what are the results for students?

For the past 10 years, The National Center on Educational Outcomes (NCEO) has surveyed state directors of special education about the participation of students with disabilities in education reform, specifically their participation in assessment and accountability systems (Thompson & Thurlow, 2001). NCEO results confirm that reform efforts continue to evolve at a rapid pace as states and districts move toward standards-based assessment and accountability systems that include every student. States vary on the status and progress they have made toward truly inclusive systems.

This article is a report on the status and progress of states defined by the U.S. Census Bureau (1990) as the 11 most rural (Arkansas, Kentucky, Maine, Mississippi, Montana, New Hampshire, North Carolina, North Dakota, South Dakota, Vermont, and West Virginia) compared to the progress of the 11 most urban states (Arizona, California, Florida, Hawaii, Illinois, Massachusetts, Nevada, New Jersey, New York, Rhode Island, and Utah). We used 1990 Census data because it is the most recent available with details at the level needed for this analysis. In the article status information reported throughout the article is illustrated by presenting anecdotal evidence from the rural education cooperative across seven counties quoted in the opening paragraph. When the state within which the cooperative is located began working to define content and performance standards in the mid 1990s, cooperative districts volunteered to become pilot partners in the effort. Members of the cooperative saw the shift to standards as an opportunity to clearly define what "success" meant for all students. Now in full implementation of standards-based reform, these rural school districts have insight into how to ensure that all students succeed.

The 2001 survey of state directors of special education (Thompson & Thurlow, 2001) had a focus on the implications of educational reform within the context of the 1997 reauthorization of the Individuals with Disabilities Education Act (IDEA). Respondents to the 2001 survey included state directors of special education from all 50 states. Responses to the survey were gathered online and via fax. Some state directors designated other state officials to complete the survey, and multiple respondents, including state assessment and accountability personnel, completed some surveys.

State assessments are important elements of standards-based reform. States and districts measure how well students are doing through assessments aligned to academic content and achievement standards. One common method of state and district-wide assessment is testing, referred to in this context as "large-scale assessment." Based upon assessment results, schools work to improve curriculum and instruction so all students can succeed.

State Assessment Participation and Performance

By law and in practice, states and districts have defined the following options for students to participate in assessment systems:

* Participation in general assessments;

* Participation in general assessments with accommodations; and,

* Participation in alternate assessments.

More than half of all states reported an increase over previous years in the test participation rates of students with disabilities on their state assessments. Several directors attributed this increase to alternate assessment participation by students who had been excluded in the past. In Figure 1 changes are shown in assessment participation rates for rural, urban, and other states. Of the 11 most rural states, five reported an increase in test participation rates, five reported that test participation rates had remained about the same, and one state was unable to make a comparison across years. In contrast, all but 2 of the 11 most urban states reported an increase in participation rates of students with disabilities over previous years.

All students are expected to participate in state assessments in the rural states of Arkansas, Montana, and Vermont and in the urban states of Florida, Illinois, New Jersey, and Rhode Island. The remaining states allow assessment exclusion for a variety of reasons, including parent refusal, students who are medically fragile, have emotional distress, are homebound or hospitalized, have limited English proficiency, and or are absent on test days. One director commented that "in theory" no one is excused, but "in reality" there are students who are absent and do not make up the tests.

Assessment Performance

Performance levels of students with disabilities on state assessments have increased in about one-fourth of the 50 states. In Figure 2 three rural states are shown to have reported increases in test performance, two stayed about the same, and one rural state reported a decrease. A state director commented that using data over time they found that students with IEPs have shown improvement in all content areas assessed. The remaining five rural states were not able to compare performance levels across years, either because data were available for only one year or because performance data on students with disabilities had not yet been disaggregated. A greater number of urban states were able to report assessment performance across years with increased performance in two states, stable levels in five states, and a decrease in test performance in one state.

According to the rural cooperative's director, educators in one member district looked for strategies to ensure that all students were making progress based on multiple measures without focusing specifically on raising test scores. But as classroom strategies were implemented to ensure that all students succeeded in standards-based settings, district staff found that performance on assessments improved as well. In the state assessment in reading and math, students progressed from the bottom 10% of districts to above average on all tests by the end of the third year of intervention. This included students who had not been successful in the past. The district motto is "take care of business, and business will take care of itself." The trend in score improvement was encouraging, although district staff was cautious about comparing scores based on a single year or only two years. With only about 100 students enrolled at each grade level in that district, staff were cautioned there would be artificial "blips" in overall school scores, reinforcing reliance on multiple measures to show the progress of schools over time.

Assessment Accommodations

Assessment accommodations are alterations in the way a test is administered without changing the actual test content or performance standard. Researchers argue accommodations should raise or "boost" performance of students who need them and not affect the performance of students who do not need them (Fuchs, Fuchs, Eaton, Hamlett, & Karns, 2000; Tindal, Helwig, & Hollenbeck, 1999).

Every state, currently, has a policy governing the use of accommodations on large-scale assessments (Thurlow, Lazarus, Thompson, & Robey, 2002). These policies vary across states with a great range in the number of students using accommodations and the variety of accommodations selected (Thompson & Thurlow, 1999). The specific accommodations listed by states have continued to increase over time.

Nearly 60 percent of all states keep track of the use of accommodations during state assessments, about half of these reported an increase in use and the other half reported stable use (Thompson & Thurlow, 2001). Some directors attributed growth in use to increased awareness and understanding by educators, parents, and students. In Figure 3 changes are shown in the use of accommodations on state assessments for rural, urban, and other states. Five rural states reported stable accommodation use, three states reporting an increase, and three states having no accommodation data to report. In contrast, a greater number of urban states showed an increase in accommodations use while only one state remained the same and four urban states did not have accommodation use data available.

The director of the educational cooperative talked about the linkage between assessment accommodations and accommodations needed for instruction. Staff members found that unless there is a thoughtful approach to choosing accommodations that appropriately support the learning of students with disabilities to the same high standards as other students, accommodations on large-scale assessments had limited impact. To have an impact there had to be changes in practice at the classroom level. The director of the cooperative told a story about how districts gradually evolved to "accommodated general education classrooms:"

Back almost 25 years ago, a couple of us were teaching social studies at the high school level in one of the districts and started wondering why some kids would get up and leave our classroom to go to the special education resource room for tests. We started talking with the special education staff, saying (only half in jest), "You've taken our history test too often-we aren't going to send our kids to your room any more." The special education teachers suggested an alternative: "What if we make you better teachers of all kids? We can give you strategics and techniques to make YOU more successful. Then the kids won't have to come to us for learning or for testing." Soon that model permeated the system. We would meet as a team with the special education resource staff, and they would critique us-our materials, our methods, our models, our outcomes. It was really a high trust situation to have a peer say "I watched you do that, it didn't go so well, let's work on what might be changed in that strategy to make it work." Over time, all of the teachers understood and made use of accommodations and strategies to help every student learn, and to show what they know!

Alternate Assessment

IDEA required all states to have alternate assessments in place by July 1, 2000. An alternate assessment is a way to measure the performance of students who are unable to participate in general large-scale assessments used by a district or state. Alternate assessments provide a mechanism for students with even the most significant disabilities to be included in the assessment system.

Every state involved some type of stakeholder group in the development of their alternate assessments (Thompson & Thurlow, 2001). All stakeholder groups included state and local special education personnel. In addition, most states included state and local assessment directors and coordinators, local school administrators, related service personnel, and general educators. Nearly all states also included parents and advocates, while a few included students and adults with disabilities. The most rural states also reported representation of nearly all of these groups in the development of their alternate assessments.

Standards Assessed by Alternate Assessments

Nearly all state alternate assessments including all but one of the most rural states, assess the same standards as general assessments-either by expanding state standards, linking a set of functional skills back to standards, or assessing standards plus an additional set of functional skills. The alignment of alternate assessments with standards has evolved a great deal during the past three years. In 1999, several states that indicated they were developing alternate assessments based on a special education curriculum are now making a connection between their alternate assessment and state standards (Thompson & Thurlow, 2001).

Alternate Assessment Approach

Several strategies have been used to show progress toward state or local content standards through alternate assessments (Thompson, Quenemoen, Thurlow, & Ysseldyke, 2001). Most states compile data at multiple points over an extended period of time-usually throughout a school year-using a variety of assessment strategies. In Figure 4 alternate assessment approaches selected by rural, urban, and other states are shown. Over half of the states, including 8 of the 11 most rural states, organize the data collected for a student's alternate assessment into some type of portfolio while others summarize the results on a checklist or rating scale. These approaches have evolved as alternate assessments have been piloted and refined. Most states, including nearly all of the 11 most rural states, have teachers scoring the alternate assessments for either their own students or for students in other schools or districts.

In one of the districts in the rural cooperative, there is an option for all students at the high school level to meet academic standards in community settings through project-based learning called the "Bridges" program. Each student develops a project aligned to specific standards with support by a mentoring team of teachers with certification in math, science, social studies, language arts, and special education. Each student builds a portfolio of work completed and prepares a final culminating event of a performance, paper, or project outcome. The option is open to all students and includes the participation of students with significant disabilities. In states using portfolios as the alternate assessment approach, a project based learning option aligned to specific content standards is an outstanding way to gather uniquely appropriate information on the progress of students with significant disabilities. Through project based learning aligned to standards, in work, home, and community settings, students with significant disabilities can develop the academic and functional skills they need to be successful and demonstrate their progress in a context of high expectations. Here, again, partners in one rural education cooperative have shown specific strategies to make all students successful.

Reporting

States indicated whether they report the assessment scores of students who take tests in various ways with approved accommodations, non-approved accommodations (sometimes called modifications or non-standard administrations), alternate assessments, and whether students who were not assessed are included in reports. Almost all states, including most rural states, report students using approved accommodations, but just over half report the scores of students who use non-approved accommodations. The same number of states report scores of alternate assessment participants; however, about one-third of states, including three of the most rural states, have not yet made a decision about how to report these scores.

Consequences of Inclusive Standards, Assessments, and Accountability

State directors of special education and others who responded to NCEO's survey (Thompson & Thurlow, 2001) were overwhelmingly positive in their response to a request to describe consequences they had observed or heard about as a result of the participation of students with disabilities in state standards, assessment, and accountability systems. One director wrote, "The benefits seem to outweigh the negative consequences." Four rural states identified an increase in the number of students with disabilities included in their accountability system. Additional positive consequences for students with disabilities identified by rural states included increases in:

* Participation in statewide assessments;

* Performance on some state assessments;

* Access to the general education curriculum;

* Networking between general and special education teachers;

* Use of accommodations, including assistive technology; and,

* Educational rigor.

In addition to those identified by rural states, other states identified increases in:

* Academic expectations;

* Access to a general high school diploma;

* Parent awareness of standards, assessments and expectations; and,

* Teacher attention to achievement of standards assessed.

These outcomes were confirmed by a teacher in the rural district where efforts of general and special educators were used to develop strategies to improve learning for every student judged by assessment data and other measures. When educational evaluators visited this teacher's classroom they asked the teacher to point out the students with disabilities. The teacher challenged the evaluators to figure this out. The teacher told us that the evaluators could not determine which students were eligible for special education services, and this was the highlight of his teaching career! He commented, "We had to climb out of the silos of separate systems for special and general educators to do that, and it worked!"

Nothing new comes without cost, however, and there have been plenty of challenges as students with disabilities are included in standards, assessments and accountability systems. State directors in rural states identified two primary challenges: some students with disabilities are currently unable to participate successfully in state assessments but do not qualify for alternate assessment; and, the increased workload for teachers. It is interesting to note that the directors in rural states did not mention several of the challenges identified by urban states. For example, in the most urban states, these challenges were mentioned:

Including students with disabilities in the accountability system makes schools and/or teachers look less effective;

* Students with disabilities are traumatized/stressed out by taking tests;

* More students than necessary may be identified for alternate assessments in order to exclude them from the reporting system;

* Students with disabilities may not graduate; and,

* Some people question how students with disabilities can access or reach the state learning standards.

Using Assessment Data for School Reform

One of the districts in the cooperative had a comprehensive testing program supplemented by additional instructional and curricular data-gathering methods. Staff in that district relied heavily on teacher teams to identify areas of strength and weakness in instruction and curriculum and to design staff development to address issues. When the standards-based state assessments were implemented, the schools looked forward to additional information about their students.

As the planning team reviewed the results, there was concerned that some students were not at the proficient levels. When team members tried to understand why some students were not succeeding, or even which students were not succeeding for whatever reasons, the test data that were sent were not helpful. While team members were able to look at results by variables like disability status, ethnicity, and free and reduced lunch groupings, these numbers did not show any clear patterns from which the team could develop an instructional improvement plan.

One of the principals asked a teacher who taught computer analysis to help the team dig deeper into the data, to really "mine" the data for better information. With a good computer data base program team members were able to look at student results in many different ways. They also added locally collected data. It was found that students with learning disabilities and cognitive disabilities were achieving at rates that exceeded expectations, but students with emotional or behavioral disabilities were not. This information in itself did not help them identify solutions. They asked staff and parents of these students to work with them to identify what was working, and what was not. Parents and teachers identified a need for specific behavior management techniques that could be used consistently at school and at home. Training on the use of these techniques became an additional part of their staff development plan.

A key lesson learned about use of data for decision-making was the linkage to staff development plans. Once this district focused on understanding the data it had from multiple measures, staff members realized they had the key to planning individual and group staff development. Each year the board sets specific short and long range goals for the schools, and the administrative staff works to identify group staff development goals, based on data to help them reach board-set goals. The board also requires each teacher to develop an individual staff development plan, again with a data-based rationale, and linked to the board set goals. Each plan must have an evaluation component that includes a review of student work generated as a result of the teacher's training and new skills. This data-based link to student results has been a powerful data-based means for improvement.

Another finding resulting from the shift to databased decision-making involved discussions about the grade levels to target in order to improve student performance on specific skills. As the director of the cooperative told us:

We look at 1st grade classrooms to ensure that 8th graders will succeed. I had a superintendent in one of the districts call in a panic after the last state assessment. He wanted to get help fixing the problem in his 8th grade math class, because he had low 8th grade math scores. I asked him, "How are your kids in 1st grade doing on problem solving?" and he replied "I don't know how the 1st graders are doing, I just know we have low 8th grade scores." So I said, "Well, there is the problem! You need to be sure kids are getting the foundational skills they need all the way along. You can't fix the problem in 8th grade without thoughtful review of your data from 1st grade in many, if not most, cases."

As shown in this example, we found the same concepts over and over in mapping the curriculum in these districts, year after year, with big holes in the curriculum, no wonder it wasn't working. I had a principal just panic when he saw the gaps and overlap in their current approach. He wanted to know how to fix it all immediately. I told him that the first thing to do is to go have a beer-relax-there arc things we can do tomorrow, but we need to lay long range plans to fix it and to fix it in first, second, third, fourth, all the way through.

Summary

The shift to standards-based reform is challenging for all states, rural and urban. Development of inclusive assessment systems to measure progress toward academic content and achievement standards, school by school, is part of that challenge. All states' data show a trend toward more inclusive participation and improved performance on state assessments for students with disabilities. There are some interesting differences, however, between rural and urban states.

* Rural states showed more stable participation rates across years than urban states, which had more states showing an increase in participation rates. Rural state directors indicated that participation rates were high in previous years in their states, thus less change was anticipated.

* Fewer rural states had data available to validate changes in assessment performance levels, but of the rural states with data, more showed an increase in performance levels than urban states with data.

* More rural than urban states had data on accommodations use and more of these states showed stable use across years; more urban states showed an increase in assessment accommodation use. Similar to the change in participation rates, this may reflect relatively high use of accommodations in previous years, and, thus, less change was anticipated.

* Nearly all rural states use a portfolio (sometimes referred to as a body of evidence) approach to alternate assessment in contrast to urban states that use a variety of approaches. A portfolio approach is seen to be more time intensive for development, training, implementation, and scoring than, for example, a checklist approach, but it also yields rich and substantive information on quality of schooling for students with the most significant disabilities. It may be that the benefits of the portfolio approach are more feasible and thus justifiable when fewer students are involved.

Each of these differences, relatively high participation and accommodations rates, performance increases where data are available, and use of portfolios or body of evidence for alternate assessment, may relate to relatively low total student populations, fewer total number of schools in the state, and smaller administrative organizations at the state and district levels. Therefore, when the policy commitment from the state and district levels is in place to ensure all students participate in and benefit from standards-based reform, "smallness" may be an asset to implementation of inclusive systems. For example, in a rural, sparsely populated state, there tend to be fewer bureaucratic levels to maneuver to get necessary changes agreed to and in place. The ability to reach all educators with necessary staff development and support may be enhanced by lower populations as well. And as suggested by the staff in the rural cooperative described throughout this article, schools and school staff in rural sparsely populated areas are forced to collaborate, across district boundaries and across general and special education in order to serve every student well. Collaborative approaches help address a common challenge in rural small states, that of limited resources.

Even so, the benefits of "smallness" only will occur if there is clear state and district policy level commitment to gather performance data for all students. The story of one rural educational cooperative illustrates how information can be used in a rural setting to make a difference in student outcomes, and thus make a difference in school performance. By putting a "face" on assessment as one aspect of school reform, the purpose and the benefit for students and schools in rural states can be shown.

References

Fuchs, L.S., Fuchs, D., Eaton, S.B., Hamlctt, C., & Karns, K. (2000). Supplementing teacher judgments about test accommodations with objective data sources. School Psychology Review, 2P(I), 65-85.

Individuals with Disabilities Education Act Amendments of 1997, 20 U.S.C. section 1400 et seq.

Thompson, S.J., Quenemoen, R.F., Thurlow, M.L., & Ysselciyke, J.E. (2001). Alternate assessments for students with Disabilities. Thousand Oaks, CA: Corwin Press.

Thompson, S.J. & Thurlow, M.L. (1999). 1999 state special education outcomes: A report on state activities at the end of the century. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Thompson, S.J. & Thurlow, M.L. (2001). 2007 state special education outcomes: A report on activities at the beginning of a new decade. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Thurlow, M.L., Lazarus, S., Thompson, S., & Robey, J. (2002). 2001 state policies on assessment participation and accommodations (Synthesis Report 46). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Tindal, G., Helwig, R., & Hollenbcck, K. (1999). An update on test accommodations: Perspectives of practice to policy. Journal of Special Education Leadership, 12 (2), 11-20.

U.S. Census Bureau (1990). 1990 Census Data Locator. Author. Retrieved October 2002, from the World Wide Web: http://vvww.lib.umich.edu/govdocs/cenindex.html.

Sandra J. Thompson, Rachel F. Quenemoen, and Martha L. Thurlow

National Center on Educational Outcomes

University of Minnesota

The National Center on Educational Outcomes is supported through a Cooperative Agreement (#H326G000001) with the Research to Practice Division, Office of Special Education Programs, U.S. Department of Education. Opinions expressed herein do not necessarily reflect those of the U.S. Department of Education or Offices within it.

Correspondence concerning this article should be addressed to Sandra J. Thompson, National Center on Educational Outcomes, University of Minnesota, 350 Elliott Hall, 75 East River Road, Minneapolis, MN 55455, Phone: 612.625.5338, Email: thomp178@umn.edu

Special acknowledgement is extended to Pete Ziegler, director of the Minnesota River Valley Education District, and two of its member districts, Yellow Medicine East and Montevideo, for giving a "face" to education reform in rural America.

Copyright American Council on Rural Special Education Fall 2003
Provided by ProQuest Information and Learning Company. All rights Reserved