Defining high-quality career and technical education: Building a framework for best practice.

Hyslop, Alisha ; Imperatore, Catherine

The term "high-quality career and technical education" has become a national catchphrase in use by policymakers, practitioners and a wide variety of influential education and workforce development stakeholders. But what is high-quality CTE? How should this term be defined, and can it be used to evaluate programs, determine areas for targeted improvements and recognize successful elements that should be scaled?

CTE programs are subject to rigorous state and federal accountability systems that provide information on key student outcomes. However, while these outcome measures can form a basis for identifying high-and low-performing programs, they are insufficient for answering underlying questions about how or why that level of achievement was attained, and on their own do not provide the information necessary to identify and replicate best practices or to determine appropriate technical assistance for targeted program improvement. For these reasons, more extensive frameworks to define and measure program inputs and progress are necessary, and have become a major topic of conversation among education and workforce development leaders and policymakers.

However, there is no single source of information on what makes for a quality CTE program, and different states, national organizations and system stakeholders place different emphases on keyelements. To help synthesize the myriad voices that are a part of the dialogue on high-quality CTE, ACTE is embarking on a multi-step project to identify a comprehensive, research-based quality CTE program framework, test the framework and integrate it into our efforts to recognize and disseminate information on best practices within CTE.

Multiple Perspectives on Quality

The first phase of our project relates to identifying current sources of information on what constitutes high-quality CTE. A preliminary review of this area yielded three major sources of information: broad statements on CTE quality, such as general white papers or organizational positions; state frameworks and policy documents; and national frameworks.

As the conversation around high-quality CTE has grown, a number of groups and thought leaders have weighed in on the discussion, yielding a number of statements on CTE quality. CTE leaders themselves were some of the first to begin the conversation about high-quality CTE. For example, in 2010, the National Association of State Directors of Career Technical Education Consortium introduced its vision for a high-quality CTE system through its paper "Reflect, Transform, Lead: A New Vision for Career and Technical Education." This vision, which was endorsed by all 50 state CTE directors, laid out five principles designed to "guide CTE's role in our nation's educational, workforce and economic advancement and success." (1) Though not listed here, under each of the principles arc programmatic and policy actions for the field to take in order to make the vision a reality. The principles include:

* CTE is critical to ensuring that the United States leads in global competitiveness.

* CTE actively partners with employers to design and provide high-quality, dynamic programs.

* CTE prepares students to succeed in further education and careers.

* CTE is delivered through comprehensive programs of study aligned to the National Career Clusters[R] Framework.

* CTE is a results-driven system that demonstrates a positive return on investment.

Groups outside the traditional CTE system have put forth ideas as well. The following year, Harvard Graduate School of Education released "Pathways to Prosperity: Meeting the Challenge of Preparing Young Americans for the 21st Century." While not specifically focused on quality CTE, the idea permeates the paper, which notes a "growing number of rigorous, high-quality national models that demonstrate what career and technical education can achieve in the 21st century." (2)

The authors of this paper address inconsistencies in program quality and call for a national effort to improve both program quality and completion. Key quality elements embedded in the paper include:

* clear pathways to all major occupations

* stronger linkages between labor market needs and educational programs

* enhanced employer role

* increased work-based learning opportunities

* greater focus on career counseling

In late 2013, as the conversation continued, the Georgetown Law Center on Poverty, Inequality, and Public Policy; the Business Roundtable; and the College Board released "The Promise of High-Quality Career and Technical Education: Improving Outcomes for Students, Firms, and the Economy." This paper outlines some broad principles around what the authors believe constitutes high-quality CTE:

* Being part of career-oriented systems in secondary and postsecondary schools, with access for both youth and adults.

* An emphasis on strong career options for all students, including those bound for two-year and four-year colleges (to relieve the stigmatization of CTE programs and avoid the "tracking" of CTE students away from college paths).

* The integration of rigorous academic curricula into CTE, along with the teaching of rigorous technical and employ-ability skills in project-based or work-based settings.

* Professional development for staff and support services for students (especially the disadvantaged or those whose academic preparation has been weak).

* The use of appropriate assessment tools and accountability based on those tools. (3)

In laying out these criteria, the authors also assert that they "believe that high-quality CTE has enormous potential to successfully prepare all Americans--including the disadvantaged--for college and careers," but they recognize the challenges in replicating and scaling quality programs.

Around that same time, the American Federation of Teachers and the Albert Shanker Institute hosted the conference, Fulfilling the Promise of a Quality Education for All: 21st Century Career and Technical Education, and released a paper outlining thoughts on high-quality CTE. (4) These groups defined high-quality CTE programs as ones that:

* Are aligned with the Common Core and Common Career Technical Core standards.

* Employ teaching strategies and curricula that integrate career and technical subjects, as well as core academic subjects, in students' programs of study.

* Have as their foundation partnerships between educational institutions and businesses, community institutions and labor unions from all sectors of the economy (private and public, for profit and not-for-profit).

* Coordinate career and technical programs and sequences between secondary and postsecondary educational institutions.

* Provide educators with high-quality professional development that is embedded in their educational workplace, focused on real issues they confront in their work and sustained over a period of time. Incorporate appropriate technology.

* Wherever possible, provide internships and other work-based learning opportunities for students.

* Use high-quality performance assessments of technical skills.

In a presentation at that conference, James Stone, director of the National Research Center for CTE, summarized the research on CTE program quality and provided these four elements as critical: rigorous programs/curriculum, effective pedagogy, a systems approach (that fosters alignment between levels and sectors of education, as well as alignment with industry) and professional development. (5)

While these documents and statements have served to magnify the focus on CTE quality and provide initial points of discussion about critical elements, a more detailed set of criteria is necessary to assess quality at the program level. To help address this, almost every state has developed tools to provide CTE programs with quality indicators. These might exist in the form of program approval processes, program evaluation tools, or even legislation or other formal state policy.

In a similar regard, a number of national frameworks exist related to quality within CTE--such as the Department of Education's Rigorous Program of Study (RPOS) Framework and the CLASP Quality Career Pathways Indicators--that address a range of indicators from the systemic level to more granular classroom specifications. These national frameworks also include model-specific quality criteria, such as those related to career academies and Linked Learning.

It is with these quality frameworks that we at ACTE began our efforts to identify a set of research-based quality indicators that might be applied broadly to the CTE enterprise.

A Closer Look at Quality

To begin to make sense of the more specific quality frameworks already in use, ACTE is partnering with the Central Regional Education Laboratory (REL Central), located in Colorado. Within this partnership, ACTE is developing a summary and crosswalk of the various program-quality frameworks already in use nationally related to CTE, while REL Central is developing a similar crosswalk of CTE programs and policies in use in its partner states. We are sharing resources and methodology as we work on these complementary pieces.

To select frameworks--a term that encompasses standards, rubrics and other documents outlining key CTE program characteristics--developed by national organizations, we started with a list of frameworks brainstormed by CTE experts. We then added to this list through online searches using CTE-related key-words like "career pathways" and "career readiness."

We focused on frameworks that primarily addressed inputs, such as programmatic elements, teaching strategies and partnerships rather than student outcomes. The Perkins Act already provides a federally mandated set of program outcome measures, and as mentioned earlier, knowledge of inputs is necessary for program improvement and quality replication.

After the master list of documents was compiled, several frameworks were eliminated as not relevant to our particular evaluation. In most cases, this was either because the framework would not produce enough data, such as a short list of one-sentence statements, or because it addressed student outcomes rather than inputs.

We then sent our compiled list of frameworks to CTE researchers for their feedback. Throughout our evaluation process, we made some adjustments to the list as we discovered new frameworks or different versions of frameworks. More than 20 frameworks were eventually selected for inclusion in the project, including the RPOS Framework and CLASP career pathway indicators mentioned earlier, as well as others such as the National Career Academy Coalition Standards of Practice and documents related to program accreditation.

To evaluate each framework, we read it multiple times and classified it in relation to the following categories created by ACTE staff:

* developing organization

* research base (how the criteria were developed)

* purpose (formative or summative)

* topic and unit of analysis (state career pathway system, career academy, etc.)

* external participants (businesses, workforce development agencies, etc.)

* formulation of criteria (inputs or outcomes or both)

* number of criteria

* level and consistency of detail within criteria

* administration of criteria (self-or third-party assessment)

* complexity of response scale (how progress on each criterion is rated)

Where possible, we standardized our process across the frameworks to ease comparability. For instance, for two of our categories--level and consistency of detail, and complexity of scale--we devised a ranking system to compare the frameworks.

The next step in the process invoked coding individual content elements of each framework to create a crosswalk showing common elements, such as the incorporation of program of study/career pathway course sequences, the provision of career guidance and the establishment of partnerships with business. Coding was based on an initial set of codes and subcodes developed by REL Central for their work, which was then tweaked as necessary to accommodate the particular documents we were evaluating.

Initial Results

While the full results of our work comparing national quality frameworks are still being finalized and will be released later this month, (you can look for the paper on the ACTE website at www.acteonlinc.org/high-qualityCTE), some initial trends did emerge. Frameworks were identified related to a variety of units of analysis, ranging from individual CTE programs and institutions to broad education and workforce development systems that involve multiple partners and services. The majority were designed for both formative and summative evaluation, and many require either the input or cooperation of external partners as well.

The most evident trend is the great variation in the level of detail, consistency and content of the frameworks. Even though we excluded very general pieces from this part of the evaluation (such as the broad white papers highlighted earlier), documents still differed widely in complexity. A few contained numerous criteria and sub-criteria with detailed explanations of components, while most were less detailed, including only high-level practices. Quality indicators embedded within frameworks were structured in a variety of ways as well, from measuring achievement based on a simple "yes" or "no" response to measuring achievement based on step-by-step goals and growth over time.

Another trend was the lack of specificity in the research base. In several frame-works, expert participation in formulating criteria was cited. Some frameworks referenced alignment to research literature, and in a few, indicators were said to be based on the organization's experiences in the field. However, only a handful included details such as specific literature used or how a group of experts was brought to consensus.

Next Steps

Our review of the current landscape of CTE program quality illustrates the need for further research and refinement in this area. While not even considering the state-level documents that were part of REL Central's work, we found that the current national documents have a great deal of overlap, but enough key differences to create confusion among practitioners looking for practical quality tools for use at the local program level.

After this first phase of the project, our next steps include:

* Reviewing literature to determine evidence supporting elements uncovered in crosswalk work.

* Developing and validating a synthesized set of program quality elements.

* Pilot-testing an evaluation instrument based on the program quality elements and revising based on feedback.

* Sharing the final instrument, and developing and disseminating supporting materials for use in states and local programs.

* Incorporating the results of our high-quality work into broader ACTE activities, which might include the awards program, conferences or other professional development activities.

ACTE and REL Central will continue to work together on these elements as appropriate as the project unfolds, and will involve other relevant partners and stakeholders as well. We hope this work will help to provide more clarity to the high-quality CTE picture that is emerging across the country. Tech

Alisha Hyslop is the director, public policy, at ACTE. E-mail heratahyslop@acteonline.org.

Catherine Imperatore is the research manager at ACTE. E-mail heratcimperatore@acteonline.org

ENDNOTES

(1.) National Association of State Directors of Career Technical Education Consortium. (2010). Reflect, transform, lead: A new vision for career technical education. Retrieved from www.careertech.org/sites/default/files/2010-Vision-Paper.pdf

(2.) Harvard Graduate School of Education. (2011). Pathways to prosperity: Meeting the challenge of preparing young Americans for the 21st century. Retrieved from www.gse.harvard.edu/sites/default/files//documents/Pathways_to_Prosperity_Feb2011-1.pdf

(3.) Holzer, H., Linn, D., & Monthey, W. (2013). The promise of high-quality career and technical education: Improving outcomes for students, firms, and the economy. Washington, DC: The College Board and the Georgetown Law Center on Poverty, Inequality, and Public Policy.

(4.) Albert Shanker Institute. (2013). A quality education for all: A career and technical education policy agenda. Retrieved from www.shankerinstitute.org/sites/shanker/files/CTE-POLICY-AGENDA-3-10-9-131.pdf

(5.) Stone, J. R. III. (2013, October). What the research is telling us about high-quality CTE. Keynote address made at a conference hosted by the United Federation of Teachers and the Albert Shanker Institute, New York, NY.

By Alisha Hyslop and Catherine Imperatore