Correlates of the ETS field test in business--grades and choice of major.

Jares, Timothy E. ; Wilcox, William ; Allen, Garth 等

INTRODUCTION

More than 660 undergraduate business programs have administered the Educational Testing Service (ETS) Major Field Test in Business (ETSB) during the past three years to nearly 130,000 undergraduate business students (Educational Testing Service, 2013). The ETSB, first administered in 1990, evaluates the student's knowledge of core business topics typically taught in business core courses. Broadly defined, the topical coverage includes an assessment of accounting; finance; management; marketing; information systems; quantitative analysis; social and legal environment; and international issues and economics.

Standardized tests such as the ETSB represent one way of attempting to measure the outcomes of business programs by measuring actual student learning. Accrediting bodies consider tests such as the ETSB in the category of direct assessment. Direct assessment, where skills and knowledge learned in a given program are measured, is strongly preferred over indirect assessment where students, employers, and faculty are asked their opinions of the level of skills and knowledge gained by the students in a given program. An advantage of standardized tests such as the ETSB is the ability for institutions to benchmark the outcomes of their programs and for individual students to gauge their performance against a sample well beyond their own university. A disadvantage, however, is that there is little published evidence supporting a correlation between outcomes on the ETSB and the student's ability to use and apply actual business knowledge for future career success (Mason et al, 2011).

The Association to Advance Collegiate Schools of Business (AACSB) has strongly emphasized the assurance of learning accreditation component over the past decade. The increase in the use of the ETSB over this period is at least in part a result of AACSB's push for assurance of learning and direct measures of that learning. ETSB results provide one commonly accepted and objective measure of student learning. The fact that so many schools publicly share those results, and often issue press releases to promote their achievements, suggests that whether these schools believe the ETSB results are correlated with actual business knowledge or future career success or not, business schools absolutely want to improve their students' ETSB results.

Though other studies, e.g., Bycio and Allen, (2007) and Mason et al. (2011) have examined the determinants of success on the ETSB, our study employs an alternative model with a relatively large data set from the University of Northern Colorado (UNC). Throughout the period evaluated in our study UNC has scored consistently in the top 10% of schools administering the ETSB. Mason et al. (2011) develop a model they argue shows that ETSB outcomes are highly predictable--in sample and for one particular university. Bycio and Allen (2007) find a strong correlation with ACT/SAT scores.

The goal of this study is two-fold. First, we augment and strengthen previous research findings that suggest measures of innate intelligence and pre-college learning experiences (e.g., ACT scores) are associated with higher performance on the ETSB. Of course business schools could use these results and improve their ETSB results by simply increasing entrance standards. Though such an action might make sense for universities with far more applicants than vacancies, such a process does not advance the common university mission of improving performance through learning across a wide spectrum of abilities. The second goal of this research is to isolate and measure the impact of the university experience on resultant ETSB scores by using a process that controls the pre-college learning experiences. In short, by better understanding how factors representing university experiences impact ETSB scores, schools can revise curricula, allocate resources, even devise extra-curricular experiences to improve student performance.

LITERATURE REVIEW

Motivated by evolving accreditation requirements and by public pressure to justify the exploding costs of higher education, the assessment literature in higher education has increased greatly in both depth and breadth in recent years (e.g., Martell & Calderon, 2005). Two streams of this literature most related to this article focus on evaluation of outcomes and/or predictability of those outcomes and a related, but more difficult to assess, value-added proposition.

Evaluation/Predictability

In one of the evaluations of ETSB results, Allen & Bycio (1997) found a high correlation between ETSB scores and student business GPA and SAT scores. Thus, measures of incoming ability (SAT scores) and measures of business learning in the aggregate (business GPA) are associated with higher ETSB scores. They also found that student motivation (e.g., awarding a portion of a course grade based on ETSB performance) was highly correlated with higher ETSB scores. Consistent with their earlier study Bycio and Allen (2007) did not find gender differences in the ETSB. In their first study, accounting majors scored higher than other majors, while in the second study finance majors scored highest. In both studies, other groups such as marketing and management tended to score lower. Allen and Bycio (1997) note that the differences they found in accounting major ETSB performance were mirrored by equivalent differences in SAT scores, suggesting the higher scores achieved by accounting majors have more to do with academic ability than a bias in the test or superiority of preparation in the accounting discipline.

In research closely aligned in purpose to our study, Mirchandani, Lynch and Hamilton (2001) sought first to explain the variation in ETSB scores of their students, and then to leverage that information to improve either their curriculum or its delivery for the benefit of their students. SAT scores explained most of the variance in their sample. A quantitative factor, comprised of course grades in calculus, accounting, finance and operations management courses, was also significantly correlated with ETSB scores. While the authors suggested higher entrance standards would likely lead to an increase in ETSB scores, they noted such a reaction was not realistic at their institution, and, most importantly, it would do little to improve the value added by the business program. This is not to say, however, that selective programs admitting only students with high ACT scores cannot lever that approach into greater value-added programs because groups of superior students unencumbered by average or below average students may be able to absorb material at a faster rate allowing the faculty to cover more topics including complex topics in greater depth. The authors also suggested an increased concentration on material from courses comprising their quantitative factor and perhaps ETS review sessions would improve ETSB performance. While both of these actions might enhance ETSB scores, it is difficult to argue the latter would significantly improve the business program.

Bagamery, Lasik, and Nixon (2005) added to the results of Mirchandani, Lynch, and Hamilton (2001) by exploring the degree to which grades in core business courses helped explain ETSB scores. Similar to Mirchandani, Lynch, and Hamilton (2001), Bagamery, Lasik, and Nixon (2005) created factors by combining course grades. The studies share a similar finding in which quantitative courses explained more ETSB variance than non-quantitative courses, and the authors note that the Accounting I grade alone explained 10% of the total variance.

Black and Duhon (2003) also took the continuous improvement approach and suggested changes at the program level such as rewarding faculty members and departments whose students do well or improve. They also pointed out analysis of factors driving ETSB results can be used to identify characteristics of at-risk students, students with a low level of student learning, so that coursework might be designed for those students with weaknesses in specific areas. While there is much commonality among the business cores of most business schools and while such business cores are well represented by the ETSB exam content, results of studies such as this might lead to customized curricular and extracurricular activities for those with at-risk characteristics. For example, Black and Duhon (2003) estimated an abbreviated model in which the ETSB score is regressed on the ACT English score, a GPA calculated from introductory economics and financial accounting courses, age, and gender. They suggested the results of their model could be used to either advise or require students to take remedial courses or limit admission to upper-level business courses.

Mason et al. (2011) challenged the value of administering the ETSB. They evaluated many more parameters than prior studies and developed a sophisticated logistic model to predict ETSB performance. Highlights from the ordinary least squares results included statistically higher performance from males and from certain majors (i.e., accounting, finance, financial services, economics, and international business). Because they found that ETSB outcomes were highly predictable, they asserted the value-added in administering the ETSB was highly suspect. Alternatively, it is rational to conclude that predictability is the result of a well-designed and consistently implemented business school curriculum, and the ETSB provides external validation of the curriculum and student achievement. Additionally, identifying those factors that are associated with higher ETSB scores can help identify areas in need of resources or deserving of rewards, or as Black and Duhon (2003) suggest, curriculum changes or customization.

Value added in higher education

The value-added literature is motivated by accountability and is based on the proposition that improvement, and not necessarily the level of learning, is the most important measure of a business college's output. In value-added studies using the ETSB, researchers used pre/post-test designs to address their research questions. Not surprisingly, Jonas, Weimer, & Herzer (2007) found a significant difference in ETSB scores for both traditional and nontraditional students. The authors did not report ACT/SAT scores, but pre-test ETSB scores provided evidence of incoming academic ability. Incoming GPAs for nontraditional students were somewhat higher than traditional incoming students' GPAs. Perhaps somewhat surprisingly, they found that nontraditional students actually increased their ETSB scores more than traditional students as a result of the undergraduate business program. The nontraditional students in the study experienced a customized, cohort-based evening program while the traditional students attended typical day classes, and that difference, as well as other possible causal factors, made it impossible to identify the true source of the causes of the value-added differences. While the observed differences in post-assessment ETSB scores could have been a result of the curriculum design or the instructors, it also could have been a function of students' life experiences and/or their motivation to learn.

Rook and Tanyel (2009) not only showed the value-added of their business program through a pre/post-test study, but they sought to explain the change in their students' performance through measures of student performance in their curriculum. They found that both the upper-level core and overall GPAs were positively correlated with the ETSB scores of those completing their course of study. Though their models explained a relatively small amount of change in either the ETS raw score or percentile, Rook and Tanyel's analysis suggested that a business student with a core GPA one point higher than another student could expect, on average, an ETSB score about 6 points higher. Rook and Tanyel's research did not explore how subsets of the curriculum or even specific course outcomes might explain differences in ETSB outcomes.

THE HISTORY OF THE ETSB AT THE MONFORT COLLEGE OF BUSINESS

Since 1992, just two years after the ETSB introduction and about the time the college began initial pursuit of AACSB accreditation, the Monfort College has required every bachelors graduate in Business Administration to complete the ETSB assessment. Though assurance of learning was not nearly as prominent in the AACSB standards at the time, college leaders felt direct measures of learning in the business core were important. Furthermore, the ETSB provided credibility with external audiences (e.g, parents, university administrators, state regulators, prospective students, etc.) for the college as an externally validated, benchmarked measure of the college curriculum and faculty. Arguably that credibility has strengthened as the number of schools administering the exam has increased significantly in the past decade, paralleling a marked increase in emphasis on assurance of learning by AACSB.

Throughout the past twenty years the faculty developed clear processes that triggered review and subsequent changes enhancing either the curriculum or the delivery of the curriculum. For example, on an annual basis if the college falls below the 60th percentile in a topical sub area (or overall) or there is an overall downtrend for two consecutive years, the department chair and faculty from that content area are required to review the results and provide a formal response to the college curriculum committee. At least partly as a result of these processes, the college has not experienced a score (overall or in any subarea) below this level in more than ten years.

There were instances, however, in which the college struggled to increase scores, particularly in certain topical sub areas. In one case it was evident that the college scores in the legal and social environment content area were consistently poor. A few years prior to the decline the college decided not to replace a permanent faculty member and instead filled those courses with adjunct faculty. Inconsistency of content delivery caused by adjunct faculty turnover appeared to be driving the poor performance. Not long after the hiring of a permanent faculty member to teach this content area, ETSB scores in the legal and social environment content area improved and stabilized. In another example, as a result of consistently poor performance on the finance portion of the ETSB and per the Department of Management's request, the college revised the required curriculum for management majors to include a second course in finance. The result of this change also led to a noticeable and consistent increase in scores for those students.

Consistent with other examples in the literature, e.g. Mirchandani, Lynch, and Hamilton (2001), our college has taken steps to ensure genuine and full student efforts when taking the exam. Obviously the college likes to see results showing its students are among the best taking the exam, but perhaps more importantly, because the results are used to make operational changes, it is important for ETSB scores to reflect actual student learning and not the fiat of a disingenuous effort. The ETSB is a required component of the business core capstone course and student performance on the exam comprises 10% of their course grade. Additionally, the college has worked for years to make the ETSB a part of the culture of the college and achievement on it a point of pride. At presentations introducing prospective students to the college, historical results are provided and the importance of continuous improvement is emphasized. Results from the ETSB are publically displayed every semester and press releases are provided. Students scoring in the top 10% nationally on the exam are recognized with a letter from the Dean, through press releases, and with their name prominently displayed on a bulletin board in the business building.

While the examples cited do illustrate attempts to use ETSB results for continuous improvement, there is a more comprehensive approach that this study will attempt to develop. In order to make appropriate inferences, colleges must consider factors that their students possess prior to their business school experience. In addition, there is a common body of curriculum that helps to add value to the student's performance. And finally, there are other demographic variables that have an association with ETSB performance. The authors are attempting to develop a model that considers all of these factors together, so that colleges can make better use of the ETSB results when making curriculum and resource decisions in hopes of continuous improvement.

SCOPE OF THE STUDY

The principal interests of the investigators/researchers are twofold: (1) Examine the relationship between ETSB results and an individual's college-level academic measures and pre-college performance measures and (2) Determine the influence of post high school academic performance and college major on ETSB results after removing the influence of a student's pre-college academic measures.

The first objective is fairly straightforward. The means, standard deviations and correlations are presented and discussed across the pre-college and college-level academic measures including the results on the ETSB. These are discussed in the first part of the analysis section.

To accomplish the second objective, a two stage step-up multiple regression analysis is developed. An initial model is developed that considers the ETSB as the dependent variable and a set of pre-college measures as independent variables. This model provides data about the influence of these pre-college measures on the ETSB. This initial model is then reduced to a subset of individually significant variables via a backward elimination procedure.

A second model is then developed with the ETSB as the dependent variable. The independent variables include two sets of variables. The first set is the reduced set of pre-college variables and the second set is the college-level performance measures. This model provides an opportunity to examine the influence of the college-level variables after controlling for the effects of the pre-college measures. That is, what are the contributions of the college-level variables after removing the influence of the pre-college measures?

Symbolically, the two models are as follows:

Model 1: ETS Field Test = f(Pre-college performance measures)

Model 2: ETS Field Test = f(College-level measures, (Pre-college performance measures))

Through this process we seek to explain the variation in our students' ETSB scores with a minimal set of pre-college and college-level measures. The next section describes the full set of dependent and independent variables included in this study.

Model Variable Explanation

Dependent Variable

ETSB. The ETSB assesses knowledge of the core areas in Business Administration. It includes content in the primary areas of Accounting, Finance, Marketing, and Management, but also in related areas such as Quantitative Analysis, Legal and Social Environment, Information Systems, and Economics and International Issues. A composite performance measure is provided for each student and content subscores are provided in the aggregate. The composite measure is used in this analysis. As described earlier, every graduating student is required to take the exam. It is administered in the capstone business course which is commonly taken in a student's final term or close to the final term. Over the sample period the college average was consistently in the top 10% of those institutions administering the exam.

Independent Variables

Pre-college measures

ACT Composite. The composite measure is an average of the four parts noted below. The optional Writing test was not included. The original standardization of the exam yielded a mean of 18 and standard deviation of 6.

ACT English. The English test covers usage/mechanics and rhetorical skills.

ACT Math. The Mathematics test covers pre-algebra, elementary algebra, intermediate algebra, plane geometry, coordinate geometry and elementary trigonometry questions.

ACT Reading. The Reading section consists of four ten-question passages, from the realms of prose, humanities, social science, and natural science

ACT Science. The Science Test measures the skills required in the natural sciences: interpretation, analysis, evaluation, reasoning, and problem solving.

High School GPA. The student's cumulative high school Grade Point Average (GPA) as reported by the high school attended.

College-level Measures

University GPA. This measure is the undergraduate GPA for all coursework taken at the University of Northern Colorado. It includes both business and non-business courses. It includes both passing and failing grades.

Business GPA. This measure is the undergraduate GPA for all business courses completed at the University of Northern Colorado. Only courses offered by the business college are included. Business program requirements not offered by the business college are not included in this measure. As with the University GPA, this measure includes both passing and failing grades.

Major. Within the Business Administration program, six emphasis areas are offered (emphasis areas are hereinafter referred to as majors thus using the nomenclature common to most universities). A student may choose to major in Accounting, Finance, General Business, Information Systems, Management, and Marketing.

Accounting 1 grade. The grade reported for the student in the BAAC 220 Accounting I class. It is an introductory financial accounting class required of all business majors.

Accounting 2 grade. The grade reported for the student in the BAAC 221 Accounting II class. It is an introductory managerial accounting class required of all business majors.

Finance grade. The grade reported for the student in the BAFN 370 Finance class. It is an introductory corporate finance class required of all business majors.

Information Systems grade. The grade reported for the student in the BACS 300 Information Systems class. It is an introductory information systems class required of all business majors.

Marketing Principles grade. The grade reported for the student in the BAMK 360 Marketing Principles class. It is an introductory marketing class required of all business majors

Management of Organizations grade. The grade reported for the student in the BAMG 350 Management of Organizations class. It is an introductory management class required of all business majors

DESCRIPTION OF SAMPLE

Four hundred and forty-eight (448) students were selected for analysis. The sample represents all students who entered the university as freshman, complete high school academic and ACT measures, completed all core requirements at the University of Northern Colorado, and completed the ETSB and graduated during the period 2007 to 2011.

Continuation and graduation requirements require each student to pass each core class with a grade of C or better. In addition, the student must maintain a 2.0 Business GPA to continue in the program and to graduate. Therefore, all individual core course grades of the students taking the ETSB are A, B, or C.

ANALYSIS OF DATA

Table 1 provides the means and standard deviations for the ACT Composite, ACT subscales and HSGPA.

The data in Table 1 support several observations.

* ACT Composite and ACT subscale means are substantially higher than the normative standard of 18 established when the exam was normed.

* Accounting, Finance and Information Systems majors demonstrate above average ACT performance for the college's graduates, while General Business, Management and Marketing majors are below the college-wide average on the Composite measure. Across the subscales, in general, this division held up.

* High School GPA is lower for Information Systems, Management and Marketing majors while other majors are at or above the overall college average.

* Within the subscales, Math and Science averages exceeded the Composite, while Reading and English averages fell below the Composite.

Table 2 presents the means and standard deviations for the ETSB, University GPA and Business GPA. Major counts are also noted. Several observations can be made from the data in the table.

* Accounting, Finance and Information System majors demonstrate above average test results while demonstrating above average University and Business GPAs. Finance majors average ETSB score was almost a standard deviation above the lowest performer, Marketing majors.

* Across all majors, the Business GPA is consistently lower than the University GPA.

* Average performance by major on the ETSB demonstrated some fidelity with Business GPA but less than perfect.

* Information Systems and General Business had small enrollments totaling less than 13 percent of total graduates.

Table 2 presents the Pearson correlations among ACT measures, the ETSB results, and High School GPA as well as University and Business GPAs.

From Table 3, several observations can be made:

* Correlations among the ACT subscales, in general, are moderate and all are positively related to each other. This was expected. These associations produced some collinearity in the initial regression model.

* ETSB correlations are positive and moderate with ACT results and weak for HSGPA. All relationships were positive and as expected.

* Both University and Business GPAs show moderate, but stronger, associations with ETSB results than the ACT subscales do with the ETSB.

* University and Business GPA are almost perfectly correlated. For the regression models presented in the next section, University GPA was dropped for collinearity reasons.

Multiple Regression Analysis--Model 1

The purpose of this first model is (1) to measure the association between pre-college variables and the ETSB results and (2) use the model results to partial out the effects of pre-college knowledge or aptitude in a subsequent analysis that relates college-level measures with the ETSB results. Demographics, such as gender, were also explored though a preliminary model, but were not found to be significant in our sample.

Table 4 presents the full and reduced models of the ETSB against the ACT subscales and HSGPA. Recall the ACT Composite is a linear transformation of the subscales and, as such, was omitted from the analysis.

The full model produced an R-squared equal to .384. ACT English and HSGPA were non-significant in the model. Through a backward elimination procedure, these two variables were removed and the subsequent reduced model R-squared was .383, a minor reduction in the model's explanatory power. The regression coefficients are in the expected direction and all are significant at the .01 level. The removal of ACT English and HSGPA was largely a product of collinearity with the other subscales.

Multiple Regression Analysis--Model 2

Model 2 includes the variables from the reduced model and adds Business GPA, University GPA, Major, and grade in each of six core business classes: Accounting 1, Accounting 2, Finance, Management, Marketing, and Information systems. Business and University GPA are overall performance measures whereas grades in individual core classes bring a focus to performance in a specific discipline.

Major and grade were implemented as binaries in the model. Each major, Accounting, Finance, Information Systems Management and Marketing were represented by a unique binary. The General Business binary was eliminated to remove the linear dependency created by its presence. For grades, a binary was created for grades of 'A' and another for grades of 'B' in each class. The binary for grades of 'C' were omitted because of their linear dependency with the 'A' and 'B' grades. A minimum of a 'C' grade was required for each of the core classes, thus creating the dependency.

Table 5 presents the Full model for the Pre-college reduced set and the full set of College-level variables. For clarity, variables' p-values are not presented for insignificant coefficients.

The R-squared for this full model is .590, substantially larger than .383 from the Stage 1 full or reduced models. College-level variables explain an additional 20 percent of the variability in ETSB results. From Table 4, one observes that Business GPA, certain majors, and grades in select core business courses are significant.

Table 6 presents the Stage 2 reduced model created by a backward elimination of the college-level variables.

The full model produced an R-squared equal to .590. Numerous factors were insignificant in the full model. Through a backward elimination procedure, these variables were removed and the subsequent reduced model R-squared was .576. As with the Pre-College model there is a minor reduction in the model's explanatory power. The regression coefficients are in the expected direction and all are significant at the .01 or .05 level. While the magnitude of some coefficients changes slightly, directionality is unchanged.

This final model suggests core business course grades, major choice and Business GPA influence ETSB results. An increase in R-squared of close to 20 percent was maintained after the elimination of non-significant variables. Consistent with Rook and Tanyel (2009), we find that a student with a Business GPA one point higher than another student should expect an ETSB score about 5 points higher. Our results also suggest that choice of the finance major add nearly 4 points to the expected ETSB score. Such an incremental change might move a student from an average ETSB score (152.4-50th percentile) to an expected score of 156 (60th percentile). The most significant incremental difference, however, is that a student earning an "A" in the introductory Finance course should expect an ETSB score more than 8 points, or a percentile improvement in the range of 20%.

Rather than interpret the differences in ETSB results as measured by the magnitude of the regression coefficients, Table 7 provides an instructive look at the mean performance on the ETSB across core business class grades, choice of major and Business GPA.

From Table 6, several observations are evident:

* For each core business class grade, mean performance on the ETSB was highest for the 'A' grade, middling for the 'B' grade and lowest for the 'C' grade. The overall means noted in the table demonstrate a remarkable consistency.

* This was true within each major and overall. The only exception to this was for Information Systems majors in the Management course. One student was in this category.

* Finance majors consistently demonstrated higher ETSB means across all core classes. Marketing majors demonstrated the lowest.

The methodology employed in this study as well as the sample size and composition suggests both caution with respect to results interpretation and opportunities for future research. First, to our knowledge, like this study, all previous related research has been conducted with samples from a single university. The literature could be greatly enhanced with the addition of studies comprised of samples from multiple universities, perhaps from varying regions.

Second, our sample size was adequate but limited because we removed those students transferring grades from core business courses considered in this study. Future studies could expand on this research by including course outcomes from community colleges and other universities.

A third limitation is that, due to course prerequisites, only students who received a grade of C or better in the core business courses were allowed to take the ETSB exam. Students who were unsuccessful in mastering the content of a course at a competent level (grade of C) were not included in this study and could have added an important piece of information as to the value-added of a course's content to success on the ETSB.

The final limitation is that the ETSB in this instance is simply a measure of the level of competency. Because there are no pre-and post-test results, we were unable to measure the amount of value added by the curriculum in a more direct fashion.

RECOMMENDATIONS FOR FUTURE RESEARCH

Key observations resulting from this study strengthen similar conclusions from other studies. Consistent with other studies, results of this study confirm that measures of aptitude (pre-college ACT scores) are indeed consistent with higher ETSB scores. Colleges without enrollment pressures could consider increasing entrance standards as a means to enhance graduate ETSB scores.

This study augments prior research in two primary ways. First, grades discriminate across ETSB performance results. Faculty issued grades consistently differentiated across performance in every core class considered. The ETSB evaluation confirms the performance observed by the individual faculty members responsible for core classes.

Second, the study describes the types of students and types of activities and experiences that highly performing ETSB test takers share. The fact that those graduating with a finance major and those earning either an 'A' or 'B' in finance are high performers is interesting and largely consistent with other studies that have found that accounting majors (e.g., Allen and Bycio, 1997) tended to perform better on the exam. Allen and Bycio concluded that the superior performance had more to do with academic ability than test bias. In our study Accounting, Finance, and CIS majors had higher ETSB averages and the higher average ACT composite scores are driven by higher subscales in Math, Science and Reading. If these subscales influence ETSB results more than English, this may help explain higher ETSB results for Accounting, Finance, and CIS majors.

It is unclear if a bias exists in the ETSB, but because our models control for academic ability, perhaps more research on test bias is worthwhile. This study shows the introductory finance course to be a strong predictor of superior ETSB performance in addition to the choice of the finance major, perhaps the problem-solving skills and the finance curriculum is structured in a way that reinforces and strengthens the content learned in other business disciplines.

From an evaluation perspective, ETSB results may offer insight into faculty performance or curricular design. As part of a faculty evaluation model, how well grades correlate with ETSB performance might be considered (as long as the course topics bear some relationship to the ETSB topics examined). Similarly, as part of the curricular evaluation process, do the topics covered in a core class include, at minimum, the topics covered on the ETSB? Lastly, ETSB performance might serve as a long-term performance improvement measure and might be considered as part of an evaluation process either for individual faculty or, perhaps, for a department.

As noted earlier in this paper, in the aggregate graduating students from the University of Northern Colorado have averaged in the top 10% of institutions administering the exam for more than ten years. One possible reason for that outcome could be a relatively high percentage of accounting and finance majors among its graduates. Further research in this area would be useful and could inform enrollment management decisions.

COMMENTARY AND CONCLUSIONS

Higher education costs continue to increase, in part due to decreases in public funding, putting greater pressure on increasing enrollments at colleges and universities. If incoming students do not immediately require measures of outcome results as central to selection criteria, many parents and other stakeholders will. Business colleges will compete with other colleges on campus for enrollments, with other business colleges, and even with community colleges that provide lower-cost alternatives for many core business offerings. This may also suggest that the greater the admissions selectivity of a business college the better the performance on the ETSB exit measures.

In summary, the results of this study suggest a process business college administrators and curriculum developers might incorporate to best prepare their students for direct assessment measures such as the ETSB. Modifications to our model could also be used to design customized experiences, perhaps online and relatively inexpensive, for students based on the results of their lower division coursework. In short, informed analysis such as this can better inform both overall curriculum design and specific experiences for learners with varying aptitudes and learning styles.

REFERENCES

Allen, J.S., & Bycio, P. (1997). An Evaluation of the Educational Testing Service Major Field Achievement Test in Business. Journal of Accounting Education, 15(4), 503-514.

Bachelor's Degree in Business. (2013). Major Field Test Comparative Data Guide. Educational Testing Service, n.d. Web. 26 Sept. 2013.

Bagamery, B.D., Lasik, J.J., & Nixon, D.R. (2005). Determinants of Success on the ETS Business Major Field Exam for Students in an Undergraduate Multisite Regional University Business Program. Journal of Education for Business, (81(1), 55-63.

Black, H.T., & Duhon, D. L. (2003). Evaluating and Improving Student Achievement in Business Programs: The Effective Use of Standardized Assessment Tests. Journal of Education for Business, 79(2), 90-98.

Bycio, P., & Allen ,J.S. (2007). Factors Associated With Performance on the Educational Testing Service (ETS) Major Field Achievement Test in Business (MFAT-B). Journal of Education for Business, 82(4), 196-201.

Jonas, P.M., Weimer, D., & Herzer , K. (2007). Comparison of Traditional and Nontraditional (Adult Education) Undergraduate Business Programs. Journal of Instructional Psychology, 28(3), 161-170.

Martell, K., & Calderon, T. (Eds.) (2005). Assessment of Student Learning in Business Schools: Best Practices Each Step of the Way. Tallahassee, FL: Association for Institutional Research.

Mason, P.M., Coleman ,B. J., Steagall, J. W., Gallo, A. A., & Fabritius, M, M. (2011). The Use of the ETS Major Field Test for Assurance of Business Content Learning: Assurance of Waste? Journal of Education for Business, 86(2), 71-77.

Mirchandani, D., Lynch, R., & Hamilton, D. (2001). Using the ETS Major Field Test in Business: Implications for Assessment. Journal of Education for Business, 77(1), 51-56.

Rook, S.P., & Tanyel, F.I. (2009). Value-added Assessment Using the Major Field Test in Business. Academy of Educational Leadership Journal, 13(3), 87-94.

Timothy E. Jares

William Wilcox

Garth Allen

Robert Lynch

University of Northern Colorado

Timothy E. Jares is an Associate Professor of Finance at the Monfort College of Business at the University of Northern Colorado. He earned his Ph.D. in finance from the University of Nebraska. Dr. Jares has published articles in the Journal of Financial Research, the Journal of Financial Services Research, Advances in Financial Education, and the Journal of Teaching in International Business.

William Wilcox is an Associate Professor of Accounting & CIS at the Monfort College of Business at the University of Northern Colorado. He earned his Ph.D. in accounting from the University of Nebraska. Dr. Wilcox has published articles in the Journal of Accounting & Public Policy, Advances in Accounting, Issues in Accounting Education, and the Journal of Marketing.

Garth Allen is an Associate Professor of Finance at the Monfort College of Business at the University of Northern Colorado. He earned his Juris Doctor at the University of Iowa. His interests and publications are in the areas of risk management, insurance, business law, education, and financial planning.

Robert Lynch is a Professor of Business Statistics at the Monfort College of Business at the University of Northern Colorado. His interests are in linear models, statistical computing and data mining. He holds a Ph.D. from the University of Northern Colorado in Applied Statistics.

Table 1
Means and Standard Deviations for Pre-College Measures

                   ACT Composite     ACT English        ACT Math

Major Emphasis   Mean    Std Dev   Mean    Std Dev   Mean    Std Dev

Accounting       23.37    3.25     22.29    4.50     24.05    3.79
Finance          23.78    3.58     22.48    4.78     24.41    3.92
Info Systems     23.63    4.14     22.05    3.97     24.32    5.31
Gen Business     22.84    3.36     22.42    3.77     22.97    3.91
Management       22.21    3.30     21.45    4.22     22.47    3.84
Marketing        22.40    3.00     22.29    4.16     21.56    3.45

Overall          22.90    3.36     22.13    4.33     23.05    3.97

                   ACT Reading       ACT Science         HS GPA

Major            Mean    Std Dev   Mean    Std Dev   Mean    Std Dev

Accounting       23.52    4.25     23.43    3.14     3.50     0.46
Finance          24.18    5.19     23.90    3.18     3.44     0.47
Info Systems     24.11    5.67     24.11    3.59     3.39     0.39
Gen Business     23.13    4.69     22.76    4.04     3.41     0.40
Management       22.53    4.57     22.35    3.48     3.34     0.46
Marketing        23.40    4.42     22.22    2.70     3.35     0.49

Overall          23.38    4.68     22.94    3.29     3.40     0.46

Table 2
College-level measures and major counts

Major            ETS Field Test    Major Count

                Mean    Std Dev    N      %

Accounting     163.98    11.17    87    18.4
Finance        167.59    12.53    98    20.8
Info Systems   163.21    13.44    19     4.0
Gen Business   160.42    11.85    38     8.1
Management     159.75    11.24    111   23.5
Marketing      157.02    11.23    119   25.2

Overall        161.66    12.20    472   100.0

Major           University GPA    Business GPA

               Mean   Std Dev   Mean   Std Dev

Accounting     3.24    0.48     3.08    0.59
Finance        3.11    0.44     2.90    0.51
Info Systems   3.21    0.29     3.07    0.38
Gen Business   2.94    0.41     2.67    0.47
Management     3.01    0.38     2.81    0.46
Marketing      3.03    0.40     2.81    0.47

Overall        3.08    0.43     2.88    0.51

Table 3
Correlations among the ETS Field Test, Pre-College and University
measures

                      Pre-College measures

           ETS Exit   ACT Comp   ACT Eng   ACT Math

ETS Exit   1.000
ACT Comp   0.606      1.000
ACT Eng    0.458      0.851      1.000
ACT Math   0.497      0.775      0.553     1.000
ACT Read   0.523      0.832      0.658     0.449
ACT Sci    0.527      0.806      0.574     0.607
HS GPA     0.205      0.402      0.379     0.374
Univ GPA   0.549      0.438      0.404     0.343
Bus GPA    0.580      0.451      0.401     0.355

           Pre-College measures          University

           ACT Read   ACT Sci   HS GPA   Univ GPA   Bus GPA

ETS Exit
ACT Comp
ACT Eng
ACT Math
ACT Read   1.000
ACT Sci    0.585      1.000
HS GPA     0.268      0.277     1.000
Univ GPA   0.343      0.364     0.516    1.000
Bus GPA    0.354      0.389     0.495    0.956      1.00

Correlations are all significant with p < .05

Table 4
Stage 1 Linear Regression Model

Full Model
[R.sup.2] = 0.384 F = 57.98 p < .01

              Coefficients   p-value

Constant        110.560
ACT English      0.960        0.526
ACT Math         0.729        <.01
ACT Reading      0.731        <.01
ACT Science      0.767        <.01
HS GPA           -0.746      0.480

Reduced Model -(Backward elimination of Full Model)
[R.sup.2] = 0.383 F = 96.64 p < .01

              Coefficients   p-value

Constant        109.017
ACT Math         0.729       <.01
ACT Reading      0.763       <.01
ACT Science   0.784          <.01

Table 5
Stage 2--Full Model--ETS Field Test against
Pre-College and College-Level Variables

[R.sup.2] = .590 F = 29.362 p < .01

                          Coefficients   p-value

Constant                    115.163

Retained Pre-University
  Variables
ACT Math                     0.316        <.02
ACT Reading                  0.628        <.01
ACT Science                  0.463        <.01

University Variables
Business GPA                 5.329        <.01
University GPA               -1.636

Accounting Major             -2.681
Finance Major                1.933        <.05
Info System Major            -2.698
Management Major             -0.897
Marketing Major              -3.560       <.05

Accounting 1 Grade A         4.075        <.01
Accounting 1 Grade B         1.807
Accounting 2 Grade A         -0.072
Accounting 2 Grade B         -0.332
Finance Grade A              8.702        <.01
Finance Grade B              2.789        <.01
Marketing Grade A            3.821        <.05
Marketing Grade B            2.270        <.05
Management Grade A           -1.358
Management Grade B           1.005        <.05
Info Systems Grade A         -1.106
Info Systems Grade B         -1.261

Table 6
Stage 2 Reduced Model--University Variables with
Retained Pre-University Variables

[R.sup.2] = .576 F = 62.566 p < .01

                           Coefficients   p-value

Constant                     109.375
Retained Pre-University
Variables
  ACT Math                    0.315        <.01
  ACT Reading                 0.615        <.01
  ACT Science                 0.475        <.01
University Variables
  GPA
    Business GPA              5.153        <.01
  Major
    Finance Major             3.620        <.01
    Marketing Major           -2.088       <.05
  Core Grades
    Accounting 1 Grade A      2.549        <.05
    Finance Grade A           8.439        <.01
    Finance Grade B           2.771        <.01
    Management Grade B        1.770        <.05

Table 7
Mean ETS Field Test result by Course Grade, Major Emphasis and Core
Class

Core Class             Accounting I              Accounting II

Grade            A        B        C        A        B        C

Emphasis        Mean     Mean     Mean     Mean     Mean     Mean

Accounting     167.84   159.97   158.88   169.86   160.16   156.73
Finance        175.28   167.11   160.68   174.32   166.77   160.00
Info Systems   174.00   164.25   152.83   169.17   161.00   160.00
Gen Business   165.80   163.26   153.92   174.25   158.07   155.40
Management     169.70   160.97   155.09   169.47   160.72   154.92
Marketing      166.75   158.10   153.72   166.00   159.95   152.95

Overall        169.98   161.82   155.52   170.80   161.54   155.27

Core Class               Finance

Grade            A        B        C

Emphasis        Mean     Mean     Mean

Accounting     173.56   161.81   157.60
Finance        179.88   166.67   160.13
Info Systems   177.50   165.67   156.88
Gen Business   168.75   163.82   158.14
Management     177.50   164.87   156.01
Marketing      174.14   161.74   153.61

Overall        175.94   164.00   155.87

Core Class            Info Systems                 Management

Grade            A        B        C        A        B        C

Emphasis        Mean     Mean     Mean     Mean     Mean     Mean

Accounting     171.06   162.74   154.00   170.11   158.97   153.50
Finance        176.63   166.66   162.88   172.26   165.63   154.20
Info Systems   169.57   160.56   156.33   163.33   162.33   176.00
Gen Business   160.33   161.57   157.56   160.91   162.00   155.14
Management     172.20   159.26   155.32   169.72   157.27   154.87
Marketing      162.94   156.87   154.10   163.79   154.74   151.07

Overall        170.12   160.90   156.58   168.33   159.22   153.91

Core Class              Marketing

Grade            A        B        C

Emphasis        Mean     Mean     Mean

Accounting     172.37   162.16   156.24
Finance        177.29   167.85   156.86
Info Systems   178.00   162.62   151.00
Gen Business   166.00   163.71   156.65
Management     171.45   161.34   153.76
Marketing      167.74   155.92   151.06

Overall        172.20   161.73   154.39

Grade            A        B        C

Overall        171.23   161.54   155.26
(unweighted)