Evaluating the assessment outcomes in the principles of marketing course.

Gerlich, R. Nicholas ; Sollosy, Marc

INTRODUCTION

Increasingly, academic programs look to assessment as an important vehicle in determining the overall veracity of a program. It is a means by which the deliverer (instructor) of an academic program can determine if the goals and objectives of the program have been achieved by the recipients (students). Institutional effectiveness is concerned with the extent to which intended outcomes are being achieved (Black & Duhon, 2003). There are two (2) fundamental schemas of assessment delivery. The first employs an instrument developed externally and is standardized across a multitude of dimensions, like the Educational Testing Service's (ETS) exam in business. The second approach employs an embedded, internally developed, instrument that explicitly measures the specific outcomes associated with a program or course.

The purpose of this paper is to examine the various factors or drivers that influence a student's performance on an embedded, course specific assessment (local). These factors or drivers are described along two (2) dimensions, environmental or external, and internal student drivers. The assessment was administered to multiple sections of students in the same foundation or principles level course. The delivery of the actual course material was done both in the traditional in class model, as well as the increasingly popular Online delivery method. The results of this study are derived in the College of Business at a public university located in the Southwestern part of the United States. The institution is mid-sized with a total enrollment of approximately 7,500 total students, 1,000 undergraduate business students, 350 graduate business students.

This manuscript is organized as follows: First, a literature review is provided. The second section of the manuscript describes the data and the model. The next section provides empirical results testing the performance of students on the assessment exam while controlling for format of delivery, professor, student grade point average (GPA), international vs. domestic status, standardized test scores (ACT), transfer students, and gender. The final section offers conclusions and implications.

LITERATURE REVIEW

Research abounds on the subject of program assessment. Assessment is a "systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development" (Palomba & Banta, 1999). Collegiate business programs are increasingly tasked with the need for ongoing assessment of student performance in their programs (Adams, et al, 2000; Bagamery, et al, 2005; Martell & Calderon, 2005; Terry, et al, 2008; Trapnell, 2005). Increasingly, since the mid-1980s, there has been a shift towards student-centered and learning-oriented assessments and accreditation (Lubinescu, et al, 2001). In fact, the AACSB imposes standards for program learning goals upon collegiate business programs aspiring to attain or maintain AACSB accreditation. These programs utilize direct measures in order to demonstrate student achievement of the stipulated goals (Martell, 2007; Pringle & Michel, 2007).

As assessment increases to build momentum, it is important to identify the internal and external audiences who will utilize the results in shaping and refining the assessment process. A comprehensive assessment process provides an institution with information that can be both shared and utilized to satisfy the needs of internal and external constituents. The internal audience (faculty, students, assessment committees, administrators and alumni) benefits by helping to define successful ongoing programs, implementing similar programs, and for improving less successful programs. Externally, the assessment data are used to demonstrate to accreditation organizations, government officials, government boards and other constituents the institution's effectiveness and accountability (Aloi, et al, 2003).

There are two principle types of assessment tests, standardized and local. Of the two, local tests require more faculty effort and other resources for test development, scoring, reporting, and improving. However, the advantage to the local instrument is that it can be tailored to a specific course or program so that the actual scores more accurately reflect the extent to which specific learning objectives are being met (Black & Duhon, 2003), along with impact of local and specific influences or drivers.

An important aspect for any instrument involves validity. Validity exists when the scores on the instrument accurately reflect achievement along the various dimensions the institution is seeking to evaluate. Validity has several dimensions, including content and context. Content validity exists when the test or instrument covers and measures the specific program or course curriculum. Context validity examines the extent to which scores (outcomes) logically correlate with other, external, variables expected to be associated with student achievement. An additional facet to validity is criterion validity, the extent to which scores of the test correlate with other variables one would expect to be associated with test performance ( Black & Duhon, 2003).

The Educational Testing Service's (ETS) exam in business has become, to many, the de facto standard of standardized assessment instruments in collegiate business programs. The literature reveals an almost universal agreement as to the principle variables examined as predictors of student performance on the ETS exam. These variables include: grade point average (GPA), standardized test scores (ACT/SAT) and gender. In addition to these variables, Mirchandani, et al, (2001) include transfer GPA and student grades in quantitative courses.

It is possible to extrapolate and utilize the same variables when examining the results on a local instrument. Terry, et al, (2008) developed a model based upon a production view of student learning to examine the determinants of performance on the business major field achievement ETS exam. Their model controlled for grade point average (GPA), standardized test scores (SAT/ACT), junior college transfer students, gender. Their findings were consistent with much of the previous research in this area, that academic ability as measured by grade point average (GPA) and scores on standardized tests (SAT/ACT) are the primary determinants of student performance on the ETS exam.

Black and Duhon's (2003) study of ETS scores conducted during three (3) semesters in 1996-1997 included an examination of an incentive as a driver in student performance. In that study, students scoring at the national 50th percentile, or better, were given an extra-point bonus, which was used in the calculation of the student's final course grade. The exclusion of this, or some other, form of incentive is not used; some students may not take the test seriously and by extension, the results may be misleading (Allen & Bycio, 1997). Terry (2007) included the impact of course formats, traditional campus courses, online courses and the newer hybrid courses on ETS scores.

METHODOLOGY

A 20-item multiple-choice comprehensive examination was given to students in each of three sections of the Principles of Marketing course at a regional university in the Spring 2008 semester. This exam was one component of a two-part final exam administered at the end of the semester, and was incentivized to motivate student preparation. The results for the three sections appear in Table 1.

The instrument was a series of questions that cover the entirety of topics presented in a typical Principles of Marketing course. The entire Marketing faculty agreed upon the corpus of questions in Fall 2007. Each of the questions were mapped to specific learning objectives for the course.

Two professors taught this course during the Spring 2008 semester. Professor 1 is a tenure-track professor with a PhD, and taught sections 1 (on-campus) and 3 (online). Professor 2 is a PTI (part-time instructor) with an MBA and 18 graduate hours of study in Marketing, and taught section 2.

Numerous studies have sought to map the relationship between student input variables and outcome assessments. Generally speaking, these models delineate multiple factors (or internal student drivers) impacting assessment scores, including Native Ability (intelligence, often measured by ACT and/or SAT scores), Student Effort (often measured by the student's cumulative GPA), and Student Traits (a vector of categorical demographic information, including gender, transfer status, and nationality). A fourth set of factors include Environment Variables, which accounts for course format (online, campus, etc.) and the individual professor of record. Figure 1 illustrates the relationships of this model. Four models were calculated in this study based on these relationships, with the broadest model specified as follows. The inclusion of specific variables is based on the work cited above in predicting ETS exam scores, with the exception of the Professor variable. This variable was included in an effort to extend the research and to search for possible hidden influencers.

[FIGURE 1 OMITTED]

The dependent variable in this study was the student's score on the 20-item exam, with each correct response earning five points. Scores could thus range from 0 to 100. Independent variables were compiled from the university Registrar's database after the semester ended, and included a variety of internally-scaled and binary variables. Cumulative GPA is on a scale from 0 to 4.00, and represents only the student's GPA since the time of their matriculation at this university. The ACT score is the student's composite score on the national entrance exam, and ranges from 1 to 36. The remaining independent variables (student nationality, transfer status, gender, course format, and professor) were binary variables coded as either a 0 or 1.

Based on upon the prior research of Terry, et al (2008), Mirchandani, et al, (2001), and Black & Duhon (2003), we hypothesize the following relationships:

H1: ACT will have a significant positive influence on a student's outcome assessment score. Terry, et al, and others are in unison in finding that ACT (or SAT), as a measure of native ability, is a strong predictor of student assessment outcomes.

H2: GPA will have a significant positive influence on a student's outcome assessment score. Similarly, student effort, as measured by cumulative GPA, is a strong predictor. Terry, et al, and numerous others concur that this variable will yield a positive and significant influence on outcomes scores.

H3: Transfer status will not have a significant influence on a student's outcome assessment score. Transfer status has been included in prior research, but has not yet been found to be a significant predictor. We thus do not predict any significant influence in either direction.

H4: Nationality will not have a significant influence on a student's outcome assessment score. Terry, et al, also included student nationality as a dichotomous variable, and found no significant influence. We likewise have no reason to predict any significance from this variable.

H5: Class Format will not have a significant influence on a student's outcome assessment score. While much research has been done comparing online and--on-campus outcomes, no trend has emerged favoring one over the other. We thus propose no significant influence from this variable.

H6: Gender will have a significant influence on a student's outcome assessment score. Terry, et al, found that females scored significantly higher on their ETS exam, and we likewise propose a similar significant relationship.

H7: Professor will not have a significant influence on a student's outcome assessment score. This variable has not been included in the prior cited work, but given that many schools employ multiple professors to teach their foundational courses, it is possible that an "instructor effect" may be apparent. In the absence of prior research, we posit there will be no difference, assuming similar effectiveness across instructors.

RESULTS

The initial sample was comprised of 151 students who took the assessment exam; this group was labeled the "full" model. The "partial" model was comprised of 63 students for whom the university had an ACT score on record. Since the university does not require an ACT score for incoming transfer students, and because a large percentage of students are transfers, the "partial" model was based on a much smaller sample.

Tables 1 and 2 contain course section means for the full and partial (ACT) groups. The overall mean score for the full group was 72.58, while the overall mean among those with ACT scores was 72.85. Given that the majority of students reporting an ACT score were four-year students at the university (and the non-ACT students were transfers), it can be concluded there is no difference in mean score between these two student groups.

Four linear regression models were then calculated, as summarized in Table 3. A 2 X 2 matrix of the models includes both "full" and "partial" models, as well as those with and without the effect calculated for the professor.

The first model (Basic Model; top-left) is built on the full data set and contains the basic vector of variables (GPA, Transfer, Nationality, Gender, and Format). In this model both Format and GPA were significant predictors at the Prob.=0.05 level, suggesting that the online format and higher overall student GPAs were positively related to student scores on the assessment exam. Only 25% of the variance is accounted for by the model, though. The data for the three sections suggested, though, that the explanatory power of Format may not be sufficient to capture other nuances of the data. A categorical "dummy" variable was created to account for the professor of these three sections, the results indicating in the second model (Basic Model + Prof; lower-left) that GPA and Professor are now significant at the Prob.=0.05 level. This manipulation resulted in the model accounting for 39% of the variance, a substantial improvement over the first model.

Of the four models created, the Partial + Professor model resulted in the highest R2 (0.458). In this model only the Gender (H6) and Professor (H7) variables were significant at the Prob.=0.05 level. GPA (H2) was marginal at Prob.= 0.06, and thus conditionally retained. Surprisingly, ACT (H1) was not significantly related, in spite of the findings in prior research. In this instance, it is possible that the Professor effect was more profound and offset any native ability students may have possessed going into the course.

Of particular interest is the effect of Gender. While we affirm the findings of Terry, et al, Gender did not appear as a significant variable in the Full Model (column 1), but did so in the Partial Model (column 2). A Key Influencers analysis of the variables revealed that female students were tightly clustered in the 76-84 score range, and once the Professor variable was added to the equation, Gender emerged as a very strong predictor.

Further analyses were conducted by calculating t-tests in both the Full and Partial models using the dichotomous independent variables with the dependent variable (test score). These results are summarized in Table 4, indicating Format and Professor yielded significant differences in test scores. The models above, though, indicate somewhat different results when variables are not isolated.

CONCLUSIONS AND FUTURE RESEARCH

The statistical models presented above illustrate the differences in outcomes scores that can accrue in one topical area in multiple format/professor situations. In this particular case, the most significant predictor was the professor, not the format. The implication of these findings is that departments and colleges engaged in the assessment process need to be cognizant of influences from different sources that may not otherwise have been observed or isolated.

In this particular study, differences in outcomes assessment were not so much a factor of course delivery format as they were who was teaching the course. The difference in scores may be attributed to the differences between a full-time doctorally-qualified professor and a part-time adjunct with a master's degree. This is not to say that non-PhD instructors are less effective than those with a PhD; rather, it illustrates the need for departments and colleges of business to carefully dissect the data when analyzing assessment scores.

This research is intentionally narrow in scope, and is thus not necessarily transferable across disciplines or schools as it stands. With only a small matrix of course sections, formats, and professors included in the study, there is room for further exploration by applying this model across disciplines, time, and more faculty. Of greater potential importance is the possibility of tracking these students through the remainder of their academic program, including their outgoing assessment exam in the capstone Strategy course.

REFERENCES

Adams, C., Thomas, R & King, K. (2000). Business Student's Ranking of Reasons for Assessment: Gender Differences, Innovation in Education and Training International.

Allen, J. S., & Bycio, P. (1997). An evaluation of the Educational Testing Service Major Field Achievement Test in Business. Journal of Accounting Education, 15, 503-514.

Aloi, S. L., Gardner, W. S. & Lusher, A. L. (2003). A framework for assess general education outcomes within majors. JGE: The Journal of General Education, 52(4), 237-252

Angelo, T.A., & Cross, K. P. (1993) Classroom assessment techniques: A handbook for college teachers (2nd ed). San Francisco: Jossey-Bass

Bagamery, B., Lasik, J., & Nixon, D. (2005). Determinants of success on the ETS Business Major Field Exam for students in an undergraduate multisite regional university business program. Journal of Education for Business, 81, 55-63.

Black, H. & Duhon, D. (2003). Evaluating and Improving Student Achievement in Business Programs: The Effective use of Standardized Assessment Tests. Journal of Education for Business, 79, 157-162.

Ebel, R. I. & Frisbie, D. A. (1991) Essentials of educational measures (5th ed.), Englewood Cliffs, NJ: Prentice-Hall.

Lubinescu, E.S., Ratcliff, J.L., & Gaffney, M.A., (2001). Two continuums collide: Accreditation and Assessment, New Directions for Higher Education, 113, 5-21.

Martell, K. (2007). Assessing student learning: Are business schools making the grade? The Journal of Education for Business, 82(4), 189-195.

Mirchandani, D., Lynch, R, & Hamilton, D. (2001). Using the ETS Major Field Test in business: Implications for assessment. The Journal of Education for Business, 77, 51-56.

Muchlitsch, J. F., & Sidle, M. W. (2002) Assessing Student Learning Outcomes: A Comparative Study of Techniques Used in Business School Disciplines. Journal of Education for Business, January/ February 2002, 125--130

Palomba, C.A. & Banta, T.W. (1999). Assessment essentials: Planning, implementing, and improving assessment in higher education. San Francisco: Jossey-Bass.

Pringle, C. & Michel, M. (2007). Assessment practices in AACSB--accredited business schools. The Journal of Education for Business, 82(4), 202-211.

Terry, N. (2000). The effectiveness of virtual learning in economics. The Journal of Economics & Economic Education Research, 1(1), 92-98

Terry, N., Mills, L., & Sollosy (2008). Performance of domestic versus international students on the ETS exam, Southwest Review of International Business Research, 19(1), 191-195

Trapnell, J. (2005). Forward. In K. Martell & T. Calderon Eds. Assessment of student learning in business schools. Tallahassee, FL: Association for Institutional Research.

R. Nicholas Gerlich, West Texas A&M University

Marc Sollosy, West Texas A&M University

Table 1: Exam Scores by Section, Format,
and Professor: Full Sample (161)

Section Avg. Score Class Size Format Professor

1 79.06 53 Classroom 1
2 60.27 55 Classroom 2
3 81.32 53 Online

Table 2: Exam Scores by Section, Format,
and Professor: ACT-Only Sample (68)

Section Avg. Score Class Size Format Professor

1 80.96 31 Classroom 1
2 60.83 24 Classroom 2
3 80.38 13 Online

Table 3:: Model Comparison

Full Model (all cases) Partial Model (ACT score available)
Basic Model Basic + ACT
n = 161 n = 68
[R.sup.2] = .206 [R.sup.2] = .274
Adj [R.sup.2] = .181 Adj [R.sup.2] = .203

 IV t prob IV t prob

Format 4.724 0.000 Format 2.087 0.041
GPA 3.124 0.002 GPA 3.249 0.002
Nat 1.303 0.194 Nat 1.046 0.300
Transfer -0.869 0.386 ACT 0.953 0.344
Gender 0.895 0.372 Transfer -0.643 0.532
 Gender 2.334 0.023

Basic Model + Prof Effect Partial Model + ACT + Prof Effect
n = 161 n = 68
[R.sup.2] = .397 [R.sup.2] = .458
Adj [R.sup.2] = .374 Adj. [R.sup.2] = .395

 IV t prob IV t prob

Format 1.116 0.266 Format 0.617 0.540
GPA 2.402 0.017 GPA 1.910 0.061
Nat 0.715 0.476 Nat 0.826 0.412
Transfer -0.435 0.664 ACT 0.901 0.371
Gender 1.276 0.204 Transfer -0.279 0.781
Prof 6.988 0.000 Gender 2.624 0.011
 Prof 4.520 0.000

Table 4: T-Tests

Full Model Partial Model

 IV t prob IV t prob

Format -4.747 0.000 Format -1.674 0.099
Gender -0.555 0.580 Gender -0.587 0.559
Transfer 0.411 0.682 Transfer 0.500 0.619
Nat -1.624 0.106 Nat -1.243 0.218
Prof -9.423 0.000 Prof -6.044 0.000