The relationship between the assessment of student learning outcomes and the marketing process of institutions: a case study.

Rey, Melanie Powell ; Powell, Kimberly K.

INTRODUCTION

Assessment of student learning outcomes and engagement is not new to higher education institutions. Accrediting bodies are requiring greater emphasis on assessment of student learning outcomes, given the critical role of student learning outcomes in higher education. Subsequently, the overall premise of assessment stems from the notion that what is important and/or critical should be measured and what is measured should be improved (Levin & Reed, 2011). When delving deep into the impact of assessment on academic units, business schools that are accredited by the Association to Advance Collegiate Schools of Business (AACSB), are one academic unit that is under constant pressure to employ assessment measures that address areas such as student learning outcomes and engagement and continuously improve based upon their assessment results. What is interesting is that business schools that are AACSB accredited devoted less than 10% of the accreditation criteria and sub-criteria to assessment prior to 2003 (Michel & Pringle, 2007).

The increased emphasis on the assessment process is a result of greater public accountability on behalf of state legislators, private donors and taxpayers who are holding colleges and universities accountable for justifying funds that are quickly diminishing in the area of higher education (Michel & Pringle, 2007). Financial pressures to streamline budgets and the need to focus on other priorities, such as healthcare, K-12 education, and crime prevention, further heighten this level of accountability. Such factors have caused state and federal governments to accept the argument that a college education should be viewed as less of a public investment to an educated citizenry and more as a consumer good of primary benefit to the student (Duderstadt & Womack, 2003).

With the onset of increased assessment, greater accountability, and reduced funding, the higher education market has become increasingly competitive for resources such as students and external funding (Duderstadt & Womack, 2003). Therefore, business schools within public universities have to change their model and approach to dealing with these challenges and parlay them into opportunities for growth and innovation. One approach is leveraging assessment results into an opportunity for marketing their respective programs. A model that can assist in this opportunity is the Identity Salience Model of Relationship Marketing Success by Arnett, German, and Hunt (2003).

Relationship marketing is very strategic for public universities given the fact that colleges and universities have an advantage over online education providers in that they can leverage the power of the social institution to forge true partnerships with students, alumni, and university supporters (Anctil, 2008b). Moreover, the integration of relationship-inducing factors such as reciprocity and satisfaction that are designed to influence identity salience, into the development of assessment tools and/or employment of these relationship components into the overall improvement of student learning outcomes and engagement, academic units can enhance retention, alumni giving, and positive word of mouth promotion of the university to others (Arnett, German, & Hunt, 2003).

To further detail the intersection of assessment and relationship marketing from an applied standpoint, this study will explore how one College of Business (COB) has leveraged its assessment results from its Graduate Exit Survey, along with employing the identity salience model of relationship marketing success theory in order to develop marketing-based strategies designed to retain and recruit students and garner greater resources. Based upon the theory and findings from the School's Graduate Exit Survey, the authors discuss relationship marketing strategies and recommendations that were developed in an effort to garner greater resources (e.g. students and funding) and how such efforts can serve as a guide for other academic units who undergo accreditation and must comply with assessment standards.

LITERATURE REVIEW

Assessment in Higher Education

Assessment in higher education has grown and is more commonplace, but challenges regarding its effectiveness still exist. According to assessment expert and director of the National Institute for Learning Outcomes Assessment, George D. Kuh, colleges and universities have plenty of assessment tools for measuring student learning outcomes, but they must learn to use them more effectively. Kuh goes on to say that although assessment has increased, there is still uncertainty regarding how this information is being transferred in such a way to change practices for the better (Hoover, 2009).

Another area of concern is faculty engagement on the assessment of student learning outcomes. According to a report released by the National Institute for Learning Outcomes Assessment, which describes how more than 1,500 colleges measure what students learn, two-thirds of university administrators (e.g., provosts) elucidate that faculty involvement would enhance the measurement of learning outcomes. Kuh contends that assessment should be a shared responsibility and therefore requires greater leadership and involvement at the departmental level (Hoover, 2009).

The utilization of assessment data for creativity and innovation is another area of concern. Kuh posits that higher education institutions have an add-on mentality. Although changes in the external environment (e.g. the changing economic landscape and its impact on higher education) would serve as a catalyst for innovative change and productivity, experts contend that universities have not utilized their assessment data for innovation during these difficult times. In fact, Kuh states that the National Institute for Learning Outcomes and Assessment does not have sufficient examples of how institutions use their respective data to innovate and move their respective programs forward (Hoover, 2009).

Higher Education Marketing

Public colleges and universities are social institutions, yet their competitive environment forces them to behave like modern businesses. The irony of this is that although students should be viewed as customers, they are not the traditional customer with specific needs and expectations. In fact, students tend to be unsure of what exactly they are looking for, but they also recognize the decision they will make is paramount to their lives from a professional and economic context. This is also compounded by the fact that college and university administrators tend to be inexperienced on the marketing side of higher education because they often come from academic fields not related to business (Anctil, 2008b).

Although higher education institutions make strides to market and advertise themselves, this area is still highly under-researched, thereby making it challenging to implement empirically driven changes and strategies (Anctil, 2008b). From a marketing perspective, Brown and Mazzarol (2009) explored the importance of institutional image to student satisfaction and loyalty in higher education in Australia. Rojas-Mendez, Vasquez-Parraga, Kara, and Cerda-Urrutia (2009) conducted a relationship-based study in higher education in Latin America. Helgesen (2008) conducted a study on relationship marketing in selling higher education. The primary focus was undergraduate, Norwegian students. Some studies, such as one conducted by Marzo-Navarro, Pedraja-Iglesias, and Rivera-Torres (2009) examines relationship marketing in higher education but in a different context (e.g. the relationship marketing approach between universities and business firms).

Despite the fact that researchers are making strides to examine higher education from a marketing context, research efforts are still limited (Anctil, 2008b). This is coupled with the fact that colleges and universities are not effectively leveraging their assessment process for greater innovation and to move their respective programs forward (Hoover, 2009). Research is also limited when examining how assessment can be utilized in selling higher education from a relationship-marketing context. Therefore, this study will address how business schools can leverage their assessment tools in order to sell their respective units in higher education.

INTERSECTION OF RELATIONSHIP BETWEEN MARKETING AND ASSESSMENT: THE CASE OF ONE COB

One AACSB International accredited College of Business (COB), which is located in the southern region of the United States, was deeply affected by environmental and economic factors including a natural disaster and substantial budget cuts. Given these circumstances, this particular business school decided to work toward developing assessment tools that were not only congruous with AACSB International mandates (e.g. student learning outcomes, engagement, strategic planning, etc.), but could be utilized to market their college of business. By integrating a relationship marketing approach aimed at retaining students, recruiting students, and increasing alumni giving, the COB employed certain relationship-inducing factors under the Identity Salience Model of Relationship Marketing Success by Arnett, German, and Hunt (2003). This model was designed to explain relationship marketing success in exchange relationships and is based primarily on social exchange in the context of nonprofit higher education marketing (Arnett, German, & Hunt, 2003). One of the assessment surveys that employ components of this model includes the COB's undergraduate Graduation Exit Survey (GES).

Survey Instrument Development

To comply with the Association to Advance Collegiate Schools of Business (AACSB International) standards, the COB is continuously participating in direct assessment measures to collect quantitative data based on the College's learning goals. The utilization of student surveys in assessment has been found to provide invaluable insights into how students have experienced higher education programs. In fact, benchmarking students' perceptions over time is a well-established practice at a number of higher education institutions that employ this approach. Student surveys that employ satisfaction based approach are intended to be used as an auspicious source of information about the students' experience from their own perspective while working to improve higher education programs (Williams, Cappuccini-Ansfield, & Kane, 2008).

The employment of satisfaction-based surveys among university students is heavily documented in research. Brunson (2010) examined service factors and satisfaction levels as contributors to student enrollment retention in a private university for it to remain competitive. A Likert-type 1-5 point survey developed by the researcher was administered to undergraduate, business majors within the institution. Findings revealed positive relationships between quality perceptions and satisfaction, between quality perceptions and commitment, and between satisfaction and commitment. Significant relationships between quality perception and satisfaction survey items occurred in faculty relationships with students. From the study, higher educational leaders were able to identify key service performance programs within their own institutions that contribute to retaining sufficient enrollment while enhancing competitiveness among programs (Brunson, 2010).

These direct assessment measures include course embedded assessment and the Knowledge Retention Assessment (KRA). In addition, the College is also participating in assessment measures to collect quantitative data in order to assess student perceptions and learning satisfaction of their academic programs in an effort to sell the COB from a relationship-based context. One assessment measure that assists in both efforts include the undergraduate Graduate Exit Survey (GES). Business majors who were in the final semester of completing their studies were required to complete this survey.

The COB has administered the GES since the Fall 2007 semester. However, ongoing skepticism existed in the level of reliability of the survey instrument among administrators in the COB. Therefore, survey reliability was assessed on the GES utilizing Cronbach's Alpha during the Summer 2010 semester. At that time, the alpha level was .013, which according to Nunally (1967) indicates that scores from this particular instrument failed to yield reliable results. After analyzing the results from Cronbach's Alpha, the following procedures were employed to increase the internal consistency of the Graduate Exit Survey during the Fall 2010 semester:

1. Demographic questions were expanded to include such categories as gender, age, number of years utilized to complete the bachelor's degree, marital status, etc.

2. Constructs were developed in order to group questions together systematically.

3. A pilot study was conducted in order to test the score reliability of the modified instrument.

4. A forced response system was implemented via Survey Monkey so that the prospective graduates would answer all questions.

After the survey was redesigned based on the aforementioned procedures during the Fall 2010 semester, a pilot study was conducted. The participants of the pilot study consisted of seventeen (17) COB students. It was observed that 100% of the respondents were prospective Fall 2010 graduates. In order to assess reliability, Cronbach's Alpha was utilized. It is important to note that Cronbach's Alpha was performed on two (i.e., Program and Coursework Satisfaction and Faculty and Student Relations) of the six newly developed constructs. The selected recommended reliability levels vary depending on the researcher. Nunally (1967) recommended a .5-.6 level for preliminary research, .8 for basic research and .9-.95 for applied research. Given that the result of the analysis for Program and Coursework Satisfaction was .863 and .843 for Faculty and Student Relationships, it was determined that the internal consistency scores yielded from the modified GES were considered reliable.

COB's Graduation Exit Survey (GES) Overview

The data from the GES for academic years (AY) 2010-11 and 2011-12 was analyzed in order to facilitate data based decision-making as it relates to program enhancement in the COB. The respondents from both AYs included a total of 273 undergraduate business majors (seventy-two Accounting majors-26.4%, eight Economics majors-2.9%, twenty-four Finance majors 8.8%, one hundred thirteen Management majors-41.4%, fifty-five Marketing majors-20.1%, and one E-Business major-.4%). Regarding gender, 134 male students (49.1%) and 139 female students (50.9%) completed the survey. The GES addresses the following domains: Demographic Information, Post-Graduation Plans, Continuing Education Plans, Program and Coursework Satisfaction, and Faculty and Student Relationships.

The Intersection of Relationship Marketing and the GES

The GES intersects with the identity salience model of relationship marketing success based on the following 3 domains within the GES: 1.) Continuing Education Plans; 2.) Program and Coursework Satisfaction; and 3.) Faculty and Student Relationships. The GES also aligns with the model due to the premise that the stronger a person's salience for a particular university identity, the more likely this person will adopt certain supportive behaviors for the university (e.g. donating money and providing resources to the university, providing positive word of mouth/recruiting for the university (Arnett, German, & Hunt, 2003). This survey was designed not only to adhere to assessment requirements by AACSB but to assess how the COB can increase students' salience for the COB's identity so the COB's recent alumni can adopt the aforementioned supportive behaviors. The relationship-inducing factors likely to induce a relationship between potential donors (e.g. alumni) and the public university are as follows:

Reciprocity

This core relationship-marketing factor implies that the non-profit higher education organization not only takes but also gives something in return (Arnett, German, & Hunt, 2003), hence the social contract (Duderstadt & Womack, 2003). Positive feelings are induced by potential donors (alumni) (Arnett, German, & Hunt, 2003) when the university has fulfilled its end of the social contract (e.g. teaching, research, and service) (Duderstadt & Womack, 2003). Consequently, these feelings positively impact the student's self-evaluation, which in turn provides a reaffirmation of the identity related to the university (Arnett, German, & Hunt, 2003). Under the Program and Coursework Satisfaction domain, 49% of respondents were satisfied with the relationship they had with faculty (Q7) and 48% were satisfied with the quality of academic advising (Q10).

Another domain, Faculty and Student Relationships, goes into detail in order to ascertain whether respondents' expectations were met in this context. For example, 46% of respondents agreed to the question that faculty members are willing to work with them (Q4). The majority of respondents agreed (52%) with the fact that students in their respective program are treated with respect (Q5) and that the rapport between faculty and students in their program is good (54%) (Q6). The majority of respondents (53%) also agreed with the statement that the overall climate of their respective program is positive (Q7). Respondents (56%) were also in agreement with the fact that their respective program content supports their research/professional goals (Q9) and that program activities foster a sense of intellectual community (58%) (Q10). Respondents were also in agreement with the program structure. For example, 55% of respondents indicated that program structure provides opportunities to engage in interdisciplinary work (Q11) and 50% of respondents affirm that program structure encourages student collaboration or teamwork (Q12).

Satisfaction

Numerous organizations focus on satisfaction in order to retain existing customers while attracting new customers. The satisfaction the person feels reaffirms their respective identity, which in turn increases identity salience. Therefore, alumni who are satisfied with their university experiences, particularly within their academic department, are more likely to place the university higher in their hierarchy of identities (Arnett, German, & Hunt, 2003). One domain in the GES that coincides with this relationship-inducing factor is the Continuing Education Plans. One relevant question (Q2) under this domain includes "Would you attend the University's MBA program?" The majority (71%) of the respondents indicated that they would attend the COB's MBA program.

The Program and Coursework Satisfaction domain directly links with this relationship-inducing factor. For example, respondents were asked to indicate their level of satisfaction regarding the intellectual quality of faculty (Q1). The majority of respondents (56%) were satisfied with the intellectual quality of faculty.

Reciprocity and Satisfaction

There were some domains and corresponding questions that were pertinent to both reciprocity and satisfaction. Under the Program and Coursework Satisfaction domain, respondents (62%) indicated that they were satisfied with the overall quality of the program (Q3). Respondents (47%) were satisfied with the helpfulness of staff members in their respective departments (Q4). The majority of respondents (61%) were satisfied with the academic standards within their respective departments (Q5) and their program's ability to integrate recent developments into their respective field (56%) (Q6). Regarding overall undergraduate level teaching quality, 64% of respondents were satisfied (Q8). Most of respondents were also satisfied with program space and facilities (49%) (Q9). The majority of respondents (94%) also indicated that they would select the COB again if they were to start their undergraduate career over again (Q12) and 78% of respondents would select the same major (Q13). Ninety-six percent of respondents indicated that they would recommend the COB at this particular university to other students (Q14).

STRATEGIES AND RECOMMENDATIONS

Although the majority of the relationship-based responses from the GES were positive, continuous improvement initiatives were developed in an effort to garner stronger relationships with new alumni. The strategies and recommendations listed below (e.g., the flipping model and academic advisement) emphasize greater faculty involvement. The service quality component employs involvement from both faculty and staff. The role of faculty is critical in retaining students (Tinto, 2007), strengthening assessment measures, implementing innovative strategies (Hoover, 2009), and helping to forge long-lasting relationships with alumni.

The Flipping Model

Based on the findings from the GES and the reciprocity and satisfaction tenets of relationship marketing, student learning had to be reevaluated to enhance learning outcomes. This is nothing new given the fact that professors can no longer pump out information and believe that their students will just understand it. Therefore, the flipping method was introduced into certain courses within the COB in order to invert the traditional classroom lecture. Flipping is the inversion of expectations in the traditional classroom lecture. Certain forms of flipping include just-in-time teaching and interactive engagement. Although flipping the classroom has been done for decades, there is a reinvigoration of this approach due to increased assessment of student learning outcomes. Courses that employ the flipping model have been shown to acquire greater gains than traditional lectures in comparable courses (Berrett, 2012).

Flipping takes numerous forms (Berrett, 2012) but the form that has been employed in the COB is interactive engagement, which includes application-based group discussions and classroom presentations based on the employment of business articles and pertinent podcasts. This technique is currently being utilized in the junior-level principles of marketing and principles of management courses. Students are no longer passive learners and therefore become more invested in their education, while developing greater identity salience.

Greater Focus on Academic Advising From a Developmental Context

The results from the GES also address academic advising. Most advisors are prescriptive advisors. To build relationships with students, a developmental advisor is preferred. In the article entitled "A Developmental View of Advising as Teaching," Crookston (1972) argued that the facilitation of learning is the duty of both teachers and advisors and that both of these roles involve working with students to improve their problem-solving and decision-making skills. This aligns with research by Eble (1988), who posits that academic advising should be considered by the faculty as an extension of their teaching role in which they can demonstrate genuine concern for their students' welfare by being available and approachable outside of the classroom and Kramer and Gardner (1983) who argued that advising is a specific teaching activity during which a student's educational choices are questioned and challenged.

Service Quality

Given the fact that higher education institutions offer a service (Anctil, 2008a; Helgesen, 2008) and questions within the GES focus on service quality, the services of the COB were evaluated from a quality-based and relationship-based standard, hence reciprocity and satisfaction. When discussing service quality from a relationship-based context, the SERVQUAL instrument by Parasuraman, Zeithaml, and Berry (1988) should also be considered. Parasuraman, Zeithaml, and Berry (1988) describe the constructs for the SERVQUAL instrument for assessing customer perceptions in service oriented and retail-based organizations. SERVQUAL is often used by businesses for building and maintaining long-term customer relationships. Based on their instrument, relevant domains for service quality include reliability, responsiveness, and empathy. From a relationship marketing context, a better quality service component (e.g. prompt response, convenient operation hours) will lead to fulfilling promises, meeting expectations, and servicing needs (Anctil, 2008a). Therefore, faculty and staff within the COB employed the following service quality components: prompt response, virtual office hours/convenient office hours, and increased individualized attention.

CONCLUSION

Business schools must learn not to view assessment as just a fulfillment of the requirements set forth by their accrediting bodies. By leveraging assessment data in order to market their respective programs from a relationship-based context, business schools can innovate and gain a competitive advantage in the higher education market. Therefore, business schools must learn to parlay these results into strategic plans and initiatives designed to garner long-lasting relationships with alumni/donors, acquire greater resources, and effectively retain and recruit students.

REFERENCES

Anctil, E.J. (2008a). Market differentiation. ASHE Higher Education Report, 34, 2, 49-88. Anctil, E. J. (2008b). Recommendations for selling higher education. ASHE Higher Education Report, 34,2, 89-98.

Arnett, D.B., German, S.D., & Hunt, S.D. (2003). The identity salience model of relationship marketing success: the case of nonprofit marketing. Journal of Marketing, 67, 2, 89-105.

Berrett, D. (2012). How 'flipping' the classroom can improve the traditional lecture. The Chronicle of Higher Education. Retrieved from http://chronicle.com/article/HowFlipping-the-Classroom/130857/

Brown, R. M., & Mazzarol, T. W. (2009). The importance of institutional image to student satisfaction and loyalty within higher education. Higher Education: the International Journal of Higher Education and Educational Planning, 58 (1), 81-95.

Brunson, K.W. (2010). Examining relationships between quality perceptions, satisfaction, and student brand loyalty in a higher educational organization. Northcentral University, 153 pages; AAT 3405888.

Crookston, B.B. (1972). A developmental view of academic advising as teaching. Journal of College Student Personnel, 13.

Duderstadt, J.J., & Womack, F.W. (2003). The future of the public university in America: Beyond the crossroads. Baltimore: The Johns Hopkins University Press.

Eble, K. E. (1988). The craft of teaching: A guide to mastering the professor's art. San Francisco, Calif: Jossey-Bass.

Helgesen, D. (2008). Marketing for higher education: A relationship marketing approach. Journal of Marketing for Higher Education, 18,1, 50-78.

Hoover, E. (2009). An expert surveys the assessment landscape. The Chronicle of Higher Education. Retrieved from http://chronicle.com/article/An-Expert-Surveys-the/48945/

Kramer, H. C., & Gardner, R. E. (1983). Advising by faculty. Washington, D.C: National Education Association.

Levin, J., & Reed, T. (2011). Making assessment work. The Chronicle of Higher Education. Retrieved from http://chronicle.com/article/Making-Assessment-Work/129266/

Marzo-Navarro, M., Pedraja-Iglesias, M., & Rivera-Torres, P. (2009). The marketing approach in relationships between universities and firms. Journal of Relationship Marketing, 8, 2, 127-147.

Michel, M., & Pringle, C. (2007). Assessment practices in AACSB-Accredited business schools. Journal of Education for Business, 82, 4, 202.

Nunnally, J. C. (1967). Psychometric Theory, 1st ed., New York: McGraw-Hill.

Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1988). SERVQUAL: A multiple-item scale for measuring consumer perceptions of service quality. Journal of Retailing, 64,1, 12--40.

Rojas-Mendez, J.I., Vasquez-Parraga, A.Z., Kara, A., & Cerda-Urrutia, A. (2009). Determinants of student loyalty in higher education: A tested relationship approach in Latin America. Latin American Business Review, 10, 21-39.

Tinto, V. (2007). Research and practice of student retention: What's next? Journal of College Student Retention, 8, 1- 19.

Williams, J., Cappuccini-Ansfield, G., & Kane, D. (2008). Student satisfaction surveys: the value in taking an historical perspective. Quality in Higher Education, 14, 2, 135-155.

Melanie Powell Rey

Kimberly K. Powell

Southern University and A & M College

Melanie Powell Rey holds a Ph.D. in Special Education (Mild/Moderate Disabilities) from Southern University and A & M College in Baton Rouge, Louisiana. Dr. Rey is a staff member in the College of Business at Southern University and A & M College and an Adjunct Professor of Speech-Language Pathology and Special Education at Southern University and A & M College. Her research interests include diversity, retention, and universal design for learning in higher education, culturally and linguistically diverse learners, autism and intellectual disabilities in the criminal justice system, and autism in sub-Saharan Africa.

Kimberly K. Powell holds a Ph.D. in Urban Higher Education from Jackson State University in Jackson, Mississippi and additional certifications in both management and marketing from Tulane University in New Orleans, Louisiana and Texas A & M University-Commerce. Dr. Powell is an Assistant Professor of Marketing at Southern University A & M College in Baton Rouge, Louisiana. Her research interests include entrepreneurial marketing, marketing for higher education, retention and diversity in higher education, sustainability in both higher education and small and minority-owned businesses, and social media marketing.