Online education in the broader context: are live applied mathematics classes superior to online?

Adams, Lynn L. ; Glenn, Lowell M. ; Adams, Nathanael L. 等

Abstract

This paper presents research showing that students taking an online applied mathematics course in operations management are scoring as well (two percent higher) on the same tests being given to the live class sections. Statistical analysis of the data collected so far indicates that on three of four tests given to all of the students (online and live) that there is no statistical difference in the scores. This study details the course management system, especially the WebCT testing setup that has been used in all of the class sections for this study, in addition to the data from 155 students that have taken the course from the same professor and coauthor of this paper.

Introduction

Comparisons between online and live classes are often difficult, because of different testing situations and other significant differences in the way live and online classes are delivered. Many articles, including a provocative paper presented in last year's ABEAI 2004 Conference "The Impact of Course Content in Selecting Business Online Classes" (Letterman, 2005), have suggested that "quantitative business courses (should) be offered in the traditional class format or partially online" as contrasted to a wholly online environment. Concerns for the online teaching of quantitative courses include, among other things, that teaching quantitative material online is difficult, problematic, and substandard. The authors of this paper would not have argued any differently a year ago. The focus of this paper is to present an ongoing case study of test scores of online verses live students, who are taking the same operations management class from the same professor, using the same text, syllabus, and other course materials, but, most significantly: the same course management system (taking the same tests with the same testing procedure online using WebCT).

The class being studied is MGMT 3450, an Operations Management class offered to juniors and seniors that is largely an applied quantitative mathematics course, which also includes some statistics. This paper will also discuss in detail how this particular OM class is being delivered, both online and live, and how that course management system is affecting the outcome of the students. This ongoing study has evaluated 155 students from the fall of 2004 to the end of the summer of 2005 in seven class sections (three online classes and six live sections). So far, the online sections have out performed the live sections by 2.49 percent on the four tests given each semester.

Background

The Business School at UVSC (Utah Valley State College) decided to risk offering a third year operations management class online in the fall of 2004, emphasizing quantitative OM skills in applied mathematics and statistics. The School of Business at UVSC has been concerned about the lack of quantitative skills of high school students coming into the business school, and the low scores of graduating business students on quantitative subjects on the senior exam given to graduating business students at UVSC. Statistics and applied mathematics have been two of the main problem areas identified by the School of Business in its graduating seniors, so the school has been working on bolstering quantitative subjects and integrating those subjects throughout the curriculum.

Part of that effort to improve student quantitative skills was to increase the rigor of the MGMT 3450 class in operations management. A new text was sought out--a text with quantitative rigor (Operations Management, 8th Ed., William J. Stevenson, 2005). A former OM instructor was reassigned in this process, while another instructor was given the majority of the OM sections. That same instructor was given the task of developing a pure online OM class to be offered in the fall of 2004. Class development occurred over the summer of 2004.

As that instructor, who is also one of the coauthors of this article, developed the online OM class, he decided to develop and use WebCT testing in all of his OM classes, both online and live. The decision to use WebCT testing was as much a decision to simplify instructor work load as it was a decision to investigate the effectiveness and value of WebCT testing. At the end of the first semester in the fall of 2004, the average test scores were higher for the online section than for the three live sections (80.83 percent average for the one online section versus 75.03, 75.56, and 79.07 percent for the three live class sections), which stunned the instructor. That first online class section consisted of only nine students, because the class, being a new class offering for UVSC online, was limited at first to 15 students, and six of the original students eventually dropped the class.

After the instructor saw that the online students scored higher that first semester, a decision was made to study the testing results for that operations management class for all of the students taught by that same instructor (Dr. Lynn L. Adams). When adjustments were made to any of the tests, syllabi, etc. the same adjustments were made to all sections at the same time. UVSC requires that all classes be graded very soon after the end of finals each semester (five days, usually), including online sections, so the online students did not have extra time on assignments or tests at the end of the semester. The conditions under which all of the OM classes were taught from the fall of 2004 to the end of the summer of 2005 created a nearly perfect environment to study the differences in online and live teaching for this specific class, delivered in the specific way that will be outlined below.

Delivery System and Testing

Every attempt has been made to keep the online delivery consistent with the live delivery where possible in teaching this specific operations management class. In addition to having the same instructor in every case, the same tests, given in the same format and delivery system, were given to all of the sections, and the online sections were kept under the same time constraints as to course length, assignments, and testing time as the live classes.

If adjustments were made to any of the tests, syllabi, etc. the same adjustments were made to all sections at the same time. For example, after the first semester the instructor found that several of the questions on the tests were undesirable, either due to wording or other problems. Those specific test questions were eliminated for the subsequent semesters in all of the sections, live or online. Each class was given four tests and all of the classes covered the same chapters. In addition, all the classes were assigned the same homework problems, and the tests covered the same material.

The first test in every class has been a 14 question applied math test that has included two linear regression questions, several forecasting questions, a quarter relatives problem, an upper and lower control limits problem, a series of questions on breakeven analysis with buy or build, and a few statistics questions relating to analyzing a small production output set. That first test has continued to be delivered in the short answer test format, rather than a multiple choice test, so that the instructor can see trends in the students' abilities, and deficiencies in the students' math skills. The instructor initially thought that in time the first test would be offered in the same manner as the other three tests, which is multiple choice, but the advantages of seeing more of what the students know coming into the class has continued to be valuable, and cheating has not been an issue.

The only aspect of the first test that has changed each semester is the input numbers and the data, which helps discourage cheating. Each test is given using WebCT, online, so each student can take each test at home or in the school computer labs. Each test is timed and the student has only one chance to take the test. Thus, the student has to take the test in one sitting. The rigor of all four of the tests is such that the students generally need most of the time given (120 minutes). On all four of the tests students are allowed to use their books, notes, calculators and spreadsheets, but not to collaborate. The time restriction makes it difficult for students to get outside help on questions. On the first test partial credit is given for incorrect answers if the student shows their work and they were correctly solving the problem, yet did not get the right answer.

The last three tests are multiple choice tests where the questions are given at random from a master question bank. Each student is given 20 questions to answer, and no partial credit is offered. The instructor built the test bank for each of the last three tests using the more difficult questions provided by the publisher of the text. Problematic or unclear questions have been systematically removed over time for all of the OM sections. True/false and yes/no questions were not used at all. The instructor feels that true/false test questions tend to raise average test scores and to not adequately assess student knowledge. But, more importantly, true/false type testing online increases the probability of cheating.

Since cheating is always a critical issue in testing, and particularly whet attempting online testing, test settings for online testing are very important. In addition to limiting student time to take a test online in WebCT, and only allowing one attempt to take each test, the bulk of the multiple choice questions are given with the answers scrambled. Even if a student happens to get the same question as another student that they have discussed the test with, the answers will most likely be in a different order. The instructor has only kept a few questions with answers like "all the above" or "a and b above", etc. because scrambling of those types of questions is problematic ("a and b above" type questions will not make sense to students when WebCT scrambles the answers). The instructor does not scramble the few questions that have been kept with "all the above" type answers.

Finally, the instructor has set WebCT to not give out the test scores or the correct answers to the students until the deadline to take the test has passed, further frustrating collaboration attempts by students. The testing period has usually been from Tuesday morning through Sunday night (they must be finished by 11:55 p.m. on Sunday or the system cuts them off with an incomplete test). As soon as the test closes to the students, at least on the last three tests that are multiple choice, the tests are graded automatically.

The instructor has offered live help to online students in his office so that the online students would have help doing the assigned problems, and has also used the Blackboard feature in WebCT with a WACOM Graphire tablet to show students how to do some of the math problems. However, through the first year of this study, the online students have only used live tutoring by the instructor a few times at the instructor's office (less than 10), and the blackboard feature has largely gone unused by the students.

Live students have had the advantage of getting a review session from the instructor and of having a number of the homework problems worked out in class. The instructor posts what the online students should expect on each test and how to study, but, still, only a few of the online students have ever accepted the offer to be given live help from the instructor.

Course Materials

All of the course syllabi, lessons, book lists, and other materials are posted on WebCT for all of the sections, not just the online classes. The entire course has 14 lessons, with each lesson containing a week's worth of material and assignments. Each lesson includes an introduction to the material and a description of each chapter of the text, one chapter of the text for each lesson, and a summary of each lesson. Unless the online students become heavily involved in online discussions they only have the provided introduction and summary for each chapter to replace the live lecture. Each lesson also includes the assignments for that week, including homework problems, book reports (two are required during the semester), production facility tours (two per semester, with a brief write up of each), and if a test is due to be taken that week. Finally, one large group project is required before the end of the semester. That project involves the students in studying an actual production system or firm and recommending specific improvements to the operations of that organization.

Two significant differences exist between the live delivery of this OM class versus the online delivery of the class. First, the online students are given two to four "discussion questions" each week that are designed to get the students analyzing and discussing the lesson material between themselves and the instructor. The instructor is more of a facilitator than a teacher/lecturer in that process if the students become immersed in the process, rather than just do the minimum to answer the discussion questions. Attendance for the online students is based on the number and quality of their responses to the weekly online questions and their other online discussions and questions. The live students do not have the discussion questions online, although the value of such discussions may lead the instructor to change that in the future. Second, the live students get a live lecture on the material that the online students do not experience. The live lectures include some other lesson material that is not currently offered online, such as short movies on production topics (provided by the publisher of the text), and a production problem using Legos (the Lego Challenge is a simple production simulation that has been used in a number of classes across the country, but will not be discussed here).

Data

The seven class sections that this study analyzed had a total of 155 students. Four sections were live classes, and three were online. The average class size for the live sections was 22.83, while the online average class size was 19.38. The average test scores are given in Table 1.

When the average of the means was taken the online classes surpassed the live sections. Although the means indicated a slight superiority for the online classes when trying to determine if the numbers were statistically significantly a clear cut advantage was not defendable statistically.

A t-test, at a 90% confidence interval, was used to determine if there was a significant difference between the online and live classes (Table 2). The results mimicked the mean of the tests but brought out some critical faults within the study. Tests two and four were clearly the same statistically with large enough t-scores to forgo any further analysis. Test one had a t-stat, of 1.39 and a critical point of 1.29. Statistically, the means are the same, but any number of variables could sway the results. It is hypothesized that the reason for the relative closeness of the t-stat, for the first test is that students in the online courses are not as ready for the first test as live students. Students assume an online course means an easier course and do not adequately prepare for the test.

The third test is the most difficult test given. As such the t-test shows there is a significant difference between the two means. The null hypothesis is rejected because the t-stat is 0.23 for a critical value of 1.29. Such a strong rejection shows a clear distinction between the online and the live classes. Both classes have much larger variations on this test than the three other tests. The reason for the variation is the heavy emphasis on statistics. Thus the average scores are lower and the standard deviation larger. Still, the online students on test three had higher average test scores than the live students (see Table 1).

Test numbers one, two, and four provide reasonably conclusive results that with time should improve given a larger population size. Test three will continue to be statistically different due to the high degree of variation no matter what the size of population is (provided the same test is given). Finally, another fault will continue with the study: the number of students in the online classes will always be smaller than the live classes as long as only one section of online continues to be offered per semester. With the central limit theorem in mind, as the data set for live classes increases and approach the true mean, the online classes will continually lag behind. However, given that three out of the four tests were statistically the same given the small population size for online is sufficient to question the long held belief that quantitative classes cannot achieve the same level of success or rigor of live sections.

As the online students continued to score as well or better than the live students this past summer, the researchers decided to find out if the online students were somehow superior to the live students. Institutional Research at Utah Valley State College was asked to provide a spread sheet of all of the students who had taken MGMT 3450 from Dr. Adams in the fall of 2004 and the spring of 2005 (data for summer of 2005 is still not available as of this writing), listing their majors, year in school, and GPA. The students that took the class live in the live morning sections had overall average GPAs of 3.145, and the live afternoon students had overall average GPAs of 3.065. The overall average GPA for the live students was 3.1227 and the overall average GPA for the online students was 3.1575.

Of the 116 students' data provided by Institutional Research at UVSC who took MGMT 3450 from Dr. Adams in the fall of 2004 and the spring of 2005, 79.3 percent were seniors, 15.5 percent were juniors, 3.4 percent sophomores, and 1.7 percent freshmen. Only two of the students had majors outside of the school of business, and the business majors were a mix of various business disciplines.

Summary and Conclusions

The research presented in this paper shows that an applied mathematics/ statistics class in operations management, using WebCT testing and a fairly rigid class structure similar to live classes, with a heavy emphasis on quantitative skills is not only possible online, but can be as effective as live courses. The writers are of the opinion that as long as the quantitative material is not overly theoretical, and sticks to applied methods and models, that students can comprehend and master the material online. In most applied courses similar to the course studied for this research, the level of the mathematics and/or statistics is a step down from the foundational theoretical mathematics required by most colleges from their respective Math departments or even from within their own business schools. Teaching quantitative material in foundational mathematics and statistics courses that is at a higher theoretical level than the applied mathematics courses required later on in course work should help junior and senior level quantitative courses in general, not just the online classes.

The authors of this paper believe, after seeing the results of this ongoing study, that a fundamental business calculus class may be successful as a hybrid online/ live class. The Economics and Finance Department of the School of Business at UVSC is contemplating the creation of a pure online class for business calculus, which may be attempted in the fall of 2006. The results of this paper along with the experience of faculty in the Economics and Finance Department at UVSC will weigh heavy in that decision.

Eliminating cheating in online testing will be an ongoing challenge given that students can use their books, calculators, and spreadsheets. This operations research class has allowed and encouraged students to use the tools they will have in the job market. In other quantitative classes that may not be the case and so adjustments will have to be made. So far, the instructor has only detected one instance of cheating, in one of the live classes. The instructor was amused when the test scores for those three students, before disciplinary action, were a D+ on that test (test 1). Setting a specific time to take a test, giving less time for each test, increasing the size of test banks for questions, and giving short answer tests rather than multiple choice questions are some of the alternatives being considered for this class if cheating ever becomes a serious problem. The set up of questions for tests on WebCT as outlined in this article should be considered as a minimum to discourage cheating.

The instructor who instigated this research is still amazed at the results. Eight years of quantitative business, fundamental mathematics, economics, and statistics teaching did not give this instructor any confidence that an online quantitative class could succeed at the same level as a live class. This study has completely changed the coauthors' belief in online delivery of quantitative subjects. Structuring both live and online classes to be as similar as possible to each other has definitely improved both types of delivery for the coauthors. The extra material set up on WebCT for live students has drastically reduced student questions on what is required, especially if the student is out of town or ill. Having gone through the process of creating an online class, the instructor of this study OM class has learned how to build additional detail into lessons and the syllabus so that both online and live students feel sufficiently sure of what is required of them and do not feel lost. That extra structure and shepherding has carried over into the live sections with great benefit, both in terms of fewer questions being asked by students and in higher teacher evaluations. A lively online discussion of key material continues to be superior to live lectures, and the writers suggest that college instructors begin to incorporate online discussions into their live curriculum.

REFERENCES

Letterman, Denise & Morris, Robert (2005), The Impact of Course Content in Selecting Buisness Online Classes. Applied Business and Entrepreneurship Association International Conference, Maui.

Stevenson, William J. (2005), Operations Management, 8th Ed. McGraw-Hill Irwin, Boston.

Lynn L. Adams, Associate Professor of Business, Phone: 801-863-6483, adamsly@uvsc.edu

Lowell M. Glenn, Department Chair of Finance and Economics and Assistant Professor of Business, Phone: 801-863-8385, glennlo@uvsc.edu

Nathanael L. Adams, MPP, Adjunct Professor of Business, adams.nathanael@gmail.com Utah Valley State College School of Business 800 West University Parkway, Orem, UT 84058.

Table 1
 Live Online

Mean Test #1 81.36792 77.5102
Mean Test #2 78.22642 85.6383
Mean Test #3 74.53922 75.20833
Mean Test #4 73.79808 79.54545
Average [mu] 76.98291 79.47557

Table 2

 Test 1 live Test 1 online

Mean 81.36792 77.5102
Variance 196.2348 284.5468
Observations 106 49
Hypothesized Mean Difference 0
df 80
t Stat 1.394001
P(T<=t) one-tail 0.083589
t Critical one-tail 1.292224
P(T<=t) two-tail 0.167177
t Critical two-tail 1.664125

 Test 2 live Test 2 online

Mean 78.22642 85.6383
Variance 118.1578 81.1055
Observations 106 47
Hypothesized Mean Difference 0
df 105
t Stat -4.39788
P(T<=t) one-tail 1.31E-05
t Critical one-tail 1.289666
P(T<=t) two-tail 2.63E-05
t Critical two-tail 1.659495

 Test 3 live Test 3 online

Mean 74.53922 75.20833
Variance 158.1519 316.977
Observations 102 48
Hypothesized Mean Difference 0
df 70
t Stat -0.23432
P(T<=t) one-tail 0.40771
t Critical one-tail 1.293763
P(T<=t) two-tail 0.815419
t Critical two-tail 1.666914

 Test 4 live Test 4 online

Mean 73.79808 79.54545
Variance 150.7258 114.9049
Observations 104 44
Hypothesized Mean Difference 0
df 92
t Stat -2.85211
P(T<=t) one-tail 0.002681
t Critical one-tail 1.290821
P(T<=t) two-tail 0.005363
t Critical two-tail 1.661585