A COMPARISON OF PLACEMENT IN FIRST-YEAR UNIVERSITY MATHEMATICS COURSES USING PAPER AND ONLINE ADMINISTRATION OF A PLACEMENT TEST

Today, many universities in the United States use mathematics placement tests in combination with high school grades and SAT scores to place students in freshman mathematics courses. In an attempt to make this process more convenient for students and universities, these tests are beginning to be given online. This paper describes the history of a university mathematics placement test, originally given in 1992, which was converted to an online format in 2005. The placement method is described and a logistic regression is used to evaluate the accuracy of the online placement procedure in comparison to the placement with the paper test.


INTRODUCTION
Mathematics departments at colleges and universities in the United States have been concerned with students' success in mathematics courses and correct placement for many years.
Although there is research that confirms the predictive ability of Math SAT scores with regard to success in individual mathematics courses (Gussett, 1974), many authors have questioned the accuracy of Math SAT as a predictor of success in college, particularly in mathematics classes (Bridgeman, 1982;Dalton, 1976;Fincher, 1974;Gougeon, 1984;Mauger and Kolmodin, 1975).
In fact, it is well documented that Math SAT combined with high school GPA and/or class rank is indeed a better indicator of achievement in college classes (Jacobs, 1995;Bacon &Norman, 1992;Fair Test, 2006;Troutman, 1978).Research also exists which indicates that a mathematics placement test, in combination with other factors, can provide an even better predictor of success in mathematics classes (Bridgeman, 1982;Edge and Friedbert, 1984;Howlett, 1978;Nobel and Sawyer, 1982;Odell and Schumacher, 1995;Smith and Schumacher, 2005).
Today, many universities in the U.S. use mathematics placement tests in combination with high school grades and SAT scores to place students in freshman mathematics courses.This process often includes placement in remedial courses.A 2005 study by the National Bureau of Economic Research of 27,000 students produced results that suggest that such remediation results in successful intervention (Bettinger, 2005).In an attempt to make the placement process more convenient for students and universities, these tests are increasingly being given online (Ager, 1993;Shermis, 1997, Sutherland, 2006).This paper describes the history of a university mathematics placement test, which was converted to an online format in 2005.A statistical analysis is also provided to compare the accuracy of the two placement methods.

HISTORY
In an attempt to reduce the number of failing students in freshman mathematics classes at a private university in northeast of the United States, in 1992, the Mathematics Department felt it needed to identify and then place mathematically deficient students in special sections.Toward this end, we began to administer a mathematics placement test in that year.Originally, a test developed by the Mathematics Association of America (MAA) was used to place students in freshman mathematics classes.At the time, students were placed into one of four classes: Math 100, a non-credit, pre-college course; Math 105, a regular pre-calculus class with business applications; Math 105HN, an honors section of this course; and Math A105, an advanced section with fewer pre-calculus topics.The use of the test, along with a student's Math SAT, was used for several years and proved to have a better correlation with final grades than Math SAT alone (Odell and Schumacher, 1995).However, the fact that the weakest students were placed in a noncredit course, Math 100, which they needed to complete in addition to three required mathematics courses, created many problems for the department since students did not want to take an additional three credits.
In 2000, a three-credit remedial course, Math E105, was developed to replace the noncredit Math 100.This course was designed to cover the same material as the regular pre-calculus class, Math 105, but met five hours a week instead of three.The extra two hours were lab classes devoted to reviewing prerequisite material.This placement system met with much less resistance from the students since they received the same credit as the regular classes.
The placement test was administered at freshman orientation in late June.Although students' displeasure with being placed in a remedial course was lessened, there were still issues with the accuracy of the placement.Also, administrative tasks, such as getting all of the students to take the test and then changing students' schedules late in the summer or early in the fall, still created problems for the university registrar.It was particularly difficult for international students to be placed properly without a placement test and few were able to attend summer orientation.
In the fall of 2004, the Dean of the College of Arts and Sciences approached the department and asked us to create an online mathematics placement test which students could take as soon as they were accepted.The goal of this change was for students to be placed in their mathematics class prior to summer orientation.As a result, when they came on campus in June, they could be given their schedule of classes for the fall.
First, we needed to research how to best accomplish this change, which was to be in place for May of 2005.We reviewed several commercial tests.We rejected this option because we wanted more control of the content of the test.We eventually decided to use the same test that we had used the previous year to make it easier to compare the placement accuracy using the online test with the previous year's paper test.The placement test is composed of 35 multiple choice questions covering algebra and pre-calculus topics.It is modeled on the MAA test and is designed to be taken in one hour without a calculator.
The test was administered online for the first time in May of 2005 and again in May of 2006.We moved from paper to online by putting the test on the university's Blackboard platform.
This had the added advantage of familiarizing the students with the classroom presentation software used in a majority of the classes taught at the university.There were several problems with the online format, but we were able to address many of them for the second administration in 2006.

THE ONLINE PLACEMENT TEST ADMINISTRATION
The main issues with which we were concerned were accessibility and reliability.Since we wanted students to be able to take the test from their homes, we had to be sure that they had adequate computer connectivity and technical skills.In addition, we obviously wanted to obtain a true picture of their ability.The test was designed to be taken in one hour without a calculator, and we wanted to be sure that these instructions were followed even though the test was taken in an unsupervised environment.
We attempted to address the concern about accessibility due to technical issues by providing exact and accurate instructions on how to take the test and which computer equipment would be needed.In our instructions, we asked the students to complete the test within one week of receiving a letter from us informing them that the test was available.This letter also included instructions as to how to access the test as well as other test instructions.
Several problems with connectivity were encountered in the first year.Given only one week to complete the test, while still enrolled in high school and possibly while taking final exams, many students experienced technical difficulties getting onto the test site or viewing the test.Some of the problems may have been due to the high traffic on our computer system.
Unfortunately, the week that the students were taking the test coincided with the end of the semester and high computer use by the undergraduates at the university.Most of the technical difficulties turned out to be due to inadequate computer equipment.Many of the students who reported difficulties were trying to access the web using a dial-up modem connection.On our end, we were overwhelmed by calls from students in the short time that we had allowed for them to take the test.
In the second year ( 2006) of the online placement system, we gave the students a longer time in which they could go online to take the test and so they experienced fewer connection problems due to university internet use.It could also be true that in 2006 more students had better equipment and fewer students were trying to access the test using a dial-up modem.We also were able to alleviate some of the viewing problems by changing the format of the test so that the test items came up one at a time rather than all at once.Some students did not have sufficient capacity to view the entire test because the bandwidth required to show the mathematical formulas using Java script for the formulas was excessive.
The reliability issue was also addressed by clearer communication.We emphasized to the students that the test was to be used for proper placement and that the placement system was designed to help them achieve success in their first freshman mathematics class.Students were encouraged to give an honest representation of their skill.We tried to convince the students that there would be no advantage in their being placed in a class that would be too difficult for them.
Several facts have led us to believe that most students followed the instructions as given.For example, the students were told to take a maximum of one hour for the test.They were given a warning when they had five minutes remaining.Although they might have ignored the instructions and continued to take the test beyond the hour, Blackboard reported the time that they spent with the test.Only a handful of students took more than the allotted time and most of these students had contacted us to complain of technical problems.Another reason that we feel that the students were honest is the strong correlation between their Math SAT score and their placement test score.The adjusted R2 between the Math SAT and the placement score was 36.3%.
Furthermore, this result is in line with the corresponding adjusted R2 values for the last three years of the paper test (2002, 2003, and 2004) which were 36.5%,36.5%, and 31.9%,respectively.Although there were certainly exceptions to this rule, when we encountered students with extremely low SAT scores and high scores on the placement test, we decided to place them in the lower class and ask them about this discrepancy.Based on discussions with such students, we adjusted their placement, if deemed advisable.Since the paper test was given in a supervised setting, we believe that is students were not submitting their own work or taking extra time on the online test, there would be some differences in these and other statistical comparisons of the results.Additional statistics will be provided in the evaluation section of this paper.

PLACEMENT METHOD
The test results were downloaded from Blackboard into an EXCEL spreadsheet and together with other admissions variables (high school GPA, class rank, and Math and Verbal SAT scores) were used to place the students in the appropriate freshman class.
Placement was done with a combination of formula and personal judgment using available data including the placement test score.The formula used was developed over several years.In 1993, the formula produced a score from the calculation: 2 x (placement test score) + (Math SAT score).The development of this formula was described in an earlier paper (Odell & Schumacher, 1995).Although the authors believed that high school achievement was a very good predictor of college success, the data for high school rank were not easily available.It was also difficult to precisely interpret the rank without looking at the high school transcript to see what type of courses the student had taken.For example, some students take more honors courses than others, which could affect the interpretation of their rank.
Using the above formula, we arrived at a score with which to determine the students' placement.Once a cutoff number was decided on, based upon the number of students who could be placed in the E105 (remedial) sections and the A105 (advanced) section, personal judgment was used to decide where to place the borderline students.In these cases, we looked at the level of difficulty of the courses taken in high school.Prior to the online test, the faculty member who had been in charge of placement had referred to each of these student's admissions folders and high school transcripts to obtain the information about which classes were taken.In 2005 and 2006, when students took the placement test online, they were also asked to self report which mathematics courses they had taken in high school and so this information was readily available along with the placement test score.

COMPARISON OF PLACEMENT: PAPER TEST VERSUS ONLINE TEST
We were concerned that there might be some erosion of the placement process by using the online test.To address this concern, logistic regression was used to try to determine if placement with the online test was consistent with the placement that had been done with the paper test, which was given in a strictly supervised setting.A logistic regression was first run on the 2004 data (paper test) with placement into the remedial Math E105 as the dependent variable and Verbal SAT (VSAT), Math SAT (MSAT), placement test score (PMT), and high school GPA (GPA) as independent variables.Although, Verbal SAT had not been used previously, we chose to include it since we felt that it might improve the accuracy of the forecasts.The 2005 values for each student were then substituted into this regression equation obtained the previous year.This equation resulted in a 94.2% prediction accuracy of placement with VSAT, MSAT, and PMT all significant coefficients of the 2004 placement equation.GPA, however, was not significant in this model.This provided strong evidence that using online placement test score and personal judgment, our placement procedure in 2005, was consistent with the previous year placement where the test had been given face to face.
The fact that high school GPA was not significant was quite interesting.In our subjective decisions involving borderline students, we learned to trust class rank much more than high school GPA.The GPAs seemed to be inflated and less correlated with both Math and Verbal SAT scores.In an attempt to get a better logistic regression fit, we ran the logistic regression with the 2005 coefficients, when the high school GPA was replaced with high school rank.The classification accuracy remained the same at 94.1%, but all of the independent variables were significant.Using class rank, however, reduced the total number of cases that were predicted since there were more missing values and the number of cases correctly predicted dropped from 614 to 429 of the 761 students.Thus, either class rank or GPA were somewhat useful for the purposes of this study but if we wish to use the logistic regression equation for future placement purposes, we will probably continue to use GPA since it will give a more complete data set.If we wish to use class rank, which is a better predictor, we will need to consider using some Data Mining statistical techniques which are designed to handle missing values.

EVALUATION
It is difficult to judge the success of student placement based on final grades because of the wide variety of teaching styles of the individual teachers.Also, although the material is the same, the individual instructors give their own exams and grades.How does one, therefore, compare an A in an honors course with an A in a Math E105?We did, however, make an attempt to see if there was consistency in grades for the remedial class (Math E105) and the regular section (Math 105) from 2004 to 2005.We decided to concentrate on these two classes rather than the advanced classes because the key issue for potential E105 students is whether or not they are placed in E105 or 105.
We established the hypothesis that there were no grade differences between final grades for all students placed either in Math E105 or Math 105 by the paper test in 2004 and the online test in 2005.In order to test this hypothesis, we divided the grades into five categories (W or F, D, C, B, and A) and ran a chi-square test of Grade by Year of Placement (paper vs. online).The results (see Table 1 below) revealed that there was no significant difference between the grades for the students tested by paper test or by online test (chi-square = 4.807, df = 4, p-value = 0.308).
To be sure that this was true in particular for the weakest students, we ran the test again for Math E105 students only (see Table 2 below) and again found no significant differences (chi-square = 6.286, df = 4, p-value = 0.179).We were also concerned about addressing the question of possible cheating on the placement test.When we compared the test scores for the online test (2005), with the paper test (2004), we did find significantly higher scores with the online test as shown in Table 3.There are, however, several possible explanations for these differences.We felt that scores might have been high because the students could take the test in their own homes at a convenient time rather than during freshman orientation on campus when they were more focused on meeting other students and learning about the school.Compared to taking a mathematics placement exam, the many other activities and presentations provided during orientation offered a great deal of "more attractive" competition for their attention.We also noted that the class of 2005 had higher average Math SAT scores (mean of 576.81) than the class of 2004 (mean of 568.58).
In addition, we wanted to see if the fact, in particular, that the weakest students did better on the online test indicated the possibility of their having cheated.To address this question, we categorized the students by their SAT's, defining the weakest as those with Math SAT ≤ 520 and total SAT ≤ 1000.The results of this inquiry are in Table 4 below.For the weakest students, based on SAT scores, Tables 4 provides evidence that students taking the online test did no better than students in the same category who took the paper test.
These results provide some evidence that these students, as a group, did not successfully cheat to obtain higher scores than they deserved.

CONCLUSION
As of the second administration of the online placement test, we are very pleased with the results.The administration is happy that we are able to complete mathematics placement by the end of May since the registration for most students can be completed in June during freshman orientation.Students are pleased because they can take the test at a time and place convenient to them and they can get their class schedule prior to the start of classes in September.This is particularly true in the case of international students who rarely can visit campus prior to the start of classes in September.An added advantage to the online placement testing turned out to be a savings in time in grading the placement tests.Since the tests are graded immediately in Blackboard and easily downloaded into an EXCEL spreadsheet, this alleviated that task.
We are also pleased that we have worked out many of the technical problems and that the number of students who are not satisfied with their placement is minimal.Finally, and most significantly, we are satisfied that the accuracy of the placement using the online test is consistent with the accuracy of the placement using the paper test.The next step in our placement process is to try to utilize the logistic regression formulas developed in this research for placement of future classes.

Table 1 :
Paper and online test grade comparison for all math 105 and math e105 students

Table 2 :
Paper and online test garde comparison for only math e105 students

Table 3 :
Overall quartiles for placement test scores

Table 4 -
Placement scores for students with MSAT < 520 and total SAT < 1000