Teachers’ Perceptions of Mathematics Content Knowledge Assessments in Professional Development Courses

The goal of this follow-up study was to generalize the findings of a previous inquiry into how assessing teachers’ mathematical knowledge within a professional development (PD) course impacted the teachers’ perspective of their learning and their learning experience. This quantitative research study examined whether the teachers’ attitudes about assessments found in the original study were generalizable to a similar population as well as whether factors involving their No Child Left Behind (NCLB) status and prior experience with the PD facilitators were factors affecting their perspectives. Results indicate that the teachers felt that they learned more mathematics, increased their learning efforts, and gained confidence in their understanding of and ability to teach mathematics because they were assessed. Additionally, the teachers’ NCLB status or prior experience in PD with the facilitators had virtually no impact on the teachers’ perceptions about assessment. Characteristics of the PD that led to these results are explained.


METHODS
The purpose of the study was two-fold. First, we wanted to verify quantitatively the results of the aforementioned qualitative study, including whether the perceptions volunteered by the initial teachers on an open-ended survey were consistent among the second set of teachers when asked specifically and whether those perceptions were generalizable statistically. We were particularly interested in the inservice teachers' perceptions regarding (a) the appropriateness of assessment, (b) their concern about the exam for program evaluation, (c) the impact on learning resulting from assessment, and (d) the support for their learning engendered by the aforementioned properties of the assessment. Second, we wanted to know if these results were influenced by NCLB status or previous professional development courses with the facilitators.
The intent was to verify that the positive reactions to the assessment were not just from teachers that were highly qualified or that had developed a rapport with the facilitators during previous PD courses. The research question for the first goal of the study was as follows: 1. Were participants' responses significantly different from neutral on survey items related to (a) the appropriateness of assessment, (b) their concern about the exam for program evaluation, (c) the impact on learning resulting from assessment, and (d) the level of support engendered by properties of the assessment?
The second goal of the study required three research questions to account for the interaction between NCLB status and previous professional development courses with the facilitators. They were as follows: 2a. Is there a significant interaction between highly qualified status and prior professional development experience with the facilitators on participants' attitudes and beliefs about being assessed?
2b. Is there a significant main effect of highly qualified status on participants' attitudes and beliefs about being assessed? 2c. Is there a significant main effect of prior professional development experience with the facilitators on participants' attitudes and beliefs about being assessed?
The participants of the study were 81 inservice teachers who mainly taught grades 5-8.
Approximately three-quarters (75%) of the teachers were female and approximately three-eighths (38%) were highly qualified. Half of the respondents reportedly taught for eight years or more. Of the 76 participants who reported their undergraduate major, 37 (49%) had a general education degree, 14 (18%) had a science degree, 10 (13%) had a social science degree, and 11 (14%) earned a mathematics or mathematics education degree.
The three PD courses each consisted of 2 weeks in the summer with a total of 80 contact hours. The content focus for the courses and associated enrollment follows: Data Analysis and Probability (32 teachers), Number and Operations (17 teachers), and Geometry (32 teachers). The facilitators' backgrounds included extensive work as mathematicians and extensive work with inservice teachers. Chamberlin and Powers served as researchers for these courses, while Novak served as the main instructor and was assisted by 2-3 other facilitators for each course.
The PD is classified as content-based (Lappan, 2000;Loucks-Horsley, Hewson, Love, & Stiles, 1998), implying that a significant portion of the time is devoted to assisting teachers with increasing their subject matter knowledge. The PD courses for this study followed a threepronged approach: deepening teachers' mathematical understanding through problem solving, using mathematical learning experiences to re-examine practice, and providing support for teachers to modify instruction. First, the courses placed teachers in the role of mathematics learners to deepen their mathematical understandings. This was done through careful facilitation of group problem-solving sessions with a strong emphasis on teachers providing coherent explanations of mathematical ideas. Although most of the activities could be adapted to a middle school classroom, the mathematics was extended to levels that were appropriate for the teachers as adult learners. Second, the teachers were asked to reflect on their mathematical learning experiences and on future applications to practice. The teachers were encouraged to examine "what just happened" in the mathematics component and to consider in essence, "Does my practice provide these kinds of learning experiences for my students?" Finally, the teachers were supported to integrate their new understandings of both mathematics and the teaching and learning of mathematics into their practice. This involved completing two classroom implementation projects and follow-up activities that supported teachers in modifying curriculum or specific instructional practices. For more information and examples of this three-pronged approach, see Chamberlin, Farmer, and Novak (in press).
In particular, teachers increased their content knowledge as well as their pedagogical understandings. Furthermore, Hill, Rowan, and Ball (2005) and Saxe, Gearhart, and Suad Nasir (2001) found that teachers' mathematical knowledge was significantly related to student achievement gains. Previous research and program evaluations have shown that PD courses similar to the courses in this study also can positively impact teachers' practices and interactions with students, teachers' knowledge of mathematics for teaching, and students' mathematics achievement (Farmer, Gerretson, & Lassak, 2003;META Associates, 2006;OMNI, 2006).

DESCRIPTION OF THE ASSESSMENTS AND ASSESSMENT PROCESS
For the study, assessment is defined as follows (NCTM, 1995): Assessment is the process of gathering evidence about a student's knowledge, or ability to use, and disposition toward mathematics and of making inferences from that evidence for a variety of purposes. (p. 3) In each PD course, mathematics assessments consisted of a mathematics exam for outside evaluation purposes, nearly daily homework assignments, 3-4 quizzes, and a mathematics content project. In designing and implementing the assessments, the facilitators strove to utilize current standards regarding assessment of K-12 students' mathematical knowledge as well as a number of important principles of assessment relevant to adult learners (Black & Wiliam, 1998;Cumming & Gal, 2000;Ginsburg & Gal, 2000;Kasworm & Marienau, 1997;Mathematical Sciences Education Board, 1993;NCTM, 1995NCTM, , 2000NCTM, , 2003National Forum on Assessment, 1995).
The mathematics exam i administered for evaluation purposes was developed by Heather Hill, Brian Rowan, and Deborah Ball from the University of Michigan's School of Education.
The instrument has been used and tested for several years and has been found to be reliable and valid. The purpose of the exam is to evaluate the effectiveness of programs that work to enhance teachers' mathematical reasoning skills and their knowledge in middle school number and operations, pre-algebra and algebra, and geometry (Hill, Schilling, & Ball, 2004). The pretest of this exam was administered the first day of the course, the posttest on the next to last day.
The nearly daily homework assignments were typically continuations of work done in class and were intended to help the teachers determine how well they understood the material.
The assignments were graded in a variety of ways. Sometimes they were graded for completion, other times for accuracy. Both letter and numerical grades were used by the facilitators.
Revisions were often allowed, especially if ideas were central to course content. The facilitators expressed to the teachers that what was most important was that they spent time trying to do the homework, not that they got it all done or all right. The facilitators wanted them to think of doing mathematics as a process, not just as getting an answer (NCTM, 2000). This included providing the teachers with time guidelines to reduce their stress: They were told that if they spent longer than a certain amount of time or if they got completely frustrated, they should quit and ask questions the next day.
In each course, three to four 30-45 minute quizzes were given, employing at least two of the following formats in each course: · Individual quizzes (the usual model for giving quizzes), · Quizzes with revisions (earn back from one-half to all the points missed), · Quizzes with partners (take a quiz with one or two other people), and · Quizzes with pair-ups (take the quiz yourself, then meet with one other person to discuss the quiz and make changes as desired).
These allowed the teachers to experience quizzes as part of their ongoing learning process. In determining the content for the quizzes, the facilitators took a "big ideas" approach, testing the most important concepts and procedures that the learners had time to learn well. The facilitators referred to this approach as "testing in the center", which stands in contrast to an approach that assumes students have learned basic material and preferentially chooses topics that are less central. Figure 1 provides an example of the second quiz in the Number and Operations course.
The facilitators also varied the difficulty and nature of the quiz tasks. Some were straightforward, involving procedures or memorization (see #4 in Quiz 2), while others required significant explanation or illustration of mathematical ideas (see #2 in Quiz 2). In addition, the facilitators allowed teachers choice among similar items (see #3 in Quiz 2). This reduced the stress on the teachers and acknowledged that learning processes are different for different individuals. This also acknowledged the time it takes to internalize and understand mathematics at the level of depth and complexity required for teaching. In addition to the homework and quizzes, the teachers in each course completed a mathematics content project. The project asked the teachers to apply what they had learned and to explore more fully the mathematical ideas from the course. See Appendix A for an example of a project from the Number and Operations course. Each project was introduced in class to help teachers develop a sense of the expectations, but the teachers mostly completed it on their own time with a due date 2 weeks after the completion of the course. The projects had focused and opened-ended questions with lots of opportunity for making and testing conjectures and exploring mathematical ideas. The emphasis was on the process of doing mathematics and the learning that happened along the way, not just on the end product.

Designing and Implementing Assessments in a Supportive Way
In designing and implementing the assessments, the facilitators had four explicit goals in mind. First, they strove to be open with the teachers about the process, talking with them about their decisions in designing and evaluating the assessments. Second, they strove to communicate their view of the on-going nature of learning mathematics. Third, they openly acknowledged and addressed affective consequences of the assessment. Finally, they strove to be flexible in the implementation and use of the assessments.
Quiz 2 1. Explain the partitive and measurement models for division. Illustrate each of your explanations with an example. 2. Explain how you would model each of the problems below using two-colored counters. Be sure to include diagrams to enhance your explanations. a. (+5) + (-8) b. (-2) -(-3) 3. Choose any two of the following three models for illustrating multiplication of fractions: pattern blocks, counters, or fraction squares. Use the models to calculate 2 1 6 5 × . Be sure to explain the problem using each model.

Being Open with the Teachers about the Assessment Process
At the beginning of each course, the facilitators shared that the assessment was motivated by three factors: to receive and provide feedback on the teachers' learning, to assign accurate grades for graduate credit and No Child Left Behind (NCLB) reporting purposes, and to meet evaluation requirements for associated funding. They also were open in explaining expectations and in answering questions about the assessments. Many times such explanations centered around the content project and the associated expectations.
As another example, prior to many of the quizzes, they asked the teachers to spend a few minutes during class helping them determine the topics that should be on the quiz. The facilitators asked the teachers to focus on main ideas -those they had spent time on and learned well. The process involved some negotiation between facilitators and the teachers, but usually resulted in substantial agreement about what would make for a 'good' quiz, i.e., one that allowed the teachers to demonstrate their learning. While having these discussions about the quizzes took time away from instruction, they advanced the learning process in some important respects. First, they provided the facilitators with information about teachers' concerns and questions as learners.
Second, they provided the teachers with information on what the facilitators wanted them to learn and why. Third, the discussions served as a review of what had been learned and as an opportunity to put mathematical ideas into perspective, seeing how they related and their importance.

Communicating the On-Going Nature of Mathematics
The teachers' understandings of a topic were expected to develop gradually as they engaged in activities and completed assessments throughout the course. The facilitators frequently communicated this philosophy and their belief that individuals often need to work through and interact with a mathematical idea several times before developing deep understanding. The facilitators challenged the teachers to do their best work but not to stress, as they would receive feedback and opportunities to revise their work. Indeed, the facilitators often emphasized the idea that professionals and mathematicians in particular proceed through such revision cycles. This helped the facilitators communicate the value of revision cycles and to express clearly to the teachers that their true aim was for them to learn.

Acknowledging and Addressing the Affective Consequences of Assessment
The facilitators also paid explicit attention to the affective consequences of the assessment. This often took the form of conversations with individual teachers when they were having difficulty with the homework. About half of this time was spent addressing the content of the homework assignment; the other half was spent talking about how the teachers were feeling about the homework and the course, with the facilitators reassuring them about their abilities and performance. Sometimes, the facilitators addressed affective issues with the whole class. For example, on days when a facilitator sensed that a large majority of the teachers were feeling overwhelmed, the facilitator often took the time for the teachers to go around the class and share how they were doing. When the teachers realized that they were not alone in feeling overwhelmed, their concerns appeared alleviated, and much laughter often accompanied these discussions. The facilitators attended to such affective issues because they believed that learners are less able to engage in cognitive tasks without first attending to current emotional issues.

Being Flexible in the Implementation and the Use of the Assessments
Going into each professional development course, the facilitators had an assessment plan in mind, but they were flexible with many of the assignments. Their choices for homework and quiz content were heavily influenced by the teachers' performance on prior assessments and their perceptions of the understandings the teachers demonstrated in class. While two or three different mathematical ideas were explored in a day, the facilitators were cautious about which of those topics they would assign to homework. They tried to balance what the teachers needed to solidify from the day's learning with what they needed for the next day. The facilitators also tried not to overwhelm the teachers either in terms of time or intensity of effort required, choosing some straightforward problems and some that pushed them to synthesize what they had been learning.

DATA SOURCES
We (the authors) developed a survey based on the findings from Chamberlin, Farmer, and Novak (in press)

ANALYSIS
To address the first goal of the study, namely to verify quantitatively the inservice teachers' perceptions of the consequences of assessment during professional development, one sample t-tests were performed. The null hypothesis of each test was that the mean response of the participants was no different than a neutral response (3.5 on the scale of 1 = Strongly Disagree to 6 = Strongly Agree). Because 37 separate t-tests were performed, a Bonferroni adjustment to the significance level was established, specifically an alpha value of .001, or roughly .05 ÷ 37, was set. The purpose of this was to reduce the likelihood of committing a Type I error, since we wanted to be certain when participants' perceptions were different than neutral.
The second goal of the study was to determine whether participants' highly qualified status or prior experience with the facilitators were mitigating factors in their perceptions. To address this goal, a two-way ANOVA with independent variables highly qualified status and prior PD experiences were performed for each survey item. Because faulty conclusions for these analyses were based on committing a Type II error (i.e., failing to reject the null hypothesis when it is in fact false), we kept the significance level at .05. To clarify this point, it was more critical to retain the statistical power for each analysis since the desired outcome was that no significant difference among the groups of participants would be found.   Table 3 presents the results of the one sample t-tests of the participants' mean scores on items rating their perception of the impact of the mathematics exam for program evaluation.

Generalizing Teachers' Perceptions
Primarily, the participants' mean score was significantly different from neutral ( Table 4 presents the results of the one sample t-tests of the participants' mean scores on items rating their perception of the impact of the assessment on mathematical learning during the professional development. With the exception of the perception of participating more because of assessments, each item was significantly different from neutral and indicated agreement by the participants. Before asking the 81 teachers about the support engendered by the properties of the assessment, we wanted to verify that these teachers felt the same properties were evident. All participants at least somewhat agreed that the assessments asked questions that were fair, covered the expected mathematics, were open-ended, that the instructors' goal was for them to learn, and that their grade was based on multiple assessments. All but one participant (99%) at least somewhat agreed that they were able to seek assistance and clarification from the instructors on the assessments, they understood what the instructors expected on the assessments, and the instructors used the assessments to make instructional decisions. Ninety-seven percent (all but two) of the participants at least somewhat agreed that they received evaluative marks or grades on the assessments that were informative and the assessments were within their range of mathematical abilities. Finally, ninety-six percent (all but three) of the participants at least somewhat agreed that explanations were provided about the requirements and expectations of the assessments, they were able to collaborate with their peers on the assessments, they completed revisions on the assessments when given the opportunity to do so, and they received helpful written feedback on the assessments. Table 5 presents the results of the one sample t-tests of the participants' mean scores on items rating their perception of the support engendered by the properties of the assessment. Each item was significantly different from neutral and indicated agreement by the participants. The use of assessment to guide instruction 73 5.14 0.72 19.40 .000   To answer the research questions, a two-way analysis of variance was performed on each of the survey items. Table 8 presents the result of these analyses. There was a significant interaction between highly qualified status and prior professional development experience with the facilitators on students' responses to the item, "The assessments covered the mathematics that I expected". There was a significant main effect of highly qualified status on the same item, which was examined in the post hoc analysis. Additionally, there was a significant main effect of highly qualified status on the item, "I received evaluative marks or grades on the assessments that were informative". There was no other significant interaction or main effect difference on any other item examined in the study.

Impact of Highly Qualified and Prior PD Experience
To determine the nature of the significant interaction on participants' expectations about the material covered on the assessments, a post hoc analysis was performed. First, Figure 2 reveals the nature of the interaction. It was apparent that those participants who had prior experience with the facilitators of the professional development had identical agreement about what was expected on the assessments whether they were highly qualified or not, but there was a distinct difference for participants who were taking the professional development course from the facilitators for the first time. Table 9 presents the results of post hoc t-test analyses that confirm the observations from the figure. There was no significant difference between highly qualified participants and those who were not highly qualified for participants who had prior professional development experience with the facilitators. However, there was a significant difference between highly qualified participants and those who where not highly qualified for participants who had prior professional development experience with the facilitators. Specifically, highly qualified participants who did not have prior experience with the facilitators of the professional development had a significantly higher mean score than participants who were not highly qualified and who did not have prior experience with the facilitators about the extent to which the assessments covered the expected mathematics.  Figure 2. Estimated marginal mean responses on "The assessments covered the mathematics that I expected" for professional development experience with facilitators by highly qualified status. An examination of the mean scores revealed the nature of the significant main effect of highly qualified status on participants' opinions about receiving marks or grades that were informative. The 26 participants who classified themselves as highly qualified had a mean score of 4.81 and a standard deviation of 0.75; the 40 participants who were not highly qualified had a mean score of 5.23 and a standard deviation of 0.58. The participants who were not highly qualified agreed significantly more than those who were highly qualified on the item, "I received evaluative marks or grades on the assessments that were informative".

DISCUSSION
The purpose of this study was two-fold: to verify quantitatively the results of the previous qualitative evaluation of the impact of assessment on teachers' perceptions of their learning and to determine if these results were influenced by NCLB status or previous PD courses with the facilitators. With regard to the first purpose, the previous results were verified. The teachers reported that because they were assessed, they felt that they learned more about the associated mathematics. In addition, they reported that the assessment caused them to increase their learning efforts by taking actions such as participating more in class, paying more attention in class, asking more questions of the instructors, revisiting the mathematical material on their own throughout the class, looking for connections among the mathematical ideas, and making their written responses more clear and thorough. The teachers also reported gaining confidence in their understanding of and ability to teach mathematics. Thus, from the teachers' perspective, the potential negative effects of assessment, such as rote learning or excessive anxiety reported by Askov, Van Horn, and Carman (1997), Kohn (1993), Strickland and Strickland (1998), and Trumbull (2000), appear to have been primarily avoided and replaced instead by positive effects for learning. We surmise that these positive effects may be due to an important aspect of the assessment process in these PD courses -the assessment and learning of mathematical topics and material was on-going and demonstrating mastery of those ideas was expected. This stands in contrast to other forms of PD in which an interesting mathematical activity may be completed but then the central mathematical concepts are never revisited or assessed.
The teachers also were not concerned about taking the exam for program evaluation as the facilitators were clear in describing the exam was for determining growth over time and was not to be included in their grade. We find this news encouraging as many teacher educators are required to administer similar such exams for external funding. For example, when   Because the PD was a graduate course and teachers could receive credit for it, the participants reported that it was appropriate to assess their mathematical knowledge. Thus, for other teacher educators offering PD courses for credit, they may likely expect that their teachers will also see assessment as appropriate. If offering PD without credit, however, the results of this study may be shared with the teachers to possibly produce similar buy-in.
Finally, the quantitative results verified that the teachers found the same properties of the assessment supportive of their learning. Thus, we see again that many of the same recommendations for assessing K-12 mathematics and for assessing adults generalize to contentbased PD courses for teachers (Chamberlin, Farmer, & Novak, in press). In particular, assessment should: · Contribute to students' learning and reflect mathematics that is most important for students to learn, · Be accompanied by clear and regular communication about expectations, requirements, and content, · Be integrated and aligned with instruction and curriculum, · Honor students' unique qualities and experiences by allowing them to demonstrate their knowledge on multiple and varied formats, · Provide students with timely and informative feedback, · Encourage self-assessment and allow for cycles of revision, and · Be an open process in which students are involved in choosing tasks, selecting criteria, selecting solution approaches, and interpreting results (Black & Wiliam, 1998;Cumming & Gal, 2000;Ginsburg & Gal, 2000;Kasworm & Marienau, 1997;MSEB, 1993;NCTM, 1995NCTM, , 2000NCTM, , 2003National Forum on Assessment, 1995).
With regard to investigating the impact of NCLB status and previous PD experience with the facilitators, these two factors had virtually no impact on the majority of the teachers' perceptions of the assessment. This is particularly encouraging, as it further verifies that using assessment in a conscientious way that follows assessment recommendations can be supportive for teachers with and without strong backgrounds in mathematics and for teachers new to working with particular teacher educators.
There were two instances, however, when these factors were significant -first, the interaction between highly qualified status and prior professional development experience with the facilitators on students' responses to the item, "The assessments covered the mathematics that I expected". It was clear that those who had taken a professional development course from the facilitators whether they were highly qualified or not had a good grasp of what was to be assessed. For those participants new to professional development with the facilitators, the ones who were highly qualified appeared to expect the material on the assessments, but the one who were not highly qualified were surprised with some of the material on the assessments. It is possible that communication of the expectation of material on assessments in professional development is communicated through experience. For those who had PD with the facilitators, prior assessment written by the facilitators likely communicated to them the style of the mathematics that would be on the assessments of the current course. For those who were highly qualified, the number of mathematics courses taken to become highly qualified may have provided the necessary experience to anticipate the expectations of the facilitators on the assessments of the course. Finally, those who did not have direct prior experience with the facilitators and who were not enculturated in the practices of mathematics assessment (i.e., not highly qualified) struggled more with their expectations of the mathematics on the assessments. It should be noted however that even this group agreed that the assessment covered the mathematics they expected (M = 5.11). Thus, this finding perhaps points to the need for the facilitators to continue thinking about how to clearly explain and illustrate the expectations and requirements of assessments.
The other significant influence of NCLB and previous PD courses with the facilitators was that highly qualified teachers who attend professional development may not benefit as much from (i.e., be informed by) evaluative marks or grades on assessment as teacher who are not highly qualified. We suspect this may be the case because the highly qualified teachers may be performing well on assessments and thereby receive less specific feedback about their understandings.
The study presented here includes one important limitation -the results are based on teacher self-reports. Doing so was appropriate in this case, as we were interested in the teachers' perception of a process that was being required of them. However, self-reported data do not provide a full picture of how the assessment may have impacted the teachers. Two additional impacts that could serve as topics for further study include how the assessments impacted the teachers' actual learning of mathematical content as evidenced on those assessments and how the assessment process impacted the teachers' use of assessment in their grades 6-8 mathematics classrooms.
Yet, we feel the results of this study are informative and encouraging for mathematics teacher educators. We hope they consider incorporating assessment into their PD in ways that reflect the recommendations for assessment and perhaps draw on ideas expressed here, while also realizing that they can use more standardized forms of assessment without damaging the rapport and openness to mathematical learning that they are trying to engender. We feel taking a proactive and conscientious approach to assessment changed the nature of the assessment from being a potentially negative experience that showed the teachers what they did not know to a positive experience that allowed the teachers to show us and themselves what they did know and understand.