UNIVERSITY STUDENTS’ KNOWLEDGE AND BIASES IN CONDITIONAL PROBABILITY REASONING

. The research question in this study was assessing possible relationships between formal knowledge of conditional probability as well as biases related to conditional probability reasoning: fallacy of the transposed conditional; fallacy of the time axis; base rate fallacy; synchronic and diachronic situations; conjunction fallacy; and confusing independence and mutually exclusiveness. Two samples of university students majoring in psychology and following the same introductory statistics course were given the CPR test before ( n = 177) and after ( n = 206) formal teaching of conditional probability. Results indicate a systematic improvement in formal understanding of conditional probability and in problem solving capacity but little change in those items related to psychological biases. is a sign of the discriminant as the better in


INTRODUCTION
Conditional probability was included by Heitele (1975) in his list of fundamental stochastic ideas that have helped probability theory to develop throughout history. These basic ideas lie at the heart of all probabilistic situations and are as simple as they are powerful. Heitele took the view that these fundamental concepts can be studied at various degrees of formalization, which are manifested in more complex cognitive and linguistic levels as one proceeds through school to university using a spiral curriculum. He also suggested that even young children may be supported to build intuitive models for these fundamental ideas that later help them to establish correct analytic knowledge. However, Heitele also pointed out the fact that these fundamental ideas are sometimes accompanied by misconceptions or errors. "There are fundamental ideas, as there are fundamental errors, and both are counterparts of each other. Such errors bridge the centuries, the ages and the cultural layers, and may be criteria of what is really fundamental" (Heitele 1975, p. 191).
The relevance of building sound knowledge and conceptions as regards to conditional probability is due to the fact that conditional probability allows us to change our degree of belief in random events when new information is available. The importance of being able to interpret conditional statements in terms of risk assessment for students in their lives outside school is also illustrated by Watson (1998) who suggests that conditional probability reasoning is a crucial part of statistical literacy, since it helps to make accurate decisions or inferences in everyday life. Borovcnik & Peard (1996) remark that probabilistic reasoning is different from logical or causal reasoning and thus, counterintuitive results in probabilistic tasks are found even at very elementary levels. This is in contrast with other branches of mathematics, where counterintuitive results are found only when working at a high degree of abstraction. These types of counterintuitive results are frequent when dealing with conditional probability and cause a number of psychological biases.
Research on the understanding of conditional probability has been carried out with both secondary school and university students (e.g. Fischbein & Gazit 1984, Tarr & Jones 1997, Watson & Moritz 2002, Tarr & Lannin 2005. Their research is summarized in previous papers published in this same journal (Díaz &de la Fuente 2006a and2007a) where we described the construction of a comprehensive instrument (Conditional Probability Reasoning, CPR test) oriented to assess both formal knowledge of conditional probability as well as related biases described in the literature. We also analysed the responses to the test from a sample of 418 university students to explore possible relationships between formal knowledge and psychological biases.
Students' performance in the formal components of the test was quite good. In particular, we observed a high percentage of correct or partly correct solutions to problems (including total probability and Bayesian problems). However, some of the biases described in the literature were widespread in these students' thinking. Moreover, results of factor analysis showed that responses to the items assessing the biases in conditional probability reasoning were unrelated to those assessing formal knowledge.
In this new paper, we continue the research described in Díaz & de la Fuente (2007a). We analyze possible improvements of both formal knowledge and biases related to conditional probability after instruction. Two new samples of psychology students (n = 177 before instruction, n = 208 after instruction) were given the CPR test that is included as appendix A to the paper.
Below, we first describe the biases assessed in the test and then analyze the responses.

BIASES RELATED TO CONDITIONAL PROBABILITY REASONING
In relation to conditional probability, Feller (1968, pg. 114) suggested that: "The notion of conditional probability is a basic tool of probability theory, and it is unfortunate that its great simplicity is somewhat obscured by a singularly clumsy terminology". A common definition of conditional probability is the following: Suppose an event B, in a sample space, for which it holds . In this case, for every event A in the same sample space, the conditional probability of A given that B happened, is defined by: Apparently, this is an easy mathematical definition, although various researchers have shown that its understanding and application are not always correct. From the mathematical point of view its difficulties are no different to other mathematical concepts. However, conditional probability is, at the same time, enriched and made more complex when it is related to several particular situations, which are discussed below. Some of the items of the CPR test directly address heuristics that might lead to idiosyncratic reconstructions of the problems and to wrong answers. Other items relate to tasks in which knowledge and perception of various aspects of the concept of conditional probability would help to find the solution.

Confusing causal and diagnostic situations
Item 7 is related to causal issues. Causation is a complex scientific concept and also a notion perceived in a natural way by human beings because, to a great extent, we organize our knowledge about the world we live in, by taking account of causes and effects. The concept of causation develops after the period of formal thought, though our perceptions about causation are sometimes biased and at other times causation and conditionality are confused (Pozo 1987). Different philosophical theories have explained causation. One feature commonly accepted (although it is not the only feature) is that if an event A is the cause of another event B, whenever A occurs, B also occurs, and therefore it holds that 1 ) . On the contrary, if then it is not true that A is a cause for B though the existence of a conditional relationship indicates that a causal relationship is possible. In some cases conditional relationships do not imply causation.
From a psychological point of view, the person who assesses a conditional probability may perceive different types of relationship between A and B depending on the context (Tversky & Kahneman 1982a).
is viewed as a causal relation, and These different relationships concern judgments of conditional probability. The impact of causal data on the judgment of the probability of a consequence is usually greater than the impact of diagnostic data on the judgment of the probability of a cause. For this reason, people tend to overestimate causally perceived conditional probabilities while they ignore diagnostic conditional probabilities. Moreover, some people confuse diagnostic and causal probabilities; this is a particular case of confusing the two directions of conditioning, i. e., and that is termed as the fallacy of the transposed conditional (Falk 1986).

The fallacy of the time axis
Items 8 and 9b directly relate the course of events to time. The time-axis fallacy is a common misconception that also relates to the concept of cause-and-effect. Most students understand and accept the fact that the outcome of an event can affect the outcome of a later event, but think that it cannot actually affect another outcome that has already happened. Falk (1989) found that many students assume that the knowledge of an event's outcome cannot be used to determine the probability of the occurrence of a previous event.
For example, in item 9b students typically argue that, as when drawing the first ball we have not yet obtained the second, the result of the second drawing is completely irrelevant in determining the probability of the previous drawing. This is false reasoning, as even when there is no causal relation from the second experiment to the first one, the information in the problem (that the second ball is black) has changed the sample space in the first experiment (there is now just one black ball and two white balls relevant to compute the probabilities in the first drawing). Similar results are reported by Gras & Totohasina (1995) who remarked that the majority of situations where conditional probabilities intervene can be thought as a reduction of sample spaces. This reduction is, however, not easy to realize in chronological situations where a series of experiments or times intervene. They introduced the term chronological conception of conditional probability for the case where students interpret the conditional probability as a temporal relationship and think that the conditioning event B should always precede event A.

The base rate fallacy
Item 2 deals with a Bayesian problem where only the inclusion of base rates would lead to a solution. The base rate fallacy is an error that occurs when the conditional probability of some hypothesis H, given some evidence E, is assessed without taking sufficient account of the "base rate" or "prior probability" of H. In calculating the probability of an event H, two types of information may be available: generic information about the frequency of H (the base rate) and specific information about the case in question (E). People who have only generic information tend to use it to calculate the probabilities, which is the rational thing to do. In contrast, when people have both types of information, they tend to make judgments of probability based entirely upon specific information, ignoring out the base rate information even if they perform calculations and do not judge the probability intuitively.
This fallacy was identified in early research by Tversky & Kahneman (1982a) when people were asked to solve problems involving Bayes' theorem. In Bayesian problems you are given statistics for a population as well as for a particular part of the population; both types of information have to be considered together to solve the problem; however, people tend to ignore the population base rate (Bar-Hillel 1983, Koehler 1996. These authors suggest that people do not employ Bayesian reasoning intuitively and instead they found wide occurrence of the baserate fallacy in both students and professionals.

Synchronic and diachronic situations
Items 10 and 13 relate to such issues. In diachronic situations the problem is formulated as a series of sequential experiments, which are carried out over time. Synchronic situations are static and do not incorporate an underlying sequence of experiments. Formally, the two situations are equivalent; however, empirical research shows that individuals may perceive them differently. Sánchez & Hernández (2003) found that 17 to 18 year-old students do not always perceive the situations as equivalent (n = 196). These students add probabilities instead of using the product rule in computing a compound probability in a synchronic format but use the correct rule in a diachronic format of the problem.

The conjunction fallacy
Item 6 is devoted to this fallacy, which occurs when it is assumed that specific conditions are more probable than a single general one. When two events can occur separately or together, the conjunction, where they overlap, cannot be more likely than the likelihood of either one of the two individual events. However, people forget this and ascribe a higher likelihood to combined events, erroneously associating representativeness of events with higher probability. Tversky & Kahneman (1982a) identified this bias and coined the term conjunction fallacy for people's unawareness that a compound probability cannot be higher than the probability of each single event.

Confusing the concepts of independence and being mutually exclusive
Item 3 deals with another common error, which is confusing independence with mutually exclusiveness (Sánchez 1996, Truran & Truran 1997): • Two events A and B are exclusive if they cannot happen simultaneously.
• Two events A and B are independent if the outcome of event A has no effect on the outcome of event B.
So, if A and B are mutually exclusive, they cannot be independent. If A and B are independent, they cannot be mutually exclusive. This is not easy for some students to understand.

Frequency versus probabilistic format
Recent research suggests that conditional probability problems are simpler when information is given in natural frequencies, instead of using probabilities, percentages or relative frequencies (Gigerenzer 1994, Gigerenzer & Hoffrage 1995, Sedlmeier 1999, Martignon & Wassner 2002. However, our own research in a small experimental setting showed that the majority of the students in the experiment were able to solve problems proposed in probability format, when they were taught to use a Bayesian table and Excel to facilitate the computations (Díaz & de la Fuente 2006b; n = 75 psychology students). A short test also corroborated an intuitive understanding of the theorem and its utility in updating conditional probabilities. From these results we conclude that is it possible to teach students more productive and generalizable methods of solving conditional probability problems with data given in probabilistic format. We also think this approach is needed in the teaching of many professionals, including psychology students who deal with conditional probability in situations such as evaluation or diagnosis.

METHOD
In spite of the amount of previous studies related to conditional probability reasoning, we found no comprehensive test to globally assess students' understanding and misconceptions on these topics. Following this need, we built a comprehensive instrument that may be used to assess the different biases and misunderstanding related to conditional probability. The construction of the CPR test (Conditional Probability Reasoning) is described in detail in Díaz &de la Fuente (2006a) and(2007a). Complete analysis and results in the test are included in Díaz (2007, chapters 4 to 6).
The aim of this paper is to compare the performance in the test before and after formal instruction in conditional probability and Bayes' theorem as regards to both students' formal knowledge of the topic as well as the aforementioned psychological biases.
In the following sections we compare the performance of 206 students (who took the test, after a teaching unit in conditional probability) with those of another group of 177 students (who took the test before the same teaching unit). The first group was taught conditional probability and Bayes' theorem with the help of tree diagrams, two-way tables, and meaningful examples for about two weeks before they completed the test. The test was given to the students as an activity in the course of data analysis that was similar for both groups of students.
For the comparison to provide conclusive evidence about the research question, the two groups should be similar in all their characteristics, except for the instruction on conditional probability (Thorndike 1991).
• All the students were majoring in psychology in the Universities of Granada (4 different groups of students) and Murcia (two different groups of students); most of them were 18 or 19 year-olds.
• They came from a varied social background with about 60 percent of girls, which is the normal proportion in psychology in Spanish universities.

•
All the students had studied conditional probability at secondary school level (when they were 14-15 years old, which is about four years before the data were collected).
• All the students were following the same statistics course and all of them had studied basic statistics (distributions, averages, spread statistical graphs, bivariate data, correlation and regression) and basic probability (simple experiments, sample space, Laplace rule) before they took the test.
• Even if the students in our sample were majoring in psychology, their social and educational characteristics are not different from other students in an introductory statistics course. We therefore consider our findings could be of interest to statistics lecturers in a wider range of specialties.
Cronbach's coefficients assure the reliability of our results; we got values of α = 0.804 for the groups combined, α = 0.753 for the instruction, and α = 0.762 for the no instruction group. In the next sections we first compare the total score, and use discriminant analysis to compare performance in both groups. We also include a detailed analysis of results in open-ended items.

OVERALL RESULTS
In Table 1 we present the statistical summaries for the total scores in both groups of students and in Figure 1, the box plots for the same variable. The maximum score in the CPR test was 30 points, as there were 22 items (14 multiple choices scored as correct or incorrect; 8 openended items that were scored as incorrect = 0 points; partially correct = 1 point, and completely correct = 2 points). A value t = 8.61 (p < 0.0001) in the t-test of difference in averages for the total score between both groups suggests a significant difference that favours the instruction group.
We can see that while the group with instruction answered more than half of the items correctly, the group without instruction did not reach this level. The lower and upper quartiles, minimum, maximum and median are all also higher in the instruction group while the spread is very similar in both groups, as can be seen in the standard deviations and box plots. This is a clear sign of the test´s discriminant validity, as well as the better results in the instruction group.
Students seem to have made overall progress in their understanding of conditional probability and in applying this concept to solve related problems as a consequence of instruction. Better results in the average score show an overall improvement in the instruction group.
However, in addition to these overall results we were interested to see whether improvement was throughout all the items or if there were some components in understanding conditional probability that were not affected by instruction. Consequently we performed a discriminant analysis to study improvement in the different items after instruction. We used the groups  In Table 2, we present percentages of correct responses to each item in both groups and pvalues for the differences between the percentages of correct responses in the two groups.
Differences always favour the group with instruction and are statistically significant, with exception of the items assessing the conjunction fallacy (item 6), the fallacy of transposed conditional and difference between causal and diagnostic reasoning (item 7), and one of the items measuring the fallacy of time axis (item 8). Note that, even if the difference is statistically significant in the other item assessing the time axis fallacy (item 9b) in fact results for that item are worse after instruction. Consequently, the CPR test may serve to discriminate between students with and without specific instruction in conditional probability if we exclude some part of the items assessing the psychological biases.
There is a substantial improvement in performance in all the open-ended problem solving tasks: • Item 13 (solving a conditional probability problem, in a single experiment), • Item 14 (solving a problem where students should apply the formula for total probability), Other items with good discrimination were:

DETAILED ANALYSIS OF SELECTED OPEN-ENDED ITEMS
Students' responses to the open-ended items are analysed in detail to take into account the degree of completeness in their solution as follows, with correct solutions, partly correct solutions, and wrong responses. Online, the reader may find the formulation of the item (appendix A), the complexity analysis (appendix B) as well of it as well as examples of the other categories of student solutions also by clicking the note symbols. In the annex, the various types of answers are also ordered by items, which may be convenient in studying the effects of each item.

Correct solutions
We consider correct those students who complete all the steps in the problem and provide a correct solution. Some examples follow: • Item 11. In this item students have to define what conditional probability is and provide a correct example. In the following, the student uses a formula in order to define the conditional probability and provides correct examples. "Simple probability P(A) gives the probability for a single event.
Conditional probability gives the probability of A provided that B happened.
Examples: Simple probability: probability that a person smokes; conditional probability; probability that a person smokes provided he is older than 50." • Item 12. Students should describe the restricted sample space in a conditional probability problem. An example of a correct answer is the following: "Gender of the children of a three children family and gender of the children in a three children family if two or more children are male ".
• Item 14. Students are asked to apply the total probability rule. In the next example, the student correctly identifies the events and their probabilities and applies the total probability formula: • Item 15. Students need to solve a conditional probability problem, in case the events are independent. In this example, the student identifies the independence in the successive throwing of the dice: " a the result does not depend on previous results". 2 / 1 s • Item 16. Students have to apply the product rule in the case of independence of events. In this response, the student correctly identifies the data and the independence of experiments and then applies the product rule for independent events: • Item 17. Students have to apply the product rule in the case of dependence of events.
This student correctly identifies the data, the dependence of experiments, and then applies the product rule for dependent events: " 6 0.327 0.36 0.91 ) ( = × = t matters t importan Lying abou P ".

Partly correct solutions
We considered partly correct those students who correctly identify the data in a problem and the type of probability to be computed, but forget some data or make some mistakes in solving the problem. Some examples follow: • Item 11 (Defining conditional probability and giving an example): "Simple probability: when there is a single event.
Conditional probability takes into account two events". This answer is partly correct because in compound probability the two events also intervene; thus it is not clear that the student discriminates between compound and conditional probability. "Single probability is the probability that a variable happens while conditional probability is the probability that a variable happen when you fix a condition". This response is imprecise as conditional probability can also be defined for events and not just for variables.
Because of poor combinatorial reasoning, the student does not complete all possible events in the first part but gives a correct restriction of the sample space in the second.
• Item 13 (Solving a conditional probability problem, in a single experiment): "Possible cases = 36; combinations: 3) (4 4), (3 2), (6 6), the probability is therefore = 2/36 = 1/18" The student correctly identifies the possible cases in the compound experiment and all the favourable cases for the product 12. As none of the numbers can be 6, he makes a correct restriction of the sample space; he considers only 2 favourable cases in applying the Laplace rule. However, he mistakes the denominator in the Laplace rule by considering the favourable cases in the unrestricted sample space; thus he obtai a wrong probability. ns • Item 14 (Solving total probability problem): The following student identifies the number of men and women and adds these numbers to get the total of people smoking. However, in applying the total probability rule he assumes the same proportion of men and women in the population, which is incorrect: "200 people; 50 men smoke and 35 women smoke; a total of 85 among 200 smoke; P(smoking) = 85/200".
• Item 15 (Solving a conditional probability problem, in the case of independence): "10/15". The student correctly identifies independence of events; however he assumes that the die is biased (in spite of what is said in the problem statement). He gives a correct frequentist estimate of probability, but the solution is partly correct, since he assumed the die is biased.
• Item 16 (Solving a product rule problem, in the case of independence): This student correctly identifies the data. But he confuses the rules for computing the probability of the union and intersection of events. Moreover he does not realise that a probability cannot be higher than 1.
• Item 17 (Solving a product rule problem, in case of dependence): In this example the student identifies the data, but he computes a conditional probability instead of a joint probability. • Item 11 (Defining conditional probability and giving an example): "Simple probability: getting a ball from an urn; conditional probability: getting a ball and then a second ball". This response is incorrect because it lacks precision. The student only remembers an experiment given by the teacher concerning sampling balls from an urn, but he does not describe the event in full (for example specifying the colour of the ball, and the composition of the urn). Moreover, the example given for conditional probability corresponds to a compound probability instead of a conditional probability.
The response is incorrect because the student only considers families with two children instead of families with three, and all the events are entical. The student is unable to find the sample space in the compound experiment; instead she adds the number of events in each sample space for the single experiment. Moreover, she computes the simple probability that none of the two terms in the product is 6 rather than a conditional probabil . ity • Item 14 (Solving total probability problem): "130". The student gives a response that is not related to the problem data.
• Item 15 (Solving a conditional probability problem, in the case of independence).
Some students assume that, after getting a majority of odd results in throwing a die, the probability of even result increases. We consider this response as incorrect, since this is an inaccurate perception of equal likelihood and independence; the student is reasoning according to the representativeness heuristics (Kahneman, Slovic, & Tversky 1982). According to this heuristic, when people are asked to judge or calculate the probability that an object or event A belongs to class or process B, probabilities are evaluated by the degree to which A is representative of B, that is, by the degree to which A resembles B.
As 10 odd results out of 15 do not seem to be a representative sample for outcomes coming from a fair die, so students in this category assume there is more likelihood to get an even result after getting 10 odd results out of 15, hich seems more representat e. w iv • Item 16 (Solving a product rule problem, in the case of independence): Some students enumerated all the cases in throwing two dice (36 cases), but were unable to cont ue the proble . in m • Item 17 (Solving a product rule problem, in the case of dependence): "55%". This response has no relationship to the problem data and the students do not provide reasoning for the same.

Summary presentation of the effect of teaching
We present in Table 3 and Figure 2 the percentage of incorrect, partially correct and totally correct solutions. We can see that there are statistically significant differences between the performance of students with and without instruction in all items. Moreover, there is an increase not only in the correct solutions, but also in the partly correct solutions for all the items.
Differences in performance are specially noticeable for solving a total probability problem (51.3% improvement in item 14); product rule problem in the case of dependence (38% improvement in item 17); solving a conditional probability problem, in the case of independence (31.5% improvement in item 15) and solving product rule problem, in the case of independence (23.6% improvement in item 16); that is, in problems that require two or three steps in the solution.
Students have first to assess independence /dependence in the events involved in the problem and then to compute at least a compound or conditional probability. In particular in a total probability problem, the students have to compute two compound probabilities and then apply the addition rule.  We can describe these types of problems as complex or compound conditional probability problems, as opposed to those problems (e.g. item 13) where only one conditional probability has to be computed. The improvement in performance is lower on this item as well as in item 11 on being able to give a correct definition of conditional probability, or in item 12 being able to restrict the sample space.

Exemplary analysis of the influence of teaching
To analyse improvements with instructions in Bayesian problems (item 18), we considered the different steps needed to solve the problem. These steps take into account the mathematical activity involved in identifying the data and in computing the different probabilities one needs to apply in the Bayes' formula (as stated in Martignon & Wassner 2002), as well as possible errors of the students along this process. A more detailed analysis of previous research related to solving Bayesian problems and the students' difficulties in each of the following steps is provided in Díaz & de la Fuente (2007b). 1. Identifying the data of the problem. The first step in solving the problems (Figure 3a) involves discriminating between the single probabilities that the ball is produced by either machine or and the conditional probabilities such as , that machine produces a defect. Students also need to distinguish the conditional probability from its inverse that a defect has been produced by machine . They need to identify this data from the problem statement as well as computing the probabilities of several events such as , i. e. that machine produces a correct ball. Although the student in Figure 3a identifies all these probabilities, he is unable to continue solving the problem. In Figure 3b, the student fails to represent the partition of the population of balls in two samples (those produced by machines and ) and the second division of each of these populations in defective and correct balls. This failure restricts the student in his problem solving process. 2. Identifying the conditional probability to be computed. In a second step, the students should identify what probability needs to be computed; in this case it is an inverse probability. This does not assure the problem is solved as one might confuse the concepts of the formulas for conditional and joint probabilities as in Figure 4, where the student incorrectly applies the product rule. 3. Computing the denominator of Bayes' formula. After identifying the conditional probability to be computed, and recalling Bayes' formula, the students need to compute the numerator and denominator in this formula, which are not directly given in the problem data. The denominator should be computed with the total probability formula. In Figure 5, the student produces a correct tree diagram, identifies all the data, then multiplies the probabilities along the different tree branches, and finally adds some of those joint probabilities to get the probability P(D) that the ball is defective. However, he confuses the numerator in Bayes'

No instruction Instruction Tests Item
formula and gets an incorrect solution that moreover is greater than 1.

Computing the inverse probability (Bayes' theorem).
In fact, the students should synthesize all the above steps and substitute the numerator (joint probability) and denominator (total probability) in the conditional probability formula to get the inverse probability. This will complete the application of Bayes' theorem ( Figure 6).  In Table 4, we present the percentage of students in each group that reached each of the steps in the above process. There is a clear improvement in the second group, since almost half of the students with specific instruction (49%) are able to correctly solve the problem; 57.3% of these students correctly compute the total probability in the numerator. On the contrary, only 4% of students from the group without instruction correctly solve the problem and only 10.7% reach successfully the level of computing the total probability. The number of blank responses or incorrect identification of data is substantially higher in the no instruction group. A chi-square test of independence between groups and classification of responses yields highly significant results (χ 2 = 108.8; p < .0001), which support our hypothesis that performance in Bayesian problems improves with specific instruction in the topic.
These results are consistent to those in the remaining open-ended items and support our conjecture than the main effect of instruction in our teaching experience favours the improvement of performance in complex conditional probability problems. Although there have been previous research studies related to solving Bayesian problems (e.g. Eddy 1982, Gigerenzer & Hoffrage 1995, Sedlmeier 1999, or Martignon & Wassner 2002, those authors are mainly interested in comparing performance when the data are given in percentages or natural frequencies. The steps above in solving Bayesian problems are inspired by Martignon & Wassner (2002). However, none of the previous authors undertook a systematic analysis of the types of errors involved in each step or provided empirical evidence of improvement in the different steps with instruction in problems given in probabilistic format.

IMPLICATIONS FOR TEACHING CONDITIONAL PROBABILITY
Our results show that students' performance in problem solving ability and in formal understanding of conditional probability improves with the specific instruction provided where we used clear examples in topics of interest for the students as well as tree diagrams as an instructional resource. We observed higher scores in the instruction group and higher percentages of correct solutions to all items related to formal mathematical knowledge.
A deeper analysis of open-ended items also suggests a clear difference in the percentages of correct and partially correct solutions in all these items. Students with instruction perform better in defining both simple and conditional probability, enumerating the sample space, and solving two-step or three-step conditional problems including total probability and Bayesian problems. These results suggest the value in including a theme related to conditional probability and Bayes' theorem in the training of undergraduates given the relevance of these topics in statistical inference and in making professional decisions.
However, our results also reflect the complex relationship between probabilistic understanding and intuition as analyzed in Borovcnik, Bentz, & Kapadia (1991), or Borovcnik & Peard (1996). Most of the biases described in the literature were widespread in both groups of students and some of them did not improve with instruction. While the base rate fallacy and the confusion between exclusive and independent events seemed to disappear in a part of the instruction group, the percentages of students showing the conjunction fallacy, the confusion between causality and conditionality, and the fallacy of time axis were similar in both groups.
Consequently we need to consider these errors and difficulties when we assess students' probabilistic reasoning if we want to get a more comprehensive idea of students' capabilities and conceptions. These difficulties also need to be recognised when organising our teaching, which should also emphasise the probabilistic reasoning and biases, instead of merely concentrating on algorithmic aspects of computing conditional probabilities and on the definitions.
These results are also consistent with the analysis by Batanero, Henry, & Parzysz (2005) who trace an analogous lack of intuition in the historical development of the discipline. Even though independence and conditional probability are informally used from the very beginning of the study of chance games, it is not until the middle of the 18th century that these two concepts were made explicit in the mathematical theory. Furthermore, von Mises (1928Mises ( , 1952 criticized the formal modern definition of independence because it is not intuitive at all. It is unsurprising that these historical difficulties recur in the students' learning of probability.
Consequently, our research reflects the need for reinforcing the study of conditional probability in teaching data analysis at university level. Moreover, it yields strong reasons for a change of teaching approaches. One simple way to convince students that their ideas or solutions to probability problems are wrong is to confront their ideas with experiments. The items included in the CPR test may help instructors either to assess the extent of these biases among their students or to build up didactic situations where students are confronted with their misconceptions about conditional probability.
For example, if students are confused with the solution to item 9b, we might organise a classroom experiment, where students working in pairs repeat many times the trial described in the item and record the results. They then can compare those trials where the second draw was a white marble and count in how many of them the first draw was also white. Experimentally, they can estimate and compare the result to the probability involved in the problem. Similar experiences can be organised as regards the other biases described in this paper.
While technical procedures can assist stochastic thinking, they need to be accompanied by a mature, critical stance which is often difficult to promote with standard teaching. This critical stance is essential for making interpretations which go beyond intuitive approaches; it is especially important because research has shown that people retain and use invalid probabilistic intuitions in many everyday and professional decisions (e.g. Kahneman & Tversky 1982b).
As suggested by Nisbett & Ross (1980, p. 280-1), students should be "given greater motivation to attend closely to the nature of the inferential tasks that they perform and the quality of their performance" and consequently "statistics should be taught in conjunction with material on intuitive strategies and inferential errors".

ANNEX -ALL APPENDICES AND ORDERED TABLES
For the ordered tables follow this link. Suppose we select at random a person from this group: a. What is the probability that the person had a heart attack b. What is the probability that the person had a heart attack and, at the same time is older than 55? c. When the person is older than 55, what is the probability that he/she had a heart attack? d. When the person had a heart attack, what is the probability of being older than 55? Item 2. (Tversky & Kahneman 1982a). A witness sees a crime involving a taxi in a city. The witness says that the taxi is blue. It is known from previous research that witnesses are correct 80% of the time when making such statements. The police also know that 15% of the taxis in the city are blue, the other 85% being green. What is the probability that a blue taxi was involved in the crime? a.  (Sánchez 1996). A standard deck of playing cards has 52 cards. There are four suits (clubs, diamonds, hearts, and spades), each of which has thirteen cards (2,... ,9,10,Jack,Queen,King,Ace). We pick a card up at random. Let A be the event "getting diamonds" and B the event "getting a Queen". Are events A and B independent? a. A and B are not independent, since there is the Queen of diamonds. b. A and B are only then independent when we first get a card to see if it is a diamond, return the card to the pack and then get a second card to see if it is a Queen. c. A and B are independent, since P(Queen of diamonds) = P(Queen) × P(diamonds). d. The events A and B are not independent, since P(Queen | diamonds) P(Queen).

APPENDIX A. ENGLISH TRANSLATION OF THE CPR TEST
≠ Item 4. There are four lamps in a box, two of which are defective. We pick up two lamps at random from the box, one after the other, without replacement. Given that the first lamp is defective, which answer is true?
a. The second lamp is more likely to be defective. b. The second lamp is most likely to be correct. c. The probabilities for the second lamp being either correct or defective are the same.
Item 5. (Eddy 1982). 10.3 % of women in a given city have a positive mammogram. The probability that a woman in this city has both positive mammogram and breast cancer is 0.8%. A mammogram given to a woman taken at random in this population was positive. What is the probability that she actually has breast cancer?  Tversky & Kahneman 1982 b). Suppose a tennis player reaches the Roland Garros final in 2005. He has to win 3 out of 5 sets to win the final. Which of the following two events is more likely or are they all equally likely? a. The player will win the first set. b. The player will win the first set but lose the match. c. Both events a. and b. are equally likely.
Item 7. (Pollatsek, et al. 1987). A cancer test is administered to all the residents in a large city. A positive result is indicative of cancer and a negative result of no cancer. Which of the following results is more likely or are they all equally likely? a. A person has in fact cancer supposed that he got a positive result. b. To have a positive test supposed that the person has cancer. c. The two events are equally likely.
Item 8. (Ojeda 1996).We throw a ball into the entrance E of a machine (see the figure). If the ball leaves the system through exit R, what is the probability that it passed through channel I? a. 1/2 b. 1/3 c. 2/3 d. Cannot be computed Item 9. (Falk 1986 Item 10. An urn contains one blue and two red marbles. We pick two marbles at random, one after the other without replacement. Which of the events below is more likely or are they equally likely? a. Getting two red marbles. b. The first marble is red and the second is blue c. The two events a) and b) are equally likely.
Item 11. Explain in your own words what a simple and a conditional probability are and provide an example. Item 13. In throwing two dice the product of the two numbers is 12.
What is the probability that none of the two numbers is a six (we take the order of the numbers into account)?
Item 14. 60% of the population in a city are men, 40% women. 50% of the men and 25% of the women smoke. We select a person from the city at random; what is the probability that this person is a smoker?
Item 15. A person throws a die and writes down the result (odd or even). It is a fair die (that is, all the numbers are equally likely). These are the results after 15 throws: Odd, even, even, odd, odd, even, odd, odd, odd, odd, even, even, odd, odd, odd. The person throws once more. What is the probability of getting an odd number this time?
Item 16. A group of students in a school take a test in mathematics and one in English. 80% of the students pass the mathematics test and 70% of the students pass the English test. Assuming that students' scores on the two tests are independent, what is the probability that a student passes both tests (mathematics and English)?
Item 17. According to a recent survey, 91% of the population in a city do lie and 36% of those lie about important matters. If we pick a person at random from this city, what is the probability that the person lies about important matters? Item 18. (Totohasina 1982). Two machines M 1 and M 2 produce balls. Machine M 1 produces 40 % and M 2 60% of balls. 5% of the balls produced by M 1 and 1% of those produced by M 2 are defective. We take a ball at random and it is defective. What is the probability that that ball was produced by machine M 1 ? APPENDIX B.

SOLUTION AND COMPLEXITY ANALYSIS FOR OPEN-ENDED PROBLEMS
To detailed analysis of items in article Item 11. Explain in your own words what a simple and a conditional probability are and provide an example.
An intuitive definition of conditional probability could be: "Conditional probability is the probability that an event A happens, given the occurrence of another event B". Another intuitive definition of conditional probability could be "number of joint occurrences of A and B, divided by the number of times that B has happened". A more formal definition is as follows: In this item we assess the recognition that a conditional probability involves the restriction of the sample space. The correct answer in part a) requires the enumeration of all elements of the sample space. In part b), the student should reduce the sample space by the given condition as follows: Some students might not succeed to complete the enumeration in part a) and omit some cases; for example they might consider order is irrelevant. We consider partly correct those responses where the sample space E is not fully completed (for example, if the order is not considered), while the sample space * E is correctly reduced from E.
Item 13. In throwing two dice the product of the two numbers is 12. What is the probability that none of the two numbers is a six (we take the order of the numbers into account)?
In this item we assess the student's competence to compute a conditional probability in the experiment "throwing two dice". In order to solve the problem correctly, the student must first define the correct sample space (36 possible cases), and then identify the cases where the product is 12 that define the restricted sample space: {(2, 6), (3, 4), (4, 3), (6, 2)}. Among these possible cases, only (3, 4) and (4, 3) have no six in it. Therefore the requested probability is equal to ½ since there are two favourable cases.

Item 14.
Of the population in a city 60% are men; 40% are women. 50% of the men and 25% of the women smoke. We select a person from the city at random; what is the probability that this person is a smoker?
To solve this problem, the student could directly apply the total probability formula. Let S denote the event "the person smokes", M "the person is male" and F "the person is female", then: The student could also use a two way table to solve this problem. He could use proportional reasoning to compute the data in the different cells. Once the table cells are filled, the problem is reduced to a simple probability problem that can be solved with the Laplace's rule: 40 people smoke out of 100 people; the probability for a person taken at random smokes is 40/100.

Does smoke
Doesn't smoke Total   Male  30  30  60  Female  10  30  40  Total  40 60 100 Item 15. A person throws a die and writes down the result (odd or even). It is a fair die (that is, all the numbers are equally likely). These are the results after 15 throws: Odd, even, even, odd, odd, even, odd, odd, odd, odd, even, even, odd, odd, odd. The person throws once more. What is the probability to get an odd number this time?
To correctly solve this problem, the student should understand that the previous occurrences do not affect the probability for the next outcome, since the successive throwing are independent. Therefore the probability to get an odd number in the next throwing is again ½. There are two possible cases (odd or even); only one of them (odd) is favourable. We consider it partly correct if the student provides a frequentist estimate for the probability (10/15) because he might have not assumed that the dice was fair. An incorrect solution is when students reason according the "gambler's fallacy" and think the probability of getting an odd number is now 5/15, since they expect the results should balance in the short run.
Item 16. A group of students in a school take a test in mathematics and one in English. 80% of the students pass the mathematics test and 70% of the students pass the English test. Assuming that students' scores on the two tests are independent, what is the probability that a student passes both tests (mathematics and English)?
In order to correctly solve this item, the students should apply the product rule, for the case of independent events. Let M denote the event "the student pass the mathematics test" and E the event "the student passes the English test". Then 0.56 0.7 0.8 ) ( Item 17. According to a recent survey, 91% of the population in a city do lie and 36% of those lie about important matters. If we pick a person at random from this city, what is the probability that the person lies about important matters? This item assesses the student's competence to solve a product rule problem when the two events are dependent. Let L be the event "the person lies", and I the event "the person lies about an important matter", then, 327 0. . Item 18. (Totohasina 1992). Two machines M 1 and M 2 produce balls. Machine M 1 produces 40 % and M 2 60% of balls. 5% of the balls produced by M 1 and 1% of those produced by M 2 are defective. We take a ball at random and it is defective. What is the probability that that ball was produced by machine M 1 ?
In this item, we assess the student's competence to solve problems that involve Bayes' theorem. Let D the event "the ball is defective·, M 1 the event "the ball is produced by machine M 1 and M 2 the event "the ball is produced by machine M 2 then: The probability can be computed by using the formula of the total probability: ) (D P