Open Type Tasks in Mathematics as a Tool for Students’ Meta-Subject Results Assessment

The relevance of the present study is due to the tasks of assessing the quality of secondary education, which determine students’ subject knowledge along with their personal achievements, and where meta-subject skills define not only the level of educational results achieved, but also the prospects for the future high-quality training for a school graduate at any age. Thus, the purpose of the study conducted is to create approaches for making the system of tasks together with the criteria of their assessment to determine the level of students’ meta-subject results. The leading methods applied are modeling systems of open type tasks, with mathematical content and systematic analysis of large samples of experimental data, based on the evaluation of the two-point scale four parameters: the optimality of students’ proposed ideas, efficiency of reasoning, originality of their answer and development degree of their solutions. A pilot study conducted since 2008 has applied open tasks with mathematical content and the criteria for their assessment formed the approach to determine the level of high school students’ meta-subject results, expressed quantitatively in the form of an integrated assessment of the relative character – meta-subject intelligence quotient. Practical use of the intelligence quotient enables the making of accurate calculations for each age group in order to show the level of students’ meta-subject results. This in turn may determine the future direction of students’ individual development, and to ensure their transition to a higher level of meta-subject skills and, consequently, higher quality of education.


Relevance of the study
Quality control improvement and quality management have become the areas of modernization of the Russian educational system. In the government-approved development priorities of the educational system of the Russian Federation, it is noted that there is a need to form a national system for evaluating the quality of education received by each citizen (Government of the Russian Federation, 2008). However, among Russian authors working on this problem (Bolotov, 2005;Zheleznova, 2014;Kolomiets, 2003;Subbeto, 2004;Tretyakov, 2002), there is no common point of view on the number of "quality criteria" or their quality indicators. According to Bolotov, "evaluation of the quality of education implies the assessment of the quality of educational achievements of students and the evaluation of the quality of the educational process. The quality of education is understood as an integral characteristic of the educational system, which reflects the extent of compliance of the actual achieved educational results with both the regulatory requirements and social and personal expectations" (Bolotov, 2014). In our opinion, building the national system of assessment of educational quality according to priority directions of development of the educational system of the Russian Federation should be based on two aspects: 1) experiences from international research on evaluating the quality of education, and 2) normative regulation of intended educational results achieved by school students.

International scientific research on educational quality assessment
Today, international research groups such as Program for International Student Assessment (PISA), Trends in Mathematics and Science Study (TIMSS), and International Assessment of Educational Progress (IAEP), study the problem of educational quality assessment. The research groups determine the quality of education as students' mastery of reading, mathematical, scientific, information and computer literacy. Students from educational institutions of Russia were actively involved in these studies, but the results of independent educational quality assessment were considered with indifference. At the same time, the foreign experience in this field can be hardly overestimated, for example, France (research group PISA), the USA (IAEP), Netherlands (TIMSS), and Germany (PIRLS) (Site of Center for Educational Quality, 2014). Considering the conceptual model of one of the foreign studies ( Figure 1), one can see that there are common approaches to the assessment of educational achievement on the subjects and common cognitive activities, which students should show while solving tasks (The main results of an international study of quality of mathematical and natural science education TIMSS-2011TIMSS- , 2013. In our opinion, the Russian educational system fixes typical cognitive activities for task solution.

The regulatory framework of intended educational results of high school students
The present federal state educational standards of general education specify the requirements for the learning results of basic educational programs of primary, secondary and secondary general education. These requirements should be taken into account while developing the system of educational quality assessment. The current educational standards of the new generation strengthens their focus on the educational results as a backbone component of standards. These standards are designed to resolve the key contradiction of modern education -contradiction between knowledge and abilities of each student to master only a part of this knowledge. Therefore, the federal state educational standards implies the focus on students' achievements, both in subject and meta-subject educational results (Russian Academy of Education, 2008). The standards specify requirements for personal, meta-subject and subject results of students, who have mastered the basic educational program of general education. Standards methodically worked out within the framework of general education are applied to assess subject results (i.e. evaluation of students' knowledge and skills). Psychological and social pedagogic diagnostic methods assess the personal results. These methods are methodologically worked out and repeatedly improved. Meta-subject results are a relatively new phenomena in the Russian educational system, so the evaluation system of metasubject results has neither been worked out, nor put into practice.

Methods of the study
The study applies the following methods: analysis of the regulatory documents; analysis of psychological, pedagogic and methodological sources; analysis of learning products; method of mental experiment; forecasting; synthesizing of facts and concepts; modeling, project method, method of expert judgment; analysis of the educational activities results; study and generalization of the experience of application of open type tasks with mathematical content, diagnostic methods; and pedagogical experiment.
The experimental work involved students of all age groups from secondary schools in all regions of the Russian Federation and some neighboring countries.
The study was conducted in three stages: 1) The "preparatory" stage. Analysis of the current state of the investigated problem in pedagogical theory and practice, and the development of research methodology; 2) The "main" stage. Development and implementation of open type tasks with mathematical content; the systematic analysis of large samples of the experimental data; verification of the methodology effectiveness for assessing the level of students' meta-subject skills formation; 3) The "final" stage. Systematization, interpretation and synthesis of the research results; refinement of the theoretical conclusions; processing and registration of the obtained results.

Meta-subject results and the problem of their assessment
We can conclude that the international experience in educational quality assessment and the requirements of the educational standards narrow the problem of educational quality assessment to assessment of meta-subject results.
Meta-subject results include the level of students' knowledge of interdisciplinary concepts and universal training activities (regulatory, cognitive and communicative); students' ability to use the knowledge in cognitive and social practice; autonomy in planning and implementation of educational activity and organization of educational cooperation with teachers and contemporaries; the ability to build individual educational trajectory; possession of learning-research, project and social skills (Kovalenko & Nikitin, 2013).
Thus, meta-subject achievements in general education are based on interdisciplinary concepts and universal learning activities. Universal learning activities can be evaluated as foreign research by recording several cognitive activities. Only "non-subject" tasks can implement the assessment of interdisciplinary concepts. This assessment requires the special tasks, which are not reduced to the subject knowledge.

Open type tasks as a tool of meta-subject results assessment
Below we describe the task of the series "Mathematics", as proposed in the framework of international research, TIMSS.
Task 1. Here are two shapes. Describe their similarities and differences ( Figure 2). The problem has a vague condition that does not determine the exact sequence of actions. Such tasks are generally called open type tasks (Gorev & Utemov, 2011). In our opinion, such tasks can act as a tool to assess the level of formation of metasubject skills (Zinovkina, Gareev, Gorev, & Utemov, 2013).

Figure 2. Illustration of the task proposed by TIMSS
Below we consider the difference between open type and closed type tasks .
Closed type tasks provide clear and unambiguous interpretation of problem conditions. As a result, the task usually has the one correct solution. Such tasks do not give the student an opportunity to form an integral meta-subject quality (Figure 3).
The following task is an example of the closed type task: Task 2. To paint 1 m 2 of ceiling you need 240 grams of paint. The paint is sold in cans weighing 2.5 kg each. What is the minimum amount of cans you need to buy to paint the 50 m 2 ceiling?
Tasks of this type are didactically valuable in practicing any particular solution method or in acquaintance with new material. These tasks are based on ready facts and knowledge; for their solution students obtain comprehensive information about the solution method, conditions, and sometimes the answer, the accuracy of which is not doubtful.
Below we consider the task of open type. The characteristics of its basic parameters are determined in Figure 4.
The following task is an example of the open type task: Task 3.You probably noticed that the larger pits and ditches that have to be overcome by a vehicle, the greater the diameter of the wheels. For example, a tractor that works the fields. Racing cars, on the other hand, have small wheels in order to drive faster on the perfect road. Is it possible to come up with a universal wheel, which would change its size depending on the condition of the road?
Open type tasks have indistinct conditions, hence it is unclear how to act, or what to use in the solution. However, the desired result is clear. These tasks require a variety of solutions that are not straight; moreover you have to overcome obstacles encountered. There are a lot of options for solutions, but there is no concept of the  right one, because it should already have been applied in order to achieve the desired results, or not. Open type tasks give possibility to apply standard knowledge in a nonstandard situation. Carrying out the tasks the student can demonstrate the ability of logic and abstract thinking that is the ability to categorize, summarize and draw analogies, to predict the result applying intuition, imagination, and fantasy. The most important is that these tasks contribute to the formation of an integral meta-subject quality.
The division of tasks into closed and open types has vague outlines; the task is not always accurately referred to the problem of a particular type. For this reason, it is advisable to consider partially open type tasks. In school practice, partially open type tasks are presented as creative tasks. In these tasks, condition, solution and answer (together or separately) have closed characters. Thus, partially open type tasks have boundary position and they are often used on the reproductive level of learning educational material.

Classification of partially open type tasks with mathematical content
Below we give the example of classification of partially open type tasks, determined in the process of analysis of educational sources (Table 1) (Mikhaylov, Gorev, & Utemov, 2014).  The condition can be interpreted ambiguously, defining not only internal angles, but also adjacent angles (acute or obtuse). Therefore, there are several variants of answers. But there is only one method of solution and clearly is defined in the formulation of the problem

Closed answer (it is clear what the result is)
On the teacher's question "What can you draw using three circles?" Peter replied: "I can draw a snowman", Bob exclaimed: "A plate of berries!" and Sergei muttered, "I can draw a propeller plane". Try to draw these objects using only three circles It's clear what the result is. However, the formulation does not provide an unambiguous method of solving it. The condition does not give full information about how the objects were painted (what size of circles they used, if those circles crossed, and so on) Closed condition and answer (it is clear what and how to search, but there are multiple answers) Normal cube scan requires strips of paper of not less than three square faces width. Is it possible to scan the cube using strips of paper of three faces width?
There is enough information for a solution. The only method of solution is the thought experiment of cube scan. There are exactly 10 answers (up to a motion)

Closed condition and answer (it is clear what to search and what to have as a result, but there are several methods of solution)
According to Archimedes' principle, any object not being at equilibrium is destined to float or sink in the liquid. However, there is a kind of liquid where an amazing thing happens, if you throw ordinary egg in it -the egg will periodically emerge and sink. What is the trick and what kind of liquid is it?
The answer is based on the use of chemical reactions, for example with hydrochloric acid. But in any case, the answer relies on the idea of using gas bubbles

Criteria for open type tasks assessment
To determine the level of formation of meta-subject character of the results, approved criteria for evaluating educational open type tasks can be used, obtained on the basis of creativity indicators' generalization by Guilford and Torrance (Guilford, 1967;Torrance, 1964) (Table 2) Thus, assessment criteria of the open type tasks define an eight-point scale. It characterizes the level of students' meta-subject results manifestation in mastering the programs.
The open type tasks system consists of six tasks directed to various types of cognitive activity. Evaluation of the open type tasks solutions may be an indicator of quantitative index of students' meta-subject achievements formation. The accuracy of estimating the level of students' meta-subject manifestation may be taken into account only according to numerous practices of carefully selected open type tasks. The total score of the six tasks can be considered as the final level of creativity.

Meta-subject intelligence quotient, based on the evaluation criteria of open type tasks
We analyzed the results of our own research conducted in 2008 within the heuristic competitions "Owlet" and "Breakthrough" and other forms of work based on open type tasks. The investigation of more than 200,000 students from various Russian regions and abroad shows that such systems may be considered as a mean of meta-subject results' evaluation in mastering basic educational programs of general education. To adjust the results, you can use the integrated assessment of the relative character -meta-subject intelligence quotient.
By analogy with the intelligence tests (such as IQ), intelligence quotient represents the ratio of "mental age" to the actual chronological age of the subject. Calculation of the intelligence quotient is carried out separately for each age group according to the common division of participants' scores. About 50% of participants should have the confidence interval 90-110 test scores (up to equal points); about 25% of participants should have the result below 90 test scores and another 25% of participants -higher than 110 test scores showed (up to equal points) ( Figure 5).
It should be noted that according to the description of the methodology of foreign studies, scales on specific informative categories and types of cognitive activity also have comparable parameters: average value of the average scores of all countries participating in the study is taken as 500, with a standard deviation 100.

Status (Is the solution detailed, or at the level of ideas?) 2
The proposed solution will give clear understanding on how to achieve results The solution has a particular method providing a quite capacious, clear and optimal result Solution is original, and is found with <5% of respondents Solution is reasoned clearly and correctly and all actions are justified 1 All in all, the solution process is clear and it is possible to achieve the results, but some elements of the solution are not thought out or vaguely explained The solution is optimal, but some aspects of the solution process can be greatly simplified The solution is rarely found in the answers: between 5-10% of respondents The decision is on the level of ideas that might be brought to reasonable justification and completion 0 According to the solution, it is not clear how to achieve the desired result The solution process is too cumbersome; the use of many techniques are unjustifiable The solution is standard and provided by more than 10% of respondents The solution process is not represented or is vague So, basing on the analysis of 33,840 participants' works of competitions held in 2015, the trends in the assessment of meta-subject intelligence quotient were identified. As a result, the participants were divided into four groups. The number of participants who fell into the first group, each parallel is the maximum, but not exceeding the level of 10% of the total number of participants in this parallel up to equal points. The number of participants who fell into the second group is determined by the intelligence quotient: this group includes participants with test scores above the level of 110 points, but not included in the first group. The number of participants belonging to the third group is determined by confidence interval of sample according to intelligence quotient of 90-110 points. The other participants are in the fourth group of points (Table 3).
Thus, the integration result of the open type tasks is closely linked to the development of students' meta-subject results of basic educational programs. Metasubject intelligence quotient may be used for meta-subject assessment, reflecting the general human ability to express the cognitive activity of the subject and his ability to assimilate new knowledge, action, complex forms of activity. In this regard, we believe that the use of meta-subject intelligence quotient in assessing learning outcomes will enhance the effectiveness of education and the creation of conditions for the formation of subject teaching models in secondary school. We believe that further research can be continued towards the development of open type tasks with the

DISCUSSIONS
Assessment of the quality of education through the evaluation of cognitive activities for quite a long time has been carried out at the international level. Summarized data of international studies' areas to assess the quality of education are given in Table 4.
Russian scholars working on the problem include Bolotov (2005), Zheleznova (2014), Kolomiets (2003), Subbeto (2004), andTretyakov (2002). However, analysis of scientific papers on the topic showed that modern views on the evaluation of the quality of education through students' achievement of meta-subject results have not been thoroughly studied, as the authors do not offer mechanisms for assessing students' achievements.

CONCLUSION
It was found that the result of the pilot study (held since 2008) using open type tasks with mathematical content and the criteria for their assessment is the new approach to determination of the level of students' achievement of meta-subject results, expressed quantitatively in the form of integrated assessment of the relative character -meta-subject intelligence quotient. It is based on the criteria of open-type tasks estimation (optimality of the proposed ideas of students, effectiveness of their reasoning, originality of their answer and degree of elaboration of their decisions). Specially selected systems which indicate the level of students' achievement of metasubject results with reliable accuracy of normal distribution for each age group that in turn may determine the future direction of development of an individual student to ensure his transition to a higher level of meta-subject skills and, consequently, better quality of education.

RECOMMENDATIONS
The paper may be useful for teachers, mentors and tutors, striving to improve the level of students' achievement of meta-subject results, to evaluate and further adjust individual educational direction of students' development at the level of their meta- Control measuring materials to assess the training of primary school leavers Mathematics, Russian language, Nature Study subject skills assimilation. Taking into account the results of this study, we can identify a number of scientific issues and prospective directions of further consideration: the deepening and widening of certain aspects discussed in the paper related to the accumulation of psychological and pedagogical potential of open type tasks in training; the development of scientific-methodological support for widespread use of meta-subject intelligence quotient to evaluate the level of students' achievement of meta-subject results.