Abstract

Diagnostic assessment test for misconceptions (ATM) help teachers identifying misconceptions of students and help students to enhance conceptual understandings. The main purpose of the study was to develop a diagnostic assessment test for misconceptions (ATM) in Mathematics at the elementary school level. At the first stage, the content was defined, and a table of specifications was made. Students’ misconceptions from previous literature were identified, and then a two-tier MCQs type misconceptions test was developed. Validity was ensured by school teachers, educationists and experts of the field. For pilot testing, the test was conducted among 60 students of three schools and item analysis was employed. Item difficulty and item discrimination through item analysis was drawn to standardize the test. Test-retest reliability was also measured by pilot testing. After testing the validity and reliability, thirty items remained in the diagnostic test. The final form of the test enables the Mathematics teachers to understand the weak areas of students.

Key Words

Diagnostic Assessment Test, Two tiers, Misconceptions, Mathematics, Elementary    School Level

Introduction

Mathematics mainly contains symbols to represent numbers (Egodawatte 2011; Moodley 2014); certain rules and theories are used to manipulate these symbols (Matuku, 2017; Ncube, 2016; Mangorsi, 2013) elaborates that algebra is further comprised of algebraic expressions, polynomial equations and properties, it is important to understand them conceptually with proper functionality to form algebraic equations. Owusu(2015) stated that learners have some prior knowledge from their surroundings and everyday experiences when they come to the classroom. Amirali and Halai (2010); Ali (2011) supported this idea and argued that everyday knowledge of mathematics has a role in learning and helps in solving problems and misconceptions. Luneta (2015); Matuku (2017) also contributed that errors and misconceptions are somehow related but mostly distinguish from each other (Egodawatte, 2011). Researchers (Aygor & Ozdag 2012; Mdaka, 2011) further added that when students make mistakes, they put some blunders and mistaken ideas formulated on false facts that are directly related to their learning of algebra. Different errors, omissions and misconceptions in algebra are due to certain reasons some of them are inability to understand in-depth concepts of algebra, lack of knowledge to recollect and proper application of algebraic rules and rote learning while performing different algebraic tasks (Bohlmann et al., 2017; Iddrisu et al., 2017).

Egodawette (2011); Pournara et al. (2016) argued that if learners have prior knowledge about algebra from their elementary level, then they may be able to deal with misconceptions at the secondary and higher secondary level (Luneta, 2015; Gumpo, 2015). This concern is stated by different researchers to represent student’s understandings. Misconceptions are defined by McAfee (2018) as general believes which are contradicted through sound evidence. All researchers are convinced that for better understandings of students, different scientific theories are need of the time. Regardless of the difference of opinions, all the terms are focused on the difference between the ideas students have in their minds and the concepts by certain scientific theories (Matuku, 2017). In most of the studies, the main purpose of research is to understand the misconceptions of students that slow down their learning. So identification of misconceptions in a considerable way becomes the first step towards a solution. The current study focused on developing a diagnostic test for misconception. Dowing (2006) presents a systemic way of developing a diagnostic test to probe these misconceptions. There is a relation between diagnosing ideas by a test and students knowledge explored by the researcher (Gurel et al., 2015). 

Most researchers put their focus on exploring conceptions of individuals. Further emphasis was on if a doctor has knowledge of only two or three diseases, he/she will be able to diagnose only them in his patients and will be failed to diagnose the rest of the diseases despite available technical support. If the doctor’s diagnosis is correct, a prescription may work; on the other hand, if the diagnosis is wrong, it may have fatal effects on the patient. This example is quite relatable in that the studies and researches are impactful for mathematics and science education research if valid and reliable diagnostic methods are used for misconceptions. Diagnostic tests are characterized as an effective tool which mainly concerned with tenaciously repeated difficulties in learning which are the main reasons behind learning difficulties among students (Gronlund, 1981). Apparently, these instruments clearly identified the difference between what is the learning of our students is what we are expecting for their learning. This study is mainly focused on the development of diagnostic assessment in mathematics and presents an overview of the diagnostic assessment test for misconception (ATM) in mathematics. 

Literature Review

Colin et al. (2002) explore that in order to determine students’ understanding, open-ended questions are commonly used in mathematics education. This may help them in writing their views and idea in a more clear way. They took proper time to think, but evaluation of results may be tiresome for the researcher.  To reduce critical aspects of interviews and open-ended questions as discussed above, another method may be used to determine learner’s perception Downing (2006), which is a diagnostic multiple-choice test, in which immediate results can be calculated and is applicable for all subjects. Different studies are being reported in the literature for the diagnoses of misconception through multiple-choice tests (Caleon & Subramaniam, 2010). To provide ease to teachers, diagnostic test is an easy and convenient way which is more valid and reliable, easy to monitor, convenient to score, and is a best way to determine the better understanding of students in education. After diagnosis of misconceptions of students teacher may be able to overcome them by using suitable approaches. Some studies which are reported against it also reported draw backs of multiple choice questions. 

Some of the limitations are, mostly student solve multiple choice questions through guess which lead to low reliability of a test. The selection of choices may not increases students in-depth conceptual understanding of ideas among students. Students are mostly bound to fill the answers provided to them; they are not able to express their own ideas. Sometimes it is not possible to provide good options for choices. By all these evidences, it may assume that ordinary multiple-choice questions are not a good way to determine correct reasoning and overestimation of students result may be calculated( Chang et al., 2010; Ery?lmaz, 2010; Kaltakç?, 2012; Pe?man & Ery?lmaz, 2010; Caleon & Subramaniam, 2010). After in-depth research in this field, some studies Gurel et al., (2015) found supporting some specially designed test which is being used for the calculation of misconception among the learners. Several studies are conducted by Widjaja et al., (2008); Wiliam (2009), Bukula (2010) stated that diagnostic analysis helped teachers to provide an opportunity to their earners to get knowledge with better understandings.

Diagnostic tests are designed to reveal specific misconceptions of individual students about a specific topic. A diagnostic test is defined by Wiliam (2009), “If formative assessment delivers additional information that addresses students difficulty in a precise manner”. 

Wiliam (2009) stated that diagnostic test is mostly conducted in the form of a carefully designed test.  Steinle and Stacey (2008) broadly elaborate that teachers plan this test by keeping in mind students misconceptions. A poorly designed may result in false conclusions, so teachers must pay attention to the construction of test so that hidden misconceptions and shortcomings are handled properly. The teacher must have in-depth knowledge about the strength and weakness of their diagnostic test, so if the analysis does not reflect the expectations of teachers, then they may be able to join it with other means of hidden misconceptions (Widjaja et al., 2010). The results of the diagnostic assessment test must be shown to teachers Baturo (, 2004) about how learners think mathematically; it totally depends upon the construction of diagnostic test how fluent students are about their understanding.

Linsell et al. (2012) ; Shute (2008); Steinle and Stacey(2012) further elaborates that diagnosing testing is very helpful for teachers to reveal strengths and weaknesses of their students, it may help them in panning of their lesion in future, which may lead to better understandings of their students. Through formative assessments, teachers may have more precise and clear views about understandings of their students, which bring positive change in their teaching practice. A study on diagnosed test was conducted in New Zealand by Linsell et al. (2012) on the learners of secondary level, which results in an in-depth understanding of their students. This ultimately helped teachers in bringing commendable changes in their teaching methods by getting detailed analysis of mathematical thinking of their students. This empowered teachers to facilitate their learners in a better way to improve their algebraic knowledge. Kettedin-Geller and Yovanoff (2009) further admired that diagnostic test may play a role bridged in fulfilling gaps between teachers teaching skills and students learning abilities. Further, another research was conducted in the United States of America where Bukula (2010) discovered misconceptions among students of grade 7th, which consequently helped her in the reformulation of her teaching skills which clearly address those difficulties and problems faced by her students in their learning. Similarly, other researchers also formulated the SMART diagnostic techniques, which apparently changed their lesson plans and teaching style to overcome the shortcoming, which is a clear direction for teachers that misconceptions can be removed as most of us try to overcome and ignore them ( Steinle, 2004; Steinle & Stacey,2012; Linsell et al., 2012).

A study was conducted to elaborate different methods for diagnosis of misconception in mathematic in which 53% were interviews, 34% open-ended questions, 32% were a multiple-choice test, multiple-tier test contributed as 13%, and 9 % were other types of diagnostic tests. Interviews have a key role in several diagnostic methods for misconception because through them, in-depth investigation can be done, and there is a possibility of a complete description of the cognitive structure of learners (Gurel et al., 2015).  Several studies are reported to find out what peoples’ mind have, how they think and what they feel about something as their perception may be clearly identified through interviews (Frankel & Wallen, 2000). Therefore, interviews are considered as one of the best sources to reveal learners view on the possibility of misconception. Although interviews have the advantage of in-depth information and better flexibility but a huge amount of time is required to enhance better generalizability. For a researcher to conduct the interview in a better way, training is also required. Moreover, if the interview has bias opinion, it may affect the results; also, data analysis is quite critical and difficult (Duit, 2004; Hammer et al., 2005; Tongchai et al., 2009; Adadan & Savasci, 2012).

In the light of the above literature researcher intended to investigate “Development of diagnostic assessment test for misconceptions in Mathematics at elementary school level”.




Purpose of the Study

The main objective of the study was to explain the methodology for developing a diagnostic test to identify students’ misconceptions in selected content areas of elementary school level mathematics. Four main areas of algebra from grade three to 7 were included in the diagnostic test.

Methodology

The diagnostic test was developed to recognize the misconceptions of students in particular subject areas involving seven steps covering four concepts of algebra.

 

Describing the Content

The first step was about the description of concepts and dividing them according to the grades. The content was selected from the Punjab Text Book Board from grade three to eight. The first step typically produced

Same as defined by Stewart (1980).

Step 1: Identifying Concepts

Concepts were further divided according to the grade and age of the students. Mainly seven concepts of algebra from grade three to seven were included. These seven concepts were (1) patterns, (2) polynomials, (3) Factorization (4) linear equations. Keeping in view these concepts, sixty questions were formed from PTB books.

Step 2: Developing a table of specification

Considering the test development method by Novak, a sketch was developed upon the concepts to be included in test development. This practice of developing the table of specification enables the researcher to consider all the content material of concepts carefully selected for test development.




 

Table 1. Table of Specification for Misconceptions Test

Level	Concept	Number of Items				Total
	Patterns
3,4 & 5	Explain and analyze patterns	1	2	2	1	6
	Use of symbolic notation representing the statement of equality.	2	2	2	1	7
6 & 7	Polynomials
	Identify algebraic expressions and basic algebraic formulas.	2	2	1	2	7
	Apply four basic operations on polynomials.	2	2	2	1	7
	Manipulate algebraic expressions using formulas	2	2	2	1	7
	Factorization
	Factorization	1	2	2	2	7
	Linear Equations
	Formulation of one and two variable linear equations	2	2	2	1	7
	Solve simultaneous linear equations	2	2	2	1	7
Total		14	16	15	10	50

Table above shows the number of items against each concept area specified concepts of mathematics.

 




Step 3: Relating Table of Specification to the Statements of Test Items

Questions drawn at the first stage explicitly connected to the table of the specification to make sure that all selected content has been included in the test development process. This was a verification of reliability that the underlying principles and questions actually analyze the subject area to be tested. To check whether the selected concepts are adequately chosen, it is necessary to have a representative coverage of concepts and questions.

 

Step 4: Content Validity

Questions from the content and table of the specification were discussed and verified with the mathematics teacher. The list of related test items and the table of the specification was corrected and updated, and any inconsistencies were eliminated as per decisions of the experts. In this way, the material being tested was carefully recorded.

 

Obtaining Information about Students' Misconceptions

A detailed review of the related literature about procedural and conceptual misconceptions of the students of the mathematics, content and collecting answers from open-ended paper-pencil questions were the second wide field for designing diagnostic tests to identify the misconceptions of students.

 

Step 5: Examining related literature

It is important to review the relevant literature and researches conducted on misconceptions before beginning new efforts to find misconceptions in the subject area. A literature review did not find any research on myths being carried out on this subject for the work on mathematics. On the other hand, extensive research has been conducted on misconceptions of students in learning mathematics.

 

 

Step 6: Unstructured Interviews of Students

Concerning obtaining a wide understanding of students about the mathematical concepts, unstructured interviews were carried out with students studying in grade eight. These interviews were helpful in understanding the weak areas and misconceptions. Responses of these interviews contribute to the creation of ideas for further probes.

 

Step 7: Developing Multiple Choice Test Items

Multiple-choice test items were written from the selected content and questions obtained at an earlier stage. Each item was focused on related statements decided at an earlier stage. Each multiple-choice item was consist of four options one true option and three distracters.

 

Developing a Two-Tier Diagnostic Test

The second tier of the test items was consisted reasons of for choosing these options accordingly. The creation of two-tier MCQs includes the third and wide stage for test development, of which the first tier needs a content response and the second tier requires to choice of justification for the response. For the algebra concepts, a diagnostic test consisting of fifty questions was prepared through a literature review. By taking into account the age group of the students, the questions in the test were prepared. Each student’s true responses were coded as 1, and false and void responses were coded as 0 after the test has been applied to 60 students from three schools and the test items were evaluated.  

Content Validity

The test's content validity was assessed by the views of three school teachers. Amendments were made on the basis of suggestions of the teachers.

 

Face Validity

To sure whether the tests fulfill the needs, what it intended to measure face validity of the test was checked by two school teachers and two experts of the field.

 

Construct Validity

To check whether the developed test is constructed according to the variables it is going to measure, construct validity was assured by a psychometrician.  

 

Item Analysis

After development test was applied to sixty students, the test item analysis was carried out by measuring the complexity and the uniqueness of the test questions, the validity and reliability survey was carried out, incorrect questions were omitted. To standardize test items item analysis of the test items was carried out which is being presented in Table 2 and Table 3.

 

Item Difficulty Description

The difficulty level of test items was considered according to the criteria on each item range from 0. 75 to 1.0 were considered as very easy, 0.25 t0 0.75 average and below 0.25 were considered hard test items. 




 

Table 2. Results of Item Difficulty Description

Item No.	Item Difficulty	Comment	Item No.	Item Difficulty	Comment
1	0.41	Average	26	0.34	Average
2	0.41	Average	27	0.36	Average
3	0.42	Average	28	0.32	Average
4	0.26	Average	29	0.24	Hard
5	0.24	Hard	30	0.20	Hard
6	0.20	Hard	31	0.31	Average
7	0.37	Average	32	0.23	Hard
8	0.34	Average	33	0.24	Hard
9	0.24	Hard	34	0.37	Average
10	0.12	Hard	35	0.23	Hard
11	0.37	Average	36	0.27	Average
12	0.39	Average	37	0.31	Average
13	0.38	Average	38	0.24	Hard
14	0.39	Average	39	0.20	Hard
15	0.40	Average	40	0.27	Average
16	0.33	Average	41	0.25	Hard
17	0.28	Average	42	0.24	Hard
18	0.24	Hard	43	0.37	Average
19	0.35	Average	44	0.32	Average
20	0.42	Average	45	0.36	Average
21	0.27	Average	46	0.33	Average
22	0.32	Average	47	0.24	Hard
23	0.24	Hard	48	0.20	Hard
24	0.33	Average	49	0.24	Hard
25	0.27	Average	50	0.24	Hard

 




The results obtained by the students were determined and sorted by the highest and lowest.  The detailed values of item analysis on item difficulty of item1,2,3,4, 7,8.11,12,13,14,15,16,1719,20-28,31,34, 36, 37, 40 43, 44,45 and 46  ranges from 0.50 to 0.68. Results of item analysis reveal that these items were average in difficulty level for the measurement of intended variable for the elementary level students.

Item Discrimination Index Criteria

Item discrimination index was conducted by point-biserial correlation. Criteria of item discrimination index were as: 0.30 and above good, 0.10-30 fair and equal to zero means no discrimination, and negative means item was constructed very poor.




 

Table 3. Item Discrimination Index results

Item No.	Item Discrimination	Comment	Item No.	Item Discrimination	Comment
1	-1	Poor	26	.30	Good
2	-1	Poor	27	.30	Good
3	.30	Good	28	.25	Fair
4	.12	Fair	29	.25	Fair
5	.13	Fair	30	.23	Fair
6	.32	Good	31	.12	Fair
7	.10	Fair	32	.-11	Fair
8	.40	Fair	33	.-12	Poor
9	-.32	Poor	34	.25	Poor
10	-.11	Poor	35	.12	Fair
11	.30	Good	36	.11	Fair
12	.36	Good	37	.12	Fair
13	.10	Fair	38	-.44	Poor
14	.12	Fair	39	-.05	Poor
15	.11	Fair	40	.12	Fair
16	.11	Fair	41	-.44	Poor
17	-.5	Poor	42	-.5	Poor
18	-.52	Poor	43	-.52	Poor
19	-.5	Poor	44	.12	Fair
20	.13	Fair	45	-.35	Poor
21	.10	Fair	46	.30	Good
22	.12	Fair	47	-.5	Poor
23	-.35	Poor	48	-.52	Poor
24	.30	Good	49	.13	Fair
25	.10	Fair	50	-.35	Poor




The table showed the Item discrimination level for the assessment test for misconceptions (ATM).  The detail of item discrimination index was the two most important item-level statistics for dichotomously scored test (write/wrong) items were the P-value and the point-biserial correlation, which represented the discrimination index of the items.  Item discrimination results show 30 test items fall in the criteria well and fair.

Conclusion

Based on the results obtained from the validity and item analysis, it was concluded that the developed test (ATM) is valid to identify misconceptions of students in mathematics. Results of item difficulty level and discrimination index were accepted and meet the criteria.  Therefore, it can be said that the assessment test for misconceptions (ATM) is valid and reliable to use.

References

• A review and comparison of diagnostic instruments to identify students' misconceptions in science. Eurasia Journal of Mathematics, Science and Technology Education, 11(5), 989-1008.
• Adadan, E., &Savasci F. (2012). An analysis of 16- 17-year-old students' understanding of solution chemistry concepts using a two-tier diagnostic instrument. International Journal of Science Education, 34(4), 513-544.
• Bakula, N. (2010). The benefits of formative assessments for teaching and learning. Science Scope, 34(1), 37-43.
• Baturo, A. R. (2004). Empowering Andrea to help year 5 students construct fraction understanding. In M. J. Hoines& A. B. Fuglestad (Eds.), Proceedings of the 28th Conference of the International Group for the Psychology of Mathematics Education (Vol. 2, pp. 95-102), Bergen, Norway: PME.
• Caleon, I. S. & Subramaniam, R. (2010). Do students know what they know and what they don't know? Using a four-tier diagnostic test to assess the nature of students' alternative conceptions. Research in Science Education, 40, 313-337.
• Cataloglu, E. &Robinett, R. W. (2002). Testing the development of student conceptual and visualization understanding in quantum mechanics through the undergraduate career. American Journal of Physics, 70(3), 238-251.
• Chang, C. Y., Yeh, T. K., &Barufaldi, J. P. (2010). The positive and negative effects of science concept tests on student conceptual understanding. International Journal of Science Education, 32(2), 265-2
• Colin, P., Chauvet, F., Viennot, L. (2002). Reading images in optics: students' difficulties and teachers' views. International Journal of Science Education, 24(3) 313-332.
• Columbia University. Clement, J., Brown, D. E., &Zietsman, A. (1989). Not all preconceptions are misconceptions: finding 'anchoring conceptions' for grounding instruction on students' intuitions. International Journal of Science Education, 11, 554-565.
• Downing, S. M. (2006). Twelve steps for effective test development. In S.M. Downing, & T. M. Haladayna (Eds.), Handbook of test development (pp. 3-25), New Jersey: Lawrence Erlbaum Associates, Inc
• Duit, R., Treagust, D. F., & Mansfield, H. (2004). Investigating student understanding as a prerequisite to improving teaching and learning in science and mathematics. In D. F. Treagust, R. Duit, & B. J. Fraser (Eds.), Improving teaching and learning in Science and Mathematics (pp. 17-31). New York: Teachers College Press.
• Frankel, J. R. & Wallen, N. E. (2000). How to design and evaluate research in education (4th ed.). US: McGraw-Hill Comp.
• Greca, I. M.,&Moreire, M. A. (2002). Mental, physical and mathematical models in the teaching and learning of physics. Science Education, 86(1), 106-121.
• Kaltakçı, D. (2012). Development and application of a four-tier test to assess pre-service physics teachers' misconceptions about geometrical optics. Unpublished PhD Thesis, Middle East Technical University, Ankara, Turkey.
• Kettedin-Geller, L., &Yovanoff , P. (2009). Diagnostic assessments in mathematics to support instructional decision making. Practical Assessment, Research and Evaluation, 14(16), 1-11.
• Klammer, J. (1998). An overview of techniques for identifying, acknowledging and overcoming alternative conceptions in physics education. (Report no: ED423121).
• Linsell, C., Tozer, L., Anakin, M., Cox, A., Jones, R., McAuslan, E., Smith, D., & Turner, G. (2012). Teaching algebra conceptually: Student achievement. In J. Dindyal, L. P. Cheng & S. F. Ng (Eds.), Mathematics education: Expanding horizons (Proceedings of the 35th annual conference of the Mathematics Education Research Group of Australasia, pp.465-472), Singapore: MERGA.
• McAfee, M. (2018). Development and Validation of a Scale to Measure Misconceptions about Educational Psychology among Pre-service Teachers.
• McDermott, L. C. (1993). How we teach and how students' learn-A mismatch?American Journal of Physics, 61(4), 295-298.
• Osborne, J. F., Black, P., Meadows, J., & Smith, M. (1993). Young children's (7-11) ideas about ight and their development. International Journal of Science Education, 15(1), 83-93.
• Peşman, H., &Eryılmaz, A. (2010). Development of a three-tier test to assess misconceptions about simple electric circuits. The Journal of Educational Research, 103, 208-222
• Sarwadi, H. R. H., &Shahrill, M. (2014). Understanding students' mathematical errors and misconceptions: The case of year 11 repeating students. Mathematics Education Trends and Research, 2014(2014), 1-10.
• Stacey, K., Price, B., Steinle, V., Chick, H., &Gvozdenko, E. (2009). SMART assessment for learning. Paper presented at the Conference of the International Society for Design and Development in Education, Cairns, Australia, September 28 - October 1.
• Steinle, V. (2004). Changes with age in students' misconceptions of decimal numbers. Thesis (Ph.D.)--University of Melbourne, Dept. of Science and Mathematics Education.
• Tongchai, A., Sharma, M. D., Johnston, I. D., Arayathanitkul, K., &Soankwan, C. (2009). Developing, evaluating and demonstrating the use of a conceptual survey in mechanical waves. International Journal of Science Education, 31(18), 2437-2457.
• Widjaja, W., Stacey, K., &Steinle, V. (2010). Locating negative decimals on the number line: Insights into the thinking of pre-service primary teachers. Journal of Mathematical Behavior, 30(1), 80-91.
• Wiliam, D. (2009). An integrative summary of the, England: Routledge.