1st supervisor: Dr. W. R. van Joolingen 2nd supervisor: Dr. P. Wilhelm University of Twente Faculty of behavioral science Enschede – June, 2007 Foreword

4. Conclusions

No significant differences between control and experimental groups were found, meaning that there is no advantage to using an analogy mapping technique in terms of correct answers on the CINS or in terms of insight into the shortcomings of the simulation. To put it in another way: Analogy mapping does not result in insights that participants would not be able to acquire by concept mapping alone, at least not in the current experimental set-up. Both groups did learn somewhat, as reflected in better post-test scores for most participants.

Other findings

Differences between evolutionists and non-evolutionists were quite large with respect to pre- and post-test CINS scores. Not only did evolutionists score better on both tests, but they also improved more between tests. This result seems to suggest that people who do not (fully) believe in evolution as an explanation for the existence of complex life, are less willing or capable to change their views on this theory, though this conclusion is certainly not decisive. For the sake of public argument, the fact that many evolutionist scored rather low (approximately 7,5 and 4,5 incorrect answers on pre- and post-test respectively) may be interpreted as demonstrating that the debate between ‘evolutionists’ and creationists is indeed somewhat based on belief, rather than understanding, on the part of both parties.

Men scored significantly better in pre- and post-test questionnaires. Women, however, did seem to improve more between pre- and post-tests. A possible explanation for this interaction effect might be that men have more affinity with biology-related or technical subjects; have learned more about it in the past, and are therefore more knowledgeable, but also less flexible with regard to changing their – partly incorrect - conceptual models of evolution. More research is needed to clarify this matter.

Reflections on the CINS and mappings

Evolution is a very difficult topic to teach, and teachers should not assume that it can be explained to students in a straightforward matter. Getting a clear picture of existing pre-conceptions, by means of the CINS for instance, may seem appropriate, but there is a mentionable caveat: A good or bad answer to a CINS question does not seem to reveal much about the student’s conception of evolution! The results from this study seem to suggest that this kind of superficial testing is, at best, only insightful into whether or not students fully understand evolution. No statements with regard to exactly how a student conceives evolution can be made on the basis of CINS questions alone. Here is why:

A relation at the 0.1 significance level was found between concept/analogy map quality and scores on likert questions, whereas no relation whatsoever was found between CINS scores and map quality. It is possible that more power and/or a higher number of Likert questions would have resulted in statistical significance of the relation between map quality and Likert scores. It will be interesting to see whether future research can explain the difference between the relation between mappings and questionnaires as diagnostic tools, and how Likert scale questions like the ones used here relate to multiple choice questions in terms of exactly what it is they measure, and what conclusions can be drawn on the basis of their outcome. For now, it seems that in the search for evidence for the existence of both correct and alternative conceptions of evolution and natural selection in students, confidence measures like those reflected in the Likert scales may be essential. Answers to the Likert question, though not like confidence measures in a strict sense, reflect some degree of doubt and resistance to conceptual change that may persist in the student.

Anderson, Fisher and Norman (2002), developers of the CINS, state that “the CINS provides a simple yet effective means of identifying the frequency of some common misconceptions among large numbers of students”, and that “because the [CINS] targets ten of the main ideas of the theory of natural selection, instructors can be assured that a student who chooses all of the correct answers has a fairly comprehensive understanding of how natural selection influences populations” (p. 967). It seems, then, that the CINS is not an actual inventory, but rather a relatively crude means for discriminating poor and good understanding of natural selection.

It seems that the CINS, based on an – admittedly valid – subdivision of conceptions in the field of natural selection, is a questionnaire that forces students to automatically fall into one of ten pre-selected categories of misconception. This is not to say that the CINS is useless, but that, preferably, it should not be used in isolation when testing student knowledge and insight, or inventorying misconceptions. It may be a worthwhile investment to develop a test that uses the advantages of likert scale confidence measures, and that is based on the conceptual subdivision similar to the CINS’s.

The current experiment required the CINS to be translated into Dutch, and this, in turn, required that the translated version was reviewed by several domain experts for validation. The expert comments were insightful, and some comments might even be things to consider for improvement of the CINS:

Question 13 reads: "Assuming ideal conditions with abundant food and space and no predators, what would happen if a pair of guppies were placed in a large pond?" The correct answer is C, which reads "The guppy population would grow slowly at first, then would grow rapidly, and thousands of guppies would fill the pond". Apparently, the authors are trying to avoid using the word 'exponentially', but the current formulation is confusing to say the least. The relative speed of growth is constant. Only the increase of absolute numbers is 'slow'. It seems that 'exponentially' is not such a bad term, and even just 'fast' seems a better choice than the current formulation.

Question 7 reads: "What type of variation is passed on to the offspring?" All four alternative answers are incorrect. According to the authors, the correct answer should be C ("All characteristics that are genetically determined"). Better would be "some characteristics..", since part of variation is, after all, random mutation. "Type of variation" also seems like a vague description that students will have a hard time interpreting.