Copyright 2020 The Author(s). Published by vgtu press

2.2.3. Problem solving test – pre- and post-
The tests were created based on the PISA 2012 mathematics questions. Six questions were 
carefully selected for each test that were developed in parallel, so that they are appropriate 
for the age group of the Form 1 students which is approximately 13 years old. Therefore, the 
difficulty level of the test selected was only at proficiency level 3 since the original test paper 
was created for 15-year-old students as PISA originally intended. Instructions were given to 
the teachers to administer the pre- problem solving test one week before the intervention and 
one week after the intervention for the post-test. Teachers were reminded that they should 
not give this test on the same day as TTCT to avoid students being too tired mentally.
2.3. Instrument validity and reliability
The TTCT and PISA problem solving questions which was adopted in this study has re-
ported very high reliability and validity globally. Therefore, no pilot study was done on them. 
However, since the interview protocol are newly created, a pilot study was done to test the 
instrument. The TTCT was reviewed four times since it was developed by Torrance in 1966. 
According to the TTCT manuals of 1966 and 1974, the range of the test–retest reliability 
coefficients is .50 to .93. However, after a few revisions, the reliability estimates of the creative 
index ranged between .89 and .94 (Kim, 2006, p. 6), which is way greater than the desired 
0.70 level. Although only certain questions from PISA 2012 were used in this research study, 
the reliability of PISA problem solving questions of 2012 were reported to be 0.88 (OECD, 
2014d). However, the reliability and validity tests were performed on the pre- and post-test 
scripts of both tests and the result is in Table 2.
Table 2. Internal reliability and eta-squared value of the instrument (source: created by authors)
Instrument
Reliability
(
α
)
Inter-item correlation
Analysis of variance 
(
η
2)
Pre-
Post-
Mean
Range
Mean
Range
Pre-
Post-
Torrance Test of 
Creative Thinking
.913
.923
.569
.149–.950 .598
.224–.948 9.917**
8.795**
Problem solving 
test
.615
.610
.212
.108–.413 .188
.268–.398 10.495** 25.508**
(** p < 0.01).

---

280
M. Khalid et al. Enhancing creativity and problem solving skills through creative problem...
The Cronbach’s alpha of TTCT shows a very high reliability while the reliability for the 
problem solving test is only at an acceptable level since it is between 0.6 and 0.7 with small 
sample (Griethuijsen et al., 2014, p. 588). The mean for inter item correlation were in the 
acceptable range of 0.15 – 0.50 while the values of correlation are somewhat in acceptable 
range of .15 and .85 for problem solving test but a bit high for TTCT. The eta-squared values 
show that the instruments are able to differentiate between the sample of different schools.
The observation checklist shows good reliability for all of the four constructs of original-
ity, elaboration, fluency and flexibility at 0.893, 0.922, 0.894 and 0.938 respectively. The pat-
tern matrix confirmatory factor analysis shows two constructs of originality and elaboration 
loading perfectly into two columns while one item from flexibility overlaps in fluency which 
otherwise behaves accordingly. Hence the instruments are quite reliable and valid for use in 
this study. A summary of research objectives, research questions, data type and collection 
strategy, sampling and sample, validity criteria addressed, and data analysis executed in this 
study is presented in the Appendix.