Introduction: Health literacy is recognized as an important determinant of individuals' ability to find relevant health information and critical appraisal of health claims as a key skill for developing overall health literacy. Children, like adults, are exposed to various unsubstantiated claims about the effects of health interventions and tend to pay little attention to the credibility of the sources of health claims. A multidisciplinary, international team of experts gathered in the project called Informed Health Choices proposes a framework for the critical appraisal of health claims in the form of Key concepts, which serve as standards for making informed decisions about health. Objective: To determine the short and long-term effects of the education based on the IHC Key Concepts on the ability of primary school children to appraise health claims about treatment effects critically. Methods and subjects: This Cluster Randomized Controlled Trial (CRCT) included six randomly selected primary schools from Split. Schools were treated as randomization units and were randomly assigned to the intervention (N=3) or control (N=3) group. The respondents in this research are students in the third and sixth grades of primary school. In the intervention schools, education lasted nine school hours, and all the educational materials of the IHC group, together with the textbook for children, were translated and adapted into Croatian and given to the teachers in the intervention group during the educational workshops. The Ministry of Education, Culture, and Science gave its approval and a positive opinion for introducing education based on IHC concepts into the curricula of selected schools. Education in the intervention schools was conducted from the beginning of March to the end of May 2021. The control schools' students and teachers were unfamiliar with the educational materials. Both groups tested students immediately after the education and six months later. The principal investigator was present during the completion of the tests in all classes. Results: The total number of children included during the first measurement was 733, with a 91% response rate. Seven hundred twenty-eight children participated in the second measurement, with a response rate of 92.03%. In the 1st measurement, there were 347 boys and 320 girls, and in the 2nd measurement, 349 boys and 321 girls. No statistically significant difference was found in the gender distribution between the intervention and control groups in the 1st measurement [χ2(1)=0.60; P=0.44], nor in the second measurement [χ2(1)=2.32; P=0.13]. According to the Rasch model and classical measurement theories, the Croatian version of the test for 3rd and 6th-grade primary school students has good metric features and measures school children's ability to evaluate health claims critically. During both measurements, the students showed good understanding and correct interpretation of the applied educational materials, with no significant difference between the 3rd and 6th-grade students and no significant deviations between the level of knowledge between the first and second measurements. Furthermore, in both measurements, we found a significant difference (P<0.001) in the proportion of correct answers on the test between third and sixth-grade students. The applied educational intervention improved the rate of correct answers on the test for the 3rd and 6th grades, which is one of the most important findings of this study. In 3rd-grade and 6th-grade students, a significant difference was found between the intervention and control groups at the P<0.001 level. The results of the 2nd measurement in the 3rd and 6th grades show that even after six months, the intervention group maintained a significantly higher proportion of correct answers than the control group. Conclusions: Education based on the IHC Key Concepts showed excellent short and long-term effects on the ability of primary school children to appraise claims about treatment effects critically. All schools from the intervention group had a higher proportion of correct answers at both measurements than schools in the control group.