Adams, M., Foutz, S., Luke, J. & Stein, J. (2007). Thinking Through Art: Isabella Stewart Gardner Museum School Partnership Program year 3 research results.
Abstract:
The Isabella Stewart Gardner Museum and an outside evaluation company, ILI, conducted research on the Gardner’s School Partnership Program (SPP), a multiple-visit museum education program for elementary school students. The study examines how participation in the program affected the development of students’ critical thinking skills. It was carried out over three individual phases, each building on and informed by the previous year’s research findings. This research summary focuses on year three of the evaluation. In this year, the researchers wanted to assess the effects of participation in a multiple-visit art museum program on students’ critical thinking skills in both a school setting, a museum setting and in terms of comprehensive academic standardized tests scores in both English language arts and mathematics. The researchers found treatment group students (those participating in the program) showed more evidence of critical thinking skills in both the school and museum setting than the control group of students who did not participate in the program. Standardized test score results were not available at the time of this report.
Key Findings:
- When the researchers compared students’ use of critical thinking skills discussing reproductions of works of art, they found that students who participated in the Gardner Museum’s education program demonstrated higher frequencies of critical thinking skills at both the school and museum sites.
- Treatment students displayed greater instances of observation, interpretation, association, comparison, flexible thinking, and evidence.
- Treatment students also spoke for longer about the works of art with an average of 28 lines as compared with 14 lines from the control group students.
- Treatment students in the museum setting averaged 45 lines of discussion in their un-facilitated small group discussions as opposed to 27 lines from the control group students.
- Treatment students at the museum also displayed greater instances of the same critical thinking skills demonstrated at the school than did the control group students.
Significance of the Findings:
The findings demonstrate there is strong evidence to suggest the School Partnership Program (SPP) was highly successful in enhancing students’ critical thinking skills. The rigorous and quasi-experimental design of the study sets it apart from most of the studies being conducted in museum education. This research provides a solid base for the larger museum field on which to better understand students’ development of critical thinking skills in an art museum multiple-visit program and, thus, apply these findings to the development of school programs that more consciously and deliberately facilitate critical thinking.
Methodology:
The researchers used a quasi-experimental research design to compare the critical thinking skills of students who participated in the Gardner’s education program (treatment group) with those of students who did not participate in the program (control group). Treatment and control students were in the third, fourth, and fifth grades and attended schools that were comparable regarding race, socio-economic status, and student test scores. Their schools were considered low- to middle-income and had a high proportion of students labeled “at-risk.” Control students did not receive regular arts instruction at school. The researchers examined three research questions: 1) how does participation in the SSP influence students’ individual critical thinking skills; 2) how does participation influence students’ critical thinking skills within a social context in the museum; and 3) how does participation influence students’ performance on standardized tests?
Researchers developed a rubric to measure critical thinking skills in the first year of the three-year study, which was refined and revised in the second year of the study. The critical thinking skills rubric identified and measured seven specific skills: observation, analysis, revision, association, interpretation, problem-finding and evaluation. At the beginning of the study the program entailed educators working with classroom teachers to design gallery visit lessons that connected with the school curriculum. By the end of the study the program had evolved to a set of specific lessons focused on developing students’ observation and interpretation skills. For all three years of the study the program involved a museum educator leading a classroom lesson before the gallery visit. Students visited the museum three or four times a year.
Researchers collected data by conducting open-ended interviews with pairs of students about a reproduction of a work of art at the school site. At the museum site, small groups of six students walked around the museum discussing works of art. The researchers’ audio-recorded these unguided museum tour conversations for analysis. A total of 135 students (64 from the treatment group and 71 from the control group) participated in the school site based interviews about a reproduction of a work of art. For the unguided museum tours 116 students (56 from the treatment group and 60 from the control group) participated. Standardized test scores from 410 students (199 from the treatment group and 211 from the control group) were going to be analyzed, presumably using ANCOVA test, but this work had not been done in time to include in the Year Three report summarized here.
To analyze the Year Three data, transcripts from school-based interviews and museum-based unguided tours were coded by three trained coders. To determine consistency between the three coders, inter-rater reliability (IRR) was assessed. For coding for the seven critical thinking skills in individual interviews IRR was 81%, though coding for evidence was 33%. In coding the “untours” IRR was 78% reliable for the critical thinking skills and 100% reliable for providing evidence. As 60% is considered an acceptable IRR rate, the researchers found the reliability rate to be sufficient though noted that the evidence category in individual interviews should be interpreted with caution.
Limitations of the Research:
Although this multi-year, multi-phase study was comprehensive, the researchers may have relied too heavily on the rubrics that they created and revised based on students’ responses in their own program. It is unclear if the rubric would work in another institution with another student population. Department of Education funding required the researchers to focus on low-income, “at-risk” student populations. This research does not tell us if critical thinking skills would improve for middle- to high-income students not at-risk for school failure. Finally, this research does not address what components of the SPP led to higher instances of critical thinking for treatment students (e.g., was it the consistency of the program, the pedagogy of the museum, follow up by teachers in the classroom, etc.?).
Questions to Guide New Research:
How would the increase or decrease of the quantity of visits affect students’ critical skills development?
How would students’ who receive regular in-school visual arts instruction have responded to this study?
How would this study be different in a different institution with a different student demographic?