Coding Photos
Evaluating Cultural Inclusivity in Classrooms
Client: Forest Grove School District, Oregon
Duration: May – September, 2018 (Note: this is the duration for this highlighted part of the project. The project itself has been ongoing since 2017)
My role: Research scientist
Disclosure: This is part of an ongoing large-scale project that included multiple sub-teams and stages. Although this is not a UX project, I highlight the photo coding portion led by me to showcase my expertise in qualitative codebook development and communication with diverse, non-technical stakeholders.
TL;DR: Skip to Outcome to glimpse my research toolbox and result.
Table of Contents
Project Overview
Background & Goal
The then-superintendent of Forest Grove School District, Oregon, was concerned that the dropout rate and achievement gaps were increasing every semester among Latinx students in the district, and that a growing number of Latinx students reported experiencing discrimination and feeling excluded at school. As a result, the superintendent reached out to my advisor, Dr. Stephanie Fryberg, to help improve the learning environment for Latinx students and racial minority students more broadly. Our team recognized that students from racial minority groups and lower socioeconomic status tended to be raised in a culture different from the mainstream White, middle-class culture at schools. Therefore, our team developed an inclusion training for teachers to help them understand cultural differences and build culturally inclusive classrooms for disadvantaged students.
We did a 2-month training during the summer. One semester after the training, we planned to evaluate teachers' post-intervention inclusion-related performance in order to 1) deliver insights and convey progress to the superintendent and the teachers, and 2) evaluate the effectiveness of the intervention.
What Are Culturally Inclusive Classrooms?
These are places where all students feel that they belong and can be successful. That is, students see themselves, their experiences, and their ways of being and knowing represented in the curriculum and explicitly acknowledged and valued in the classroom.
My Input
Among many indicators of inclusion-related performance, I proposed to use classroom visual reminders (e.g., posters, slogans, photos, work display board, etc.) because teachers explicitly learned during the training that adding visual reminders of inclusive culture could convey to students that their cultural and/or racial backgrounds were recognized and valued. Because group representation is my research focus, I self-nominated to lead this part of the performance assessment.
Goal: translate photos of inclusive visual reminders into quantifiable data that can be used to compare pre- vs. post-intervention performance.
Team
The large project was directed by Dr. Stephanie Fryberg. Some other leading figures on the project included Dr. Hazel Rose Markus and Dr. Laura Brady I led all aspects of the photo coding portion, and had a team consisting of 9 undergraduate research assistants working as coders.
Process
Here is a high-level overview of what the process looked like. I led all phases. Except for Phase 2 in which I trained and supervised 9 coders coding photos, I worked on all the other phases on my own while presenting progress and insights at research team meetings to solicit feedback from other team members.
Timeline wise, Phase 1 took 5 days; Phases 2 spanned across 2 months due to the large amount of photos and the restriction on the coders' working hours; Phase 3 took 2 days; Phase 4 took 1 week.
Phase 1
For each teacher participating in the inclusion training, we took photos of all the visual reminders in their classrooms both 1 week before the training and 1 semester after the training. In the end, I had over 1,800 photos. Below is an overview of the codebook development process:
Initial Codebook Development: I randomly selected 50 photos. I annotated all visual reminders (e.g., poster, slogan, student work, artwork) that conveyed the idea(s) of inclusion, diversity, and/or equity. Affinity diagramming reveals initial recurring themes, which made up the initial codebook.
Codebook Refinement: I randomly selected another 50 photos and applied the initial codebook to check whether new themes emerged, documenting any changes to the codebook. This process was repeated 2 times.
I completed the final codebook after presenting the working codebook to at research team meetings to solicit and incorporate feedback.
Notes were clustered using affinity diagramming to reveal initial recurring themes from the first set of 50 photos.
A Glance at the Themes Identified in the Final Codebook
Phases 2
Training Coders
In this project, all the coding was completed in Excel.
To help my coders effectively and successfully complete the coding task, I first wrote a detailed manual to guide coders step-by-step through the coding process (click here to see the manual).
Then, I led a 2-hour workshop with my coders, going through the manual and then practicing coding 15 photos to detect any technical issues and confusions. After the workshop, I let my coders do their job while holding a weekly check-in meeting.
Each photo was independently coded by 3 coders.
Ensuring Inter-Coder Agreement
Each week, I examined whether 3 coders agreed on how to code the same visual reminder. If there was a disagreement, the visual reminder would be returned to the 3 coders to discuss and re-code. If disagreement continued, coders would put the photo on a disagreement list. The whole team discussed how to properly code the photos on the disagreement list at the weekly check-in meeting. Phases 2 ended when ALL visual reminders in the 1,800+ photos had unanimous codes.
Below are two examples of the coding spreadsheet for photos:
Example 1:
In conclusion, 1 (out of 1) visual reminder in this photo would be coded as an inclusive visual reminder and conveying one type of culturally inclusive message.
Example 2:
Part of the photo is blurred due to restrictions.
In conclusion, 3 (out of 4) visual reminders in this photo would be coded as inclusive visual reminders and conveying two different types of culturally inclusive message.
Phases 3 & 4
Analyzing
To compare the inclusive visual reminders that teachers put up pre- vs. post-intervention, I used two indices, one being the type of inclusive messages conveyed in the classroom and the other being the quantity of inclusive visual reminders. For the former, I simply counted how many different (sub)categories were coded in each classroom before the intervention as well as after the intervention. For the latter, I decided to quantify inclusive visual reminders with percentage (the amount of inclusive visual reminders divided by the total amount of all visual reminders in a classroom).
Results: Analysis showed that, one semester after the intervention, the percentages of inclusive visual reminders and the types of inclusive messages conveyed by these visual reminders in most classrooms had increased compared to pre-intervention.
Delivering to Stakeholders
The large project team presented progress in inclusive classroom building that the intervention achieved to 2 groups of stakeholders: the superintendent (along with her staff) and the trained teachers. I was in charge of the deliverables related to inclusive visual reminders. I focused on the big picture insights when delivering to the superintendent and individual performance when delivering to each teacher.
Deliverables to the superintendent and her staff:
Key Progress: The intervention improved both the quantity of inclusive visual reminders and the type of conveyed inclusive messages.
Note: numbers are redacted due to restrictions.
Since the intervention, the amount of inclusive visual reminders in the school district increased by ☐☐% overall and increased by ☐☐% on average in each classroom.
The number of classrooms without any inclusive visual reminders dropped from ☐☐ to ☐☐.
Teachers conveyed more diverse types of inclusive messages post-intervention. Before the intervention, inclusive visual reminders were predominantly bilingual slogans and posters of female/racial minority role models. After intervention, we observed more visual reminders encouraging students to learn from and collaborate with each other, respect differences, and be aware of and stand against prejudice and discrimination.
Deliverables to individual teachers: I made each teacher a personalized progress report regarding putting up inclusive visual reminders. The report contained 1) visualizations of their post-intervention performance compared to their own pre-intervention performance and compared to the average post-intervention performance among teachers across the school district, along with a short summary of the takeaways from the visualizations; and 2) actionable recommendations for how to keep improving based on what types of inclusive messages were under-displayed (or not displayed at all) in their classrooms. Here is a sample of the personalized progress report.
Data and insights in this report are made up. Real reports cannot be shared here due to restrictions.
Outcome
By using an inductive approach to codebook development and directing a team of 9 coders to code 1,800+ photos using the codebook, I achieved the primary goal of translating photo content into data that quantified the extent to which teachers put up inclusive visual reminders. My analysis supported that the intervention was effective in providing teachers with tools to build culturally inclusive environments for their students.
Reflections
Obstacle & Solution
Ambiguity in visual reminders contained in photos: We did not have a manual or explicit instruction for photo taking, so photos did not follow the same format and could cause confusion when coding. For example, some photos only had one visual reminder such as a poster, but some photos ended up with multiple visual reminders clustered together such as 2 posters along with a student work display board and part of a bookshelf. Another challenge that I went into was that sometimes how many visual reminders were in a photo was open to interpretation. Here is an example:
Part of the photo is blurred due to restrictions.
In the photo on the left side, while at first glance, the blue circle seemed to contain two posters, a closer look revealed that they were the same poster but in two languages, so they should be considered as one visual reminder. As a results, this photo contained 4 unique visual reminders (as labeled in the photo on the right side).
Solution: In the manual I wrote to guide coders step-by-step through the coding process, I provided detailed instructions regarding how to identify the number of visual reminders in a photo and what order coders should follow when coding when more than one visual reminder were present (click here to see the manual instructions).
What Would I Do Differently?
A lot of extra work during the coding process could be attributed to a lack of standard protocol for photo taking. In the future when I need to code photos, I would get involved earlier in the process and write a manual for how to take photos of the artifacts.
I wish that we set up a goal before intervention regarding what increase in the amount of inclusive visual reminders we expected post-intervention so that the comparison would be more meaningful. Currently, it was hard to tell whether the increase we accomplished was ideal.