On Assessing Diversity…
My question specifically, is “How can I assess/evaluate professional development and courses for diversity and inclusion?”
Some initial thoughts…
I was recently reviewing a course from a content provider to see if I wanted to integrate some of the work into my own teaching materials and an “ah-hah” moment came to me. I looked at the materials to assess the quality of content, how interactive it was, but then I realized the materials were missing something. It wasn’t diverse. But how did I know it wasn’t diverse? I didn’t have criteria or a list of components, but I knew something was “off” and should be different. If I choose to pass on these materials, how would I explain things beyond “I have a gut feeling.”
This then got me thinking about my own materials and could I really say I developed courses that contained diversity? What criteria was I even using to determine that? I had a vague idea in my mind, but I thought it was time to dig into this more and look at having a process. Now, I am wary of a process because diversity is a human issue. This means it’s complex, layered, highly personal, relational, and more; and by developing a process or a method of evaluation, I could be collapsing this complicated topic into a paragraph or two.
I recognize this is a nuanced and complex question to navigate and that I will do my best to be thoughtful and demonstrate care in my approach.
Can it be measured?
In 2011, Thomas Laird ran a study to see if it was possible to measure the diversity inclusivity of college courses. The study relies on 12 survey items, categorized into 9 groups, that address ways faculty include diversity into their courses.
Below is a re-creation of the table of diversity items from Laird’s research.
|Itema||Related Course Elementb|
|a||Students gain an understanding of how to connect their learning to societal problems or issues||Purpose/goals|
|b||Students develop skills necessary to work effectively with people from various cultural backgrounds||Purpose/goals|
|c||The course content emphasizes contributions to the field by people from multiple cultures||Content|
|d||The course covers topics from multiple theoretical perspectives||Foundations/perspectives|
|e||You explore your own cultural and intellectual limitations as part of class preparation||Instructor(s)|
|f||You address your potential biases about course-related issues during class||Instructor(s)|
|g||You try to learn about student characteristics in order to improve class instruction||Learners|
|h||You vary your teaching methods to encourage the active participation of all students||Pedagogy|
|i||You work on creating a classroom atmosphere that is conducive to student learning||Classroom environment|
|j||You try to empower students through their class participation||Pedagogy|
|k||You evaluate student learning using multiple techniques||Assessment/Evaluation|
|l||You adjust aspects of the course (e.g., pace, content, or assignments) based on student learning needs||Adjustment|
aFaculty members were asked how much each item happened in their course sections. Response options were 1 = Very little, 2 = Some, 3 = Quite a bit, and 4 = Very much
bCourse elements from Nelson Laird’s (2010) model
He found that “Courses inclusive of diversity also do not all meet diversity requirements and, further, an individual course may be inclusive in some aspects and lack inclusivity in others. This study also shows that many, many faculty bring diversity into their courses through an inclusive learning approach, which combines knowing student needs, creating a classroom environment conducive to learning, and varying teaching and evaluation methods to reach different types of students. (Laird, 2011)”
While these 12 items or strategies provide important information about a course and can be used as indicators in a course, these indicators aren’t the only thing instructors should use to determine whether their course is diverse. Zaretta Hammond, who wrote Culturally Responsive Teaching & The Brain, noted in an interview with Jennifer Gonzalez for Cult of Pedagogy, one of the misconceptions about culturally responsive teaching is that it’s all about choosing the right strategies. I interpret that to mean that even if the above 12 strategies were employed, or implemented into a class, that Zaretta is saying it’s more complicated than that to make a classroom diverse – that instructors also need to regularly reflect and review their instruction practices and structures while also keeping their own identities and the identities of their students in mind. (Cult of Pedagogy, 2017).
I believe it is important to have a process that can evaluate how diverse materials are – if not, how can I explain how my own work is diverse? I do implement some of the strategies listed above, but do I have a robust process that also allows me and others I work with to interrogate and reflect on the strategies that have been implemented? Currently, the answer is no. So what can I do?
I think Thomas Nelson Laird is onto something in a presentation he gave at Penn State University in 2011. In the presentation he proposes using a self-assessment worksheet based on the 9 major groups identified in his research study.
The worksheet looks something like this, where the major groups are listed and instructors identify any area within those categories of their course to make more inclusive.
An example might be the content and I could be more inclusive by working to include more diverse authors and perspectives in my courses. In addition to self-assessment, instructors should also take into account student experiences and feedback. This could be done through regular surveys on and reflection activities in the course as well as the final course evaluation at the end of the term.
If I were to add this to a course review process, then I would likely modify the Online Learning Consortium’s OSCQR Rubric to incorporate a 7th category for Diversity & Inclusivity and add Nelson Laird’s work to it. The new section could look something like this:
7. Diversity Equity & Inclusion
While I like this format and I think it makes giving feedback easier for reviewers, I would like to remove Sufficiently present and rename Not applicable to Existing evidence as columns in a diversity review. This way reviewers can highlight work already in progress and encourage new and additional strategies or refinement of existing strategies in the Action Plan field.
So for me, a classroom rubric might look like the following:
Diversity & Inclusion Rubric for Classes
|Minor Revision (1/2 hour or less)||Moderate Revision (1/2 – 2 hours)||Major Revision (2+ hours)||Existing Evidence||Action Plan & Additional Comments|
If I were reviewing content from a provider, I would simplify this this rubric further into Minor Issues, Moderate Concerns, Major Concerns, Existing Evidence, and a space for additional comments.
Diversity & Inclusion Rubric for PD and Content Providers
|Minor Concerns||Moderate Concerns||Major Concerns||Existing Evidence||Additional Comments|
In addition to these rubrics, I would also create instructions that would help explain category and provide reviewers with examples of things they might see in each category.
As I think about my proposed rubrics above, I recognize that there is still more work to be done. I think in each of these main categories I would like to expand them and create sub-sections, like the ISTE Standards (e.g. 3a, 3b, 3c, 3d). These categories are very broad, and I believe the 12 survey items from Nelson Laird’s survey would provide a good start.
By using a rubric like this I can also see other uses, like when I evaluate technology tools, that I’ll be able to better explain concerns, be aware of impacts, and hopefully, improve diversity and inclusivity in in my team’s work and in my own work. That said, if this rubric is used for class evaluation, it should still be paired with thoughtful reflection from the instructor and instructors should also consider who their students are, because some strategies may work better with some groups of students than others for a variety of reasons.
While there are concerns of collapsing a complex topic like diversity, into a rubric, I think it’s still worth the practice because it creates the opportunity for conversation and self-reflection. I would not advocate on this becoming a tool used in personnel practices, but instead, for thoughtful feedback and to help supplement decisions made on technologies/software and content selected that will improve diversity and inclusivity across an institution.
Nelson Laird, T. (2011). Measuring the Diversity Inclusivity of College Courses. Research in Higher Education, 52(6), 572–588. https://doi-org.ezproxy.spu.edu/10.1007/s11162-010-9210-3
Gonzalez, J. & Hammond, Z. (2017). Cult of pedagogy. Culturally Responsive Teaching: 4 Misconceptions. https://www.cultofpedagogy.com/culturally-responsive-misconceptions/ [podcast]
Nelson Laird. T. (2011). Including diversity: A strategy for improving teaching and learning in all courses and curricula. http://equity.psu.edu/workshop/assets/pdf/fall11/morn_ses.pdf