On Assessing Diversity…

K.Park/ August 1, 2021/ 4 comments

Question

My question specifically, is “How can I assess/evaluate professional development and courses for diversity and inclusion?”

Some initial thoughts…

I was recently reviewing a course from a content provider to see if I wanted to integrate some of the work into my own teaching materials and an “ah-hah” moment came to me.  I looked at the materials to assess the quality of content, how interactive it was, but then I realized the materials were missing something.  It wasn’t diverse. But how did I know it wasn’t diverse?  I didn’t have criteria or a list of components, but I knew something was “off” and should be different.  If I choose to pass on these materials, how would I explain things beyond “I have a gut feeling.”

This then got me thinking about my own materials and could I really say I developed courses that contained diversity?  What criteria was I even using to determine that?  I had a vague idea in my mind, but I thought it was time to dig into this more and look at having a process.  Now, I am wary of a process because diversity is a human issue.  This means it’s complex, layered, highly personal, relational, and more; and by developing a process or a method of evaluation, I could be collapsing this complicated topic into a paragraph or two. 

I recognize this is a nuanced and complex question to navigate and that I will do my best to be thoughtful and demonstrate care in my approach.

Can it be measured?

In 2011, Thomas Laird ran a study to see if it was possible to measure the diversity inclusivity of college courses.  The study relies on 12 survey items, categorized into 9 groups, that address ways faculty include diversity into their courses.

Below is a re-creation of the table of diversity items from Laird’s research.

ItemaRelated Course Elementb
aStudents gain an understanding of how to connect their learning to societal problems or issuesPurpose/goals
bStudents develop skills necessary to work effectively with people from various cultural backgroundsPurpose/goals
cThe course content emphasizes contributions to the field by people from multiple culturesContent
dThe course covers topics from multiple theoretical perspectivesFoundations/perspectives
eYou explore your own cultural and intellectual limitations as part of class preparationInstructor(s)
fYou address your potential biases about course-related issues during classInstructor(s)
gYou try to learn about student characteristics in order to improve class instructionLearners
hYou vary your teaching methods to encourage the active participation of all studentsPedagogy
iYou work on creating a classroom atmosphere that is conducive to student learningClassroom environment
jYou try to empower students through their class participationPedagogy
kYou evaluate student learning using multiple techniquesAssessment/Evaluation
lYou adjust aspects of the course (e.g., pace, content, or assignments) based on student learning needsAdjustment

aFaculty members were asked how much each item happened in their course sections. Response options were 1 = Very little, 2 = Some, 3 = Quite a bit, and 4 = Very much

bCourse elements from Nelson Laird’s (2010) model

He found that “Courses inclusive of diversity also do not all meet diversity requirements and, further, an individual course may be inclusive in some aspects and lack inclusivity in others. This study also shows that many, many faculty bring diversity into their courses through an inclusive learning approach, which combines knowing student needs, creating a classroom environment conducive to learning, and varying teaching and evaluation methods to reach different types of students. (Laird, 2011)”

While these 12 items or strategies provide important information about a course and can be used as indicators in a course, these indicators aren’t the only thing instructors should use to determine whether their course is diverse. Zaretta Hammond, who wrote Culturally Responsive Teaching & The Brain, noted in an interview with Jennifer Gonzalez for Cult of Pedagogy, one of the misconceptions about culturally responsive teaching is that it’s all about choosing the right strategies. I interpret that to mean that even if the above 12 strategies were employed, or implemented into a class, that Zaretta is saying it’s more complicated than that to make a classroom diverse – that instructors also need to regularly reflect and review their instruction practices and structures while also keeping their own identities and the identities of their students in mind. (Cult of Pedagogy, 2017).

Next Steps

I believe it is important to have a process that can evaluate how diverse materials are – if not, how can I explain how my own work is diverse?  I do implement some of the strategies listed above, but do I have a robust process that also allows me and others I work with to interrogate and reflect on the strategies that have been implemented?  Currently, the answer is no.  So what can I do?

I think Thomas Nelson Laird is onto something in a presentation he gave at Penn State University in 2011.  In the presentation he proposes using a self-assessment worksheet based on the 9 major groups identified in his research study. 

The worksheet looks something like this, where the major groups are listed and instructors identify any area within those categories of their course to make more inclusive.

Diversity Self-Assessment Tool by Thomas F. Nelson Laird

An example might be the content and I could be more inclusive by working to include more diverse authors and perspectives in my courses.  In addition to self-assessment, instructors should also take into account student experiences and feedback.  This could be done through regular surveys on and reflection activities in the course as well as the final course evaluation at the end of the term. 

If I were to add this to a course review process, then I would likely modify the Online Learning Consortium’s OSCQR Rubric to incorporate a 7th category for Diversity & Inclusivity and add Nelson Laird’s work to it.  The new section could look something like this:

7. Diversity Equity & Inclusion

Sufficiently presentMinor Revision (1/2 hour or less)Moderate Revision (1/2 – 2 hours)Major Revision (2+ hours)Not ApplicableAction Plan
Purpose/goals      
Content      
Foundations/perspectives      
Learners      
Instructor(s)      
Pedagogy      
Classroom environment      
Assessment/evaluation      
Adjustment      

While I like this format and I think it makes giving feedback easier for reviewers, I would like to remove Sufficiently present and rename Not applicable to Existing evidence as columns in a diversity review.  This way reviewers can highlight work already in progress and encourage new and additional strategies or refinement of existing strategies in the Action Plan field.

So for me, a classroom rubric might look like the following:

Diversity & Inclusion Rubric for Classes

Minor Revision (1/2 hour or less)Moderate Revision (1/2 – 2 hours)Major Revision (2+ hours)Existing EvidenceAction Plan & Additional Comments
Purpose/goals     
Content     
Foundations/perspectives     
Learners     
Instructor(s)     
Pedagogy     
Classroom environment     
Assessment/evaluation     
Adjustment     

If I were reviewing content from a provider, I would simplify this this rubric further into Minor Issues, Moderate Concerns, Major Concerns, Existing Evidence, and a space for additional comments.

Diversity & Inclusion Rubric for PD and Content Providers

Minor ConcernsModerate ConcernsMajor ConcernsExisting EvidenceAdditional Comments
Purpose/goals     
Content     
Foundations/perspectives     
Learners     
Instructor(s)     
Pedagogy     
Classroom environment     
Assessment/evaluation     
Adjustment     

In addition to these rubrics, I would also create instructions that would help explain category and provide reviewers with examples of things they might see in each category. 

Conclusion

As I think about my proposed rubrics above, I recognize that there is still more work to be done.  I think in each of these main categories I would like to expand them and create sub-sections, like the ISTE Standards (e.g. 3a, 3b, 3c, 3d).  These categories are very broad, and I believe the 12 survey items from Nelson Laird’s survey would provide a good start.

By using a rubric like this I can also see other uses, like when I evaluate technology tools, that I’ll be able to better explain concerns, be aware of impacts, and hopefully, improve diversity and inclusivity in in my team’s work and in my own work. That said, if this rubric is used for class evaluation, it should still be paired with thoughtful reflection from the instructor and instructors should also consider who their students are, because some strategies may work better with some groups of students than others for a variety of reasons.

While there are concerns of collapsing a complex topic like diversity, into a rubric, I think it’s still worth the practice because it creates the opportunity for conversation and self-reflection.  I would not advocate on this becoming a tool used in personnel practices, but instead, for thoughtful feedback and to help supplement decisions made on technologies/software and content selected that will improve diversity and inclusivity across an institution.

References

Nelson Laird, T. (2011). Measuring the Diversity Inclusivity of College Courses. Research in Higher Education52(6), 572–588. https://doi-org.ezproxy.spu.edu/10.1007/s11162-010-9210-3

Gonzalez, J. & Hammond, Z. (2017). Cult of pedagogy.  Culturally Responsive Teaching: 4 Misconceptions. https://www.cultofpedagogy.com/culturally-responsive-misconceptions/ [podcast]

Nelson Laird. T. (2011). Including diversity: A strategy for improving teaching and learning in all courses and curricula. http://equity.psu.edu/workshop/assets/pdf/fall11/morn_ses.pdf

Share this Post

4 Comments

  1. Yes, diversity is a human issue, but it still, can be measured. I think the way your proposal is able to help. It is a good measuring tool and it also can step by step help teachers try to catch up and follow up with the rubric. Nice post, it is very useful information. Thank a lots.

  2. I deeply appreciate watching your thought process unfold as you provide varying iterations of the rubric/assessment guide. It’s an extremely effective way of communicating your rationale for the final product (even if you will continue to make adjustments and expand it moving forward). I also appreciate your attention to the complex nature of “measuring” the presence of diversity and inclusion in a given course, and that one can’t simply “check all the boxes” with certain strategies and automatically make a course/classroom appropriately diverse, equitable, and inclusive. As you’ve noted, opportunity for regular self-reflection is key, and these rubrics seem to work really nicely toward that end! Well done!

  3. I like that there is a rubric to help determine diversity in a course or classroom, or whatever. I thought it was interesting the way you hashed that out. I also agree that the need for student evaluations are important for determining diversity. I wonder, do you think a self-assessment is more important over student evaluations or surveys? Are they equally important? Anyway, good job!

    1. From my perspective, I think both are needed for a full(er) picture, but one with out the other cannot be the sole determining factor on whether or not someone is meeting diversity and inclusion efforts. I would hope more for critical reflection that shows growth and consideration of student perspectives.

      When I worked as a grocery clerk, I learned that most people who have a good experience, don’t say something (leave feedback), but those who have a poor experience are something like three times more likely to say something. I think in some ways that can happen with Student Feedback on courses, plus there’s concerns about low response rates, skipped questions, and the feedback can be contradictory, probably even more so when it comes to a topic like diversity and inclusivity. So that’s not to say that I don’t value student feedback – it’s more to say that there are some additional considerations and that’s why I don’t really feel comfortable at this point to introduce a measurement tool that impacts personnel decisions. I think there would be more issues to work though that I haven’t really thought about in context of this post.

Leave a Comment

Your email address will not be published.

*
*

This site uses Akismet to reduce spam. Learn how your comment data is processed.