How To Improve Racial and Gender Inclusion in Survey Design

Measuring racial and gender diversity in surveys is facing new challenges for the market research and insights industry. Here is the transcript for the presentation I did on this topic as part of the May 2021 Virtual Town Hall panel discussion “Online Research: Essential Ideas for Inclusive Outcomes,” organized by the Insights Association and InnovateMR.

In my section I discussed:

• The evolution of the racial categories used today in many market research surveys.
• Challenges involved in measuring diversity.
• The current state of questions related to gender identity and sexual orientation.
• Practical survey design recommendations.

The webinar took place on May 7, 2021. Give it a listen (~13 minutes) or read the transcript below.

 Introduction

00:00:00

Thank you, Melanie and InnovateMR for the opportunity to speak today. Hello everybody, my name is Michaela Mora. I am the president of Relevant Insights, a research agency in the Dallas area, Texas. We have been doing qualitative and quantitative research and UX research for the last 13 years.

I’m also one of the founding members of the Multicultural Insights Collective, a coalition of researchers with experience in research, diversity, and multicultural research.

Today, I’ll talk about key moments in the evolution of the race categories that we all use today in our surveys and the challenges we are facing to capture the racial and cultural diversity that has been growing steadily for more than 50 years in the U.S. I will also touch on the current state of questions related to gender identity and sexual orientation since this is an area that needs rethinking given the public discourse and changes in attitudes, especially in the younger generations. Then I’ll close my section with some practical survey design recommendations.

00:01:11

Most market research surveys use a set of standard categories to measure race and ethnicity. To understand why we use those categories it helps to know a little bit about their history in the Census because that’s where they come from.

I strongly recommend reading the book “What is your race?” by Kenneth Pruitt. He is a professor in Social Affairs at Columbia and was the director of the Census between 1998 and 2001.

And he does a great job at showing the evolution of what he calls statistical races at the Census in the context of social, economic, and cultural changes in the United States.

His thesis is that racial categories that we use today are no accidents of history. They have been constructed by the government and used with political purposes and policy consequences.

Evolution of Race Categories in the U.S. Census

It is a long history of 230 years and 24 censuses with many changes reflecting and shaping our social reality.

I created this timeline to show changes specifically related to the race categories. I won’t have time to go through all of them, so I really recommend reading the book.

The race categories we have today have been molded by initially a racist system promoting segregation, hierarchy, and restriction of interracial unions, and then started to take the current form under the influence of the civil rights movement, which changed the purpose of these categories as they became useful when the government began to enact policies to promote integration and equality.

Although these categories started with a color line to separate blacks and whites, the immigration of workers who were needed to grow the country introduced the nativity line in the racial categories very early on, and this nativity line continues to play a big role in their definitions today. There are other factors that have shaped the racial categories such as identity politics, the interest in gaining recognition by certain groups, and demographics, and attitudinal changes towards diversity over time.

00:03:34

One important moment in the history of the race categories was the standardization that took place in 1977 across federal agencies.

So, these are the categories that were used in the 1980 census. I want to draw your attention to the mix of criteria, on the left column, that were used and are still with us today. The standardization didn’t mean a coherent set of categories, so depending on the group race could mean color, descend group, culture, tribal affiliation, or nationality.

They were modified mainly to make public and official how affirmative action should extend beyond black Americans, and so the term People of Color became a common label to refer to non-white groups indicating greater diversity.

00:04:21

As you can see, in the most recent 2020 Census, all those mixed criteria were still there. Another pivotal change towards the measurement of diversity happened in the 2000 Census where the racial categories became a multiple-choice question, with a mark one or more races, which has been the question format since then.

00:04:47

We can see it in the 2020 census there. However, the most significant change in the 2020 was the increased focus on the nativity line. Here you see the Census started to ask about the origins for the white and black categories, and there are also more examples of nationalities for other categories.

It is important to understand that racial and ethnic self-identification, which is what we ask in the Census and on our surveys, is highly personal and can change as one’s relationship with our own identity changes, so by focusing on nationality and country of origin the census seems to be trying to put some distance from the color line and find a way to promote a post-racial society that is based on our geographic origins while reflecting the changes in the population.

Challenges to Measuring Racial Diversity

00:05:46

If you check the definition of each category here, on the right column, you can see the greater focus on nationality or the nativity line as the population has grown mainly thanks to immigration. The personal and evolving nature of racial self-identification influenced by the nativity line is posing big challenges for how we measure race in surveys to represent diversity.

I want to show you how this may be affecting the responses from non-racial [non-white] and ethnic groups in market research surveys.

Asians in America

00:06:22

Next, in the case of Asians, back in 2012 the Pew Research Center did a study showing that the Asian or Asian American label didn’t resonate with people from countries from the Asian continent. Most people from those countries describe themselves by the country of origin, and as you saw in the Census form, there is no Asian American category. They just listed nationalities, so the Asian category is really an aggregate of nationalities.

Hispanics in America

00:06:53

If we look at Hispanics, the Pew Center also found that many don’t see a shared culture with other Hispanics in the U.S., and they are more likely to select “some of the race” in any survey. We all have seen that, and this is because the race categories don’t resonate with them either. Like Asians, many Hispanics also describe themselves by the country of origin. At the same time, the Hispanic identity fades with generations which confirms the impact of country of origin.

By the way, if you have been thinking about using Latinx instead of Hispanic or Latino in your surveys, please don’t. Only 14 percent of Hispanics have heard of that label, and only 3 percent use it. You can add it to increase reach among younger Hispanics but don’t use it as a substitute.

Black America

00:07:48

In the case of black Americans, recent research by Gallup indicated that most say it doesn’t make a difference if they are referred to as blacks or African Americans.

However, other research has shown that these terms have different associations for many black Americans. In qualitative research, the authors of the book “What’s Black About It?” found that the term Black is often associated with intimacy and familiarity with each other, while the African American label is preferred for public and formal references.

At the same time, black Hispanics and immigrants from Africa and the Caribbean, a group that has grown, doubled since 2000 doesn’t often feel represented by the term African American so again nationality seems to be having more weight given the term “African” in the label.

As you can see the diversity within the established racial and ethnic groups, as we know them, is growing mainly in connection with immigration.

Race Questions – Practical Survey Design Recommendations

00:08:55

To conclude this section about racial categories, and before I go to gender and sexual orientation, I wanted to share some practical recommendations to create more inclusive racial categories and analysis in surveys.

Please, consider not only self-identification but also nationality, generation, number of years lived in the U.S., education in the U.S., languages poke at home and other situations, and media consumption. These variables are strongly correlated with racial and ethnic identification.

Also, allow for multiple-choice, use several labels to describe a group, if there is a fragmentation in their use, test preferences for labels to avoid being offensive, if possible, add examples of nationalities, and try to analyze by different combinations of race and ethnicity.

Sex, Gender, and Sexual Orientation Concepts

00:09:59

Now, sex, gender, and sexual orientation. The measurement of these concepts has been studied by federal agencies concerned with the health of and benefits for the LGBT community. There are still many versions out there of how to ask questions about this.

The government seems to be the one with the resources to do the research that can help standardize these questions, so I suggest you monitor their work.

To ask these questions, we need to consider the definitions of some key terms.

Sex is used to refer to biological traits like male and female and intersex.

Gender is about the internal sense of gender or belonging to a gender community, so this is about identity. Sexual orientation is more complicated as it has three components: sexual attraction, sexual behavior, and also self-identification with an orientation.

Key Challenge to Measuring Gender Diversity

A big challenge here is to find labels to capture gender diversity and that resonate with most people. There are many labels circulating in the media and social media, but they can mean different things to different people. Even across federal agencies, there are variations of how these questions are asked.

00:11:25

Of all the versions I have seen, I like the format recommended by the UCLA William Institute of Law. They recommend asking questions separately, about gender identity and sexual orientation, with clear, common terms, and with an option to add your own labels. They also wrote a paper that I recommend reading explaining the rationale behind their recommendation for the wording and order of these questions. I have not seen that explained for other versions used out there.  

Gender Questions – Practical Survey Design Recommendations

00:11:59

Here are some practical applications. Don’t assume that terms that seem popular, like for example, non-binary, or queer, or other, mean the same thing to everybody so try to use tested and validated terms. This is a very sensitive topic.

Please separate gender identity from sexual orientation. They are not the same.

Use a sequence of questions. Don’t try to compress elements of gender identity and sexual orientation all in one question to save time and money which tends to create confusion for respondents and confound variables for analysis.

Use follow-up open-ended questions when categories are not chosen so people can write in their own labels.

Always, always include Refuse and Don’t know options to respect privacy and allow for someone who doesn’t know how to answer these questions.

Recommended Reading

Well, that’s all I have for you today, but before I finish my section, I want to leave you with some recommended reading about these topics that may help you in your survey design. The next one also shows the books about gender, or sources about gender. Thank you.