Why do a poll?
Polling is one way, among many, to actively listen to the people we serve – in this case, a representative sample of all Coloradans. By having 20-minute conversations with a sample of people who look like the population of our state, we can better understand the priorities and attitudes of a wide range of Coloradans. The Colorado Health Foundation’s annual poll aims to take the pulse of Coloradans on a range of important health issues each year to inform policy far into the future.
Through Pulse: The Colorado Health Foundation Poll, we interviewed more than 2,450 Colorado adults in August of 2021, a large sample that allows us to dig into differences in perspectives based on race, ethnicity, income, geography, and a host of other factors. In fact, one of the upsides of this kind of quantitative research is that it allows us to discern patterns in what different groups of Coloradans are thinking or experiencing.
Through the poll, we learn a lot about how people are feeling and what priorities they have – and we share those with policymakers, community leaders, the media and anyone else who is interested. These findings also inform our work to advance health equity.
Polling has its limitations, too – namely that we only have about 20 minutes with each Coloradan we interview. That’s why there are other research methods we use to understand even more about Coloradans’ views and what drives those. For example, we’ve used qualitative research like focus groups to dive deeper into topics with smaller groups of people so we can really understand the mindsets of Coloradans.
How were people chosen to participate in the poll?
In 2021, Pulse gathered responses from two samples.
The primary way in which we get responses to the poll is through a probability sampling methodology. That means that participants were randomly selected from a list of all Colorado adults with addresses with every adult having the same chance of being selected. Addresses were matched to phone numbers and email addresses, and participants were contacted by phone call, text message, email and mail with an invitation to complete the survey. We use data from the U.S. Census Bureau to ensure that the sample of people we interviewed matches the characteristics of the entire population of adults who call the state “home.” In 2021, the total number of respondents in the probability-based sample is 2,392.
The secondary sample uses a convenience sampling methodology, where respondents come from a group of people easy to contact or to reach. This component of our methodology is new in 2021, and we added it to increase the number of Asian Americans and Pacific Islanders (AAPI) as well as Native Americans interviewed. In our 2020 poll, the number of interviews done with these populations was too low to report data that might be representative of their perspectives and experiences. For 2021, we partnered with community organizations that serve AAPI Coloradans and Native Americans living in Colorado. Those organizations sent an invitation to participate in Pulse out to their networks via email, social media and text. Because of these partnerships, we were able to increase interviews with these populations. In 2021, the total number of respondents who completed the survey from this kind of outreach is 101.
Importantly, we specifically chose to conduct this research with adults – not just voters – living in Colorado. This decision follows our commitment to listen to all Coloradans, not just those who are registered or likely to vote. It also distinguishes our approach from much of the other polling that is conducted in our state on behalf of candidates or issue campaigns. Our purpose is not to predict the outcome of elections, but rather to listen to the people of our state and accurately reflect what Coloradans are thinking and experiencing.
What about people experiencing homelessness who don’t have addresses or people who are unstably housed and move frequently?
People experiencing homelessness and housing insecurity are among some of the most challenging folks to include in quantitative research projects. With an address-based sample, it’s likely we didn’t reach many people who have been chronically homeless. This is a limitation of our methodology, and it’s one that challenges us to use other methodologies to listen to people experiencing homelessness in order to understand their opinions and experiences. Even with the limitations of this methodology, Pulse found that a significant number of Coloradans are struggling to afford housing in 2021 – with 21% of Coloradans worried they might lose their home in the next year because they can’t afford the rent or mortgage. That number jumps to 44% for Coloradans living on low incomes – a reminder that concerns about housing affordability and stability are not limited to those currently experiencing homelessness.
What are oversamples, and why do you use them?
Through Pulse: The Colorado Health Foundation Poll, we seek to hear from all Coloradans, even population groups that make up only a small percentage of residents of our state overall. That’s why we oversampled Black/African American adults, Asian Americans and Pacific Islanders, and Native Americans. Importantly, because Hispanic Coloradans make up more than 20% of the state’s total population, we are able to achieve a large number of interviews without oversampling.
Here’s an example of how oversampling works: Black people make up around 4% of Colorado’s population; in a purely representative sample, that means they’d make up roughly 4% of all respondents. Given our commitment to improving the health of communities of color, we interviewed more Black Coloradans than just 4% of our sample so that we’d better understand their experiences, priorities and opinions.
It can be confusing to understand how oversamples like this don’t skew a poll’s overall results away from being representative of Colorado’s population as a whole. But they don’t – and that’s because of an additional step to the analysis process called “weighting.” According to the American Association for Public Opinion Research, “Weighting adjusts the poll data in an attempt to ensure that the sample more accurately reflects the characteristics of the population from which it was drawn. Weighting is used to adjust the relative contribution of the respondents, but it does not involve any changes to the actual answers to survey questions.”
In short, we do more interviews with populations that are mission-centric for us so that we’re better able to understand their opinions when we look at how they responded. This allows us to have greater confidence that we’re accurately reflecting the views of these groups of Coloradans when we report data about their opinions and experiences as a group. Then, when we’re reporting data about Coloradans overall, the weighting process balances everything out so that the topline numbers aren’t skewed by our oversamples.
Is there a way to look at the responses of people who are undocumented?
Pulse: The Colorado Health Foundation Poll asks participants if they were born inside of or outside of the United States. The full results show how those two sub-groups compare in their answers to each question. The poll does not ask about documentation status for people not born in the U.S. In our current political environment, immigrants – both documented and undocumented – may live in fear of deportation or other consequences. Asking about immigration status could increase that fear and likely make many immigrants decline to participate in the poll. While we would love to know the priorities and experiences of undocumented immigrants living in Colorado, we do not want to create additional fear by asking about immigration status in our poll. We are challenging ourselves to listen to and explore the perspectives of people who are undocumented in other ways.
How can the data be broken down geographically? Can I get data specific to a county/region of the state?
A poll the size of Pulse – 2,493 respondents in 2021 – allows for some geographic breakdowns. On our interactive data dashboard, you can see the data divided by region: Eastern Plains, Colorado Springs, Pueblo, Larimer and Weld, Denver Metro Area, and Western Slope. In addition to regions, the crosstabs show the data broken out by the counties with the greatest numbers of respondents. County-specific data is available for any county where we gathered more than 100 responses. In 2021, those counties include: Adams, Arapahoe, Boulder, Denver, El Paso, Jefferson, Larimer, Pueblo, and Weld.
What’s a “margin of error,” and why does it change depending on what part of a poll we’re reading?
When conducting a poll, we only talk to a sample of the population, and while the sample represents the population, we’d get a truer result if we interviewed every single person in the population. The margin of error describes how close we can reasonably expect a survey result to be to the true population value if we were able to hear from every single person. A margin of error of plus or minus 3 percentage points at the 95% confidence level means that if we fielded the same survey 100 times, we would expect the result to be within 3 percentage points of the true population value 95 of those times.
In a poll, the most commonly cited margin of error applies to the entire sample (for Pulse in 2021, that’s 2,493 adults living in Colorado with a margin of error of +/-2.7%). When we look at sub-groups within the population – such as Hispanic/Latinx adults living in Colorado – the margin of error gets bigger because the sample size gets smaller. While there are a number of factors that influence the margin of error, the predominant factor is sample size – the bigger it is (i.e., the more people you hear from), the smaller the margin of error (i.e., the closer you’ll be to representing the results if you talked to every single person).
Haven’t election polls proven that polling isn’t reliable?
Pulse: The Colorado Health Foundation Poll doesn’t aim to predict the results of any election. Interviews are conducted with Colorado adults, not just voters, because our goal is to know more about the experiences and feelings of all Coloradans to help us bring health closer in reach.
We understand that the experiences of recent polling trying to predict election outcomes and failing to do so has caused a lot of people to feel skeptical. The American Association for Public Opinion Research (AAPOR) has evaluated electoral polling from 2016 and 2020 to advise pollsters on how to avoid errors in the future. You can learn more about that work here.
Pulse: The Colorado Health Foundation Poll takes a number of steps to ensure that we’re accurately reflecting the opinions of Coloradans. First, Pulse is a multi-modal survey that reaches and interviews respondents on landlines, via texts and calls to cell phones, and through the mail. Second, Pulse is in the field with interviews for a month, allowing more time to get a representative sample of Coloradans to participate. Third, we interview respondents in both English and Spanish so the poll is more accessible. Finally, we ask Coloradans about the issues that are impacting their lives – not partisan politics – so they can trust the questions we’re asking and are more willing to participate.
What about people who don’t have landlines or refuse to answer the phone?
To be sure, one of the most significant challenges facing the polling industry is the reality that people often don’t answer their phones. Pulse seeks to mitigate this challenge by contacting Coloradans via multiple modes. We send emails, text messages and postcards – in addition to calling – in an effort to reach a representative sample of our state.
Can we really trust these results? Does The Colorado Health Foundation want to bias the responses to show support for specific policies we are advocating for?
Pulse: The Colorado Health Foundation Poll is committed to transparency; that’s why we’ve released everything – every question we asked and all the results we collected. The poll is also conducted as a partnership with both Democratic and Republican pollsters, and we worked hard to make sure our questions were presented in a balanced way. The poll has no predetermined outcome, and the data speaks for itself.
What languages was the poll conducted in?
Respondents were able to complete the poll in two languages: English and Spanish. According to U.S. Census Bureau data, English is the most spoken language in the state, and Spanish is the second most spoken language. We conducted the poll in these two languages to ensure we could reach a representative sample of adults in Colorado. The results of the poll are also available in both English and Spanish.
Although our sample is representative, we know that we weren’t able to include Coloradans who don’t speak English or Spanish. German and Chinese (including Mandarin, Cantonese) are the next two most-spoken languages in the state – each with fewer than 24,000 speakers. Because we want to ensure that Pulse reflects the priorities of all Coloradans, we are considering what additional languages may need to be integrated into the various ways we listen to Coloradans.