There’s Only One Way to Get Into This Club, and Wishing Won’t Make It So

When you’re walking in your local mall and someone asks you to respond to a quick survey question, do you provide an answer?  When you’re surfing the internet and a one-question survey “pops up” on the side of your screen, do you provide an answer?  When you’re watching your favorite network news program and the anchor asks you to text or tweet a response to the question on the screen, do you provide an answer?

If you do, you’re participating in what is known as convenience sampling, a data collection procedure in which survey respondents are included in the sample at the convenience of the researcher (and the respondent).  This sampling methodology is in sharp contrast to (simple) random sampling, a data collection procedure in which the researcher insures each member of the population of interest has an equal probability of being included in the sample.

Image Source:

Let Me Choose My Own Fate, I Promise I Won’t Bias You (Wink Wink)

One of the problems with convenience sampling is that it allows respondents to self-select, meaning they get to choose whether to participate.  Self-selection is a problem because it means the people who choose to participate may be systematically different from those who choose not to participate.  And if the respondents are different from the non-respondents, then the information you glean from the respondents isn’t representative of the entire population of interest (i.e. it’s biased).

Think about the political opinion polls you see on network news programs.  Who do you think responds to those questions?  It’s people who are highly opinionated and passionate about the topic.  Anyone who is relatively indifferent or apathetic doesn’t take the time to respond to the poll.  Since the people who respond are systematically different from the people who don’t respond, the opinions collected by the poll aren’t representative of the U.S. population at-large.

Image Source:

If you’re only interested in the poll results because you find it entertaining to see which highly opinionated, passionate group spends the time to text or tweet their responses, there’s no real problem; however, if you’re interested in collecting public sentiments that are representative of the U.S. population at-large, an on-air political opinion poll is just about as worthless as it gets.

Our Program Helped the People Who Participated, So the Program Must Be Effective…Right?

Self-selection isn’t only a problem for survey research.  It’s also a problem for program evaluation, which means both government agencies and companies need to understand self-selection and how to avoid or correct for it.  Let’s say a government agency or company has developed a weight loss program and wants to be able to claim (legitimately) that the program can help anyone lose weight (i.e. that the program is effective).

One (unscientific) way for the government agency or company to evaluate the program is to advertise for participants, enroll everyone who responds to the advertisement, weigh them before they go through the program, weigh them again after they complete the program, and finally analyze the difference between the pre-program and post-program weights.

There are many problems with the design of this program evaluation (e.g. where’s the control group?), but one important problem is that participants self-select into the program.  People who choose to participate in a weight loss program are (very likely) different from people who don’t voluntarily choose to participate.  If the two groups are systematically different, then the “results of the program” (e.g. the average amount of weight lost by the participants) may not be due to the program at all.

Image Source:

Without using a (scientific) random sampling methodology to reduce the probability of self-selection or using advanced statistical estimation techniques to correct for self-selection, it wouldn’t be appropriate for the government agency or company to claim that the program is effective.

Self-Select into Good Decision Making By Checking For Self-Selection

We’re often interested in acquiring information from a sample to learn about a specific population of interest.  There are many ways to select people for a sample, including convenience sampling and simple random sampling.  One problem with convenience sampling and other methodologies that allow individuals to choose whether to be included in the sample is self-selection.

Self-selection is a problem because those who choose to participate may be systematically different from those who don’t participate.  If that’s the case, the results of the survey, poll, or program evaluation won’t be representative of the specific population of interest.

Remember, always make sure you understand the sampling methodology, and whether self-selection might be an issue, before you form an opinion or make a decision based on information from a survey, poll, or program study; otherwise you could end up forming an inaccurate judgment or making an unwise decision based on flawed (i.e. unrepresentative) information.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s