evaluating statistical claims, studies, & experiments
On the SAT, you’ll be asked to interpret surveys, evaluate the validity of statistical claims, and distinguish between observational studies and experiments. The key: know how to generalize (and when you can’t), recognize bias, spot sampling mistakes, and understand the limits of a sample or study design.
samples, populations, and generalizing
Sample: A smaller group *actually studied* or surveyed.
Population: The bigger group you want to say something about.
Generalize: To make claims about the whole population based on your sample.
- You can only generalize to the group your sample represents. If you randomly survey 50 students from one school, you can talk about that school—but not all students everywhere.
- Random sampling is what makes generalization possible. If you only ask your friends, it’s a biased sample!
- On the SAT, questions will ask about the "largest population" to which a result can be applied. If your sample is limited, your claim must be too.
observational studies vs. experiments
- Observational Study: You collect data without changing anything. Example: Surveying teens about phone use.
- Experiment: You assign treatments or conditions to test cause-and-effect. Example: Giving half a group a new teaching method and comparing test scores.
- Key difference: Only randomized experiments can show cause-and-effect. Observational studies can show an association but not causation!
If nobody in the study was assigned to groups or given a treatment, it’s observational. If groups were randomly assigned and something was changed, it’s an experiment.
sampling bias, validity, and margin of error
- Sampling bias happens if the sample isn’t random or leaves out groups (ex: an online poll for smartphone users misses people without smartphones).
- Margin of error is a range that likely contains the true value for the population. If a survey says 65% ± 3%, the actual percent could be anywhere from 62% to 68%.
- SAT trap: Just because a result is true for the sample doesn’t mean it’s exactly true for everyone. Only random samples with a stated margin of error give you a range for the whole population.
statistical claims & what’s valid?
- If the sample isn’t random or is too small: You can’t trust the results for the population.
- If a study finds a relationship (ex: kids who eat breakfast score higher): Association doesn’t prove cause! Maybe kids who eat breakfast also get more sleep.
- If the claim is about cause-and-effect: Check if it was a randomized experiment (not just a survey).
- Generalizing beyond the group: Only generalize to the group that matches your sampling method.
Try these real SAT-style statistical reasoning questions:
Question 1
A sample of 40 fourth-grade students was selected at random from a certain school. The 40 students completed a survey about the morning announcements, and 32 thought the announcements were helpful. Which of the following is the largest population to which the results of the survey can be applied?
Question 2
Residents of a town were surveyed to determine whether they are satisfied with the concession stand at the local park. A random sample of 200 residents was selected. All 200 responded, and 87% said they are satisfied. Based on this information, which of the following statements must be true?
I. Of all the town residents, 87% would say they are satisfied with the concession stand at the local park.
II. If another random sample of 200 residents were surveyed, 87% would say they are satisfied.
Question 3
A survey was conducted using a sample of history professors selected at random from the California State Universities. The professors surveyed were asked to name the publishers of their current texts. What is the largest population to which the results of the survey can be generalized?
evaluating statistical claims & studies mastered
You now know how to spot valid claims, recognize the limits of studies, and avoid common SAT statistical traps.
Next Up
35: area and volume
master area, surface area, and volume for all shapes