Photo by Charles Deluvio on Unsplash
Using graphs to demonstrate the correlations
When conducting statistical analysis, especially during experimental design, one practical issue that one cannot avoid is to determine the sample size for the experiment. For example, when designing the layout of a web page, we want to know whether increasing the size of the click button will increase the click-through probability. In this case, AB testing is an experimental method that is commonly used to solve this problem.
Moving to the details of this experiment, you will first decide how many users I will need to assign to the experiment group, and how many we need for the control group. The sample size is closely related to four variables, standard error of the sample, statistical power, confidence level, and the effect size of this experiment.
In this article, we will demonstrate their relationships with the sample size by graphs. Specifically, we will discuss different scenarios with one-tail hypothesis testing.
Standard Error and Sample Size
The standard error of a statistic corresponds with the standard deviation of a parameter. Since it is nearly impossible to know the population distribution in most cases, we can estimate the standard deviation of a parameter by calculating the standard error of a sampling distribution. The standard error measures the dispersion of the distribution. As the sample size gets larger, the dispersion gets smaller, and the mean of the distribution is closer to the population mean (Central Limit Theory). Thus, the sample size is negatively correlated with the standard error of a sample. The graph below shows how distributions shape differently with different sample sizes:
As the sample size gets larger, the sampling distribution has less dispersion and is more centered in by the mean of the distribution, whereas the flatter curve indicates a distribution with higher dispersion since the data points are scattered across all values.
Understanding the negative correlation between sample size and standard error help conduct the experiment. In the experiment design, it is essential to constantly monitor the standard error to see if we need to increase the sample size. For example, in our previous example, we want to see whether increasing the size of the bottom increases the click-through rate. The target value we need to measure in both the control group and the experiment group is the click-through rate, and it is a proportion calculated as:
the standard error for a proportion statistic is:
The standard error is at the highest when the proportion is at 0.5. When conducting the experiment, if observing p getting close to 0.5(or 1-p getting close to 0.5), the standard error is increasing. To maintain the same standard error, we need to increase N, which is the sample size, to reduce the standard error to its original level.
Statistical Power and Sample Size
Statistical power is also called sensitivity. It is calculated by 1- β, where β is the Type II error. Higher power means you are less likely to make a Type II error, which is failing to reject the null hypothesis when the null hypothesis is false. As stated here:
In other words, when reject region increases (acceptance range decreases), it is likely to reject. Thus, Type I error increases while Type II error decreases. The graph below plots the relationship among statistical power, Type I error (α) and Type II error (β) for a one-tail hypothesis testing. After choosing a confidence level (1-α), the blue shaded area is the size of power for this particular analysis.
From the graph, it is obvious that statistical power (1- β) is closely related to Type II error (β). When β decreases, statistical power (1- β) increases. Statistical power is also affected to Type I error (α), when α increases, β decreases, statistical power (1- β) increases.
The red line in the middle decides the tradeoff between the acceptance range and the rejection range, which determines the statistical power. How does the sample size affect the statistical power? To answer this question, we need to change the sample size and see how statistical power changes. Since Type I error also changes corresponded with the sample size, we need to hold it constant to uncover the relationship between the sample size and the statistical power. The graph below illustrates their relationship:
When the sample size increases, the distribution will be more concentrated around the mean. To hold Type I error constant, we need to decrease the critical value (indicated by the red and pink vertical line). As a result, the new acceptance range is smaller. As stated above, when it is less likely to accept, it is more likely to reject, and thus increases statistical power. The graph illustrates that statistical power and sample size have a positive correlation with each other. When the experiment requires higher statistical power, you need to increase the sample size.
Confidence Level and Sample Size
As stated above, the confidence level (1- α) is also closely related to the sample size, as shown in the graph below:
As the acceptance range keeps unchanged for both blue and black distributions, the statistical power remains unchanged. As the sample size gets larger (from black to blue), the Type I error (from the red shade to the pink shade) gets smaller. For one-tail hypothesis testing, when Type I error decreases, the confidence level (1-α) increases. Thus, the sample size and confidence level are also positively correlated with each other.
Effect Size and Sample Size
The effect size is the practical significant level of an experiment. It is set by the experiment designer based on practical situations. For example, when we want to check whether increasing the size of the bottom in the webpage increases the click-through probabilities, we need to define how much of a difference we are measuring between the experiment group and the control group is practically significant. Is a 0.1 difference significant enough to attract new customers or generate significant economic profits? This is the question the experiment designer has to consider. Once the effect size is set, we can use it to decide the sample size, and their relationship is demonstrated in the graph below:
As the sample size increases, the distribution get more pointy (black curves to pink curves. To keep the confidence level the same, we need to move the critical value to the left (from the red vertical line to the purple vertical line). If we do not move the alternative hypothesis distribution, the statistical power will decrease. To maintain the constant power, we need to move the alternative hypothesis distribution to the left, thus the effective effect decreases as sample size increases. Their correlation is negative.
How to interpret the correlations discussed above?
In summary, we have the following correlations between the sample size and other variables:
To interpret, or better memorizing the relationship, we can see that when we need to reduce errors, for both Type I and Type II error, we need to increase the sample size. A larger sample size makes the sample a better representative for the population, and it is a better sample to use for statistical analysis. As the sample size gets larger, it is easier to detect the difference between the experiment and control group, even though the difference is smaller.
How to calculate the sample size given other variables?
There are many ways to calculate the sample size, and a lot of programming languages have the packages to calculate it for you. For example, the pwr() package in R can do the work. Compared to knowing the exact formula, it is more important to understand the relationships behind the formula. Hope this article helps you understand the relationships. Thank you for reading!
Here is the list of all my blog posts. Check them out if you are interested!