In statistics, the power of a binary hypothesis test is the probability that the test correctly rejects the null hypothesis () when a specific alternative hypothesis () is true. It is commonly denoted by , and represents the chances of a true positive detection conditional on the actual existence of an effect to detect. Statistical power ranges from 0 to 1, and as the power of a test increases, the probability of making a type II error by wrongly failing to reject the null hypothesis decreases.
Notation
This article uses the following notation:
β = probability of a Type II error, known as a "false negative" 1 − β = probability of a "true positive", i.e., correctly rejecting the null hypothesis. "1 − β" is also known as the power of the test. α = probability of a Type I error, known as a "false positive" 1 − α = probability of a "true negative", i.e., correctly not rejecting the null hypothesis
Description
For a type II error probability of , the corresponding statistical power is 1 − . For example, if experiment E has a statistical power of 0.7, and experiment F has a statistical power of 0.95, then there is a stronger probability that experiment E had a type II error than experiment F. This reduces experiment E's sensitivity to detect significant effects. However, experiment E is consequently more reliable than experiment F due to its lower probability of a type I error. It can be equivalently thought of as the probability of accepting the alternative hypothesis () when it is true – that is, the ability of a test to detect a specific effect, if that specific effect actually exists. Thus,
If is not an equality but rather simply the negation of (so for example with for some unobserved population parameter we have simply ) then power cannot be calculated unless probabilities are known for all possible values of the parameter that violate the null hypothesis. Thus one generally refers to a test's power against a specific alternative hypothesis.
As the power increases, there is a decreasing probability of a type II error, also called the false negative rate () since the power is equal to 1 − . A similar concept is the type I error probability, also referred to as the false positive rate or the level of a test under the null hypothesis.
In the context of binary classification, the power of a test is called its statistical sensitivity, its true positive rate, or its probability of detection.
Power analysis
A related concept is "power analysis". Power analysis can be used to calculate the minimum sample size required so that one can be reasonably likely to detect an effect of a given size. For example: "How many times do I need to toss a coin to conclude it is rigged by a certain amount?" Power analysis can also be used to calculate the minimum effect size that is likely to be detected in a study using a given sample size. In addition, the concept of power is used to make comparisons between different statistical testing procedures: for example, between a parametric test and a nonparametric test of the same hypothesis.
Rule of thumb
Lehr's (rough) rule of thumb says that the sample size (each group) for a two-sided two-sample t-test with power 80% () and significance level should be:
where is an estimate of the population variance and the to-be-detected difference in the mean values of both samples. For a one sample t-test 16 is to be replaced with 8. The advantage of the rule of thumb is that it can be memorized easily and that it can be rearranged for . However, a full power analysis should always be performed to confirm and refine this estimate.
Background
Statistical tests use data from samples to assess, or make inferences about, a statistical population. In the concrete setting of a two-sample comparison, the goal is to assess whether the mean values of some attribute obtained for individuals in two sub-populations differ. For example, to test the null hypothesis that the mean scores of men and women on a test do not differ, samples of men and women are drawn, the test is administered to them, and the mean score of one group is compared to that of the other group using a statistical test such as the two-sample z-test. The power of the test is the probability that the test will find a statistically significant difference between men and women, as a function of the size of the true difference between those two populations.
Factors influencing power
Statistical power may depend on a number of factors. Some factors may be particular to a specific testing situation, but at a minimum, power nearly always depends on the following three factors: