Statistical Power for ANOVA / ANCOVA / Repeated measures ANOVA

Statistical Power for ANOVA, ANCOVA and Repeated measures ANOVA

XLSTAT offers tools to apply analysis of variance (ANOVA), repeated measures analysis of variance and analysis of covariance (ANCOVA). XLSTAT-Power estimates the power or calculates the necessary number of observations associated with these models.

What is statistical power for ANOVA, ANCOVA and Repeated measures ANOVA?

What is statistical power?

When testing a hypothesis using a statistical test, there are several decisions to take:

• The null hypothesis H₀ and the alternative hypothesis H_a.
• The statistical test to use.
• The type I error also known as alpha. It occurs when one rejects the null hypothesis when it is true. It is a significance level set a priori for each test and is usually set to 5%.

The type II error or beta is less studied but is of great importance. In fact, it represents the probability that one does not reject the null hypothesis when it is false. We cannot fix it up front, but based on other parameters of the model we can try to minimize it. The power of a test is calculated as 1-beta and represents the probability that we reject the null hypothesis when it is false.

We therefore wish to maximize the power of the test. The XLSTAT-Power module calculates the power (and beta) when other parameters are known. For a given power, it also allows to calculate the sample size that is necessary to reach that power.

What are ANOVA, ANCOVA and repeated measures ANOVA?

Analysis of variance (ANOVA) is a tool used to partition the observed variance in a particular variable into components attributable to different sources of variation.

If p is the number of factors, the anova model is written as follows:

y_i = β₀ + ∑_j=1...q β_k(i,j),j + ε_i

where y_i is the value observed for the dependent variable for observation i, k_(i,j) is the index of the category (or level) of factor j for observation i and ε_iis the error of the model.

The hypotheses used in ANOVA are identical to those used in linear regression: the errors ε_i follow the same normal distribution N(0,s) and are independent. It is recommended to check retrospectively that the underlying hypotheses have been correctly verified. The hypothesis of normality of can be checked by analyzing certain charts on residuals or by using a normality test. The independence of the residuals can be checked by analyzing certain charts or by using the Durbin Watson test.  

ANCOVA (ANalysis of COVAriance) can be seen as a mix of ANOVA and linear regression as the dependent variable is of the same type, the model is linear and the hypotheses are identical. In reality it is more correct to consider ANOVA and linear regression as special cases of ANCOVA.

If p is the number of quantitative variables, and q the number of factors (the qualitative variables including the interactions between qualitative variables), the ANCOVA model is written as follows:

yi = β0 + ∑j=1...p βjxij + ∑j=1...q βk(i,j),j + εi

where yi is the value observed for the dependent variable for observation i, xij is the value taken by quantitative variable j for observation i, k(i,j) is the index of the category of factor j for observation i and εi is the error of the model.

The hypotheses used in ANOVA are identical to those used in linear regression and ANOVA: the errors εi follow the same normal distribution N(0,s) and are independent. To check them, you can use the same tests as mentioned above. 

Repeated measures ANOVA is a regular one in which the measures are repeated several times on a same data sample. That way, we can take the repetition effect into account. Several result tables are displayed for each repetition, as well as a table that enables us to sum up the repetition and group effects.
 

The statistical power calculations are usually done before the experiment is conducted. The main application of power calculations is to estimate the number of observations necessary to properly conduct an experiment. XLSTAT can therefore test:

• In the case of a one-way ANOVA or more fixed factors and interactions, as well as in the case of ANCOVA:
  • H₀: The means of the groups of the tested factor are equal.
  • H_a: At least one of the means is different from another.
• In the case of repeated measures ANOVA for a within-subjects factor:
  • H₀: The means of the groups of the within subjects factor are equal.
  • H_a: At least one of the means is different from another.
• In the case of repeated measures ANOVA for a between-subjects factor:
  • H₀: Les The means of the groups of the between subjects factor are equal.
  • H_a: At least one of the means is different from another.
• In the case of repeated measures ANOVA for an interaction between a within-subjects factor and a between-subjects factor:
  • H₀: The means of the groups of the within-between subjects interaction are equal.
  • H_a: At least one of the means is different from another.

Options for Statistical Power for ANOVA/ANCOVA/Repeated measures ANOVA in XLSTAT

Similarly to the Statistical Power for Cox model, you can choose to calculate the size of your data sample based on a set power, or to calculate the power reachable when using a set sample size. 

In this analysis, you have to specify which statistic test you want to study. Depending on your choice, you will have to specify several parameters sur as the alpha (significance threshold for type I error), the degrees of freedom of your model, the number of groups included in it...

Effect size for ANOVA, ANCOVA and Repeated measures ANOVA

This concept is very important in power calculations. Indeed, Cohen (1988) developed this concept. The effect size is a quantity that will allow calculating the power of a test without entering any parameters but will tell if the effect to be tested is weak or strong. For example, in the context of an ANOVA-type model, conventions of magnitude of the effect size are:

• f=0.1, the effect is small.
• f=0.25, the effect is moderate.
• f=0.4, the effect is strong.

XLSTAT-Power allows you to enter directly the effect size but also allows you to enter parameters of the model that will calculate the effect size. We detail the calculations below:

• Using variances: We can use the variances of the model to define the size of the effect. With var_explained being the variance explained by the explanatory factors that we wish to test and var_error being the variance of the error or residual variance, we have: f = √var_explained / var_error
• Using the direct approach: We enter the estimated value of eta² which is the ratio between the explained variance by the studied factor and the total variance of the model. For more details on eta², please refer to Cohen (1988, chap. 8.2). We have: f = √η² / (1 – η²)
• Using the means of each group (in the case of one-way ANOVA or within subjects repeated measures ANOVA): We select a vector with the averages for each group. It is also possible to have groups of different sizes, in this case, you must also select a vector with different sizes (the standard option assumes that all groups have equal size). We have: f = √Σ_i (m_i - m)² / k / SD_intra with m_i mean of group i, m mean of the means, k number of groups and SD_intra within-group standard deviation.
• When an ANCOVA is performed, a term has to be added to the model in order to take into account the quantitative predictors. The effect size is then multiplied by f = √1 / (1 – ρ²) where ρ² is the theoretical value of the square multiple correlation coefficient associated to the quantitative predictors.

Once the effect size is defined, power and necessary sample size can be computed.

Results for the Statistical Power of ANOVA, ANCOVA and repeated measures ANOVA in XLSTAT

Calculations for the Statistical Power of ANOVA, ANCOVA and Repeated measures ANOVA

The power of a test is usually obtained by using the associated non-central distribution. For this specific case we will use the Fisher non-central distribution to compute the power.

We first introduce some notations:

• NbGroup: Number of groups we wish to test.
• N: sample size.
• NumeratorDF: Numerator degrees of freedom for the F distribution (see below for more details).
• NbRep: Number of repetition (measures) for repeated measures ANOVA.
• ρ: Correlation between measures for repeated measures ANOVA.
• ε: Geisser-Greenhouse non sphericity correction.
• NbPred: Number of predictors in an ANCOVA model.

For each method, we give the first and second degrees of freedom and the non-centrality parameter:

• One-way ANOVA: DF₁ = NbGroup – 1; DF₂ = N – NbGroup; NCP = f²N
• ANOVA with fixed effects and interactions: DF₁ = NumeratorDF; DF₂ = N – NbGroup; NCP = f²N
• Repeated measures ANOVA within-subjects factor: DF₁ = NbRep – 1; DF₂ = (N – NbGroup)(NbRep – 1); NCP = f²*N*NbRep*ε / (1 – ρ)
• Repeated measures ANOVA between-subjects factor: DF₁ = NbGroup – 1; DF₂ = N – NbGroup; NCP = f²*N*NbRep / [1 + ρ(NbRep – 1)]
• Repeated measures ANOVA interaction between a within-subject factor and a between-subject factor: DF₁ = (NbRep – 1)(NbGroup – 1); DF₂ = (N – NbGroup)(NbRep – 1); NCP = f²*N*NbRep*ε / (1 – ρ)
• ANCOVA: DF₁ = NumeratorDF; DF₂ = N – NbGroup – NbPred – 1; NCP = f²N

Calculating sample size for ANOVA, ANCOVA and Repeated measures ANOVA taking into account the statistical power

To calculate the number of observations required, XLSTAT uses an algorithm that searches for the root of a function. It is called the Van Wijngaarden-Dekker-Brent algorithm (Brent, 1973). This algorithm is adapted to the case where the derivatives of the function are not known. It tries to find the root of:

power (N) - expected_power

We then obtain the size N such that the test has a power as close as possible to the desired power.

Numerator degrees of freedom for ANOVA, ANCOVA and Repeated measures ANOVA

In the framework of an ANOVA with fixed factor and interactions or an ANCOVA; XLSTAT-Power proposes to enter the number of degrees of freedom for the numerator of the non-central F distribution. This is due to the fact that many different models can be tested and computing numerator degrees of freedom is a simple way to test all kind of models. Practically, the numerator degrees of freedom is equal to the number of group associated to the factor minus one in the case of a fixed factor. When interactions are studied, it is equal to the product of the degrees of freedom associated to each factor included in the interaction.

Suppose we have a 3-factor model, A (2 groups), B (3 groups), C (3 groups), 3 second order interactions A*B, A*C and B*C and one third-order interaction A*B*C We have 3*3*2=18 groups. To test the main effects A, we have: NbGroups=18 and NumeratorDF=(2-1)=1. To test the interactions, eg A*B, we have NbGroups=18 and NumeratorDF=(2-1)(3-1)=2. If you wish to test the third order interaction (A*B*C), we have NbGroups=18 and NumeratorDF=(2-1)(3-1)(3-1)=4. In the case of an ANCOVA, the calculations will be similar.