Difference-in-difference power analysis

This is an interactive document to assess the power of a 2x2 interaction analysis for a linear model assuming a Gaussian family and a mixed design (i.e., one within-subjects factor and one between-subjects factor). This could be used to conduct a power analysis for a difference-in-difference study with two time points (e.g., before vs. after) and two groups (e.g., control vs. treatment).

To use this document, click on ‘Run Code’ in each code chunk to run the code. Make sure to run the code chunk in the ‘Parameters’ section before running the code chunks in the ‘Visualize parameters’ and ‘Power analysis’ section.

Parameters

For a 2x2 interaction with one within-subjects factor there are 12 parameters that need to be set:

  • The means in each group at each time point (parameters starting with M_)
  • The standard deviations in each group at each time point (parameters starting with SD_)
  • The sample size in each group (parameters starting with N_)
  • The correlations of the repeated measures in each group (parameters starting with r_)

To simplify this, you can assume that some parameters are identical to other parameters (e.g., set the same value for each standard deviation).

Use the code chunk below to tweak the parameter values or simply hit ‘Run Code’ to rely on the default values.

After the parameters section, the parameters are separately stored as means and variance-covariance matrices (called Sigma).

Visualize parameters

To help set the right parameter values, you can use the code here to visualize the data that the parameters produce. The code below simulates the data once, producing a sample of the data that will exactly match the parameter values. It then visualizes the means and distribution of data points.

If you want to set the means based on proportions of each Likert option, per group, you can use the extra section at the bottom to convert proportions to means.

Power analysis

Here we conduct the power analysis by repeatedly simulating (s times) data with the parameters previously specified. We extract the relevant p-value and store it so we can calculate the power as the number of times a significant effect is obtained divided by the number of simulations (s). You can increase s to obtain a more precise estimate of the power.

For a difference-in-difference analysis, the crucial test is that of the interaction term between the two conditions (control vs. treatment) and time (pre vs. post). The code below calculates the power for this test.

Caution

Note: Depending on the number of simulations, this can take a while to run.

Extra: Estimating mean differences

You can use the code below to help set the means for each group by converting proportions of responses (e.g., the proportion of participants in the treatment condition who selected a ‘4’) to means for use in the power analysis. Set the proportions of each response (ranging from 1 to 5) in each group and see what the mean is in each group and their difference.