Multi-Level Modeling: Monte Carlo Simulations

The Monte Carlo simulations reported in the paper were performed as follows.

1. Data were simulated in MLn, with a pre-specified variance-covariance matrix. The variance and covariance quantities in this matrix determine whether the matrix is spherical, and whether the data have homogeneous variance. For a particular variance-covariance matrix, 100 data sets were simulated. Each data set was simulated to have n=24 trials, k=3 conditions, and j=24 participants (1728 observations). Condition means were simulated to be {-0.2, 0, +0.2}. Total variance was simulated to be unity.
   For each data set, the fixed effect was tested (using chi square, with df=2, alpha=.05, critical value is 5.99) and the result was logged (example log).
   The 100 data sets were written as a large transposed matrix (100x1728, row-by-row) to a text file (example). Simulations were controlled by macro files for MLn (example macro).
2. The transposed matrix was re-transposed to a conventional data layout, with rows representing observations and column representing data sets (1728x100, column-by-column, example). This was done in S-Plus 2000 using this script.
3. Data were then read into SPSS 11.0 in multivariate layout, with 72 observations (columns) per participant and 100 (data sets) x 24 (participants) = 2400 rows.
   Data were re-partitioned into chunks corresponding to separate data sets, and fed into a repeated-measures ANOVA using the GLM command for repeated measures. This was done in SPSS 11.0 using this example syntax script and output mode set to "draft".
4. The resulting output (LARGE example) was copied into a plain-text editor, and sorted. The 100 test statistics for the 100 data sets thus were grouped together. These 100 (text lines reporting) statistics were re-sorted and classified by their significance value. Finally, the number of significant test statistics (at alpha=.05) was noted down.
5. These numbers were edited into an S-Plus script that constructs the corresponding figure in the paper. The figure was saved in native S-Plus format, in Windows Metafile format and in Encapsulated PostScript, for further text processing.