Evidence that -mixed- really is slower than competing packages

[paulvonhippel]

Over the years various users have posted to Statalist reporting that multilevel models were too slow to converge, if they converged at all. Other users posted suggestions to achieve faster convergence, but these rarely made much difference. Today I tripped over this paper by McCoach et al (2018), which confirmed that Stata really was much slower than other packages at fitting multilevel models.

Figures 5 and 6 show that Stata was 5 to 10 times slower than HLM or MPlus, and 2 to 3 times slower than SAS or lme4 in R.
Table 4 shows that Stata failed to converge 10 to 100 times more often than HLM or MPlus, but lme4 was even worse.

I should say that the paper was published in 2018 and used Stata 14.1. I am using Stata 16.1 but haven't noticed any improvement. Has anyone tried -mixed- in Stata 18?

[Leonardo Guizzetti]

In my experience -mixed- seems a littler faster from Stata 17 to 18, but I don't believe it's a substantial amount. The speed (or slowness, depending on one's perspective) doesn't much bother me for most applications because I will only run a limited number of models and my dataset sizes tend not to be very large. I do hesitate to run Monte Carlo simulations that require -mixed- though. In those cases I reach for SAS or Mplus because they'll be much faster (especially where REML and Kenward-Roger corrections are used). Is it perfect? No. But I can reasonably work around it with the tools as my disposal, which is a good skill to hone in as much as knowing one's primary software as best as possible.

[Joseph Coveney]

Has anyone tried -mixed- in Stata 18?

I don't disagree with the contention that Stata is much slower than at least Mplus, but the current release might not be quite so dismal as implied in that article.

It's just an N of 1 (the setup is too artificial for me to have interest in anything more extensive) and assumes that I've got their simulation conditions correct (their two figures of equations on Page 15 seem inconsistent, and the treatment in the passage at the top of Page 19 is a little hard to follow), but on my aging laptop I got speeds for Simulation Conditions 1 and 15 of 18 seconds and 13 seconds, respectively, which are in the neighborhood of R's lme4 shown in their Figure 6. Again, single replication on a different machine, but certainly not the ranges of 100 seconds and 40 seconds shown for Stata in Figures 5 and 6.

Do-file and log-file attached.

If anyone has the inclination to go for 500 replications each, then I recommend cutting off the iterations at 10 or so for Case 1 using the iterate() option. These things are typically going to converge in about that if they're going to converge at all to anything worth waiting for, and in real practice, it makes no sense to sit there and stare at the same warning message and log-likelihood value roll by for 300 iterations, especially if you're just doing a specification search on which predictors warrant random slopes. Also, the nostderr option will save time for Case 1, too, inasmuch as Stata's not going to report standard errors, anyway, for the random effects parameters in that case.

[Joseph Coveney]

It dawned on me afterward that a portion of the difference in speed might result from different "flavors" of Stata. The authors don't go into detail in the article about the "flavor" of Stata Release 14.1 that they used or the hardware (number of cores or processors). I don't have access to the online appendix, but they might have covered such details there.

[paulvonhippel]

No,I don't think the problem was that the authors used the wrong flavor of Stata. I use Stata MP and I routinely encounter slow runtimes and nonconvergence. It sounds like Joseph Coveney 's machine is faster than the machine that the authors were using 5 years ago, but that doesn't change the fact that Stata runs these models much slower and with more convergence problems than MPlus and HLM. The conditions that the authors simulated in 2018 may run in less than 20 seconds on a more recent computer, but I routinely run into situations where Stata's -mixed- runs for 15 minutes or more, or just doesn't converge.

After years of user complaints, now verified by simulations, I doubt users who report problems with Stata's -mixed- are doing something wrong. The problem is with Stata's implementation of -mixed-.

I would recommend that StataCorp or an enterprising user take a look under the hood, see what -mixed- is doing less efficiently than competing software, and publish a command that runs faster.

[Erik Ruzek]

I believe Stata has gotten faster with v17+, which uses the Intel MKL libraries for computation. From here:

Last, but not least, the mixed command for fitting multilevel mixed-effects models is faster. In our timings, models with 10,000 panels, 10 time periods, and 5 random slope parameters run 2 to 3 times faster in Stata 17 than in Stata 16. Similar speed improvements occurred for different numbers of panels, time periods, and slope coefficients.

Paul, have you looked into MLwiN and the associated Stata program runmlwin? The cost of MLwiN is not too high and the license is lifetime w/ free upgrades as they get pushed out. I compared the timings of Joseph's code (McCoach20231011.do) using Stata 16 (4-core) mixed to MLwiN via runmlwin and got the following:

timer list
*Condition 15
   1:     20.67 /        1 =      20.6710   // Stata 16 mixed
   2:      0.85 /        1 =       0.8480    // MLwiN 3.05 via runmlwin
*Condition 1   
   3:     37.11 /        1 =      37.1080  // Stata 16 mixed
   4:      0.86 /        1 =       0.8610  // MLwiN 3.05 via runmlwin

I've used MLwiN via runmlwin to run mixed models on data with millions of records that mixed would choke on. That said, MLwiN cannot run the full variety of models that mixed can. For example, models with alternative covariance structures are severely limited in MLwiN. But MLwiN also has a really fast Gibbs sampler for outcomes and models that maximum likelihood has troubles with. I have gotten many times my money worth out of my MLwiN license.

[paulvonhippel]

Thanks, Eric! Very helpful. Your comparison of -mixed- and -runmlwin- running on the same data and computer show that -runmlwin- is 24 to 43 times faster! Maybe that will inspire Stata to improve -mixed-.

In 2004-2006, Sean Reardon published a package that accomplished something similar by calling HLM from Stata:

https://econpapers.repec.org/softwar...de/s448001.htm

But I believe it's gone out of date with new versions of HLM and Stata. sean reardon , how do you fit these models nowadays?

Best,
Paul

[paulvonhippel]

On the subject of MLWin's cost, the price sheet says that a one-user license is £400 to £800 GBP, or about $500 to $1000 USD.

https://www.bristol.ac.uk/cmm/software/mlwin/ordering/

The low end of that range is comparable to the academic cost of Stata/MP.

Readers can form their own opinions about how much they value the improvement in runtime. It probably depends on what other packages they have available, and how time they spend waiting on -mixed-.

[Erik Ruzek]

Subjective, of course, but I am highly satisfied with the MLwiN + runmlwin combo. I purchased MLwiN in 2019 and they not only give you updates for free but they also have a great online message board for Q&A with the developers.

[Stephen Jenkins]

FWIW I ran Joseph Coveney's do-file, and got timings of

. 
. timer list
   1:     13.89 /        1 =      13.8930
   2:     15.61 /        1 =      15.6100

Stata/MP4 18.0 for Windows (64-bit x86-64).
Processor Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz, 3696 Mhz, 10 Core(s), 20 Logical Processor(s)

Joseph reported the following:

. timer list
   1:     12.93 /        1 =      12.9250
   2:     17.60 /        1 =      17.5970

[Ariel Linden]

Hi All,

I know that this thread was started 8 months ago, but I just came across it now. I would like to add that I use -mixed- a tremendous amount in my work where I have visits nested within patients who in turn are nested within doctors (or hospitals). I am using Stata MP with 12 cores (version 18) and over 600 g of RAM!

With sample sizes as small as 26,000, mixed takes over 2 weeks to run! Based on some other sample sizes, I have some analyses that have taken over 3 months! This is really a ridiculous when you think about it. I have to plan when I can run these jobs so that I don't need the server for that period of time!

In any case, I feel this is definitely something that StataCorp has to figure out. If there are programs available (such as MLwiN) that can complete the analysis in a fraction of the time, then I see no reason why Stata cannot figure this out.

Sorry for complaining!

Cheers,

Ariel