Making Mata libraries for multiple Stata versions, gracefully

[David Roodman]

I'm writing a package soon to be posted on SSC that will include an ado front end an mlib file. The problem I'm having is that I want the package to:

* Take advantage of a time-saving function introduced in Stata 13, panelsum().
* Be compatible with older versions of Stata, say, back to Stata 11.

I have written a function that checks the user's Stata version, uses panelsum() the version is >=13 and otherwise bypasses it with a Mata implementation of panelsum(), which is significantly slower. My idea s to compile that in Stata 11, and then share the resulting mlib for use in all versions 11 and higher.

As described, this actually doesn't work. It compiles, but fails at run time. Why? When I run my function in Stata 11, it tries to compile (or whatever the verb is) my function from object code to machine code and it chokes on the reference panelsum(), a function it does not know. It cannot link that reference. The fact that my function will never _call_ panelsum() in Stata 11 does not help.

I can fix that problem through indirection: create a function called _panelsum() that simply passes its arguments to panelsum(). My main function then refers to _panelsum() instead. At runtime, Stata 11 is happy with this because now all functions referenced by my main function clearly exist. And when it runs the main function, by design it never actually hits the call to _panelsum(), so it never wanders into that function, where it would again choke on the troublesome reference to panelsum().

But then I hit another problem. If I compile this in Stata 14, it runs as intended in Stata 14. If I compile this in Stata 11, it runs as intended in Stata 11. But if I compile this in Stata 11, it fails in Stata 14. Thus I cannot distribute my Stata 11-compiled mlib in the expectation that it will work with newer Statas.

What's going on? I guess when Stata 11 compiles my _panelsum(), with its undefined reference to future-built-in panelsum(), it produces object code that Stata 14 cannot comprehend at runtime. It does not recognize my intended reference to the new function.

All of which leads to my question: what is the most graceful way to produce an mlib compatible with multiple versions of Stata, yet taking advantage of built-in functions available in only some?

One elegant solution would to have -net install- run (compile) my mata file one time at installation, producing an mlib compatible with the user's Stata version. This would also let the user take advantage of any speed-ups in Mata made between version 11 and the user's version. But I am pessimistic that can be done.

Something else that might work is writing the ado code to check what version of Stata compiled it, and recompiling it if the version is old. This could be even better in that, if the user upgraded, the user's mlibs would be upgraded too. But I don't know how to do that.

I could also put the Mata code in the ado, so it compiles every time it is loaded. Ugly.

The fallback is to write to versions of the program, one for 11, one for 13. The two copies would be the same but all the functions in the 11 version would have "11" in their names. Each would be compiled in the corresponding Stata version. The ado front end would determine which to call. Workable but also ugly.

Any advice?
--David

[Niels Henrik Bruun]

Hi Dave
This is the exact same problem I had in http://www.statalist.org/forums/foru...sions-of-stata
Hope you find help there.

[David Roodman]

Thanks, Niels. In one of those posts, you write, "I think a solution could be to attach the Mata code in one or more do-file (some version dependent) and letting the ado be verifying the existence and version of the mlib before using it." How do you do check the existence and version of an mlib from an ado? I haven't found anything on that.
--David

[wbuchanan]

David Roodman you could write a simple .ado that compiles the library without the user being aware if the .mlib file does not exist. That's what I've been doing with some of my stuff lately (e.g., http://github.com/wbuchanan/brewsche...brewscheme.ado) to some degree. I've not added tests to check for an existing mata library file yet, but it wouldn't be much to add something like that to the existing code base. Then you wouldn't have to worry about asking the end user to compile things directly. You still get the benefit of a single compile but may need to do a bit more work if there are updated to the underlying mata libraries in the future.

[David Roodman]

Ah. Maybe this. Include in the library a function that captures the version in which it is compiled:

string scalar mymlibversion() return("`c(stata_version)'")

The "`c(stata_version)'" will be translated at compile time.

Then at run time the ado file can do:

mata st_local("mymlibversion", mymlibversion())
if `mymlibversion' != c(stata_version) {
  qui findfile "mycode.mata"
  run "`r(fn)'"
}

That's pretty graceful. You would compile and share this in the oldest version of Stata you were supporting.


For now, I have just put the full mata code in the ado file, so that it is compiled every time it is loaded (meaning I guess once per Stata session). I have to admit the time cost is imperceptible. That said, I have watched it take ~5 seconds to compile xtabond2.mata or cmp.mata. Here is an example, on a pretty fast laptop in Stata 14.1:

. run c:\ado\plus/c/cmp.mata
r; t=4.65 8:38:28

--David

[Niels Henrik Bruun]

Hi David
I haven't thought a lot about it.
But my first thought was simply to let version be a part of the mlib file name.
Checking for the existence and hence also the version of a file should be quite easy from the ado file.
If you find out something more elegant I would like to hear about it.
Do your code from #5 work?

[David Roodman]

Yes, it is working!...after one bug fix. I edited the code in post #5 so that mymlibversion() returns a string rather than a real. Hit reload in your browser.

[David Roodman]

wbuchanan, if I understand right, your idea it to share the .mata file but not the .mlib. Then at run time, the ado checks if the mlib exists and compiles if not. I think that wouldn't work for me because I always end up revising my programs over time, so sometimes mata files would need to be recompiled even when an (old) mlib already exists.

[David Roodman]

My solution is now live in the package "boottest", just posted on SSC. Look at the tops of the ado and mata files.

[Mark Schaffer]

David,

Too late to be helpful (unless you change your mind) ... but we had the same set of issues with ivreg2 except worse, because the mlib was shared across multiple packages (ivreg2, ranktest, avar).

What we switched to about a year ago:

1. A master ivreg2.ado that runs under the minimum Stata required (Stata 8).

2. Within ivreg2.ado, a parent ivreg2 program that checks the Stata version. The parent ivreg2 then forks to the appropriate ado,namely the most recent one that supports the calling Stata version. E.g., if the calling version of Stata is Stata 10, then it forks to ivreg210.ado.

3. The current ivreg2 - the one that supports the most recent version(s) of Stata - is the one that uses the shared mlib.

4. Older versions of ivreg2 - ivreg210, ivreg29, et al. - are self-contained w.r.t. Mata code and do not use the shared mlib.

This setup works because when we add support for new features etc., we do it only for the current ivreg2. We don't go back and recode ivreg210, ivreg29, et al. Makes maintenance and development manageable.

Another useful feature is that it supports version control. This is handy for various reasons - sometimes we want to check/compare results across different versions of the program, and sometimes when there's a bug report, the easy temporary workaround is to revert to an earlier version using version control.

Maybe also useful and relevant - we added a whichlivreg2() Mata program to the mlib. Reports the version of the mlib is being used, the version of Stata under which it was compiled, etc. (Come to think of it, we should probably put some of this information in r(.) macros.)

[wbuchanan]

david Roodman I just started wrapping up something that might make it a bit easier to trigger compilation on the user's side based on file system properties (e.g., file creation/modification dates). I still need to clean up the return values a bit to make it easier to do those types of comparisons, but the code would look something like:

filesys `"`c(sysdir_plus)'l/lbootstrap.mlib"', attr

if date(substr(`"`r(modifiedon)'"', 1, 10), "CY-M-D")  < `distrodate' {

    // trigger compilation of the library

}

Things got a bit messed up with the formatting the first time I posted. I figured this could provide additional flexibility with regards to handling compilation at the user side. The current version of things can be found at:

https://github.com/wbuchanan/StataFileSystem

[David Roodman]

Thanks Mark Schaffer and wbuchanan. I'm sure there are advantages to both your methods. I find my solution, with 6 lines of code, pretty adequate and elegant, so for now I'm going to stick with that.

[wbuchanan]

David Roodman doesn't this still create issues if you update the underlying mata library and the user is still using the same version of Stata? I've tweaked things a bit during the day to make it a bit easier and help to avoid some of the casting stuff above:

filesys `"`c(sysdir_plus)'l/lboottest.mlib"', attr
if `r(creatednum)' < clock("17dec2015 00:00:00", "DMYhms") qui: do `"`c(sysdir_plus)'b/boottest.mata"'

Two lines and can then be made to recompile based on the date you update the library by hardcoding the date `clock("17dec2015 00:00:00", "DMYhms")`. It won't avoid issues that could arise from different versions of Stata, but should make it easier to force the compilation of the library to the end user. It's also a bit shorter now too.

I just threw a helpfile into the repository and the readme has a few examples and shows what the program returns as well.

[David Roodman]

No, I don't think it's a problem because my packages all include the mlib, compiled in the oldest Stata version supported. So a command like adoupdate will assure that the latest mlib is installed. Then, in the case of my new command, if a newly updated mlib is run in a newer version of Stata, it will be recompiled one time. (The packages all include the .mata file too.)

[wbuchanan]

David Roodman makes sense. I was trying to approach things more from the side of distributing the mata source and not having to distributed a precompiled mlib file, but your approach definitely makes sense too.