Mata in different versions of Stata

[Niels Henrik Bruun]

Hi
I've made a Stata command that works from Stata 13 onwards because it uses xl().
I would like to extend the command to version 12.
And therefore I've made mata function string_matrix2csv.

My problem is not to parse the mata block containing xl() when I'm in version 12.
I've tried -_caller()- but then I can't test it in version 13.1 which is the only version I've got immediate access to.
I've also tried the mata function callersversion but that doesn't seem to work in version 12, at least not in a way so I don't the mata with xl() parsed as well.
And then command causes an error in Stata 12.

Does anyone have some good advice regarding this?

Thank you in advance

[wbuchanan]

If you use a service like GitHub/GitLab or other repository, you could always ask others to help test your code. I need to do something similar, but still have my copy of Stata 12 installed so I can test things (when needed).

[Niels Henrik Bruun]

wbuchanan Your suggestion is good. But my question was more on how to handle different variants of mata code in different version of Stata.

[wbuchanan]

Niels Henrik Bruun is this more along the lines of what you were asking about (e.g., testing the version of Stata from within Mata). And I tested the logic of it across version 12 - 14.

. version
version 12.1

. mata
------------------------------------------------- mata (type end to exit) --------
: if (c("version") >= 12 & c("version") < 13) {
> printf("This is a version 12 variant of Stata")
> }
> else {
> printf("This is some other version of Stata")
> }
This is a version 12 variant of Stata
:

. version
version 13.1

. mata
------------------------------------------------- mata (type end to exit) -----------------------
: if (c("version") >= 12 & c("version") < 13) {
> printf("This is a version 12 variant of Stata")
> }
> else {
> printf("This is some other version of Stata")
> }
This is some other version of Stata

. version
version 14.1

. mata
------------------------------------------------- mata (type end to exit) ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
: if (c("version") >= 12 & c("version") < 13) {
> printf("This is a version 12 variant of Stata")
> }
> else {
> printf("This is some other version of Stata")
> }
This is some other version of Stata

[daniel klein]

Unfortunately the strategy suggested by William's is unlikely to solve Niels problem for two reasons.

First, c(version) gives the version currently set by version, not the version (release) of Stata running. This is great if Niels wants to support version control, but as I understand he wants c(stata_version) instead.

Howevere, even c(stata_version) will fail for the purpose, as I understand it. While in Stata - a scripting language - this approach will do, in Mata - a compiled language - it will most likely not. In Stata, if we code

if (c(stata_version > 13)) {
    do something
}

and run this in, say version 12, then Stata interprets the first line, finds that the expression evaluates to false and stops executing. Mata, running on Stata 12, on the other hand, will try and compile the complete code at compiling time. It will, thus complain not to know something which is available only in Stata 13.

Will write a litte more later.

Best
Daniel

[Niels Henrik Bruun]

Thank you both. Daniel : I expected that something you describe were the case. But I couldn't formulate it any better than above. Loking forward to hear the rest.

[daniel klein]

Do not expect much from me, I just happen to have a similar problem. So here is the little more.

My approach so far is to have separate pre-compiled versions of the Mata code, then set up the main ado-file as illustrated above

program main
    version minimum_version_required
    
    local version = c(stata_version)
    
    if (`version' < #) {
        mata : myfunction#()
    }
    else if (`version' < #) {
        mata : myfunction#()
    }
    else {
        assert 0
            /* internal error */
    }
end

excluding any Mata code from the ado-file.

There a couple of things to be aware of. First, you must compile the Mata functions using a version/copy of Stata as indicated - version control will not do the trick. You also want to make sure the remainder (if any) of your main ado-file runs correctly under the (minimum) version indicated at the beginning of that file. To be sure, you again need the respective copy of Stata and run the ado-file on it - version control will not suffice. Alternatively, you can scroll thru the whatsnew help and hope to catch any changes that might bite.

Now, this strategy is obviously less convenient with (i) the number of versions needed per function, (ii) the number of functions in total, (iii) the dependencies among those functions. I will have to experiment with Mata libraries, hoping that perhaps there is a possibility to store functions compiled with different versions into one library, so I will not need to pass along a thousand files, when sharing the program/command.

Maybe an alternative is to provide different "complete'" packages, which is quite often the case on SSC. However, personally I do not like users to type mycommand in their (a)do-files one day and then needing to change that to read mycommand12 (then mycommand13, then ...) at a later point in time, when I update the command. This why I still prefer the approach outlined above, despite the downside of passing along many files. Of course, even better would be to have some sort of version control for user-written stuff from StataCorp instead. But I do think this is going to happen, as I cannot really imagine a way to implement something like it, even if they were willing to.

I do currently see no way to do something similar as the above from within Mata functions, for the reasons outlined earlier - Mata is a compiled language.

Best
Daniel

[Niels Henrik Bruun]

Hi Daniel
Thank you very much.
This is actually a bit worse than I feared.
I've recently decided not to use compiling of Mata/mlib due to the fact that you have to compile for each version of Stata.
Also the code is then accessible for the user as documentation.

I've had imagined something like the below in the worst case:
program main version minimum_version_required local version = c(stata_version) if (`version' < #) { run code#.do // code# contains the version # code of myfunction mata : myfunction() } else if (`version' < ##) { run code##.do // code## contains the version ## code of myfunction mata : myfunction() } else { assert 0 /* internal error */ } end Now I think I have to limit my code to version 13 and up.

[daniel klein]

Hm, I think this could be simplified a bit to read

program main
    version minimum_version_required
    local version = c(stata_version)
    if (`version' < #) {
        run code#.do // code# contains the version # code of myfunction
    }
    else if (`version' < ##) {
        run code##.do // code## contains the version ## code of myfunction
    }
    else {
        assert 0
        /* internal error */
    }
    mata :  myfunction() // WRITE THIS LINE ONCE ONLY
end

Actually, this should work. It has the drawback that the function would be re-compiled anytime the user calls your command, which would certainly eat up the speed advantage of compiled code. You could get around this, including a "setup" command and tell the user that he is to run

setup_mycommand

before running mycommand. But this seems a bit awkward and I honestly fail to see the advantage of distributing several .do files instead of several .mo files (or libraries). Regarding the documentation issue, there is nothing wrong with sending a myfunction.mata (or.do file), containing the source code, along with the .ado, and .mo files (or libraries). Ben Jann has yet another way of providing the source code for all functions in his moremata package, despite distributing "only" three Mata libraries.

If you are fine with providing pre-compiled code, you could still think in the direction William suggest. I have Stata version 11 and 12 installed, and I would not bother compiling the code for you, run a (short) certification script of yours and send you the log and compiled files. I would guess you will find others that would do the same on Statalist, as long as the time they need to spend on this is reasonable.

All the best
Daniel

[wbuchanan]

Another somewhat related option to what daniel klein suggested would be to create an .ado file that serves a similar function as a make file. Here's an example from something I've been working on recently.

// Drop program if already loaded in memory
cap prog drop buildd3

// Define program to automate compiling mata classes for D3js wrappers
prog def buildd3, rclass

    // Version of Stata to use
    version 12
    
    // Define syntax structure for program
    syntax anything(name=what id="D3 Classes or All") [, DIsplay MLib MOSave ///   
        dir(passthru) REPlace size(integer 2048) noPATH ]
    
    // Clear any existing class from mata
    mata: mata clear
    
    // Build a class list macro to store the names of the mata classes
    loc classlist ""
    
    // Check for path option
    if `"`path'"' != "nopath" {
    
        // Set Directory for locating Mata source
        loc location `c(sysdir_plus)'d/
        
    } // End IF Block for nopath option
    
    // If user wants to compile the mega class
    if `: word count `what'' == 1 & inlist(`"`what'"', "all", `"""all"""') {
    
        // Drop class if exists in mata already
        cap mata: mata drop d3()
        cap mata: mata drop header()
        cap mata: mata drop doc()
        cap mata: mata drop filedoc()
        
        // Compile the HTML Header/Footer class
        run `"`location'header.mata"'
        
        // Compile the mega d3 class
        run `"`location'd3.mata"'
        
        // Add d3 to class list
        loc classlist d3() header() doc() filebase()
        
    } // End IF Block for all classes
    
    // If what argument is not all
    else {
    
        // Compile header class
        cap mata: mata drop header()
        cap mata: mata drop doc()
        cap mata: mata drop filedoc()
        
        // Compile the drag behavior class
        run `"`location'header.mata"'

        // Return the classes contained in the header.mata file
        loc classlist `classlist' header() doc() filebase()
        
        // Loop over the arguments passed to the program
        forv i = 1/`: word count `what'' {
        
            // Get the first word from the passed arguments
            loc classes `: word `i' of `what''
            
            // Check for valid arguments
            if !inlist(`"`classes'"', "behavior", "core", "geo", "geom",      ///   
            "layout", "scales", "svg", "time") {
            
                // Print error message to console
                di as err "`classes' is not a valid class to compile.  "     ///   
                "Must be one of: " as res "behavior, core, geo, geom, "      ///   
                "layout, scale, svg, or time"
                
                // Error out of program
                err 198
                
            } // End IF Block to check for valid arguments
            
            // If it is a valid argument
            else {
            
                // Drop existing class definitions if present
                cap mata: mata drop `classes'()
                
                // Compile the drag behavior class
                run `"`location'`classes'.mata"'
                
                // Return the classes defined here
                loc classlist `classlist' `classes'()
                
            } // End ELSE Block for compiling individual classes
            

        } // End Loop over arguments passed to program's parameter
        
    } // End ELSE Block for individual class compilation
        
    // Loop over the classes in the class list
    forv i = 1/`: word count `classlist'' {
    
        // Return the name of the class in a macro
        loc nm `: word `i' of `classlist'' 
        
        // Remove parentheses
        loc lnm `: subinstr loc nm `"()"' "", all'
        
        // Return the nmae of the class
        ret loc `lnm' "`nm'"
        
        // Check for MoSave option
        if `"`mosave'"' != "" & `"`mlib'"' == "" {
        
            // Save the mata objects
            mata: mata mosave `nm', `dir' complete `replace'
        
        } // End IF Block for mata mo save
        
    } // End Loop over the class list

    // If display option is turned on
    if "`display'" != "" {
        
        // Print message to screen
        di as res "Compiled the classes : `classlist'" _n _continue
        
        // Describe the classes/methods
        mata: mata d, all
                    
    } // End IF Block for display option
        
    // Check for MLib option
    if `"`mlib'"' != "" {

        // Create the library
        mata: mata mlib create libd3, `dir' size(`size') `replace'

        // Add the first class to the library
        mata: mata mlib add libd3 *(), `dir' complete
        
        // Index the mata classes/functions
        mata: mata mlib index
            
    } // End IF Block to build custom d3 library for user
    
// End of program
end

In this way, the end user can compile the code without having to be fully aware of what is happening in the backend and you could use the scripting flexibility in Stata to test the runtime environment version. If the library is sufficiently complex, some users may only want a limited portion of the functionality and something like this allows them to build what they need/want to satisfy their unique use case.

[Niels Henrik Bruun]

Sorry for the lousy code block in #8. It appeared ok when I wrote it.
Thank you both for some very inspiring comments.
I hear you and I'll think about it.
However my code in this case isn't that time consuming so far and although it is very nice of the Mata community I can act more independent by not compiling.
On the other hand it is good to know that it is possible to get others to compile the code.

Finally, inspired by your comments I think a solution could be to attach the Mata code in one or more do-file (some version dependent) and letting the ado be verifying the existence and version of the mlib before using it.
If the mlib is not present or not of proper version the mlib is compiled or overwritten fitting the users current version.
This way the user himself compiles the code when necessary.
I guess that is what you suggest to me in the last 2 comments.

Thank you very much

[wbuchanan]

Niels Henrik Bruun that definitely works and is definitely a viable solution. Best of luck with the program.

[wbuchanan]

Niels Henrik Bruun not sure if you were keeping up on a somewhat related thread started by David Roodman, but I recently built something that builds on your last post.

filesys `"`c(sysdir_plus)'l/libmymatalibrary.mlib"', attr
if `r(creatednum)' < clock("17dec2015 00:00:00", "DMYhms") qui: do `"`c(sysdir_plus)'l/libmymatalibrary.mata"'

So, one of the issues that I thought of about the solution you came to above is what would happen if there were updates to the mata source itself. Since the file would already exist, it wouldn't trigger compiling the source again. So I put together a quick Java plugin that returns filesystem properties so you could test whether or not the file was created prior to your distribution date as a way of forcing a recompilation if the library already exists. I've been using Mata a bit more in some of my own work so it was fairly helpful for me as well (and to simplify some of the installation process for the brewscheme package).

The source and pre-compiled JAR files are all available at: https://github.com/wbuchanan/StataFileSystem.

[Niels Henrik Bruun]

wbuchanan Thank you very much. However I do follow all Mata posts. In the near future I'm going to play with this. Again thank you very much

[wbuchanan]

Not a problem. I thought it might be useful/helpful for others as well.