high dimensional matrices in Mata

[Ingrid Qiu]

I am wondering whether there are high dimensional matrices in Mata. Right now I can just use pointer for high dimensional matrices. I think high dimensional matrices are quite essential for matrix computation. I strongly suggest the developer of Mata consider incorporating high dimensional matrices into Mata. What do you Statalisters think?

[Matthew J. Baker]

Ingrid --

While they aren't quite as user-friendly as their counterparts in, say, Python, you can use associative arrays in Mata, which support quite general indexing schemes and can even be saved using the conventional approaches in Mata.

For example, you could make a four-dimensional "matrix" by using four indices:

mata:
A=asarray_create("real",4)
asarray(A,(1,1,1,1),"stuff")
asarray(A,(1,1,1,2),"more stuff")
asarray(A,(1,1,1,10),runiform(10,10))

To recover items, use the array and the index. E.g.:

asarray(A,(1,1,1,10))

This really just scratches the surface - the Mata Matrix Programming Manual M-5 has a pretty good discussion of associative arrays. Take a look at page 308 in the manual, to which I've added a link below.

Hope that helps!

Matt

[Matthew J. Baker]

Ingrid --

Sorry link didn't work - but hopefully you get the idea to look in the Mata manual!

Best,

Matt

[Christophe Kolodziejczyk]

Ingrid
You should start a post entitled "Wishlist for Stata 15" in the Stata forum (if it isn't already done), where you mention your wish of having N-dimensional arrays as built-in functions in Mata.
Better than asarray would be objects that have more than 2 indices. Actually other softwares like Gauss or Matlab have them, so why not Stata/Mata?
Best
Christophe

[Ingrid Qiu]

Hi Matt and Christophe,

Thank you for your suggestions! I will try to use asarray in my current codes, but I do think Mata should incorporate high dimensional matrices. I feel like Mata is an important tool for econometricians, equally important as Matlab. Although econometricians mainly use Matlab, but most applied economists use Stata. So econometricians always want to have a Stata version of their Matlab codes. So I think the computation ability of Mata should be improved. And I think it will greatly benefit everyone if Mata can be more compatible with Matlab. Although we can use C plug-in, I am afraid that C is not so good at matrix manipulation as Matlab.

I will start a new post and share my thoughts with the developer of Mata. Thanks again.

Best,
Ingrid

[Matthew J. Baker]

Ingrid and Christophe --

I think it is very useful to think about how Mata and Stata might be made more useful to users, and how to achieve more seamless integration with other languages. I hasten to add that it is a really terrific idea on StataCorp's part to have an integrated compiled (Mata) and dynamic (Stata) language available to programmers. I really like Python, but achieving this integration in Python via C++ is a bit more arduous (to me at least) than with Stata and Mata. It is especially nifty how these two things can work together in one .ado file.

Anyways - one thing I thought I would stress that might be a bit lost in the conversation is that associative arrays are very flexible objects. One can use arbitrarily large-dimensional indices, use string indices, include all sorts of objects, ect. in an associative array. And the documentation is quite good on how it all works, so please give it a look!

Best,

Matt

[Ingrid Qiu]

Hi Matt,

Sure, I will have a close look at the associative arrays.

I agree with your comments for the combination of Stata and Mata. These two very different programming environments would just confuse new users. And one thing that make me very uncomfortable is that these two environments are not perfectly integrated together. Take my own experience as an example. I want to do quantile regressions in a loop. If I do these in Stata, an obvious drawback is that Stata is not good at looping. If I do this in Mata, an obvious drawback is that I cannot use the qreg command in Mata. Finally, I converted a Matlab quantile regression function into a Mata code. Unfortunately, this greatly increases the computation time. And now I am considering writing a C plug-in to do loops in order to shorten the computation time. Unfortunately, a C plug-in cannot be directly called in Mata. I have to say that this freaks me out. I wish Stata can be more user-friendly in later version.

Ingrid

[wbuchanan]

Ingrid Qiu
If you've already started the hard work of developing something in C, why would you need to resort to Mata anyway? You can call plugin-based functions directly from Stata and the overhead of interpreting a single line of code is fairly minimal.

[wbuchanan]

Matthew J. Baker
Have you had any luck trying to nest arrays? I've been trying to figure out how I can create an array that would store arrays and when I try to use the get function I end up with : type mismatch: exp.exp: transmorphic found where struct expected. I've copied/pasted more code below to provide a better working example:

fh = fopen("~/Desktop/Programs/StataPrograms/StataIPEDSAll/ipedsdb.csv", "r")
colraw = fget(fh)
colnms = select(tokens(colraw, ","), tokens(colraw, ",") :!= ",")
ipedsdb = J(963, 5, "")
for (i = 1; i <= rows(ipedsdb); i++) {
    line = fget(fh)
    ipedsdb[i, (1..5)] = select(tokens(line, ","), tokens(line, ",") :!= ",")
}

class AssociativeArray scalar ipedsLookup(string matrix ipedsdb, string colvector schyrs) {
    real scalar i, j, k
    string scalar thisSchoolYear, thisSurvey, thisTitle
    string colvector surveys, titles, links
    class AssociativeArray scalar ipedsArray, schoolYearArray, surveyArray, titleArray
    ipedsArray = AssociativeArray()
    ipedsArray.reinit("string", 1, rows(schyrs))
    schoolYearArray = AssociativeArray()
    schoolYearArray.reinit("string", 1)
    surveyArray = AssociativeArray()
    surveyArray.reinit("string", 1)
    titleArray = AssociativeArray()
    titleArray.reinit("string", 1)
    for(i = 1; i <= rows(schyrs); i++) {
        thisSchoolYear = schyrs[i, 1]
        surveys = uniqrows(select(ipedsdb[., 2], ipedsdb[., 1] :== thisSchoolYear))
        for(j = 1; j <= rows(surveys); j++) {
            thisSurvey = surveys[j, 1]
            titles = select(ipedsdb[., 3], rowsum(ipedsdb[., (1, 2)] :== (thisSchoolYear, thisSurvey)) :== 2)
            for(k = 1; k <= rows(titles); k++) {
                thisTitle = titles[k, 1]
                links = select(ipedsdb[., (4, 5)], rowsum(ipedsdb[., (1, 2, 3)] :== (thisSchoolYear, thisSurvey, thisTitle)) :== 3)
                titleArray.put("dataset", links[1, 1])
                titleArray.put("script", links[1, 2])
                surveyArray.put(thisTitle, titleArray)
                titleArray.clear()
            }
            schoolYearArray.put(thisSurvey, surveyArray)
            surveyArray.clear()
        }
        ipedsArray.put(thisSchoolYear, schoolYearArray)
        schoolYearArray.clear()
    }
    return(ipedsArray)
}
ipedsArray = ipedsLookup(ipedsdb, schyrs)