Limit on maximum matrix length from J()?

[Andrew Maurer]

Hi Statalist,

I noticed that mata errors out when trying to produce a matrix using J() of length 2.2*10^9 or larger. I did the tests below and have found that:

1. The restriction is on the length of the matrix produced by J, not the total size (ie I can produce a matrix 2*10^9 rows X 2 columns, but not a matrix 2.2*10^9 rows long)
2. The restriction does not apply to all matrices, just on the J() function. Ie - I can produce a matrix 4*10^9 rows long by stacking 2 matrices of length 2*10^9 on top of each other
3. The limit is not reported in help limits, nor is it reported in the conformability section of help mata J()

Edit:

• I realized the actual limit is 2,147,483,647, also the maximum size of a stata "long" variable. Does anyone have any thoughts on how to get around this most efficiently?
• I wonder if the issue might result from the internal declaration of the J() function. Perhaps the function is declared as having arguments r,c,val, where r and c are declared as int data type in C? The 4-byte int data type has a maximum value of 2,147,483,647.

cap mata mata drop testfn1() testfn2() testfn3() testfn4() 
mata
void testfn1() {
    A = J(2100000000,1,1)
    // Works
}
void testfn2() {
    A = J(2200000000,1,1)
    // Error: J():  3300  argument out of range
}
void testfn3() {
    A = J(2100000000,2,1)
    // Works
}
void testfn4() {
    A = J(2000000000,1,1)
    B = A\A
    // Works
    // but takes up more RAM than expected...
    // eye-balled task manager around 60gb.
    // peak usage should be (space of A) + (space of B)?
    // ie (2*10^9*8/10^9 + 4*10^9*8/10^9) = 48gb
}
end

mata testfn1()
mata testfn2()
mata testfn3()
mata testfn4()

Thank you,
Andrew Maurer