Why is sort so slow? Sorting data mata vs. R

[Paul Corral]

mata is fast for many things, but sorting a vector is not one of them.

Below I generate a random vector containing 1 million observations. I sort it 100 times. This process takes mata a little over 75 seconds. The same exercise in R takes around 17 seconds. This is a considerable time difference which only gets worse the larger the number of observations.

If I have one item in my wishlist is for Stata to update their sort, in mata and in Stata. Please make the update backwards compatible, not only for Stata 15.

[Maarten Buis]

You can speed Mata up a bit by using sort(y) instead of y[order(y,1),.]:

mata
y=rnormal(1e6,1,10,5)
timer_clear()
for(i=1;i<=100;i++) {
    timer_on(1)
    x1=sort(x,1)
    timer_off(1)
    timer_on(2)
    x2=y[order(y,1),.]
    timer_off(2)
}
timer()
end

In general Stata is very good at forward compatibility. For example, beta4 part of the betafit package on SSC was last touched 18 years ago and still works in the current version of Stata. I think that is pretty impressive. But StataCorp is a commercial company, which makes them obviously reluctant to do backward compatibility. They to have bills (and Bills) to pay.

[Paul Corral]

Thanks for the reply Maarten.

Mata's sort command uses order, therefore there should be no difference. If you try running the code below, which is a slightly modified version of your code, you'll notice that the time difference is minimal. Regardless, the difference with other software is quite noticeable (~70 seconds vs. ~18 seconds) and leaves Stata at a considerable disadvantage.

mata
y=rnormal(1e6,1,10,5)
timer_clear()
timer_on(1)
for(i=1;i<=100;i++) {
    x1=sort(y,1)    
}
timer_off(1)

timer_on(2)
for(i=1;i<=100;i++) {
    x2=y[order(y,1),.]    
}
timer_off(2)


timer()
end