stata mp 15 computing speed

[Vishal Sharma]

hi, I recently upgraded to stata mp 15 in order to use the multi processor capabilities. however, when i run the reshape command on a VERY large data set ( 2557 binary variables) with 1.6 million observations, i noticed that stata is not utilizing all of the cpu's to their max. is there a setting in stata mp i need to adjust in order to take advantage of the super computer i m using?

thanks

vishal

[Attaullah Shah]

Why not try fastreshape

ssc install fastreshape

[Vishal Sharma]

i tried sreshape but it seems to have a variable maximum limit.. i have 7600 variables i need to reshape

[William Lisowski]

From the StataCorp marketing page for Stata/MP at

https://www.stata.com/statamp/

we can download the Stata/MP Performance Report PDF. In that report, on page 31 (03 October 2017 version) we see that that the reshape long command does not scale linearly in the number of cores. On page 135 we see in figure 376 that for desktop systems with 1-8 cores, there's little improvement before a jump at 7 cores. On page 230, in the plot below the heading reshape long we see that after the jump at 7 cores, little further improvement is made and regardless of the number of additional cores, reshape long runs at about 3 times the speed of a single core.

I note that Attaullah Shah recommended fastreshape but you have not commented on the results with that package.

There was also a long discussion about reshape speeds and techniques for improving it earlier on Statalist at

https://www.statalist.org/forums/for...reshape-faster

And there was a fair amount of thoughtful advice given to you earlier at

https://www.statalist.org/forums/for...eshape-command

from which I notice that you attempted to speed up reshape by reshaping subsets of observations, but one of the recommendations in

http://www.nber.org/stata/efficient/reshape.html

linked to in the long discussion suggests reshaping a single variable at a time.