[Marcos Almeida]

Friedman's ANOVA. Why not?

Yes, I know this is a basic test. Yes, I know we have some community-contributed programs that can delve with it (some with challenging strategies for the novice, such as transposition; some needing a good understand of what is being done, since the output doesn't underline 'Friedman', because it was originally created to fulfill a different task).

In short, the very fact that Friedman's ANOVA is available in several quite 'basic' statistical software prompts us, Stata lovers and users, to spot this missing issue.

[jerome falken]

There is no reason to link it to libpng12 and libpng16 really; please migrate the code relying on libpng12 (likely the icons from the menubar, etc.) to the latest libpng libraries. Modern linux distributions no longer include libpng12.

$ ldd xstata
linux-vdso.so.1 (0x00007ffff23c8000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f9f50b0f000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f9f508f1000)
libgtk-x11-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgtk-x11-2.0.so.0 (0x00007f9f504aa000)
libgdk-x11-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0 (0x00007f9f503f3000)
libatk-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libatk-1.0.so.0 (0x00007f9f503ca000)
libgio-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0 (0x00007f9f5020c000)
libpangoft2-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpangoft2-1.0.so.0 (0x00007f9f501f1000)
libpangocairo-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so.0 (0x00007f9f501e1000)
libgdk_pixbuf-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0 (0x00007f9f501bb000)
libcairo.so.2 => /usr/lib/x86_64-linux-gnu/libcairo.so.2 (0x00007f9f5009b000)
libpango-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0 (0x00007f9f50050000)
libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007f9f4ff94000)
libfontconfig.so.1 => /usr/lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007f9f4ff4c000)
libgobject-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007f9f4fef7000)
libgmodule-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007f9f4fef1000)
libgthread-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgthread-2.0.so.0 (0x00007f9f4feec000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f9f4fee2000)
libglib-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007f9f4fdc3000)
libpng12.so.0 => /home/XXX/lib/libpng12.so.0 (0x00007f9f4fb9a000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f9f4fb80000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9f4fb5f000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9f4f99e000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9f4f81b000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f9f4f816000)
/lib64/ld-linux-x86-64.so.2 (0x00007f9f50cc6000)
libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f9f4f6d3000)
libXcomposite.so.1 => /usr/lib/x86_64-linux-gnu/libXcomposite.so.1 (0x00007f9f4f4d0000)
libXdamage.so.1 => /usr/lib/x86_64-linux-gnu/libXdamage.so.1 (0x00007f9f4f4cb000)
libXfixes.so.3 => /usr/lib/x86_64-linux-gnu/libXfixes.so.3 (0x00007f9f4f2c5000)
libXrender.so.1 => /usr/lib/x86_64-linux-gnu/libXrender.so.1 (0x00007f9f4f0bb000)
libXinerama.so.1 => /usr/lib/x86_64-linux-gnu/libXinerama.so.1 (0x00007f9f4f0b6000)
libXi.so.6 => /usr/lib/x86_64-linux-gnu/libXi.so.6 (0x00007f9f4eea4000)
libXrandr.so.2 => /usr/lib/x86_64-linux-gnu/libXrandr.so.2 (0x00007f9f4ec99000)
libXcursor.so.1 => /usr/lib/x86_64-linux-gnu/libXcursor.so.1 (0x00007f9f4ec8c000)
libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f9f4ea7a000)
libmount.so.1 => /lib/x86_64-linux-gnu/libmount.so.1 (0x00007f9f4ea1b000)
libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f9f4e7f3000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f9f4e7d7000)
libharfbuzz.so.0 => /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0 (0x00007f9f4e6dc000)
libpixman-1.so.0 => /usr/lib/x86_64-linux-gnu/libpixman-1.so.0 (0x00007f9f4e636000)
libpng16.so.16 => /usr/lib/x86_64-linux-gnu/libpng16.so.16 (0x00007f9f4e5fd000)
libxcb-shm.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-shm.so.0 (0x00007f9f4e5f8000)
libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f9f4e5cc000)
libxcb-render.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-render.so.0 (0x00007f9f4e5bd000)
libthai.so.0 => /usr/lib/x86_64-linux-gnu/libthai.so.0 (0x00007f9f4e5b2000)
libfribidi.so.0 => /usr/lib/x86_64-linux-gnu/libfribidi.so.0 (0x00007f9f4e595000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f9f4e558000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f9f4e54d000)
libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f9f4e543000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f9f4e4cf000)
libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007f9f4e47a000)
libgraphite2.so.3 => /usr/lib/x86_64-linux-gnu/libgraphite2.so.3 (0x00007f9f4e44d000)
libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f9f4e247000)
libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f9f4e041000)
libdatrie.so.1 => /usr/lib/x86_64-linux-gnu/libdatrie.so.1 (0x00007f9f4e037000)
libbsd.so.0 => /usr/lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f9f4e01d000)

[Adrien Bouguen]

Including gsort option into bysort i. e. that I can write bys X Y (-Z): gen newvar=mean().

[Leonardo Guizzetti]

Including gsort option into bysort i. e. that I can write bys X Y (-Z): gen newvar=mean().

Indeed, it would be nice to generalize sort so that -gsort- will set the indicator that the data have been sorted.

[Niels Henrik Bruun]

Looking into the Stata documentation today I was quite surprised to discover that there is no way of inserting a Mata string matrix into a word document.
So I suggest that _docx_add_mata can take a string matrix as well as a real matrix as argument.

Likewise for pdf and fillMataMatrix.

The reason is that it might be needed to handle formatting etc in Mata and then export the formatted result.
And there is no reason I can see for limitating matrices to be only real.

[Joro Kolev]

Not that I have any hope that this will be done, but one of the three major way in which Stata torments me is that the -by- construct does not allow multiple operations.

I would love to have something like this (imaginary code follows, this does not work):

by group: summ x \ gen newvar = r(mean)

[FernandoRios]

Hi Joro,
that is true, -by- doesnt allow for multiple operations like that, directly, but you can implement or use -egen- functions.
for example

by group:egen newvar=mean(x)

does exactly what you describe.
Granted, for more complicated problems it requires bit of programming.
Fernando

[Clyde Schechter]

@291, 292: And more generally, if you wrap a group of commands inside a program, then -runby-, written by Robert Picard and me, and available from SSC, can executed by groups. -runby- is also much faster than looping over -levelsof- variables, which is the conventional workaround for lack of by-ability.

[Joro Kolev]

Stata returns error message when you try to equate a local or a scalar to a matrix expression which evaluates to a scalar. The fix to this "feature" of Stata is to split the operation on two lines, like I am showing below. There are two problems with that. 1) This "feature" of Stata does not seem to be clearly explained anywhere, and it is always a trial and error figuring out which matrix expression Stata will consider evaluation, and which picking up an element. 2) I want to be able to do this in one line, leading example being when I am bootstrapping a statistic which is a matrix expression evaluating to a scalar. If I could do this on one line, I could directly use the canned bootstrap. Currently I need to write a whole program to be able to implement the two line solution.

Example:

. sysuse auto, clear
(1978 Automobile Data)

. qui reg price mpg headroom

. local rowsplusone = rowsof(e(V))+1
matrix operators that return matrices not allowed in this context
r(509);

. mat tempmat =  rowsof(e(V))+1

. local rowsplusone = tempmat[1,1]

. mat list tempmat

symmetric tempmat[1,1]
    c1
r1   4

. dis `rowsplusone'
4

[Bjarte Aagnes]

Ref #294

* Stata/MP 16.1 for Windows (64-bit x86-64) Revision 30 Jun 2020

. sysuse auto, clear

. qui reg price mpg headroom

. local rowsplusone = rowsof(e(V))+1

. mac dir _rowsplusone
_rowsplusone:   4

[Joro Kolev]

This is fascinating, Bjarte. If Stata Corp have fixed this, it would be beautiful !

Can you try please this a bit more meaty and realistic example, this is actually what I need this one line feature for:

. sysuse auto, clear
(1978 Automobile Data)

. qui reg  price mpg headroom

. matlist e(b)*invsym(e(V))*e(b)'

             |        y1 
-------------+-----------
          y1 |  427.7852 

.  bootstrap w=(e(b)*invsym(e(V))*e(b)') : reg price mpg headroom
(running regress on estimation sample)
matrix operators that return matrices not allowed in this context
error in expression: e(b)*invsym(e(V))*e(b)'
r(509);

. version
version 15.1

Ref #294

* Stata/MP 16.1 for Windows (64-bit x86-64) Revision 30 Jun 2020

. sysuse auto, clear

. qui reg price mpg headroom

. local rowsplusone = rowsof(e(V))+1

. mac dir _rowsplusone
_rowsplusone: 4

[Bjarte Aagnes]

ref #296

. sysuse auto, clear
(1978 Automobile Data)

. qui reg  price mpg headroom

. 
. matlist e(b)*invsym(e(V))*e(b)'

             |        y1 
-------------+-----------
          y1 |  427.7852 

. 
. bootstrap w=(e(b)*invsym(e(V))*e(b)') : reg price mpg headroom
(running regress on estimation sample)
unknown function invsym()
error in expression: e(b)*invsym(e(V))*e(b)'
r(133);

[wbuchanan]

Adopt the POSIX epoch for date/datetime variables. This could potentially affect older files, but should make it easier to use the existing APIs for other languages which tend to use dates/datetimes based on a different epoch date. (This could potentially also have an effect on the datetime format that includes corrections for leap seconds).

[Christopher Bratt]

For me, there are still primarily three drawbacks in Stata compared to other programs: 1. not easy to develop reproducable research with flexible options (compared to R Markdown), 2. difficulties handling output from analyses (compared to R, where results are objects accessible as data frames and easy to manipulate), and (3) speed.

Speed seems to be a recurring problem. Are there some fundamental, old choices in Stata once made and now preventing Stata from running at the speed of other programs?

A recent Twitter post stated:
"Just ran a regression with about 100 million observations.
In Stata (MP, 16): 97 minutes
In Julia (using FixedEffectsModels.jl): 13 seconds."

https://twitter.com/clibassi/status/1289948339962646528

Is it possible to rewrite some of the fundamentals of Stata to improve speed?

[Clyde Schechter]

@ #298. I think this is a terrible idea. If the rest of the world used the POSIX epoch and Stata were the only outlier, than sure, it would be the thing to do. But anarchy still reins on this. I suspect that among Stata users (or at least Stata users who are Forum members) there are many more people who work with Microsoft Excel than with various APIs that rely on the POSIX epoch. Excel doesn't even have a consistent epoch base across versions! -import excel- deals with this properly when importing date variables from Excel. It similarly deals correctly with importation of dates from SAS and SPSS. Perhaps over time Stata will extend the -import- command to work with some of the more popular APIs.

But at the moment, adopting the suggestion in #298 would provide convenience for what I believe is a small number of Stata users while breaking enormous amounts of existing code. And really, if you are a sophisticated enough programmer to be bringing in data from a range of API's, how hard is it to just add a line of code to correct date variables for different a different epoch convention?