Download (install) all packages in SSC?

[River Huang]

Dear All, Is it possible to install all the packages in SSC using one simple command (whatever that is)?

[Nick Cox]

Only if someone writes that command.

Kit Baum and I wrote the original ssc command, later rewritten by StataCorp, and I don't think we ever discussed or even imagined that anyone would want to download everything -- and that was in the life of Stata 7, more than 15 years ago, when the archive was much smaller.

Manifestly the current official command doesn't offer that either, so why not?

I want to underline that it's not a good idea. Here I speak for myself, but also guess that the company view (or Kit's) would not be much different.

1. SSC is an archive. Programs can be removed from that archive, but the bigger deal is that many files stay there just in case someone wants to run old code, or because they are just old stuff doing no harm. There is no systematic, periodic review to purge old stuff or decide what's good and worth keeping. That's much tricker than people might imagine. Suppose something is superseded by a command in a new version of Stata. It might still be exactly what somebody who hasn't upgraded wants. Or there are many really ancient do-files out there using old stuff. Typically they still work fine, but they could be broken if the SSC commands they used disappeared. The last time I looked at SSC download data I saw some extraordinary downloads of things long since superseded by changes in Stata. That's also not very surprising and not much of a problem.

2. SSC has a simple, rigid overall structure such that any command.ado, command.sthlp or other file can only exist uniquely. But necessarily nothing rules out absolutely a clash with filenames also in use by official Stata, by community-contributed commands published in the Stata Journal, or indeed with any other Stata files written by the community or a user. That's not a hypothetical. One class of examples: Many programs start out on SSC and are then revised for publication in the Stata Journal. Now download to Stata includes checks if files already exist. But a mass download could cause lots of clashes that create problems which could be tough for a experienced user who understands how these files are used, and extremely puzzling or frustrating for the naive or novice user who thinks, "I'll just download everything and work out later what's useful. It's all free and the space requirements are trivial."

My main reaction is still the puzzlement I started with. Really, why would anyone want them to use or even to look at them all?

[River Huang]

Dear Nick, Thanks for the reply. My question was just for curiosity. (I agree that very few people will do that).

[Richard Williams]

I'm with Nick. I can't imagine wanting to download thousands of programs, most of which I don't want or need and often don't understand. There are plenty of programs I did once choose to download and now don't remember why.

If you desperately wanted to do it though, maybe one of the programs described here (none of which I have tried) could do the trick?

https://www.makeuseof.com/tag/how-do...fline-reading/

[Richard Williams]

I do wish ssc kept archives of some old programs, e.g. a version that only requires Stata 9 while the newest version requires Stata 13. What many authors do, though, is create a renamed version of the old file and make it available on ssc, e.g. oglm9 is out there for somebody who is condemned to using something older than Stata 11.2.

[Nick Cox]

Again, SSC is just an archive.

Kit has some minimal rules. For example, I am sure that he wouldn't accept do files on their own or programs without help files, and he's sensitive -- and discreet -- to obvious poor quality, reports of plagiarism, and the like.

But, necessarily, he is dependent on what authors submit, or do not submit.

I have a great deal of sympathy with e.g. authors who have long since lost access to previous versions of Stata and don't feel obliged to support previous versions of their programs indefinitely for the sake of users who didn't upgrade. (I know it costs money, but programmers' time and effort is also involved here. Free products carry fewer obligations.)

To forestall a possible comment: If you want GitHub, you know where to find it.

(This is a deliberate variant on an old programmers' joke: Dennis Ritchie, the principal architect of C, was often bombarded with questions on why C did not include this, that and the other; in short, why it was not much bigger and more complicated. His stock reply was "If you want PL/I, you know where to find it". The story has particular piquancy for those who remember PL/I, a versatile but bloated language that fell into oblivion as rapidly as it rose to attention.)

[Alan Riley (StataCorp)]

I'll second Nick Cox's thoughts. It is so easy to install something on an as-needed basis that there is no good reason to clutter your PLUS directory with the thousands of packages on the SSC all at once. And, if someone did that, they would be asking for pain every time they ran adoupdate to make sure their community-contributed packages were up-to-date.

I have seen a couple of organizations pull the entire archive for only one reason, and that was a data "clean room" where no internet connection was allowed, in or out, for security reasons.

[Christiaan Righolt]

On some of our data analysis systems there is no internet connection and no instant way to get files in. So we wrote a tool to do this and install STB and SJ packages as well.

Note the warnings above: THIS CODE MAY DO IRREPARABLE HARM TO YOUR SETUP!!!

Christiaan H. Righolt and Salaheddin M. Mahmud
Vaccine and Drug Evaluation Centre (https://vdec.ca)
Department of Community Health Sciences
University of Manitoba, Winnipeg, Manitoba

You need one generic utility to retrieve package information from SSC, STB, and/or SJ and additional code to install each package. Packages are typically installed in the PLUS directory (see help sysdir) and auxillary files are typically installed in the working directory (see help cd). You should copy the code below into a do-file (and run the file to install all packages).

Package read utility
This utility grabs the package description list, as written to a smcl file, and returns a list of packages that can be installed one by one using internal commands.

program define readPackageList, rclass
  args fileName
  cap file close fh  //fh=filehandle; close if already in use
  file open fh using "`fileName'", read
  file read fh line
  while r(eof)==0 {
    //look for line formatted as "@net:describe pkgname!pkgname@"
    if index("`line'", "@net:describe")    > 0 {
      // b=> index of character just after the string "@net:describe"
      local b =index("`line'", "@net:describe") + length("@net:describe") + 1
      local e =index("`line'", "!")
      local pkgname= substr("`line'", `b', `e'-`b')
      local pkglist "`pkglist' `pkgname'"
    }
    file read fh line
  }
  file close fh
  return local pkglist="`pkglist'"
end

Add this code as well:

global dryRun=0
global savename test1.txt
set more off

SSC
SSC packages start with a lower case letter or \_. The code below describes all available packages starting with each symbol and installs them one by one.

foreach d in `c(alpha)' _ {
      noi ssc desc `d' , saving($savename, replace)
      readPackageList "$savename"
    noi di "Found packages: " r(pkglist)
    foreach a in `r(pkglist)' {
    noi di in yellow "Found package: `a'"
    if $dryRun !=1 {
      cap ssc install `a', all replace
      if _rc != 0 {
          noi di in red "Error downloading `a'"
      }
    }    
    }    
}

Stata Technical Bulletin
STB, now superseded by SJ, has had 61 issues, the code below describes the packages available for each issue and installs them one by one.

foreach issue of numlist 1/61 {
  log using $savename, replace smcl
  net from http://www.stata.com/stb/stb`issue'/
  log close
  readPackageList "$savename"
  foreach a in `r(pkglist)' {
    noi di in yellow "Found package: `a'"
    if $dryRun !=1 {
      net install `a', all replace
    }    
  }
}

Stata Journal
SJ has up to 4 issues a volume; each volume spans a calender year starting in 2001. The code below describes the packages available for each issue and installs them one by one.

local current_year = substr("$S_DATE",-4,4)
local current_vol = `current_year' - 2000
foreach vol of numlist 1/`current_vol' {
  foreach issue of numlist 1/4 {
    // See whether it exists if yes, then actually install it
    cap net from http://www.stata-journal.com/software/sj`vol'-`issue'
    if _rc==0 {
      log using $savename, replace smcl
      net from http://www.stata-journal.com/software/sj`vol'-`issue'
      log close
      readPackageList "$savename"
      foreach a in `r(pkglist)' {
        noi di in yellow "Found package: `a'"
        if $dryRun !=1 {
          cap net install `a', all replace
          if _rc != 0 {
            noi di in red "Error installing `a'"
          }
        }    
      }
    }
  }
}

[Christiaan Righolt]

If you do this on an existing Stata installation, you might want to install to and run from a clean directory so it doesn't cause trouble for your existing packages.

[River Huang]

Dear Christiaan, Many thanks for these interesting procedures. I will give it a try.