Introducing statacons: An SCons-based build tool for Stata

[Brian Quistorff]

To the Stata community,

We are pleased to announce statacons, an SCons-based build tool for Stata.
With statacons, users can manage their workflows efficiently, rebuilding parts of a project if and only if necessary. statacons avoids both errors of inclusion (unnecessary, time-consuming rebuilds of parts of a project for which no inputs have changed) and of exclusion (failing to rebuild parts of a project whose inputs have changed). The user defines relationships between inputs and outputs in a build script, and then statacons tracks file changes and decides what to rebuild. The build script, called an SConstruct, documents the workflow. statacons can use the SConstruct to provide simple visualizations of the workflow and the current status of all or selected components. statacons does not require changes to existing code, is extensible via Python, is git-friendly, and complements literate programming tools such as dyndoc, texdoc and markdoc.

Please see below for links to our forthcoming Stata Journal paper, our installation guide, and additional resources.

We look forward to comments from the Stata community.

To submit feature requests or bug reports, please open an issue on our project repository: https://github.com/bquistorff/statacons/issues

Sincerely,

The statacons team:
Raymond Guiteras (NCSU)
Ahnjeong Kim (NCSU)
Brian Quistorff (Bureau of Economic Analysis)
Clayson Shumway (NCSU)

Paper:
Guiteras, Raymond, Ahnjeong Kim, Brian Quistorff and Clayson Shumway, "statacons: An SCons-based build tool for Stata," CEnREP Working Paper 22-001, May 2022, https://go.ncsu.edu/cenrep-wp-22-001. Accepted at The Stata Journal. PDF: https://osf.io/preprints/metaarxiv/qesx6/download
Abstract: This paper presents statacons, an SCons-based build tool for Stata. Because of the integration of Stata and Python in recent versions of Stata, we are able to adapt SCons for Stata workflows without the use of an external shell or extensive configuration. We discuss the usefulness of build tools generally, provide examples of the use of statacons in Stata workflows, present key elements of the syntax of statacons, and discuss extensions, alternatives, and limitations. Appendices provide installation instructions and recommendations for collaborative workflows.

Installation:
See our Installation Guide: bquistorff.github.io/statacons/installation.html.
Dependencies: statacons requires Stata 16 or higher, Python 3.5 or higher (3.8 required for some advanced options) and SCons 4.3 or higher. We provide information on installing python and SCons in the Installation Guide.
Additional resources:
The Project Web Page at https://bquistorff.github.io/statacons/index.html hosts or links to all statacons resources, including

• The paper: Guiteras, Raymond, Ahnjeong Kim, Brian Quistorff and Clayson Shumway, "statacons: An SCons-based build tool for Stata," CEnREP Working Paper 22-001, May 2022, https://go.ncsu.edu/cenrep-wp-22-001. Accepted at the Stata Journal. PDF: https://osf.io/preprints/metaarxiv/qesx6/download
• The Installation Guide: bquistorff.github.io/statacons/installation.html.
• A web tutorial based on Software Carpentry's make lesson: https://bquistorff.github.io/statacons/swc.html
• The project Wiki Page with additional advanced features, a troubleshooting guide, ideas for future improvements, and fixes for common errors (user contributions welcome): https://github.com/bquistorff/statacons/wiki
• Documentation for statacons, stataconsign, and pystatacons
• The MetaArXiV site archiving all pre-publication drafts of the paper: https://osf.io/preprints/metaarxiv/qesx6/.
• The OSF site archiving all versions of the software and replication code and data: https://osf.io/gbh4m/

[Bert Lloyd]

Statacons team,

This looks interesting. I would like to try out statacons, what do you recommend a user do to get started?

- BL

[Raymond Guiteras]

Bert Lloyd , thanks for your question. We're glad you are interested in trying statacons.

Here's what we recommend for getting started

Preliminary: install Python, SCons, pystatacons and the statacons package following our instruction guide here:

https://bquistorff.github.io/stataco...tallation.html

First, work along with the Introductory Example in Section 2 of the paper. The code and data are here:

https://github.com/bquistorff/statac...ons-lesson.zip

Second, we have a web tutorial (inspired by Software Carpentry's make lesson) here:

https://bquistorff.github.io/statacons/swc.html

The first four lessons (up to and including "Exercise: Write Two New Rules") cover the basics.

For more advanced syntax and features, read through the paper and see the additional examples on the project website.

If you decide to use statacons in your own work, our main advice is to start small. If you are adding statacons to a large existing project, we recommend that you start with a small, standalone piece of that project to gain some experience.

As you go, you may find our troubleshooting guide useful:

https://github.com/bquistorff/statac...roubleshooting

Best,

Raymond (on behalf of the statacons team)

[Bert Lloyd]

Raymond Guiteras , Brian Quistorff and the statacons team,

Thank you for providing a startup guide.

You mention that statacons is "git-friendly" - the project I think is best suited to trying first happens to use subversion instead of git. Do you think that statacons is equally subversion-friendly?

- BL

[Brian Quistorff]

It should be fine. We provide some .gitignore template files to indicate what shouldn't be under control. I think you'd have translate to SVN ignore instructions (though I haven't been a Subversion user in a long time!).

[Raymond Guiteras]

Just to add to Brian Quistorff 's suggestions: we provide a step-by-step example of using GitHub with statacons on our project Wiki page:

https://github.com/bquistorff/statac...he-SCons-cache

https://github.com/bquistorff/statac...code-on-GitHub

https://github.com/bquistorff/statac...he-SCons-cache

To get started with Subversion, you might try to reproduce this example for yourself, but using Subversion instead of GitHub.

Best,
Raymond

[Raymond Guiteras]

statacons has been updated to v3.0.1 for compatibility with SCons 4.5.

To update, enter the following in Stata:

net install statacons, from(https://raw.github.com/bquistorff/statacons/main/) force replace

Notes:

1. This affects only runscons.py.
2. updating to statacons v3.0.1 is only necessary for SCons 4.5 or greater, but is compatible with SCons 4.3 or 4.4.

See the Project Web Page at https://bquistorff.github.io/statacons/index.html for full installation instructions.

We are grateful to Andreas Gustafsson for catching this issue with statacons and SCons 4.5 and for filing a bug report on the issues page of our repo:

https://github.com/bquistorff/statacons/issues

[Raymond Guiteras]

Dear Statalist,

statacons and pystatacons have been updated to v3.0.2.

We recommend updating both the full statacons Stata package (which includes stataconsign, complete_datasignature and a few auxiliary programs) and the pystatacons Python package. To update:

1. Update statacons by entering the following in Stata:

net install statacons, from(https://raw.github.com/bquistorff/statacons/main/) force replace

To verify that statacons has been updated, enter:

which statacons

If the update was successful, you will see:

*! version 3.0.2 February 22 2024 statacons team

2. Update pystatacons by entering the following at the appropriate terminal prompt:

pip install --upgrade pystatacons

To verify that pystatacons has been updated, enter

pip show pystatacons

If the update was successful, you will see:

Version: 3.0.2

This update fixes some minor bugs:

1. pywin32 has been added as a requirement of pystatacons on Windows.

2. pystatacons previously used pkg_resources, which has been deprecated as an API, leading to a deprecation warning. The updated pystatacons no longer uses pkg_resources.

3. complete_datasignature returned an error when dataset or variable notes included quotation marks. This has been fixed.

Please report any bugs as issues on the Statacons github page:

https://github.com/bquistorff/statacons/issues

[Raymond Guiteras]

I neglected to mention that version 3.0.2 also includes an improvement to complete_datasignature contributed by Michael Stepner

The order in which Stata saves dataset characteristics is not consistent, leading complete_datasignature to false positive detections of changes to datasets with characteristics. Michael Stepner identified the problem and contributed a bug fix, for which we are grateful.