Copilot for Stata?

[Tarjei W. Havneraas]

Does anyone know if something similar to copilot is being developed or considered for Stata? Seem to be early days but highly relevant (e.g., Nature article)

[Daniel Schaefer]

Maybe some day, but I doubt it's a priority right now.

Smart code suggestions and code completion has been a thing for at least a couple of decades now. Traditionally, this is done by the compiler and is typically better in languages that have strong type systems or abstractions that are created with mathematical rigor. In my mind, Haskell and C# both stick out as having particularly powerful compilers that (together with the IDE) are already very helpful at assisting you in writing code. Apparently this is a major selling point for rust. Arguably the first step for Stata should be to write the interpreter in such a way that it is better at identifying patterns and idioms in ado, then providing some UI tools that show the user what they should be doing when they violate those idioms. Support for this kind of thing exists in Stata, but is not very strong.¹

The 2017 google white paper on transformers has undoubtedly lead to a renaissance in AI technology, particularly when it comes to natural language processing. This technology does, of course, have a few drawbacks. For instance, is it legal (or ethical) to monetize an AI model that is trained on publicly available information? Would it be ethical to train a model using posts on this forum and in doing so monetize the work of forum members? I'm honestly not sure.

There are also technical problems. Transformer models are data hungry, which means they are large and inefficient, often requiring far more computational resources than are available on a single desktop computer.² You can get around this if you have a big datacenter where you can host the model and provide access through an API, but that requires a great deal of technical overhead and investment. You also need a massive amount of training data, which can be hard to collect and store (though github is well positioned for this). Large-language transformer models are also, frankly, bad at reasoning. At its core, GPT3 and related technologies are sophisticated, altra-high-dimensional word association machines. They are great (amazing really) at analogy and metaphor, but it is quite easy to show that they often make basic errors in reasoning, and are not all that good at identifying and regurgitating factual information.³ What this means is that the quality of the training data really matters if you want to get useful information back from a prompt, because the model can't distinguish between good and bad answers - just answers that are "nearer" or "further away" from the given prompt in vector space. This need for accurate training data brings up a whole host of logistical issues, and remember, these models absolutely require a great deal of training data, which means that quality is sometimes sacrificed for quantity, meaning that you might sometimes get incorrect or problematic recommendations.

There are so many other technical problems here. For example, how do you continually and efficiently update the model as coding standards and practices change? "Continual learning" is a difficult problem in machine learning, and an active area of research that is far from solved.

So, long story short, the technology looks cool and useful, mostly insofar as it can provide you with information you might otherwise google, but faster, but I bet you anything it is still fairly error prone and inconsistent. Moreover, there are ways to create programing recommendation systems that aren't heuristic/machine-learning style systems that might improve your quality of life even more in the short term.

Finally, if you want something like this for Stata now and don't want to wait, it might be possible to train your own model using fauxpilot. Fair warning, it would be a fairly significant project. If you have time to get into the weeds, check out this discussion board post.


-- NOTES --


¹As a particularly egregious example of this in Stata, occasionally when I'm doing something particularly weird, the text editor will highlight some code that it identifies as containing an error, but the code will execute as expected. This behavior is very strange since in my view the program that executes the code and the one that identifies errors should be one-in-the-same.

² You can get a motherboard that supports multiple graphic cards and some specialized graphics cards from Nvidia (these are upwards of $5K a piece) if you don't want to pay for the use of someone else's datacenter. Might be worth it if you're a professional. You also need a ton of RAM and disk space.

³ I play around with these in sandboxes every chance I get, and I honestly think when they get facts or reason through things correctly, it's more by chance than anything else. Large language models on their own are often catastrophically wrong, although it is worth noting that there is other technology separate from the large language model that insures copilot suggestions are correct more often than they would otherwise be for the large language model alone. Of course, correct-answer lookup is also an ongoing area of research in and of itself.

[William Lisowski]

Also this experimental support for the experience Daniel Schafer reports in his footnote 3.

Do Users Write More Insecure Code with AI Assistants?

[Tarjei W. Havneraas]

Thank you for the very informative reply, Daniel! There was a several things here I was not aware of which certainly is interesting to know regarding future development of software.

[paulvonhippel]

Any update on this? It's been about a year.....

[FernandoRios]

I have been using copilot doing some simple codes with vsc and Stata
it works , but not all the time.
if copilot learns from users interaction, it may get better as
more Stata users work with this.

[paulvonhippel]

How do you run Stata from Visual Studio Code?

[FernandoRios]

I have been using nbstata with python for dynamic documents or class notes
not much more in terms of advanced programming

[Mikko Rönkkö]

Here is an example https://youtu.be/cOwmYXYkxWA?si=BAEvahXgN9EgGV1j&t=229 of Copilot-like function in Stata with Visual Studio Code, Cllne, and Claude 3.5 Sonnet.

(I hold no commercial interest in the YouTube channel: there are no adds)

[FernandoRios]

Dear Mikko
Thank you! This is really good.
Best wishes