[Daniel Schaefer]

I'm really enjoying watching all of the ChatGPT content coming out in the programing and medical communities. That said, I wonder how one might go about systematically evaluating the quality of the programming advice. To what extent does the model get things wrong, and are there systematic conditions under which it generates incorrect responses?

[Carlo Lazzaro]

George:
yes, I was impressed too!

[George Ford]

That's cool. ChatGPT is a co-author of that editorial.

[Carlo Lazzaro]

Some interesting issues concering the (mis)use of AI at large are covered here (https://pubmed.ncbi.nlm.nih.gov/36549229/).

[Joro Kolev]

Nooooo... ChatGPT has come to take our jobs !

[George Ford]

I've asked it a bunch of questions and it makes a lot of factual errors. It will provide similar language for similarly worded questions. I think these artifacts will be the "tell." I suspect this will not be unlike the migration of "regression by hand" to statistical software. It will make writing papers easier for sure, but it will never replace a clever question.

[Felix Bittmann]

Wow, I wonder what happens now when students all submit super well done statistical analyses for their homework... interesting times ahead.

[Jared Greathouse]

It got the code for the nation (state), not each of the 16 Spanish states, but hey, either way, this would work nicely if the dataset could be found.

It would also be super useful to scrape together Wikipedia tables if they had useful data

[George Ford]

As for the Spanish GDP. I don't use Python so don't know if it works.

[George Ford]

This could be useful.

[Jared Greathouse]

I wonder how nice it would play with data collection. Like, even if I asked "Where might I get Spanish state GDP/cap from 1990-2010", or even if I asked it to scrape together python code to do that (if I fed it like a link or something).

Either way, I don't take think it'll overtake ado programmers any time soon

[George Ford]

For those who don't want to write it up themselves.

[George Ford]

It seems like it could answer a bunch of the more simple coding questions that get asked here.

[Jared Greathouse]

Yep that's my friend's command. When i asked him, he said augsynth is pretty much completely Ridge regression instead of OLS followed by ridge. Justin also noted how R and Stata optimize sort of differently, so he said they might not be totally comparable. But I'll see! Maybe I'll test them side by side.

But yeah, chatgpt seems super useful, and it could be sharpened over time for programming

[George Ford]

Might look at this.

https://twitter.com/aspaglayan/status/1378124828524179456?lang=en