Different coeffecients, significance levels on Stata

[Nick Baradar]

Hello everyone,


I am running a panel data fixed effects regression and sometimes when I run the command, it gives slightly different coeffecients together with slightly different p-values, I wonder if anyone else has encountered the same issue and whether there is a particular reason.

The command is the following:

xtreg DV IV Interactions Controls, fe vce(robust)

Thank you,
Nick

[Clyde Schechter]

The problem most likely arises from something in the code that precedes the -xtreg- command itself. Something in the data management is resulting in different data being used when you get to -xtreg-.

The usual cause of indeterminate behavior like this is commands that explicitly or covertly sort the data on a sort key that does not uniquely determine the order of the data set, and then perform some sort-order dependent operation. For example, consider

bysort a b c: keep if _n == 1

If a, b, and c do not uniquely identify observations in the data set, then the order of observations within the groups defined by a, b, and c is randomized. Consequently which observation is kept is indeterminate and you can end up with a different data set each time you run the code. Subsequently performing a regression will therefore also produce varying results on repetitions of the code.

So you will need to scrutinize the code for any command that sorts the data, and ascertain whether they may be the source of this indeterminacy, and, if so, either fix them so that the sort key uniquely specifies observations, or remove the operations on the sorted data whose results are order-dependent. This task can be difficult because some commands may sort the data behind the scenes without your awareness.

[Nick Baradar]

Dear Clyde,

Thank you so much for your feedback. Currently, I've reviewed my do-file and haven't identified any commands that would introduce randomness to the process. Nonetheless, I've been considering a potential approach and would appreciate your opinion on it.

I'm curious about the feasibility of comparing two datasets in Stata to pinpoint the specific variables where the observations differ. My guess is that this would enable me to examine the corresponding operations involving these variables within my do-file. However, I am not aware of a command on Stata that would perform it and the data is too large to perform it on Excel.

Warm regards,
Nick

[Clyde Schechter]

-help cf-

[Nick Baradar]

Thank you very much, Clyde I did compare the datasets that yielded different results, and they were identical. It's very unfortunate, honestly, because now my only concern is the command itself.

[Joseph Coveney]

. . . I did compare the datasets that yielded different results, and they were identical.

Did you compare them at the state immediately before executing xtreg?

[Nick Baradar]

Yes, Joseph, indeed, right before executing the command. Interestingly, the command produces only two variations of results, which, although very close to each other, still exhibit differences and are significant in terms of their interpretation. I run the command on R, and it yields consistent results. I guess there is some randomness in the way "xtreg" with the option "fe vce(robust)" treats certain observations.

[Joseph Coveney]

I guess there is some randomness in the way "xtreg" with the option "fe vce(robust)" treats certain observations.

If so, then it's a bug and I urge you to contact StataCorp technical services and have them look into fixing it.

[John Mullahy]

If you set a seed immediately before the xtreg command does this behavior persist? E.g.

set seed 2345
xtreg ...

[Nick Baradar]

Hello John, thank you for your input. I would have assumed this should work, but interestingly, it did not work either! I am preparing an email to send my do-file to StataCorp technical services, currently working on anonymizing it, which takes a good amount of time.

[Nick Baradar]

I should perhaps make a note in the Stata 19 wishlist to add a command that would systematically anonymize the do-files