How much computer specs do I have to analyze large data for fixed effect

[Jeon Joan]

Hi
I have very large daily data.
The number of rows is 1 million and the number of columns is 50.
However, I anticipate that the number of columns will increase to approximately 5,000 by adding daily dummy variables, weekly dummy variables, and weekly x city dummy variables.
I would like to use this data to analyze fixed effects.
My computer spec has 4 core CPU and 8 gigabytes of RAM.

Results of two experiments in STAT 13 SE version
The computer stopped for 8 hours and 10 hours, respectively.

I do not know what to do, since this is the first time I have to turn around this big data.
Is there a good way?
I am willing to add RAM if necessary.

Thanks for reading
Thanks for any advice.

[FernandoRios]

You probably do not need to create those additional variables. Look into the community contributed command reghdfe by Sergio Correira
best regards

Fernando

[Jeon Joan]

thank a lot Fernando
i will check reghdfe

[Sergio Correia]

Hi Jeon,

If you only have 8gb of RAM, consider using the "compact" and "pool(1)" options of reghdfe, as well as adding all the dummies and fixed effect as categorical variables in absorb(). With that, you should be able to run the regression in a couple of minutes.

[Jeon Joan]

Thank! Sergio
I did not know what to do.
If I can finish the paper, it will be thanks to you.
However, from yesterday, "ssc install reghdfe" in the command window, the result window, "connection timed out" came out.
Is there any other installation method?

[FernandoRios]

Search for reghdfe GitHub, Sergio has a webpage there where the lastest version of of that command is kept
i believe there are instructions of how to install it from there too.
best

[River Huang]

Please try:

// https://github.com/sergiocorreia/reghdfe

// reghdfe

* Install ftools (remove program if it existed previously)
cap ado uninstall ftools
net install ftools, from("https://raw.githubusercontent.com/sergiocorreia/ftools/master/src/") replace

* Install reghdfe 5.x
cap ado uninstall reghdfe
net install reghdfe, from("https://raw.githubusercontent.com/sergiocorreia/reghdfe/master/src/") replace

* Install boottest for Stata 11 and 12
if (c(version)<13) cap ado uninstall boottest
if (c(version)<13) ssc install boottest, replace

* Install moremata (sometimes used by ftools but not needed for reghdfe)
cap ssc install moremata, replace

ftools, compile
reghdfe, compile

[Jeon Joan]

thank a lot, guys :D