We are using ODBC Load to import large datasets from a MySQL database into stata.
Basically, we are doing following:
1) In the database we generate a view containing all the (many) columns, joining tables as needed.
2) In stata we generate a global macro with a comma-separated list of variables, and a macro containing the connection string.
3) Then we load data using :
odbc load, exec("SELECT $longvarlist FROM ViewThatJoinsTables ") ///
connectionstring("$OurConnection")
This works perfectly up to about 1'800 columns. With more columns, stata frequently crashes. If it does not, however, the load works remarkably fast.
Is there a way to improve our solution to get more stable results? (reducing the number of columns is no option, unfortunately)
Basically, we are doing following:
1) In the database we generate a view containing all the (many) columns, joining tables as needed.
2) In stata we generate a global macro with a comma-separated list of variables, and a macro containing the connection string.
3) Then we load data using :
odbc load, exec("SELECT $longvarlist FROM ViewThatJoinsTables ") ///
connectionstring("$OurConnection")
This works perfectly up to about 1'800 columns. With more columns, stata frequently crashes. If it does not, however, the load works remarkably fast.
Is there a way to improve our solution to get more stable results? (reducing the number of columns is no option, unfortunately)
