[txttool] Command Error Interpretation

[Julius Doyle]

So I'm trying to do a text analysis on what is roughly 22,000+ qualitative (string) descriptions.

Using the following command:

txttool incidentsummaries, gen(incident_summary)

I get the following message:

[ st_addvar(): 3300 argument out of range
mm_txttool(): - function returned error
<istmt>: - function returned error

r(3300); ]

What am I doing wrong? What does this mean?

[Mike Lacy]

I suspect you're not doing anything wrong. After examining the documentation for Mata error 3300 and for -st_addvar-, I can't get the latter command to generate this error by making simple mistakes, such as trying to add an existing variable or trying to add a variable to a dataset with 0 observations. I would be inclined to guess there's some kind of limitation or bug in -txttool- and contact the author. (It appears that the work of -txttool- is done by modules of Mata code that are already compiled and not something a user could examine.)

Regarding a stab at diagnosis: Can you get the command to work on smaller subsets of your data, say 50 observations, then 500, then 5000, etc.? Or, perhaps are some of your incident summaries very large and does eliminating them permit the command to work? Can you get -txttool- to work on *any* example text?

[Jeremy Skog]

Hello,

I was getting a similar error. After some investigation it looks like the error is caused by long strings. When I dropped all observations with a string length of greater than 2,000 the txttool command ran correctly. I suspect that there is some issue with cells longer than 2,048 characters as these observations had grayed out text when the command returned and error, but all the grayed out observations were dropped when the error ran.

Unfortunately I am not sure of how to update the Mata code to work around this restriction. Please post an update if you have an idea.

[Jeremy Skog]

Hello,

I was getting a similar error. After some investigation it looks like the error is caused by long strings. When I dropped all observations with a string length of greater than 2,000 the txttool command ran correctly. I suspect that there is some issue with cells longer than 2,048 characters as these observations had grayed out text when the command returned and error, but all the grayed out observations were dropped when the command ran.

Unfortunately I am not sure of how to update the Mata code to work around this restriction. Please post an update if you have an idea.

[Mike Lacy]

Good find. I'd suspect this does have to do with some program feature of -txttool- rather than any limit of Mata. Per my comment above, I don't think the source code for -txttool- is accessible to users.