ASTILE: New Package - Fast and byable alternative for XTILE

[Attaullah Shah]

The slow speed of xtile has long irritated me . As an attempt to find a speedy alternative, I posted astile program on this forum (http://www.statalist.org/forums/foru...tile-vs-astile). However, that version had issues and was abandoned. Even so, my motivation for a speedy alternative never died. Thanks to Kit Baum, new version of astile is now available on SSC. Here is the description of the program and some speed tests. To install the package, type

ssc install astile
help astile

Title

astile - Creates variable containing quantile categories


Syntax

astile newvar = exp [if] [in] [, nquantiles(#) by(varlist)]


Description

astile creates a new variable that categorizes exp by its quantiles. For example, we might be interested in making 10 size-based portfolios. This will involve placing the smallest 10% firms in portfolio 1, next 10% in portfolio 2, and so on. astile creates a new variable as specified in the newvar option from the existing variable which is specified in the = exp. Values of the newvar ranges from 1, 2, 3, ... up to n, where n is the maximum number of quantile groups specified in the nq option. For example, if we want to make 10 portfolios, values of the newvar will range from 1 to 10.


astile is faster than Stata official xtile. It's speed efficiency matters more in larger data sets or when the quantile categories are created multiple times, e.g, we might want to create portfolios in each year or each month. Unlike Stata's official xtile, astile is byable.

Options

astile has the following two optional options.

1. nquantiles
The nq(#) option specifies the number of quantiles. The default value of nq is 2, that is the median.

2. by
astile is byable. Hence, it can be run on groups as specified by option by(varlist).


Example 1: Create 10 groups of firms based on thier market value

.

 webuse grunfeld
 
    . astile size10=mvalue, nq(10)

Example 2: Create 5 groups of firms based on thier market value in each year

. webuse grunfeld
 
    . astile size5=mvalue, nq(5) by(year)

Limitatons

This version of astile does not support weights, altdef and cutpoint options that are available in the official xtile function. In the next version, I plan to include some of these options.

SPEED COMPARISON
The following tests are performed using Stata 14.2. The test results might vary from computer to computer based on CPU speed.

Without by Option

* To generate an example data set of one million observations

clear
set obs 1000
gen year=_n+1000
expand 1000
bys year: gen id=_n
gen size=uniform()*100

timer clear

timer on 1
egen xt10=xtile(size), nq(10) // from egenmore package
timer off 1


timer on 2
astile as10=size, nq(10)
timer off 2

timer on 3
xtile of10=size, nq(10) // Stata official
timer off 3

assert as10==of10
timer list
   1:      4.71 /        1 =       4.7130
   2:      3.61 /        1 =       3.6140
   3:      9.01 /        1 =       9.0050

* Both xtile (from egenmore package and astile perform three time faster than the official xtile, with marginal speed efficiency for astile over xtile from egenmore.

With By Option

Since the official xtile does not have a by option, I would compare astile with xtile from egenmore package.

. timer clear

. timer on 1

bys year: egen yxt10=xtile(size), nq(10)
timer off 1
timer on 2
bys year: astile yas10=size, nq(10)
timer off 2
timer list

 1:   1037.37 /        1 =    1037.3700
 2:    198.31 /        1 =     198.3080

assert yxt10==yas10

[Nick Cox]

Fast always beats slow with nothing else said, so speeding up slow commands is clearly good.

But as with Attaullah's asrol, discussed at

http://www.statalist.org/forums/foru...peed-advantage

there is a claim here of speed without full access to the code: the Mata code at the heart of this is presented as compiled code and programmers can't inspect it.

I feel more or less the same way about this practice as I do about magic, such as a card trick or someone sawn in half as far as you see, except that they aren't. With magic, evidently someone has a smart trick but we don't know what it is. Similarly, I can't learn from code I can't see and I can't fully discuss it either (e.g. what are its hidden assumptions, if any).

I don't know why some programmers hide Mata code in programs made public.

That said, there is plenty of scope for various comments on the details.

1. The egen function xtile() in egenmore (SSC) was written for Stata 8.2 by Ulrich Kohler. Uli can speak for himself, but it's offered as a convenience wrapper. In Attaullah's second example, there are 1000 distinct groups to loop over, so yes, it's slow. I've not seen anyone before wanting to bin using quantiles, but separately within 1000 distinct groups, but someone will tell me they do often want this. There is no reason to use this function if you only want a single group, as xtile does that directly.

2. The comparison has missed a serious alternative candidate, Michael Stepner wrote fastxtile and it's been on SSC since 2014. That program has several useful bells and whistles more than astile, just not (so far as I can see) support for by:.

3. There is another benchmark with bins ideally of equal size when the quantiles are regularly spaced in probability terms, as they are in the only version of quantile binning tacked by astile. You just round the ranks suitably. That method is exact if (and only if) ties do not bite at quantile boundaries. With a million random uniforms generated as a float that's possible but very unlikely. But it's a reference level.

I tweaked Attaullah's script by leaving out the generation of identifiers (irrelevant) and adding an explicit seed, important for reproducibility, and adding calls to Michael's code and what I am calling the naive approach of rounding ranks.

Here are my results:

. clear

.
. set obs 1000
number of observations (_N) was 0, now 1,000

. set seed 2803

. gen year=_n+1000

. expand 1000
(999,000 observations created)

. gen size=uniform()*100

.
. timer clear

.
. timer on 1

. egen xt10=xtile(size), nq(10) // from egenmore package

. timer off 1

.
. timer on 2

. astile as10=size, nq(10)

. timer off 2

.
. timer on 3

. xtile of10=size, nq(10) // Stata official

. timer off 3

.
. timer on 4

. fastxtile ms10=size, nq(10)

. timer off 4

.
. timer on 5

. sort size

. gen naive10 = ceil(10 * _n/_N)

. timer off 5

.
. assert as10==of10

. assert ms10==of10

. assert naive10==of10

.
. timer list
   1:      1.70 /        1 =       1.7000
   2:      1.33 /        1 =       1.3260
   3:      2.50 /        1 =       2.4960
   4:      1.20 /        1 =       1.2010
   5:      0.86 /        1 =       0.8580

.
. timer clear

. timer on 1

. bys year: egen yxt10=xtile(size), nq(10)

. timer off 1

.
. timer on 2

. bys year: astile yas10=size, nq(10)

. timer off 2

.
. timer on 3

. bysort year (size): gen ynaive10 = ceil(10 * _n/_N)

. timer off 3

.
. timer list
   1:    772.65 /        1 =     772.6540
   2:    111.14 /        1 =     111.1440
   3:      1.05 /        1 =       1.0540

.
. assert yxt10==yas10

. assert yxt10==ynaive10

The new details are that

1. fastxtile is faster yet than astile for the simpler problem. Personally I don't mind waiting 2 seconds for 1 million values to be binned. I can check Statalist while I am waiting. (If I had 1 billion values, I would care.) But if speed is claimed, astile is not, on this limited evidence, the fastest beast in the zoo.

2. While Attaullah's program is as before much faster than the egen function for the groupwise problem, naive code is 100 times faster and (in this case; I was lucky) gives the same result. That doesn't mean it's better code, as it won't handle ties properly. The question is what is astile doing to be so slow when the only difference is fixing what happens with ties?

Good news then: scope for many further improvements is implied.

[Michael Stepner]

There is in fact another creature in the zoo! Mathieu Gomez created an egen wrapper for fastxtile, which is available in SSC under egenmisc. I have not used this code or inspected it, but I believe it is based on my fastxtile code.

I can't blame Atullah for being unaware of these alternatives: it's hard to keep up with the growing catalog of options. During the time Stata 14 was in development, I advocated for replacing xtile with fastxtile. My own tests show that they produce identical output, but Stata has their own certification scripts which they could subject fastxtile to. I wouldn't be offended if they removed the additional functionality before placing it in mainline Stata, if they'd prefer not to add any new bells and whistles to xtile. And I've released the code to the public domain under CC0, so there are no licensing issues to speak of.

If anyone at StataCorp would like to replace xtile with fastxtile for Stata 15, they'd have my full support in the endeavor.

[Nick Cox]

Michael: Glad you spotted this thread.

StataCorp can speak for themselves, but by and large they don't adopt user code any more. I guess the reason is three-fold:

1. Really good user code such as your own fastxtile is already accessible and easy to download and use, so the margin in StataCorp adopting it is smaller than you think. Otherwise put, the main StataCorp solution is providing net install so that it's trivial for most people to pull stuff across the net. (But I don't forget that a large class of users are behind firewalls that make that difficult if not impossible.) The success of this strategy is indicated indirectly by the large difficulties that many users have in keeping track of what is user-written and what is not!

2. Adopting user-written stuff creates a burden of testing, documentation and user support that could be spent otherwise on really big projects that users would hardly take on, such as multiple imputation or Bayes.

3. We all understand best what we create ourselves. That's really important. So, StataCorp developers look for problems that users are running into and then create their own solutions. (Sometimes, they take several years to get to that, but the company has no interest in implementing transient fads that researchers will only play with for a couple of years before the next version of sliced bread arrives.)

[Michael Stepner]

Nick: Your effective use of an @-tag brought this thread into my email inbox

I understand the reasoning for why StataCorp doesn't generally adopt user code. And of course they would need to decide whether it's worth the effort to adopt fastxtile. This may be a rare case where the benefit exceeds the cost. StataCorp could run fastxtile through their existing certification scripts, since it follows the same syntax as xtile. They could remove the bells and whistles if they'd like to keep the xtile documentation largely unchanged (save updating Author and Date). Someone at StataCorp would likely need to read the code to double-check that it's sensible, but fortunately it's well-structured and only 200 lines.

Despite the fact that user scripts are easily obtainable, defaults are powerful. I would guess the ratio of xtile to fastxtile users is something like 10,000:1. Perhaps StataCorp would deem it worthwhile to spend a couple employee hours on this. It would likely save their user base many hundreds of hours in the aggregate (although most users might save only a few seconds or minutes of their time).

Nick, to be honest I don't know the appropriate way to get this suggestion in front of the right person at StataCorp. The last time I made the suggestion, it was at an annual Stata Conference. Do you have advice about how I should suggest the replacement of xtile with fastxtile to StataCorp? I wouldn't be offended if they decide not to follow it.

[Nick Cox]

http://www.stata.com/company/contact/ gives a general email address. They will know who to contact.

[Ariel Karlinsky]

I agree with michael. while user-added capabilities are a great feature of stata, I would have loved to see stata adopt some of the most used and known user packages into vanilla stata. as was stated in this thread, it's sometimes very hard to know what you don't know - so that's a big limitation on whether or not you install user packages. The fact that the stata FAQ itself has some mentions of user-packages as well is, sort of an admission of that. I'm talking for example about spmap, which I originally came upon by chance as I thought that stata has no geo-mapping capabilities at all.

When programming code is well written and publicly available, I would have liked to see stata-corp utilize that "openness" for the benefit of all users.

[Attaullah Shah]

Dear Nick, thanks for your time. Several of your comments need my reply. First of all, I think you have read my first post where I posted version 1 of astile. The post is located here http://www.statalist.org/forums/foru...tile-vs-astile. The 'so-called naive 'approach that you have posted here for making qunatile groups was actually used in that first version. But then you commented and used "MUST and SHOULD "

That aside, any serious program in this territory, in Stata or in any other language,

1. must handle ties intelligently

2. must handle missing data intelligently

3. should support groupwise calculations

4. should support equal or unequal bins (e.g. some people might want to bin with boundaries at 5, 10, 25, 50, 75, 90, 95% points of a distribution and not just according to so many quantiles equally spaced in probability terms).

5. should lend itself to application to several variables

The use of "must" and "should" matches my suggestions on what is essential and desirable respectively.

The so-called naive approach is 100 time faster, but does handle ties and missing values. So I think it is not suggested for any serious analysis. See the following example

clear
webuse grunfeld
sort mvalue
replace mvalue = . in 4
replace mvalue=. in 5
replace mvalue=. in 18
replace mvalue=. in 198
replace mvalue=. in 12
gen naive1 = ceil(10 * _n/_N)
xtile off10=mvalue, nq(10)
astile size10=mvalue, nq(10)
assert size10 ==off10

. assert naive1 ==off10
21 contradictions in 200 observations
assertion is false
r(9);

Nick further commented that

I don't know why some programmers hide Mata code in programs made public.

. I think he has already answered this here http://www.stata.com/statalist/archi.../msg01124.html

[Nick Cox]

Attaullah: Let me rephrase one point more precisely: I don't know why you are hiding your code and even when the point is raised you don't explain. Your decision, to hide or not, to explain or not, and no obligation that you don't take on yourself, but code hidden cannot be discussed.

The role of the naive code is already explained, to act as a benchmark showing how fast the code could run in perfect conditions.. You're right that missing values are ignored and so the code will not handle those correctly. I was just running the code on the test problem you set out yourself. If you post a test problem with missings, the naive code could be adapted and compared.

Thanks for reminding me of the ideals I laid out. Points #4 and #5 cited in #8 are presumably now on the astile agenda.

A main theme of this thread is exactly what you have made it: claims about speed for your code. Just with asrol, astile runs faster than what you compared it with, and well and good, but corrections and qualifications are needed. fastxtile is faster for the first problem, and astile is still very slow for your second problem and I don't know why.

As Michael Stepner commented, there is also an egen version of his command to run on the second problem.

[Attaullah Shah]

Michael Stepner, You are right. I was unaware of fastxtile. As compared to the posted version of astile, fastxtile is faster by few seconds without using the by option, and with the by option, the difference increases. I have been working on the next version of astile, which still needs certification. So far, results of that beta version show that it is even faster than fastxtile. I am working on the certification, once ready, I shall post that version first on Statalist and then update the SSC version.

[Attaullah Shah]

Nick, I have my own reasons of hiding the code, as does the Stata Corporation, which you have yourself defended on different occasions, one of which I have already referenced

whereof one does not want to speak, thereof one can be silent

( http://www.stata.com/statalist/archi.../msg01124.html). My objective of posting about my new programs on Statalist is to spread the word among users so that if someone wants to use it, they should know.

but corrections and qualifications are needed

I claimed astile was faster than Stata Official

[Nick Cox]

Hidden code: As I said, it's your decision and I won't press the point further.

What remains interesting to anyone concerned with these calculations is which programs are correct, most versatile and fast(est).

As I write, the last post in the thread on asrol is my post #16 at http://www.statalist.org/forums/foru...dvantage/page2 with a suggestion that the help wording remains misleading. That suggestion still holds.

[Attaullah Shah]

I plan to change the help file in the next version, which brings significant changes in the way asrol understands different data structures, thereby overcoming its present limitation of not efficiently handling data with too many gaps in the range variable. But till then, you are right, the help file needs correction.

[Attaullah Shah]

I am posting version 3.0 of astile here. My initial tests show that version 3.0 of the astile program is reliable and is the fastest among available alternative programs, such as xtile, fastxtile, and xtile from egenmore. I have posted the new version on my website and can be downloaded by typing the below command

net install astile, from(https://sites.google.com/site/imspeshawar)

If proxy settings do not allow the download, it can be manually installed by copying the attached files to:

copy and paste
astile.ado
astile.sthlp
 in the following folder
C:/ado/plus/a

For some reason lasn.mlib cannot be attached, so here is the direct link https://sites.google.com/site/imspes...edirects=0&d=1
and copy lasn.mlib to

C:/ado/plus/l

Since I have already shown speed tests for official xtile and xtile from egenmore, following are the speed tests for fastxtile( Michael Stepner) and fastxtile from egenmisc (SSC)

clear

. set seed 1234

. set obs 1000
obs was 0, now 1000

. gen year=_n+1000

. expand 1000
(999000 observations created)

. gen size=uniform()*100

.
.
. *Withou by option
. timer clear

. timer on 1

. egen fast10=fastxtile(size), nq(10)

. timer off 1

. timer on 2

. astile as10=size, nq(10)

. timer off 2

. timer list
   1:     18.17 /        1 =      18.1720
   2:      5.42 /        1 =       5.4200

. . assert as10 ==fast10

.
.
. timer clear

.
. timer on 1

. bys year: egen yxt10=fastxtile(size), nq(10)

. timer off 1

. timer on 2

. bys year: astile yas10=size, nq(10)

. timer off 2

. timer list
   1:     19.95 /        1 =      19.9500
   2:      7.16 /        1 =       7.1620

. assert yxt10 ==yas10

.
end of do-file

[Nick Cox]

Thanks for the extra news. Some small points of clarification from me:

1. The egen function fastxtile is not a wrapper for @Michael Stepner's code. Matthieu Gomez, its author, only used the same name. I think Michael was guessing from the name that it was a wrapper but looking at the code contradicts that. There is no Mata code there, for example.

2. The only advantage of any egen function to users lies in their support for a by: or by() calculation, as calling code from within egen can only slow it down.

Some small points of clarification now requested:

1. I don't see here timings for fastxtile and your new astile.

2. What's puzzling now is that your timings for the new astile show that it is slower than it was in post #1 (5.4 s rather than 3.6s). Is this a different problem? (Looks the same to me.) Are you using a different machine?