Most appropriate way to compare two similar models - coefficient or R-squared?

[Sam Murgatroyd]

Hello!

I want to compare which of two variables better explains my dependent variable, which is country-level cigarette consumption - I'll call the two independent variables I want to compare Most Interesting Variable 1 (MIV1) and Most Interesting Variable 2 (MIV2).

Because MIV1 and MIV2 are highly correlated, I don't want to include them in the same regression. So I have decided to split the model of factors that explain country-level cigarette consumption into two models that only differ in whether they include MIV1 or MIV2. I therefore run a model as shown in eq(1) and a model as shown in eq(2), where X_jt is a vector of country-level macroeconomic factors. The regression is (initially) run using OLS with country and year fixed effects. Standard errors are clustered at the country-level and the panel of 56 countries over 5 years is balanced in both cases.

Y_jt= β₀ + β₁MIV1_jt + β₂X_jt + δ_t + α_j + u_it (Eq1)

Y_jt= β₀ + β₁MIV2_jt + β₂X_jt + δ_t + α_j + u_it (Eq2)

Question:

1. Does one use the size and statistical significance of β₁ in each model, or the R-squared of each model, as the basis of saying whether MIV1 or MIV2 better explains the dependent variable, y?

2. If it is best-practice to compare the R-squares between models that only differ by one independent variable as in eq1 and eq2, how should one conduct such a comparison in the case of a dynamic panel model which includes the first lag of the dependent variable when this is run with something like systems GMM, which doesn't give you an R-squared?

Thank you!

Sam

[Richard Williams]

Do you absolutely have to choose one or the other? Are they each supposed to be measuring the same thing? Or are they supposed to be measures of two different but highly correlated things? If they are different operationalizations of the same concept I can see going with the one that seems to be better measured, but if two different things are being measured and both make theoretical sense to include in the model I hate to just drop one if I don't need to.

If they are measured the same way, would it make sense to just add them together or otherwise constrain their coefficients to be equal?

If you absolutely positively have to pick one and only one, are the comparisons you propose in conflict with each other? My guess is you would get the same winner either way.

[Sam Murgatroyd]

Hi Richard Williams,

Thank you for the response!

The two variables I am interested in are related, but they measure different concepts.

There is a debate in my field of study about which of two "benchmarks" governments should target when it comes to setting tobacco taxes. One best practice policy rule is to target the affordability of tobacco products such that the affordability of these products is decreasing over time. In this literature, affordability is measured with something called the Relative Income Price, calculated as the ratio of the price of 100 packs of the most-sold brand of cigarette to per capita GDP price (=price/income). The other benchmark is based on a targeted tobacco tax incidence - the best 'practice' rule says that governments should strive to set their tax rates such that the tobacco tax incidence is 75% of the retail price.

I have data on the RIP and the total tax incidence for a panel of 56 countries, and I am interested in seeing which measure - the Relative Income Price, or the total tax incidence - has best explained per capita cigarette consumption between 2010 and 2020 (data on each of these measures is only available every two years). But, because both the RIP and the tax incidence are a function of the retail price, including them in the same regression is problematic due to collinearity.

I am interested in ascertaining the most appropriate way to use regression analysis on a static panel in Stata to answer the question of which one - RIP or tax incidence - offers more explanatory power of per capita cigarette consumption at the country level. To this end, I am confused about whether I use the R-squared from each model, or the coefficients obtained on my two variables of interest in each separate regression, to inform this discussion.

If the advice is to compare the model R-squareds in the static panel, I become even more confused on how one goes about doing this kind of comparison in the dynamic panel setting, where the first lag of the dependent variable is included, and the estimation technique does not return a fit statistic like R-squared (e.g systems GMM).


Thank you!


Sam

[Richard Williams]

I am still reluctant to drop a variable. Have you actually tried having both in the model at the same time? Small sample sizes will sometimes limit how many variable you can include, but if both vars make good theoretical sense I would try to keep both if I could.

But, if you can only keep one, I think you keep the one that is more statistically significant. Which I think would be the same as the var that does the most for R Square.

[Sam Murgatroyd]

Thank you for your response, Richard. I appreciate your time on this. FYI: My original proposal was to include both, but this was rejected by people that I need to listen to, so I am following orders. Thank you!

[Clyde Schechter]

Perhaps I am misunderstanding #3, but it seems to me that there is no possible way to compare these policy approaches statistically. Targeting tax incidence at 75% of the retail price is a single policy with no free parameters. There is some guesswork involved in how to do that given that manufacturers, wholesales, and retailers have some flexibility to blunt the impact of the tax by lowering their part of the price--so it may take a few iterations to get to 75%, but the policy has a single, clearly defined goal. By contrast, the RIP method is a one-parameter family of policies. Two jurisdictions might target two different levels of the ratio (and, again, due to pricing flexibility it may take a few tries before they reach their targets) with, presumably, correspondingly different impacts on smoking prevalence (and, perhaps more important, incidence of smoking among previously non-smoking youth). One might due better than the 75% incidence ratio and the other worse just because of the different target ratios chosen. It's like asking whether one light is brighter than another when one is on a variable dimmer switch and the other gets a fixed electrical input. It makes no sense. What am I missing here?

[Sam Murgatroyd]

Hi Clyde,

Apologies that my post caused confusion.

The 75% incidence target was established in a World Bank paper in the late 1990s as a way of guiding countries (mostly LMICs who had historically not used excise taxes on tobacco all that much at the time) on the rate at which to set tobacco taxes. The advice was: set the tobacco tax rate (and adjust it at least annually) so that the total tax incidence (sum of excise, VAT, and any other taxes = 75%). A country's tobacco tax incidence is measured and monitored by the WHO percentage of retail price of the most-sold cigarette brand made up of excise taxes, sales taxes and import duties.

The affordability targeting rule developed in the early 2000s because fast economic growth in many LMICs at the time made it clear that its not only the price of tobacco that matters in terms of reducing consumption; but also the extent to which people can afford to buy these products. This led to another school of thought that says - instead of targeting a particular tax incidence, countries should rather target reductions in affordability over time. In practice, this means taking income growth developments into consideration when setting/adjusting your tobacco tax rates such that the RIP is increasing over time. Affordability is measured by the WHO as the Relative Income Price (RIP) - the ratio of the price of 100 packs of the most-sold cigarette brand to GDP per capita (the amount of GDP per capita required to buy 100 packs of the most-sold brand in a country).

I am not only the policy side of this and I hear you on the complications of getting this right in practice. I just am an interested student that sees these two benchmarks, and wonders which metric - the RIP or the tax incidence - has better explained observed cigarette consumption. I have data on the observed RIP and the observed tax incidence for 56 countries between 2010 and 2020 (with data available every 2 years, 2010, 2012, ..., 2020). Very few countries have a tax burden = 75% of the retail price but the tax burden has been increasing over time in many countries; the RIP has been increasing (cigarettes have become less affordable over time) in some countries, while in others, it has been decreasing.

Using the the observed tax burden and the observed RIP for this sample countries, I want to investigate the relative importance of the tax burden vs the RIP in explaining cigarette consumption in the sampled countries. My question in this forum arose because I have seen dominance analysis used to compare the relative importance of independent variables in a single regression; but putting RIP and the tax incidence in the same equation as dominance analysis requires is not an option for me since both RIP and the tax incidence are a function of the price of the most-sold brand of cigarette. So I split the equation into two models that I want to compare (one model includes the observed RIP and one includes the observed total tax incidence) and wondered how I would go about comparing these identical models that only differ in respect of whether they include the RIP or the tax incidence in the case of a static fixed effects OLS model, and a dynamic version of the model estimated using systems GMM.

Thank you!

Sam

[Richard Williams]

This is outside my area of substantive expertise but, regardless of what you’ve been told, I would at least try it with both variables in the model. If the standard errors skyrocket because of multicollinearity, that helps justify your decision to reluctantly use only one. On the other hand, if multicollinearity isn’t that big of a problem, then the model is better specified with both in there and you can more directly compare their effects.

[Jeff Wooldridge]

Sam: It's hard to admit, but I don't think we have a good way to choose between a static and dynamic model. They identify different effects on x. Whether we hold fixed past y makes a difference for how we interpret the coefficient on x. If the choice is based on fit, the dynamic regression will almost always win. It's unusual that the lagged outcome would not be the best predictor of the current outcome. As you've implied, it's not easy to obtain a goodness-of-fit statistic with fixed effects when these are removed by Arellano-Bond differencing.

The causal inference literature is making some headway, I think, by applying "local projections." But I haven't looked at it carefully to know whether what is being identified under what assumptions makes the most sense. I think you can try it both ways. When you control for lagged y, the coefficient on x is best interpreted as an immediate or short term effect. Rather than included lagged y, you can included lagged x and estimate a kind of distributed lag.

[Sam Murgatroyd]

Thank you, Richard, and Jeff, for your inputs on this. @Jeff, I will look into the local projections, this is the first I have heard of it. I also appreciate the clarification on the difference between including lagged y and lagged x in the specifications, I will play around with both.

Sam