insignificant margins of significant interaction term?

[Sita Gupta]

Hello,

I test a regression model with two continous variables (brand share and shop share) . Both variables are a ratio so they are between 0 and 1.

The interaction term is significant. The single coefficients of brand and shop are however not significant. As far as I know this is not a problem.

Now I computed the margins as follows:

margins, dydx(brand) at (shop =0.0(0.1)1.0)) vsquish


This results in:

	Delta-method
	dy/dx	Std. Err.	z	P>z	[95% Conf.	Interval]
brand
_at
1	20.84446	13.55386	1.54	0.124	-5.720606	47.40953
2	18.65248	12.61064	1.48	0.139	-6.063927	43.36889
3	16.46049	11.74239	1.40	0.161	-6.554165	39.47515
4	14.26851	10.96691	1.30	0.193	-7.22624	35.76326
5	12.07652	10.30517	1.17	0.241	-8.121247	32.27429
6	9.884538	9.780296	1.01	0.312	-9.28449	29.05357
7	7.692553	9.415191	0.82	0.414	-10.76088	26.14599
8	5.500568	9.228843	0.60	0.551	-12.58763	23.58877
9	3.308583	9.232081	0.36	0.720	-14.78596	21.40313
10	1.116598	9.42471	0.12	0.906	-17.35549	19.58869
11	-1.075388	9.795564	-0.11	0.913	-20.27434	18.12357

So the p- value is insignificant at every point.

Does that mean that I dont have any significant effect of my interaction term?

Thank you very much!

[Clyde Schechter]

You don't provide any information about the regression that was done. Would I be correct in assuming that you had a c.brand#c.shop interaction term in the model?

If so, there is nothing surprising here. The interaction term tells you whether or not the marginal effect of brand depends on the value of shop. If that coefficient was statistically significant, and if you believe in statistical significance tests for this kind of thing, then you would conclude that the marginal effect of brand does depend on the value of shop. You can even see that very clearly in your -margins- output. Depending on the value of shop, the marginal effect of brand might be as big as 20.8 or as small as -1.1.

The p-values in the -margins- output answer a completely different question. The marginal effects shown in the table are the marginal effects themselves, and the p-values test the hypotheses that each of these effects is different from zero. They tell you nothing about whether they differ from each other. Again, on the assumption that you take p-values seriously in this context, you would have to conclude that your data do not support the conclusion that the effect of brand is different from zero at any of the levels of shop you examined. That does not mean that you can conclude that any or all of them are equal to zero. It just means that your data are not sufficiently copious or precise to determine whether or not they are positive or negative or zero. So you have a bunch of marginal effects that are very imprecisely estimated by your data, but the data happen to enable you to conclude that they are not all equal to each other, even though it cannot precisely tell you whether they are positive, negative, or zero.

[Sita Gupta]

Thank you a lot Clyde! you are right, I used the interaction term c.brand#c.shop.

I still have one question to see if I understood it correctly. Lets assume the p value ist significant for the marginal effects if brand is between 1 and 4, but not if brand is between 5 and 10. Would I conclude that the joint effect of the two variables is only true until 4? And if brand gets larger, we have to reject that there is a significant effect?

Thanks again!

[Clyde Schechter]

No. In fact, you should never interpret statistical significance of anything like that. I can't say it loud or often enough: a non-statistically significant finding does NOT mean there is no joint effect or the joint effect is not true. It just means that the data do not identify the effect with sufficient precision to know its direction. If you are in the habit of saying that X is not related to Y when you have a non statistically significant coefficient of X in a regression of Y on X, a very widespread habit, then work fastidiously on breaking that habit because it's just dead wrong. It just means that your data cannot tell you to what extent or in which direction X and Y are associated. This can arise from the data being too scanty (insufficient sample size), or too noisy relative to the actual effect. It is possible that the true effect is zero, but real zero effects are quite uncommon in the world, or at least in the world of things that people study. So of all the reasons an effect could be found to not be statistically significant, the actual absence of an effect is the least likely, and it should be your explanation of last resort only.

In the case of continuous by continuous interactions, there is another consideration that reinforces it. If you regress Y on c.X##c.Z, algebraically you are saying that the effect of X on Y is a function of Z. In particular, it is a linear function of Z. That is effect of X on Y = c0 + c1*Z. Now, unless c1 turns out to be zero, this means that there is always some value of Z, namely -c0/c1, for which the effect of X on Y is actually zero! So the use of an interaction model forces you to accept that for some value of Z, the effect of X is zero. And for values of Z near that, the effect of X will be close to zero, and will therefore also be statistically insignificant. This is an inherent feature of interaction models with continuous variables. Now, sometimes it will happen that this range of Z where the effect of X must be close to zero lies outside the range of Z's that are actually observed in the data, or even in the real world. But often this is not the case, which means that in many such models there will be actual instantiations of values of Z for which the effect of X is near zero. Conversely, when Z if far from that value -c0/c1, the effect of X on Y will be large (large negative on one side, and large positive on the other). And these large values may be statistically significant.

Consequently the finding that the effect of X on Y is sometimes statistically significant and sometimes not statistically significant tells you nothing at all: it is baked into the use of a continuous by continuous interaction model. It is only interesting to the extent that it pushes you to explore how those "effect of X is significant" and "effect of X is not significant" regions of Z overlap with actual real-world values of Z.

So, when dealing with this kind of model, when looking at the effects of X on Y at particular values of Z, you should pay essentially no attention to statistical significance, and pay no attention at all to changes from statistically significant to not within the range. The effect of X on Y varies continuously with Z, and really it is the coefficient c1 (which is also the coefficient of the interaction term in your regression output) which is important: it tells you the extent to which the effect of X on Y depends on Z.

[Hovhannes Nahapetyan]

Hi Clyde, I have two questions regarding above discussion. 1) You mention "Depending on the value of shop, the marginal effect of brand might be as big as 20.8 or as small as -1.1" - however the confidence interval or when point estimate is 20.8 contains -1.1 and therefore the slope for brand variable is not statistically different at both points. 2) at all levels of shop confidence interval for brand contains 0 ( I agree with you that sometimes we should not pay attention to statistical significance but for the sake of argument, let's do this time) which means the coefficients for brand at all levels of shop are not statistically different from zero. If this is the case, they can't be different from each other as zero is a zero. This in case makes it very strange that interaction term is significant which would mean that a zero is different from another zero in a statistically different way (or any way for that matter). Could you please elaborate on this. Thanks

[Clyde Schechter]

at all levels of shop confidence interval for brand contains 0 ( I agree with you that sometimes we should not pay attention to statistical significance but for the sake of argument, let's do this time) which means the coefficients for brand at all levels of shop are not statistically different from zero. If this is the case, they can't be different from each other as zero is a zero.

This is completely and absolutely false and the fact that so many people believe this to be true is one of the strongest reasons for abandoning the concept of statistical significance. "Equality" in the sense of "not statistically significantly different from" is not a transitive relationship.

Even if we take statistical significance seriously, we have to use it in ways that are consistent with its definition. It was never true that a statistically non-significant result implies that the effect is zero. It implies only that we don't know whether it is zero or not. You can have two things that each may or may not be different from zero, and yet still be quite confident that they are different. The output shown in this thread is a good example of exactly that phenomenon.

[Hovhannes Nahapetyan]

Hi Clyde, thank you for your reply. I want to ask a few more questions if i may: 1) you said "If you are in the habit of saying that X is not related to Y when you have a non statistically significant coefficient of X in a regression of Y on X, a very widespread habit, then work fastidiously on breaking that habit because it's just dead wrong. It just means that your data cannot tell you to what extent or in which direction X and Y are associated. This can arise from the data being too scanty (insufficient sample size), or too noisy relative to the actual effect. It is possible that the true effect is zero, but real zero effects are quite uncommon in the world, or at least in the world of things that people study". Is there a concern that this would open up the door for people to claim there is a relationship even in the case of spurious correlation? After all such relationships are also very common and one could always bring up with excuse of scanty data to argue for relationship that does not exist. 2) "algebraically you are saying that the effect of X on Y is a function of Z. In particular, it is a linear function of Z. That is effect of X on Y = c0 + c1*Z. Now, unless c1 turns out to be zero, this means that there is always some value of Z, namely -c0/c1, for which the effect of X on Y is actually zero! " in my analysis both c0 and c1 are positive and so there is no such point; 3) " You can have two things that each may or may not be different from zero, and yet still be quite confident that they are different" but if they may or may not be zero then we could assume that there is some probability for both of them to be zero (and indeed in this above example both confidence intervals contain zero) in which case they can't be different from each other since both are zero. 4) this is probably of utmost concern to me- in my analysis all conditional slopes are greater than zero i.e. slope of x1 at all levels of x2 is greater than zero. So say when x2= 4 slope for x1 =10 and when x2 = 5 the slope of x1=12 ( both are obviously positive and their relevant confidence intervals do not contain zero). However, my purpose for this interaction model is primarily to see if this difference of 2 between the x1 slopes is real (statistically significant) or just a chance which in case would be the answer to the question 'does the effect of x1 depend on x2). How do I find out answer to that question in this particular case? Thank you!

[Hovhannes Nahapetyan]

Oh I left out an additional information in " in my analysis both c0 and c1 are positive and so there is no such point" - x2 cannot be less then zero

[Clyde Schechter]

Is there a concern that this would open up the door for people to claim there is a relationship even in the case of spurious correlation? After all such relationships are also very common and one could always bring up with excuse of scanty data to argue for relationship that does not exist.

A proper presentation of results is to show the effect estimate along with a measure of its uncertainty (i.e. a standard error or a confidence interval). If that is done and somebody tries to claim they have found an important relationship when the estimate itself is small or the confidence interval is wide enough to include relationships that would be trivial (including, but not limited to zero), then they are transparently misrepresenting their findings and that will be obvious to anybody who is paying attention. One of the problems with relying on statistical significance to decide whether a relationship "exists" is that the p-value depends not only on the actual effect size, but also on the noise in the data and the sample size. And there is no way to tease out which of those three is driving the p-value from just looking at the p-value alone. The important thing is to focus on the effect estimate and its uncertainty and to evaluate those in terms of what is a real-world meaningful effect. If you have a huge sample size, even tiny effects of no real-world importance will often be "statistically significant." Using statistical significance is an ill-fated attempt to turn uncertainty into certainty. There is by now a large literature on all that is wrong with statistical significance as a concept.

See https://www.tandfonline.com/doi/full...5.2019.1583913 for the "executive summary" and
https://www.tandfonline.com/toc/utas20/73/sup1 for another 43 supporting articles. Or https://www.nature.com/articles/d41586-019-00857-9 for the tl;dr.

in my analysis both c0 and c1 are positive and so there is no such point;

Yes there is such a point, algebraically. It just doesn't happen to be a possible value of x in your data. The point I was making is that with continuous interactions, the difference in slope depends on the value of x, so it is not possible to make a blanket statement about the differences, and, in particular, that algebraically, there is always a value of x for which the difference is zero, and, for that matter, any other value you specify.

but if they may or may not be zero then we could assume that there is some probability for both of them to be zero (and indeed in this above example both confidence intervals contain zero) in which case they can't be different from each other since both are zero

You contradict yourself. Yes the may both be zero, but from that you cannot derive that both are zero, and hence you cannot derive that they can't be different from each other. Also, in nearly all real world situations, the probability that either coefficient is exactly zero is usually, wait for it, exactly zero. That is, the effects we are estimating are continuous, and it is exceedingly rare for such an effect to have a spike at zero (or any other value). So an exactly zero result has zero probability (which is not the same as saying it can't happen.)

this is probably of utmost concern to me- in my analysis all conditional slopes are greater than zero i.e. slope of x1 at all levels of x2 is greater than zero. So say when x2= 4 slope for x1 =10 and when x2 = 5 the slope of x1=12 ( both are obviously positive and their relevant confidence intervals do not contain zero). However, my purpose for this interaction model is primarily to see if this difference of 2 between the x1 slopes is real (statistically significant) or just a chance which in case would be the answer to the question 'does the effect of x1 depend on x2). How do I find out answer to that question in this particular case?

Well, since I have clealry said that I think that the whole idea of trying to decide whether such a difference is "real" or "statistically significant" is misguided, I am not going to point you in that direction.

Suppose, however, your model is something like -regress outcome i.x1##c.x2- where x1 is a dichotomous predictor and x2 is a continuous one that modifies the effect of var1. To estimate the difference in the effect of x1 when x2 = 4 or 5, you can just run -lincom (5-4)*_b[1.x1# c.x2]- and Stata will provide you with the estimated effect difference along with its standard error and confidence interval.

[Hovhannes Nahapetyan]

Thank you for the reply and the links Clyde.