This post is about the impact of survivorship bias on various types of strategies. I want to look at mean-reversion as well as trend-follow strategies to see how their performance gets impacted by using today’s stock index constitution vs. a historical correct definition aka survivorship bias free.
Test setup
- Indices: NDX100, SP100, SP500
- All trades are close/close, no trading cost
- 25 concurrent positions / 4% size
- Data adjusted for splits and cash dividend
.
Mean Reversion
For the first test I run a simple mean-reversion strategy. Enter the trade when RSI3 < 25 and exit when RSI3 > 75. Of course one can argue that these aren’t the perfect levels. However it should be good enough to understand the impact. In case multiple entries occur the ones with the lowest RSI3 values are picked.
The strategy performed significantly worse on a survivorship bias free definition. This can be seen among all three major indices.
.
Trend Following (long only)
In a second step I looked into trend following. Buying 25 stocks with the highest RSI14 value using a rotational trading approach.
Again, the performance difference is very significant on all three indices.
.
Trend Following (short only)
At the end I want to look into SHORT trades using a trend following approach. Shorting the 25 stocks with the lowest RSI14 (rotational).
In this case the take away is different. The survivorship bias free version is performing somewhat better on all three indices. I think this is due to the fact that the system has more candidates to choose from, especially in past years when the survivorship bias is more pronounced.
.
Summary
In all cases the results were significantly different and therefore leading to faulty decisions during the strategy development process. One can not get an accurate picture of his or her strategy without considering survivorship-bias. I’m stressing this point because I continue to see people using today’s index definition in order to backtest strategies. Should you have an interest on testing your index-stock strategies in a reliable way then read this for further information.
# Frank
Hi
Frank I used your code of rotational system here in Poland and it works.
After testing I have the same conclusion that “current” index test outperform bias free.
And some question , about your rotational code.
Once I saw strange thing program change ranking after buy after 2 weeks like “look into the future” in terms of ranking.
It was only once but now I will be more watching because it can be done and seen only in current time not in backtesting.
“Check code” in AB returns that code looking into the future , but looping in AFL code sometimes gives false positive results and I don’t know….
Best Regards
pit
Hi Pit,
I trade the system in one way or the other for about 18month now. I don’t experience any issues with Amibroker.
Regards,
Frank
Thanks
for your replay.
But it was almost impossible not noticed for me that after 4 weeks I see that one of the traded symbol I bought 2 weeks ago not 4. Strange isn’t it ?
I will must look at this more.
Pit
The current index membership is roughly based on current market cap. Stocks that have been removed have declined in value relative to stocks that have been added or stayed in. It is always a good idea to short stocks for a few months before they are removed from the indices, because the stocks that are removed are almost by definition the worst performing stocks in the index. This can’t be done in real-time, but it explains why a short-only strategy will do better with the survivorship bias-free stocks. In particular, the stocks that started to fall and recovered may still be in the index, but the stocks that just kept falling have been removed.
It might be interesting to look at the relative performance of stocks in the Russell 2000 that have never been in the Russell 1000 vs the stocks that were downgraded from the Russell 1000 to the Russell 2000. I would expect the stocks that have never been large-caps to outperform the stocks that used to be large-caps but aren’t any more.