The impact of survivorship bias

This post is about the impact of survivorship bias on various types of strategies. I want to look at mean-reversion as well as trend-follow strategies to see how their performance gets impacted by using today’s stock index constitution vs. a historical correct definition aka survivorship bias free.

Test setup

  • Indices: NDX100, SP100, SP500
  • All trades are close/close, no trading cost
  • 25 concurrent positions / 4% size
  • Data adjusted for splits and cash dividend

.

Mean Reversion

For the first test I run a simple mean-reversion strategy. Enter the trade when RSI3 < 25 and exit when RSI3 > 75. Of course one can argue that these aren’t the perfect levels. However it should be good enough to understand the impact. In case multiple entries occur the ones with the lowest RSI3 values are picked.

The strategy performed significantly worse on a survivorship bias free definition. This can be seen among all three major indices.

.

Trend Following (long only)

In a second step I looked into trend following. Buying 25 stocks with the highest RSI14 value using a rotational trading approach.

Again, the performance difference is very significant on all three indices.

.

Trend Following (short only)

At the end I want to look into SHORT trades using a trend following approach. Shorting the 25 stocks with the lowest RSI14 (rotational).

In this case the take away is different. The survivorship bias free version is performing somewhat better on all three indices. I think this is due to the fact that the system has more candidates to choose from, especially in past years when the survivorship bias is more pronounced.
.

Summary

In all cases the results were significantly different and therefore leading to faulty decisions during the strategy development process. One can not get an accurate picture of his or her strategy without considering survivorship-bias. I’m stressing this point because I continue to see people using today’s index definition in order to backtest strategies. Should you have an interest on testing your index-stock strategies in a reliable way then read this for further information.

# Frank

About these ads

Comments

  1. Hi

    Frank I used your code of rotational system here in Poland and it works.
    After testing I have the same conclusion that “current” index test outperform bias free.

    And some question , about your rotational code.
    Once I saw strange thing program change ranking after buy after 2 weeks like “look into the future” in terms of ranking.
    It was only once but now I will be more watching because it can be done and seen only in current time not in backtesting.
    “Check code” in AB returns that code looking into the future , but looping in AFL code sometimes gives false positive results and I don’t know….

    Best Regards

    pit

  2. Thanks

    for your replay.
    But it was almost impossible not noticed for me that after 4 weeks I see that one of the traded symbol I bought 2 weeks ago not 4. Strange isn’t it ?
    I will must look at this more.

    Pit

  3. The current index membership is roughly based on current market cap. Stocks that have been removed have declined in value relative to stocks that have been added or stayed in. It is always a good idea to short stocks for a few months before they are removed from the indices, because the stocks that are removed are almost by definition the worst performing stocks in the index. This can’t be done in real-time, but it explains why a short-only strategy will do better with the survivorship bias-free stocks. In particular, the stocks that started to fall and recovered may still be in the index, but the stocks that just kept falling have been removed.

    It might be interesting to look at the relative performance of stocks in the Russell 2000 that have never been in the Russell 1000 vs the stocks that were downgraded from the Russell 1000 to the Russell 2000. I would expect the stocks that have never been large-caps to outperform the stocks that used to be large-caps but aren’t any more.

  4. Nice article, however we found that Survivorship Bias can either overstate or understate your result when using the same model across different stock series. See Survivorship Bias: neither Myth nor Fact..

    Our backtesting was done over the period 1999 to 2013, over what time period do your backtests span?

Trackbacks

  1. [...] how bad is the effect of SB on system results?  Quite frankly, pretty darn bad.  This study, which compared the returns of mean reversion and momentum systems when run on biased and unbiased [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 304 other followers

%d bloggers like this: