The impact of survivorship free backtesting

This post is about the learning how survivorship is impacting system design with a focus on rotational trading. As posted here, the last couple of weeks I’ve worked on creating a NASDAQ 100 database that doesn’t have a survivorship bias. It’s been quite a tricky and tedious task. Tricky, because I first had to figure out how to do the technical implementation. Tricky, because it takes a lot of time to transform the raw data into Amibroker AFL code.

The problem

Let me first describe the actual problem! Often times you test an idea based on stocks being part of a particular index, e.g. Nasdaq 100. Why does one focus his or her testing on an index? Because the members of this particular index are BIG. And size often correlates with the number of stocks being traded, so liquidity shouldn’t be an issue. Also because, often institutional investors focus on these stocks as well. Furthermore being part of a particular index required companies to comply with higher accounting standards etc.

It’s well known that a system should be tested over an extended period of time (bars) in order to ensure the system has been through different kind of markets (bull, bear, side, low/high volatility, etc.). So you are inclined to include as many as possible historical bars. Typically I use about 3000 daily bars for my test (~ 12 years). So let’s check how many of today’s NDX-100 members have been part of the NDX-100 in the past.

So effectively I tested my strategies against a VERY different environment than the one that existed in reality, e.g. only 60 of today’s members were part of the index in 2005. To say it differently, 40 stocks got dismissed because of poor stock performance, bankruptcy or acquisition.

The impact

A few weeks ago I presented a very simple rotational system in order to introduce the idea and to give the interested reader a starting point.   The rules of this system:  only in the market when the QQQQ’s are  trading above MA30 or MA200, Ranking: TSI

Let’s review the impact of survivorship bias:

- A performance drop of about 50% in absolute as well as risk adjusted performance

- Significantly higher max. draw down

The results consider only trades starting 01.01.2001. The same test going back to 1995 resulted in about the same performance drop.

.

.

Background

Data quality: I got historical index constitution all the way back 1995. The completeness until 1997 is very good, before that i miss a very few stocks.

Technical implementation: all historical data is maintained in a Microsoft Excel sheet. I then turned this data into one gigantic Amibroker AFL function. So this function can be called with ticker name + date and will then return TRUE/FALSE depending if this ticker has been part of the index at the given date.  Once I did this, it has become extremely simple and easy to test existing systems. Just needed to add a call function statement and run the system against the newly created watchlist (containing all current + historical tickers). Within minutes i got the results (after i spend many many days building the database).

Test

Availability

I’m strongly convinced that testing these kinds of systems without a survivorship free database has severe impact on system design and performance. The decisions you take during the system designed can be flawed as your data suggest a much more positive environment than the one existed in reality. So I encourage you to either build your own survivorship free database or get one from me! As said, i build that database for my personal use.  Though I’m not in the business of selling software or data. So donate 500$ to UNICEF (link), send me the UNICEF donation confirmation and i will send you an email containing: Excel file, Amibroker AFL file as well as Watchlist file (historical stock price data isn’t included) .   This way you can do something good to your trading business and support UNICEF in their effort to build schools for children in Africa.

.

Conclusion

I’m not going to build any kind of system based on an index I don’t have survivorship free data for.

About these ads

Comments

  1. Very interesting and helpful analysis; I will now be wary of systems that do not account for survivorship bias. I did a similar test on a simple NASDAQ 100 system I threw together, and accounting for survivorship bias had a huge negative effect on performance too. I could only compile end of year lists for the NASDAQ 100 because my data source wasn’t that good: I used archive.org’s Wayback Machine in conjunction with the NASDAQ 100 members list (http://dynamic.nasdaq.com/dynamic/nasdaq100_activity.stm) and historical changes (http://www.nasdaq.com/indexshares/historical_data.stm). Where did you get your delisted data? Also it looks like your performance isn’t too terrible after accounting for survivorship bias, so I wonder what effect a QQQQ short hedge would have (like in David Varadi’s post here: http://cssanalytics.wordpress.com/2009/11/25/performance-of-the-livermore-active-issues-index-created-from-the-nasdaq-100/)…

    • Riccardo Ronco says:

      alternatively it makes sense to stop building models on stocks and focus on ETFs on multi-asset classes when it comes to the private investors. Total return indices are available for most of them now so a test there would cut at the source this issue.

    • Hi guys,

      i am testing some models on sp500 on yahoo data with average holding period of 2/3 days ….do you think the performance would be so different with a surv. free database?

      thanks
      Giacomo

      • Hello Giacomo,

        it depends what you mean with “… so different”. I would expect a 10-20% better performance using data with a survivorship bias. That’s not the main issue, for “ME” the main issue is that this set of data will lead you into a somewhat different direction than a survivorship free database.

        If you don’t have the historical correct data, I only see these options:
        - Use ETFs (don’t trade stocks)
        - Do not focus on specific indices, rather include all US stocks (with a volume filter). Of course you should get delisted data as well. Premiumdata / Norgate offers that for a fair price

        Frank

  2. This is a very helpful post and I hope you have more thoughts to share on the subject in future posts. I am curious if you think this would be as drastic with an index such as the S&P 500, 2000 to 2003 was an asynchronous period for the Nasdaq as a whole. Also I am wondering what your average holding period per trade was on this test; I have a guess that rotational systems which average a short holding period , say 2 to 3 days will preform better than those holding 2 to 3 weeks – in terms of performing worse with survivorship free data.

    An interesting study might be to see if you can single out any red-flag behavior in these stocks before they get delisted, do they exhibit any similar traits in terms of relative strength, volume action, etc etc. Would be nice if you could detect somewhat trouble stocks to avoid.

    -D

    • Hi,

      i would expect a similar performance drop as the SP500 also had a high turnover. Might write an article for the Traders journal about rotational trading later this month. Will consider your questions / comments and answer later.

      Frank

  3. Usually I do not read post on blogs, but I would like to say
    that this write-up very forced me to try and do so!
    Your writing style has been surprised me. Thanks, very
    nice post.

  4. Jane Fox says:

    Please beware about using only ETFs, they have survivorship bias as well.

Trackbacks

  1. [...] impact of survivorship bias (link): I re-run the same system based on a database with survivorship bias. Basically i used the [...]

  2. [...] on NDX100 stocks (survivorship free: historical index adjustment + delisted [...]

  3. [...] on NDX100 stocks (survivorship free: historical index adjustment + delisted [...]

  4. [...] database starting 1990.  I did a similar exercise for Nasdaq 100 stocks already see  (here and here). With this post I want to share some of my S&P500 specific findings that might be relevant to [...]

  5. [...] of survivorship free backtesting (link) I do plan to continue this blog in the year to come. However, I would like to give you [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 305 other followers

%d bloggers like this: