My past posts focused a lot on the mechanics of trading system development. Though we have to keep in mind: our backtests can only be as good as the data we run it on. With this post I want to take a different route and share some insight about my data providers and some of the recent improvements to the data service I use.
To understand what I use I want to give you some background about my trading systems. All trading systems are based on daily bars. My systems are using the close price to generate signals, but some of them are trading at the OPEN the next day and others are trading at the CLOSE.
The ones trading at the CLOSE do use the close price from the daily bar 15-30sec before the close and thereby anticipating the ENTRY/EXIT signal at the close. Of course this could result in an inconsistency between your current market position and your back-tested results. I’ve been doing that for a few years and it turned out not to be an issue.
To execute this setup I do need two types of data providers: one providing me with historical data and the other for real-time data. My real-time data provider is iqFeed. They are reliable and have a fast AmiBroker integration. So why do I need a second data provider: iqFeed and most of the other real-time data providers are very weak when it comes to historical data. I don’t rely on them for my backtests. One of the reasons is that I want to have an excessive long history of currently listed and de-listed stocks. And even more important I want to have correctly adjusted data.
My data provider for historical data is NORGATE / premiumdata. I’ve been using them for quite some time now. They have proven to be extremely reliable. During the time I used them I didn’t have a single day of service outage. Their service is fairly priced and VERY well integrated with AmiBroker. Though, they also have some room for improvement. With this post I want to share some of these recent improvements. Let me make one thing very clear: I’m not associated with NORGATE and I do not receive any kick-back for writing this posts.
.
Data update technology has undergone significant change
- Different type of data subscriptions can be maintained in a single AmiBroker workspace, e.g. US stocks, FOREX and FUTURES. This way you don’t have to switch between databases or even use different data update tools.
- The entire database update takes less than 60 sec. and can be further automated
- The data update UI has undergone significant changes and is now more user friendly and intuitive to configure.
(click on the image to enlarge it)
.
Even tighter AmiBroker integration
- Data padding capabilities introduced: Allows you to repeat the close of the previous date when a time series is missing data at any given bar. There are various settings how that can be handled.
- In the past all data has been stored in MetaStock format, which turned out to be a drag on performance when handling a larger number of stocks. So this has been definitely addressed.
- Any chart annotations, trendlines etc. are retained when a security changes its symbol or exchange
- Support for 4 level GICS
(click on the image to enlarge it)
.
CASH dividend adjusted data
- While all data has been adjusted for major capital events, cash dividend adjustment has always been missing. You now have the option to include or exclude cash dividends from your backtest. In case you want to include it HIGH, LOW, OPEN AND CLOSE are adjusted accordingly. My next post will be focusing on the impact of cash dividend adjustments.
- Of course the cash dividend is also adjusted for stock splits and other major capital events.
(click on the image to enlarge it)
.
I did this post on a yet to be released version of their offering. In case you are an existing customer of NORGATE premiumdata and want to participate in their early beta, go contact NORGATE support.
Let me wrap-up this post. I really like where NORGATE is taking it’s data service offering. It’s become faster, even better integrated, got cash adjusted data and that for the same fair price. This definitely looks like as they are listening to their customers. So next on my wishlist: historical fundamentals 
- Frank
re: fundamental history, what about this:
http://www.trading-tools.com/historical-fundamental-data-downloader.htm
Looks pretty interesting, I’ve been using EODData.com for both daily and one minute bars, but they don’t adjust the time series. Is there a way to tell when adjustments have been applied, or do you need to query the old values ? I’d be uploading the data into a mysql DB so it would be nice to know when to reload the entire series.
Hi Stewie,
it all happens automatically, so you want notice it as a user.
Frank
Great post Frank!
Do you know if Norgate has any plans of providing real-time data? This would eliminate the need for 2 data providers.
Also, do you know how Norgates historical delisted database stacks up to CSI’s? I’m curious how one would know if a database is accurate or more accurate than another one? What is the reference point in order to make that validation?
Again, thanks for the great post!
Hi Ronen,
Thanks for your feedback (and sorry for the late reply).
I’m not aware of Norgate moving into that direction. They focus on more markets… I’ve spend a little time with CSI’s database recently. CSI’s focus is on FUTURES, that’s were they are coming from. I didn’t develop an objective method to compare the data….
Frank
Hi Frank, what about the quality of the Norgate data itself? Have you ever tried to measure it in any way?
Hi Philipp,
no I didn’t try to measure it.
Frank
Frank
I also want to trade just before market close, essentially using daily bars. If the market close system I am trading requires a long history of data (say 60 days), how do you make that work? Are you storing a long history of data in your iqfeed database? Or do you somehow merge the two?
Thanks
Bruce
Hi Bruce,
all my systems require a long history to run and get the indicators calculated. Running on pure iqFeed that’s not a problem, a backfill takes a very few seconds (even 3000 bars). The main issue is: iqFeed is anadjusted data (cash dividend)…. Currently I’m trying something different: using my Norgate database adjusted for cash dividends and getting the last bar from IB.
Regards,
Frank
I’ve been reading your blog lately and I’ve noticed that you now have a survivorship bias free database of the Nasdaq 100 and the S&P 500. What do you do with a ticker once it leaves the Nasdaq 100? Do you remove it from the Amibroker Nasdaq 100 watchlist? If so, how do you perform a backtest including the older removed tickers? I hope you answer my question. Thank you!
Hello Dan,
I got a watchlist containing all index members (also the delisted ones and the ones not in the index anymore). Then I got a AFL function that returns true/false for a given stock at a given date.
so my buy statement looks like this: BUY = YourCriteria and IsMember( Name(), Date() );
Frank
Hey Frank,
Thanks for your prompt reply. Perhaps I’m being thick here, but is there a list of some sort which you use that contains when a ticker was removed from the, say, Nasdaq 100 when you run your backtest?
So from what I understand based on what you mentioned in your reply is that you have a 1 watchlist (nasdaq100) that contains all the members incuding delisted and ones that were removed from the index. You then run your backtest against that watchlist. Which then goes back to my other question… Where do you store a delisted/removed date for each ticker symbol in your nasdaq 100 watchlist where your AFL can check against?
Much thanks Frank and your blog is great! Keep up the good work so beginners like myself have someone to look up to!
Hello Dan,
all data regarding start / end of index membership is in the AFL file. It’s called with Symbolname and date, it returns TRUE/FALSE.
Frank
Thank you Frank! I get the picture now.
Hi Frank,
I’m just following on your attempt to use Norgate EOD with IB latest bar (RT). How did that go and if you are satisfied would you kindly share some of your thoughts re that setup. I was thinking about the same issue and believe it’s important to try to replicate as much as possible the back-testing environment when going for real trading.
BTW, I find your research very practical and focused, so BIG thank you for sharing that knowledge.
Hello Kariim_j,
Thanks for your feedback!
That works very well. I’ve been doing that for about a year or so. No issues.
Regards,
Frank
Great that it’s working fine for you, Frank. But, sorry to be so dense, as I’m interested in how you set up the database. Norgate is a Metastock format and uses a different plugin than IB obviously, so I’m curious, how you did bring them together?
Any insights would be greatly appreciated, or if you know of any help available out there if it’s too much asking!
Cheers.
Hi,
i just change the data-source from Norgate to IB. IB backfill to one bar only.
So you use the history from the Norgate and the most recent bar from IB. A little trick!
Regards,
Frank
That’s cool, It didn’t occur to me.
Much appreciated, Frank.
Hi Frank,
I’m very much interested in understanding how you have achieved what I assume is one database that started with Norgate historical EOD data and then changed to use IB plugin
database to download most recent bar from IB.
I have tried to change the DB setting from the (Norgate) Metastock plugin to IB, and set to EOD, only fill 1 day – however, I seem to lose all my data once I press OK (i.e. the chart window says there are 0 bars to display). Amibroker support have also advised I should actually be creating a new database and not change the plugin setting.
Any help would be appreciated.
Regards,
Bobby
Hi Bobby,
I’m using this setup for 12+ month now. Works very well.
Make sure you have “Local data storage” enabled.
Regards,
Frank
Hi Frank,
What version of Amibroker are you using? and which market (Us or ASX) have you got this setup working with?
I’ll write up my exact steps for the implementation for your eyes to glance over if possible as the historical data seems to wipe out once the plug in is changed to IB and a connection to TWS is established for backfill.
Thanks,
Bobby