As time has passed, I've been progressively including more and more functionality into Spartan. Initially, I was only tracking forex but have started tracking stocks and client sentiment data as well. I've found it invaluable to have my own source of data. In the long term, I'll continue to do this no matter what. Why? It's fun. Things I want to add: - Tracking option chains: This is probably doable under the current architecture, just treat price of each contract as a standalone instrument, similarly with greeks (or maybe those would be backed out algorithmically using BS or whatever else) - Earnings data: estimated, initial, updated
Just keeping this here as a note for myself, eventually will start storing it and see relations to daily returns: Code: curl -q -o - https://api.stocktwits.com/api/2/streams/trending.json 2> /dev/null | jq '.messages | .[] | .symbols | .[] | .symbol' | sort | uniq -c | sort -nr
First week where IB didn't randomly go down and stay down for hours. Though it did of course randomly stop publishing quotes as you can see in the "number of quotes per 10 minute period. Sometimes I wonder what they do over there.
Funny to see that each hour has one 10-minute bar which is substantially lower than the rest of that one hour period.
I want to start storing option data because there is a definite handicap for me without the ability to play with historical data. Obviously, I'm not going to store tick data. I don't think it makes sense to store every strike, since I won't be looking at holding options for longer than a few weeks. The interface I'd like to have for beginning to track this data would be to create a new row in the instruments table, something like: Code: insert into instrument(symbol,type) values('SPY','OPTION') Then Spartan would automagically begin tracking contracts and the underlying for up to 6 months in future with a 10% interval on either side. 10% because in my data, SPY moves a median of 5% for any given 6 month period in the last decade so this should be OK for my immediate purposes. In terms of expiration, it looks like there are ~6+5 contracts for each strike (50 strikes for SPY) over a 6 month period if you include weeklies. I would also track some greeks as they come in from the data feed. So this probably works out to hourly bars for 11*50 new contracts just to track SPY. Currently, Spartan tracks only 31 instruments and has a database of about 300MB. So this means that after 6 months, I can expect my DB to be between 5-8G in size at the current rate. Should be fine. But I can't back it up locally every hour, in that case. In comparison, AMD/INTC have ~10 strikes to track so over a 6 month period, they would add a relatively smaller amount. In terms of the table layout, I'd probably have a child table, something simple like this specifying the contracts: * instrument_id * parent_instrument_id (i.e., the underlying) * expiry_date * strike * is_call Then I can re-use the rest of the mechanisms for pulling and storing the data. I'd likely have a separate table for option-specific data attributes like the greeks even though I could probably re-use the mechanism used for storing prices.
Compounding One of the things that is very important to me is to compound on work that I've done before. As I mentioned elsewhere on the site I have begun to use a desktop wiki (moinmoin) to track my research. I've been experimenting with a trading journal on the wiki and one of the steps in that I've got in prep work for the day is a check list, where one of the items is to check the economic calendar. Previously, I had a link to investing.com's calendar but I wanted it to be more efficient than a link, I wanted to embed it to avoid having to visit a new site. Unfortunately, investing.com does not allow embedding in an iframe but I realized I've actually been tracking economic calendar data for a while now. A little bit of code later, and it's available inline in my journal template. This updates dynamically every time I visit the page and the entire purpose is to keep me efficient. The ugly, hacky code is below. Code: import sqlite3 import pandas as pd import os.path dbpath = os.path.expanduser("~/src/noobymcnoob/spartan/run/db.sqlite") def highlight_todays_date(df): if df['diff'] < 0: return ['background-color:lightgrey; opacity: 0.25']*len(df) if df['diff'] >= 1: return ['opacity: 0.25']*len(df) else: return ['']*len(df) def execute(macro,args): currencies = '(select distinct currency from economic_event)' if args is not None and args != "": currencies = args.split(',') currencies = ["'" + c + "'" for c in currencies] currencies = "("+",".join(currencies)+")" conn = sqlite3.connect(dbpath) try: df = pd.read_sql("select julianday(datetime)-julianday('now','localtime') as diff,date(datetime) as date,time(datetime) as time,currency,type,impact,description,forecast,actual from economic_event where currency in %(currencies)s and datetime(datetime) BETWEEN datetime('now','-24 hours') AND datetime('now','24 hours') order by datetime asc" % dict(currencies=currencies), conn) html = ",".join(df.currency.unique()) + df.style.hide_index().hide_columns(['diff']).apply(highlight_todays_date,axis=1).render() return html finally: conn.close() if __name__ == "__main__": print(execute(None,None))
This was much easier than I expected, less than 100 lines of code. Every morning, Spartan will pull down all strikes expiring within the next 10 days that are within 1 ATR on either side of the last closing price. Next, I am going to be keeping 2 weeks of tick data around for all instruments. That will be fun I'm sure! Currently tracked options (edit looks like it was still updating):