Hi there! Anyone here knowing for a data vendor for retail traders as well as software or programing solution (python preferably + opensource SQL) for retrieve at a daily basis tick data (Level1 time&Sales) and then post-process it to store in a custom time compressed database (not traditional OHLCV)...? I know of tickwrite (tickdata.com) solution, but quite expensive for a mortal trader! I could pay for a Reuters Eikon "cheaper version (Xenith)" to retrieve data via it's API for example. What I want to do is, from each given stock at a specific exchange listing; from it's timeStamp, last price and size from data vendor, create own time dependent arrays (minute, hour, day, etc) to store them locally with fields like this ones: TIME - Last TimeStamp of session. LAST - The last closing price of session. HIGH - Maximum price of session. LOW - Minimum price of session. UT - Cumulative count of upticks from session. UTV - Cumulative count of size for UpTicks from session. UMF - Cumulative MoneyFlow (Price*size) of UpTicks from session. DT - Same as UT but for DownTicks. DTV - Same as UTV but for DownTicks. DMF - Same as UMF but for DownTicks. Additional to this, store some Fundamental key data like: Shares Outstanding, Shares Float, Classification Code(GSIC,etc), P/E, P/S, EPS, P/B. Thanks in advance!
It's complicated, and will cost more in time and/or money than you have available, assuming you have the experience in the first place. The problem is not the conversion, there are algos on the web for that, the problem is the performance to reduce latency. We're using inmemory and even with it 100% optimized have had to shift to an institutional system which we're testing now. If you're not looking for realtime then go to StackExchange, there are a lot of resources for converting to OHLC.
DTN IQFeed has historical tick data available via their API. Use whatever language you want to retrieve it. Process it any way you like. Keep in mind that if you are downloading massive amounts of tick data, do it after hours.
Thanks a lot for your quick reply's @birzos - Im not looking to do HFT, realtime, or auto-trading hocuspocus stuff (maybe later)...! I only want quality DATA IN to do QUALITY RESEARCH OUT to generate filters for watchlist's, and the real good data comes like the good meat: RAW (Tick based); then organize it with standard industry-sectors, make own composites for breath, custom weighted indexes, etc...), end-of-day, end-of-week basis! My approach is that oldie OHLCV compression structure lacks of good information for today's markets (Total volume alone is worthless). Buying and Selling Pressure at volume are loss in the process for example, and that's why Im interested of doing own candlestick structure construction with this information implied in (In fact, Im only adding more data to simple volume bars). @dholliday - Yup, I see IQFeed have much better data and more openly API's for deliver data! I currently payed 1 month for 3 for Metastock's Xenith and they are doing good work in the API side, specially with python, but still need to play more and see how much limitations they put for the retail version of Eikon. Nowadays I see that collecting and/or store quality data is a real mess. Some retail trader "cheap" feeds like eSignal, IQFeed, or even now Metastock Xenith's (Ripped version of Reuter's Eikon) have good timeseries price data but lack one way or the other of some key fundamental ones (Trying to find good CurrentSharesOutstanding + CurrentSharesFloat report is driving me crazy). Sticking to traditional platforms like Amibroker, NinjaTrader, Quantshare, etc, put you in additional limitations regarding of connectivity "plugins" or proprietary database handling. If they don't open more their platforms, they will get nowhere IMO! Now I see everywere on the web that the modern way to go is adopting open programmatically solutions like python, .Net or java to make custom research, making use of open API's and opensource database sub-systems, then maybe use one of these platforms just as mere charting monitoring, but still in the language arena, is hard to find all-in-one guides to start on. Cheers, XeL.
IQFeed has some fundamental data delivered when you subscribe to a symbol (via the API). There is no historical fundamental data. You would need to get a trial subscription to see if the fields are filled in and correct/up to date. Fundamental Message Format The Fundamental Data Message Format consists of a string, starting with a capital F, separated by commas. The termination character for this message is a <LF>. The fields in the comma separated string are listed in the table below. Ord Field Name Field Type Description of field Origin of data 1 F string The character F, indicating a fundamental data message 2 Symbol string The Symbol ID to match with watch request 3 Exchange ID string Deprecated Use Listed Market (field 45 below) instead 4 PE float Price/Earnings ratio Provided by exchange or 3rd party data provider 5 Average Volume integer Average daily volume (4 week average) Calculated by DTN 6 52 Week High float Highest price of the last 52 weeks. For futures, this is the contract High. Provided by DTN 7 52 Week Low float Lowest price of the last 52 weeks. For futures, this is the contract Low. Provided by DTN 8 Calendar year high float High price for the current calendar year. Provided by DTN 9 Calendar year low float Low price for the current calendar year. Provided by DTN 10 Dividend yield float The annual dividends per share paid by the company divided by the current market price per share of stock sent as a percentage. Provided by exchange or 3rd party data provider 11 Dividend amount float The current quarter actual dividend Provided by exchange or 3rd party data provider 12 Dividend rate float The annualized amount at which a dividend is expected to be paid by a company Provided by exchange or 3rd party data provider 13 Pay date MM/DD/YYYY Date on which a company made its last dividend payment Provided by exchange or 3rd party data provider 14 Ex-dividend date MM/DD/YYYY The actual date in which a stock goes ex-dividend, typically about 3 weeks before the dividend is paid to shareholders of record. Also the amount of the dividend is reflected in a reduction of the share price on this date. Provided by exchange or 3rd party data provider 15 (Reserved) 16 (Reserved) 17 (Reserved) 18 Short Interest integer The total number of shares of a security that have been sold short by customers and securities firms that have not been repurchased to settle outstanding short positions in the market Provided by 3rd party data provider. 19 (Reserved) 20 Current year earnings per share float The portion of a company's profit allocated to each outstanding share of common stock Provided by exchange or 3rd party data provider 21 Next year earnings per share float The total amount of earnings per share a company is estimated to accumulate over the next four quarters of the current fiscal year Provided by exchange or 3rd party data provider 22 Five-year growth percentage float Earnings Per Share growth rate over a five year period. Provided by exchange or 3rd party data provider 23 Fiscal year end integer The two digit month that the fiscal year ends for a company. Provided by exchange or 3rd party data provider 24 (Reserved) 25 Company name string Company name or contract description Provided by DTN 26 Root Option symbol (there may be more than one) string A space separated list Provided by exchange or 3rd party data provider 27 Percent held by institutions float A percentage of outstanding shares held by banks and institutions. Provided by exchange or 3rd party data provider 28 Beta float A coefficient measuring a stock’s relative volatility. It is the covariance of a stock in relation to the rest of the stock market. 30 day historical volatility. Provided by exchange or 3rd party data provider 29 Leaps (there may be more than one) string Long term equity anticipation securities Provided by exchange or 3rd party data provider 30 Current assets float The amount of total current assets held by a company as of a specific date in Millions (lastADate) Provided by exchange or 3rd party data provider 31 Current liabilities float The amount of total current liabilities held by a company as of a specific date in Millions (lastADate). Provided by exchange or 3rd party data provider 32 Balance sheet date MM/DD/YYYY Last date that a company issued their quarterly report. Provided by exchange or 3rd party data provider 33 Long-term debt float The amount of long term debt held by a company as of a specific date in Millions(lastADate). Provided by exchange or 3rd party data provider 34 Common shares outstanding float The amount of common shares outstanding. Provided by exchange or 3rd party data provider 35 (Reserved) 36 Split factor 1 float[ ]MM/DD/YYYY A float a space, then MM/DD/YYYY Provided by exchange or 3rd party data provider 37 Split factor 2 float[ ]MM/DD/YYYY A float a space, then MM/DD/YYYY Provided by exchange or 3rd party data provider 38 (Reserved) 39 (Reserved) string Deprecated - Formerly Last Traded Market which is provided in the Update Message 40 Format Code string Display format code, See: Price Format Codes Provided by DTN 41 Precision integer Number of decimal digits Provided by DTN 42 SIC integer Federally designed numbering system identifying companies by industry. This 4 digit number corresponds to a specific industry. Provided by exchange or 3rd party data provider 43 Historical Volatility float 30-trading day volatility that it is calculated using Black-Scholes [Wiki] [Invest] model. Provided by DTN 44 Security Type string The security type code, See: Security Types Provided by DTN 45 Listed Market string The listing market ID, See: Listed Markets Provided by DTN 46 52 Week High Date MM/DD/YYYY The date of the highest price of the last 52 weeks. For futures, this is the contract High Date. Provided by DTN 47 52 Week Low Date MM/DD/YYYY The date of the lowest price of the last 52 weeks. For futures, this is the contract Low Date. Provided by DTN 48 Calendar Year High Date MM/DD/YYYY Date at which the High price for the current calendar year occurred. Provided by DTN 49 Calendar Year Low Date MM/DD/YYYY Date at which the Low price for the current calendar year occurred. Provided by DTN 50 Year End Close float Price of Year End Close. (Equities Only) Provided by DTN 51 Maturity Date MM/DD/YYYY Date of maturity for a Bond. Provided by DTN 52 Coupon Rate float Interest Rate for a Bond. Provided by exchange or 3rd party data provider 53 Expiration Date MM/DD/YYYY IEOptions, Futures, FutureOptions, and SSFutures only IEOptions expiration dates are provided by the exchange. All others provided by DTN 54 Strike Price float IEOptions only Provided by exchange or 3rd party data provider 55 NAICS integer North American Industry Classification System 3rd party data provider 56 Exchange Root Text The root symbol that you can find this symbol listed under at the exchange. Provided by the Exchange >>have good timeseries price data but lack one way or the other of some key fundamental ones (Trying to find good CurrentSharesOutstanding + CurrentSharesFloat report is driving me crazy). <<