Ah, I see. Apologies I mistook this for an idle banter thread but this is actually very interesting. I would say the first thing to keep in mind is your goal is a technological one and should be implemented in a matter that does not consider business functionality. That is if you want the best efficiency you are probably going to want something that is only integers, if it happens to match ISIN then great, but it wont. If it's only speed your looking for then maybe the stream the ticks go through should be aware of what symbol it's for? If not just however many bytes you need to represent all of your contracts. I'm working on a tick fed simulator myself but I wasn't planning on actually analyzing ticks (I wasn't planning on such short term for my first system) but I didn't consider how many raw ticks it could handle. How many ticks are you expecting to have to process? What's the goal? I'm just wondering as my own was around 100k per second (very fair estimate that I have yet to measure, hopefully more like 500k) but this was going to be for 5second+ data not raw ticks so I figured I would be fine. As for what natural identifier you have for all instruments, I would say ISIN. It sounds like equities all have properly formed ISINs and you can always create fake ones for Options/Futures and/or use some hybrid that considers the exchanges (eg, CME + SecurityID) and just be sure you allow a lot of characters or not a set amount of characters anyways. If any of that didn't make sense please tell me, I've been programming a lot so communication is a little rough sometimes.
I am not sure ISIN really fits with what you're doing. regarding broker count... apparently there are over 500K currently registered brokers : ------- http://www.finra.org/Investors/ToolsCalculators/BrokerCheck/index.htm "approximately 660,000 currently registered brokers and 5,100 currently registered securities firms. " -------- but if you have security id and traded exchange that should cover the real data. ID=>Security 32763 => GOOG 28700 => YHOO 10028979 => CL June 09 10131598 => ES Apr 08 10131466 => ES June 08 problem is that sometimes different brokers use different symbols for same security... especially for indicies. so in addition to above table, you also need: BrokerSymbol=>ID /SPX => 10131598 /SPX => 10131466 or some variation, so your designations are portable across brokers. along same lines, it would be cool if you had some space reserved for arbitrary indicies (groups of securities). eg : GroupID => ID 5000 => 28700 5000 => 32763 // group 5000 is the 'search engine' indicie just my 02
You're correct. When I said broker, I wasn't referring to that kind of broker as an individual. Let me use the term "provider" instead as in, "data provider". Data provider refers to the source of the tick data. So for example, you may receive tick data for GOOG both from Interactive Brokers and Sterling. That tick data in the GOOG example above has the same securityId and same exchangeId but will have a different providerId. So, it's necessary to separate ticks by those three id's and even further for futures and options by expiration and strike price. Also, sometimes the same security gets traded on multiple exchanges. Actually, this is where we differ. See if the following makes sense. It seems very important that within the system we use a single id to refer to a security regardless of which provider that data comes from. Why? It's because many users express interest in receive the same data from at least 2 data providers. That's usually their broker plus another like eSignal or iqFeed. But they want the system to compare matching securities and switchover as a backup in case of failure or also to compare them to catch "bad ticks" or missing ticks from one or the other. In order for that to happen, it's critical that the system see the same securityId for both while the providerId (broker) will be different. Make sense? So portability as you describe is needed but in the form of a common securityId. Excellent point. Proposal This proposal incorporates all the issues above plus those related to high speed tick processing. 1. I successfully wrote code the converts an ISIN to and from a ulong of 64 bits using unit tests. 2. Experimentation shows that requires 60 bits which leaves 4 bits. 3. In the remaining 4 bits which makes 16 compinations, we use it as a "header" to identify what the security id represents. A. Security header value 0 means that the rest is a standard ISIN. So we must convert broker tick symbols into ISINs. FYI, in the US the ISINs are simply the CUSIP plus the country code US. Are those provided by brokers like Interactive Brokers on the tick feed? If not, it seems we have create a translation table. Do they at least provide the CUSIP? (Which can easily be switched to the ISIN by adding the US country code.) https://www.access.cusip.com/home.htm?userFrom=CUSIP B. Header value 1 means that the security isn't covered by starndard ISIN. This applies to futures or currencies so TZ will define a its own ISIN for these which can be similar to standard ISIN. Z. Header value 2 means that the security is a custom index or spread defined by a user. If it becomes commonly used, alternately, it can be added to the TZ ISIN (value id). 4. Provider ID will be a uint value with 32 bits. TradeLink must define a custom list of providers that it supports like Echo, Interactive Brokers, etc. 6. Exchange ID will be a uint value which comes from the MIC international exchange/market code with a header for TL/TZ custom ones if they aren't defined by the standard like perhaps currencies pairs which don't have a central exchange. http://www.iso15022.org/MIC/homepageMIC.htm Conclusion The above provides good international portability since it uses the standards defined for that purpose. As far as cost: The list of MIC's are easily and freely downloadable. The list if CUSIPs for US ISINs require a subscription (at the above link) and no price is listed for subscription. We need to find out if cost is prohibitive for any of this. Each country administors it's own section of the ISINs so eventually a database of all the ISINs for countries support by Tradelink will be needed.