Database is not only small, it is also incredibly space wasting. Once you deal with real data amounts that is a serious difference. For example I use Nanex tapes. A Day has about 1.5 gb data - sadly in SQL form it uses about 50gb. HUGH difference. So we extract the tapes into files with the ata wen eed, using a highly compresssing and read speed optimized binary format (variable length, a trade can be down to one byte, that includes tick accurate timestamp, volume, price). Then run along the files in playback etc. Keep in memory only what is needed - especially when you do algo the need for historical access is close to zero (except loading data, but you never buffer it).
Iti s not so hard. byte 1: 2x4 bit. first 4 bit: packet type, which is trade 2nd 4 bit: option length, which is 0 Decoding: Trade, same teimstamp, same price, same volume like last one (which happens quite often). Otional fields are: * TIme delta (in "ticks", predefined in another message, our feed has 25ms there as granularity) * Price (as integer in ticks, against predefined, this is a delta (1 byte normally enough) * Value, against as delta normally * Security (to change between securities) Not that hard. You have to understand that most things are not jumping around like mad. Depending on market it is quite normal to get similar priced events quite often. Decoding is cheap.