Streaming Data Feeds may soon get easier

Discussion in 'Data Sets and Feeds' started by nitro, Mar 17, 2006.

  1. nitro


  2. Any ideas how long this compression takes? Still don't think it can be specialized binary protocols..
  3. nitro


    I don't know, but I can't imagine that if it weren't competitive with what they are doing now it would be adopted.

    Best is probably just to keep an ear to the ground and see how it developes.

  4. Yeah, well it's definately good news, hopeuflly it'll develop into something. I wish people would start looking into STCP rather than UDP for multicast dissemenation.

  5. nitro


    What advantage does SCTP have over UDP multicast? I can't tell from that article.


    - multi-homed
    - connection-oriented similiar to TCP
    - automatic rebroadcast/assembly of dropped packets
    - multiple streams within one connection
    - smarter routing
    - handles congestion more easily than udp

    lots of others I missed, but seems interesting.

  7. Sigh, I guess everybody is entitled to create their own pseudo-compression routines. What about those of us that have zlib routines written in assembler? Being a former cryptographer, I keep on wondering why don't more people in the financial services business look at ASN.1 (using PER, for instance). It is the standard defined by ISO and ITU-T, and it is implementation / transport independent.

    All routers and OSes have implemented ASN.1 (by definition of X.500 and X.680 compliance).

  8. Probably because it has been put in the 'too hard' and too complicated basket. And because of the somewhat unreasonable predjudice of some in the TCP/IP community against ISO protocols.

    You cannot reasonably work with big ASN.1 protocols without an ASN.1 compiler. It's several years since I had anything to do with it, but in those days, there was just one free ASN.1 compiler that I could find. And commercial compilers were quite expensive. This tended to discourge it's uptake outside of the telecomms world and in particular experimenting/prototyping with it.

    Also the original ASN.1 specs had 'macros' which were truly horrible and very difficult for a compiler to deal with. I belive they were later dropped, but application layer protocols such as ROSE used them.

    I don't think your point about routers and OS supporting X.500 is particularly valid as X.500 (and friends) contains a specification for a concrete transfer syntax specified in ASN.1. ASN.1 as it's name implies is an abstract presentation syntax.

    I do agree that ASN.1 would be a decent choice for this type of thing. It's encoding is quite efficient and way better than text based protocols. And it comprehensively supports all simple and compound data types that might be needed.

    I've always liked SUN's XDR for it's simplicity and ease of use though the lack of support for arbitrary length integers is not good for things cryptographic.
  9. nitro


    That is very interesting. Thanks for the links and explanation.

  10. I couldn't agree more. The funny thing about finance is that as much as people in finance like to think about they are doing complicated things, the technology aspect reluctant to adopt another that seems overly complicated.

    I still use the ASN.1 compiler in OpenSSL (formerly SSLeay) on occasion, it is very good and very fast, I contributed a bit to the PKCS12 and PKCS11 implementations. It is not a compiler per se, it is a set of hand crafted routines parsing for X.509 family, but it works very well. While ideally a compiler would be nice, but people don't use FIX routines derived from a compiler either, they use custom written FIX routines specifically for that version / product / destination / etc.

    All I was trying to point out that through Microsoft CryptAPI, Cisco SNMP, etc, most of the devices already have a working (but concrete and specially written for a single purpose) ASN.1 implementation somewhere in the system. The implementation may not be usable, but it is just a matter of extending the libraries to handle an ASN.1 version of FIX (for instance), etc. In terms of speed of parsing, if one can do software IPSec with > 10 mb/s of throughput, I can't imagine speed being a problem.
    #10     Mar 18, 2006