I did not run those tests myself I just took the numbers from their site. I was looking for a ballpark number before doing my own testing. Looks like Redis stores data as strings so you will need to parse data anyway to get ints/doubles. How fast is your Redis? How do you store data? Do you use sorted set where score is the time of a tick?
true, but the strings are binary safe, you can as well handle it as binary with string key look ups. So, there is no per se parsing between strings. A string is just defined as the format of choice to hold data, whether a byte array, string, integer, or image. If you just want to deal with binary data you just store as byte arrays and then deserialize yourself with protocol buffers which are a lot faster than most other serialization libraries. Here is a basic intro to the data types: http://redis.io/topics/data-types-intro If you take a look at Booksleeve, the async C# API for Redis you find that it takes byte arrays as input, so serialization/deserialization is done on the user's side. But again, I do not want to come across as the one who thinks Redis solves all problems or is the product of choice here, it is not. Binary flat files are the storage of choice here imho. And even if I did that on a server I would not use REST to serve the data. Even the slowest .Net Windows based tcp stack is faster in delivering the data than any REST API could ever accomplish.
- "No longer advocate Redis?" - not criticizing client/server? Hmmm. Lol. youporn? nah, hardcore is where it's at. kink.com. Now if they were using Redis, I would be impressed. Yes, yes. We are all impressed with your incredible knowledge, KDB and such. In honor of this discussion, Jtrader should christen his new client/server app as "RedisDB 2.0". Lol. Nah, that would be too arrogant.
I use redis only to do inter-proc communication and, as described, to quickly serve data that my app outputs and which I like to analyze in R. Other than that I run my own binary data store with query algorithm that can lookup and load a requested data packet between start and end time stamp. The packet location can be determined in a matter of single digit milliseconds regardless of size of file. I store tick data depending on what data we talk about. For example, fx tick based data is stored in 16byte data structures, 8 bytes for the time stamp, 4 bytes each for bid and offer.
+1 for redis. It's quite easy to get started with redis : http://try.redis.io/ It's quite adequate for your specific use-case.
Very simple to setup and beats the heck out of having to code DTOs, requests, replies to get a simple REST interface going.
I am not too familiar what exists in Java but I guess your link points to an object serialization library for Java, correct? It can be used to serialize objects regardless of storage type (file, database, memory). In the .Net world the most efficient libraries I found to be Protocol Buffers (protobuf-net) for binary serialization and the Service Stack serializers (json serializer). Not sure what is best in town in Java world...