The open source thread

science_trader · Oct 10, 2005

Quote from Josh_h34:

You're right that I want to avail myself of the realtime scheduling class on solaris.
The reason Java can't cut it for me is because I reduced the periodicity of my bars from 15 min to 1 min, now I have to monitor complex breadth indicators on about 800 highly liquid issues once a minute and the time series became 180,000 bars long per symbol.
That's 140MB of raw data, not to mention multiple other blocks of memory used for sorting, indicator data etc.

With all due respect for Java which I like a lot it takes more than a minute to do the number crunching and make some sense out of the new data. Even a minute of number crunching is too long for processing 1 min bars.
Rather than try to parallelize computations I opted for a high end workstation with lots of RAM and a fast vectorizing FPU and now I am looking to rewrite the code in c++ for speed.
Hopefully compiler optimizations will cut the processing time to 30 seconds or less.
What's the best c++ api out there for order entry and management?
More...

Have you tried a profiling of your java program ? I used to find that some very little thing in your program can slow it up in a way you would not have suspected.

However, it's quite true that java garbage collector and memory management is really slow and unpredictable.

prt_systems · Oct 11, 2005

Quote from Josh_h34:

.....
With all due respect for Java which I like a lot it takes more than a minute to do the number crunching and make some sense out of the new data. Even a minute of number crunching is too long for processing 1 min bars.
Rather than try to parallelize computations I opted for a high end workstation with lots of RAM and a fast vectorizing FPU and now I am looking to rewrite the code in c++ for speed.
Hopefully compiler optimizations will cut the processing time to 30 seconds or less.
What's the best c++ api out there for order entry and management?
More...

Well.... if you have done this before then you should have no problem. However if you have never tackled this type of project then do not underestimate the amount of time it takes. The first project i did like this years ago took quite a bit of time due to the learning curve. Even today, having built several of these I still find that the testing and optimizing aspects take significant time and effort.

HLB · Oct 11, 2005

I had the exact dilemma. For the non-trading projects I usually use C++, but IB C++ API is MFC-based, and their Java API looks much more solid and stable. Also I need multithreading, since number cunching is done in separate thread, while the main thread continue to receive market data. Multithreading is also much easier in Java. But number crunching itself is done in C++ via JNI. It's also preffered not to use double's in Java, but hold prices as integers.

Quote from Josh_h34:

You're right that I want to avail myself of the realtime scheduling class on solaris.
The reason Java can't cut it for me is because I reduced the periodicity of my bars from 15 min to 1 min, now I have to monitor complex breadth indicators on about 800 highly liquid issues once a minute and the time series became 180,000 bars long per symbol.
That's 140MB of raw data, not to mention multiple other blocks of memory used for sorting, indicator data etc.

With all due respect for Java which I like a lot it takes more than a minute to do the number crunching and make some sense out of the new data. Even a minute of number crunching is too long for processing 1 min bars.
Rather than try to parallelize computations I opted for a high end workstation with lots of RAM and a fast vectorizing FPU and now I am looking to rewrite the code in c++ for speed.
Hopefully compiler optimizations will cut the processing time to 30 seconds or less.
What's the best c++ api out there for order entry and management?
More...

HLB · Oct 11, 2005

Quote from Josh_h34:

I reduced the periodicity of my bars from 15 min to 1 min, now I have to monitor complex breadth indicators on about 800 highly liquid issues once a minute and the time series became 180,000 bars long per symbol.
That's 140MB of raw data, not to mention multiple other blocks of memory used for sorting, indicator data etc.

More...

140MB / 800 / 180,000 ~ 1 B per bar

How do you compact one bar into single byte? Bar is at least 4 (5 with volume) integers. 2 bits per price?

forextrades · Oct 11, 2005

Can't you just run the data feed on Linux through a Windows emulator? Then just convert the data so it works on your charting software on Linux.

dcraig · Oct 11, 2005

Quote from forextrades:

Can't you just run the data feed on Linux through a Windows emulator? Then just convert the data so it works on your charting software on Linux.
More...

Probably with wine or vmware. But all such arrangements are to be avoided if possible. Simplicity should always be aimed for and adding extra software components for which there is no good architectural reason does not promote reliablity. It also complicates system monitoring and error recovery.

constantnone · Nov 7, 2005

How about this one:

https://humaitrader.dev.java.net/

It seems progress quickly.

mattgundersen · Apr 14, 2006

Wow, that EclipseTrader is nice. Humai is pretty cool too.

I see that both of them are open ended for datafeeds and both already have support for multiple feeds (realtime and EOD).

Humai has a nice set of line tools and he's been expanding the studies adding a neat interactive volume profile thing.

I also saw someone from www.opentick.com asking the guy @ EclipseTrader to integrate his package to opentick feed.

What would really be nice is for someone to hook one of these into the data feed that comes from IB's TWS.

m.