You certainly are on a much smarter track than Rufus. (Expert knowledge of C is still indispensible, but use it only when you really have to! ) Once you start touching things like 'matrix packages', you should take a look at Python. You got the right idea, but you haven't seen anything yet. nononsense
Well rufus has been doing this for quite a bit longer than I.. don't know if java/OO was all that popular then. I've had a look at python and it just feels dirty.. I hate loosly typed languages, makes my brain do weird things. I'm open to the possibility but I just can't bring myself to make the switch now. Also, one of the things I like about my matrix package is the verbosity.. I know many cringe at this, but it's almost like reading pseudo-code.. it just makes sense. One of my main gripes about matlab/single char operators/mathematic notation is it's terseness and thinking it requires just to figure out what something is doing.
Stephen, Maybe you should look once again. Having done Java before Python, I find Java a "high clutter" language in the sense that you have to occupy yourself with lots more useless stuff to do the same thing. Personally, I would say, the less clutter, the less likelihood of brain damage. I do really a lot of mathematics with python enjoying the luxury of the availability in python of the best and most seasoned libraries in the field. I have not yet missed for a wink the former drag of having to type everything explicitly. This is inherent in the design of python. No more need for typing at all. Certainly nothing loose ever! nono
Which libraries specifically? I've learned a lot writing my own matrix package.. I think it is robust enough to release publically but don't want to go thru the trouble right now.. specifically most matrix libs/languages are fairly ineffecient for instance, a matrix transpose actually creates a fully copy of the matrix, whereas my package makes the new transposed matrix share the same underlying native buffer and only swaps the num rows/cols and the col/row increments.. many of my native fortran libs can deal with matrices such as these and it is not necessary to do a full matrix copy before doing some operation. If a native func actually needs a column-major matrix when I just do 'matrix.trans().copy()' before passing it along in the case where a full copy really is necessary. Same concept goes for matrix slicing, reversing.. swapping, etc. I've reimplemented the unscented kalman filter with matrix package, translated directly from the matlab version.. it is about 15% faster and infinitely more readable and fully object oriented. Things like this make it more apparant about the memory access and runtime of your algorithms and makes it more readily apparent where optimizations can happen.
I agree to some extent, writing my own libraries would let me do a lot more customization and tuning than if I just blindly use some pre-built package. But there is another side to this coin, for instance, I found the native Linux thread scheduling inherently inefficient (and Alan Cox admitted it as much), so I ended up rewriting a good portion of the native threads library (it is being reviewed by the kernel dev-group now), yes, it is quite a bit finer grained and faster, but it also caused some problems in stability (hehe) that I had to track down. Like-wise for memory management, I resolved to write my own memory management, so that my code won't be allocating memory or even worse, garbage collecting when my system need to response the most, of course managing my own heaps is problematic as well, so I had to spend more effort to track down a few rather nasty bugs. The upshot is that yes my system is quite a bit quicker and cleaner, and I have full control, but the trade off is that I now have a large code base that I have to maintain and fix bugs. Oh yeah, I am not *that* old, but I do remember Gosling's first talk about Java (and the somewhat buggy JDK 1.0.0). I just happen to have started coding fairly early (mid 80s). I learned OO concepts from Smalltalk-80, so C++ for me is a weird 'bastardization" of OO concepts (what? no metaclasses? what about meta-meta-classes?).
Excellent observations, I've experienced just the same thing.. I've definitely had much more headaches debugging my packages than I would have if I had used something else. However, in this case, there really are no high performance math packages available for java.. well, I think there is 1 or two but they are quite crappy. One thing I've learned from all this: unit tests are your friend. I used to absolutely hate writing test code, now I almost like it.. it's saved way more time than I spent writing the tests. lol. My first experience with OO was Objective C.. I thought it was pretty nice. One thing about java that really bugs me is the lack of multiple inheritance.
Hi Stephen, To get a quick idea, start looking at: http://www.scipy.org/ http://www.r-project.org/ http://rpy.sourceforge.net/ http://www.gnuplot.info/ http://gnuplot-py.sourceforge.net/ also MLab.py and matplotlib.py modules. These above libraries contain an unbelievable wealth of thoroughly tested and optimized code going back 20 years or more. In fact some falls back on hard to beat Fortran. Most of this would be impossible to duplicate during an individual's lifetime. Before you code - first look. To acquire these for other platforms cost a fortune (or used to?). In fact MLab is still very pricey! Your experience of transcribing MatLab code into any serious language is revealing! For those who don't know, doing this, quickly teaches the difference! (You ought to try some in Python.)
Rufus, Talking about thread scheduling. If you insist on too much granularity, you will run into trouble on any OS, not only linux. In fact Linux or Unix might do much better "out of the box" than others. This is a fundamental limitation of every standard OS. As you know, many real time operating systems (RTOS) exist. One of the major differences is the much greater flexibility and granularity of thread scheduling possible. RTOS can be found for compact embedded systems and also for large scale industrial real time control applications. In trying to better standard OS thread scheduling facilities, will always quickly confront you with this basic limitation. BTW, I use often the Python thread.py and threading.py facilities. This covers most of my needs more than adequately. Years ago, I once did a small RTOS by myself. Indeed, bug fixing & maintaining is quite a chore! nono
Stephen, That dumb new 10min posting change rule makes me add this in a new post. You should also look at: http://new.scipy.org/Wiki In fact a new version is just released. I still have to look at it myself.