Tom, Most of the successful traders I met in person or by phone/email weren't the best programmers in the world. Most of them knew nothing about programing, but they have the best ideas. And I mean really really good ideas ... out there.. far away from mainstream. Most often than not the existing technologies/platforms are not suitable for these ideas. But by knowing what they want to do, I know what it has to be done. And somehow I have figured out the solution for easily 90% of these massive data/massive calculation/real time response ideas from the technological perspective. Thanks, ~mhtrader~
I don't know if some of you know that lot methods in the Object class are implemented in C++ and not in .NET itself.( source file ecall.cpp/comobject.cpp on the Shared Source CLI ). The main reason is because C++ is faster. In the system namespace along I found close to 200 methods of the .NET framework implemented in C++ ( of course you can't do that yourself, Microsoft don't let you do that, they reserver that trick only for themselves... at least that's what I found around 5 years ago ) But besides that. Here is one of the method that is most often used in the framework Object.MemberwiseClone. Sometimes you don't even call that method but it is inserted in the generated .NET code for you when objects are copied( value types, ref types, arrays ). This will be equivalent to a mem move or a copy constructor in C++. Here is the code from Microsoft: PHP: FCIMPL1(Object*, ObjectNative::Clone, Object* pThisUNSAFE) { CONTRACTL { MODE_COOPERATIVE; DISABLED(GC_TRIGGERS); // can't use this in an FCALL because we're in forbid gc mode until we setup a H_M_F. THROWS; SO_TOLERANT; } CONTRACTL_END; OBJECTREF refClone = NULL; OBJECTREF refThis = ObjectToOBJECTREF(pThisUNSAFE); if (refThis == NULL) FCThrow(kNullReferenceException); HELPER_METHOD_FRAME_BEGIN_RET_ATTRIB_2(Frame::FRAME_ATTR_RETURNOBJ, refClone, refThis); // ObjectNative::Clone() ensures that the source and destination are always in // the same context. MethodTable* pMT; DWORD cb; pMT = refThis->GetMethodTable(); // assert that String has overloaded the Clone() method _ASSERTE(pMT != g_pStringClass); cb = pMT->GetBaseSize() - sizeof(ObjHeader); if (pMT->IsArray()) { BASEARRAYREF base = (BASEARRAYREF)refThis; cb += base->GetNumComponents() * pMT->GetComponentSize(); refClone = DupArrayForCloning(base); } else { // We don't need to call the <cinit> because we know // that it has been called....(It was called before this was created) refClone = AllocateObject(pMT); } // copy contents of "this" to the clone memcpyGCRefs(OBJECTREFToObject(refClone), OBJECTREFToObject(refThis), cb); HELPER_METHOD_FRAME_END(); return OBJECTREFToObject(refClone); } FCIMPLEND Another hurtful example. Would you like to call myArray.length in C#?? ( in C++ is myVector.size() will be probably 2 or 3 assembler instruction ). Here is what happens when you call myArray.length. A function call into the CLR to this C++ function: PHP: FCIMPL2(INT32, Array_GetLength, ArrayBase* array, unsigned int dimension) { CONTRACTL { THROWS; DISABLED(GC_TRIGGERS); MODE_COOPERATIVE; SO_TOLERANT; } CONTRACTL_END; VALIDATEOBJECTREF(array); if (array==NULL) FCThrow(kNullReferenceException); unsigned int rank = array->GetRank(); if (dimension >= rank) FCThrow(kIndexOutOfRangeException); return array->GetBoundsPtr()[dimension]; } FCIMPLEND I think that illustrates enough. Thanks, ~mhtrader~
Certain C++ compilers have been doing it for years. From Agner`s CPU blog (talking about Intel's C++ and Fortran compilers): "the compiler or library can make multiple versions of a piece of code, each optimized for a certain processor and instruction set, for example SSE2, SSE3, etc. The system includes a function that detects which type of CPU it is running on and chooses the optimal code path for that CPU. This is called a CPU dispatcher." This is something Microsoft C++ compilers do too. If configured to optimize for speed, they inline many functions that haven't been explicitly defined as "inline". In fact, such optimization even happens across object modules at link time.
Thank you, Tom, for adding the acronym AOT=Ahead-Of-Time for the type of compilation/optimization done by C++ compilers. When I wrote earlier, I had to say "compile-time optimization vs JIT", but I only did that for lack for knowing the AOT vs JIT. Aside from AOT vs JIT, the subject of optimization is broad. We can all agree that taking calculations out of the loop is a valuable tactic, but actually, like most optimization methods, it depends on the hardware. As our servers get more cores, we are quickly coming to fine-grain parallel computing. By fine-grain parallelism, I mean using many cores to execute the same instructions concurrently. The importance of this can be seen by the growing support by Microsoft and Intel. I have spent plenty of time tuning socket servers built on coarse-grain parallelism. (Have four threads paused waiting for the next batch of quotes. If a second batch arrives before the first is completed, wake a second thread to leap-frog on the first batch. Once you have four threads chewing on a batch, other threads on the system are getting starved so yield, sort, bunch, and try to catch-up. ...) With fine-grain parallelism (distributing operations across multiple cores), memory management is all so important. In the example of taking a calculation out of the loop, this is often a poor choice when using parallelism. If it the inputs need already be pulled to each core, then often, it is faster to let each core produce its own copy. Optimizers will still keep it to one calculation per core, but the issue isn't as clear-cut as it originally sounded. LeeD, I am glad that you were right about loops of addition being optimized to multiplication in certain situations. I agree with many on this thread, that it is the delivery of useful features to market that matters the most. The software architects with whom I have worked are far more concerned will timely, scalability (network bandwidth, security, and redundancy) then CPU load. Code optimization is something that can be done post-profiling. (That is the point of having a profiler isn't it.) I have code that correlated most futures and equities multiple ways in multiple timeframes. It spends hours doing floating-point. The thing that mattered most when I optimized it, was memory management. The memory cache is big enough to fit a few data histories. I reorganized the calculation to load a few series in cache, and then compute the combinations. I wouldn't be surprised if a .Net JIT version with this clumping wasn't faster than an AOT version done in the unconsidered way. Mhtrader, the Advanced Market System's platform (not available for sale at this time) does support .Net plug-in studies which can update every trade, Level I quote, and/or every Level II quote for a whole portfolio (1000 equity symbols tested.) It is AOT compiled, but the idea of doing this is not unique. My earlier point that this trend began as a business/features discussion is still valid. Having a product and being a successful business are different things. Facebook is not in the news for having solved a technical challenge which allowed social networking to become a reality. Tom, I disagree with you that a basic design decision can be a truly fatal flaw in a platform. It is not that I disagree that design decisions can be truly hampering. Quite the opposite, I have consistently found that as one takes the concept and makes the reality, there are a myriad decisions to make. And as one makes decisions, following decisions depend on earlier choices creating design paths. Much like the game of chess, many architectural decisions are intractable. That is why people are shifting to iterative design principles such as Agile. Just as with a trade, one doesn't want to overcommit to an earlier decision. On the other hand, to get the iteration process to converge in a timely manner, it is often necessary to curtail discussions similar to the AOT vs JIT flame.
Kind of reminds my of Wayne Walters Tickzoom project. Also a solo developer who sold a few licenses of his thing (also claiming unparalleled performance and so on), and now the project appears to be stale since years. I guess the thread is still on this board somewhere. If you pay up and get Orc or Apama or some such, atleast you will know the project wont be ditched the next day. And how is a single guy supposed to be able to maintain all the broker and exchange-connections - let alone support clients? The intentions may be good (like i suppose Wayne's were), but im not sure how this is supposed to work out..
This thread is really a good one. At least if it comes to compilers and optimizations. Iâm really happy to learn something here and to read interesting things about compilers! Since Iâm a developer since the Sinclair ZX80 I think that I âm a little bit aware about the compiler basics â especially if it touches the Microsoft VC++ world - which is my âhomeâ environment. I know the Tickzoom project and since Iâm also a âone man showâ I can understand his idea to setup Tickzoom and try to see what is possible in the market. Today people expect software to be free (or close to be free) which make live for developer much more difficult and it makes it hard to place new products in the market (because development cost time & money). If people wouldnât try to realize their ideas, the world would stop to develop (just my 2 cents). The problem here is how should Wayne pay the cost to support his clients, for maintenance and for further development? This is easy for big companies, banks and brokers â but for new companies and products it could be impossibleâ¦but maybe some new product and ideas would be healthy for everybody???? Daniel
Visual C# 2010 :: Mono C# compiler version 2.8.2.0 http://shootout.alioth.debian.org/demo/benchmark.php?test=all&lang=csc&lang2=csharp Visual C# 2010 :: Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode, sharing) http://shootout.alioth.debian.org/demo/benchmark.php?test=all&lang=csc&lang2=java Vista
This thread (and many others lately, in the automated trading forum) is suggesting me a little story. Imagine i am a scientist writing a paper for a scientific journal. I am quite ambitious: I want to to propose a new primality test which runs in time O((log(n))^2) . Now, the editor and the publisher of the journal, are also impatient (i have anticipated them my research) because the publication of a breakthrough will make the journal among the top ones, and probably the publisher a rich man. Now, i begin my research to reach and consolidate this result. But, while a do so, i begin to be concerned: what editor/technology should i use to write the paper? Should I use Latex. Will Word do? Or maybe Open office will be faster ? Hmm, I am just wondering if Word is able to take advantage of all the cores of my machine Perhaps, i could open 2 windows of Word and use 2 keyboards. <b>I could type in one window with my left hand, and in the other one with my right hand</b>, thus parallelizing my production ) So i get so carried out with all the technical details and concerns around some aspects my task that some time passes (in the meantime i also had to learn c++ because word wasn't clearly fast enough and had to rewrite my own word processor. Then, noticing it was still too slow . I rewrote the all thing in assembly, and now it seems ok). Then, after several months, i get the call from the editor. Clearly, i am so enthusiast to tell him all the progresses i have made and how fast, parallelized, multithreaded, and great my typing can be. Wow! The editor is quite impressed and pleased. But after several minutes of listening all my wonders, then he asks me: ... ok ok sure very good! All that is beautiful. But <b>where is the result </b> ? :eek: Tom