my point is very simple: why should something NOT work, just because it is beyond my current ability? if nearest additional sharpe ratio was higherMathITOrWhatever method, i'd send all forces into that direction. my "problem" is that i have not yet exploited all the "easier" things ... IMHO it is really not useful to academically discuss technology. i had wonderfully skilled people on my desk, who could not at all bring their know how down to earth of system trading. and i had people with much less academic background and they produced additional sharpe almost from day one. and the other way round. it is much more about focus than background. i guess it is the same with every level of complexity. some nn-technician will tell you for years why he needs additional RAM-Data-Money to "make it" and another guy will sit down and just do "it". that happens on all levels. just, if you are managing the whole thing and you hire people you look up to, you might experience year after year the RAM-Data_Money story ...
Very good points, man. It basically comes down to two extremes: hacker and theoretician. The hacker 'just does', and this is the easier thing because all you need is dedication and a computer. I traditionally fall under this category because all my computer skills are self-taught. The theoretician is a pure thinker who slaves over logic, equations, proofs, etc approaching the problem from the point of view of logical inducation and small steps of truth, to arrive at greater truths. Hackers can create some very amazing stuff if the complexity or scope of the project is limited (e.g. code some wicked fast sorting/retrieving/balanced trees, queing models, etc). Of course there are some extremely talented hackers that can create very large and complicated systems that work, but these types are the exception rather than the rule. The theorist does the same thing as a hacker, except everything is 'proven' to be true (within the bounds of the assumptions made beforehand). And the theorists speaks a different language of notations, proofs, theorems, etc. These theorems and proofs can be turned directly into working code by hackers who can bridge the gap between these two worlds, enough theory to be dangerous, enough code to be lethal. So designing algorithms, creating models of equations to represent observed/experimental data, etc, is a process of theorizing/fitting to experimental data/modifying theory if needed/verify again, etc. This has served science well, reductionism, but we are enering a new realm of science: complexity. Where instead of reducing science to simple behaviours, we combine existing simple behaviours to see how they interact to form global/complex/intricate and seemingly "random" behaviour, also, complexity is an extension of chaos theory to systems beyond deterministic nonlinear equations. One of the top guys in this field is one of Time managinzes top-100 influential people Geoffrey West: Master of Complexity The coworker I mentioned ending up leaving the AI field because of the problems you mentioned. The vast amount of areas of research is expanding so quickly, that there is no time to master it all, because by the time you master one theory there are 2 new ones already popped up and you must decided which one to devote your time to studying, and even then there may be many many old/standard methods that you havent even taken a look at yet either. He worked with theoretical physicsts to take the abstract theory and turn it into working code, but this type of thing only goes on in deep-pocket institutions or crazy guys sitting in their basement with stacks of CPUs, or in mysterious no-sign-out-front wall-street shops. He became so consumped with the research and theory and coding that he was becoming consumed with it, living room stacked with books, journals, etc so thick you couldnt walk thru it. He finally decided to leave it all behind and live a simple, relaxed life with his wife and kids. Not that he failed, this stuff is in use in space/technology applications today, but not stuff the average person uses or knows about. man, I think you ultimately need to use your intuition to decide what to pursue, and it takes a leap-of-faith to have confidence that you are not wasting your time. But, if your understanding is increased, but the application isnt directly useful, that can still be considered useful because you learned and can apply that knowledge to the next area, so that you contintually approach the solution.
i think you are right. it will boil down to intuition. first i wanted to do some agent based stuff, but i am afraid that the number of agents will open up for too much fitting ... as happens with nn. someone proposed taboo theory to me. have no clue what that is about ...