Zzzz1: You're still thinking 100% in terms of workstation and not server. I can't help you anymore. There's an old saying that "Programmers make bad admins". It's true in many cases. I started out with difficulties but learned quickly. Your statement about worrying about optimization later is incorrect, unless you have lots of free time on your hands and enjoy mundane programming. A little forethought in the design can save you hours/days of headaches later. I'm not talking about taking optimization all the way down to the assembler level. You still ignored my language question, but it doesn't matter anymore. fan27 hasn't stated anything about his algorithms except for back testing. I've been doing various complex back tests and data processing in my scratch pad C++ program for years now. I know the difference between heavy and light, inefficient and efficient. fan27: You got the points. Since you've mentioned loading from multiple sources, write your load functions so that they can handle any source you want and multiple sources together. A little extra glue code will offer some nice flexibility in the future. An example might be: load compressed data for a really long test using old data that isn't accessed much any more, load regular binary data data for the recent months, and get the most recent ticks from a database.
In case you have missed the thread title, this is about "Go", hence its not about language choices! And maybe you have experience in areas you claimed expertise in, most certainly do you seem to have very little experience in trading system architectural design and workflow. You refuse until now to even look at the consumers of data, almost stubbornly so. By the way, most every seasoned developer and project manager will tell you that early optimizations are the killer of most projects. You make it sound as if one must choose, either well thought-out design at initiation, or later profiling and optimizations. Good design does both. Your diatribe about kernels and what have you clearly demonstrates that you tend to over-optimize prematurely when you do not even know what the bottlenecks are. Let's agree to disagree because I could not disagree with any of your points more. P.S.: Regarding your language question, I am willing to make a bet with you that I can write a C# based simple data loader, streamer, and backtester, including strategy code that will consume pricing data equally fast as any C++ code you present. In fact I will finish the project most likely in 1/4-1/2 the time it would take you. That is because you do not seem to understand that the bottleneck is in the strategy code and a well versed C# coder can write core statistical algorithms and other basic array and algebra functions as efficiently as a C++ coder. Language really does not matter in this particular case, except that you with your C++ development most likely take 100-300% longer than I would in C#.
This very appropriately popped up on the side: For my point, I'll assume we all have something to learn, both from more experienced software devs, but also from our own experience with programming, an experential and even subjective kind of learning that's not possible to truly master from books alone. It is true that good design is often very much about separating concerns. In software development abstraction is a tool for that. However, I'll argue abstractions and separating concerns are not ends by themselves, but more the kind of useful tools that allow us to easier reason about our designs, ie. they are used due to our limitations. The reason we separate concerns, is really as a way to cope with the inherent and emergent complexity of all systems. We quickly lose the overview and control (after 7 items), and then use divide-and-conquer as a useful strategy to cope. The end result, us coping, has some value, while separation and abstraction need some justifications for value, because the costs are so much more than just the extra effort. One such justification might be that it allows us to measure separate components, to compare different versions of them and even exchange them, without affecting the rest of the system. These may be good justifications, but only if you truly need it, and also have identified your required / correct boundaries in your DDD. If you program your entire life and never really need it, it's the wrong justification for all the implied costs. You may find further justifications, such as pride, that far outweight any perceived costs. Most just follow trends and "design patterns", never figuring out why they do the things that they do though, and thus have never really identified any value in doing it the way they do it, or found a better/novel way to do it. About being afraid of refactoring, rework and rewrites, that is a sign of bad design, too complex components and over/early-optimalisation. "Business logic" may have some value, the code should not (but may still). Marry your code, and you risk it stagnating with you. In fact, for extreme creative and innovative endeavours, I'd argue you should plan to trash all your code and implementations, or at least make entire new creations. If creativity and innovation is top priority, this is an effective method to truly test out different approaches and to radically improve. If your goal is to create a milking cow, of course follow that strategy by all means and don't disturb the cow! These are just thoughts from mostly Go-related inputs, and since I don't know the problem, I can't really provide concrete recommendations. However, I do think it's worth considering more practical approaches rather than designing cathedrals every time a new program needs to be made. It doesn't make sense to scale software until you truly need it, and at that time only will the requirements become a bit clearer. Maybe the software never takes off, then all the extra work is just wasted. The better way than just guessing, is to do and then measure. Start somewhere and improve. For a business, there are other more important areas than technology that needs focus first, so it may not help to scale the software if the business fails, though if the business is threatened because of performance, by all means do tune and scale! But again, it all depends on the problem and/or your goals. There are just so many so-called "truths" about software development and programming, that shuts off people's brains these days, so is still a healthy reminder.
Very well put, thank you for sharing your thoughts. I have never heard from any of the developers I came in touch with that they would not significantly rewrite code as the project grew and matured. It sounds on the surface inefficient but really seems necessary during the growth process and adjusting to different parameters over time. A lot of people make the erroneous assumption that projects are either indefinitely funded or that deadlines can be pushed farther and farther. In the business world that is simply not the case. Too many early optimizations, and a project implementation will not perform at all by the agreed deadline. Too few design consideration early on might later prove very costly. A balance has to be found and it depends on a lot of variables as you pointed out. I believe all projects should strictly start with the end user in mind. What are the needs that ought to be met. What are the opportunities, the limitations. Where are the bottlenecks. Answering those questions early on already makes a major difference between success and failure.
Speaking of projects, as they are defined and implemented, often tend to become a root cause for problems, not a continuous solution. This is because, projects by definition, are time-bound, resource-limited and goal-limiting processes. So how can projects address long-term value appreciation, continuous improvements and maintenance? The truth is, that's not what they are designed for.. Yet, projects keep binding up ever more resources, time and mindshare in most backwards organization, until the top brass leaders give them up! Projects may work well where the requirements are 100% clear upfront and never need any changes (ie. any change to an RFC leads to "hell"). Static requirements or even fully known requirements never truly happens in today's fast-paced world. So projects are becoming marginalized and their damage will be minimized. They can work well though when an organization truly need to prioritize larger cross-cutting changes and there's enough management support, but the costs to the rest of the organization is often huge while that is going on, as with any organizational change. Essentially, projects are like little black holes in your organization sucking up energy. And I'm not being mean here, that's what they do, all the while giving pretense of solving real-time problems (projects do NOT solve real-time problems). Back in the software development-world, which is fast becoming central to all organizations, because the best organizations are technology-driven, not gut-feel driven, changes in software and organization are becoming too complex, competitive and rapid for old-fashioned management-methodologies like projects to keep up. We do need cross-functional teams though, and we need to keep the good teams. This is better done by not disrupting their work every year creating new "projects" by constantly killing good previous efforts and conveniently forgetting their original goals. There's going to be alot more changes due to the "old" ways of working not really keeping up, least of all keeping their promises, even when on-budget and with high throughput (garbage in -> garbage out). Software is never truly done, unless your company has bankrupted itself. So the definition of done is that it's not, or shouldn't be if you want to stay in business. Sorry for the little rant, your points still holds. EDIT: Of course "Agile" may/may not solve these problems. To solve the root problems though, one must address the origins of the problems (root causes), ie. lack of understanding of how to exploit IT instead of being victimized by IT. "Projects" are not the problem itself, just a symptom, a tool, like a hammer.. Sadly, when top leaders hears of "Agile" they picture that their coworkers will be more productive hanging upside-down doing their tasks, or some other magical thinking that will somehow "fix IT". Often the worst advice about "Agile" is from the "Agile" consultants themselves, or even if they provide sound advice, it's not heeded or understood.
But Agile Software development solves all these problems...right. (Spoken tounge and cheek from someone who works in an "Agile" shop).
I see most of your points and they seem valid. I guess some CS engineers believe there is always a perfect solution which collides with business realities. Let's rename "projects" simply to "work" and "agile" to "flexible cross team management". As in trading and finance, the IT sector is littered with parasites that feed off the few productive elements and most others just feed off the few. But even the top performers in this industry need to realize most projects, work, specs, are fluid all the time. Requirements change, new opportunities arise, existing doors close due to regulation. Your typical average CS major comes out of school thinking that he will be sat on a team that remains static, and that once project specs are agreed on, nothing would change. Nothing could be further from the truth. I am science trained but very early on got into trading professionally and I understand the desire for beauty and elegance in the scientific field. But we need to accept that projects are just a lose definition and initial pathway to a vision. As the vision takes on more concrete shapes, specifications are changed, adjusted, added upon. We are witness to many well executed projects and many total failures. The level of success comes down to people managing and implementing projects rather than whether the environment is agile or whatever. I am absolutely convinced that we will hysterically laugh about the whole agile hysteria in a few years and embrace a different concept that some crazy professor dreamed up. In the end those are just concepts and nothing beats a dynamic team structure where everyone works towards one common goal. This applies to every industry. It's simply work that needs to get done and the idea, vision, dynamic team structure, sharing and pooling of resources, and experience of management will determine the outcome.
@Zzzz1: You and what real world admin experience? Your C# example is hilarious. And I'll say again: I'm not talking about optimizing down to assembler. I'm talking about thinking ahead to use efficient code and designs from the start. A backtesting framework hosted on a limited server isn't rocket science. From there, y'all can argue endlessly about algorithm efficiencies. Projects that don't think ahead and design a good modular flow will end up poorly written and difficult to maintain. This is why so many programs out there suck. Those projects end up getting rewritten for every major version, and they still suck, getting even more bloated. Those are the black hole money pits. Y'all can throw around any philosophical terms y'all like, but if the project managers and designers are incompetant morons, it's doomed from the start. That's real world experience.
You seem to be a wizard server administrator. (administrator by your own admission, not trying to talk down your skill set). But that does not make every project manager, developer, trader an idiot. So, to answer your question what real world experience I have on the administrative side: none! I only traded professionally at banks and hedge funds for 15 years and consider myself proficient enough in C# to not be afraid to compete with seasoned C++ developers for everything but the most latency sensitive and memory efficient applications. I look at things from an end user perspective and am entirely self taught in anything CS related. So I apologize that I am not measuring up to your admin standard. I hope we find some common ground down the road. Here we need to agree to disagree on this topic, though in your most recent posts you kind of came full circle after you started schooling us about the importance of kernel considerations to get a simple backtest and trading architecture going.
Some thoughts on why its hard to get all things "right": Many business functions and nobody with a complete understanding of all of them. The people creating the software are rarely the same people having to maintain, or God forbid, further develop the software. Ie. instead of actively including admins from the start, throw some build over the silo wall over to operations and then deal with the complaints. Software and requirements are never "done", but ever-evolving beasts. Needs and circumstances change all the time. Time and cost-considerations put limitations on how "good" software is at any given point in time. Software development is nothing like erecting bridges or buildings. Very tempting to get too much into the HOW too early, regardless of which area you work in, instead of sharing the same WHY and WHAT. Though the HOW is important too, ie. if existing software has clearly identified deficiencies you want to mitigate or the product should have certain qualities, ie. around high-availability and responsetimes. Marketing and user interface (UX) seems more valued than properly functioning software "under the hood". You rarely get a chance to "do things right" from the start, as you have to work under various constraints, even as a startup.