How do you guys organize code that you routinely use? As a non-native programmer, I'm slow creating/finding useable code so I rely mainly on existing code. I then have notes on the code, how to use it etc. and when i need to use it i have to look in my notes to remember everything. But, I feel there is a better method out there for quickly implementing code. For example I would like to have a selection process for data, data type, then methods to apply hypothesis testing/transformations/data building to choose from. Right now I'm investigating existing software and programming tools to see if I can find a balance between flexibility and user interface. I use R quite a bit, so im also checking out some of the gui's that are available.I have feelng Visual Studio or something similar could help, but I couldn't find a dumbed down answer.
Honest suggestion would be to take a programming course. Nothing too serious, but enough to no longer have to rely on tools, or on other peoples code. Once you can write your own code, you can do anything. Just comes down to time, and motivation. Build you own code base that you fully understand -- then you're unstoppable.
"How do you guys organize code that you routinely use?" and for that, programmers use "Snippet Managers". It's a common practice of course. No one can remember the thousands or millions of lines of codes that are routinely used. Edit: I should clarify that snippet managers are designed for programmers to store *their own* source code.. not to cheat!
Simple method that works for codebases of up to several MB (perhaps 100-200 code snippets max, even more if you are very disciplined): Store all the code snippets in one single textfile. Put tags in front of the snippets that describe it. When snippet needed: Search for appropriate tags with text editor of your choice. Paste and copy.
I essentially do this with python but with a little more structure. I have several base classes with re-usable code snippets defined as methods. I derive new classes from there. It is very little overhead compared to independent programs but it is a little more work. I used to do all my analysis exclusively in R, and when I ran into this problem of hundreds of scripts everywhere with a lot of repeated functions inside, I considered building an R package but it seemed like a lot of overhead and a learning curve, and since I knew python already, I went with python (+matplotlib). But I still miss the easy plotting and stats functions in R, and I often create text output from my python code to view and analyze my results in R. I have to admit, I still haven't found the optimal solution.
By the way, I also know of this package: http://www.knime.org/ I am thinking that may be I can use this to organize my code. This might be a lot of work as I probably need to define a constant interface between data readers, analyzers, and output modules but the beauty of this is once I create such a convention, then putting together back tests and little experiments and statistical things might be a snap (literally just snapping together the components into a pipeline). It will take some real work / time to get a critical mass of components together though. But I am thinking it might be worth it eventually.
Another question is: what happens when things go wrong? Your computer dies, or you accidentally overwrite some code and now it doesn't work? A useful thing to use is something called "version control" like Github (have to pay for private code hosting but has a better system) or BitBucket (free private code hosting) which will not only back up your files, but also keep track of the changes you've made to them so that if you mess something up you can go back to an older version. I'd say using a single text file is not the best idea, these can get unwieldy very quickly when you're dealing with thousands of lines of code. Since you're mainly dealing with simple snippets, I'd say learning to program is a great idea. You're not going to building the code equivalent of skyscrapers or space shuttles, so a single course in programming is easily enough for you to be able to understand and even modify some of the snippets to get them to do what you want them to do. On top of that, you'll be able to even write your own snippets in the case that you don't have a snippet that does what you want!