Can anybody drop a brief outline of what I'll need to accomplish this. What are the easiest languages/programs to use?
maybe if you gave a little more detail as to where your pulling data from.. and what your going to be using the data with..
FX exchange prices provided by my ECN. I want to execute simple technical backtests over different time periods. Eventually I want to use DBs for bigger projects.
Look up ETL. Something like Microsoft SSIS is probably what you are looking for. Not sure if there is an open source alternative. It's a 4GL tool that allows you to visually program your data load. 4GL simply means its graphically. I really think tools like SSIS and Ab Initio are a big improvement over basic bcp.
Does anyone have a book written that specifically covers continuously writing to a database from a feed?
Take a look at SQLite >>> sqlite.org They provide c libraries so you can stream and save tick data into standalone sql databases. Clean - Fast and Compact: No server required. You are able to serve these database files using SQLabs - CubeSQL and feed virtually any system. If your excel centric you may want to take a look at RICHClient put out by Olaf Schmidt and available at thecommon.net. One time pain setting it up but it has served us well for many years now.
Are you getting data at tick resolution or snapshots? So suppose you're getting something like 500ms snapshots. Just write up a handler which parses your feed into a sql statement with the data embedded, then pass it from the handler into a queue. Then just allow a thread to access that queue and process it.