HOME FORUMS BROKERS SOFTWARE BOOKS CONTACT US
Elite Trader Your Account  •  Become a Member  •  Help  •  Search    
    Forums ›› Technically Speaking ›› Programming ›› database selection......  


Post A Reply
    Page 2 of 2:   1   2  
Jay_Ap
 

Registered: Jun 2012
Posts: 29

 

08-03-12 10:54 AM


Quote from cdcaveman:

ok .. there is some good to Mysql because any web application will already come prebuilt with a mysql db.. meaning i wouldn't have to add a datasource and connection to the application to query the inserted options data... its just automatically available.. what do you mean by admin headache? i wouldn't be hosting the db myself.. it would be on a hosting account.



I highly recommend Mysql as it is easy to work with, stable, and available on just about all hosted environments. It even has a pretty decent query browser tool that allows you to query data and view and modify it in an Excel like grid. You can also dump the queried data out as csv files.

One word of caution, you need to really think about table structure and the storage engine you will choose (i.e. innodb, myisam, etc). This will depend on how much data you are going to suck into the database.

I use Mysql with my production trading system and it works remarkably well.

    Edit/Delete Quote Complain
cdcaveman
 

Registered: Aug 2011
Posts: 3514

 

08-03-12 09:11 PM

thats cool.. i use innob with a ecommerce site i own... its funny how you learn things.. i didn't know what engine i was using untill my hosting account told me they don't support innob and that they don't back them up.. haha but you know you learn things when you have to actually fix things instead of reverting back to old instances i guess i have messed around with mysql quite a bit.. Zend framework php etc.. started out in coldfusion etc.. now is there anything in the data that could use the benefit of relational tables? i mean theoretcially a p and c for calls takes up more space then a boolean value.. did you consider all that? sort of doesn't make sense cause that seems to be the only place you could crunch things.. i was just thinking of increasing the specificity of what the db gets so that it can not just grow as quickly as it would in just one table.. i'm imagining you guys are just dumping it into one table right? its only when you get odd sales and errors on data that you need some cleansing which is what the bought data from like tick data is suppose to have done to it.. I just can't see buying data that my broker already gives to me.. its ALOT of money considering the amount of money i'm trading to buy a few years of a few symbols of options chains

    Edit/Delete Quote Complain
Jay_Ap
 

Registered: Jun 2012
Posts: 29

 

08-03-12 10:29 PM


Quote from cdcaveman:

thats cool.. i use innob with a ecommerce site i own... its funny how you learn things.. i didn't know what engine i was using untill my hosting account told me they don't support innob and that they don't back them up.. haha but you know you learn things when you have to actually fix things instead of reverting back to old instances i guess i have messed around with mysql quite a bit.. Zend framework php etc.. started out in coldfusion etc.. now is there anything in the data that could use the benefit of relational tables? i mean theoretcially a p and c for calls takes up more space then a boolean value.. did you consider all that? sort of doesn't make sense cause that seems to be the only place you could crunch things.. i was just thinking of increasing the specificity of what the db gets so that it can not just grow as quickly as it would in just one table.. i'm imagining you guys are just dumping it into one table right? its only when you get odd sales and errors on data that you need some cleansing which is what the bought data from like tick data is suppose to have done to it.. I just can't see buying data that my broker already gives to me.. its ALOT of money considering the amount of money i'm trading to buy a few years of a few symbols of options chains



I would recommend splitting the data across a few relational tables just so it doesn't grow super huge. So, descriptive info like ticker should be put in a separate table with a autoincrementing primary key. This key number is then used as the foreign key ID in the table holding the price time series data.

Sometimes it's a tradeoff between splitting the data too much (i.e. normalizing) such that it's a pain in the ass to query across tables, versus just dumping all the data in one table so it's easy to work with. If you just want to do ad-hoc analysis, you may just want to dump it all in one table. If you want to do something more systematic, I'd go with normalizing the data.

    Edit/Delete Quote Complain
cdcaveman
 

Registered: Aug 2011
Posts: 3514

 

08-03-12 11:26 PM

i completely understand foreign keys and their related complexities in quering db's. ... its just one of those things i've learned.. you say. . "oh no big deal for now" and then later you find out you should have jumped the complexity hurdle early on i haven't even started doing this... i just was reading some of Augen's books and messing around with excel and the sheet twsdde from interactive brokers... and a hoadley finance add in. .. do you clean the data at all? for example i traded a strangle on weekly opex for aapl and then i reversed into a bull put credit spread after for a decent rip on both... and i noticed that the otm 515 put that expired worthless posted a. 92 cent offer at expiration when it had been trading less then half that for the last hour of trading... i'm sure you have found some not so realevant data that you trashed or at least extracted to examine for reasons later... either way i appreciate your feedback.. i know our doing this at a much higer level then i am... so your feed backc helps

    Edit/Delete Quote Complain
cdcaveman
 

Registered: Aug 2011
Posts: 3514

 

08-20-12 10:03 PM


Quote from RPEX:

Hi this is a really interesting thread, i was considering doing something similar with futures options closing prices. But i don't have that much experience with mysql, and was looking for some pointers on table structure for options data. I want to analyse the data both in time series [say a calculation for ATM IV] and cross sectional [vol surface].

Would you go as far as having a separate tables for each date?

How do you treat the underlying. Obviously no option price really means anything without an underlying price. But i wouldn't want to keep a field just for that - surely that's what the relational model seeks to avoid.

Absolutely any ideas / ways of thinking would be great.



This is the way i think about it alot of times... Do i blog about my trades.. And potientially give away my edge.. Or become some chump fuck that only makes money telling other peole how to trade... If these guys really had good great trading methodologies.. They sure as fuck woulldnt be selling it.. Ok ok ok back to the subject at hand
The fact that your asking about table structure in the manner that you are tells me you know nothing about how databases work... Which is no big deal the majority of things in life i feel are made out to be more complex then they are so people feel good about knowing something you don't... Now.. I would sugguest picking up a mysql book.. And just rippping through it cover to cover.. Its stupid simple.. First you learn update delete select etc.. Then you learn about inner joins, joins etc.. Then you learn about indexing quieries optimization triggers. . Etc.. But the whole idea here is to reduce redundent data.. You don't wanna duplicate entries down an entire table... For example puts & calls could be reduced to even a booleon 1 or 0 value if yo uwanted... As the more data you put in there the more exponeitailly inflated your db can get... Think about tick data on every strike on every futures contract every day.... Shit adds up.. Now.. That being said.. If your ripping down prices, time and sales, open interest, volume etc.. There isn't much gonna be duplicated... Thats why i thought basically you would have only one table.. Looking like an excel spreadsheet.. Makes for super easy queries to.. so. . Do you need a seperate table for every strike price? i'm not sure.. I'm working on my ecommerce store right now and i lost track of what i was thinking on this.. I'll obvsiouly have to sit down and start figuring this out.. Cause even if i don't publish it to a website.. I'm definilty gonna start ripping down options data. . Cause you just can't get it once it expires without paying for it.. Especially not in detail.. Obviously the more tables the more complex the quieries. . I bet you someone wrote some stuff about this somewhere.. Or we could post in some db forums and get some advice on it.. Cause i know that it always ends up being alot more data then you think... Then you have to figure if you rip down data from your desktop connection to your IB connection.. If you lose your internet. Or you ccomputer crashes your gonna have a huge gap... I'm sure we could hook into the api from a server that has crazy better uptime and rip it down to a db their.. But you literally would have to have root access to the server so you could install trader work station and hhook into it..

    Edit/Delete Quote Complain
cdcaveman
 

Registered: Aug 2011
Posts: 3514

 

08-20-12 10:39 PM


Quote from RPEX:

What are you on drugs?



Actually i'm sober a really long time in AA haha how ironic..

    Edit/Delete Quote Complain
    Page 2 of 2:   1   2  
Post A Reply


Receive an email whenever a new post is added to this thread by subscribing to it.
 
Rate This Thread:

Forum Jump:
 

 

   Conduct Rules  -  Privacy Policy  -  Day Trader -  Day Trader Forum -  Best Trading Software -  Sitemap Copyright © 2013, Elite Trader. All rights reserved.    
 
WHILE YOU'RE HERE, TAKE A MINUTE TO VISIT SOME OF OUR SPONSORS:
Advantage Futures
Futures Brokerage & Clearing
AMP Global Clearing
Futures and FX Trading
Bright Trading
Professional Equities Trading
CTS
Futures Trading Software
DaytradingBias.com
Professional Trading Analytics
ECHOtrade
Professional Trading Firm
eSignal
Trading Software Provider
FXCM
Forex Trading Services
Global Futures
Futures, Options & FX Trading
Interactive Brokers
Pro Gateway to World Markets
JC Trading Group
Direct Access Trading
MB Trading
Direct Access Trading
MultiCharts
Trading Software Provider
NinjaTrader
Trading Software Provider
OANDA
Currency Trading
optionshouse
Option Trading & Education
Rithmic
Futures Trade Execution Platform
SpeedTrader
Direct Access Trading
SpreadProfessor
Spread Trading Instruction
thinkorswim by TD Ameritrade
Direct Access TradingAdvertisement
TradersStudio
System Building & Backtesting
Trading Technologies
Trading Software Provider
Trend Following
Trading Systems Provider