HOME FORUMS BROKERS SOFTWARE BOOKS CONTACT US
Elite Trader Your Account  •  Become a Member  •  Help  •  Search    
    Forums ›› Technically Speaking ›› Programming ›› Data storage for backtesting  


Post A Reply
    Page 4 of 4:   1  2  3   4  
PocketChange
 

Registered: Jul 2008
Posts: 2037

 

08-31-12 10:03 PM

This is one of those exception to the "rules" cases. The views are simple and result columns are all contained in the index... just the group by (1 min, 5 min, 10min etc) clause changes.

When doing an indexed lookup of a row, the usual procedure is to do a binary search on the index to find the index entry, then extract the rowid from the index and use that rowid to do a binary search on the original table. Thus a typical indexed lookup involves two binary searches.

If, however, all columns that were to be fetched from the table are already available in the index itself, SQL will use the values contained in the index and will never look up the original table row. This saves one binary search for each row and can make many queries run twice as fast.

For this specific case calling a 1 min view, 2 min view, 5 min view etc performance is already optimized with one table simplicity. K.I.S.S.




Quote from Kevin Schmit:

Especially on MySQL, which came by views lately, and still doesn't do them too well. Even worse when you have views referencing views. Use "EXPLAIN" to see what is going on with them, you will be shocked at the inefficiency of what is going on under the hood, at least compared to a highly optimized commercial database.

    Edit/Delete Quote Complain
cdcaveman
 

Registered: Aug 2011
Posts: 3517

 

08-31-12 11:57 PM

to much information can be poison... the guy wants only 1 min data to serve to a website... why does everything have to be brought out to an exponent of 10 in complexity.. no need.

    Edit/Delete Quote Complain
Kevin Schmit
 

Registered: Nov 2005
Posts: 568

 

09-01-12 04:39 AM


Quote from PocketChange:

This is one of those exception to the "rules" cases. The views are simple and result columns are all contained in the index...... all columns that were to be fetched from the table are already available in the index itself, SQL will use the values contained in the index and will never look up the original table row. This saves one binary search for each row and can make many queries run twice as fast.


You're right, thanks for pointing this out. I should look at what is being done before regurgitating my standard opinion of MySQL views.

    Edit/Delete Quote Complain
sle
 

Registered: Apr 2003
Posts: 1609

 

09-01-12 05:00 PM

I hear stories that Infobright is the way to go for dealing with financial data on the cheap. It is a column-oriented database, with a scalable cost solution ( "Community" edition is free, "Enterprize" edition is not). As a bonus, all of the tools for MySQL work with it and you don't have to learn Q to do queries. Also, for me, there is an added benefit vs KDB of being able to store binary objects (vol surfaces).

Does anyone have any experience or at least heard more concrete stories about this product? Any color would be helpful.

    Edit/Delete Quote Complain
    Page 4 of 4:   1  2  3   4  
Post A Reply


Receive an email whenever a new post is added to this thread by subscribing to it.
 
Rate This Thread:

Forum Jump:
 

 

   Conduct Rules  -  Privacy Policy  -  Day Trader -  Day Trader Forum -  Best Trading Software -  Sitemap Copyright © 2013, Elite Trader. All rights reserved.    
 
WHILE YOU'RE HERE, TAKE A MINUTE TO VISIT SOME OF OUR SPONSORS:
Advantage Futures
Futures Brokerage & Clearing
AMP Global Clearing
Futures and FX Trading
Bright Trading
Professional Equities Trading
CTS
Futures Trading Software
DaytradingBias.com
Professional Trading Analytics
ECHOtrade
Professional Trading Firm
eSignal
Trading Software Provider
FXCM
Forex Trading Services
Global Futures
Futures, Options & FX Trading
Interactive Brokers
Pro Gateway to World Markets
JC Trading Group
Direct Access Trading
MB Trading
Direct Access Trading
MultiCharts
Trading Software Provider
NinjaTrader
Trading Software Provider
OANDA
Currency Trading
optionshouse
Option Trading & Education
Rithmic
Futures Trade Execution Platform
SpeedTrader
Direct Access Trading
SpreadProfessor
Spread Trading Instruction
thinkorswim by TD Ameritrade
Direct Access TradingAdvertisement
TradersStudio
System Building & Backtesting
Trading Technologies
Trading Software Provider
Trend Following
Trading Systems Provider