Data sets show different data

Discussion in 'Data Sets and Feeds' started by hedron, Nov 8, 2014.

  1. hedron

    hedron

    First, sorry if this makes the width exceed the width of the browser. I don't know how to contain it with img tags.

    Anyway, depending on the source I get different candles. I firmly believe that the ToS candles are the actual correct ones and that the yahoo and google sources are giving me garbage OR amibroker isn't displaying them correctly.

    This is ToS:

    [​IMG]

    This is Amibroker with yahoo data:

    [​IMG]

    Amibroker with google data:

    [​IMG]

    The reason I suspect it might be amibroker displaying it incorrectly is because if you goto google.com/finance you will see that it is the same as the ToS output. Same thing with yahoo. The charts at yahoo.com are the same as ToS. Why would google and yahoo give any other data than what it uses for their own charts on their websites?

    Is there a way to fix this?
     
    Last edited: Nov 8, 2014
  2. jharmon

    jharmon

    Broken record: Garbage in garbage out.
     
  3. hedron

    hedron

    Then how come there isn't garbage being outputted from google.com/finance ?
     
  4. jharmon

    jharmon

    Yahoo and Google do not have quality data. If you want quality (and support) for data you'll have to pay for it.

    Apart from quality issues (they actually don't care whether the data is accurate) you might also find that different data sources are used (consolidated tape, BATS-only or primary-exchange-only trades). Maybe even different order types are considered eligible for each bar. You simply don't know and they keep changing their service and providers they use without notice too.

    If you were a manufacturer, you'd want consistency from your suppliers and the ability to access support on their products.
     
  5. hedron

    hedron

    Then ToS doesn't have quality data either?
     
  6. jharmon

    jharmon

    It's not their core business
     
  7. hedron

    hedron

    I really don't think you're understanding the issue.

    Why are there two outputs to the same data?
     
  8. jharmon

    jharmon

    Clearly it's different data or at least a different interpretation of the data.

    Some vendors (e.g. on a 5 minute chart) put the 16:00 bar as all trades as 15:55:00.001 through 16:00:00.000
    Other vendors put the 16:00:00 bar as 15:55:00 thorugh 15:59:59.999

    I prefer the former as the logic works at all timeframes (tick, minute, hourly, daily)

    Without any clear documentation or support channel available you just never know what they are feeding you.
     
  9. M.ST.

    M.ST.


    All mumbo jumbo. There is nothing incorrect. Just as jharmon has said garbage in, garbage out.

    Amiquote 3.10 downloads from Google finance intraday data server
    Sample link using RHT
    http://www.google.com/finance/getprices?q=RHT&x=NYSE&i=300&p=2d&f=d,o,h,l,c,v

    300 -> means 300 / 60 = 5 (min).
    2d means recent two days of data.
    So since today is Tuesday Nov 11th and US markets are still closed today the upper link downloads data from Fri Nov 7th to Mon Nov 10th.

    So Nov 7th data of upper link used today is the following one.
    If you wanna use that link tomorrow downloading data till Nov 7th then you need to input 3d instead of 2d and so on. Maximum is 60 business days (12 weeks, 3 months) back to get 5-minute data from Google.

    Code:
    EXCHANGE%3DNYSE
    MARKET_OPEN_MINUTE=570
    MARKET_CLOSE_MINUTE=960
    INTERVAL=300
    COLUMNS=DATE,CLOSE,HIGH,LOW,OPEN,VOLUME
    DATA=
    TIMEZONE_OFFSET=-300
    a1415370900,60.7,60.84,60.6,60.84,18423
    1,60.81,60.81,60.59,60.66,10616
    2,60.94,60.95,60.63,60.74,29125
    3,60.87,60.94,60.65,60.94,9498
    4,60.84,60.96,60.78,60.82,9610
    5,60.7,60.84,60.68,60.84,5464
    6,60.56,60.71,60.535,60.66,13900
    7,60.59,60.6,60.52,60.53,8000
    8,60.72,60.72,60.61,60.61,16787
    9,60.83,60.83,60.6501,60.68,10899
    10,60.815,60.82,60.73,60.82,8123
    11,60.75,60.81,60.75,60.77,4241
    12,60.69,60.7338,60.68,60.73,5829
    13,60.73,60.76,60.66,60.67,8094
    14,60.725,60.74,60.71,60.72,6992
    15,60.78,60.79,60.71,60.71,6697
    16,60.73,60.765,60.72,60.765,9100
    17,60.78,60.82,60.75,60.75,6838
    18,60.76,60.8,60.72,60.77,5500
    19,60.735,60.76,60.72,60.76,5500
    20,60.815,60.82,60.72,60.74,6255
    21,60.87,60.88,60.78,60.82,8588
    22,60.88,60.89,60.83,60.884,7986
    23,60.87,60.9,60.87,60.89,3400
    24,60.82,60.93,60.8,60.88,9400
    25,60.78,60.8,60.705,60.8,10100
    26,60.7,60.78,60.69,60.77,13250
    27,60.74,60.74,60.69,60.69,3900
    28,60.74,60.76,60.68,60.745,24510
    29,60.81,60.83,60.745,60.745,6109
    30,60.88,60.91,60.815,60.815,6751
    31,60.845,60.87,60.81,60.87,7663
    32,60.66,60.84,60.66,60.84,13289
    33,60.78,60.78,60.66,60.66,7810
    34,60.85,60.8736,60.78,60.78,8335
    35,60.8,60.85,60.79,60.85,6210
    36,60.835,60.8504,60.8,60.81,5266
    37,60.88,60.9,60.85,60.85,6800
    38,60.865,60.875,60.845,60.87,4947
    39,60.93,60.93,60.87,60.87,6828
    40,60.97,60.98,60.91,60.93,9109
    41,61.04,61.04,60.98,60.98,22262
    42,61.07,61.08,61.04,61.05,8781
    43,61.07,61.1,61.06,61.07,13765
    44,61.105,61.11,61.065,61.07,5972
    45,61.1899,61.19,61.11,61.11,6672
    46,61.15,61.19,61.14,61.18,9610
    47,61.19,61.22,61.13,61.16,43486
    48,61.25,61.26,61.19,61.195,7689
    49,61.24,61.25,61.23,61.25,6082
    50,61.205,61.24,61.17,61.24,5400
    51,61.19,61.21,61.14,61.2,5610
    52,61.23,61.23,61.16,61.18,6713
    53,61.24,61.24,61.1901,61.22,6208
    54,61.28,61.3,61.24,61.24,14179
    55,61.275,61.29,61.27,61.29,4771
    56,61.235,61.27,61.23,61.27,5711
    57,61.27,61.27,61.22,61.23,11149
    58,61.29,61.29,61.245,61.27,9020
    59,61.2368,61.29,61.2,61.29,12986
    60,61.19,61.24,61.19,61.23,11167
    61,61.14,61.18,61.1,61.18,11000
    62,61.165,61.19,61.15,61.15,7085
    63,61.1999,61.1999,61.16,61.16,10304
    64,61.11,61.19,61.1,61.19,9253
    65,61.11,61.11,61.05,61.105,8421
    66,61.2,61.2,61.095,61.095,13123
    67,61.23,61.23,61.19,61.2,11567
    68,61.19,61.23,61.16,61.23,17010
    69,61.15,61.2,61.12,61.2,19596
    70,61.18,61.19,61.12,61.15,20143
    71,61.15,61.2,61.14,61.185,21864
    72,61.19,61.19,61.13,61.15,23201
    73,61.1201,61.2,61.12,61.19,32782
    74,61.11,61.15,61.09,61.12,36196
    75,61.14,61.17,61.11,61.11,47670
    76,61.16,61.19,61.12,61.14,49427
    77,61.28,61.29,61.16,61.16,143895
    The price data is CHLO

    Now do yourself a favor and chart that one in Excel.
    You will get this display of symbol RHT

    [​IMG]


    If you download two days of 5-minute data via Amiquote 3.10 then you will get the same chart display of Nov 7th google data in AmiBroker 5.80.
    Symbol RTH in AmiBroker

    [​IMG]



    Same input, same output from same source.
     
    Last edited: Nov 11, 2014
  10. M.ST.

    M.ST.

    If you download Google 1-minute data instead of 5-minute data then go to formats folder of AmiBroker directory and open aqgi.format file with notepad or any other text editor and add line

    $TIMESHIFT -0.01666666667

    at the bottom.
     
    #10     Nov 11, 2014
    hedron likes this.