Live Streaming Historical Data

Discussion in 'Automated Trading' started by wackytacky, Nov 28, 2016.

  1. Okay. So you meant is I use some kind of scheduler that runs at the end of time interval and it appends the new data to an already existing data?

    secondly,
    Where do I store this data?
    Memory or some database?
     
    #11     Nov 30, 2016
  2. MarcG

    MarcG

    Depends if you want the bar to be closed on time or existing ticks.
    On existing ticks you can just use sth like below. Whereas I prefer to have the current bar also already in the existing array/data structure and not separate it

    Depends. Whether you want to request the data over and over from your provider or have it on your system.
    I store data in plain files because I can make better use of resources.

    1 Minute Example in Golang, pseudo, without consideration if tick data might cross with historical data:

    type Tick struct {
    Time int64
    C int64
    V int64
    }

    type OHLCVBar struct {
    Tick
    O int64
    H int64
    L int64
    }

    type Ctx struct {
    Bars []OHLCVBar
    BarCur OHLCVBar
    LenBarTime int64
    }

    func (self *Ctx)OnRTData(tick *Tick) {
    isNextBar := OHLCVTimeBasedAddTick(&self.BarCur, tick)
    if isNextBar {
    self.Bars = append(self.Bars, self.BarCur)

    StoreBarToDB(self.BarCur)

    barNext := OHLCVBar{}
    CalculateNextBarTime(&barNext, LenBarTime)
    barNext.O = tick.C
    barNext.H = tick.C
    barNext.L = tick.C
    barNext.C = tick.C
    barNext.V = tick.V
    self.BarCur = barNext
    }
    }

    // assumes that OHLCVBar Time is on end of period
    func OHLCVTimeBasedAddTick(barCur *OHLCVBar, tick *Tick) (isNextBar bool) {
    if tick.Time >= barCur.Time {
    return true
    }

    barCur.C = tick.C
    barCur.V += tick.V

    barCur.H = MAX(barCur.H, tick.C)
    barCur.L = MIN(barCur.L, tick.C)

    return false
    }
     
    #12     Nov 30, 2016
  3. JackRab

    JackRab

    So, you don't know anything about coding? You'd better read some books before you go live mate....
     
    #13     Nov 30, 2016
    CBC likes this.
  4. JackRab

    JackRab

    @MarcG
    Now, I'm not trying to be a dick... but if @wackytacky doesn't now proper coding maybe don't give him code at all.
    Because, if there's a mistake in it or it doesn't work with the rest of whatever he's got it could end up in a drama.

    (PS. I don't know coding either, but I do know that things can easily be overlooked if you don't know what you're doing)
     
    #14     Nov 30, 2016
  5. MarcG

    MarcG

    Just a little help for him and to be honest, I don't want to be impolite but I really don't care if his account gets killed with my code or not. One important part of software development / engineering is testing. If I hadn't tested my code which integrated code from others in the past ... man I would have lost my job and my account a 1000 times.
    Sadly, when it comes to money or software people often act brainless.
    So ultimately, it is his responsibility ... if he wants me to give some ideas or have a short look at his final code before going real, no problem but the rest is up to him.
     
    #15     Dec 1, 2016
    JackRab likes this.
  6. Offcourse it is my responsibility to review source code before going live.
     
    #16     Dec 1, 2016
  7. Code:
    import csv
    from kiteconnect import WebSocket
    api=' '
    public=' '
    userid=' '
    
    kws=WebSocket(api,public,userid)
    
    def on_tick(tick,ws):
         crudeoilm = open('crudeoilm.csv', 'a')
         crudeoilWriter = csv.writer(crudeoilm)
         crudeoilWriter.writerow([tick])
        
    def on_connect(ws):
        ws.subscribe([53315591]) # Instrument token for Crudeoil Contract Dec-2016
        ws.set_mode(ws.MODE_LTP,[53315591])
      
    kws.on_tick=on_tick
    kws.on_connect=on_connect
    
    kws.connect()

    The above is the code I used to fetch Tick data and send it to CSV file, Will it work?
    The attached file is the output of CSV file.
    The thing is the Json data is pasted to one particular cell in CSV, rather than having Keys and values of it in different cell. How do I manipulate that?
    Also how do I use this streaming live data and make an OHLC 5min data out of it?

    This is the link to docs-
    https://kite.trade//docs/pykiteconnect/#kiteconnect.WebSocket
     
    #17     Dec 1, 2016
  8. MarcG

    MarcG

    Again: Better you start with basics of programming.
    There are many very good python tutorials out there.
    You get a JSON string representing a tick via your Websocket. JSON is not comma separated and it makes no sense to split it in some way here.
    Neither it makes sense to open a file and a writer on it each time a tick comes in.
     
    #18     Dec 1, 2016
    wackytacky likes this.
  9. Any tutorials you would like to suggest that are particularly for 'building automated trading systems in python'
     
    #19     Dec 1, 2016