Anyone using Python on a large scale trading application?

Discussion in 'App Development' started by chromosome, Dec 11, 2011.

  1. Hi Christophe,

    Sorry for the late response, have been very busy lately cranking up the non-profitable algos an masse...
    I have prototyped only some stuff with ChartDirector and do not have nice working example, but will try to help with some snippets.

    That's right, the whole image is created at once, and it's really not the environment to deal with the interactive, real-time charts in common way. It is definitely geared toward the offline chart handling. Having said that, I am very impressed with the library speed and it is really coming very close to be a non-issue. One thing, you have to handle the image in memory, without saving it to file. It took me several attempts to figure out how to process the output from ChartDir in wxPython.
    Code:
    stream = cStringIO.StringIO( c1.makeChart2(0) )
    img = wx.ImageFromStream( stream )
    stream.close()
    self.canvas.bitmap = wx.BitmapFromImage( img )
    
    If you are using the fixed y_scale, then you already know your boundaries. X_scale is your time line (i.e. number of data points) and Y_scale is your price range. If you are using the auto scale, you have to pack the plot first to get the Y_scale min/max.
    Code:
            #--- pack plot area so we can put some custom lines
            c1.packPlotArea( x1, y1, w, h )
            y_axis_min = c1.yAxis().getMinValue()
            y_axis_max = c1.yAxis().getMaxValue()
    
    Then you get the coords of your trend line, add line to the chart and adjust z-order if needed.
    Code:
    x1_coor = c1.getXCoor( trendline_start_datapoint )
    x2_coor = c1.getXCoor( trendline_end_datapoint )
    y1_coor = c1.getYCoor( trendline_start_pricepoint )
    y2_coor = c1.getYCoor( trendline_end_pricepoint )
    cp_line = c1.addLine( x1_coor, y1_coor, x2_coor, y2_coor, c1.dashLineColor(0x40C020,pychartdir.DotLine), 2 )
    cp_line.setZOrder( pychartdir.PlotAreaZ )
    
    When it comes to interaction, that's where it becomes awkward. I got something working by using the <getHTMLImageMap> function. It is meant for the web app, and the output is the html string, which gives you the info about all image objects, so you can have a clickable charts (there is a working example in docs). However, in wxPython you have to parse the output and put the info in dictionary and process it through the mouse and graphics events. Inside the Frame where you render the image, you have to do this:
    Code:
            stream = cStringIO.StringIO( c1.makeChart2(0) )
            #--------------------------------------------------------------------------------------------#
            # create image maps
            image_map = c1.getHTMLImageMap('a','x={x} T={xLabel} H={high} L={low} C={close}')
            #<area shape="rect" coords="889,379,895,398" href="a?x=99 T=01:00 H=99232 L=99147 C=99177">
            y_axis_map = c1.yAxis().getHTMLImageMap('b','{value}')
            #<area shape="rect" coords="901,490,935,505" href="b?P=98700">
            #<area shape="rect" coords="901,467,935,482" href="b?P=98800">
            #<area shape="rect" coords="901,445,935,460" href="b?P=98900">
            self.chart_data_info = []   #[ ((x1,y1,x2,y2),'x=99 T=01:00 H=99232 L=99147 C=99177'),...]
            s = image_map.split('">')
            for line in s:
                a = line.partition('coords="')
                b = a[2].partition('" href="a?')
                if b[0].strip() == '': continue
                self.chart_data_info.append( (tuple(b[0].split(',')), b[2]) )
    
            self.y_axis_info = []   #[ ((x1,y1,x2,y2),'98700'),...]
            s = y_axis_map.split('">')
            for line in s:
                a = line.partition('coords="')
                b = a[2].partition('" href="b?')
                if b[0].strip() == '': continue
                self.y_axis_info.append( (tuple(b[0].split(',')), int(float(b[2]))) )
            #--------------------------------------------------------------------------------------------#
            img = wx.ImageFromStream( stream )
            stream.close()
            self.canvas.bitmap = wx.BitmapFromImage( img )
            self.Refresh()
            self.canvas.bitmap_status = 'DONE'
    
    This sample gets the info about the HLC bars, and the comment lines are output sample that is parsed. You have to read the docs and understand the <getHTMLImageMap> function and the output.
    I am attaching the ImageCanvas class I have wrote, it is rather generic and you have to assign the instance to the canvas of your wxFrame. Inside the ImageCanvas, the on_mouse_motion() event is processing the info dictionary from the above as well as cross-hair, scale label, repainting, etc. By setting the status of <bitmap_status> var in your wxFrame, you are controlling the kickoff of the rendering logic, when graphic context is idle.
    The whole thing will give you back the info about your trend line and you could simply show a dialog for the input of the new coords, or if you want mouse action, play with the mouse events and get the new coord via clicks. Lastly, you have to render the image again.

    If you want to highlight the bar, you would have to have a "shadow" data series, with single bar in it, the one that you would get from the above described procedure. You would be rendering the image every time when mouse gives you the new location. Although I believe that, probably, you are not going to see any issues, it might start showing some flickering under the heavy load.
    Instead of "highlighting" I was simply showing the bar info in the info box. That would be fast and efficient way for you comments as well. You put the info/comments in dictionary and do this:
    Code:
            #--- add some text in upper corner (e.g. legend) It is a dictionary {'title':'text',...}
            if self.legend:
                h_offset = 100; v_offset = 15; yy1 = y1 + v_offset
                for legend_title,legend_txt in self.legend.items():
                    c1.addText( x1,         yy1, legend_title, font, 10, 0x838d74 )
                    c1.addText( x1+h_offset,yy1, legend_txt,   font, 10, 0x838d74 )
                    yy1 += v_offset
    
    You can make your info/comments persistent in many ways (pickle, file, dbs, etc).

    Once again, it starts getting awkward at some point, but considering how quickly you can build variety of charts and being very fast, the library is great tool and can take you long way before you hit the wall with some super-short refresh time requirement.

    Hope this help.
    Dr. Sheldon Cooper
     
    #31     Dec 26, 2011
  2. Thank you for finding it interesting. In a certain way, the issue of performance is a bit too overrated nowadays. 20+ years ago you had to pay much more attention, while today's hardware is heavily compensating for deficiencies of the modern software. You still get 100% difference between the tool A and tool B, however while ago that was 1/2 day vs whole day of processing and today that's 5 min vs 10 min. Of course there are specific issues, and the key is to pick your battles and come equipped with the right tools.

    For most of the time, I have to feel comfortable with programming environment, I like when I enjoy it, when it is logical and has the common sense in the structure, so when I need something new I was never using before, I find it easily. Because it is in logical place. To me. I am referring to Python, and apparently bunch of people feels the same way. I guess not too many, there is another thread on this forum about the language of the future and Python is not even on the list. Go figure... I have started many years ago with COBOL and it's been a while since I was enjoying programming as I am now with Python.

    As far as .Net and other proprietary stuff (Java too, it's free to use, but it's not open source), my main issue is corporate control over the OS and development tools. We could have had order of magnitude better, more stable, more sophisticated operating systems and development tools many years ago. And the things are not getting better, bigApple takes one of the best OS's ever, make some small changes (usually not for the better) just so they can stamp the name and patent. Corporate environment inherently produces inferior results in software arena, period. I've been living that misery for 25 years and haven't seen a single exception. When up to 5 people are in the business, the code is still reasonably good, when it gets big or purchased buy the big co, nobody gives a sh**t any more and it just morphs into a big blob of code that nobody wants to touch any more. Then it comes the business cycle and you get the "new release", and on and on... And the bs about the support, well tell me how many times you got help from the knowledgeable MS tech support vs from somebody from the user community. I've been seeing loads of money paid for tech support and I never, ever got any help or resolution when I needed it.
    On the other side, you have to be a real trail blazer to encounter an issue that is not already commented and documented somewhere in the open source community, and even then there is always somebody (world is a large place) who will try to help for the very simple reason: people love good stuff, love to work with good stuff and they DO CARE... simple like that.

    And if you feel comfortable with .Net (or whatever else), that's perfectly fine, but I am pretty sure sooner or later you will be disappointed with new release, missing features or direction where it's going, just because some corporate schmuck who never did more then 10 lines of code in his/her life decided something is not cool anymore. Not saying that open source projects can't go wrong way or different way, but you and bunch of other people can (and will) fork it and continue to make it better, while on the other side if you decide to make your better version of .Net, MS lawyers will remind you of copyright infringement (as per them, MS invented bits and bytes) and you should be happy with what you get, and don't forget your monthly payment for tech support and don't forget to upgrade at reasonable price when we give you the same sh**t in a new box....

    Peace out,
    Dr. Sheldon Cooper
     
    #32     Dec 26, 2011
  3. monti1a

    monti1a

    Damn 6,

    you know your shit! wish we had more people like you around :)
     
    #33     Dec 26, 2011
  4. rosy2

    rosy2

    struct.unpack() ?
     
    #34     Dec 26, 2011
  5. unpack what? nothing is packed to start with...
     
    #35     Dec 26, 2011
  6. rosy2

    rosy2

    you can use unpack for text. you dont need to loop over the url to parse it.
     
    #36     Dec 27, 2011
  7. It wouldn't really help much... It is not looping over the url, it's the lines created with split.
    As I said, it is quite awkward thing, because the library does not provide any other way to get the info about the objects on the chart.
    The function <getHTMLImageMap> (as the name suggests) provides the string with html tags that you could embed in the web page and create clickable chart. Those tags carry info about the object (coords and other stuff, it is relatively configurable as far as I remember) and that gives you the possibility to display object info on the web page.
    My environment was not browser and the html formatted tags were not helping (quite opposite). I needed to extract the coords and bar info, so I could handle it in mouse event.
    The output of the function is string with tags of (relatively) unknown length. I could figure out the number of objects in the current rendering snapshot, but that would definitely make it beyond awkward complicated. As a result, I can't specify format mask for the unpack, let alone that it would be veeeery looong (actually curious how long mask/string unpack can efficiently handle?). So I split it in lines using end of tag (">) as delimiter. Then it loops over those lines (single tags) and parse them. At that time I could have used unpack, however the performance was not the issue (assuming the unpack would be faster), and splitting/partitioning is a bit more flexible, meaning if something changes in the func output spec it has greater chances to survive vs more rigid unpack format mask. Radical change would screw you either way. I guess I could have used html parser, but I hate web crap (no offense to the good people in web development community) and that stuff would be too heavy for the simple parse; as much as I was able to see, it's mostly python native classes.
    So much about that, I am not really doing anything about that right now, I wanted to help Christophe and had to dig in the old code.


    I am now deep in trouble with some concepts which I am trying to integrate with clustering,NN's, SVM,etc... Some parts of that I would not be willing to discuss in public forum, but I would definitely love general discussion and would help where I can, particularly with some code snippets, ideas, etc.

    If anybody is doing serious work in that area and would like more in depth discussion, you can pm (I guess that thing is working) or contact here:
    ''.join(['nohtyp'[::-1],'@','ranidapurg'[::-1],'.','moc'[::-1]])

    Regards,
    Dr. Sheldon Cooper
     
    #37     Dec 27, 2011
  8. Zzoom

    Zzoom

    Great post.

    My experience is: FORTRAN, RPG, Java, C++, VB, C#, Perl, Python.

    Given the choice I'll take Python backed up by C++ every time.
     
    #38     Jan 7, 2012
  9. ycomp

    ycomp

    does anyone know if the JForex API is Thread-Safe? I assume it must be from the nature of the thing and from the examples but I just haven't read it anywhere - and nobody answers anything on the Dukascopy JForex forums, don't know why that forum exists really.
     
    #39     Oct 20, 2015
  10. Butterfly

    Butterfly

    wise words and refreshing to read,

    the problem is kids these days have no clue about programming and all they can do is use the latest fashion in programming to learn

    for those in the game since the beginning, we had not much programming choices and yet we created those choices by being more creative in programming

    today it's all about complicated and pointless framework because they have no fucking clue or too lazy to do the ground work
     
    #40     Oct 31, 2015