Visualizing Large Data Sets

Discussion in 'Trading' started by kmiklas, Jul 29, 2021.

  1. kmiklas

    kmiklas

    Brethren:

    Any recommendations on how to visualize and manipulate large (100,000 + points per set) data sets? I feel like this is beyond Excel.

    Is there any software out there with the horsepower to handle this volume of data?

    Thanks, and have a profitable day.
    -- Keith XD
     
    Last edited: Jul 29, 2021
  2. 2rosy

    2rosy

  3. tiddlywinks

    tiddlywinks

  4. Tableau
     
  5. sef88

    sef88

    Python
     
    Snarkhund likes this.
  6. Atikon

    Atikon

    Aren't you a programmer? Why do you ask for this 5 cent advise in this forum? Matplotlib, seaborn and bokeh should come to mind
     
  7. kmiklas

    kmiklas

    1. I am a programmer. What does that have to do with knowing the data visualization market?
    2. As a programmer I can build something like this, but it's a lot of work, so I'm hoping to use 3rd party stuff.
    3. This forum's advice is at least 10 cent; perhaps even 25.
    4. I'm hoping for something really slick. I'm hoping that when I CTRL-scroll it will conflate and/or deconstruct data into candlesticks that make sense
    5. Thank you for the recommendations
    6. I feel like I'm going to end up building something custom anyway
     
  8. kmiklas

    kmiklas

  9. rb7

    rb7

    Seems very cool.
    If you want to entertain yourself that's probably a good way to do it.
    But if you want to make money from the market...I don't know.
    And btw, Excel can support up to 1,000,000 rows. So 100,000 shouldn't be a problem.
     
    kmiklas likes this.
  10. I know that my recommendation may seem hard but I would advise you to learn data visualisation with python and matplotlib.
    I understand that some people don't want to hear anything about programming but python is better than any other app because it is extremely versitile. I mean that there are lots of computer programs which require files of special formats or the apps which should be bought and their free trial cannot bring you anything useful. Python is absolutely free of charge. You need just download it and use. More than that, it is very easy to visualise data using python library matplotlib. You just need to write a couple of lines of code and Bob's your uncle. Especially, if you are working with huge datasets, python will be a really nice solution to your problem.
    To conclude, I should say that it only sounds a bit of scary that you need to do some programming in order to make some chart or graphs, but it is extremely easy in reality. Give it a try!
     
    #10     Aug 4, 2021