How about using selenium for the web-crawling in IB webpage?

Discussion in 'App Development' started by jk90029, Mar 16, 2016.

  1. d08

    d08

    You keep asking the same question over and over again hoping for a different answer.
    It's called "scraping", not "crawling" (different thing) or "scrapping" (I'd like to scrap your idea).

    If you decided to scrape, you can just use python, perl or any language and some web interface module. It should take you no more than 1-2 days to learn this and get it working as basic scraping is very simple.

    PS. $10k for someone to write IB API stuff specifically for you isn't that crazy, after all you're talking about programming in the field of finance.

    PS22222222222222222222. I can't believe so many people here tried to help you sincerely.
     
    #21     Mar 20, 2016
  2. Fortunately, it was successful to use C# to communicate web page of broker, with selenium and DOM, which was introduced just few years ago.

    using OpenQA.Selenium;
    using OpenQA.Selenium.IE;
    using OpenQA.Selenium.Support.UI;

    Please note that most selenium books are published in the last 3 years, as shown at http://www.amazon.com/s/ref=nb_sb_n...ywords=selenium+book&rh=i:aps,k:selenium+book
     
    Last edited: Mar 20, 2016
    #22     Mar 20, 2016
  3. I am not so rich man.
    My friends and family can help me for such a work, like delivering and communicating between broker server and me.
    There might be rich man somewhere to pay > 10K.

    However, main trading logic for buy/sell costs, of course, a lot more than 10K or even 100K.
    Already I spent the logic programming (some thousands of line) myself more than a few years.

    As a further (hypothetical) example, if some ask me "Give me 100K if and only if I gave annual 15% logic to jk90029", then I gladly accept it.
    or tell me "Give me 1000K if and only if I gave annual 20% logic to jk90029"
     
    Last edited: Mar 20, 2016
    #23     Mar 20, 2016
  4. I feel to say sorry to those who makes money for programming to other traders.
     
    #24     Mar 22, 2016
  5. d08

    d08

    Why are you sorry? There's plenty of demand for their services (I'm not one of them).

    And I'm quite sure you'll commit some newbie mistakes using writing your code as you don't exactly know what you're doing. Good coding skills come with experience.
     
    #25     Mar 22, 2016
  6. Thanks for good comment.

    I agree / Programming skill is not sufficient for a good trader with fair PnL, although it is necessary for sure. Good logic is 95% and good programming skill is only 5%.

    It has been ONLY three years since selenium is used widely.
    Possibly, after next 10 years, selenium with DOM and html5 may be popular for simple implementation to many individual brokers. (Different brokers have different API)

    So far, successfully my stock holdings (and order info) could be downloaded to local PC by web-login.

    Next week, should program codes to send 100 lines of buy/sell order to broker server, of course by my helper with selenium experience with fair labour charge.
     
    Last edited: Mar 22, 2016
    #26     Mar 22, 2016
  7. d08

    d08

    Selenium is just a tool and the idea isn't new at all. There are many like it. After all web crawling and scraping has been around since the 1990s as already mentioned.
     
    #27     Mar 22, 2016
  8. Agree, but there is some advance in selenium, compared with 90's technology such as PERL, if I heard correctly.

    For example, even if the webpage is changed a little bit by webmaster, selenium can detect and adjust it by use of DOM.
    Back in 90's, there was no concept of DOM.
     
    #28     Mar 22, 2016
  9. oly

    oly

    Modern web parsing libraries also know about DOM, such as bs4 for python. Honestly I think it's moot since you'd be better off using IB and ibpy, but it's your windmill to tilt at :)
     
    #29     Mar 22, 2016
    d08 likes this.
  10. I heard beautiful Soup is another option for parsing. Tell me if there is some advance.

    Please use plain and easy English for me.

    I am not talking about particular broker, such as IB.
    Unlike other traders, I like to use brokers of at least three countries.
     
    #30     Mar 22, 2016