how to do web scrapping

Discussion in 'App Development' started by gmst, Feb 7, 2013.

  1. gmst

    gmst

    Thanks looks interesting. I have never any python before. I might just enroll.

    As an aside, Coursera says all courses are free on it.

    How does coursera make money?
    Why do professors upload their courses on coursera, if they are not getting paid anything? I know few people will do it for a good cause. But many professors at my university were "extremely interested" in keeping their intellectual property private and wanted to monetize their IP in as many ways as possible.

    Hell - one professor never gave us powerpoints rather just printed slides and gave them to us so that we couldn't pass-on the powerpoints electronically to others.
     
    #21     Feb 9, 2013
  2. Bob111

    Bob111

    which one? VB6 or VBA?
    it's fairly simple in VB6

    just drop Internet Transfer Control on form and do this:

    http://msdn.microsoft.com/en-us/library/aa733648(v=vs.60).aspx

    what you trying to do anyway? someone might have coded already
     
    #22     Feb 9, 2013
  3. gmst

    gmst

    I mainly use vba with excel but since vb6 is not too different. I will see into your link and try to work.

    Just trying to learn this skill first. Once I pick up the skill, I have a lot of ideas that I would like to test going forward. Before I can test any idea, I will need some data for it.

    In the beginning, it will be mostly shooting in the dark and hopefully pick up few things in the process. Following are some of the specific ideas that I am planning to collect data from:

    Basic Stuff Mostly about market internals and some news
    1) http://online.wsj.com/mdc/public/page/2_3022-mfgppl-moneyflow.html
    2) collect information from google/yahoo finance

    Advanced Stuff This will be a long time thing.
    1) collect information from twitter on a particular stock
    2) Characterize this information/news on a sentimental basis (bullish/bearish)
    3) Try to build some kind of regression model to see if tweets can help me in my trading.
     
    #23     Feb 10, 2013
  4. http://en.wikipedia.org/wiki/Coursera
     
    #24     Feb 10, 2013
  5. Bob111

    Bob111

    yahoo data -not a problem,goog-if you place too many requests too fast-you going to get an error. same with yahoo's YQL
    you can use this-

    http://www.gummy-stuff.org/Yahoo-data.htm

    as for page from wsj-you can use web query

    play with this internet transfer control. it's really simple. PM me,if you need examples for VB6 for yahoo. i have some pieces here and there. but from my practice-it's sometimes easier to code from scratch,than figure out someone's code :)

    remember-with this control-you gong to get a string with HTML code. same stuff that you see,if you go on any page and see page source.then you have to go thru this data and pick pieces you need.

    you can use this link for excel's web querry(link on page):

    http://online.wsj.com/mdc/public/npage/2_3045-mfgppl-mfxml2csv.html
     
    #25     Feb 10, 2013
  6. gmst

    gmst

    Thanks Bob, very helpful of you. :)

    Let me see what I can do. I will get back if I need some example codes from you.
     
    #26     Feb 10, 2013
  7. hft_boy

    hft_boy

    #27     Apr 7, 2013
  8. ddecker

    ddecker

    #28     Apr 8, 2013
  9. Anyone got examples, ideas, starters for a C++ project for web scrapes?
     
    #29     Jul 10, 2013
  10. #30     Jul 10, 2013