best distributed file system for homebrew hedge funds

Discussion in 'Automated Trading' started by lolatency, Mar 14, 2009.

  1. I'm losing my mind managing the volumes and volumes of data I have.

    What's the best distributed file system that can be used from both windows and linux that's not samba?

    NFS? I know NFS is insecure, but it doesn't matter for my homebrew network. I just need to make a few sizable volumes available on all boxes.
  2. wenzi


    'Best' for what? Just serving data to running applications? File storage ?

    If you are dealing with Windows, networking is not as nice as Linux or OSX. Give NFS a try.
  3. ATLien


    Need more of a use case scenario to give you a good recommendation.

    I like HDFS.
  4. rosy2


  5. I'm trying to speed up data-mining. Right now, I have R-code that runs; however, R is not multi-threaded. So it runs really slow, or I have to manually cut the files and run multiple instances of R. Also, R is really bad with large files.

    What I want to do is write the data to a network file system and then use something like Gearman ( to do the functions on the data from the file.

    Samba can do what I want, but what kind of bugs me about samba is that when I am windows, it tries to reconnect those drives and if a box is down, I end up getting some really terrible boot ups. Maybe I can use samba?

    I looked at AFS before. AFS looks hard to maintain and looks complex. Is there a decent tutorial? I -love- programming, but I hate system administration.
  6. ATLien


  7. Baywolf


  8. Can't say I enjoy system admin myself, but this is the sort of problem than can be solved by system admin. If slow boots because of unavailable file systems is the problem, then why don't you just mount whatever you want in parallel. (man page for mount says -F option will do the trick).

    If smbmount won't do this trick, just fork it from the startup scripts. (/etc/init.d/ is the place to start looking [Ubuntu 8.10]). Or create your own script to mount network fs.

    Might be a quick and easy solution with few changes until you decide on a network fs to use permanently.
  9. Baywolf


    I also don't enjoy mucking with startup scripts. I noticed my installation of redhat was trying to mount a network fs before the networking was even established. stupid...

    fyi, I use autofs to handle the auto mounting on startup.

    from ubuntu: apt-get -y install autofs
    #10     Mar 14, 2009