Python - Read and split lines from text file into indexes.

Discussion in 'App Development' started by OTM-Options, Apr 28, 2015.

  1. jj1111

    jj1111

    #51     May 8, 2015
    i960 and eusdaiki like this.
  2. Python CSV to PDF Script

    Now on to the next step, a Python CSV to PDF script. I will be using the Python pyfpdf PDF generation library, which is based on FPDF the php library. I have used FPDF before and it is very good. More info: pyfpdf - PDF generation library

    Formatting will include:
    • Alternate row colors.
    • Choice of fonts, bold, centered and colored text. Red for negative numbers.
    • Column headlines at the top of each page.
    • Additional info at the top of the first page.
    • Grid lines and page numbers.

    I expect to take a few months to complete this project and will post back once completed. Does anyone know of a better Python PDF library than pyfpdf?


    :)
     
    #52     May 8, 2015
  3. I never put someone that fast on ignore. You added zero value and troll. Both together is too much. Adios amigo


     
    Last edited: May 8, 2015
    #53     May 8, 2015
  4. Omg. I am out of here. This is the most ridiculous programming thread so far for sure. Linux clowns masturbating all over Python but cannot even properly solve the original simple task. Instead it turns into a little afternoon tea python library chatter between aunties and when confronting performance with hard cold numbers, trolls literally crawl out of the closet. And now the next project dealing with color cosmetics which lasts for several month. Some of you people are outright bizarre. No wonder hardly any developer is ever promoted to a junior trading position in a professional environment. Too many clowns who love to listen to their own talk instead of answering the fucking question.

     
    Last edited: May 8, 2015
    #54     May 8, 2015
  5. jj1111

    jj1111

    to be clear, I was trolling using my own troll code, not calling your code tro-- oh what does it matter...

    sound_effect.sad_trombone()

     
    #55     May 8, 2015
    i960 likes this.
  6. I have made some changes to the Python CSV to CSV script:
    • Added a format_currency function to change 12345.67 into $12,345.67 and -12345.67 into ($-12,345.67).
    • Instead of calling the write_file function with each loop I have now concatenated each loop together into a string and then write to file at the end of the script.
    • Only the row counter is printed to the terminal while the script is running, instead of all the data. This has made a big improvement on the performance of the script.
    • The 1,000,000 Line Test now comes in at 1 minute 28 seconds, down from 2 minutes 53 seconds. More info: The 1,000,000 Line Test


    Updated CSV to CSV script. 133 lines and 4.7Kb in size.
    Code:
    #!/usr/bin/python
    # Python version 2.7.6
    
    import datetime
    import time
    
    def timer(label):
      ts = time.time()
      st = datetime.datetime.fromtimestamp(ts).strftime('%H:%M:%S')
      timer_out = open("timer.txt",'a')
      timer_out.write(label + ": " + st + "\n")
      timer_out.close()
    
    timer("Start")
    
    def format_currency(value):
      if value < 0:
      results = '${:.2f}'.format(value)
      value = "(" + results + ")"
      return value
      else:
      return '${:.2f}'.format(value)
    
    csv_in = "TransactionHistory_22523594.csv"
    csv_out = "transaction_history.csv"
    header = "Row,Date,Buy/Sell,QTY,Security,Price,Debit,"\
      "Credit,Commission,Total Amount,Currency" + '\n'
    
    footer_commission = 0
    footer_debit = 0
    footer_credit = 0
    footer_total_amount = 0
    counter = 1
    split = ","
    join = ","
    row_out = ""
    
    def write_file(row,write_to):
      f_out = open(write_to,'a')
      f_out.write(row + "\n")
      f_out.close()
    
    write_file(header,csv_out)
    
    for line in reversed(list(open(csv_in))):
      if len(line.strip()) != 0 :
      line = line.strip()
      column = line.split(split)
      if column[2] == "Buy" or column[2] == "Sell" or column[2] == "Expired":
      row_counter = '{0:03d}'.format(counter)
      transaction_date = column[0]
      buy_sell = column[2]
      qty = column[5]
      security = column[4]
      price = column[6]
      if price == "":
      price = "0"
      total_amount = column[8]
      currency = column[9]
      transaction_date = (datetime.datetime.strptime\
      (transaction_date, "%Y-%m-%d").strftime("%a %b %d"))
      qty = abs(int(qty))
      price = float(price)
      total_amount = float(total_amount)
      amount = qty * price * 100
      amount = int(amount)
    
      abs_total_amount = abs(float(total_amount))
      if column[8] >= "0" and  column[8] <= "1":
      commission = 0
      else:
      commission = abs(abs_total_amount - amount)
      if total_amount <= 1:
      debit = amount
      credit = 0
      else:
      debit = 0
      credit = amount
      if debit > abs_total_amount:
      credit = debit
      commission = debit
      debit = 0
    
      footer_debit = (footer_debit + debit)
      footer_credit = (footer_credit + credit)
      footer_commission = (footer_commission + commission)
      footer_total_amount = (footer_total_amount + total_amount)
    
      qty = str(abs(qty))
      price = format_currency(price)
      debit = format_currency(debit)
      credit = format_currency(credit)
    
      if debit == "$0.00":
      debit = ""
      if credit == "$0.00":
      credit = ""
      commission = format_currency(commission)
      if commission == "0":
      commission = ""
      total_amount = format_currency(total_amount)
      counter = counter + 1
    
      row = (row_counter + join + transaction_date + join + buy_sell + join \
      + qty + join + security + join + price + join + debit + join + credit + join \
      + commission + join + total_amount + join + currency + "\n")
    
      print row_counter
      row_out = row_out + row
    
    pl = (footer_debit + footer_commission)
    pl_percent = ((footer_credit - (pl)) / pl * 100)
    pl_debit = format_currency(pl)
    footer_debit = format_currency(footer_debit)
    footer_credit = format_currency(footer_credit)
    footer_commission = format_currency(footer_commission)
    footer_total_amount = format_currency(footer_total_amount)
    pl_percent = str(pl_percent)
    
    join2x = (join + join)
    join5x = (join + join + join + join + join)
    footer = (join5x + "Subtotal" + join + footer_debit + join + footer_credit + join \
      + footer_commission + join + footer_total_amount + join + currency + "\n \n" \
      + join5x + join2x + "Total Debit" + join2x + pl_debit + join + currency + "\n" \
      + join5x + join2x + "Total Credit" + join2x + footer_credit + join + currency + "\n" \
      + join5x + join2x + "P/L" + join + pl_percent + " %" + join + footer_total_amount \
      + join + currency)
    
    write_file(row_out,csv_out)
    write_file(footer,csv_out)
    timer("Finish")
    
    input("\n\nPress the Enter key to exit.")
    
    :)
     
    Last edited: May 9, 2015
    #56     May 9, 2015
  7. Posting to congratulate yourself on your own prior post? Maybe an all time low for ET, which is saying something.
     
    #57     May 11, 2015
  8. jj1111

    jj1111

    I took a shower after that post. After I'd finished my cigarette, of course.
     
    #58     May 11, 2015
  9. 2rosy

    2rosy

    a few people did solve his simple task. At least the thread title task. I am more than willing to write this for anyone for my minimum fee which so far scares 100% of retail traders away.
     
    #59     May 12, 2015
    jj1111 likes this.
  10. Here is the first post of OP: " have a text file with hundreds of lines and 10 columns of data separated by commas. I want to split the lines at the commas into 10 indexes and access each index individually. The code below only works on the first index - items[0] - and will print the first column and all the rows. If I change it to items[1] it will crash."

    -> Not one person provided a solution that OP requested. If Pandas is used then fine, why not. But the data tables still need to be arranged then in columnar index as OP requested. I do not think I was stickler for details, and I challenged anyone to provide a solution in Python that does what OP asked. I am still curious about the time it takes with an optimized Python version. I found it ironic that those who attack Microsoft as if there is no tomorrow argue for Linux because Windows has too many built-in latencies but then use the possibly slowest language to solve even the simplest of problems and are not man enough to face up to a performance comparison. Outright bizarre.


     
    #60     May 12, 2015