Thursday, 8 August 2013

Reading random lines from large CSV and write into a different CSV file

Reading random lines from large CSV and write into a different CSV file

Using this site, I have ways to read random lines from a CSV file but when
I write those random lines into a different csv file, i lose some
formatting. This is what I mean:
Snippet of the csv file I am reading from:
Case,Event,P01,P02,P03,P04,P05,P06,P07,P08,P09,P10,P11,P12,P13,P14,P15,P16,P17,P18,P19,P20,P21,P22,P23,P24,P25,P26,P27,P28,P29,P30
C000167,E03468,58,10,81,-0.3406615026224174,-5.581162038780728E-4,0.650243470511574,0.9011615038586112,0.829330406449827,-0.7616995269258996,-0.0019909209274398174,-0.025834173961019672,,,,,,,,,,,,,,,,,,30,
C000167,E03468,58,10,81,-0.3406615026224174,-5.581162038780728E-4,0.6517651862747824,0.9074693464398635,0.8380469158984926,-0.7589916620900861,-0.0019954144276356335,-0.03222025640621825,,,,,,,,,,,,,,,,,,30,
C000167,E03468,58,10,81,-0.3468052064483135,-5.581162038780728E-4,0.6522422131598,0.9025930670926058,0.8332814044565442,-0.7508680675826261,-0.001998391716108935,-0.03533445710480293,,,,,,,,,,,,,,,,,,30,
After reading N random lines from the above file, this is what I get:
Snippet of the output file:
"Case,Event,P01,P02,P03,P04,P05,P06,P07,P08,P09,P10,P11,P12,P13,P14,P15,P16,P17,P18,P19,P20,P21,P22,P23,P24,P25,P26,P27,P28,P29,P30
"
"C001685,E35590,58,10,81,-0.3654481697820668,-5.581162038780728E-4,0.655619563505728,0.8856242052995037,0.7811768584356523,-0.6824944804781753,-0.0018236965581893878,0.08998255581671266,0.46818235899722466,0.5349178991074381,0.0023192678347080547,0.0023192853642003794,-1.379960600462541E-4,-0.0029589652839934494,0.0190440944409012,-0.006310411153684246,0.2667578790879315,0.8519823892954023,0.005696496533938292,0.6083983010037551,-0.15781372752517367,0.8433075356454734,0.8147766325543522,0.6320612846744368,-0.47415970209573,34,1.521461493699171
"
Notice the quotes I get. When I read the ouput file, the original 32
columns are now reduced to 1 column.
The code I used is below:
import random
import csv
import os
os.remove("train_select.csv")
file_size=1146880-600
f=open("train.csv",'rb')
dialect = csv.Sniffer().sniff(f.readline())
f.seek(0)
csv_file=open("train_select.csv",'wb')
case_writer = csv.writer(csv_file,dialect)
f.seek(0)
random_line=f.readline()
case_writer.writerow([random_line])
for i in range(0,20):
offset=random.randrange(file_size)
f.seek(offset)
f.readline()
random_line=f.readline()
case_writer.writerow([random_line])
f.close()
csv_file.close()

No comments:

Post a Comment