Fun with text files



  • I have a 2GB text file with about 2.3 million records It is updated weekly.
    The data is in a 200 character wide fixed width format.
    0_1476776411160_upload-b327eaf2-c018-4414-9952-21d88798e719

    In order to do what needs done, it is being brought into SQL with a BULK INSERT.
    That step takes just under a minute on my machine. Not aware of any better method for this step.
    0_1476776311383_upload-ff7d8a8f-ef17-47e1-94d3-e4280e180cea

    The next thing it does is execute a stored proc that inserts each line into a new table with everything substringed out into pieces. that only taker another 40 seconds or so.
    0_1476776814203_upload-ef6e4460-1ee9-4107-a625-c22464cd930d

    Then do a ton of recursive querying to link things together under certain criteria. Only 2 minutes per day there.
    0_1476777094734_upload-36c32989-befa-4f49-a0be-73f29a6dc4c9

    Related: looking at code your wrote 6 years ago is sometimes scary.



  • @JaredBusch said in Fun with text files:

    Related: looking at code your wrote 6 years ago is sometimes scary.

    Especially when it works and you're not exactly sure why! lol.



  • at "200 character width" I started thinking about the COBOL class I took in college.