Caltraders suggestion sounds good theoretically, that means, if you have more than 2GB of memory (perhaps 4GB is needed), if you are running with less physical memory, you will use the file system, in memory or not. So that solution probably will not be any better than your first try. autotrader
Nope: the header is in memory... it should not be that large. Even interday text data should not be GByte size ... Even if it were there are ways to allow some paging - slower but will still work reasonably well and should be faster than the original technique if programmed efficiently...... A good lesson is that if you rely on third party utilities you often get stuck with stuff that does not do the job: you should always get source with the utility (although if the utility is poorly written then getting access to poorly engineered source code is not much help .....)
If you are going to write your own code using existing ZIP archive libraries, then you are still no better than just using Winziip or Winrar to move your zip file contents, as your code would just be doing the same thing as what the existing programs do. Unless Caltrader implies you read the "RAW" file and parse it yourself from scratch (which I didn't think is what he was suggesting), which means you need to know and code all the different compression / decompression algorithms yourself. at which case I will suggest that you allow plenty of time for writing that code and back up your original file as I can tell you from experience that you can easily corrupt your zip file if you do not know what you are doing. my AutoTrader program reads and writes to Zip files on the fly, so I know what directly accessing a Zip file is like. I write code everyday for a leading software company, and my suggestion is just a practical solution. It is your call which path you are willing to take. autotrader
With all due respect to your suggestions and background .... Writing the code from scratch to read and write the zipfile format is not difficult for a programmer of average ability. If you want to use Java the code is available for free from Sun. If you want to use another language in which you a competent then you could probably roll your own solution in a day. Like you we write code for several leading companies including some leading government labs. In our opinion this is not a difficult task for anyone on staff and we have done this several times for several projects - no we aren't looking for work, no advertisement, everyone is very busy, and our code isn't for sale since we think that this is pretty easy for anyone to do on their own. FYI there is no need to go to all the trouble of writing compression routines: zipfiles are already compressed and you merely need to extract the component files from the archive and copy them to a new archive or to a new location - if I correctly read what the orginal poster wanted to do. This is simple stuff ....
At the risk of sounding really stupid - wouldn't that still involve the step I'm trying to avoid? By "merely need to extract" you mean it would still be decompressed first and then compressed when moved to the new archive, right? Thanks for the advice guys. I've certainly learned more about this than I thought there was to it
Nope. Thats the idea: to avoid having to decompress-recompress. To merely move files from one archive to another or from one location in storage to another you dont need nor should you want to perform these operations .....
Great. Honestly though, it's such a simple task I would have never assumed it's not already available as a feature of some compression utility. Thanks for pointing me in the right direction guys.
Vlad, if this is a one-time effort, I would suggest sucking all of the files into SQL Server using DTS. Then, you can query or export them however you please.