Another Marginally Useful Tool - BatchConcat

• Sep 21, 2006
• 6 Comments
• Print

UPDATE: In one comment, szeryf points out something I didn’t know and invalidates the need for the tool I wrote. This is why I post this stuff, so someone can tell me about a better way! Thanks szeryf! I’ve updated the post to point out the better technique.

Based on my recent spate of posts, you can probably guess that I am working on improving a particular build process. 

In this situation, I have a pre-build step to concatenate a bunch of files into a single file.  I tried to do this with a simple command like so:

FOR %%A in (*.sql) do CALL COPY /Y Output.sql + %%A Output.sql

Yeah, that would work, but it is so sloooooow.

Szeryf points out that I can simply pass *.sql to the COPY command and get the same result.

copy *.sql output.sql

This ends up running plenty fast as it doesn’t dumbly iterate over every file calling COPY once per file. Instead it lets COPY handle that internally and more efficiently. How did I not know about this?

So I wrote a one minute app by simply scavenging the code from BatchEncode and concatenating text files instead.

USAGE: batchconcat EXTENSION ENCODING OUTPUT
	 sourcedir:	source directory path
	 extension:	examples... .sql, .txt
	 output:	         the resulting file.
	 encoding:	optional: utf7, utf8, unicode, 
                        bigendianunicode, ascii

	 All paths must be fully qualified.
	 USE AT YOUR OWN RISK! NO WARRANTY IS GIVEN NOR IMPLIED

This ended up being mighty fast!

I figure someone out there might need to do this exact same thing in their build process and won’t mind using such crappy code.

What others have said

szeryf Sep 21, 2006 3:39 AM

# re: Another Marginally Useful Tool - BatchConcat

no need to write separete app when you can:

copy *.sql output.sql

works on my winxp.

Matt Ellis Sep 21, 2006 3:57 AM

# re: Another Marginally Useful Tool - BatchConcat

Or:

type *.sql > file.txt

szeryf Sep 21, 2006 4:05 AM

# re: Another Marginally Useful Tool - BatchConcat

type doesn't work the same way, because it inserts the filenames in between. so it will probably spoil your *.sql file.

Steve Harman Sep 21, 2006 11:43 AM

# re: Another Marginally Useful Tool - BatchConcat

@szeryf, AFAIK that approach suffers from the same problem that Phil mentioned in the original post... namely it's "sloooooow".

While speed has no real impact on the build being successful, it is annoying when a developer has to wait an extra x seconds each time he/she hits CTRL+SHIFT+B. And I think that's what Phil was trying to get around with this rough-and-dirty app.

I would bet that the biggest bottleneck with the "copy" approach is that each "copy" command has to write to the disk, but in Phil's custom app, he can use a TextWriter and only have to flush to disk once.

Haacked Sep 21, 2006 12:19 PM

# re: Another Marginally Useful Tool - BatchConcat

@szeryf Thanks! I didn't realize that syntax worked. Doh!

@Steve: What I was doing was slow because I was iterating one file at a time. But passing in *.sql to copy is really fast.

Haacked Sep 21, 2006 3:20 PM

# re: Another Marginally Useful Tool - BatchConcat

One issue with COPY *.SQL is that there's no easy way to tell it to insert a newline between files.

That could come to bite you. Just make sure each file you concatenate ends with a newline.

I'm using COPY /b *.sql because it's UTF-8 and not ASCII.

What do you have to say?

Title:

Name:

Email: (never displayed) (will show your gravatar)

Website:

Comment: Allowed tags: blockquote, a, strong, em, p, u, strike, super, sub, code

Please add 1 and 2 and type the answer here: