I hesitate to blog this because this tool is really really really really rough, quick, and dirty. As in it needs a big ol’ box of Tide.

I needed to convert a bunch of UTF-16 text files into UTF-8 so I spent five minutes writing a little console app to do it.

This thing literally has no exception handling etc, but it gets the job done for my needs and I thought others might find it useful if they have exactly the same need.

Hey, feel free to clean up the code and send it back to me, or point me in the direction of some free tool I should’ve used all along.

USAGE: batchencode extension encoding [backup] extension: file extension with the dot. ex .sql, .txt encoding: values... utf7, utf8, unicode, bigendianunicode, ascii backup: optional fully qualified (sorry) backup directory.

What others have said

Sam Smoot Sep 20, 2006 9:49 PM

# re: Batch Encode Text Files

Ruby! :-)

File::open(ARGV.first + '.output', 'wb+') do |f|
f << Iconv.new('utf-8', 'utf-16').iconv(ARGF.read)
end

Ok, granted, Iconv can be a bit flaky on Windows (and you probably won't have it by default unless you installed with the OneClickInstaller). Sometimes it just throws an exception when the input is obviously perfectly valid for the given character-set. I've had better luck with rbuconv but it'll be significantly slower since it's pure-Ruby, so it's really best suited to files under... oh... a hundred Mb or so?

Either way, I've taken a special interest in encodings, so interesting post!

Now if only someone would show us the Python (IronPython even? As long as you don't just cheat and use the .NET BCL) version. :-)