I have just finished testing my Flac library. 191 file of 4068 flac files in my library directory fail the -t option of Flac 1.1.4.

Of the same number of files in my backup copy, 212 fail. It seems as though there is at least one unreliable link in my system. The master copy is on a new 1 terabyte My Book™ Premium Edition™ II external HDD connected via Firewire 400. The backup is a Maxtor OneTouch III 500Gb external HDD connected via USB 2.0. I have a standard older Dell, Windows XP Home, all the normal anti-virus, etc. I'm a 30 year IT guy so I generally know what I am doing.

I recently moved my library from the Maxtor to the WD drive, formatted the Maxtor, and copied my library of MP3 and Flac files back to the Maxtor to form a backup. Neither copy process generated errors. The library directory is 275 GB.

I have not yet analysed all the files that are failing, but it seems as though individual files in an album directory fail the test. It does seem random. Flacs that I created in the last 2 weeks are failing to verify.

I also have to wonder whether any of the programs I use to create (dbPowerAmp & Flac 1.1.4) , tag (Tag&Rename, Foobar & MediaMonkey), manage (MediaMonkey), play (MediaMonkey, Foobar & iPod) and sync (MediaMonkey) flacs are also part of the problem.

Bottom line though: I now cannot rely on my flac library to provide a reliable backup to my CDs. Further, if I have this problem with 1:20 flac files, I probably have the same problem with the MP3 files in the same library directory as well. I have not noticed any glitches playing any of these MP3 files either, but that may not mean anything if the odd bit changing here or there does not blow up the player. So how can I test these MP3s or do I need to go back to the original CD and re-encode?

If I have to re-encode my entire 600+ CD library, how best to do this? Flac, APE, WV, Apple Lossless? Do I need to modify the test .bat file to set the Read-Only attribute on and stop anything changing the file once it is tagged and verified. I probably need to also create a regular (nightly?) scheduled .bat to test the library directory and list any files in error. Does a backup to another external HDD make any sense if the file copy itself is unreliable.

So, it's back to the drawing board. Any suggestions welcome while I contemplate this disaster.
Rats!!!!!

Sure this sounds like a hardware failure.
I've got a Dell laptop and a Maxtor OneTouch II (IIRC) and the firewire link between them is totally unreliable. Lots of files were corrupted because of that. On the other hand copying via USB2 seems to work alright so far (both USB2 and firewire are supported by the drive). And my other Maxtor, older USB2 model, works fine. So I've got mixed feelings about Maxtor.

What I've learned I should do when making backup: now I always use md5 on almost all files, not only flacs.

(I'm not sure if new flac supports md5 internally? I know wavpack has internal checksum, not sure about flac)


Which version of MediaMonkey have you been using?
Check this thread: Mediamonkey bug tagging flacs


There are some tools which are supposed to verify mp3s. However I would simply play all mp3s in foobar through the dummy output plugin. In foobar 0.8.3 it was called "speed-meter pop-up" something. Foobar then decodes the mp3s as fast as it can and pop ups a warning on mp3 stream errors. (Edit: in 0.8.3 it was not output plugin but a dummy "encoder" - Run Conversion - choose this "speed meter")
I'm sure foobar 0.9 should have something similar.

The two ext HDDs are connected via an old-ish USB/Firewire card, Belkin I think. The drives themselves are both reasonably new (WD <30 days, Maxtor ~9 months) so I have to doubt that the drives themselves are the problem. The idea of the USB/Firewire card dropping bits moving from USB to Firewire is feasible I guess. Maybe it is worth trying a new USB card in the Dell.

All the software are latest versions. MediaMonkey is 2.5.5.998.

It certainly seems worthwhile running a batch file to test all flacs and flag any errors every now and then (weekly?). Clearly, we can't take for granted that hardware or software errors are not going to cause problems over time.

I'd also have to think that md5 or CRC32 checksums on files/directories would be a good idea. And I also have to wonder whether I should allow MediaMonkey (or anything else) to write to Flac tags (ratings, etc) once the file has been encoded, tagged and checksummed. So setting the read-only attribute on files could be a good idea.

I like the idea of being able to run a test against the flac file. However, I have to assume that the test itself is robust. Is there any possibility that this could be a problem?

I am going to update the flac-verify.bat batch file I have adapted from Synthetic Soul's post to do a "silent" test, if that fails, then re-run the test and pipe the output to a date stamped log file. If the file passes the test, then set the read-only attribute on the file. I'll also create a batch file that will create a .sfv file for every album directory in the library.

This is really a downer...

Just finished reading the Media Monkey forum and discovered that there was a bug with 2.5.5.996 that caused corruption of flac files. This may also be a factor in a number of the corrupt files in my library. Although I generally did my tagging with Tag&Rename, I updated the song rating in MediaMonkey.

So maybe the idea of setting the read-only attribute has merit. Once you're done with tagging, rating, comments, picture tags, etc., test the file one last time and then lock it up. Then test regularly.

It would be nice to think that all software developers can get this stuff right, but that's unrealistic. And I'm not being critical. Having been a programmer in my dim past, I know how hard this stuff can be.

I don't think there is any problem with such flac testing. Not that I know of.
In fact, even though FLAC doesn't have internal checksum of uncompressed wav data, it stores the checksums of all compressed frames, and these are actually verified during a test. So if a flac file passes the test, I think it is safe to say the file is OK.

And, presumably, the converse is true: if the flac file fails the test, then there really is a problem with either the audio or the tag content of the file, or both.

Not that it matters, but just as a side note: I think MediaMonkey cannot store ratings in FLACs. It can save ratings to mp3s in id3v2, but for FLACs it can only keep it in the library, so it seems. Edit: Maybe I'm wrong on this one. the manual says it should be supported. I will have to check why it didn't write ratings in my flacs... Nevermind.

yet it does, FLAC was the first codec to have that feature.


if the files encoded with --verify correctly, it's practically guaranteed that corruption was after encoding. even without --verify, if it were FLAC's fault then you'd be the first person in 8 years of its existence to find such a problem.

I'd bet on the firewire transfer. (edit: or possibly mediamonkey, I think there was a bug a while back where tag editing corrupted the FLAC file but the audio was still recoverable.)

Josh

I'm betting on both. The firewire/usb idea tends to be supported given that the copy back from the (firewire) WD to the (USB) Maxtor introduced new file corruptions. The WD version has 191 corrupt files, and the Maxtor has 212, so new ones got created during the last backup copy.

And rating data is stored in the tag by MediaMonkey. 2.5.5.996 certainly corrupted flac files if tags changed.

I'll just have to go through the 191 corrupt files on the master copy of the library, re-encode flac-flac, or re-rip from the CD, or trash any that I can't recover.

Once that's done, set read-only attributes on files that have been checked & tagged, create an .sfv file for each album directory, and set up a weekly test schedule for both the flac's and the .sfv files.

I'll post the batch files that I create to automate this. Thanks to Synthetic Soul for getting me started with flac-verify.bat. Talk about pulling at a thread!

I'll probably have to re-generate the MediaMonkey library database to be sure I get all the track tag data correctly read.

Any other suggestions are most welcome. This is gonna take weeks.

If that refers to the flac files corrupt by the MediaMonkey bug, then it is certainly not the way to go! Please check that MediaMonkey forum thread one more time. They made a tool to fix the corrupt flac header instead of just skipping it.

Got it: FlacFixTool.zip. Thanks for the heads-up kdo.

Edit: although another thread at the MediaMonkey forum suggests that simply recoding the flac files with 1.1.4 back to flac solves the problem. I'll try both to see what works.