Quickpar/par2 are good recommendations.

However, as I mentioned earlier dvddisaser is (IMHO) the application the thread-opener needs. The tool specifically address bitrot of data files on optical media. And it's engineered specifically to help you toward that goal. And it is Free Software.

Read the website, it has very useful information and gives multiple proven techniques for reducing your long term data-loss risks:

http://dvdisaster.berlios.de/en/

-brendan




FYI, only CDs have the severe label side vulnerability. The metal layer for DVDs is actually in the middle of the wafer, equidistant from both the read side and the label side. If you think about it, here's a good proof: explain why dual sided DVDs exist, but dual sided CDs do not.

-brendan

Not to mention that CDs and DVDs have multiple layers of built-in error correction. A scratch through the data layer isn't necessarily going to result in data loss.

I thought of bringing this up earlier, but considering how many misconceptions the OP had, I really didn't know where to begin.

EDIT: spelling/grammar

EDIT2: But you're right brendan; you have to go pretty deep to scratch the data layer on a DVD.

EDIT3: errr, forget it, I've already been put in my place.

Yes, most magnetic and optical media store, at the bit-level, both error detection and error correction data. The drives themselves can detect and correct the raw data read off of the disc/disk. Most contemporary magenetic media has the ability to "refresh" failing data and can even rewrite and/or reallocate the corrected data back onto the media. Optical drives generally cannot do this, so their bitrot tends to accumulate faster.

You can add an additional layer of error detection/correction by using an array that includes redundancy.

In addition, the PATA/SATA/SCSI/USB/Firewire/etc. connections generally include error detection on the communications and will retransmit several times if you have a timing/power/interference/cabling issue.

Once the data gets onto the main board, you are at the mercy of bad RAM, bad software, good software with esoteric bugs, cosmic rays, gremlins, the NSA and fat fingers. No file format can protect you against that.

There is only one thing that can: backups.

-brendan

...and to go full-circle, does MAC 3.99 have (or does anyone have any reason to think that it has) any esoteric bugs?

Sure it has them. If you encode files at full moon you must not forget to dance naked around an oak tree, else you will hear voices in your music

DVD-Rs would be a better choice than CD-Rs, since they have better error correction capabilities (though I don't know how good is it in the long term vs a CD), provided you don't scratch it up or destroy the disk.
That, and there's more space to waste on redundancy.
Speaking of DVDisaster, I need to try it out, it's a very cool tool.

I'm not sure that NTFS/FAT/ISO9660 has correction, but it has indeed a very strong CRC check! It choke when I tried to restore many of my old data CD-R's. According to thread opener, Windows (etc) would just disregard any kind of CRC check and copy the files corrupted.

Well, here are my 2 cents:

I also use MAC for archiving my rips. It gives slightly better compression than WavPack and FLAC, the tagging is fine, and it is fast ENOUGH.

I had one cd image which I compressed with mac 3.97 that did not decode with mac 3.99, out of around 20, but it decoded fine on MAC 3.97, I reencoded, and was satisfied. The MAC gui has a verify, which check the files after packing. In case of doubt, I would suggest to check the files before archiving.

The error DETECTION for MAC is more AS robust AS FLAC, since, as posted above, it has an md5 of the audio data and a CRC per frame, but I am not sure about WavePack (CORRECTED ACOORDING TO INFORMATION BELOW BY jcoalson). I am not sure which method the MAC front-end uses, but I gess it uses both. So as far as error DETECTION goes, I suggest to stick to MAC. Besides, there are ways to decode through errors. You get a glitch on the audio, but most of it is fine. For fast checks, I use QuickSFV to generate MD5 sums of the files, and check after I burn.

I use par2, quickpar (links above) and and ICE ECC to generate error correction files. ICE ECC is faster than PAR2, and you can generate correction for full directories, but if the filenames aren't the same when it checks, it doesn't recognize the files. It is a problem specially with long filenames, which nero truncates when burning. Also, if you change the file name or directory structure before burning, or miss one file, it will complain when regenarating the file. However, PAR2 is easyer to use at command line, which is fine to to program that uses so much time. Make one script (or bat file), let it run, and go to sleep. I have not been able to use ICE ECC from command line, but I didn't try very hard. And PAR2 files will be recognized even if the name is changed. Besides, ICE ECC has a very handy option, to fit the amount of recovery blocks to a certain size, say DVD or CD. I didn't know about dvdisaster, I discovered on this (lengthy) thread, but seems worth a look. I still prefer to keep recovery data on the DVD, thought, than make it after I burn.

I burn my rips to a DVD, for a lot of reasons. But when looking for my 100 and counting back-up DVD's, I am considering buying an external HD case, and a big drive. In the long run, will be cheaper. I store a copy of ICE ECC executable (it works without installation) and the MAC command line encoder on the DVD, in case they dissapear.

I think RAID's are cool but overkill for what you want, and I particularly don't trust them very much. The biggest problem is if you'll be able to read the media, be it DVD, CD, or even SATA drives for very long. I once had to find a 5-1/4" drive to read a floppy, and it was quite hard. I believe that with DVD's or HD's you won't have to change the medium so often in your lifetime, but one day you'll have to move the data to a new medium.

This post is long, the thread is even longer. I suggest to make a new thread about which is the best medium and recovery strategy for archiving.

please do not add yet more misinformation to this thread. FLAC was the first format to have MD5 checking (it had it from the beginning), and it has frame CRCs (from the beginning). later MAC added md5 checksums ala FLAC.

Josh

I have no reason to doubt tht MAC stores an md5 checksum of the decoded data, but I have never worked with a program that actually uses this (EDIT: unless MAC uses this upon decoding or verification).

The MAC 4.01 beta GUI does have the "Quick Verify" option. But based on how quickly it works, there is absolutely no way that the md5 checksum it uses is for the decoded data. Rather, I am certain that it is a checksum on the encoded data.

I do find this useful since it is a very fast way to verify whether an ape file was corrupted. I've gotten into the habbit of verifying what I burn to DVD and MAC's quick verify has made this very easy.

EDIT:
I hope you're not trying to say that MAC can encode or decode faster than WavPack or flac, cause that most certainly isn't true!

Sorry, I was just following from another post on this thread. And I did not know that FLAC had CRC's stored for each frame, it wasn't mentioned before on this thread either.
Now, a question: when you test a flac file (from command line, flac -t <flacfile>) is the CRC of each frame verified, or just the md5 for the audio stream?

Most likely both are tested at the same time.

both, otherwise why have them? test mode is the same as decode mode except no output file is written.

"Normal" Iso-CDs have a third (or second, depending how you count) layer of error protection/detection: Every sector stores 2KiB of user data, plus a 4-byte checksum and 276 Bytes of ECC data. This is an addition to the 8 Bytes ECC data per frame (one frame contains 24 bytes of user data) that Audio-CDs already have for C1/C2 error detection/correction. So a read error from a CD is quite unlikely to be missed by a good drive, and most small errors (especially scratches from inside to outside, small holes) wil be correctable. Circular scratches that destroy long parts of the track in a row are much more problematic, so the tip printed on the box is important: Do not use circular motions when cleaning a cd.

Some of you listed some possible solutions. None of them are that simple and are pretty time consuming. I'll try to see if I can implement them. Using raid 5 or something similar seems to be the simplest solution. Anyway, thanks for the info.
I hope that CDs will be obsolete very soon and we can finally start using higher quality DVD-audio or SACDs. That will mean I will have to rebuild my entire library yet again. Right now I am in the process of rebuilding it because when I started about four years ago there was no one to tell me to use lossless format. As a result I now have several thousand crappy mp3s (at 320). Sometimes you have to hate technology!

OMG here we go... again...

You seem to know what you're talking about.

How can you be sure that your lossless files will be "really lossless" and not just "Monkey lossless"? SCNR

Btw, didn't you say that "this thread is over anyway" in post #42? So, stop the trolling now.

Dude, we join forums to share verifiable knowledge and to learn from each other. Please adopt an attitude of learning, or else this forum can tell you that your presence is not needed. As it is, the only thing we are learning from you now is that trolls come in all shapes and sizes.

C'mon, it's not fair to tell him to stop just because he started to reuse different stuff to refuel the dying embers of this thread!

Well .. he said thank you.

Then again it's the most expensive and noisy solution (but that can be OK for some)


That's the trolling again ... I'll bite just once.
I hate to burst your bubble but the Hi-Rez audio formats are dying, there might stay an niche market for them for a while. Download Music shops are more likely to replace CD with mostly lossy (often DRMed) "crap" as you would call it.

It's sad that only a few shops sells lossless files. Then again you can question why 24bit/96khz files are not available, since we don't have the CD medium. Another thought; could lossy 96khz files encoded from a 24bit source sound better than the CD medium we are used to? Personally i've not been able to do such tests myself yet

What's your problem people? You'll turn anything I say into personal matters. Only a low life can constantly attack someone, especially in a pack, just to feel better. If "trolls" come in all shapes and sizes, this thread has indeed showed it, which ironically included one of your moderators. Grow up. Thanks for the warm welcome, but no thanks. Good bye.

And off to Head-Fi he is...

Ah, not so much any more. The Infrant ReadyNAS NV+ (four drive SATA RAID-5) is ~$600 for the empty enclosure and with the NV+'s new fanless PSU, it's pretty darn quiet (assuming you purchase relatively quiet drives).

-brendan

Do you think the "Excelsius" over there is "our" Excelsius XS"?
If yes, then seanyseansean was not right in post #26

(the other one destroyed his tweeters by listening too loud - but of course his speakers are cap.)

Sometimes the only thing I want to do is cry.

No, you are. We are merely trying to enlighten you but you insist in not listening.
You should know that when you repeatedly post unverified information you are breaking the forum rules and most likely that will be disputed by other members.

WE'LL MISS YOU

To be honest, I'm surprised this thread lasted so long.