Hiya,

I keep all my binaural recordings on my machine.. I've started squashing them into APE files ready for backup on dvd-rs ... i just copied some across to my dvd burning machine only to discover on verification that some were corrupt... testing the original files seems to be ok so worryingly something is going wrong in transmission across my LAN

anyways it got me to thinking about how fragile lossless audio is... compared to waves say... i assume some random error corruption in a wave would result in some terrible noise for a period.. but around that the sound would be ok wouldn't it?!!? how do lossless codecs work on this... am i right in assuming that monkeys audio saying CRC failure means that i can't use the file at all, or does it just mean that the audio that is output fails the CRC somewhere!?!?

DVD-Rs are cheap so if lossless codecs are more fragile i'll just use waves, or are these any better!?!? Thanks for your help

Jim

> DVD-Rs are cheap

I think we have identified your problem.

Yes, there's a tradeoff between compression ratio and resync span. A checksum error doesn't tell you how damaged the file is, maybe it'll just decode the last bit to a '1' instead of a '0' or vice-versa.

You might want to stay with monkey's audio, use good DVDR and/or burn them as slow as possible, and look into adding redundant parity data ("PAR-archives").

sure or that checksum error could mean that its totally trashed!?!? its frightening that copying the file across the network resulted in this corruption... i have so many troubles with my network sometimes. more worrying was a file after that failed completely and got some horrible windows errors.. could the hdd be on the way out all the more crucial to get these recordings backed up whilst i can



Ok will do a search... can Monkeys Audio do this!??

I just wanted to know what formats would do with spurious error data in the middle of the stream (say a few megabytes (sure a huge bad case scenario)) ... I was assuming so long as the headers were alright plain waves would survive...

I guess the big issue is this kind of failure when i get it on optical discs usually means you can't even get any part of the file .. I dunno.. I will probably make 2 dvds of each dvd so i have a redundant backup of each...

just wanted to point out that at the moment no dvd media is involved... i will be using good quality discs and will burn at 1x.... I have a lot of success with video playback with these discs on normal players which i think is testament to them being burnt well and of good quality in the first place.. Zero Defex is their name..

It might be wiser to use flac instead, because a single error in a flac file will still make the rest of the file decode correctly.

My recommendations is using flac and fill the rest of the CD/DVD with PAR2 recovery data.

the matroska container will allow the user to spend a flexible amount of bytes for error protection, but we have no plans for Monkeey audio in MKA yet, due to the unusual licensing that it has.

Expect Wavpack4 and FLAC to be the first supported lossless audio formats in matroska container .....

ahh i understand so this error protection (matroska aside) is seperate from the actual files themselves... will try and look into this.. how much space is needed for good recovery?? so flac is good at recovering.. hmm (as i have all apes at the moment)

ok i am playing with quickpar now... what kind of redundancys do you think is needed to be effective at all... i'm testing it now on a corrupt file, paring a correct one and seeing if it can repair the corrupted one... its only 300meg file, and yet its going to create 30-40meg par file.. this will scale up to needing quite a lot on a dvd... like nearly 800meg or something!??! not done the math.. quite a lot but if it protects then fair enough... how effective would 200meg for a full dvd be?!? its surely not magic, i guess you are gambling as to whether a fix will work or not the smaller restore files you create..

I spotted these programs for cds, CD Data Rescue or CDCheck... getting files off them when they are screwed, will these scale up and work with DVDs??!? not clear on their homepages as to whether they will or not...

I am having some success with quickpar tho i am also getting really strange occurences where its almost as if something is being cached...

if i have in directory A

file A
par files for file A

and also i have in directory B

file B (which is a corrupted version of file A)
par files for file A (copied across)

then the restore should just work!?!? sometimes it just says it cant find any blocks in the data files... in the end i had to move all files to other directories, restart the program to get it to consider it... anyone worked with this before got any idea why this would do it?

I think the problem may have been that in directory B from before

i had

file B (corrupt file to be worked on)
copy of file B (i wasn't sure if it was going to be backed up and wanted to keep the messed up one for testing smaller and smaller restoration redundancy)
file A (a previously fixed version of this file from a larger restoration redundancy set)

it came up first time saying misnamed or something and bizarrely it picked up the other files with different names, as well as the one i wanted it to look at.. it was then that no matter what i did it was confused and wouldn't work..

it managed to restore my 360meg APE file with restore files only up to 4meg in size at like 2% redundancy or something... quite impressive not sure if i'd risk going that low for real.. I dont understand how this all works and what the risks are.. the tables outlined elsewhere on features aren't too clear to me either, must just be being stupid..

the corrupt APE tho is not decompressable via monkeys it stops and u have no wave that is awful... if i lost my hour long recording to effectively 10 seconds of crap in there i dont think thats on at all...

Yes QuickPar uses a cache, and I too have found it annoying...

Quote from the my post about it here:

http://www.hydrogenaudio.org/forums/index....ndpost&p=129397

Thanks Madah, makes sense... it seems to mostly work as is tho. i just got worried, i'd read that paths werent stored but then quickpar has a path option in there...

anyways just got it working with the lowest redundancy of 0.1% .. that was enough to fix this corrupt file of 360meg with just 400k of backup data... WOW... i'd love to know how much of a risk you are taking at such low redundancy rates... i may have to post on the other post where i saw it but the table just wasnt clear...

it was in here

http://www.hydrogenaudio.org/forums/index.php?showtopic=9236



anyone like to decrypt it for me?! i've left the options untouched

for yenc 3000 lines

384k block size
402k article size

are these fine for DVD sized 4.3gig backups (tho i guess i'd need to have 4gig of files for 300meg redundancy, or perhaps even more)

[SIZE=1]ok well it looks like i am going to have to move over to flac for my serious archiving.. that coupled with some par2 seems to be at least nice protection... i am curious with flac tho, what will happen when a part of the file is corrupted.. when you got to flac decode it to wave do you just get all the valid data, joined together with the bad segments of audio just effectively cut out of the stream.. if its like this then fine... sounds nice and easily workable, and at least hopefully i would still get a nice amount of material left in the horrible case of having corruptions beyond par2 repair.. is it that easy or are there special things needed to be done when creating flac files to be more error resistant/recoverable ... and are there special options required to extract from corrupt flac files.. sorry its just i've not really used flac much and dont know how to use it


edit: ok well it looks alright.. looked at the help.. force thru errors makes sense and other than that i dont think i need anything else.. sector align borders dont look obvious, or neccessary (ok i dont get it i assume quality 8 is the best option for highest compression, unless the time taken to do it is hugely longer for small increases.. I used to use HIGH with monkeys audio as that was the highest setting that still had decoding with 1% cpu..

and finally...

i will happily decode all APE files to waves to then flac them up, but isn't there something i can use that will go from APE -> FLAC directly ... i guess foobar could but i've never really set it up to do any encoding... is that the only option...!? is it flawless, able to verify along the way etc...

Cheers... quite a lot of enlightenment in just a couple of hours on this one all i need to find out now is what people are using for their redundancy % on average, and i'm all set Thanks

dbPowerAmp will do that for you, and it preserves tags.

I did a lot of APE-> FLAC and some FLAC-> APE with foobar2000 : all tags (>20 different fields) are preserved too. I did some bit-to-bit comparison in order to check the perfect identity of transcoding, and no difference was detected. It seems to be safe.
I used Case dedicated componant for it.

OK, since I created that table, I'll interpret it for you. :-)

The table was designed to compare how much PAR2 data is actually required to achieve the same level of protection and ability to repair that a particular amount of PAR1 data would give you. It assumed that you were posting RAR files.

The calculations assumed that you had the following:

Original Files
--------------
Number of RAR files: 30
Size of each RAR file: 15 MB
Total size: 450 MB
Article posting size: 0.5 MB
Total number of articles: 900

PAR1 files
----------
Number of PAR files: 5
Size of each PAR file: 15MB
Total size: 75 MB
Article posting size: 0.5 MB
Total number of articles: 150

PAR2 files
----------
Number of recovery blocks: 5
Size of recovery block: 0.5 MB
Total size: 2.5 MB
Article posting size: 0.5 MB
Total number of articles: 5

i.e. you are posting either 75 MB worth of PAR1 files or 2.5 MB worth of PAR2 files.

The table assumes that under normal circumstances articles posted on UseNet are lost entirely randomly and that they are lost at some particular average rate. What that rate is varies from time to time and from system to system.

Now supposing the rate at which articles were being lost averaged at around 1 in every 1000 articles. The laws of probability tell us that some of the time there will be no articles lost, most of the time one article would be lost, and that occasionally larger numbers of articles will be lost.

It turns out that for this example, the probability that enough articles would be lost that you would require 6 or more PAR1 files for repair is between 1 in 1,000 and 1 in 10,000. i.e. between 99.9% and 99.99% of the time 5 PAR1 files are sufficient protection.

You will note however that the table also tells you that 5 PAR2 recovery blocks would also be sufficient for repair between 99.9% and 99.99% of the time.

The top row of the table tells you that when the rate at which articles are lost is as high as 1 in 170, you only have a 50% chance that 5 PAR1 (or 5 PAR2 blocks) would be enough for repair.

If you look at the top row of the earlier table I posted in that topic however you will note that it indicates that when you post as much PAR2 data (in total) as PAR1 data, then even if the average rate of article loss was as low as 1 in 10, then you would only have a chance of 1 in 100,000 that repair would not be possible.

I would like to hear what others do to protect their FLAC backups. How many par files do they create per album? If they ever had problems, was their method successful?

Thanks PeterBClements i kinda get it, fairly confusing stuff... quite a lot of variables to consider... i ended up going for 4gig data, 400meg recovery files.. i ran into some trouble

http://www.hydrogenaudio.org/forums/index....howtopic=13500&

damaged par files straight away after making them.. I've yet to try making those files on another machine but have tried it twice with that machine, both failing.. its a brought (not built by myself) p4 2.8ghz machine.. that i was always kinda relying on being high-ish quality (having not had my bumbling hands all over the equipment on build, static-ing out some chips here and there) but peoples suggestions on there are just that, that its a memory or some other hardware problem possibly... as the authority on all this could you have a quick look and make sure there isn't anything in the software that could cause this to happen... thanks

Can we summarize, which lossless codecs produce totally corrupted files , if there is a single error in the file, and which codecs would produce only a single click inside, if corrupted data in middle of file ?


totally corrupted by an error:

Monkeys ape (but there a switch in winamp deocder to ignore errors !),





Not totally corrupted an error:

FLAC
Monkeys ape by decoding in winamp with switch







What about ?

Wavepack Lossy
Wavpack hybrid (lossy + correction file = lossless)
wavpack Lossless

Optimfrog
Optimfrog Dualstream

I would think about replacing your network cards. I had a network card fail on me once, but it took a while to realise because everything looked ok. But every time I tried to copy files across it they got corrupted and trying to play LAN games resulted in me getting disconnected everytime.

Worth checking out I think, network cards are dirt cheap these days.

Current version of WavPack gets totally corrupted by an error.

WavPack 4 (that is in Alpha/Beta phase) will be error resistant.

No, Winamp+in_APE.dll+"Ignore Bitstream Errors", but 1 bit error trash ~5sec.
In general most codecs split input in frames, soo it is decoder choise to decode remaining frames.

Does that mean you're going to add flexible redundancy regardless of the actual codec?

Yes.