I'm in the process of backing up 500 CD-DA discs in FLAC format (using EAC). I ran a shell script to test (flac.exe -t; flac version 1.1.1) the 4000 FLAC files on my hard drive so far and found that flac.exe reported errors for 36 files. All errors are the same:

<filename>: *** Got error code 0:FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC

The files decompress fine, but now I'm not sure if I got the exact WAVs back. What does this error message mean, and how severe is the error? Moreover, is there a way to avoid it when compressing WAVs to FLAC in EAC?

if a file tests fine and the md5sum is non-zero (do "metaflac --show-md5sum file.flac") then you got all the audio data back. not sure what is causing the error though. if verify was on during compression and there were no errors then, I would guess hard drive corruption. unless repeating the test/decode on the same file yields different errors each time, in which case it is probably bad ram or overclocking.

Josh

Josh, thanks for your reply. It's always a delight to get an answer from the developer himself.

I tested all md5 sums and they were all non-zero. That means I've got files that fail the flac -t test, but have non-zero md5s.

All FLACs were encoded by EAC, with the following additional command line options:

-T "artist=%a" -T "title=%t" -T "album=%g" -T "date=%y" -T "tracknumber=%n" -T "genre=%m" %s --best --verify

That is, verify was on during encoding. AFAIK EAC does not keep track of FLAC encoding errors, so it's impossible to say whether they encoded without errors. I have always assumed they did because I have the "check from external programs return code" enabled in EAC's compression options and never got a warning.

I think I can rule out hard drive corruption because the bad FLACs occur on two different drives (Seagate/NTFS and Lacie/FAT32). Repeating the test/decode on the same file yields the same error each time, so we can can also rule out bad ram or overclocking.

So the perpetrator must be the encoding process performed through EAC. Could you perhaps explain what generates the FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC error during encoding?

you mean during decoding right? that's a decoder error. it means the decoder was looking for a sync point that begins each frame and there wasn't one. usually caused by missing or corrupt data at that point; if so, there can be frame losses.

I don't know what else could be going on. when a wave file is encoded with --verify and it passes, flac has already done everything that decoding would do. so decoding the exact same flac file again should not produce any errors unless a different decoder with a bug was used.

to debug any more I'll need a sample of a file that is giving the error.

Josh

Yes, the error that I get is a decoder error. What I meant is that the error in the FLAC must be created when EAC is processing the encode queue. EAC does not log the queue processing, so it is impossible to track down failed encodes afterwards, even if --verify was used. What would be the best way to provide you with a sample FLAC?

the best way is if you can host it somewhere accessible via http/ftp. the sourceforge bugtracker (http://sourceforge.net/tracker/?atid=11347...478&func=browse) allows uploads but has a size limit that is usually too big for flac files.

I guess I'm still confused how a file can decode fine but repeatedly give sync errors when testing since they do the same thing (testing just doesn't write the wave file).

Josh

OK, I will upload a buggy FLAC tonight. Not sure if the file actually decodes fine. Decoding produces a WAV with non-zero md5 sum, but since I don't keep md5 files for ripped CD's I can't compare md5 checksums. I tried reproducing EAC's CRC checksums (see http://www.digital-inn.de/showthread.php?t=28754) but that turned out to be not so easy.

I had an interesting situation with the same error.

On two of my FLAC's (using the latest EAC and the latest FLAC codec), i received that error (the warning pops up in foobar, but it plays fine). it also shows that error when i "test" the track with the FLAC frontend, and also when i decode it (it decodes fine, but the warning still shows in the window).

I then proceeded to re-encode it using EAC from the original disk. I received the same error, so it was something that seems to be inherent in the wav file extracted from the CD.

THis happened on two tracks, on separate disks. And both times, i tried to re-rip it from the original source and it still gave the same error.

I just decoded it and re-encoded it back to FLAC, and it is fine now

any idea what it is? looking at the situation, it seems it is not data corruption.

no idea.

to make sure it is not flac, start with a wave file and encode that. if you don't get the same error every time at the same place with the same file it is unlikely to be flac.

Josh

I looked at your files quickly... the sync error seems to be caused by the ID3v1 tag at the end of the files, so it's not affecting the decoding. flac is supposed to be id3 tag aware so I'll look at why it's reporting a sync error in this case. the audio is fine though.

Josh

Thanks Josh, that's good news. I guess now I don't have to re-rip the source discs for the buggy FLACs.

OK, found the problem. it is a decoder bug that is exposed when there is an id3 tag at the end of the file AND the last frame of the audio stream has a certain blocksize. integrity of the audio is not affected but the sync error is annoying. no need to re-encode either.

it's fixed in CVS and will be in the next release.

Josh

interesting. yea it has happened to me a bunch, and i just decoded and reencoded (which stripped the id3 tag, i guess). i did it because foobar always pops up a message and i wanted it to go away

so a fix would be to set EAC to not "add ID3 tag", huh? in fact, the ID3 tag really does nothing and has no place/purpose in a flac file, cuz it uses a different tagging system?

yeah, if you can avoid id3 tags on FLAC files, do it, use native FLAC tags instead. the spec does not require compliant implementations to support id3. even though the standard tools do there's no guarantee it will stay like that.

Josh

Josh, many thanks for sorting this out. Glad to hear I don't have to re-rip/re-encode.

Cheers,
Zennon

I was wondering if you could help me out. I have a bunch of flacs (with no access to the original sources) that I encoded but give the LOST_SYNC error when verified. The strange thing is they will play fine in Media Monkey if a non LOST_SYNC file is played first (?). Anyway, is there anyway to fix the files? flac -d doesn't work. It gives the same error, and if you tell decode through errors is says something about the source file being to big for a single wav chunk. Note: Yes, these were originally encoded with an id3 tag, silly of me...

Any ideas on how to fix the files? I do have a 4MB sample (its an intro to a "song") that I could email over if anyone needs to see it. Feel free to reply here or email me at maps1@nesurf.com.

How about using an ID3 editor to just chop the ID3v1 tag off? A dumb editor which doesn't recognize FLAC would be enough, in which case renaming the .flac to .mp3 will allow you to edit tags.

Alternatively, you could chop 128 bytes off from the end of the file for the same effect.

If someone would write a CLI Windows utility to remove any/all IDv1 and IDv2 tags from FLAC files that would be a great asset. Unfortunately too many tag editors and audio programs that convert/save to FLAC are using IDv1 or IDv2 tags today inside FLAC files. Some use both IDv tags and Vorbis tags, others just use IDv tags.

I have written to some developers about this and about the Spread FLAC campaign, but many audio apps authors appear nowadays to reside in China (about 80% of them which I have written to), and it seems they don't know much about the FLAC format, let alone about the need to use Vorbis style tags inside FLAC files. Maybe Josh should publish the FLAC developer docs in Chinese

It wouldn't hurt, Josh, to put a real emphasis on your site, in your docs and in your program code comments for developers to use Vorbis tags only and to stay away from using IDv1 or IDv2 tags inside FLAC files that their programs edit or create, as many developers are so used to the MP3 tagging formats. Just my 2 cents.

Well, have you tried Case's Tag?

if the lost sync is at the end of the file it could be tags but I'd have to see the file. I'll email you for the sample.

as for the chinese docs... good idea but someone will have to volunteer for the translation. our russian version of the FLAC site has fallen out of date too for lack of maintenance.

and yes, now that vorbis comment adoption has increased it's probably time for more stern wording on the site. I'm hesitant to pull support altogether because id3v2 is being used in some products for album art.

Josh

I believe I'm right in saying you can either remove all tags, or just ID3v2, using Tag.

You can't specify to just remove ID3v1.

Obviously it would be fine to remove all (--remove) if there were no VORBIS tags present as well.

got the file. it's corrupted. I can see in a hex editor that there are big blocks of zeroes where the middle of the id3v2 tag and beginning of the FLAC stream should be.

flac -dF is currently not able to decode through errors like this.

Josh

You can --force APEv2 --remove all (syntax might be wrong, but you get the idea.)

Can't it do the same with FLAC?

I uses Case's Tag in a batch file to solve this problem. The batch file works as follows:

1. Creates an empty text file with the .tag extension
2. Uses Case's Tag to:
i) copy all tags from the flac file to the .tag file with --force ape2 & --nocheck
ii) remove all tags from the flac file
iii) copy the tags from the .tag file back to the flac file. Case's tag will automatically use the right format.
3. Deletes .tag file

One caveat: Do not stop the process in the middle of removing the tags from the flac file as the flac file will likely become corrupted. I.e. backup your flac files first and delete the backups when the process is done. I haven't incorporated the backup process into the batchfile yet but I plan to when I get some time.

I've tried using --force flac --remove in a single command but for some reason it often fails to remove the ID3V1 tags.

So no way to fix it yet? No flac repair tools out there yet huh? Will using Case's Tag as stated above have any effect?

By the way, I repeat, the files play and sound great in Media Monkey (if I play a song that does not have this problem first), so I'm a little confused by your statement that the begining of the FLAC stream has a bunch of zeros in it...

P.S. I forgot to mention, that if there is a way to repari the file I do not mind loosing any of the tags as there are a plenty of programs that will be able to retag them from their filename/directory struction.

no, the damage is too severe. it is possible to recover frames on a frame-by-frame basis totally ignoring the metadata but I still have to add that functionality to -F.

another FLAC developer has written a frame decoder that includes recovery but I haven't tried it. it may work for your files:
http://phoenix.inf.upol.cz/~lichvarm/affd/affd-0.1.0.tar.gz


not on this file I don't think. even the id3v2 tag is corrupted.


can't explain that.

Josh

Thanks for the help. I'll check out affd. Keep up the great work!*

*Particularly if it includes -F functionality that totally ignores metaflac or a repair tool in the future.

I have the same problem. 3 songs out of 7 albums gave me that error message when playing them in Winamp. When I re-ripped those cd's it worked fine.

Still have those files... ...still haven't found a way to fix them (flacdec says no valid frames). Anything on the horizon?

Recap: These files ARE corrupt. But the "music" part of the file is 100% O.K. (it will play GREAT in MediaMonkey if non-corrupt files are played first). The corruption likely occured due to ID3 tagging and re-tagging.

CORRECTION: flacdec was able to recover each one of my corrupted files!

I just happened across this thread, but coincidentally I just tried the tool that Josh mentioned above. In my case I am sure the files are corrupted due to a failing hard drive (that I am frantically trying to back up). In my case it is impossible to recover the exact original audio content, but my plan is to repair as best I can and to make note of the problem. My logs for one corrupted track are below:
=-=-=-=-=
Test:
flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.

Track03.flac: testing, 79% complete
Track03.flac: *** Got error code 0:FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC
Track03.flac: *** Got error code 2:FLAC__STREAM_DECODER_ERROR_STATUS_FRAME_CRC_MISMATCH
Track03.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_SEARCH_FOR_FRAME_SYNC
-------------------------------------------------------------------------------
Recovery with affd:
F:\FlacRepair>flacdec -r -f wav -v Track03.flac
Track03.wav.00000 [44100Hz 2ch 16bit], fsn: 0
1682 [4608] -> 1684 [4608]
missing 1 frames
samples decoded: 7755264
Track03.wav.00001 [44100Hz 2ch 16bit], fsn: 1684
samples decoded: 1906848
-------------------------------------------------------------------------------
Lame-O repair attempts:
Track 3 wavs rejoined in Soundforge 8, pencil tool used to smooth splice transition, minor "glitch/pop" audible at 2:55.865
Track 3 Version 2 - Crossfade on .05 seconds: no audible "glitch", but slight "shortening" detectable in word "Millionaire".
Track 3 Version 3 - Crossfade on .01 seconds, still no audible "glitch", "shortening" less noticeable.
Track 3 Version 4 - Crossfade on .001 second - Best, still no "glitch", still slightly audible.

Maybe I could have obtained better results in repairing using a hex editor?
(edit - well I tried to answer that for myself, and even without knowing the exact details of the compression algorithm, it is clear that if certain information concerning a frame is lost/corrupted, then the whole frame will be undecodeable - "no way back home").

ok, I just checked in a fix for -F that will allow flac to decode even if the metadata is gone. KingTubba's file now decodes completely, although since the STREAMINFO is gone there's no way to do the MD5 check or tell if there is audio missing off the beginning.

Josh

[quote name='jcoalson' date='Jun 13 2006, 19:46' post='402734']
ok, I just checked in a fix for -F that will allow flac to decode even if the metadata is gone. KingTubba's file now decodes completely, although since the STREAMINFO is gone there's no way to do the MD5 check or tell if there is audio missing off the beginning.

Josh


Hi, when will this new feature be available to use? It sounds like exactly what I need. I have about 400 albums that were on a hard drive that become corrupted. Although I managed to recvover all the individual flac files fr the drive, I have just discovered that I cannot actually play any of them and show no tags in any editor. I get the ERROR_LOST_SYCN when flac trester is run but the file looks to be the correct size, so I am hoping it is just the lack of metadata that is causing the error. thanks.

hopefully some time this month. most of the new major features are done.

Josh

Is there something like single commit of this that I can patch the Ubuntu 1.1.2 source package with?
I also had sporadic bad blocks once on my HD once and am still encoutering broken files that I put into a "quarantine" folder (which holds 41 items by now).
Often the corrupt part was very small (like 4096b?) and for very precious songs I knocked in the end of the corresponding vorbis file in an audio editor and recompressed the file.

sorry, code is too different. there will be a beta available in a couple of weeks but the current CVS code can be used for recovery if you really need it.

Josh

affd was updated to 0.1.1 (last year, actually)
http://phoenix.inf.upol.cz/~lichvarm/affd/

added to RareWares/Debian


later

hi,

i have several flac files on my laptop hdd. several month after encoding them it appeared that there are errors in one of my files. it didn't play along, but after a few minutes the player jumped to the next track. after some research and testing here are the sad facts:

http://viktor.podzone.org:8080/tmp/tests.ods

it has 2 sheets, hda3 is my /home partition, sda1 is an external backup drive. some strange things:

1)errors are different with exhibition cd2
2)on one drive it exists and on the other it doesn't (the "shivers" track)

in theory these files on the 2 drives are the copies of each others, thats why its so strange.

my previous hdd popped out, i replaced it with a new 160gb samsung drive. then it seemed to work fine, until these corruptions. BUT, other files (documents for example) don't seem to be corrupted, i can access & use them.

so, what's happening with my flac files?

if you need samples, i can host them no probz, just let me know what you need. thanks.

i think ive realised the source of the problem. once short before i replaced my hdd i copied my stuff to the new hdd, so the files could got corrupt on my broken hdd and somehow the command line didnt show read error on all errors (coz there were many errors but i still could copy a lot of files without warnings).

i guess this gotta be a reason... im gonna rerip my cds (fortunately they are genuine and i have them at home) and encode them with flac 1.1.4. also, ive already introduced a new rule for my music folder:

chmod 0400 music/*

this way no crazy app can f*ck with my files ;-) some time ago i just realized that "unkonw artist" tags appeared randomly in my files... but since ive used several different players that time i couldnt find out which jerk was joking me... so whatever, it seems to be hard to keep my collection clear

offtopic: what about flake?
"Increased compression and dramatic speedups for both encoding and decoding"
is this supposed to mean that the improvements made by justin have been merged into flac or are these independent changes?

they're independent, flake is for the most part not based on libFLAC.

i see.

i have now located the corrupt files, moved them to the broken dir, and reencoded the old ones with 1.1.4. retested everything, and yet it seems to be fine. i will run tests regulary, and hopefully no further errors appear

btw, may i suggest to make the -V option to be default?

also, id like to thank your work, ive been using flac for quite some time, and i really love lossless (especially coz i collect audio cds and wanna achieve the best quality - what is better then lossless ? :-)

Hey viktor... I don't want to jump over your post, but I am having similar problems with my flacs. Not sure if this is hardware or software, but it sure seems like we need to regularly run tests on our flac library to check for errors.

yea i surely will, at least to recognize hdd failurs in time. this is the simplest way.

its more likely that you also have hw problems.

i cant be sure in my case, but now im waiting for any error to occur again. in theory my hw is fixed now, so it shouldnt.

well see.

anyway, havent you had any hw problems recently?

It looks like it might be a problem with the firewire connection. I have moved the ext hdd to a USB2 port and will correct the corrupted files. There was also a problem with a previous release of Media Monkey that might have also corrupted some files. The problem with these types of things is that corrupt files are "hidden" until you try to use them again. In my case, I was playing around with Synthetic Soul's flac-verify.bat file and just happened to stumble on a corrupt flac file. Pure chance.

So now I have created a batch file that I can schedule overnight to check the entire flac library directory and flag any corrupted files into a log file. I have also created a batch file that can check an album directory, create a log file of the complete flac -t output for every track and store this in the album directory. It also creates a .swf file for the directory. Once this is done, I'll also include a check of the .swf file in the library check procedure to see if throws up any errors on all other files (jpg, etc) in my library.

The bottom line is that, if we're wanting to archive our CD collection with a lossless copy, we must have a system to regularly test the archive so that we can catch problems before they get out of control. If I had been able to see that I was getting corrupt flac files when I only had one or two, this would not have become such a big problem. I should also have checked my entire flac directory when I moved it from one external hdd to another before I formatted the source hdd.

Lessons learned the hard way I'm afraid.

hmm i think its not that serious.. once you located the source of problems. however, for me its simpler to check my flac files since i have only two dirs of music, in both of them i run a flac -t *.flac and voila! hope the best...