Please wait a bit! Joseph Pohm has just reported some decoding errors i want to fix first.

No need to worry: I am quite sure that i allready know the reason and the fix will not significantly affect the compression performance. Somehow i allready had the feeling, that some slight modification necessary for the PreFilter could give some trouble. But my test corpus and the sample files of Josef decoded well hence i released the V0.08.


Nice to see some improvements with the PreFilter.

Thanks

Thomas

The tests at home have nearly completed. My decoding batch files decode the YAA files and record the time taken; once this is done they then check the MD5 hash of the decoded WAVs against the MD5 hashes of the source files (using FSUM).

This morning I noticed a problem with file "13.wav". I bought the file into work and have just encoded and decoded it again with all settings. These are the results:


As you can see, this demonstrates what I briefly noticed at home this morning: There is an issue when encoding this file with the preset "Extra".

I didn't have the time to do a thorough examination this morning, and I didn't have all the results in, but with the data I did have this was the only file and the only settings that had a problem. I have run the same decoding batch file on the "FLAC" corpus I have here at work and no files or settings have an issue. Therefore, out of 78 files encoded using 11 settings this is the only evidence I have.

The Yalac_diag.txt report can be found here.

Let me know if I can provide you with any more information.

Too late, by the time I get home today all data should be in.

Ah, I'm glad it's not just me (see above, posted before I had seen this response), and I'm glad you already have an idea what might be happening.

Yes, the Fast pre-filter definately seems worth a go. Until I can look at my timings then I can't see what performance hit there is, but from looking at Shade[ST]'s results it doesn't appear to be much at all, if any.

It seems as if i have fixed the error. I could only check one of Josef's files. I will sent him the corrected version and let him try it. If everything seems ok, i will sent the corrections to the other testers. This error should not have affected compression by more than about 0.01 percent.

The error would be more likely to show up on preset EXTRA, hence i am quite confident, that you are suffering from the same error.

Sorry for the inconvenience

Thomas

Ok, thanks Thomas. Looks like I'm starting my tests all over.

FYI, I have just been looking at the spreadsheet for my "FLAC" corpus, and here are a few more figures for you:

Very interesting! It lightens me to see, that not only Josef's files (which my encoder sometimes finds very special...) can achive a significantly bigger advantage (your best case) than the average.

The main purpose of search level NORMAL is to avoid using the prefilter, if it would hurt. But it's effect seems not very significant.

BTW: I just sent Josef the fixed V0.08a.

Edit: Just learned, that there is a difference between "lightens" and "enlightens"...

http://synthetic-soul.co.uk/comparison/yalac/ now compares 0.07 and 0.08 (?all=1 to view all versions tested so far).

http://synthetic-soul.co.uk/comparison/lossless/ now compares 0.08 with other codecs (?all=1 to view the additional prefilter settings).

Plans for V0.09

Before i forget:

Many thanks! It's nice to see some improvement with the PreFilter.

Actually i wanted to pause the work on compression improvements after the introduction of the PreFilter and instead begin the work on the file format (error tolerance etc.) to prepare a public release. But some analysis of the PreFilter results has changed this. In addition to its main purpose the PreFilter seems to generate some nice side effects, which help compression in another way than i would have expected. After some thinking i came to the conclusion, that this (and possibly bigger) improvements could be achieved outside of the PreFilter by some other modifications.

A quick and dirty implementation looks promising: About 0.20 percent improvement for preset FAST, 0.30 percent for NORMAL! The encoding speed penality is small for those presets, but because it grows with higher predictor orders may be significant (about 10 percent) for HIGH. The effect on decoding speed is very small.

Because the new method is in no way optimized yet, i would expect some more improvements soon.

But caution: My quick and dirty implementation may contain errors! There is a very small chance, that i am overestimating the possible compression improvements!

Sorry, i can not resist. I have to evaluate this for V0.09.

No need to apologize TBeck. There are lots of great lossless encoders available for free right now, so there's no hurry towards a public release. You do what you need to do to make your encoder the best it can be so that it becomes the new paradigm in Lossless compression!

I have a feeling that your encoder will set a new standard, so we certainly want you to be very methodical, careful, and thoughtful in terms of how you design it, rather than try to rush a release just for the sake of having it publicly available, then try to hack it later with improvements and have to struggle to maintain backwards compatibility and such.

I'm very impressed with your commitment and the consistency that you have been working on this. You're an inspiration for all developers, for sure!

Many thanks!

I appreciate encouraging posts very much (even if i am not always responding)! They are especially helpful, if i just spent several days of work for possible optimizations, which in the end didn't work...

And you are right: I should not hurry. Better develop a well thought design, instead of having to fix bugs or correct shortcomings of a too early public release.

Thomas

A little late, but I had was requested to submit the test results conducted on version 0.08a.

Using -c0 and -c2 switches show a tradeoff with ratio vs. decoding speed, as discussed earlier.

Now, returning to your previous discussion...

Current progress (V0.09)

There is still some more to do before the next evaluation release. But there will be no important changes for the performance. Therefore i like to present some data:



FASTEST encodes 60 percent faster than FAST. The compression penality is only 1.14 percent on this file set. Decoding is only 10 percent faster. I would expect a bigger advantage, but possibly my system is too slow.

NORMAL encodes 15 and HIGH 20 percent faster.

Sorry, i couldn't resist. I am quite happy, that i could make FASTEST much faster than i had expected.

I took the liberty of adding your previous 0/07 results.


No nasty surprises there. It seems to me to be an excellent mix of additional compression and speed in the faster presets. The addition of Fastest is extremely welcome, and its stats are extremely impressive. From my intitial understanding it seems what little compromises there are are all in the right places.

I'm looking forward to comparing it on my corpus; it seems to make it more real when using your own data/files. From past experience I expect to achieve very similar results though (you often accurately predict the benefits seen by my results when releasing new versions).

Ah, better than nothing I guess...

Thanks.


I hope you don't mind... I would like to increase the predictor order of FASTEST from 8 to 16, which makes encoding 10 percent slower:

On other test file sets this increase of the predictor order nearly halves the compression difference to FAST. And my P3-800 still encodes 56 * faster than real time...

Yow will be able to evaluate the speed advantage of 8 predictors via command line switches. If you find it important, i may change the preset again.

As it was with 8 predictors, YALAC looked like my perfect contender. I'm looking forward to testing it as soon as it's out, to see if results are uniform with what I've expected!

Peace,
Tristan.

V0.09 is done

Changes:

Compression algorithms:

- New PreFilter option "Sensitivity": Tries to limit the PreFilter usage to those frames, that benefit from it by a significant amount.
- New channel decorrelation method "Difference". While beeing nearly always inferior to the standard decorrelation method, it has two advantages: It's much faster and it compresses better, if both channels contain exactly the same signal (mono signal copied to both channels, seems to happen sometimes).

Presets:

- New preset TURBO, which encodes up to 60 % faster than FAST.
- NORMAL encodes 10 % faster and compresses slightly better.
- HIGH encodes 30 % faster and compresses slightly worse.
- EXTRA encodes 15 % faster.
- INSANE encodes 10 % faster.

Command line:

- New test cases to evaluate the PreFilter-Sensitivity-option:

-c0 = PreFilter off
-c1 = Low
-c2 = Medium
-c3 = High (same sensitivity as in V0.08)

Caution: The preset numbers have changed because of the insertion of TURBO!
0 is now TURBO, 1 = FAST and so on.

Bug fixes:

- Fixed the bug that Synthetic Soul has reported. The smaller frames of the new preset TURBO could easily reproduce it...

Other:

- Clean up of source code. Could give new errors...
- File format changed!

Release:

I hope to send the new version to the following testers within the next 24 hours:

Destroid
Josef Pohm
Shade[ST]
Skymmer
Synthetic Soul

Only reason for this selection: I haven't heard from the other testers within the last 10 days and i don't want to fill their mail box with new versions they possibly currently don't need.

Any of them can sent me an email anytime, and i will sent the current version.

What should be tested:

- Comparison with V0.09: Speed and compression performance.
- Try preset TURBO with 16 instead of 8 predictors: -p04. If this is not encoding significantly slower, we may change the preset.
- Not too important, but you could evaluate the PreFilter Sensitivity: Try preset HIGH with -c0 (off), c1 (Low) and c2 (Medium). No need to check -c3, because this is the default for HIGH.

Plans for V 0.10:

- Try to speed up the PreFilter.
- Possibly evaluate a new method to improve the compression.


Edit: EXTRA encodes only 15 % faster.
Edit 2: Added Skymmer to the tester list.

Edit: Never mind... You are comparing the new presets with the 0.8 presets, I was confused.
So what is the new distribution of compression speed?

I hit a disk IO wall encoding with TURBO, but note the lower CPU usage in addition to the speed advantage over FASTEST. Over all, all the presets encode notably faster. I was disappointed that the material I used was harder to reduce than I'd hoped, so even tenths of a percent can be important here.

For the next test I'll have my hard disk and RAM upgrades, we'll see if SATA 1.5Gbps does anything

edit: typo fixed, added other compressors' high scores

Destroid's results are greatly inspiring; I'll try the same experiment with an Edith Piaf and an Oliver Jones albums; From files that weren't previously mp3s, this time...

Peace,
Tristan.

Yalac 0.09 vs The Rest of The World (?all=1 to show non-standard (-pT4; -pH c-0; pH c-1; pH c-2))
http://www.synthetic-soul.co.uk/comparison/lossless/

Yalac 0.09 vs Yalac 0.08 (?all=1 to show all Yalac versions)
http://www.synthetic-soul.co.uk/comparison/yalac/

The IO throttling makes my results for Turbo inconclusive, although I suppose that is an argument to go for 16 predictors instead of 8, and therefore pertinent.

I must say I still like the idea of using 8, otherwise Fastest isn't the fastest it can be. That said, my IO-limited results are an argument against it, for sure.

I will try to add some CPU-only figures to this post ASAP.

Edit: And lo


Dig these in chart form baby.

NB: All MD5 hashes match, including the troublesome (for 0.8(a)) "13.wav".

Yalac Turbo, according to my corpus, compresses better than WavPack default and FLAC -8. It is also faster at both encoding and decoding.

Yalac 0.09 is providing better compression than 0.08 in all presets.

The improvement in High is superb: 0.4% better compression with a decent increase in encoding speed.

Normal compresses slightly better, and is again faster than 0.08. I would still perhaps like to see it encode faster.

That said, there is currently still a nice spread of encoding speeds (122.71x; 83.08x; 39.16x; 18.12x; 10.75x; 5.05x).

I really can't decide about Turbo. My heart says 8, but my head says 16. My stomach on the other hand is saying "Hmm... breakfast"!

Not according to the results so far:


You're spot on with the 30% for my CPU+IO results.

First, again thanks for your great work! And for providing such welcome results...

Currently i see no way to speed up NORMAL further without a significant compression penality. We could reduce the maximum predictor order from 128 to 96 or 64. This possibly would not affect your test corpus much, but makes a very noticeable difference on other file sets.


Yes. That's the way i wanted it to be: While you can not predict the compression advantage of higher presets on individual file sets, you should at least be able to easily predict the increase of the compression time.


That's a very healthy reaction! And possibly the second best after trowing dices for the deceision...


Well, it was hard do predict. High now uses only 192 instead of 256 predictors. Some files of my test corpus really don't like this. But your test corpus anyway does not benefit from higher predictor orders. And HIGH now tries the simple new difference method for the channel decorrelation, that has been added for TURBO. Possibly it helps sometimes. If so, then V0.10 may compress even a little better, because it too contains the Mid-Side variant of the classic difference decorrelation.

And i performed some very slight modifications on some encoder parts; maybe that they help a bit.


Oh yes!

The channel sensitivity option is a quick and dirty approach to solve this problem: With the default sensitivity (same as with V0.08) the PreFilter is even beeing applied, if it does not help compression at all. Unfortunately this decreases decoding speed.

Now i try to predict the PreFilter advantage and disable it (for an individual frame), if the predicted advantage is below a criterion (percent of improvement) set by the sensitivity level:

High: 0.00 = Use it if it does not hurt...
Medium: 0.3
Low: 1.00

But this prediction is not reliable. The deceision is performed at the PreFilter stage and the PreFilter is in front of the whole processing chain of the encoder. You can not reliable predict it's effect on the successing processing steps!

I have selected the sensitivity parameters by simply trying different values on my test corpus. There is no guarantee, that they work well on other file sets.

But on your and destroids test set they perform as expected:

HIGH (sensitivity): Maximum compression advantage but maximum decoding speed penality too.

MEDIUM: Should provide 50 to 70 percent of the advantage of HIGH, but should half the decoding speed penality compared to OFF.

LOW: Only use the PreFilter on the special files (of Joseph Pohm...), which benefit extremely from its usage. Your files are not among them...

With regard to Turbo, I have dug out a few CPU-only results for FLAC and WavPack to compare with Yalac.


Firstly, I was suprised at how fast WavPack -f is; I've never really paid it any attention before. I assume that it would be comparible with FLAC -2 or -3, at a guess.

But, to the reason for this table. I was very pleased to see both Turbo settings producing such excellent results, and that Fast isn't far off FLAC -0 (when comparing speed).

I was going to say that the table suggests that 8 predictors aren't required, and that 16 could easily be used. This does ring true, but you could equally say that another 0.3% compression is not required! When looking at the faster settings Yalac is providing much better compression than the codecs to which I am comparing it. The fact that Yalac Turbo is faster, at both encoding and decoding, than FLAC -0, and provides an additional 5% compression, is astounding. Considering FLAC's upcoming improvement, it is possible that Turbo will no longer compress better than FLAC -8, but we will see; it seems Thomas has even further plans up his sleeve as well.

I guess I would have to say, as long as further processes do not slow down Turbo to any relevant degree, 16 predictors makes most sense. Why not take that little more compression if it means Yalac is still top of the table?

I have no idea what impact the inclusion of error checking, seeking, and tagging will have.

Exactly my thinking!

And thanks for the table. And not to forget: for your speed graph from the previous post! It took me some time to find it....


To give an first estimate: About 0.02 percent for seeking and error checking when using 250 ms frames. More for Fast and Turbo because of their smaller frames.

But i don't know, how large the average tagging info is.

Discussion regarding PreFilter vs Parcor Coefficients moved to Yalac – Evaluation and optimization. I have tried to make the split as neat as possible. It's a difficult call, as this thread also doubles at the news update thread; however I have moved 27 posts which are not in any way related to comparing Yalac with other codecs.

Discussion regarding tagging moved to new thread: Yalac: Tagging Scheme, as this will surely require further discussion, although perhaps not quite yet maybe.

Current Progress (V0.10)

Maybe this will later be moved to "Evaluation and optimization"...

Bad News: The time of significant compression or speed improvements every two weeks is now over!

This had to happen sooner or later...

There is only one thing left, that i will try to possibly improve compression on some problem files. But no guarantee!

For the speed: No matter where i am looking into my code, speed optimizations everywhere. You can expect about 10 percent more speed for the fast presets, when i disable my self check code. And possibly cpu specific optimizations could help a bit, but not enough for me to perform them.

This time the list of improvements is quite small:

Presets

- Removed preset INSANE. With one exception any of it's exclusive options provided to litttle additional compression to pay for the speed penality.
- Combined the power of Partition search level Extra (from INSANE) with the speed of level High, which is used by preset EXTRA. Makes preset Extra less than 10 percent slower, but provides at least 70 percent of the compression advantage of old search level extra. With the PreFilter turned on, Search level High (old Extra) is quite useless, but if someone disables the PreFilter to achieve higher decoding speeds, the new Partition search level High can be very advantegous for some files.
- Preset EXTRA now uses the new (classical) channel decorrelation method Mid-Side. Again; not too useful if the PreFilter is turned on, but very fast and therefore a valuable option for faster presets. Makes Extra about 3 percent slower.

Need for speed...

or not?!

Do we really need a TURBO preset, if even FAST is allready beeing limited by today's hard disk performance? Would it at least be useful for transcoding purposes, if the disk-io allready has been performed by the transcoder software (Well, then it's the transcoder software that has to wait for disk-io, our super fast turbo mode isn't responsible, but does this make a difference?)?

My personal opinion (after some introspection...): That can not convince me. At least not, if we are talking about file encoding, decoding and playback on a PC.

But many of you were talking about possible hardware support and the need to take care for the possible limitations. And this surely is very important!

But for me this are some different things, that also should be handled seperately!

Therefore some new (not really) approach for V0.10:

1) I will keep the the presets FAST, NORMAL, HIGH and EXTRA for archival and playback on a PC.

2) The hardware (player) compatibility should be achieved by the definition of profiles, which limit the maximum cpu and memory requirements for decoding.

Basically the profiles are a additional set of presets, hence some kind of TURBO will still be avail...

V0.10 will contain a preliminary definition of 4 profiles (possibly to many), which may get redifined, when i know more about the limitations of hardware players.

Possibly nothing new or surprising for you, but for me a good way to get out of the never ending speed discussion (within my head)...

Great stuff. It's exciting to hear that you've got most of the codec worked out. Do you think that you're just about ready to work on the other features, like error correction, seeking and tagging?

Would it make any sense at this juncture to port it to C or something so that it could be made cross-platform compatible? I thought perhaps it would be less work in the long run to do it now, rather than writing the whole program, then having to do it over again.

I'm glad that you're thinking about compatibility on other platforms, and hardware, because that's one of Monkey's Audio's shortcomings.

Probably. But i just found some new optimization i want to evaluate first. It compresses some files more than 1 percent better, too much to be ignored.

Afterwards i will care for error correction and seeking. Then for tagging.

But it is quite probable, that the progress will slow down. Since my first post at April 1st i have spent nearly my whole time on YALAC! But soon i will have to go back to some work, i get paid for!


The allready very compact source of the encoder core (without any file handling or interface) is currently more than 700 KByte big! Very much work to translate it. Possibly i will first translate the far less complex decoder source to C.


But it is really a long way to go... Currently i am only preparing Yalac for such later extensions (of it's possibilities).

Current progress (V0.10)

Evaluation of compression efficiency and speed of the new Hardware profile presets of V0.10 and comparison to preset FAST of V0.09.

Performed on my primary test sets "rw" and "songs".

Test system: P3-800

All tests performed without file output.


Preset

Hardware profiles Level0/1/2 with 8/16/32 predictors and frame durations 63/100/131 ms. Preset Fast taken from V0.09.

Evaluation

Fast/Normal/High determines the evaluation depth of the encoder. This should only affect encoding but not decoding speed.

Compression

Compression ratio in percent.

Encoding / Decoding speed

Multiple of real time. Because decoding speed should not depend on the Evaluation level, i only tested High. MMX is the speed of the optimized assembler code, Pascal the speed of pure Pascal code.

Short summary

Level 2 with Evaluation High is about on par with preset FAST although it's implementation is far less complex.

I must admit that I am finding this new system quite confusing. I'm not very familiar with OptimFROG but it is reminding me of its --mode and --optimize switches.

One suggestion? Perhaps you could find the best balances in that matrix and allow users to use presets to access them? E.g.:

YALACC.EXE --level 0 --evaluation fast c:\path\to\myfile.wav

... becomes:

YALACC.EXE --preset turbo c:\path\to\myfile.wav

I have just realised that this isn't a new idea - FLAC, LAME, etc. all do this already - and you were probably already planning to. ... ah, well; I'll leave it here anyway.

Edit: Hang on, have I got the wrong end of the stick?
Sorry, does this mean that the above are additional to the existing presets, or replacements?

As I understand it, they are different sets of presets. The normal presets are for playback on a computer, the new hardware profile presets are for playback on devices with limited hardware resources (low complexity).

The hardware presets can be played on a computer of course, only they may not be as optimal as the normal presets in terms of speed and compression ratio. So normally you would use the normal presets, and convert to a hardware preset when you want to play on a hardware device (with Yalac impressive speed that should not be that painful!)

Thanks for your input jido. On reflection I believe that you are right.

Hmm... I am still confused though. I wouldn't want to have to decide on which of the nine combinations would fit my particular hardware (if I had some). I assume things will become clearer to me as they become more relevant (i.e.: when there is hardware support, perhaps a combo will be issued a preset name, like --rockbox or something...).

Jido is right:

Thanks for making it clear! I should have written some more explaination for the new presets...

But:

Yes, there are 9 possible combinations: 3 Profiles * 3 Evaluation levels.

How to make your choice:

1) Select the maximum hardware profile your hardware can support.

If you don't know or want highest compatibility to other devices, you may select 0 or 1.

I like your idea to select the profile by specifying it's name. On the other hand, it would not be too difficult to look into a list, which contains the maximum possible level for each hardware. And you anyway may have to do this, even with your named switch option, because it often will not be clear, how exactly the name of your hardware has to be specified. What if there are different models: Rockbox A3-B and Rockbox A4-9?

2) Select the evaluation level based upon your time constraints for encoding.

The hardware compatibility of a profile does not depend on the evaluation level. Select a higher evaluation level, if you are willing to wait longer for a bit more compression.


You know, that we all together have done this for the old PC-Presets. There is now a very good balance between encoding speed and compression ratio. But here we never really cared about the effect on decoding speed and hence the hardware requirements!

But we have to to this for the hardware profiles. The most important factor determining the (minimum) decoding speed is the predictor order. Not surprising: this one is the main difference between our well balanced PC-Presets.

But with the hardware profiles we are restricted to a certain predictor order. Within a profile we can only vary those encoder options, which have no effect on the decoding speed, and this is, what the evaluation level does. And because of the constraints of a particular profile, there is far less variation possible.

If you find three evaluation levels too confusing: Just ignore them and use the default (Normal). It anyhow provides the best speed compression ratio. Fast is only there to make the good old TURBO-Preset vailable.

I would be happy with less than 3 profiles. I would like to remove Level 0. But there is some conversation with Josh Coalson, and he told me, that he had good reasons to restrict FLAC -8 to 12 predictors for maximum hardware compatibility. Currently it is not clear, if Yalac would achieve the same decoding speed with 8 or 16 predictors.

Sorry for the confusion! But all this (thinking about hardware support) is very new for me.

But it's good, that i have written about it. Your feedback shows me possible failures and shortcomings of my concept.

Thanks

Thomas

Hmm... haven't heard of any new developments lately. Hopefully that doesn't mean that TBeck couldn't sustain this project. I'd hate to see this be relegated to vaporware after getting so close!

Hopefully something can be released soon enough that will demonstrate YALAC's capabilities well, but be able to maintain future compatibility with upcoming versions and revisions. I'm no expert on the subject, but I thought that implementing seeking, error correction, and tagging would be fairly easy because good existing algorithms can be used.

After the many years of work something really strange had to happen to prevent me from finishing and releasing it. But i allready wrote, that i will not always be able to spend most of my time (for the last 3 month i did hardly anything else...) for yalac, hence development will progress a bit slower.

And i want a robust and extensible public release. I don't want to have to change the file format after a public release. The design takes some time.

Some things seem to be easier than they really are. For instance it is easy to add some error recognition code to yalac to detect damaged frames. But this check is usually beeing performed, when the decoder allready has tried to decode the whole frame. Damaged data can create invalid states within the decoder, which have to be detected and handled. Because decoding speed is very important for me, i am currently trying to design the decoder for maximum error tolerance without the need to frequently check the input data for invalid states, which could take a significant amount of time.

I am still working hard on Yalac, but currently there is not much that would be worth to be posted.

Thomas

Current progress (V0.10)

Well, this time it will take more than my usual 2 weeks to get the next version done, because i have to make more fundamental deceisions, for instance about the file format. But now it's at least clear, what the next version will bring.

Done:

- Slightly better compression for preset EXTRA.
- 2 hardware profiles.
- Each frame contains a CRC32 Checksum for error recognition.
- Each frame starts with a sync code, to make streaming possible and data recovery of damaged files easier.
- Further preparations for later streaming support.
- More error tolerance within the decoder.

Preset Turbo will be a bit slower, because the checksum generation takes some time and because i have increased the frame size from 63 to 100 ms to compensate the space requirements of sync code and checksum.

To do:

My primary goal for V0.10 is far more error tolerance and a function to repair damaged files. I had to deal a bit earlier than i wanted with the streaming implementation, because some of the features needed for streaming are also important for the error resistance.

V0.10 will bring a new GUI function to manually repair damaged files. Later versions will support some automatic recovery while decoding. It's anyhow needed for streaming.

I would like the testers to damage files (be bad!) and then try the new repair function. Probably i will provide a little tool, that will randomly replace some bits of the files.

Unfortunately i can not predict, how long the implementation of the repair function will take. It's very probable, that i will find, that many parts of the decoder have to be rewritten for more error tolerance, when i start the testing of damaged files. Maybe a week...


BTW: Thanks to Joseph Pohm and Synthetic Soul for updating Joseph Pohm's comparison.

Warning: A comparison performed on a limited file set is nearly never representative for the average performance of a codec! For instance this set is the only one so far, where Yalac High comes so close to OptimFrog High. These files are really something special!

Thomas


Edit: Warning added.

Does this make sense?

Good question!

I wanted to ask the same in the Yalac-format-development-thread...

It's very possible, that the file header will contain a MD5 check sum for the whole files. This should allready provide some safety.

The frame crc's should be able to detect more errors and their location (I hope so). What is it worth?

If the errors are so minimal, that you wouldn't notice them, you will anyhow want to ignore the check sum errors.

If you can hear them, you don't need a check sum to verify this.

They may be useful, if you don't want to listen to the file to check for errors...

And it's a question of the error probability; if there are not many, one MD5 check sum should be sufficient.

To be honest, i myself could live witout frame checksums. But most other audio compressors are using them, and i would feel bad, if users would not trust yalac because it does not use them...

Thomas

A CRC for each frame is possible in LAME, but with a penalty of 16 bits per frame. I'd go with MD5 for the raw audio data like in FLAC and WavPack. In this case, you either know if a file is corrupt or not, but that's sufficient IMHO.

Edit: Why do you "expect" data loss? I never had any problem with data corruption with my lossless files.

Or, you could implement a CRC-16 code, maybe, in each frame. I don't think the system needs to be _that_ sturdy...

The advantage of per-frame error checking is that you can mute the audio when you detect an error, thereby avoiding loud bursts of noise. Also, if you are recovering a corrupt file it's nice to have the option of throwing away (or muting) the bad blocks. If you somehow lost the global MD5 you would have no idea whether what you had was mostly garbage or perfectly fine.

How much would this actually impact the compression ratio?

I'm not really sure in what context you'd get so many bad frames... usenet posting of lossless image files, maybe?

Thanks!

Current progress (V0.10)

And again a report... The main reason for the increased report frequency: I am now working on features which are more important for the practical usability for average users than some specific encoder options. I want to make sure, that i don't miss necessary features.

Decompression and error recovery

Now you can deceide, if you want to decompress files or only test their integrity.

Errors can be corrected on the fly while decompressing.

You can choose, how damaged files should be handled:

1) Skip: Don't decompress them.

2) Repair: Make a new file from the valid frames.

You can choose, how damaged frames within each file should be handled:

1) Skip: The damaged data is beeing cut out.

2) Mute: Replace the damaged data with a frame of silence. This can be useful, if you want to perform manual corrections with an audio editor on the recovered file.

For each damaged file an error log is beeing created:


It contains:

- The frame count the file header reported.
- The count of valid frames.
- The count of damaged data blocks which have been skipped.
- Skipped end is 1, if there was a damaged data block after the last recovered frame.
- The count of crc errors. Most errors may be detected by the decoder, but some will go through and than be detected by the checksum.

Skipped data blocks contains a list of the defective data blocks:

- The position and size of the block within the compressed source file in bytes.
- The position and size of the block within the decompressed file in samples. With this info it is easy to find the affected file parts within an audio editor.

Looks great. In a software decoder implementation, ideal fuction would be play and silence bad frames. It's nice to check for errors too, though, with a skip function -- if compressing a CD, for example.

Thanks.

But i forgot to implement two other useful options to handle corrupted frames:

3) Insert a frame of silence, but let it perform a fadeout (go down to 0) for the previous frame and a fadein for the next frame to avoid clicks.

4) Insert no frame of silence (skip), but instead perform a fadeout for the previous frame and a fadein for the next frame within their data to avoid clicks.

Are these things really functions of the codec? I thought it would be the decoder, or even the player that is responsible for behavior like this.

Surely not.

The interface to the codec is simple: Here are n bytes of data representing one frame of audio data of format x. Decompress it. The codec knows nothing about sync codes, crc, streaming, metadata, files etc.

But woudn't users like to compress and decompress files?

I am not sure, if i have understood your question...

Don't you think you are paying a bit too much work into error correction? I think I never had a file with corrupt frames, and the only scenario I could imagine would be an audio stream over WLAN, not the best way to transport lossless files to be saved on another computer.

It's possibly a matter of personal taste. Some people are being eased by error recognition and recovery. That's one reason for me. Others:

- I have read many of the older threads of this forum and found many posts regarding verifying and error correction.
- I anyhow needed sync codes and checksums for the preparation of streaming.
- I knew, that my decoder could crash when encountering data errors. I had to test and fix it. Not to much additional work to implement the whole thing.
- The comparioson table in the wiki contains a field for error recovery. I wouldn't like it to be red, if yalac should be drawn up (right?).
- Some trauma from the release of the first evaluation version: one tester had nothing better to do than damaging the files and reporting errors and crashes...
- I never before had used checksums and wanted to learn a bit more about them.

I store my lossless "archive" on CDs. I use Monkey's Audio for this, because it compresses good enough so I can always find two albums together to put on one CD. However, when I can I put the files in a rar archive with recovery information to fill up the rest of the available space on the CD, because I would hate to lose my lossless copied due to holes in the CD. If this protection can be provided by the audio codec itself, I'd consider this a point to chose that format.