This thread is off-topic 50% of the bloody time! (see file format posts, and previous splits I've made)

I was leaving the Open Office discussion until it had run it's course, especially as I am partly to blame.


Anyway, while I'm here: Anyone done much Damage testing yet? Why does nobody post any results here anymore? I like to see other people's work as well as my own.

I'm hoping to actually perform some Damage tests very soon, and was wondering what others have already tested...

I did a bit of damage testing. I didn't do anything too in depth... just random damage. It results in missing frames in the playback... just little gaps in the audio.

It might be interesting to wipe out the first or last bit of the file, and see if it still works without the headers and such intact.

I'll ready my chainsaw tonight : I'll try doing some damage testing on the lastest version of the encoder. Won't be much or much interesting, probably, but I'm going to change the files by hand.. Try to generate some valid compressed audio frames with mismatching CRC, or something. Maybe some obvious (cut-and-paste) steganography, too... (I know, obvious and steganography don't go together)

It's been a while since I ran TAK...

Yes, I was thinking along these lines. I've done some preliminary tests with Damage and it seems that Yalac/TAK can easily report and ignore.

Let's see what happens when we chop its head and feet off...

"head and feet"! Huh, i could not imagine, that testers could be that much cruel to innocent bits.
Really. I wasn't prepared.

But thanks for working careful!

Possibly this test is less interesting for you than the earlier tests.

Even more thanks!

Thomas

in monkey's the patent "danger" is in his hybrid rice-range coder implementation, which I think is genuinely novel. but because of submarining you never know.

I have done experiments years back with range coding in FLAC. range coding itself is for the most part believed to be patent free. as soon as you start using range coding for losslessly coding the residual of an audio signal, which includes a statistical modelling part, you're in unknown territory, and since it goes in the decoder that is even more worrying.

unfortunate.

Josh

My quite old implementation (from 2000) is quite similar to Monkey's. Possibly a bit more elaborated because of the use of more probability models from which the encoder can choose the best fitting one. For me that seemed to be a logical step if you have started with rice coding and wanted to overcome it's limitations.

Allready two with the same idea, hence i guess that it's quite probable that others had the same or a similar idea and patented it.



I had little success with adaptive models. Too slow (especially when decoding) and too little advantage over my approach with a limited set of static (order 0) models. But possibly i have not tried it hard enough...

a simple possible improvement with a rangecoder can be made when encoding the signs.
you need often less than a bit if you for example test the current sign against the last one.
i think that in your approach an adaptive model can improve the current rice-codes
because the static lpc-predictor often lefts out some of the linearity. But as you already
mentioned it's a bit slow. Dmitry Subbotin has a fast public-domain rangecoder by the way.

You maybe right. I tried it in the early days of Yalac, before i knew about arithmetic compression. Without arithmetic compression it wasn't advantegous because i could not efficiently represent the quite small probability differences. But i will try it again with range coding. Thanks!

But i wouldn't expect more than about 0.3 percent better compression (best case).


Oh yes, it's fast! For my evaluations i just switched from Michael Schindler's implementation to this one. It's really fast and compresses my data only about 0.03 percent worse.

Thomas

Well, a first quick and simple approach improves compression by about 0.04 percent...

Possibly it works better if only one Lpc-Filter is beeing used. But yalac currently sends the signal through up to 4 different filters. If you visually inspect the residuals after the last filter in an audio editor, you will not find many regularities. Ok, a statistical analysis will still find some.

Without performance hits? Every fraction of an inch gets us closer to REAL ULTIMATE POWER!

Tom, you're definitly the best. Even better than a ninja !

this is the sign-encoding? do you use an adaptive model? if, make sure the model
adapts very fast.

i could post a very complex entropy coder, if you like...just to see how things could be done

Thanks for your offer. I know a bit about adaptive modelling (i have played with it earlier).

But i doubt that i will use it in Yalac, because

- i am a bit dogmatic: no (complex) adaption on the decoder side. Decoding should be as fast as possible. (From my experience model parameters would need too much space, therefore you had to perform the model adaption in the encoder and decoder).
- The residuals (and signs) after the last filter stage look quite random. If you are using only one prediction filter you will usually find regularities within the residuals (i agree with you), but as i wrote earlier, Yalac is using a cascade of up to 4 filters.

I guess it could be advantegous for some signals to limit yalac's filter chain to only one filter and then use the sign compression afterwards. I suspect that especially low frequency signals sometimes could compress better than when using the whole filter chain without sign compression. This could be an interesting option.

But currently i am a bit to lazy to evaluate this possibility (which would need many changes of my code). Maybe later...

Not to forget: Adaptive range coding could generally be very advantegous for low level files (for instance classic music).

In the meantime i have tried two other optimizations:

1) Improved compression of the filter coefficients. Nice for files which are benefiting from medium or high predictor orders. Results from my primary test file set: TURBO 0.03 percent, FAST 0.06 percent, NORMAL 0.10 percent better compression.

This may not be much, but if you remember, that i thought about implementing (the slower) Range encoding to get about 0.12 percent better compression...

This optimization will have no significant effect on decoding speed.

2) Evaluation of the PreFilter and looking for speed ups. I wrote earlier, that i did not fully understand, why it sometimes helps. Now i know more about it.

The prefilter produced a dramatic compression improvement (more than 3 percent) on some of Joseph Pohm's special files. I could now implement a much faster filter which would achieve nearly the same compression advantage as the old implementation on those files.

But i see no chance for a general speed up of the PreFilter for average files. Sorry...

Thomas,

I would expect the DAMAGE command:

DAMAGE file.yaa -f a 1024 -p 1024 2048 -r $571ff

... to damage 1024 bytes in the bytes between 1024 and 2048, but only five bytes appear to be changed.

Is this because the 128 per MB kicks in?

Also, the command line:

DAMAGE file.yaa -e i 24 r 01010101 i 16 r 1010 i 8 -f a 1024 -r $571ff

... fails with "Command line error: - expected".

What am I doing wrong?

If I want to damage right at the end how do I set the first -p value, i.e.: -p <?> -1 ?

That's exactly the reason...


Nothing. It's my mistake. I did an error while parsing the error definition -e. Sorry! I just fixed the bug and will sent you the new version soon.


Well, you should be able to specify an offset to the file end as first parameter. Unfortunately i have not implemented it yet... I will look at it this today. I wanted to reply first.

Thanks for testing this buggy little thing! Strange to find more errors here than in the average yalac release.

Thomas

Edit: I have changed the specification of the error positions. You can now specify negative values for positions relative to the file end:

-p -100 -50 Damage between FileSize - 100 and FileSize - 50
-p -100 -1 Damage between FileSize - 100 and FileSize - 1 (= FileEnd, because counting starts at 0)

Or should we write -99 0 instead of -100 - 1?

Thanks for the response Thomas.

I look forward to the new version of Damage. I have to say though, I'm finding it hard to come up with anything to foil the Yalac decompressor so far.

0 or -1? I guess 0 is OK as (I assume) it cannot be used as a starting position. I.e.: I assume positive numbers run from 1 to n, where n is the filesize.

So -p 1 1024 would be the first 1024 bytes and -p -1023 0 would be the last...

Hmm, looking at it like that -p -1024 -1 looks a better format.


Then you would say -p 1 100 is the first one hundred bytes and -p -100 -1 is the last hundred bytes. To me that seems easier to work with... I think.

That's the least i can do!

I just sent you the V1.01. I hope it works better now, although i have done it in a hurry...


Very good news! Thanks.


Sorry, it's a bit different now (at least for this version):

-p 0 99 slecets the first 100 bytes,
-p -99 0 selects the last 100 bytes.

TBeck, I was curious about whether or not there is some way to send a signal to the audio player (i.e. foobar) if errors in the file are detected, so that the user could be alerted that there is a problem with the file.

I know that foobar will pop up an error console reporting some kinds of sync errors and such, but is there a way for the actual decoder to communicate with the player software so that such errors could be reported explicitly and logged? This could be handy for weeding out bad files and such.

Has anyone heard from TBeck lately? I was just wondering if there have been any updates to this project. It's been quite a while...

Finding a job can take some time. Let's just hope his project hasn't met a void without the community being able to continue his work.

Thanks for asking!

I am still alive!

And i have been working on Yalac. And i had to take some deceisions.

There has been no new release, because i wasn't sure, if the small improvements would justify the work of the testers. And finally i came to the conclusion, that it's now really time to stop looking for more improvements. It's possible, that i will have some new ideas sometime, but obviously i can not force them.

V0.11 will be the last version with new compression methods. Some further improvements can be achieved by optimizing the encoder parameters of the existing methods. But this has no high priority for me.

What will be new in V0.11:

- Better compression of the filter coefficients. Can give 0.1 percent better compression for preset NORMAL, a bit less for the other presets. A tiny bit but damn fast.
- New Wasted-Bits-option to remove the least significant bits of the samples, if they are all zero. Especially useful, if 16 bit samples have been converted to 24-bit by simply shifting 8 bits of zero in. To my surprise even some 16-bit files are benefiting from this option. That's strange but nice, because each wasted bit usually improves compression by up to 6 percent (for 16-Bit files).
- Presets switched back to V0.09 and somewhat modified.

I have tried far more optimizations but with little success.

What i am doing now:

- Removing code from the encoder which has only been needed for the evaluation of new compression methods.
- Cleaning up and simplifying (if possible) of the source code.
- Look for design errors.

Next things to do:

- Support for 8-Bit samples (Is anyone using them?)
- tweaking of some paramters for 24-Bit samples and other sampling rates than 44.1K.

After this has been done i will have to complete the file format (container, support for metadata).

Possibly i will publish the specification of my simple container format and ask for feedback before finalizing it.

All this will take some time, especially because it looks as if i will have far less time than before for my work on Yalac. Please don't expect a public release within the next 1 or 2 months. Sorry...

If one of the testers wants to evaluate the performance of the final encoder engine, i will release a V0.11.

Thomas

I guess this is done automatically?

I see Josef is lingering at this moment, and he's probably the best candidate, but I am happy to run a test on the latest version, as it takes very little user time (something I have little to none of at the moment) for me to run my scripts and report the results.

While I'm on: I felt very frustrated that I got little time to test the error tollerence of 0.10, and never reported any results (everything I did test acted as expected BTW). Did you get enough feedback from other testers?

Are you talking about uncompressed audio formats? For windows wave this isn't true. I don't know, if there are more exotic formats which are doing it.

If you are talking about other lossless compressors: If i remember it right, FLAC has this option always enabled and Mpeg4Als too, if the mode -7 has been selected.

a bit of misunderstanding here. i only wanted to know if you manually have to specify
the wasted bits or if yalac searches for them as in other compressors like mpeg4als.

Oh...

Currently it is always switched on and can not be disabled.

The analysis of the samples is beeing performed as part of another necessary anaylsis and shouldn't effect encoding speed. If the analysis has detected some wasted bits, there will be a small slowdown of encoding, but that should be ok for the huge compression advantage.

The only reason why one would want to disable this option could be the small speed penality on the decoding side (only if there are wasted bits). Possibly it could be just a little too much for some weak hardware, but i really don't know.

Thanks for the update TBeck. I'm glad to hear that you're still working on things.

Current progress (V0.11)

Oh yes, i am still working on it...

Most work is going into the preparation of a public release. There is really much to do. Currently i am looking at my whole source code, removing code which was only necessary for evaluations of new compression methods. Then i look at the file structure with the later documentation work in mind. Frequently i am finding inconsistencies or unnecessary complex code parts, which have to be simplified. This work isn't too interesting but needs much concentration.

The last two days went into the hopefully final modification of the preset system (I can't remember, how often i have been talking about final presets...).

I want to thank two members for helping me to build them:

- Joseph Pohm for another great report about possible preset variations.
- Synthetic Soul for always telling me: 'Good work, but i still would like to see Normal to be a bit faster...'.

And what's new?

Earlier presets had been constructed to make each higher preset encoding about two times slower than the previous one. This may be a bit unflexible, especially if someone has particular encoding speed requirements, for instance: I want the maximum compression possible at the speed my grabber can grab audio from CD.

Therefore a finer resolution of the encoding speed was needed. Now there are two new presets, one between old fast and normal and one between old normal and high. Now each higher preset should be about 1.41 (square root of 2) times slower than the previous one.

Now we have presets 0 to 5, called Turbo, Fast, Light, Normal, High and Extra.

- Turbo is very similar to old Turbo of V0.09.
- Fast is nearly identical to old Fast of V0.09.
- Light is basically Fast with 64 instead of 32 predictors. (Hi Neil, i hope, you like it!)
- Normal is nearly identical to old Normal of V0.09.
- High is now using only 128 predictors like Normal, but is using the PreFilter like old high.
- Extra is quite similar to old High, but using 256 predictors.

Now some data for my primary sample set "rw":

(Test system: P3-800. Encoding without output.)

"Diff" is the compression advantage over the previous (lower) preset.
"Ratio" is the slowdown compared to the previous (lower) preset.

There is no preset called "Max".

I have kept the good old evaluation levels Extra and Max. They can increase the compression of a preset without making decoding slower. That's especially nice for the presets Turbo, Fast and Light, which are candidates for possible later hardware presets or profiles. (Please look at my earlier posts regarding profiles and evaluation levels for a more detailed explaination.)

Too achieve the maximum compression (at any price) use preset Extra with evaluation level Max. But be warned: That's really insanely slow.

Probably i will soon release a V0.11 for the remaining testers, to evaluate the performance of the (so called) final presets.

Thomas

P.S.: Hi Skymmer, we had some email problems in the past. If you still want to test Yalac, please send me a mail with your email adress.

It's great to hear that we may have an official release soon. I'm sure the additional testing and feedback will come in useful.

May I ask whether tagging will be available, and if so what type?

Thanks for the mention; I'm not sure if I should be proud or not. Professional nusiance!

I look forward to having a play with 0.11 and comparing the new presets. Especiall Light.

Edit: Spelling

V0.11 is done

Compression algorithms:

- Better compression of the filter coefficients. Can give 0.1 percent better compression for preset NORMAL, a bit less for the other presets. A tiny bit but damn fast.
- New Wasted-Bits-option to remove the least significant bits of the samples, if they are all zero. Especially useful, if 16 bit samples have been converted to 24-bit by simply shifting 8 bits of zero in. To my surprise even some 16-bit files are benefiting from this option. That's strange but nice, because each wasted bit usually improves compression by up to 6 percent (for 16-Bit files).
- Some speed up of the PreFilter. Looses only 0.01 percent compression on my file sets.

Presets:

- Earlier presets had been constructed to make each higher preset encoding about two times slower than the previous one. This may be a bit unflexible, especially if someone has particular encoding speed requirements, for instance: I want the maximum compression possible at the speed my grabber can grab audio from CD. Therefore a finer resolution of the encoding speed was needed. Now there are two new presets, one between old fast and normal and one between old normal and high. Now each higher preset should be about 1.41 (square root of 2) times slower than the previous one.
- Now we have presets 0 to 5, called TURBO, FAST, LIGHT, NORMAL, HIGH and EXTRA.
- TURBO is using 16 instead of 12 predictors and a frame duration of 102 instead of 94 ms.
- FAST is using a frame duration of 102 ms.
- LIGHT is basically Fast with 64 instead of 32 predictors and a frame duration of 125 ms.
- NORMAL is nearly identical to old Normal of V0.09.
- HIGH is basically Normal with PreFilter enabled.
- EXTRA is quite similar to old High, but using 256 predictors. BitCoder-Optimize Choice disabled.

Release:

I hope to send the new version to the following testers within the next hours:

Josef Pohm
Synthetic Soul
(Skymmer, if he sends me an email adress...)

Only reason for this selection: I haven't heard from the other testers within the last 10 days and i don't want to fill their mail box with new versions they possibly currently don't need.

Any of them can sent me an email anytime, and i will sent the current version.

What should be tested:

- Comparison with V0.10: Speed and compression performance of the presets.
- Please put only the basic presets p0 to p5 without the additional evaluation level into your primary comparison. Otherwise it looks a bit confusing to me. Exception: You may try p52 to obtain the current maximum compression.

Plans for V 0.12:

- Futher preparation of a public release.
- Possibly i will try to speed up the PreFilter. I would like it to be so fast, that it could be included into presets NORMAL or even LIGHT.

Surely. But sorry if i should have raised wrong expectations: It's still much to do before a public release.

Possibly the first public release will not support tagging, because

- i want a public release as soon as possible.
- the omission of such useful features like tagging could prevent people from doing too serious things with Yalac. This could be a good thing! The first public release may still contain bugs which we couldn't find with our still limited test sets. It would be real horror if some of the first real users would convert their whole collections to (a buggy) Yalac before it has been tested by more people.


I could not translate "nusiance"...

But my statement was only a little bit ironical! It needed some persistance to break up some of my old conceptions about the preset structure!



Hopefully you will not be diappointed. The main advantage of Light over Fast is the increased predictor order. But we allready know, that your file set usually will not benefit too much from more predictors.

Thomas

"Professional nuisance" würd ich in etwa mit "berufsmäßige Nervensäge" übersetzen . "nusiance" ist wahrscheinlich nur ein Tippfehler.

Yes, I doubt the typo helped.

I'm sure MedO's German explanation is a lot more useful (thank you), but I would explain a nuisance as being a pest, or someone who is annoying or inconvenient.

Anyway, I hope I have been of some use.

I will try to start my tests running tomorrow morning.

Well, it seems so long since I did this that I hope I did it right. At first glance (I'm out of time now) things appear to be in order.

Yalac vs The Rest Of The World
http://synthetic-soul.co.uk/comparison/lossless/

Yalac 0.11 vs Yalac 0.10
http://synthetic-soul.co.uk/comparison/yalac/

I'll analyse at a later date.

Synthetic Soul: Considered adding flake to the test?

Kinda off topic, but i'm just blown away by these results. YALAC Turbo actually compresses than Flac 0 AND encodes/decodes faster. That's just crazy.

when this thing comes out, it's gonna be my primary lossless format!

Yay for Tbeck.

btw. is YALAC the final name?

I'm presuming you mean it compresses more, faster, and decodes faster too.

hehe yeah.

changed the original post.

so is yalac the final name?

Tom stated in another post that the actual final name was to be TAK. We'll see, though.

Oh yes! Again, many thanks!

As always i have been very curious. Because the compression performance of lossless compressors always depends so much on the selection of the test files, i can never be sure, if my own evaluations are representative. That's one reason why your work is that important for me! Especially because your test file set is so different from mine, for instance because it does not benefit from the higher predictor orders of the stronger presets.

But the new presets system seems to work fine for your files too:

- Higher presets are compressing at least a bit better than lower ones.
- The speed differences of the presets are as expected.
- New Light compresses nearly as well as Normal of V0.10, while encoding at a similar speed as WavPack default .
- Fast and Turbo both are stronger now with only a tiny encoding speed penality.

Overall not a breakthrough in peformance, but possibly a nice improvement of the usability, because of the better resolution of the preset speeds. For me it was worth the effort.

And not to forget: Even if the new "Wasted-Bits-Option" could not help your files, i am very happy to have it done now. It has been the last of my list of features, that overall aren't too important, but seem to be requested by users now and then (i have read some posts about it):

- Why is compressor xy performing bad on stereo files with mono content?
- Why isn't it able to handle silence perfectly?
- Why is it performing bad on some 24-Bit files (with wasted bits)?

Now Yalac should have no more weaknesses in this areas (fingers crossed).

BTW: The strongest compression can be achieved by preset EXTRA + evaluation MAX (-p52). Not too important. I would expect only about 0.07 percent better compression on your file set.

Possibly there will be one last modification of the presets: I would like to apply the PreFilter to preset Normal. Because i don't want to slow it down, i have to look for some speed up of the PreFilter.

Thomas

PS: Thanks to Shade[ST] for answering questions!

Nice results. The decoding speed (complexity) is specially impressive.

Thanks...








(anyone get it?)

As things stand the table uses official, released, builds only.

However, with the upcoming WavPack 4.4, FLAC 1.1.3, and also Flake, it would be interesting to see how these new encoders compare.

I may run some tests with WavPack 4.4a3, FLAC 1.1.2_CVS (using -A tukey(0.5) and -A tukey(0.5)+flattop) and wisodev's Flake build if I get time, but I am more interested in testing officially released builds, and I have little time at the moment.

Thanks for the idea, anyway.


Yes, I don't know why I didn't just use the additional evaluation levels on all presets to get the full range in the first place. I will add them in the next day or so.

Caution: For presets High and Extra the evaluation level Extra is useless, because the basic presets allready are using all the options evaluation Extra could add.

Sorry for not telling earlier.

Thomas

I'd second the interest in seeing Yalac tested against other 'development build' encoders: it'd be more like comparing state-of-the-art oranges to state-of-the-art oranges. Or apples to apples. you know what I mean. (am I making sense? it's hard to tell sometimes)

Kinda.

The who point is to compare Yalac/TAK to existing, official encoders though, i.e.: those encoders that 99% of users are using. Yalac/TAK gets updated with each alpha because it is the reason for the comparison.

I would be more than happy to update the results for FLAC and WavPack when 1.1.3 and 4.4 are released. Both will warrant new data. As a WavPack user I have my own motives to test 4.4 against 4.3.

I'm interested to see the results for Flake, but I don't really understand where it fits into the equation. Justin Ruggles' response in his thread answers some of my questions, but fails to fill me with confidence. Will Flake end up replacing FLAC.EXE as the FLAC encoder?

FLAC 1.1.3 is my main concern though, as I can test now with 1.1.2_CVS, and use settings that I think will be used in 1.1.3 following Josh's posts #77 and #79 (-A tukey(0.5), and -8 with -A tukey(0.5)+flattop for the new -9), but I'm not certain whether this will equate to 1.1.3's results or not. What's the point of posting irrelevant data?

All that said, I am interested as well, so if I find a spare hour or so then you may well see them reported.

I have now added High Max and Extra Max. Extra Max provides an additional 0.087% compression - not a bad guess.

You will not be surprised to hear, that i am also very interested into the performance of Flake and the upcoming new Flac release. Here is a comparison i have just performed.

Participiants:

Yalac V0.11a: slightly improved over V0.11.
FLAC 1.1.2:
Flake 0.10: file flake-0.10-win32-i686.zip from sourceforge.

Test system:

P3-800 with a (slow) 40 GB HD.

Results


Ratio is the compression ratio in percent, EncoTime the duration of encoding in seconds. Encoding speed of Yalac's presets Turbo and Fast is allready being limited by my slow 40 GB harddisk.

For the test sets:

"rw" contains 46 files from http://www.rarewares.org/test_samples/. Should be the whole dir. I call this my primary test set and prefer to use it for my tuning.

"songs" contains CD-Ripps of the beginning of the following songs:

This set contains many files that are benefiting from higher predictor orders. It may be less representative than set rw, but i keep it because it has been my primary set for many years.

Some remarks

Flake is performing really great compared to FLAC. Much faster and significantly better compression! Congratualations! I am very curious, if a possible combination of the new methods of Flake and the upcoming FLAC will provide even better compression.

While testing Flake, i have been a bit irritated: My tool for the calculation of the compression ratio gave me worse ratios than Flake itself reported. My pocket calculator tells me, that the my tool is right.

Does Flake (and possibly FLAC, i didn't check it) ignore the overhead of metadata and streaminfo when calculating the compression ratio?

flake has improvements that are also in FLAC CVS. the next release should provide comparable compression for the same predictor order. flake will still be faster at encoding.

the compression ratio flac shows is based on the final file size (which includes metadata).

Josh

Hi Josh,

great news!


Thanks for making it clear. Possibly it's a small bug in Flake. The compression ratio it reported was about 0.25 to 0.30 percent too good for the files i have checked.

Thomas



I really don't like the boards automatic concatenating of posts.

This should have been a separate post:

Because i am currently tuning Yalac for high resolution files, here the results of another test:


Test system is the same as before. Looks as if Flake doesn't support 24 Bit yet, therefore no results for it.

For the test sets:

"sh_2444" contains 5 files with 24 bit and 44 or 48 Khz.

"sh_2496" contains 6 files with 24 bit and 96 Khz.

Unfortunately i don't have more real high resolution files.