Content

This thread should be used to discuss the first alpha release of my lossless audio compressor TAK (formerly known as YALAC):

1. Bug reports

That's the most important purpose of this alpha testing!

Please try to provide as many details as possible about the conditions which created the error. And please keep the affected files. I may ask you for them.

2. Compression results

They should no longer be posted in the "YALAC - Comparisons" - thread.

3. Ideas for improvements and feature requests

I am aware, that the first release is missing many important features, especially tagging and plugin support for media players. No need to tell me about this...

I will collect your ideas and requests and try to implement them in later releases.


Participiants

Primary the alpha testers, because only they will have the opportunity to test TAK...

This should be the last closed test, the beta will be a public release.

Can you add a link to the binaries? That would come in handy at this place...

unEDIT: Didn't see your response in time.

Request for testers

No, the alpha version isn't done yet, but nevertheless i wanted to start this thread, because

- i want to force myself to release the alpha. I am still not really happy with the design and code quality of some higher level parts of the compressor (especially streaming and file io, the encoder core itself looks quite good). But i know myself, if i try it longer, i tend to get more and more demanding.

- i need some more testers for the alpha.

I will sent the alpha to my core testers:

Destroid
JohanDeBock
Josef Pohm
Synthetic Soul

Other earlier testers may sent me a mail, if they want to try the alpha.

Then i would like to invite up to 10 new testers.

Formal qualification:

"Member of hydrogen since 3 months or more and at least 5 posts."

Take it as a simple "i am really interested into audio processing" indicator.

Please contact me via personal mail.

BTW: I hope to release the alpha until next monday (fingers crossed).

Thomas


New testers list

1. Thundik81
2. Qest
3. Gnerma
4. Omion
5. Zergen
6. Kanak
7. TrNSZ
8. Night Rain
9. audiofreak

edit: new tester added

V0.14 Alpha is done

Let's search for bugs!

The file format will no longer change after this release. Later versions will be backwards compatible.

Changes

Encoder:

- Frame partition resolution 256 is back. It still has a very bad speed/compression ratio, but i hope to improve it later.

Presets:

- Additional evaluation MAX now enables Frame partition resolution 256. A tiny bit better compression, but with a considerable speed penality for higher presets.

Functionality:

- Better error handling: The program should no longer be aborted, if a file io error occurs.
- Removed output of internal encoder diagnostics if "Protocol level" Results & Diagnostics is selected.

File format:

- Generally faster seeking, especially if the seek table can not be used.
- Stronger protection (CRC) of meta data structures. Depending on the frame size up to 0.01 percent compression penality (preset TURBO).

Command line:

- New switch to enable the Verify function for encoding.
- I finally managed to remove some modules which were only needed for the GUI version. The exe is much smaller now and it should take a bit less time to initialize the program. Possibly the processing of single small files will be a bit faster.

Other:

- Fixed the bug reported by Destroid for V0.13: Decoding of a big file (about 700 MB) gave the error "Meta data damaged". Nevertheless the audio data itself could be decoded without any errors.
- Some other bugs removed.
- Rework of the ReadmMe.

Release:

I hope to send the new version to the following testers within the next 24 hours:

Destroid
JohanDeBock
Josef Pohm
Synthetic Soul
Thundik81
Qest
Gnerma
Omion
Zergen
Kanak

Any of the earlier testers not listed here can sent me an email anytime, and i will send the current version.

P.S.: Possibly i will get a faster internet connection within the next 24 hours. If i should encounter technical difficulties, there could be some delay.

What should be tested:

This time i don't want to tell you in detail what to do... Try all the things i never thought of and find the bugs i could not find! That's the most important thing.

Less important:

- Comparisons with other compressors are always welcome...
- You could try some encoding with verify enabled, to check, if it throws false alarms.

Plans for V 0.15 Beta:

- Bug fixes, if neccessary.

I would be happy to release the public beta in early january.

Thomas

why don't you just join the FLAC team man.....you'd be a great asset helping them!!!

Because this is his encoder and it is better.

so instead of fighting an uphill battle it would be great for a merge! no need to add formats and such, FLAC is so popular anyway

I don't think he's competing with anyone, he just wants to make his own lossless codec. Besides, merging two completely different codecs is not as easy as you may think, even if they're both lossless.

I could not have said it better...

I always was primary interested into trying what i can achieve in lossless audio compression. For many years i did it just for my own fun without telling anyone about it! Now it's fun for me to make something usable of it.


I totally agree.

There has been some conversation with Josh Coalson about an integration of Tak technology into FLAC. And i could imagine, that this will happen sometime in the future (if Josh still wants to and if he has not improved FLAC so much, that an integration of Tak would make no more sense...).

But i would not like to do this with the current implementation of Tak. Reasons:

1) The current codec is quite complex. There are about 600 KB of source code (core of the codec only) which had to be translated to C. This would be very much work (i am no wiz in C). I really don't like to spend so much of my free time for something so little interesting. I am in it for fun and would prefer to spend this time for codec improvements.

2) The code complexity is also a bad thing for the integration into FLAC or other open source projects. More code = more chances for bugs and more trouble with adaptions to other systems.

3) Tak is very fast on average, but the peak processing requirements can be quite high. In the discussion of Flake Josh warned Justin, that more than 12 predictors could increase the cpu requirements so much, that some hardware decoders were no longer able to play the files without stuttering. Hey, Tak is using up to 256 predictors! And even with only 12 predictors it's cpu requirements would be considerably higher than with FLAC.

After the public release of Tak i want to evaluate the possibility to build a new codec, based upon Tak technology but with far less code complexity and less cpu requirements. This could be a candidate for a inclusion into FLAC. But i have to try it, before i can say, if it is possible...

First i did not want to answer to this questions in this tread, but probably it was time for an update on this matter.

There is a lot of frustration among certain users regarding the sheer amount of competing projects in certain fields, with no single one doing everything they need. One of those fields is audio players under Unix, for instance. I can understand that some users can feel the same way about lossless codecs, with FLAC being well supported, but not so fast and not so efficient, others being almost not supported at all and even slower but much more efficient (OptimFROG [...]), others being faster and more efficient but more or less well supported (WavPack), etc... So, your explanation, for once, makes sense, explains a lot, and is very welcome as far as I'm concerned.

Gnerma checking in. I've just started toying with it and here are a few things:

* No hi-def support in this release? I tried a stereo 24/96 WAV and got an error. The file is 2.31 gb uncompressed.
* Also got an error trying to compress a 2.45 gb 16/44 WAV.
* UI is slow to respond when using slow compression levels.
* UI checks to see if file exists when "Test" is attempted.

I'm not exactly sure if things like this are what you're looking for but if I'll report everything I find to be even slightly off.

The current implementation supports up to 24/96. But some parts of my code (file and wave io) are limited to file sizes up to 2 GB or exactly 2.147.483.647 Bytes. It's not a restriction of the encoder itself or my Tak container (supports up to more then 100 GB).

I hope, you got a somewhat senseful error message ("Wave file not supported") or did it simply crash?

To be honest, i didn't know that wave files can be bigger than 2 GB. I thought, there was the same restriction as with avi files...


You mean if you want to stop the process?

Well, i wanted it to be as fast as possible and therefore it checks not too often for input. If it's annoying, it should be changed.


I wanted "Test" to be an exact simulation of the real encode/decode process with file io. Wouldn't i be irritating for the user, if the test succeeds but the real processing gives an error ("file exists")?

After a bit of thinking: No, this check should be disabled. It could happen frequently that the user wants to check the just encoded files with the test option. Then it's quite probable to have both the source and the encoded files in the same folder. It would be annoying to always have to enable the overwrite option for the test...


Not really. I wanted to hear that Tak is the greatest!

Seriously: Thanks! It's exactly what i wanted!

I really didn't know that the size limitation would become so soon relevant! This will be among the top items on my to do list!

Many thanks

Thomas

Yes, 2 GB limit if you are too dumb to use an unsigned long for the size values... Forgive me, but when i started my work on Tak, the Delphi 3 compiler i was using did not support unsigned longs...

It should be quite easy to fix this and increase the size limit to 4 GB. Would this be sufficent for now?

(Currently i don't want to deal with wave files containing more than one audio data chunk.)

Well, looks as if there will be a second alpha release after this test...

Full testing is coming soon, but I've not yet found any problems with any files; I compressed a few GB in different modes as well as my small standard test set here. Only 16-bit 44.1khz input was tested up to this point. I think a slower, higher compressing mode would be appropriate as well, considering the current performance. More work tomorrow, but so far, this is very nice.

(Edit: 1.73Ghz Pentium M laptop)

Encoded 7 albums so far and found no issues at all. The speed and compression levels are astounding. This is a wonderful codec. Using a E6600 @ 4 GHZ.

I am simply dumbfounded. I expected good results, but this is outstanding. I've tested about 5 albums and everything has been consistantly impressive across the board.

No bugs to report so far.

It's beautiful, TBeck. Beautiful.

I am really grateful for this fast feedback! Thank you!

It's a very nice compensation for the many lonely nights with so little sleep...

BTW: Looks as if i have fixed the 2 GB bug. Now for the next two items on the new todo list...

Thomas

Yes, the error in both instances was wave file not supported. The greater than 2gb 16/44 file is actually a 4 disc album that I'd spliced together because I'm weird like that. Not something too uncommon for people on forums like this though I'd wager.

I'm sure 4gb would be fine for most everybody but ideally the compressor should support any valid WAV file. If that is 400 tb then so be it.



Well what seems to be happening is the UI is only getting updated when its absolutely necessary, or when the compressor finishes 1% of the operation. So if you're switching between windows you could end up getting a square with the dead image from the last window instead of the contents of the UI, until it updates anyway. I personally don't like this kind of behavior I suppose because it gives the user the impression that the program is unpolished. This is just me of course..

Stopping the process isn't a problem.

Modified. Both test modes (encoding/decoding) now ignore the presence of the destination files.


Fixed. Now up to 4 GB should be supported.

Breaking this limit will need considerable more work. I don't want to do it for the first public release. Sorry...


Improved. Window updates should now be performed at least once every second. Well, this is not really exact, because the update latency currently depends on the system speed. But if it's fast enough on my good old P3-800, it should be ok for now.

Later i will implement an option to let you control the task priority, but not for the first public release.

Thank you for all your findings!

Thomas

I agree. There are some optimizations possible for the current codec, but i wouln't expect more than an average improvement of about 0.3 percent.

Possibly i will work on a new codec similar to the current one but with more methods to improve the compression. I allready worked on it but there were some really complicated optimization problems which needed much evaluation, too much to perform it before this release.

But the file format is prepared for the addition of new codecs...

I'll just bump in with a few questions although I'm not an alpha tester...

Wouldn't it be better to use a seperate thread for the encoding so that the UI isn't affected by it in any way? I think there are many problems when both tasks are coupled.

It seems that so far there's only a GUI version, so is a CLI encoder/decoder planned?

I am always a bit hesitant to use multi threading, because i don't have a multi core system and therefore am not able to check for thread synchronisation issues. If this changes, i will care for it and also built a multicore encoder (should be easy with TAK's design).


There are TAK, the GUI version and TAKC, the command line version. But you are right, in the beginning there was only a GUI version.

TAK vs The Rest of The World:
http://www.synthetic-soul.co.uk/comparison/lossless/

TAK 0.14 vs TAK 0.13:
http://www.synthetic-soul.co.uk/comparison/tak/

Sorry for the delay, I thought I better update to FLAC 1.1.3 and WavPack 4.40 at the same time, and those WavPack x modes take some time! Unfortunately it looks like I screwed them up somehow, so I'll have to spend a couple of days running them again! I have reverted back to 4.4a3 for now.

Once I've got decent WavPack 4.40 results, I will do some more timing tests with TAK, when verifying while encoding, as I think it will be interesting to compare to encoding with no verification. After that I'll try to find some time for damage.

VERY nice.

I just encoded 58GB of music, and even preset 0 easily beats FLAC level 7.

Here's some compression results:


I also tried to trip up the codec with strange signals like adding DC offset, extreme clipping, odd sample rates, etc. No problems whatsoever. This is looking really good!

I'll see if I can do some more tests soon.

Ok, for this time...

Thank you!

Sorry for the trouble.

That's a very nice idea! I never really evaluated it.



Fine!

Although i have to admit, that i am not familar with FLAC -7.


That's really great! Thank you for such an exhaustive and creative testing!

This all really diminishes my fear to release a public version...

Well, I meant FLAC preset level 7 (the command line is "flac -7", which is why I worded it that way)

I edited my previous post to clear up any misunderstanding.

Hmm. I've got a file that compresses better with Tak Normal than Tak High /Extra High.

The song i tried was "drug me" by Dead Kennedys. Here's the result:

Original Size: 20369896 (bytes)
Tak Turbo: 8743793
Tak Fast: 8732490
Tak Normal: 8500839
Tak High: 8666097
Tak Extra High: 8598122


More thorough testing coming soon.

I've compressed Seabound - Double Crosser Limited Edition 2-CD:



Hope that is useful, funny to see how many times the max mode of a lower setting outperforms the next standard setting (and still decodes faster). Edit: Also funny to see how different the results are here than on the previous test I ran.

Oh yes.

What differentiates the higher from the lower presets?

1) A better evaluation of some compression parameters. The higher presets are adding more and more of the evaluation options. Max can do the same for the lower presets. If the compression advantage of a higher presets is caused by the better evaluation, then a lower preset + max can achieve the same.

2) Higher presets can use more predictors. But the usefullnes of more predictors varies very much depending on the files. Some are happy with 4 - 8 predictors, most like up to 32 (seems to be the sweet spot) and quite rare files benefit enormously from up to 256 (or more, earlier Tak used up to 512) predictors.

3) High and Extra can use the Prefilter. Very efficient on some files, small effect (0.1 to 0.2) percent most of the time, even can hurt sometimes. Unfortunately it is often beeing applied even if it's advantage isn't significant. But if applied, it will definitely reduce the decoding speed. Well, that's the price we have to pay for the very fast encoding: It's based upon fast estimations which work perfectly most of the time but sometimes fail. A very very slow insane mode could cure this.


Yes, this is always a bit strange. It's really difficult to create a representative test set to judge codec performance.

Your first set was a very Tak friendly one. Only in rare cases it will outperform for instance Monkey's Extra High.

Your second set is closer to my average impression. Well, possibly a bit worse.

The truth may lie somewhere in between.

Thank you again!

Thomas

The first set was actually a rip of the re-release of 1978 Warm Leatherette/T.V.O.D. single by The Normal. This Seabound release would probably fall under Electronic/Industrial/EBM.

I'm running a TAK only comparison of Seabound - Beyond Flatline now, and after that is completed I will run the full lossless compressor test against my other test set, which includes some classical and jazz pieces. I'm also going to test Atari Teenage Riot - 60 Second Wipeout, which is really a codec killer, containing lots of white noise and non-musical digital nonsense, as well as Telarc's release of Ray Brown's SuperBass (CD-83393).

Quick Edit: No problems yet during encoding/decoding, all data so far verifies under all modes. Haven't tired any crazy sample rates or bit-depths yet.

These two are a little confusing to me. I think I know what you mean, but it could be clearer...

BTW, the biggest part of the new WavPack release is the work I put into the -x modes levels 1-3 (which are now completely new) and I see you're not using any of them for your tests. The plain -x option (no param) is now completely usable in fast and normal mode. It gives a nice compression boost with not much of an encoding speed hit and (like before) no decoding speed hit. At this point I think that the only time it makes sense to not use it is when you're really in a hurry to encode something, and I've thought of making it the standard operation.

Just a thought, and thanks for your testing...

Thanks! I was just about to ask what to switches to recommend for the latest WavPack evaluation. I was just using: -f, [default] and -h. I think the faster-decoding modes are appropriate to use when comparing to TAK.

I've fixed that bryant.


I will add -x, -x[1-3] on my next test. I actually only use -hhx6 myself, with speed not being much of an issue, but acceptable decoding time with high compression (as well as abundant features) making WavPack my compressor of choice. If there is really no hit, maybe you can default the x option in 4.41.

Interesting. Thank you!

This can happen, because the encoder only estimates the effect of some compression methods and sometimes this fails and it makes the wrong choice. Most probably it is applying the Prefilter although it hurts or it is using more predictors than neccessary.

I have one file in my test corpus with some similar failure.

Would it be possible to send me about 20 to 30 seconds (if it's legal in your country!) of this file which are able to reproduce this effect? I would like to look for the reason of this misbehaviour.

Thomas

Here is the test for Seabound - Beyond Flatline.



Also interesting how well certain settings do, but all are close. The next set of tests is running, then tests on high definition material.

Edit: Only about a 1% spread from 0 to 4m.

I've now emcoded 29 albums and various "problem tracks" and I haven't found any anomalies whatsoever. This will be my codec of choice once a Foobar plugin is released. This is the biggest change in speed and compression I've seen in any lossless codec in years.
I'll do some 24/96 multichannel later.

Important question

I am tempted to release a second alpha until wednesday.

It fixes the 2 GB bug. For this i had to switch to Int64 types in several places of the code. Unfortunately the code parts where the default Int32 types and the Int64 types are mixed have some chance to create errors (i allready found some by myself). It is really neccessary to test this new version before the beta release.

I think it would be ok to use the new version for our testing,

- because anything else looks ok and i wouldn't expect to encounter more severe failures of the current alpha (V0.14)
- and because i don't want to ask you to repeat your testing for a new alpha released after christmas.

Compression and speed should not be affected by the bug fixes.

What do you think?

As for me, it's ok

Sure.

I'll be interested to try this codec once it enters the beta/stable stage!


If I decide to use lossless compression again (i.e. weigh up the need for with a 279 GB disk and limited CD collection), I may be joining you!

OK, comparison is showing WavPack 4.40 now.

I've run the -v tests today also, so hopefully tonight I'll get opportunity to collate and post figures.

Here's CPU-only results, showing the difference between using -v (verify) while encoding, and not.


I may compile these for CPU+IO rates as well, but the obvious assumption is that my hard drive throttling will narrow those differences a fair bit. The fastest I seem to be able to get writing to disk is ~80x, so even with Turbo it should drop to around 130-140%.

I had the opportunity to test (a part of) the file.

First the compression ratio for any preset, each with default and maximum evaluation:


There is something in Max evaluation which really helps Fast: 1.32 better compression. It is also present in Normal default, where the advantage of Max is much smaller.

Let's find out what in evaluation max is responsible. Try Fast with each single option that would be added by evaluation max:


Sorted by compression.

Obviously "Optimize quantization" is the winner. That's very rare and therefore very interesting for me.

Now for High. Let's use maximum evaluation and play a bit with the PreFilter sensitivity:


Huh, better disable the PreFilter for this file...

Well, sometimes my fast parameter estimation fails. I will evaluate this later.

V0.15 Alpha is done

Changes:

Improved:

- Support for wave files of up to 4 GB Size (2 GB before). To be exact: 4 GB - 15 Bytes...
- When test encoding/decoding files existing destination files are beeing ignored.
- GUI version responds faster when using slow presets.

And an update for the Damage tool (V1.02):

- Support for files of up to 32 GB Size (2 GB before).

Release:

I hope to send the new version to the alpha testers within the next hours.

What should be tested:

No changes. Keep on your good work!

Feedback of the two testers who found the 2 GB bug would be nice.

Thomas

Thanks for the update Thomas. No change on the my greater than 2gb WAVs. The program still throws "Wave file not supported" for both.

Too bad...

Is there any meta data embedded?

Would it be possible to send me the first 100 KByte of the file?

Probably there is something in the file header that my very simple wave file reader can not handle.

Sorry...

P.S. Currently Tak is not able to handle files with more than two channels!

edit: P.S. added.

I've uploaded my test results for the Drug Me Sample.

Download Links:
HTM: http://www.freewebs.com/kanak/DrugMe.htm
PDF: http://www.freewebs.com/kanak/DrugMe.pdf
XLS: http://www.freewebs.com/kanak/DrugMe.xls

Song Details:
Song Tested: Drug Me by Dead Kennedys (Album: Fresh Fruit For Rotting Vegetables)
Song Length: 1:55
Uncompressed Size: 19.4 MB

Test Bed:
Dell e1505 Laptop: Intel Core 2 Duo T7200 @ 2 Ghz, 1 GB Ram, 120 GB HDD 5400 RPM

Purpose:
0. To see if I have configured Synthetic Soul's scripts properly by doing a "real world" test on a small file.
1. To test TAK
2. To see if the anomalous behavior of TAK on this file (compresses better with TAK Normal than TAK High or Tak Extra High (See Here, and here) is also shown by other Encoders
3. To test the maximum compression + experimental option in the experimental version of OptimFrog (version 4.600ex)
4. To test the new -x switches in latest wavpack (version 4.40.0)

Encoders Versions and Switches Tested
1. TAK 0.15 Alpha: All Switches + Max for each
2. FLAC 1.1.3: -0 to -8
3. Wavpack 4.40.0: All Switches + (x1 through x6 for each)
4. Monkey's Audio 4.01: Fast through Insane
5. Optimfrog 4.600 ex: Default (no switches) and (--maximumcompression --experimental)
6. ALS RM18: Default (no switches), -7 ("maximum compression"), -a -o32

Remarks
1. The "anomalous behavior" is also shown by Monkey's Audio (the compressed size is smaller with Extra High than with Insane), also by FLAC (-1, and -2 compress better than -3), and by Wavpack (-fx5 and -fx6)
2. Decoding speed figures are pretty "unreliable" because the song is so short. It definitely can't be used to establish trends across different switch combinations for the same codec. But it's still somewhat valid across different codecs. Nevertheless, the TAK's decoding speed, especially at higher compressions, is unmatched by any other codec. Kudos to Thomas.
3. Tak specific comments:
Amazing Performance. TAK Turbo encodes faster AND compresses better AND has better decoding speed than FLAC-8. Similarly, TAK Normal trounces Wavpack even at -hhx6. However, there's still room for improvement at the maximum compression settings (TAK can't match OFR even at default OR monkey's at insane)
4. Wavpack switches
The -x1 to -x3 switches do seem more usable (i.e. don't have a significant performance hit) than before, although the performance with the -x4 to -x6 switches are pretty horrid (as warned).

Future Plans
- Test on larger files (album images for different albums across genres)
- (Hopefully) Include WMA Lossless/ALAC
- TAK Specific: Use Damage tool to test error resilience


Edit: Corrected some typos

Thanks for having the patience for testing so many of the WavPack modes!

BTW, that is an interesting sample! Would it be possible for me to get a clip of it also for my test corpus? I would appreciate it...

Wow, thanks for providing such a nice amount of extremely interesting data.
I'm a DK fan too (course talking about the Jello era), gonna fiddle with that track a bit myself.

Moderation: Edited quote.

Very nice! Synthetic Soul's scripts seem to be really useful!


I'm glad that you are testing Mpeg4Als. I am always very interested into it's results because it is using similar methods as TAK (and FLAC).

From my experience Mpeg4Als -7 is on average only up to 0.3 percent better than TAK's strongest mode. But on this sample it really outperforms TAK by more than 4 percent!

I allready knew that such samples exists. I have an idea how i can tune TAK to perform much better on them. Unfortunately this would be incompatible with the current encoder.

But i anyhow want to add some more codecs to TAK in the future.

The current codec is an all in one solution. It has to cover the whole spectrum from highest speed to highest compression. Some compromises where necessary.

I will probably add two specialized codecs: One for an even higher (decoding) speed and one for higher compression. The latter will sacrifice some decoding speed for higher compression, but not too much. TAK has to be fast, otherwise it would be quite useless.

But now stop dreaming about the future and back to the work. There is still the bug in the handling of some large wave files...


Great!

Thomas