TAK 2.2.0 development Options

[_m²_]

I cut 2 samples from another file. Now I know why I liked o100.

CODE

uncompressed                    17280080
flac -8                          9544125
wavpack -hhx6                    7571878
als -a -e -p -g5 -s1 -o60        7059115
als -a -e -p -g5 -s2 -o60        7027939
als -a -e -p -g5 -s3 -o60        6958482
als -a -e -p -g5 -s6 -o60        6937060
als -a -e -p -g5 -s6 -o100       6933432
als -a -e -p -g5 -s6 -o200       6934788
als -a -e -p -g5 -s6 -o400       6934958
als -a -e -p -g5 -s6 -o800       6934920
als -b -a -e -p -g5 -s6 -o100    6868655
als -l -b -a -e -p -g5 -s6 -o100 6868655
als -7 -o100 -s6                 6865530
als -z3                          7118789

CODE

uncompressed                     34560080
flac -8                          16740526
wavpack -hhx6                    13449682
als -a -e -p -g5 -s1 -o60        11882158
als -a -e -p -g5 -s2 -o60        11814740
als -a -e -p -g5 -s3 -o60        11727356
als -a -e -p -g5 -s6 -o60        11672213
als -a -e -p -g5 -s6 -o100       11657250
als -a -e -p -g5 -s6 -o200       11669488
als -a -e -p -g5 -s6 -o400       11675959
als -a -e -p -g5 -s6 -o800       11675990
als -b -a -e -p -g5 -s6 -o100    11505858
als -l -b -a -e -p -g5 -s6 -o100 11505858
als -7 -o100 -s6                 11525750
als -z3                          11740206

EDIT1: updated with -7
EDIT2: 2nd and last update, these files are not worth digging more.
EDIT3: Typo.

This post has been edited by _m²_: Apr 25 2011, 20:33

[TBeck]

als -a -e -p -g5 -s6 -o100

I don't think this is optimal, nevertheless i am just testing it. This may take some hours.

So far 4 files have been processed, each of them worse than with TAK -p4m.

[_m²_]

als -a -e -p -g5 -s6 -o100

I don't think this is optimal, nevertheless i am just testing it. This may take some hours.

So far 4 files have been processed, each of them worse than with TAK -p4m.

I updated the topic before I saw your answer, it's indeed suboptimal.
And I suggest trying -s2 and -s3. I think -s6 is specific to this album, though I have only memory to back it up.
ADDED: But it's nice to see that you beat it. In general, I like TAK more than ALS and anyway - progress is good.

This post has been edited by _m²_: Apr 25 2011, 20:02

[TBeck]

I updated the topic before I saw your answer, it's indeed suboptimal.
And I suggest trying -s2 and -s3. I think -s6 is specific to this album, though I have only memory to back it up.

The addition of -b is important. Currently i am testing -7 as base line. That's really slow! I started the test 8 hours ago... The strongest (and slowest mode) of TAK does the work in a couple of minutes.

Because of the slowness i can't test all parameter variations on all files. I have selected 3 files with good oppurtunities for channel decorrelation for an iterative test. Now my secondary PC is busy too...

First Results: -7 -s2 gave me significantly worse results than -7 alone. -7 -t2 is just running and it seems to be advantegous. Therefore i will evaluate -t a bit more.

When i have determined the optimum t-parameter for the 3 files, i will try it with the whole test corpus. Well, i really dislike tests with such limited file sets, but because of the slowness anything else isn't practicable. At least not now.

For the final test i probably will reduce the maximum predictor count to 160 (-o 160). The default for -7 is 1023 for 44 Khz samples! 160 is an interesting choice, because it's also the maximum TAK is using.

ADDED: But it's nice to see that you beat it. In general, I like TAK more than ALS and anyway - progress is good.

-7 compresses some file sets better than the current version of TAK. This get's interesting.

[TBeck]

First Results: -7 -s2 gave me significantly worse results than -7 alone. -7 -t2 is just running and it seems to be advantegous. Therefore i will evaluate -t a bit more.

When i have determined the optimum t-parameter for the 3 files, i will try it with the whole test corpus. Well, i really dislike tests with such limited file sets, but because of the slowness anything else isn't practicable. At least not now.

Done. Well, with only 3 files from my DTS-file set m6_44_b:

CODE

              ALS -7
Add           none    -s2     -t2     -t3     -t6
---------------------------------------------------
DTS: 16 bit, 44 khz, 6 channels (5.1)
---------------------------------------------------          
m6_44_b_02    45.28   45.51   45.24   45.19   45.15
m6_44_b_03    41.33   41.87   41.32   41.23   40.81
m6_44_b_04    45.12   45.25   45.08   45.04   44.95
---------------------------------------------------

-7 -t6 was the best. I just started a respective test with my complete DTS-set. Well, this will take another 10 hours or more...

But the results of the first run of Mpeg4Als are now ready:

CODE

           TAK             FLAC    FLAC    WavPack ALS
Mode       -p2m    -p4m    -8      -8 -l32 -hhx    -7      -7 -t6
-----------------------------------------------------------------          
DTS: 16 bit, 44 khz, 6 channels (5.1)
-----------------------------------------------------------------          
m6_44_a    30.22   29.38   33.17   33.01   32.42   28.67    
m6_44_b    41.69   41.41   45.04   45.00   44.35   42.21
m6_44_c    41.28   41.06   42.69   42.66   42.18   40.60
m6_44_d    40.18   39.76   43.23   43.16   42.09   40.45
-----------------------------------------------------------------

TAK -p4m is better on set b and d, ALS on set a and c.

Sets b and d are those, where TAK's channel decorrelation is most efficient. The following tables show it's advantage for the 4 file sets:

CODE

-------------------------------------------
DTS: 16 bit, 44 khz, 6 channels (5.1)
-------------------------------------------
m6_44_a    0.34    
m6_44_b    1.96
m6_44_c    0.28
m6_44_d    1.48
-------------------------------------------

ALS's better performance on a and c may be attributed to it's high predictor count of 1023 vs. TAK's 160. A quick test of 2 files with -7 -o160 resulted in a significant decrease of ALS's advantage.

BTW: I used mp4alsRM23 for the tests.

This post has been edited by TBeck: Apr 26 2011, 07:45

[TBeck]

-7 -t6 was the best. I just started a respective test with my complete DTS-set. Well, this will take another 10 hours or more...

No, it wasn't so fast...

The last column now contains the results for Mpeg4Als -7 -t6:

CODE

           TAK             FLAC    FLAC    WavPack ALS
Mode       -p2m    -p4m    -8      -8 -l32 -hhx    -7      -7 -t6
-----------------------------------------------------------------          
DTS: 16 bit, 44 khz, 6 channels (5.1)
-----------------------------------------------------------------          
m6_44_a    30.22   29.38   33.17   33.01   32.42   28.67   28.65
m6_44_b    41.69   41.41   45.04   45.00   44.35   42.21   41.98
m6_44_c    41.28   41.06   42.69   42.66   42.18   40.60   40.57
m6_44_d    40.18   39.76   43.23   43.16   42.09   40.45   40.13
-----------------------------------------------------------------

[Chaser]

Your findings are very interesting. I wish you good look with your further tuning, though I am uncertain whether spending A LOT of time on 0.5% is worth the hassle before "constructing a sound basement". I hope you understand, what I intend to stay.

Nevertheless I am impressed how fast you implemented a multi-channel encoder that competes that well in such a short time.

[_m²_]

Like I said, I did a more extensive test of various als settings. Took almost 2 days and 1000 file-settings pairs. It happened so fast, because I cut short samples with the intention of maximizing variance.
You can download full summary here.
In short, "-7 -t6" seems best, though I found a case where "-z3 -t6" destroyed it...it's very strange because on all other files -z3 is a clear looser.

[TBeck]

Your findings are very interesting. I wish you good look with your further tuning, though I am uncertain whether spending A LOT of time on 0.5% is worth the hassle before "constructing a sound basement". I hope you understand, what I intend to stay.

I think i do.

Nevertheless I am impressed how fast you implemented a multi-channel encoder that competes that well in such a short time.

The multi-channel-codec adds nothing really new, instead it uses the existing channel decorrelation filters i have already heavily tuned for stereo audio.

I know there exist more complex decorrelation approaches for multi channel audio, but i doubt, they are as fast as TAK's filters. Furthermore i want to avoid potentially patented technology. And i like to find simple but efficient solutions for potentially complex problems.

Like I said, I did a more extensive test of various als settings. Took almost 2 days and 1000 file-settings pairs. It happened so fast, because I cut short samples with the intention of maximizing variance.
You can download full summary here.
In short, "-7 -t6" seems best, though I found a case where "-z3 -t6" destroyed it...it's very strange because on all other files -z3 is a clear looser.

Thank you! Then it's probably ok to limit my evaluation to -7 -t6.

I am just exploring some tunings. The first of them improves the compression by about 0.10 to 0.15 percent for some of my file sets.

In the meantime another ALS test is finished. Now i am using -7 -t4 (the maximum for 4 channels) and -o160. The latter limits the predictor count of the primary filter to 160 predictors as TAK is using. This does make sense for my evaluations, because i don't want to explore the effect of higher predictor orders but the efficiency of my ny channel decorrelation.

CODE

           TAK             FLAC    FLAC    WavPack ALS
Mode       -p2m    -p4m    -8      -8 -l32 -hhx    -7 -t4 -o160
------------------------------------------------------------------------------------------
Vinyl-Rip: 24 bit, 96 khz, 4 channels (quadro), denoised and decklicked          
------------------------------------------------------------------------------------------          
m4_a       65.46   65.36   67.19   67.19   66.84   65.43   Pink Floyd - Wish you were here
m4_b       62.30   62.22   63.94   63.93   63.42   62.10   Santana - Caravanserai
------------------------------------------------------------------------------------------

[TBeck]

I have got some more multichannel albums. Since it isn't practicable to use a large corpus for the evaluation of the new codec, i had to select an adequate files subset. I performed a lot of tests on all files to determine file properties that are relevant for the efficiency of the various compression algorithms. Then i selected between 2 and 3 files from each album which were in this respect representative for the whole album. This took two days but it will pay off because it will save me a lot of time in the future (when improvingthe the codec i will have to test the files over and over again).

As a side effect i can now present a new comparison:

CODE

           TAK             FLAC    FLAC    WavPack ALS
Mode       -p2m    -p4m    -8      -8 -l32 -hhx    -7 -o160 -t4/6
-----------------------------------------------------------------          
mc_4_dts   52.95   52.03   56.42   55.89   55.30   52.11
mc_4_dva   64.06   63.98   65.93   65.88   65.10   63.92
mc_6_dts   39.46   39.04   42.28   42.21   41.41   39.24
mc_6_dva   54.52   54.41   56.74   56.72   57.40   54.74
-----------------------------------------------------------------

For the file sets:

mc_4_dts = DTS: 16 bit, 44 khz, 4 channels (quadrophonic), 15 files
mc_4_dva = DVD-A: 24 bit, 96 khz, 4 channels (quadrophonic), 15 files
mc_6_dts = DTS: 16 bit, 44 khz, 6 channels (5.1 surround), 12 files
mc_6_dva = DVD-A: 24 bit, 96 khz, 6 channels (5.1 surround), 3 files

[Destroid]

Good gracious! ALS is such a dog encoding-speed-wise I never took it seriously, but it is a multi-channel contender. Thanks to the testers so for tying up their machines for a week

Lately I have been learning DVD audio extraction. I'm glad I did since I will be ready to test multichannel whenever the beta may appear. I have at least one DVD that has DTS of a live rock concert- any Zepplin fans out there?

[Destroid]

(Can't edit above post)

Highly disappointing to discover these DTS tracks have slipped-through any MPAA loudness standards- quite a lot of brick-wall compression/limiting on Front Left/Right Almost prefer the AC3 versions Maybe it would be worth trying losslessly compressing decoded AC3 for as ridiculous as it sounds...

[TBeck]

Good gracious! ALS is such a dog encoding-speed-wise I never took it seriously, but it is a multi-channel contender. Thanks to the testers so for tying up their machines for a week

How true... It's really slow but probably the strongest competitor for multi-channel audio.

I have made good progress. It took me several days to construct optimally balanced multi-channel presets from the codecs internal encoder options. Even with TAK's high speed i had to spend a lot of time waiting for the many tests to complete. This provided an opportunity to think more about algorithms to speed up the encoding. Some of them worked well and may even be applied to stereo audio. I still have to check this, but i am confident that i can make the standard codec a bit faster too.

Then i will perform a lot of testing and finally prepare an alpha release of TAK 2.2. I can't tell you, how long this will take. Maybe 2 weeks.

[_m²_]

How about going for 'stronger' instead of 'faster'?
TAK is a rocket already.

[Destroid]

@_m²_ On impulse I wanted to disagree, but there might be something to that suggestion. The question I had was how much does disk IO affect these super-size uncompressed source files being converted to considerably-sized TAK files?

Seems mechanical drives would suffer the worst, so in that scenario consider: if, for example, the -p0 settings are 1% faster than -p1 settings (because of suffocated disk IO) the size reduction makes -p1 the obvious choice and -p0 *almost* totally obsolete in for multichannel use.

Guess I may as well add, I don't plan to get an SSD this year either, maybe next year- not sure if that puts me in the user category of the majority or the minority

[TBeck]

How about going for 'stronger' instead of 'faster'?
TAK is a rocket already.

I see hardly any opportunity to make it significantly stronger without affecting the decoding speed. Furthermore most possible modifications would result in some format incompatibilities. Last but not least: I always try to avoid possibly patented technologies.

A compatible modification would consist in an increase of the maximum predictor count from 160 to 256 predictors. Unfortunately this would only improve the compression of those files, which already achieve relatively high ratios. To be more precise: Those files, which compress significantly better when going from p3 to p4. That's mostly classical music or more general music with a high dynamic range.

@_m²_ On impulse I wanted to disagree, but there might be something to that suggestion. The question I had was how much does disk IO affect these super-size uncompressed source files being converted to considerably-sized TAK files?

Well, when testing the speed, i always try to eliminate the effect of disk-io... But surely i am aware of the fact, that speed increases of TAK's fastest presets p0/p1 usually have little practical effect on todays average hardware because of the disk speed limitations.

Seems mechanical drives would suffer the worst, so in that scenario consider: if, for example, the -p0 settings are 1% faster than -p1 settings (because of suffocated disk IO) the size reduction makes -p1 the obvious choice and -p0 *almost* totally obsolete in for multichannel use.

There are also those additional evaluation levels like for instance -p0e, p0m and so on...

Ok, i am sure not many people are using them. And this does make sense, if you don't care about a small decrease of the decoding speed.

This post has been edited by TBeck: May 20 2011, 04:00

[_m²_]

A compatible modification would consist in an increase of the maximum predictor count from 160 to 256 predictors. Unfortunately this would only improve the compression of those files, which already achieve relatively high ratios. To be more precise: Those files, which compress significantly better when going from p3 to p4. That's mostly classical music or more general music with a high dynamic range.

From my observation, while moving from TAK for OptimFROG, I save the most on classical music and the least on thrash metal.

If you can make a file 1% smaller, why does it matter whether the file is 1/2 or 1/3 of the original?

[TBeck]

A compatible modification would consist in an increase of the maximum predictor count from 160 to 256 predictors. Unfortunately this would only improve the compression of those files, which already achieve relatively high ratios. To be more precise: Those files, which compress significantly better when going from p3 to p4. That's mostly classical music or more general music with a high dynamic range.

From my observation, while moving from TAK for OptimFROG, I save the most on classical music and the least on thrash metal.

If you can make a file 1% smaller, why does it matter whether the file is 1/2 or 1/3 of the original?

An increase of the maximum predictor count will decrease the encoding speed (a little) and the worst case (minimal) decoding speed by a significant amount. The first is ok, but the latter is relevant to me because i still have later hardware implementations in mind. Important: No, i don't want to (and will not) discuss going open source issues, but this is still a relevant option affecting my design decisions.

Nevertheless i could be tempted to accept some decoding speed decreases if it would help to compress those files (usually dynamically compressed, and they become more) better where TAK is less efficient than Monkey's Audio. For more dynamical files TAK is already performing very well compared to Monkey's, as can be seen in FLAC's comparison (which is quite outdated regarding TAK, new versions are both faster and stronger now. And V2.2. will be faster again.).

Ok, here i am a bit ambivalent, as you have pointed out.

I don't see OptimFrog as a competitor, this because of different goals. OptimFrog wants to achieve the maximum compression ratio regardless of encoding and decoding speed. TAK want's to bring you the maximum compression at a decoding speed comparable to FLAC. This definitely limits my design decisions.

This post has been edited by TBeck: May 21 2011, 13:22

[Destroid]

Well, when testing the speed, i always try to eliminate the effect of disk-io... But surely i am aware of the fact, that speed increases of TAK's fastest presets p0/p1 usually have little practical effect on todays average hardware because of the disk speed limitations.
...
Ok, i am sure not many people are using them. And this does make sense, if you don't care about a small decrease of the decoding speed

I didn't mean to sound like "-p0 is useless," I think some devices will greatly benefit from plain -p0 setting. I guess I was hinting that any upcoming multi-channel encode speed tests (which I plan on doing) might take into account the disk IO bottleneck. In the past I used TIMER.EXE "kernel time" to spot disk IO bottlenecks with decoding. I was curious if there was a better way to measure the 'performance offset' caused by disk IO saturation.

I see hardly any opportunity to make it significantly stronger without affecting the decoding speed. Furthermore most possible modifications would result in some format incompatibilities.

I think it is smart for TAK to keep decoding speed a priority. I was always impressed by the encoding speed and compression figures too. However, users that are archiving and seldom access those archived files might not see the advantage of TAK's decoding speed, which adds up greatly over time- advantages like quick decompression on workstations and better battery life on portable portable media players.

An increase of the maximum predictor count will decrease the encoding speed (a little) and the worst case (minimal) decoding speed by a significant amount. The first is ok, but the latter is relevant to me...
...
Nevertheless i could be tempted to accept some decoding speed decreases if it would help to compress those files (usually dynamically compressed...

The increase in predictors could be what the 'archival users' would like, and those users might not notice the decoding performance. A suggestion I can make is increased predictors would be an option only available to -p4 (and then pray someone doesn't complain that -p4x works but -p1x isn't smaller than -p1m).
As for dynamically compressed material- it appears, unfortunately, that it's here to stay. Further up in this thread was my disappointment that dynamic compression found its way into DTS too, which led to my half-joking comment of "compressing decoded AC3" just because the loudness standard is enforced with those files. Yes, I plan to do it anyway- just because I want to compare how TAK handles the loud DTS and the normal, non-brickwalled audio.

[_m²_]

However, users that are archiving and seldom access those archived files might not see the advantage of TAK's decoding speed, which adds up greatly over time- advantages like quick decompression on workstations and better battery life on portable portable media players.

I regularly access my music and for me the difference is small as long as I can achieve realtime decoding speed. For portable players I have ogg, it's files are small and decoding is efficient.

This post has been edited by _m²_: May 22 2011, 21:39

[TBeck]

In the past I used TIMER.EXE "kernel time" to spot disk IO bottlenecks with decoding. I was curious if there was a better way to measure the 'performance offset' caused by disk IO saturation.

I plan to perform an own comparison for TAK's homepage (i also plan to put a lot more info there, but somehow i never find the time to do it...) and then i will test it with a ramdisk. I have no clue if this is a good approach. I'll have to try it.

The increase in predictors could be what the 'archival users' would like, and those users might not notice the decoding performance. A suggestion I can make is increased predictors would be an option only available to -p4 (and then pray someone doesn't complain that -p4x works but -p1x isn't smaller than -p1m).

And then there are always users who judge the speed only by the strongest setting.

If i increase the predictor count, this will be done for p4.

Unfortunately i am not feeling very comfortable with such an increase, because

- The advantage will not be big. My primary test corpus, which contains quite some files that benefit from more predictors, compresses only about 0.15 percent (relative to the original file size) better. It's up to about 1 percent for rare files only. This is surely not enough to attract users who prefer OptimFrog.

- Bigger frames would help a bit, and frames which depend on previous frames even more. But i prefer small independent frames, which limit the possible loss in case of a bit error to only a small amount of samples.

- Mpeg4Als -7 is doing the above. Furthermore it uses higher bit resolution multiplications for the filters. This may bring an advantage of about 0.3 to 0.4 percent, but at the cost of also much slower processing. I wouldn't want to sacrifice most of TAK's speed advantages for this little. Ok, i am sure i could write an ALS-encoder which is a lot faster than the reference implementation, but still significantly slower than TAK.

- I am quite sure, i would have to use similar technologies as the symmetrical codecs to be able to compete with them compression wise. But this would be a whole new project. I was not joking when i said i invested thousands of hours in TAK's development. Ok, it would be less for such a new codec, because i already know a lot which would be useful.

[Destroid]

I realized my suggestion from my prior post was lacking forethought, but it was too late to edit. Perhaps it's safe to say TAK is not the "end all" for lossless codecs, and- most likely- there is no such thing as the one codec that is best at everything. But I also think TAK has tangible and enviable advantages.

Disregard my unenlightened suggestions, thanks!

[TBeck]

I realized my suggestion from my prior post was lacking forethought, but it was too late to edit. Perhaps it's safe to say TAK is not the "end all" for lossless codecs, and- most likely- there is no such thing as the one codec that is best at everything. But I also think TAK has tangible and enviable advantages.

Disregard my unenlightened suggestions, thanks!

oh no!

Sometimes my statements may sound a bit short and therefore harsh, because my english still is not really fluid.

And i had to think about it before i could answer. I know, i thought about this issue in the past, but forgot it. I should write a FAQ, especially for myself...

[Ljubo44]

Where I can to download TAK 2.2.0 encoder

[marc2003]

read the thread?

Then i will perform a lot of testing and finally prepare an alpha release of TAK 2.2. I can't tell you, how long this will take. Maybe 2 weeks.