Yet another lossless audio compressor..., Would it make any sense? Options

[TBeck]

Hi,

(sorry, my english isn't very good...)

i have been working (for fun) on lossless audio compression since about 1997. Finally i would like to bring this thing (especially the never-ending-search for just a tenth of a percent more compression...) to an end. In the light of the big bunch of existing Compressors, i am not quite sure, if it would be of any use to add one more Compressor to the public. The preparations for a useful release would be much further work, and i wouldn't like to waste my time for something not needed.

My Compressor uses similar techniques like FLAC, but far more elaborated. Compression ratios lie between Monkey's Audios High- and Extra-High-Mode (Can be better than Extra High at the expense of a considerable increase of encoding time). Encoding Speed is a bit slower than Monkey's, Decoding Speed is much higher on most Files. Seek-Times should also be better cause of the maximum (independent) frame length of 250 ms.

I would like to read some opinions. Would it make any sense to release it?

Thanks

Thomas

[kornchild2002]

In the beginning, I really didn't understand lossless (this was back in 2003). I didn't really see the 10% compression ratio as being significantly smaller than the original wav and, to me, a much smaller mp3 had the same sound.

Now I understand the importance of lossless encoders in that they can retain tags (something wav files can't do), some offer better than 10% compression ratios, and many programs now support them.

I would welcome another lossless encoder. After all, competition is good. Your encoder may not use newer technology when compared with Apple lossless for Monkey's Audio but still, it would be nice to see.

I am currently in the process of ripping my entire library to Apple lossless (I live in a iPod world).

Then again, my interest is purely for curiousity. Sadly, I am afraid that many people are set with their lossless encoders with most people going with FLAC. Unless your lossless encoder would obtain a high compatibility with current software and hardware, then I really don't see it going anywhere beyond curiosity and testing. Please, don't take this in the wrong way what so ever. I would love to see a lossless format developed by a HA poster. I just don't know if it would have any practical use or not.

[TBeck]

Unless your lossless encoder would obtain a high compatibility with current software and hardware, then I really don't see it going anywhere beyond curiosity and testing.  Please, don't take this in the wrong way what so ever.  I would love to see a lossless format developed by a HA poster.  I just don't know if it would have any practical use or not.

Thanks.

My Thinking goes into the same direction. Sigh...

Building of the compression engine has been very much work. The creation and promotion of a new (free) format, which seems to be necessary to make the technology useful, would be even more work. My biggest respect for Josh Coalson, who has made FLAC some standard.

I'm in doubt that i myself would be able to establish some new standard. And i'm not sure, if it would make sense. It's a pity, that i am too late. Some years ago my work possibly would have had a chance to enrich the development of FLAC.

Thomas

[jcoalson]

Building of the compression engine has been very much work. The creation and promotion of a new (free) format, which seems to be necessary to make the technology useful, would be even more work.

I would agree with that... from my experience, development of the actual compression algorithm takes the least part of time for a successful codec. algorithm-wise FLAC is not that much different than shorten. the vast majority of time for me was spent in trying to make it useful (format spec, portable libraries, docs, test suites, all the features people want in a codec like a metadata system, support tools, etc.) and external stuff like project administration, releases, ...

I'm in doubt that i myself would be able to establish some new standard. And i'm not sure, if it would make sense. It's a pity, that i am too late. Some years ago my work possibly would have had a chance to enrich the development of FLAC.

it still could if it's compatible with the FLAC goals; it's not too late. your table doesn't have the FLAC decoding times to compare against but if you are getting an extra 10% without more decode complexity that is very promising.

Josh

[TBeck]

it still could if it's compatible with the FLAC goals; it's not too late.  your table doesn't have the FLAC decoding times to compare against but if you are getting an extra 10% without more decode complexity that is very promising.

Josh

Here some more comparison now including Timings for FLAC.
Cause i had to use a stop watch for flac, i only tested the
biggest files to reduce the effect of the measuring equipment
(my thumb isn't too accurate...).

The first two columns now include some variations of the HIGH-Mode
with reduced predictor numbers. They could compress better without
increasing decoding time, if they used the same methods for paramter
calculation as Extra or Insane modes. I just wasn't to lazy.

The bottom rows show the speed, that TAK achieves without the use
of MMX-Instructions or general assembler optimizations. This could be
interesting, if the code would be ported to some platform without such
instructions.

CODE

           TAK     TAK     TAK                       FLAC
Mode:       High    High    High    Extra   Insane |  -8     |
Predictors: 32      64      128     256     384    |         |
---------------------------------------------------+---------+
Song_02     48,96   48,69   48.41   47.87   47.73  |  51.03  |
Song_04     34,19   33,84   33.15   32.59   32.56  |  37.27  |
Song_06     34,30   34,04   33.74   33.34   33.20  |  37.04  |
Song_08     45,74   45,31   44.97   44.56   44.45  |  49.74  |
Song_10     56,84   56,71   56.41   56.00   55.94  |  59.10  |
Song_12     54,13   53,93   53.86   53.33   53.27  |  57.62  |
Song_14     49,14   49,07   48.97   48.51   48.44  |  51.87  |
Song_16     74,17   74,16   74.16   73.82   73.79  |  75.95  |
---------------------------------------------------+---------+
Sum:        48,40   48,15   47.86   47.41   47.33  |  51.35  |
---------------------------------------------------+---------+
Times with the use of MMX:                         |         |
EncoTime:   41,59   45,14   53.01  270.94  595.41  | 191.00  |
DecoTime:   11,35   12,24   13.50   14.90   15.19  |  20.00  |
---------------------------------------------------+---------+
Times without the use of MMX:                      |         |
EncoTime:   63,08   73,87   91,65  638,81 1350,89  | ---.--  |
DecoTime:   16,40   19,46   24,77   31,46   33,38  | ---.--  |
---------------------------------------------------+---------+

To be honest, the comparison of the three leftmost columns of TAK with FLAC isn't
quite fair, because TAK actually doesn't measure the time needed for Disk-IO (my
40 GB Disk is quite slow). This shouldn't significantly affect the validity of
the other two modes, were the calculation overhead is much higher.

Possibly interesting to see, that the MMX-Implementation of the Modes with
low predictor order isn't considerably faster than the implementation without.
That's caused by some overhead introduced by the scaling and other preparations
of the data needed for the use of MMX.

Thomas

[jcoalson]

it still could if it's compatible with the FLAC goals; it's not too late.  your table doesn't have the FLAC decoding times to compare against but if you are getting an extra 10% without more decode complexity that is very promising.

Here some more comparison now including Timings for FLAC.
Cause i had to use a stop watch for flac, i only tested the
biggest files to reduce the effect of the measuring equipment
(my thumb isn't too accurate...).

Thomas, this definitely looks promising. the design of FLAC allows new methods to be added while preserving backwards compatibility. the main reasons I haven't added any so far is because they all either 1) had too little compresion gain; 2) significantly increased decode complexity; 3) were patent encumbered.

it's hard to say in this crazy world but assuming you're method doesn't run afoul of some patent (will take some research), you seem to have overcome all 3. there are other methods I have not added to FLAC just because the compression increase wasn't worth it, since the code is running in many devices and it would be some work to get it out to all the manufactures. but a consistent +10% would be worth it; at the same time I could add all the little stuff I haven't put in yet.

as you know FLAC and libFLAC are meant to be as open and free as possible so if you are OK with that, there is no reason your method cannot be incorporated. (if you're looking to make some economic benefit from it, that's much harder. the only I options see for that are to join a giant like dolby or free it and use any success of it as a reference.)

the next step would be to go over the details of your method. that could be done here or on the flac -dev list where more FLAC devs would see it.

Josh