IPB

Welcome Guest ( Log In | Register )

2 Pages V  < 1 2  
Reply to this topicStart new topic
New lossless codec comparison (Jan '13), Lots of test results, lots of graphs
Soap
post Apr 5 2013, 23:44
Post #26





Group: Members
Posts: 1001
Joined: 19-November 06
Member No.: 37767



QUOTE (spicymeatball77 @ Apr 5 2013, 14:13) *
I transcoded my FLACs from -8 to -5 and there was *very* little size increase. This will likely help out my battery life at a tiny size cost. Thanks for the input guys.


saratoga can probably speak as to the specifics of that player, but on Porta Player devices the difference in decode speed would equal ZERO, not a little, but zero power savings as the CPU doesn't clock lower than 30Mhz. Whereas the size difference would equal (probably insignificant but) real power consumption differences due to increased storage access. Flash obviously cuts the power consumption of storage access down vs spinning rust, but the number is real.


--------------------
Creature of habit.
Go to the top of the page
+Quote Post
ChronoSphere
post Apr 7 2013, 14:31
Post #27





Group: Members
Posts: 399
Joined: 11-March 07
Member No.: 41384



QUOTE (spicymeatball77 @ Apr 5 2013, 15:26) *
I've got a bit of a noob question. I just RockBox'd a Sansa Clip+ and I'm trying to find the best lossless encoding that balances file size and battery life. Right now I'm using FLAC-8. I'm assuming that on the decoding graph a higher value on the X-axis equates to better decompression ie. better battery life?
I've ran a test on my clip+ with an album encoded with flac -11 (CUETools's flaCL) and the same with wavpack -hhx and the difference was only about 10 minutes in runtime, so I'd say lossless encoding options don't affect battery life that much anymore.

This post has been edited by ChronoSphere: Apr 7 2013, 14:31
Go to the top of the page
+Quote Post
TBeck
post Jun 21 2013, 22:19
Post #28


TAK Developer


Group: Developer
Posts: 1095
Joined: 1-April 06
Member No.: 29051



I should have replied a lot earlier, because your publication immediately made me very happy.

Eventually a lossless codec comparison i feel i can trust!

In the past it was Synthetic Soul's comparison, although his sample selection was not sufficiently diverse (at least for my purposes).

When i am thinking about tuning opportunities for my codec, i often look at your results.

I think, you aren't getting as much feedback as you deserved here at hydrogen, but hey, your comparison has been published on the FLAC web site, and i am sure, the guys over there too know what constitutes a meaningful lossless codec comparison.

I must admit, i am selfish. I wanted to thank you but i also want to encourage you to continue your valuable work. Because it's so important to me, to have an independent instance, which verifies the outcome of my work.

Again

thank you! rolleyes.gif

Thomas
Go to the top of the page
+Quote Post
ktf
post Jun 22 2013, 10:43
Post #29





Group: Members
Posts: 304
Joined: 22-March 09
Member No.: 68263



QUOTE (TBeck @ Jun 21 2013, 23:19) *
I should have replied a lot earlier, because your publication immediately made me very happy.

Eventually a lossless codec comparison i feel i can trust!

Well, that's nice to hear!

QUOTE
In the past it was Synthetic Soul's comparison, although his sample selection was not sufficiently diverse (at least for my purposes).

As the issue is brought up now, I would like to ask for a little help once again. I've been looking for more music to make the sample selection even more diverse, and I now have this list of 43 albums, where the previous comparison was with 29 albums.

  • 30 Seconds to Mars - This Is War (rock)
  • Abba - Gold: Greatest Hits (1992) (pop)
  • Albert King - Years Gone By (blues)
  • Alicia Keys - The Diary of Alicia Keys (R&B, soul)
  • Alison Krauss Union Station - New Favorite (country, bluegrass)
  • Armik - Rain Dance (flamenco/jazz)
  • Café del Mar - Volumen nueve (chill-out)
  • Coldplay - Parachutes (pop)
  • Confido Domino Minsk - Sacred choral music from White Russia (choral, religious)
  • Daft Punk - Alive 2007 (electronic)
  • Dan Brown - Angels and Demons (audiobook)
  • Enya - Amarantine (new age)
  • Fanfare Ciocarlia - Baro Biao: World Wide Wedding (world music, brass)
  • Gilberto Santa Rosa - Esencia (salsa)
  • Giuseppe Verdi - Messa da Requiem (Berliner Philharmoniker feat. conductor Claudio Abbado) (orchestral, romantic)
  • Howard Shore - The Hobbit: An Unexpected Journey (orchestral, OST)
  • Iron Maiden - Brave New World (metal)
  • James Brown - Love, Power, Peace (Live at the Olympia, Paris, 1971) (funk)
  • John Coltrane - Live at the half note, vol. 1 (jazz)
  • Joss Stone - Mind, Body & Soul (soul)
  • J.S. Bach - Magnificat (orchestral, baroque)
  • Koninklijke Militaire Kapel - [no name] (military brass)
  • Kraftwerk - Autobahn (electronic)
  • Lana del Rey - Born to Die (pop)
  • Lil Wayne - Tha Carter III (hip hop)
  • Martin O'Donnell & Michael Salvatori - Halo: Reach (ambient-ish, OST)
  • Metallica - Death Magnetic (metal)
  • Michael Bublé - meets madison square garden (pop, jazz)
  • Mike Oldfield - Tubular Bells (2000 remaster) (not classifiable)
  • 'N Sync - Home for Christmas (pop)
  • Nicholas Hooper - Harry Potter and the Order of the Phoenix (soundtrack)
  • Peter Tosh - Equal Rights (2011 legacy edition) (reggae)
  • Rosenberg Trio - Djangologists (jazz)
  • Rush - Grace under Pressure (1997 remaster) (rock)
  • System of a Down - Mezmerize (metal)
  • Szakcsi - Virágom, virágom (world music)
  • Tiësto - In Search of Sunrise 7: Asia (dance)
  • The Lumineers - The Lumineers (folk)
  • Various - Jeff Waynes Musical Version of The War of the Worlds (radio drama-ish, rock, orchestral)
  • Waylon Jennings - Honky Tonk Heroes (country)
  • Various - Latin Village, CD 1: Salsa (salsa)
  • Xzibit - At the Speed of Life (hip hop)
  • Yann Tiersen - Le Fabuleux Destin d'Amélie Poulain (minimalism)


The bold items are new. I think this is a nice list so far, any comments?

QUOTE
I must admit, i am selfish. I wanted to thank you but i also want to encourage you to continue your valuable work. Because it's so important to me, to have an independent instance, which verifies the outcome of my work.

But the comparison wasn't totally fair yet. I ran your codec in Wine, and while one could argue Wine doesn't add much overhead, it's a little weird I didn't run FLAC trough wine too, for example.

Anyway, the next version of this comparison will have all codecs tested on a Windows computer, as I finally figured out a utility that reported the CPU cycles used by a certain program instead of time elapsed. I rewrote my scripts to fit in a Windows enviroment, technically I'm ready to run it, but I am waiting for WavPack 4.70 final to be released (so I can test new versions of FLAC, TAK, Monkeys Audio and WavPack), so the result will be published somewhere this summer I suppose.

If there are still any remarks on the test setup, please tell me. I can still fix things now smile.gif

edit: oh, and for the record, I will be testing on per-track basis instead of per-album basis now. Every album will be weighted equally just as the previous comparison, but now I will be able to identify outliers. Maybe something interesting pops up.

This post has been edited by ktf: Jun 22 2013, 11:42


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
ozok
post Jun 22 2013, 10:49
Post #30





Group: Members
Posts: 292
Joined: 9-December 12
From: Eskişehir
Member No.: 105075



QUOTE (ktf @ Jun 22 2013, 12:43) *
...
Anyway, the next version of this comparison will have all codecs tested on a Windows computer, as I finally figured out a utility that reported the CPU cycles used by a certain program instead of time elapsed.
...

What is the name of this utility?
Go to the top of the page
+Quote Post
ktf
post Jun 22 2013, 11:07
Post #31





Group: Members
Posts: 304
Joined: 22-March 09
Member No.: 68263



QUOTE (ozok @ Jun 22 2013, 11:49) *
What is the name of this utility?

It's timeit.exe, from the Windows Server 2003 Resource Kit Tools. However, it doesn't work with Windows 7 (I run Windows XP) but I found something that indicated powershell might have similar capabilities, but I didn't test. If anyone is going to dive into this, please report any finding, I'm not entirely sure this timeit utility reports are resembling the truth. It says process time, but the results I got back were sometimes weird for no apparent reason.

This post has been edited by ktf: Jun 22 2013, 11:30


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
ChronoSphere
post Jun 22 2013, 13:16
Post #32





Group: Members
Posts: 399
Joined: 11-March 07
Member No.: 41384



QUOTE (ChronoSphere @ Apr 7 2013, 15:31) *
QUOTE (spicymeatball77 @ Apr 5 2013, 15:26) *
I've got a bit of a noob question. I just RockBox'd a Sansa Clip+ and I'm trying to find the best lossless encoding that balances file size and battery life. Right now I'm using FLAC-8. I'm assuming that on the decoding graph a higher value on the X-axis equates to better decompression ie. better battery life?
I've ran a test on my clip+ with an album encoded with flac -11 (CUETools's flaCL) and the same with wavpack -hhx and the difference was only about 10 minutes in runtime, so I'd say lossless encoding options don't affect battery life that much anymore.
To update on this, I ran some more tests and the differences are bigger. I found out flac -11 sometimes fails to decode on the clip+ (probably because it's not a subset file), so I switched to flac -8 and battery wise, flac is a clear winner:

CODE
Flac -8:         16:09:49
Wv -hh -b384x:    11:17:12
mp3 v0:            11:48:20
vorbis q5:        12:17:56
opus 160:        10:17:00


Complete battery_bench logs here
Go to the top of the page
+Quote Post
marc2003
post Jun 22 2013, 13:35
Post #33





Group: Members
Posts: 4326
Joined: 27-January 05
From: England
Member No.: 19379



QUOTE (ktf @ Jun 22 2013, 11:07) *
It's timeit.exe, from the Windows Server 2003 Resource Kit Tools. However, it doesn't work with Windows 7 (I run Windows XP) but I found something that indicated powershell might have similar capabilities, but I didn't test.


just yesterday i was wondering if there was a tool for benchmarking command line programs and i found this.

QUOTE (lvqcl @ Feb 18 2012, 18:32) *


it works on windows 7. i also tried the powershell command. you can see the results here - i did a quick test of flac and takc on a single album image.

https://dl.dropboxusercontent.com/u/2280132.../june/bench.png

This post has been edited by marc2003: Jun 22 2013, 13:40
Go to the top of the page
+Quote Post
Wombat
post Jun 22 2013, 14:13
Post #34





Group: Members
Posts: 950
Joined: 7-October 01
Member No.: 235



I hope this time your Windows computer can run flacCL. Would be nice to see it included.
Go to the top of the page
+Quote Post
ozok
post Jun 22 2013, 15:57
Post #35





Group: Members
Posts: 292
Joined: 9-December 12
From: Eskişehir
Member No.: 105075



Thanks ktf.
Go to the top of the page
+Quote Post
ktf
post Jun 23 2013, 08:16
Post #36





Group: Members
Posts: 304
Joined: 22-March 09
Member No.: 68263



QUOTE (marc2003 @ Jun 22 2013, 14:35) *
just yesterday i was wondering if there was a tool for benchmarking command line programs and i found this.

Yes, I saw that one too, but the problem is it reports running time, not CPU time used. I just took another look at powershell, it seems to report real time as well, not CPU time.

QUOTE (Wombat @ Jun 22 2013, 15:13) *
I hope this time your Windows computer can run flacCL. Would be nice to see it included.

I'm sorry, the test needs several weeks to run, so I can only use an old computer that's not longer used from before the GPGPU era to do this work.

However, I might as well add a section on different FLAC-encoders run on a different computer, but I will have to work that out.

This post has been edited by ktf: Jun 23 2013, 08:23


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
lvqcl
post Jun 23 2013, 08:51
Post #37





Group: Developer
Posts: 3209
Joined: 2-December 07
Member No.: 49183



QUOTE (ktf @ Jun 23 2013, 11:16) *
Yes, I saw that one too, but the problem is it reports running time, not CPU time used.

I thought that "Process Time" is CPU time. What is the difference between them?
Go to the top of the page
+Quote Post
[JAZ]
post Jun 23 2013, 10:19
Post #38





Group: Members
Posts: 1706
Joined: 24-June 02
From: Catalunya(Spain)
Member No.: 2383



Since common PCs do have more than one running core for several years now, there is a distinction between the time required for a task (elapsed time) and the time required for one working unit to complete it (cpu time).

In Windows, there's a gauge showing the usage from 0 to 100%, but in linux, the "top" utility can show usages of 150%, 200% or more. This is translated as using more than the time that a single working unit is able to provide.

This post has been edited by [JAZ]: Jun 23 2013, 10:19
Go to the top of the page
+Quote Post
lvqcl
post Jun 23 2013, 11:25
Post #39





Group: Developer
Posts: 3209
Joined: 2-December 07
Member No.: 49183



QUOTE (ktf @ Jun 23 2013, 11:16) *
I'm sorry, the test needs several weeks to run, so I can only use an old computer that's not longer used from before the GPGPU era to do this work.

What CPU does it have and what instructions does it support (SSE, SSE2, etc.)?
Go to the top of the page
+Quote Post
ktf
post Jun 23 2013, 14:10
Post #40





Group: Members
Posts: 304
Joined: 22-March 09
Member No.: 68263



QUOTE (lvqcl @ Jun 23 2013, 09:51) *
I thought that "Process Time" is CPU time. What is the difference between them?

You're right. I don't know why I concluded that 7bench is not able to measure CPU-time, because it clearly can. It does give the same results as timeit (and the same pecularities, like having a smallest increment of 0.015s at my system) so I suppose it uses the same kernel hooks.

QUOTE (lvqcl @ Jun 23 2013, 12:25) *
What CPU does it have and what instructions does it support (SSE, SSE2, etc.)?

If you really want to know, the introduction date of this processor was March 10, 2005, I bought the laptop around the time the first dual-cores hit the mainstream market. Details are below. Wikipedia says: MMX, Enhanced 3DNow!, SSE, SSE2, SSE3, AMD64, PowerNow!, NX Bit
CODE
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 36
model name : AMD Turion™ 64 Mobile Technology ML-34
stepping : 2
cpu MHz : 1800.141
cache size : 1024 KB
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt
lm 3dnowext 3dnow rep_good pni lahf_lm
bogomips : 3600.28
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc


This post has been edited by ktf: Jun 23 2013, 14:14


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
TBeck
post Jun 23 2013, 22:14
Post #41


TAK Developer


Group: Developer
Posts: 1095
Joined: 1-April 06
Member No.: 29051



QUOTE (ktf @ Jun 22 2013, 11:43) *
As the issue is brought up now, I would like to ask for a little help once again. I've been looking for more music to make the sample selection even more diverse, and I now have this list of 43 albums, where the previous comparison was with 29 albums.

Usually more is better. As long as the codec relevant audio properties and there share of all audio files can't be fully determined, i don't know a better way to improve the representativity.

Do you know if your selection contains some hypercompressed files? Since they are so common know, this seems to be one more important category. I don't really like to ask for it, because i know that TAK performs less well on them...

I don't thinky you would need plenty such files. From my experience the effect of other codec relevant audio properties usually is quite small, if a file is hypercompressed.

QUOTE (ktf @ Jun 22 2013, 11:43) *
But the comparison wasn't totally fair yet. I ran your codec in Wine, and while one could argue Wine doesn't add much overhead, it's a little weird I didn't run FLAC trough wine too, for example.

I wouldn't expect a significant effect. I have examined some comparisons which ran in Wine, and the results were in line with windows comparisons. I really like that you care so much about so many details, but i think it's nearly impossible to create a practicable comparison that everybody regards as fair. One could criticize, that you will be using a quite old cpu, because it lacks some instruction set extensions or because possibly newer compilers will not generate optimal code for it. On the other hand there are also good arguments against the use of a newer cpu. But someone will always complain.

I myself feel comfortable with your cpu choice. While it dosen't support SSSE3, what will make TAK's stronger presets somewhat slower, it's microarchitecture doesn't differ too much from more recent cpus.

Well, i probably would protest, if you were using one of those strange Pentium 4 - cpus...

QUOTE (ktf @ Jun 22 2013, 11:43) *
edit: oh, and for the record, I will be testing on per-track basis instead of per-album basis now. Every album will be weighted equally just as the previous comparison, but now I will be able to identify outliers. Maybe something interesting pops up.

Great!
Go to the top of the page
+Quote Post
ktf
post Jun 23 2013, 23:12
Post #42





Group: Members
Posts: 304
Joined: 22-March 09
Member No.: 68263



QUOTE (TBeck @ Jun 23 2013, 23:14) *
Do you know if your selection contains some hypercompressed files? Since they are so common know, this seems to be one more important category. I don't really like to ask for it, because i know that TAK performs less well on them...

I added Metallica's Death Magnetic just because it is probably the best known example of hyper-compressed stuff.

QUOTE
I myself feel comfortable with your cpu choice. While it dosen't support SSSE3, what will make TAK's stronger presets somewhat slower, it's microarchitecture doesn't differ too much from more recent cpus.

Yeah, another complaint might be that I run in 32-bit, while 64-bit is more or less standard these days. Sadly I don't have other machines available. It's either this with Windows XP or the setup as used in the previous comparison. The first results are very, very similar to my 64-bit linux environment (Intel Core2Duo T9600), the only codec that gets a significant performance hit is FLAC's decoding, which is about 20% slower.


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
ktf
post Aug 11 2013, 11:19
Post #43





Group: Members
Posts: 304
Joined: 22-March 09
Member No.: 68263



I just finished a new revision of this codec comparison. It features new versions of FLAC, TAK, Monkey's Audio, WavPack and OptimFROG, runs on Windows instead of Linux and contrary to what I said earlier, the codecs were run on an AMD A4-3400. Furthermore, the test corpus has been expanded from 29 albums to 43 albums. WMA Lossless is now tested properly, refalac has been added and Real Lossless has been dropped. (edit: OS was Windows 7 SP1 64-bit)

No big changes in the results, except TAK being a whole lot faster.

The PDF and raw results are available here: http://www.icer.nl/losslesstest




This post has been edited by ktf: Aug 11 2013, 11:19


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
ChronoSphere
post Aug 11 2013, 11:31
Post #44





Group: Members
Posts: 399
Joined: 11-March 07
Member No.: 41384



It's interesting to see that TAK starts where FLAC stops (decoding vs. compression), it's like they're subsets of the same codec.
Go to the top of the page
+Quote Post
DARcode
post Aug 11 2013, 12:43
Post #45





Group: Members (Donating)
Posts: 679
Joined: 10-January 05
From: Italy
Member No.: 18968



Fantastic work, many thanks for your time and sharing it, keep it up.


--------------------
WavPack 4.70.0 -b384hx6cmv/qaac 2.32 -V 100
Go to the top of the page
+Quote Post
Propheticus
post Aug 11 2013, 16:04
Post #46





Group: Members
Posts: 218
Joined: 10-September 11
Member No.: 93615



Interesting to see the 4th node of TAK (from right to left) is faster than the 2nd and 3rd node while offering better compression. This renders these two, but especially the 3rd, rather useless IMO. Same counts for the 5th and 6th (7th is better and faster), and the 9th (10th is better).

This post has been edited by Propheticus: Aug 11 2013, 16:08
Go to the top of the page
+Quote Post
TBeck
post Aug 11 2013, 17:49
Post #47


TAK Developer


Group: Developer
Posts: 1095
Joined: 1-April 06
Member No.: 29051



From TAK's Readme:

QUOTE
Each preset selects a set of internal encoder options. Some of them affect only the encoding speed, the others also the decoding speed.

Slower decoding usually means higher hardware requirements (memory size and/or cpu power) for a playback device. Some devices may be too weak to support the more demanding internal encoder options. But any option, which only affects the encoding speed and complexity, is always applicable.

Therefore each preset consists of two components:
  • The profile selects internal encoder options which affect both encoding and decoding speed. Profiles are declared as a number from 0 to 4 (fastest to strongest).
  • The evaluation level selects only internal encoder options, which have no effect on the decoding speed. The evaluation levels are named Standard, Extra and Max and abbreviated as s, e and m (fastest to strongest).

A hardware manufacturer supporting TAK has only to specify the strongest profile it's hardware can decode, because the evaluation level does not affect the hardware requirements.

Hint: If you want higher compression and fast encoding, and are able to accept some decrease in decoding speed, it is usually preferable to select a higher profile instead of increasing the evaluation level.
Go to the top of the page
+Quote Post
Propheticus
post Aug 11 2013, 23:47
Post #48





Group: Members
Posts: 218
Joined: 10-September 11
Member No.: 93615



The CPU used in this test might have been too fast compared to mobile devices to show, but I don't see any big variations in the decoding graph between the mentioned profiles.
Go to the top of the page
+Quote Post
TBeck
post Aug 12 2013, 08:25
Post #49


TAK Developer


Group: Developer
Posts: 1095
Joined: 1-April 06
Member No.: 29051



Depending on the profile TAK is using up to 4, 12, 32, 80 or 160 predictors for it's primary filter. A corresponding number of multiplications per sample has to be performed. FLAC supports up to 32 predictors, but it's restricted subset which is recommended for maximum compatibility is limited to 12 (preset -8). And i remember reports on devices not capable to cope with 32 predictors.

I have no practical experience with the DSPs or cpus used in mobile devices, but from what i have read here at hydrogen and at Rockbox, higher predictor counts seem to be critical.

Indeed the decoding speed results show only a minor effect of the perdictor count, but that's misleading. Reasons:

  • TAK will only use as many predictors as are advantegous. Because not many files benefit from high predictor orders, the average predictor count of the profile p4 might be for instance 70 (or even less) and not 160. But if you want to be able to decode any file without stuttering on a slow device, the device has to be powerful enough to execute the maximum of 160 multiplications in realtime.
  • On a desktop system multiplications are really fast, especially if you are using SIMD instructions like MMX or SSEx. On a mobile device they can be a lot slower.


BTW: I think, TAK could be faster in this test. On the AMD A4-3400 it will only use MMX instructions although SSE2 is supported. Currently TAK will only use SSE2 is SSSE3 is supported too. My fault...
Go to the top of the page
+Quote Post
Propheticus
post Aug 12 2013, 08:59
Post #50





Group: Members
Posts: 218
Joined: 10-September 11
Member No.: 93615



Thats a clear explanation. The lack of difference makes sense now. Thanks.
Go to the top of the page
+Quote Post

2 Pages V  < 1 2
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 19th April 2014 - 08:22