Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Earguy\'s improved Digital Ear (Read 26081 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Earguy\'s improved Digital Ear

Earguy Digital Ear based on Frank Baumgarte's work is now improved.
Quote
Hello all-

First some catch-up.  I finally had some time, as of late, to get back into the ear model project.
I reread Frank Baumgarte's PhD Dissertation paper (twice) and discovered that I had used the incorrect parameters for the model.  The parameters I used were meant for using the ear model as an encoder, and not for using the ear model as a typical listener.  The reason for this mistake is that the paper is written in German and I can only read what Babel translates for me.
I have since placed the parameters I'm interested in (the ones representing a typical listener) into the ear model program.

I am using a set of 100 random ten second samples from my CD collection ranging from classical to techno.  I encode all 100 wavs using the compression tool, plus settings, that I'm interested in evaluating.  I then decode the 100 clips back into wavs using WinAmp 2.78c with the proper decoding plugin for the compression tool used.  Then, using CEP, I batch process both the original wavs and encoded/decoded wavs to 88200 sampling rate, mono, 32-bit float (making sure to align the waveform samples).  The ear model program, named Virtual Listener (VL for short), then "listens" to both the reference wavs and test wavs simultaneously and determines if it hears a difference between the two.

I have run several different compression tools and settings and created a quick web page to display the results:
Virtual Listener results

If there are other settings that you would like to see "evaluated", let me know.
Juha Laaksonheimo

Earguy\'s improved Digital Ear

Reply #1 –
*cough* I hope Todd is not afraid because of the discussion after EAQUAL release..
I'm particularly interested how well Virtual Listener hears pre-echo.
Juha Laaksonheimo

Earguy\'s improved Digital Ear

Reply #2 –
I really hope Todd (let's get personal  ) also includes some ogg results, I'm really interested in those.
I took a quick look at the results and am I correct that alt-preset insane is the best??
That would be a surprise, considering MPC.
-->xmixahlx<-- learn the truth about audio-compression

Earguy\'s improved Digital Ear

Reply #3 –
Do you know what FFT window size it uses?

edit: The score of lame makes me suspect a fft window size of 1024 maybe, at that frequency resolution the time resolution it's equal to 11.61 msecs (insufficient), but but it is only a supposition  .

Earguy\'s improved Digital Ear

Reply #4 –
Holy cow!
Does this suggest that lame extreme is better than mpc extreme?????





Jan.

Earguy\'s improved Digital Ear

Reply #5 –
Quote
Originally posted by Jansemanden
Holy cow!
Does this suggest that lame extreme is better than mpc extreme?????
Jan.
Heh, why always when new quality measurement tool is introduced people go crazy?
I don't even see MPC xtreme results there. Only mpc standard and insane.

Again, this is just a tool, not necesseraly "the truth".
Juha Laaksonheimo

Earguy\'s improved Digital Ear

Reply #6 –
I know it isn't the truth, but it is staggering how Lame clearly wins this test over mpc.

Earguy\'s improved Digital Ear

Reply #7 –
Quote
Originally posted by Jansemanden
I know it isn't the truth, but it is staggering how Lame clearly wins this test over mpc.
Umm, in my opinion even this test shows mpc wins clearly. Look at the bitrates and average difference values... And Lame extreme (244kbps average) is barely better than MPC standard (172kbps average).
239kbps average MPC is clearly better than 244kbps average Lame. I really don't see any staggering win by Lame???
Juha Laaksonheimo

Earguy\'s improved Digital Ear

Reply #8 –
As i already said on the r3 board, he didn't use Frank Klemm's latest MPC encoder, which has a bugfix for low frequencies... maybe that's why the results are less good than MP3 in the bass regions?

Earguy\'s improved Digital Ear

Reply #9 –
The biggest difference between EAQUAL and Virtual Listener is that EAQUAL works in the frequency domain whereas VL works in the time domain.  So, to answer your question amp there is no FFT window size. 

EAQUAL wasn't meant to be a quality measurement tool, just a psymodel tweaking tool.  Frank Baumgarte's ear model, which VL is based on, IS meant to represent a human ear to measure audible differences.  Since everyone hears differently, consider VL to be just another person with the particular traits that it always listens the same way and never just fatigued.  It also is the ultimate ABX test in that it can listen to both the reference and test wavs simultaneously.

Also, even though LAME --alt-preset insane appears better than MPC -insane, look at the avg bitrate difference, MPC is still pretty impressive.  There are more compressors/settings I am running and I will keep adding to the results (Ogg is definately on my to-do list).

JohnV, what would be the best way to have VL listen for pre-echo?  I need some ideas on conducting a pre-echo test.

NOTE:
On a PIII 1GHz it takes VL 1 hour to listen to 1 minute of music.  So the 100*10 sec samples takes 16 hours and 40 minutes to process.  That means that one compressor/setting per day will be analyzed and added to the results, keep checking and do with the results as you wish.

Earguy\'s improved Digital Ear

Reply #10 –
The difference I notice was the difference mpc have at 1400hz-5000hz where lame has pratically no difference.

Earguy\'s improved Digital Ear

Reply #11 –
Wow, three posts when I was composing my post...

CiTay- I will rerun the MPC --insane test with Klemm's latest encoder to see if there is an improvement.

Earguy\'s improved Digital Ear

Reply #12 –
Quote
Originally posted by Jansemanden The difference I notice was the difference mpc have at 1400hz-5000hz where lame has pratically no difference.
And the bitrate advantage for mp3 in that case is "only" 84kbps.
Juha Laaksonheimo

Earguy\'s improved Digital Ear

Reply #13 –
Quote
Originally posted by EarGuy
JohnV, what would be the best way to have VL listen for pre-echo?  I need some ideas on conducting a pre-echo test.
In my opinion, it would be best if you tested some pre-echo test clips, and give codec quality score for each clip separately. Also would be good if bitrates between codecs were pretty close to each other. That way we can find out if VL really is good at detecting pre-echo.

Some pre-echo test clips:
http://sivut.koti.soon.fi/julaak/blips.flac
http://www.ff123.net/samples/drone_short.flac
http://sivut.koti.soon.fi/julaak/death2.flac
http://www.ff123.net/samples/florida_seq.flac
http://lame.sourceforge.net/download/samples/castanets.wav
http://sivut.koti.soon.fi/julaak/castanets_si02.flac
http://lame.sourceforge.net/download/samples/fatboy.wav
http://lame.sourceforge.net/download/samples/testsignal2.wav
Juha Laaksonheimo

Earguy\'s improved Digital Ear

Reply #14 –
But mpc was supose to always be better than lame at bitrates above 180 or something.

Earguy\'s improved Digital Ear

Reply #15 –
Quote
Originally posted by Jansemanden
But mpc was supose to always be better than lame at bitrates above 180 or something.
Well it sure looks like it is. Or do you mean that under 160kbps mpc might be equal to mp3, and suddenly 180kbps mpc should be always higher quality than 320kbps mp3?
Again remember that this is only one artifical listener. Don't do any final conclusions based on only it.
Juha Laaksonheimo

Earguy\'s improved Digital Ear

Reply #16 –
It would also be nice to compare VL against the group listening tests, particularly dogies.wav and wayitis.wav.  If you are interested in doing this, Earguy, I can make the files available.

EAQUAL pretty much failed to achieve a good comparison on dogies.wav, in my opinion.  Perhaps VL does better?

ff123

Earguy\'s improved Digital Ear

Reply #17 –
Quote
Well it sure looks like it is. Or do you mean that under 160kbps mpc might be equal to mp3, and suddenly 180kbps mpc should be always higher quality than 320kbps mp3?


No, but we always tell newbies that if they are after quality and can effort bitrates 180+ mpc is the superior choice.
This does not suggest that as I see it. I don't know much about this testing method and I'm not saing what I think I see it what I should figure from this but I just said that it looked strange to me.



Jan.

Earguy\'s improved Digital Ear

Reply #18 –
Quote
Originally posted by Jansemanden
No, but we always tell newbies that if they are after quality and can effort bitrates 180+ mpc is the superior choice.
Well, I see it differently. mpc is clearly superior at equal bitrate level, even according to this test.
Juha Laaksonheimo

Earguy\'s improved Digital Ear

Reply #19 –
ff123- Yes I'm interested.  Could you put the dogies and wayitis files somewhere and email me the location?

Earguy\'s improved Digital Ear

Reply #20 –
Quote
Originally posted by JohnV
Well, I see it differently. mpc is clearly superior at equal bitrate level, even according to this test.

Agreed.

MPC SV7 (1.7.9c) -standard has a lower avg diff value (18% smaller) than LAME 3.91 --aps and yet it requires 14% less disk space.

MPC SV7 (1.7.9c) -insane has a lower avg diff value (48% smaller) than LAME 3.91 --ape and yet it requires 2% less disk space.

LAME 3.91 -api pulls in a very good performance, but how does MPC perform at that bitrate? LAME 3.91 --api may be better than MPC -insane, but it requires 35% more disk space.

I've never used the 1.7.x encoders, just Klemm's 0.9x encoders, so I'd be curious to see how they stand up.

EDIT: Dibrom's custom tweaks on the 3 alt-presets look like they really pay off. Compare --ap 192 to --aps: the latter has 47% smaller avg diff value but only requires 3% more disk space. If it wasn't for the new 3 presets, LAME wouldn't stand a chance against MPC if --ap 192 is any indication.

Earguy\'s improved Digital Ear

Reply #21 –
And of course, we can't download the Virtual Listener application, can't we?

Earguy\'s improved Digital Ear

Reply #22 –
Earguy,

I'll upload to my samples site tonight.  I've already taken care of any offset in the files to within one or two samples (45 microsec offset error at most) and of course they're all level matched.

If you test lame cbr/abr presets, you should keep in mind that some scaling might be done.

Criteria for saying that VL successfully correlates with group listening tests on the samples I provide:

dogies:  mpc should come in first, ogg, lame, and wma is the next group down, which should be significantly worse than mpc (although the order within the group is allowed to be mixed up).  xing should be significantly worse than the ogg/lame/wma group.

wayitis:  mpc/ogg should be first (with order allowed to be mixed up).  aac/lame should be next, and wma/xing should be last.

ff123

Earguy\'s improved Digital Ear

Reply #23 –
I don't know why everyone finds these results so shocking except as a result of the constant harping about MPC being the "best" encooder there is. You ask about pre echo artifacts - has anyone EVER detected ANY artifacts on an alt insane LAME file???

Earguy\'s improved Digital Ear

Reply #24 –
The results aren't shocking. They demonstrate that MPC is superior to LAME at equal bitrates...something we've known for a while. The constant harping about MPC is deserved.

Artifacts can be conceivably found in any lossy format, including --api. Maybe these artifacts are benign or rare, but don't kid yourself into thinking that --api is perfect because you aren't aware of any artifacted samples. I think dibrom might be able to point you in the right direction since he is so familiar with the alt-preset code...and what still needs improvement. I don't bother with --api because I don't care for the fixed 320kbps size. Even tweaked MPC xtreme files stay below 320kbps 99% of the time...and you can bet that MPC has less chance of hiccuping than MP3 on the tough stuff. MP3 has design flaws that cannot be eliminated with bitrate alone. You can use many encoding tricks to minimize the effects, but MP3 has a more lazy time resolution than other formats.

I do have a grudge against transform encoders that aren't time critical. If you decode a string of MP3s and paste the resulting WAVs together, the musical information won't match up. Try a live album and see what I mean...the gaps between tracks are annoying. Sure you can make one big MP3, but it's not convenient.

And LAME is too slow. That's not necessarily an indictment against the development team, but the transform encoders require a ton of horsepower to generate a high level of quality. MPC standard encodes 2.5x faster than LAME --aps. That can be a important advantage.