Help - Search - Members - Calendar
Full Version: 3.94: call for testing on preset standard
Hydrogenaudio Forums > Lossy Audio Compression > MP3 > MP3 - Tech
Pages: 1, 2
Gabriel
In the new 3.94 compilations you should see some new presets appearing: standard1 and standard2.

I would like you to test standard/standard1/standard2

They winner would (ideally) replace --preset standard.
I'm waiting for your comments.


edit:
Now, there should be a preset standard3.

This is a modified standard2. So if you prefer this one over standard2, I think that you can discard standard2 from your tests.

This standard3 should be better for pre-echo.




(ps: some options used might be surprising for you, but hopefully the blind tests will not take your potential surprise into consideration)
Linkin
where can i get lame 3.94?
SK1
Get 3.94 alpha binaries from john33's "others" page.
mithrandir
The 2003-01-14 compilation on john33's site mustn't be the "new 3.94 compilation" because it errors when you try --preset standard1.
WEST
QUOTE (mithrandir @ Jan 19 2003 - 09:06 AM)
The 2003-01-14 compilation on john33's site mustn't be the "new 3.94 compilation" because it errors when you try --preset standard1.

My compilation has this presets smile.gif
SK1
To make it more clear, his compilations on http://uawest.cjb.net smile.gif
nebob
Here you go. ICL7 with generic optimizations for all CPUs. No compression, extra switches or any similar tomfoolery.

As for the actual tests, aps0 seems to be the slowest and aps1 the fastest. aps0 highest br, aps2 lowest. They all sound pretty much the same to me though.
Gabriel
To make it more clear: I am not very interested in speed results for this test, but mainly in quality results.
mithrandir
Quick ABX test on harp40_1.wav

standard (155kbps): 8/8
standard1 (155kbps): 12/16
standard2 (147kbps): 14/16
standard 3.90.2 (159kbps): 13/16

I failed to ABX standard1, passed on standard/standard2/standard 3.90.2. I probably could ABX standard1 if I concentrated harder and could zoom in on that little annoyance I hear.
guruboolez
Tried on one of my favorite harpsichord sample :

CODE
STD1
1L Rating: 3.0
1L Comment:
---------------------------------------
STD2
2R Rating: 3.6
2R Comment:
---------------------------------------
STD0
3R Rating: 3.1
3R Comment:
---------------------------------------


Std0 = lame 3.90.2 --alt-preset standard
STD0 & STD1 are really close. The 0.1 difference is only fuzzy feeling.


Tried a second time :

CODE
STD2
1R Rating: 4.0
1R Comment:
---------------------------------------
STD1
2L Rating: 3.0
2L Comment:
---------------------------------------
STD0
3L Rating: 3.3


Std2 won again. This time, the old preset seemed to be a bit superior to Std1.

Third trial :

CODE
STD2
1L Rating: 4.2
1L Comment:
---------------------------------------
STD0
2L Rating: 3.0
2L Comment:
---------------------------------------
STD1
3L Rating: 3.3
3L Comment:
---------------------------------------



Preset Nr.2 is clearly superior to the old one and --preset 1 on this sample.
I take a lokk to the bitrate :
STD1 = 179 kbps
STD2 = 197 kbps
STDold = 184 kbps
Gabriel
Bitrates should be similar, but could you please also post your bitrates (just in case one preset would be deviating too much).

It would also be interesting to test against previous "reference" Lame versions.
Wombat
Birds.flac is without artifacts to me only with standard2!

Wombat
guruboolez
Pre-echo test with castanet2.wav sample (first attacks) :

CODE
File         Trial #1            Trial #2            Trial #3

STD0           3.5                 3.5                 3.5
STD1           2.1                 2.2                 2.0
STD2           2.0                 2.0                 2.2



The old --alt-preset standard has more sharpness ; the new presets are one or two steps behind.

EDIT :
Std0 = 189 kbps / 10.9% short-blocks
Std1 = 181 kbps / 12.4% short-blocks
Std2 = 176 kbps / 12.4% short-blocks*

*encspot 2 values


EDIT : table corrected. STD old is now show as the winner. Sorry
Wombat
Sophia2.flac produces small plops with standard2

standard and standard1 is surprising good - no probs!

Wombat
Gabriel
Guruboolez: what are you using as std0? Is it 3.90.2 or 3.94?
SK1
OK i tested with castanets.wav.
standard1 is the best, then the normal standard, worst is standard2.
Actually, as mentioned before, it's right, the old standard has more sharpness with the first attacks, but later standard1 is better, which is pretty weird..

Gabriel you wanted avarage kbps: standard = 234, standard1 = 225, standard2 = 221.

And, i really don't understand why WEST's compile is so much bigger than john33's blink.gif, 660kb compared to 194kb.
guruboolez
QUOTE (Gabriel @ Jan 19 2003 - 07:34 PM)
Guruboolez: what are you using as std0? Is it 3.90.2 or 3.94?

The HA reference version : original 3.90.2.
Gabriel
Guruboolez: it would help if you also test 3.94 preset standard (specially on sharpness/preecho test samples)
SK1
Used another good pre-echo sample, BlackBird.wav (don't remember where i got it..)
Again, standard1 performs best, then standard, worst is standard2. (all of them by the way are far worse than the original!)
standard = 199kbos, standard1 = 192kbps, standard2 = 200kbps... interesting.
[proxima]
I've done a test with le_monde.flac preecho sample at 2.2 - 3.4 seconds

---------------------------------------
File: std394a8.wav
Rating: 4.0
---------------------------------------
File: std1.wav
Rating: 3.5
---------------------------------------
File: std2.wav
Rating: 3.0
---------------------------------------
File: std392.wav
Rating: 2.9
---------------------------------------

v3.94a8 --preset standard is the winner.
guruboolez
QUOTE (Gabriel @ Jan 19 2003 - 07:49 PM)
Guruboolez: it would help if you also test 3.94 preset standard (specially on sharpness/preecho test samples)

I did. But with four files, evaluation is more complex. I'm now tired by testing, and the three last sessions I performed are totally incoherent.

Global results are :

Winners for 5 first sessions : --alt-preset standard (3.90.2) and --alt-preset standard (3.94)
--preset standard1 & --preset standard2 had sytematically the worst notation (with sometime one if the 'winner').

I must try later, or tomorrow, for a confirmation, or an invalidation. The harpsichord test was easier to perform comparison.
mithrandir
QUOTE (SK1 @ Jan 19 2003 - 01:35 PM)
And, i really don't understand why WEST's compile is so much bigger than john33's blink.gif, 660kb compared to 194kb.

He probably doesn't use UPX compression.
john33
QUOTE (mithrandir @ Jan 19 2003 - 07:24 PM)
QUOTE (SK1 @ Jan 19 2003 - 01:35 PM)
And, i really don't understand why WEST's compile is so much bigger than john33's blink.gif, 660kb compared to 194kb.

He probably doesn't use UPX compression.

Quite correct, he doesn't. Sorry that I don't have a compile of the latest libs up yet. It's compiled ready to go, but my ISP has been playing silly bu**ers all day and I can't get to upload at the moment. sad.gif
WEST
QUOTE (john33 @ Jan 19 2003 - 12:24 PM)
QUOTE (mithrandir @ Jan 19 2003 - 07:24 PM)
QUOTE (SK1 @ Jan 19 2003 - 01:35 PM)
And, i really don't understand why WEST's compile is so much bigger than john33's blink.gif, 660kb compared to 194kb.

He probably doesn't use UPX compression.

Quite correct, he doesn't. Sorry that I don't have a compile of the latest libs up yet. It's compiled ready to go, but my ISP has been playing silly bu**ers all day and I can't get to upload at the moment. sad.gif

My compile same as yours, but without UPX compression... blink.gif
john33
New compile now available. smile.gif
cookie
QUOTE (john33 @ Jan 19 2003 - 10:42 PM)
New compile now available. smile.gif

where? unsure.gif
LordofStars
http://homepage.ntlworld.com/jfe1205/others.html


Quite a few other useful programs as well.
ViPER1313
ABX Results 41_30.wav Sample (Used first 2 sec of clip)

Orig vs 3.94 --preset-standard || 12/12 (Warble, pretty easy to ABX)
Orig vs 3.94 --preset-standard1 || 12/12 (Same as above, not nearly as bad)
Orig vs 3.94 --preset-standard2 || 12/12 (Worse of the 3, easy to ABX)

--Preset-standard has a minor warble in the first 2 seconds of the clip. Standard1 is better in this effect,
and is easily the best of the three presets (I had to listen to both samples multiple times at points to hear
the difference.) Standard2 is the worst of the three (didn't even have to listen to both samples to pick the
right one.)

EncSpot Info-

--preset-standard || 223kbps, 73%ss / 27%ms, 84.5% Long / 15.5% Short

--preset-standard1 || 210kbps, 76.9%ss / 23.1%ms, 84.5% Long / 15.5% Short

--preset-standard2 || 189kbps, 76.4%ss / 23.6%ms, 84.5% Long / 15.5% Short


P.S. - I could also ABX this with Vorbis 1.0 -q5, although not for the same reason. Guess i'll have to use -q6 from now on dry.gif
LordofStars
I was able to abx 10/10 --preset standard 3.94a8 against the original wav. I found a simple easy to distinguish artifact, it seems that it was the abx playe that I was using... I will have to decode the files to wav and try again using the same program.
[edit]
Hihat.wav - original
hihat1.wav - 3.94a8 preset standard 11/11
hihat2.wav - 3.94a8 preset standard1 12/12
hihat3.wav - 3.94a8 preset standard2 16/16
hihat4.wav - 3.90.2 preset standard 13/13


Apparently the abx program I am using has a problem with its mp3 decoder. Therefore before I was hearing a decoder introduced artifact. This time I focused on preecho.
I found the order to be 4312.
Gabriel
According to the first posted results, it seems that standard1 and standard2 have a worst preecho compared to 3.90.2
Vietwoojagig
Hi Gabriel,

this kind of testing does't seem to me like a blind test an so not very helpful.

You should place different encoded versions of your test-set decoded as a wav plus the origial wav on your home-page.

Then the people can tell you, which one sounds most like the original, without any notice of the used encoder or bit-rate.

OK, the test-set is more limited but thats the way it should work.

Another advantage: you really know the used switches and versions of the encoder. And no one has to do the encoding-thing by himself.
KikeG
I'd say people doing this kind of tests already know that the tests must be blind, it's a common procedure for any serious testing. About posting wav files, there are quite many problem samples, and it's easier if everyone tries the ones they prefer, using a randomized blind procedure as the implemented on ff123's abc/hr. That's why people here show his results with this program.
Gabriel
I want people to try with any sample they want and know. By choosing the samples myself, I would reduce the range of tested tracks.
We know the version of the encoder used, because standard1/standard2 are only available in 1 version.

For the blind thing, people here are already testing blindly.
Gabriel
For those who are looking at the ammount of short/long blocks: lame --verbose directly indicates the numbers.
LordofStars
I'd agree gabriel. Standard1 and 2 have worse preecho than standard 3.94 or standard 3.90.2 Standard2 seems to be better than standard1 it required me to listen to a and b once and x at least twice while standard1 i only had to listen to x for the most part.
Gabriel
Now, there should be a preset standard3.

This is a modified standard2. So if you prefer this one over standard2, I think that you can discard standard2 from your tests.

This standard3 should be better for pre-echo.
john33
QUOTE (Gabriel @ Jan 20 2003 - 06:11 PM)
Now, there should be a preset standard3.

This is a modified standard2. So if you prefer this one over standard2, I think that you can discard standard2 from your tests.

This standard3 should be better for pre-echo.

New compile just uploaded. wink.gif
LordofStars
Standard3 much better on the hihat sample. I'm not sure what preecho is but it seems to be much more sharp than other encodes and doesn't smear across frames. ABX results 11/13.

I was checking out presetc and the major changes I see are a new psymodel. Can this psymodel be invoked with a switch? for testing exp_nspsytune against nspsytune and gpsycho? If not the possibility for this switch would be nice.
Gabriel
A long time ago, Naoki started some work on a new psymodel. But he take a break, and his work on this psymodel is still preliminary.

You can see a lot of changes in 3.94 from Takehiro (some enhancements and many fixes) that are related to Nspsytune. Those are already used in 3.94. Other changes are in experimental branch, where Takehiro is experimenting. But it is way too early to include those.
amano
hmm. john33. why don't you provide a link?
SK1
You need to learn to look at people's signatures sometimes smile.gif.
And even look if "WWW" is availible for the person who said his new something is availible.
mirror 1, others page.
Anyway, i mentioned the link early at the first page of this topic..so also read the whole thing biggrin.gif..
quellcore
QUOTE (amano @ Jan 21 2003 - 01:05 PM)
hmm. john33. why don't you provide a link?

... because in his signature there are some links. Goto mirror1, scroll down to 'others' (bottom of the page) and you're there where you want to go! wink.gif

EDIT: Damn, too late again!
WEST
QUOTE (SK1 @ Jan 21 2003 - 04:22 AM)
You need to learn to look at people's signatures sometimes smile.gif.
And even look if "WWW" is availible for the person who said his new something is availible.
mirror 1, others page.
Anyway, i mentioned the link early at the first page of this topic..so also read the whole thing biggrin.gif..

You can use alternative ICL 7 (without UPX compression) compile from me smile.gif My Webpage
SK1
Just wondering WEST, do you have something against UPX compression? smile.gif Why not use it?
nyaochi
I abxed on gekkou-intro.flac (intro of "Gekko-so" by Sunny Day Service).
CODE
ABX Results:
Original vs gekko-1s-std.mp3.wav
   10 out of 10, pval < 0.001
Original vs gekko-1s-std2.mp3.wav
   10 out of 10, pval < 0.001
Original vs gekko-1s-std3.mp3.wav
   13 out of 14, pval < 0.001
Original vs gekko-1s-std1.mp3.wav
   11 out of 11, pval < 0.001
Original vs gekko-1s.3.90.2.mp3.wav
   10 out of 10, pval < 0.001


Sadly to say, I can abxed all the encodings laugh.gif But standard2 and standard3 are very close to the original, while standard1, standard (both on 3.90.2 and 3.94alpha) are very easy to tell the differences.
WEST
QUOTE (SK1 @ Jan 21 2003 - 04:48 AM)
Just wondering WEST, do you have something against UPX compression? smile.gif Why not use it?

This problem already was discussed in one of forums, binaries compressed with UPX work with mistakes or do not work at all on old processors, and on some processors AMD
imi
I need to get the latest in_mp3.dll plugging for winamp2. Is there 1 available as lame 3.94 sounds crap compared to 3.90 on my computer!!!
Chun-Yu
QUOTE (SK1 @ Jan 21 2003 - 07:48 AM)
Why not use it?

Here's my $0.02:
It's really slow on older processors that are running anti-virus software (I've tried UPX on a Celeron 333 running NAV and programs took forever to load). Also, the way Windows handles virtual memory with executable files works best with uncompressed files (maybe I'll explain more later when I'm not busy smile.gif).
mithrandir
Perhaps we could try a standard variant that has greater sfb21 masking (default is 3.75dB) and use the saved bits on something else, like maybe using -V1 instead -V2.
Gabriel
btw standard2/standard3 are not even using V2 or V1, and it seems that they are interesting.
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2009 Invision Power Services, Inc.