LAME 3.96b regression examples, Post any possible regressions here! Options

[Jebus]

This is NOT a general 3.96b ABX results thread. This is just for people to post (any) samples which regress from 3.90.3. This means that I only want cases where 3.96 is WORSE than 3.90.3.

Please try to keep this on-topic. The idea is that if this thread stays empty, we should push for 3.96 to become official. And if it doesn't stay empty, samples will help the LAME devs with future versions.

Any and all settings are fine... just make sure you post how you encoded it, and link to samples.

This post has been edited by Jebus: Mar 9 2004, 10:47

[Moitah]

Here's a sample which I can ABX (15/16) when encoded with 3.96 --preset standard: Dream Theater - The Dance of Eternity (8 sec, FLAC). Listen to the part where the snair hits 4 times, about 1.5 seconds in. I first noticed this while testing --preset 128, it happened in 3.96 but I didn't notice the same artifact in 3.90.3 --alt-preset 128.

[PVNC]

I tried a 28-second sample from Tool's track (-) Ions. I chose this to test first, as it gets an average bitrate of 384 kbps using Vorbis GT3b1 at q5.

Original sample is in FLAC format.

I encoded mp3 versions using only the commandline --alt-preset 128 in RazorLame. I used LAME 3.90.3 modified and 3.96b1.

FLAC and mp3 versions have ReplayGain information applied.

I found that the 3.96b1 version has some kind of distortion during the lower volume parts of this clip - between the peak volume levels of the buzzing. I was able to use this to ABX 3.96 from 3.90.3 8/8.

(-) Ions

[guruboolez]

Isn't the upload forum a better place for this topic? Maybe should someone create a topic dedicated to this collection, or move this one.

[Dologan]

Also, I would encourage people to also post improvements from the old encoder, since otherwise we might be skewing our preception. Usually, improvements also introduce other minor flaws that nonetheless end up making the thing a good trade-off.

[PVNC]

Isn't the upload forum a better place for this topic? Maybe should someone create a topic dedicated to this collection, or move this one.

I agree. I didn't know that my uploaded sample would end up on the portal. It was rather embarrassing.

[westgroveg]

CODE

ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\ringing-edit--APS-3.90.3.wav
   9 out of 10, pval = 0.011

CODE

ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\ringing-edit--APS-3.96b1.wav
  9 out of 10, pval = 0.011

CODE

ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\ringing-edit--APS-3.96b1.wav
   13 out of 16, pval = 0.011

There is a HF hiss present (0:01-0:02).

This post has been edited by westgroveg: Mar 8 2004, 09:29

Attached File(s)

 ringing_edit.wav ( 1.08MB ) Number of downloads: 275

 

[westgroveg]

CODE

ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\castanets-edit-2-APS-3.96b1.wav
   12 out of 17, pval = 0.072

Guitar sounds watery.

Attached File(s)

 castanets_edit_2.wav ( 172.13K ) Number of downloads: 245

 

[westgroveg]

Moitah's The Dance of Eternity sample uploaded at HA,

Attached File(s)

 doesnair.flac ( 872.58K ) Number of downloads: 232

 

[Moitah]

Play - Is It Love (4 sec, FLAC) (try this link first, to save HA bandwidth)

--preset standard: This takes a lot of concentration for me to ABX, I got 22/32, 23/32, and 14/16 which adds up to 59/80 (p<0.001). The first time the high note is played on the guitar (1.2 secs in) sounds smoother in the original.

This post has been edited by Moitah: Mar 8 2004, 19:40

Attached File(s)

 isitloveintro.flac ( 447.13K ) Number of downloads: 218

 

[amano]

CODE

ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\castanets-edit-2-APS-3.96b1.wav
   12 out of 17, pval = 0.072

Guitar sounds watery.

As this is a regression thread, it dosn't make sense to just post just non-transparent samples. You have to compare them to the ABX results of 3.90.3.

And please don't post ABX results that aren't statistically valid. 12 out of 17 is certainly not a statistically convincing result. Otherwise this thread will be spammed with useless information.

This post has been edited by amano: Mar 8 2004, 21:37

[amano]

Play - Is It Love (4 sec, FLAC) (try this link first, to save HA bandwidth)

--preset standard: This takes a lot of concentration for me to ABX, I got 22/32, 23/32, and 14/16 which adds up to 59/80 (p<0.001).  The first time the high note is played on the guitar (1.2 secs in) sounds smoother in the original.

I don't think that is valid to add up session results (maybe some other more in depth in the ABX science may tell us for sure).

And please compare your results with 3.90.3. To be fair chose the same session procedure. and try to ABX 3.90.3 against 3.96.

EDIT: I think everyone should act according to tigre's proposals in the LAME poll thread:

# Posting results in the thread requires:

    * Upload or link to sample
    * ABX results Original<->3.90.3, Original<->3.96, 3.90.3<->3.96, with detailed description of the difference(s)
    * Report about software/hardware used: Soundcard (resampling?), Player/ABXtool, DSPs (shouldn't be allowed, besides resampling to 48kHz and volume reduction/replaygain to prevent clipping <- both a 'must'), Amplifier, Speakers/Headphones

# Results must be confirmed by someone else before they are included in 'official' statistic, p-values must be < 0.05 for at least 2 people.

This post has been edited by amano: Mar 8 2004, 21:35

[indybrett]

Since we have a regression thread, maybe we should also have a progression thread.

[westgroveg]

As this is a regression thread, it dosn't make sense to just post just non-transparent samples. You have to compare them to the ABX results of 3.90.3.

And please don't post ABX results that aren't statistically valid. 12 out of 17 is certainly not a statistically convincing result. Otherwise this thread will be spammed with useless information.

It was the only test I did with 3.96, I also did 1 ABX test with 3.90.3 got 5/5 thus my post of the sample.

Anyway your not a moderator, you didn't start the thread or even post a sample so I don't really see what your interest is, you just must like telling everyone what to do .

(HA Wiki)

A difference is concluded to be heard when 13 correct identifications out of 16 trials is achieved

This is what we are expected to score for a valid problem sample, people, not what amano wants us to do.

This post has been edited by westgroveg: Mar 9 2004, 08:13

[Jebus]

Westgroveg, i appreciate the results you are posting but PLEASE, i specifically requested that ONLY regressions from 3.90.3 be posted here, NOT general ABX results.

EDIT: Corrected Westgroveg's name

This post has been edited by Jebus: Mar 9 2004, 09:45

[westgroveg]

Westgroveq, i appreciate the results you are posting but PLEASE, i specifically requested that ONLY regressions from 3.90.3 be posted here, NOT general ABX results.

My first sample as you can see I was able to ABX with both 3.90.3 & 3.96

My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?

& my nck is westgroveg.

[freakngoat]

My first sample as you can see I was able to ABX with both 3.90.3 & 3.96

My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?

Sorry, don't mean to butt in, but they mean your ABX results aren't worse with 3.96 than they were with 3.90.3; they appear to be equal or even better. Regression testing means running past tests over again on a new version of software to make sure that nothing broke or got worse (regressed) with the newer version.

[Jebus]

Westgroveq, i appreciate the results you are posting but PLEASE, i specifically requested that ONLY regressions from 3.90.3 be posted here, NOT general ABX results.

My first sample as you can see I was able to ABX with both 3.90.3 & 3.96

My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?

& my nck is westgroveg.

As Freakngoat just posted, the idea is that we ONLY post cases where 3.96 is WORSE than 3.90.3. You just posted cases where they both mess up - which isn't the point of this thread.

[westgroveg]

My first sample as you can see I was able to ABX with both 3.90.3 & 3.96

My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?

Sorry, don't mean to butt in, but they mean your ABX results aren't worse with 3.96 than they were with 3.90.3; they appear to be equal or even better. Regression testing means running past tests over again on a new version of software to make sure that nothing broke or got worse (regressed) with the newer version.

Oh, sorry I misunderstood, I thought the thread asked, samples which are problematic for both 3.90.3 & 3.96, sorry Jebus.

[Jebus]

np

I added a clarification sentence to the first post

[westgroveg]

I just had a go at PVNC's (-) Ions sample (I used the contained mp3's for ABX)

CODE

ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\3.96b1 (-) Ions.wav
   16 out of 16, pval < 0.001
C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\3.96b1 (-) Ions.wav vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\3.90.3 (-) Ions.wav
   10 out of 10, pval < 0.001

Both have a huge distortion (0:02-->0:04), 3.90.3 sounds slightly better.

This post has been edited by westgroveg: Mar 9 2004, 11:54

[westgroveg]

Here's a sample which I can ABX (15/16) when encoded with 3.96 --preset standard: Dream Theater - The Dance of Eternity (8 sec, FLAC).  Listen to the part where the snair hits 4 times, about 1.5 seconds in.  I first noticed this while testing --preset 128, it happened in 3.96 but I didn't notice the same artifact in 3.90.3 --alt-preset 128.

I find these rock samples hard to ABX, can you tell us what type of artifact to listen for?

[Moitah]

I find these rock samples hard to ABX, can you tell us what type of artifact to listen for?

I'm not sure what it's called, pre-echo maybe. If you start with 128 it should be easier.

I need to do more testing with my second sample, it might be happening in 3.90.3 as well.

[2Bdecided]

I don't think that is valid to add up session results (maybe some other more in depth in the ABX science may tell us for sure).

You should always add all ABX result together for the same sample. It is statistically valid.

What you should never do is do more than one set of ABX tests, but only report some of them - you always need to report all the ABX testing you did with a sample. Selective ABX results are not valid.


Also, watching the result, and stopping when it's "statistically valid" isn't right either, but I haven't seen an explanation of what the correct statistics are in this case. The advice has always been to decide how many trials you will take before you start. But if you watch your result, then the "standard" statistics (which are found in most ABX programs) are wrong.

Cheers,
David.

[Moitah]

My second sample isn't a regression, I ABXed 16/20 and 12/12 (total 28/32, pval < 0.001) for 3.90.3 --alt-preset standard (I think I was able to ABX 3.90.3 better because I knew what to listen for, not because it sounded worse than 3.96).