Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Multi-Format Listening Test (Read 3816 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Multi-Format Listening Test

Synopsis:
I'm planning to conduct a listening test with ABC/HR 0.53b for Java very soon amongst some few individuals and myself, and I want to make sure I have everything in order before I do anything.
Here are the formats that are going to be tested:
•   MP3: LAME 3.98.2 through Audacity 1.3.11 Beta
•   Ogg Vorbis: aoTuV beta 5.7  though OggdropXPd 1.9.0
•   AAC: Nero 1.5.1  through foobar2000 1.0 , Quicktime/Apple  through iTunes 9.0.3.15
•   Windows Media Audio: Standard 9 & Pro 10 Versions through Windows Media Player 11 ,
•   Musepack: Musepack SV8  through foobar2000 1.0

For each codec, I'm going to use at least 3 samples per adjustable bit rate setting, so there will be at least 15 tests per each codec. I know that the minimum amount of tests aren't exactly enough for an acceptable confidence interval, since at the moment, I might only have a couple individuals on hand to participate. The volunteers are willing to donate a couple of hours in to testing, and I will assign each volunteer to a certain format.

Details

Here's some more detailed information about the codecs:

MP3
  • Encoder: LAME through Audacity 1.3.11
  • VBR; Variable Speed: Standard
  • 3 Samples per Bit Rate Setting (15 Tests)
  • Channel Mode: Stereo
  • Decoder: ???

  • V8 (65 – 105 kbps) [Low]
  • V6 (95 – 135 kbps)
  • V4 (145 – 185 kbps)
  • V2 (170 – 210 kbps)
  • V0(220 – 260 kbps) [Highest]


Ogg Vorbis
  • Encoder: aoTuV b5.7 through OggdropXPd
  • Standard Quality Mode (VBR)
  • 3 Samples per Bit Rate Setting (15 Tests)
  • Decoder: Oggdec v1.9.6

  • q 2.0 ( ~96 kbps) [Low]
  • q 4.0 (~128 kbps)
  • q 6.0 (~192 kbps)
  • q 8.0 (~256 kbps)
  • q 10.0 (~499.8 kbps) [Highest]


AAC
  • Encoder #1: Nero 1.51 through foobar2000
  • VBR
  • 3 Samples per Bit Rate Setting (15 Tests)
   
  • Q .20 ( ~48 kbps) [Low]
  • Q .40 (~125 kbps)
  • Q .60 (~225 kbps)
  • Q .80 (~325 kbps)
  • Q 1.0 (~400 kbps) [Highest]

  • Encoder #2:Quicktime/ iTunes (Apple) AAC Encoder through iTunes
  • VBR
  • 3 Samples per Bit Rate Setting (15 Tests)
  • Decoder: ???

  • ~80 kbps
  • ~128 kbps
  • ~192 kbps
  • ~256 kbps
  • ~320 kbps


Windows Media Audio (Encoder: Windows Media Player 11)
  • VBR (WMA Standard); CBR/??? (WMA Pro)
  • 3 Samples per Bit Rate Setting (15 Tests)
  • Decoder: ???

  • Format #1: WMA Standard
  • 40 – 75 kbps
  • 50 – 95 kbps
  • 85 – 148 kbps
  • 136 – 215 kbps
  • 240 – 355 kbps

  • Format #2: WMA Professional
  • 64 kbps
  • 96 kbps
  • 128 kbps
  • 160 kbps
  • 192 kbps


Musepack (SV8)
  • Encoder: Musepack Encoder (MPC) through foobar2000
  • Decoder: Musepack Decoder
  • VBR
  • 3 Samples per Bit Rate Setting (15 Tests)

  • Q3 (~90 kbps)
  • Q4 (~128 kbps)
  • Q6 (~205 kbps)
  • Q8 (~275 kbps)
  • Q10 (~350 kbps)


So now that I've decided on what codecs & bitrates to test, I need to know what kind of samples I need for the testing, and what sort of decoders I need. I need the decoders to decode each compressed file to load them onto ABC/HR 0.53b for Java.

Questions/Things Needed
  • What Decoder should I use for all MP3, WMA,  AAC formats?
  • Is there a decoder that can work for both for both WMA Standard and Pro?
  • What sort of samples would be good for testing?
  • How can I encode the Nero AAC format so that it comes out as AAC-LC?
  • How many samples for each adjustible bit rate setting should I really use that won't fatigue any of the testees?


Please, feel free criticize any part of the of the plans for the testing. Any is welcome


Multi-Format Listening Test

Reply #1 –
Quote from: ezlez on
For each codec, I'm going to use at least 3 samples per adjustable bit rate setting, so there will be at least 15 tests per each codec. I know that the minimum amount of tests aren't exactly enough for an acceptable confidence interval, since at the moment, I might only have a couple individuals on hand to participate. The volunteers are willing to donate a couple of hours in to testing, and I will assign each volunteer to a certain format.

What do you actually want to test? If you assign each tester his own format, you're only going to find that V2 is better than V6 which is not a very useful result as such.
"We cannot win against obsession. They care, we don't. They win."

Multi-Format Listening Test

Reply #2 –
And why MP3 through Audacity, Ogg Vorbis through OggDropXP, AAC through foobar2000 ?
Why not all through foobar2000?
 

Multi-Format Listening Test

Reply #3 –
Answers:
1. foobar2000 can decode all those formats.
3. & 5. Follow the discussions:
  http://www.hydrogenaudio.org/forums/index....showtopic=77584
  http://www.hydrogenaudio.org/forums/index....showtopic=77272
as well as discussions for older public tests.
4. Use NeroAACEncoder -? to get complete list of supported options. -lc is to force LC profile. You definitely shouldn't use -lc for q0.2 and for all other formats it is anyhow already the default.