So thanks to this forum, I've learned all about ABXing, double-blind tests, and how LAME --preset-standard is supposed to be transparent to just about anyone but if I really want to know where my transparency threshold is, I need to do the damn work myself. Fine by me. But in all my reading, I can't seem to find the answer to a simple question which I need to know before I can start testing:

What sounds exactly am I supposed to use to test?

Should I use my own music? If so, which genre--I listen to everything from classical to new age to world to synth-heavy electronics and even some rock.

Or are there pre-selected samples somewhere which have already been encoded at several codecs/bitrates, and which are supposed to be a good test?

If I use my own music, how do I go about picking which songs to encode?

And how long should each sample be? Should I use a whole four-minute song, and do 16 guesses in a single listen? Or should I just use a small section cut out from the song, like 20 seconds, and make one guess per listen on that? How do I choose which 20 seconds are the best to test?

How many different samples would it be good for me to test before I think I've got a decent picture of what's transparent to me?

Should I use headphones or speakers to test (see specs below)? If headphones, should I plug them in directly to my sound card, or is it OK if I plug them into the jack on the volume control on my speakers? (The volume knob also affects that headphone jack, so the signal is amplified through that.)

Finally, let me see if I have the procedure right: I rip some part of a music CD to CD-quality WAV, encode that WAV to whatever codecs and bitrates I want to test, then decode those all back to WAV, and load them up either one-by-one in an ABX program, or just load the encoded ones into ABC/HR. Is that right?

Why would I want to use ABX over ABC/HR, or vice versa?

Here's my setup (not great, I know, but I'm not a huge audiophile anyway):

Chaintech AV-710 with VIA drivers 5.12 (I think), on hi-sample-rate mode
Speakers: Logitech Z-2200 2.1
Headphones: Sennheiser HD-497

For the purposes of this test, it's possible for me to get access to a set of Klipsch ProMedia 2.1 for a short while. Anyone recommend me using that instead? (It'd be a bit of trouble to bring it here, but I could.)

Headphones are generally better suited for ABX than speakers

Cymbal crashes are very easy to abx at low bitrates. Trumpets are easy at higher bitrates on some codecs (they sound burry). Large stereo separation is interesting to test as well...

You can use your own music, and samples should be short so you don't lose too much concentration (fatigue)

Headphones should be plugged directly on your sound card or via a headphone amplifier.

Headphones will make it easier to abx than speakers.
3-4 samples will be enough, if you diversify, otherwise, if you have very various tastes, take 20 samples, approx.

If you have multiple codecs to test, use ABC/HR

Your procedure to prepare samples is right.

3-4 samples is definately not enough as there is an odd chance that 1-2 are killer samples!. Go for at least at least a dozen if not more. There are basicaly two types of music to test: 1) compressed loud 2) quite tonal.

At low bitrates when you clearly hear differences, abx pval of 5% and less will do. With more subtle differences one must aim for near perfect scores (1% or less) for 1 trial or do multiple trials, add them up and get a near perfect score. Do not give answers unless you are sure.

You may find ff123's Artifact Training Page helpful.

Regards,
Madrigal

Thanks for the advice, but no one's really answered my main question, which is:

If I use my own music, how do I decide what songs to use, and what sections of those songs to use, and how long should each sample/section I cut out be?

Actually, if you read correctly, I did.

If you want numbers, pick samples out that are 5-15 seconds, so you can concentrate on a specific part only;

as for what songs to use, you can choose pretty much anything, provided it's from the original medium. Most material has a very diverse frequency range, and features transients, etc.. (as I said, symbal crashes test for transient smearing, trumpets test for warbling, background hisses can test for noise masking, and pretty much any stereo song can test for stereo collapse, but you'll have better chances on older recordings (1950s jazz, Coltrane or Davis);

If you like, you can mention songs you're thinking about on this board, some people ought to know them and can suggest passages to encode.

Peace,
Tristan.

Oh, thanks. When you can said I could use my own music, I wasn't sure if that really meant anything.

Finding the most interesting parts for testing is a matter of experience (and sometimes luck). With a good encoder, there are maybe a few percent of a complete track which may not be transparent. In other words, if you randomly cut short samples from your favorites tracks, it's likely that you're going to miss important issues. On one side testing only problem samples may lead you to wrong and biased conclusions (because these problems should be quite rare and rarity isn't can't make a rule). But on the other side, if you're looking for full and perfect transparency (even with lossy encoders full transparency is possible) testing problem samples would be the first thing to do.


Now if your purpose is only curiosity ("but if I really want to know where my transparency threshold is...") there's no reason to discard samples coming from different musical genres. You should then take a look to the upload forum which contains several problems samples.
But if your purpose is more practical (testing before encoding your own library) you should rather focuse on your own tracks and considering foreign samples for training purpose. Finding the right and good samples is a matter of experience you probably don't have at the moment. Don't worry and test what you like (your favorite songs, their best moments): it makes test more enjoying IMO.


Not too long. I personaly find 3...5 second samples more practical than 30 seconds ones. Keep in mind that several ABX softwares allow you to choose yout own range during the test. Therefore, using 20...40 seconds long samples shouldn't be a problem. Personally I wouldn't use full tracks. Keep in mind that testing audio codecs is only a matter of comparison. And you can't listen two sounds at the same moment. Therefore, your memory plays an very, very important role. Using very shourt samples could only help your memory and yourself.



Testing 100 times the same kind of sound would be a waste of time. 12...20 samples, various enough, should give a good idea of the performance of the tested encoders. The choice of samples is more important than the number. I would in your case spend a complete afternoon to think and finally build a nice sample gallery.
I personally built my own gallery of 150 samples for classical music, and it tooks me a good week, if not more. But now I'm glad to have it, and it's a useful (and exhausting ) tool.


Both if you can. Headphones are much better for testing purpose IMO. They reveal more details.


ABX tool is only here to confirm the validity of your choice. If you don't have that much experience in testing, I would recommend you to not discard this step. Therefore: ABCHR rating + ABX confirmation should be performed.


The high sampling rate mode of your soundcard won't help you at all. Your hardware is clearly good enough to perform listening tests in good conditions. There's no need to bring Klipsch speakers and I'd rather privilege the Seenheiser headphones.