There has been endless discussion recently in the UK about the quality of our digital radio services (DAB). They typically use MPEG layer II (mp2) at 128kbps JS. The quality is interesting!

A campaign to get something done about this has been partially sucessful - many Radio stations are now available via the digital TV platforms at 192kbps mp2, which is an improvement.


What I, and others, would like to do is to run a listening test, comparing various bitrates, and (possibly) the two codecs in use worldwide in digital radio (mp2 and AAC). The results may or may not be a forgone conclusion, but the reason for doing this is to get some independent, unbiased information about what people actually hear, and, specifically, whether 128kbps mp2 is good enough.


This is a plea for help. I have followed listening tests here and elsewhere in the past, and carried them out myself. However, I need suggestions:

1. which tool is best for this test?
2. would anyone be willing to host this test?
3. what samples would you suggest?
4. would it be appropriate to use "processed" samples?
(in radio, you don't get the digital bits from the CD going to the encoder; you get A>D, mixing desk, dynamic range compression, D>A encoder at least)
5. do these sound sensible: mp2 128 160 192 256, AAC 96 128
6. which encoders should I use? Realtime hardware codecs are used in radio broadcast, but obviously software would be easier

I (and many other people!) would like to be able to take the results of this test to the regulator in the UK and say "look, 128kbps mp2 isn't good enough", and for them not to be able to pick any obvious holes in the test methodology.

Any help and suggestions greatly appreciated!

Cheers,
David.

P.S. for endless discussion over DAB audio quality, see alt.radio.digital
http://groups.google.com/groups?q=alt.radi...&oe=UTF-8&hl=en or your newsreader

It would be nice doing a test like that. Here, in Italy, are sold some car stereo with DAB, but there are very few stations here.

I have another question to add: I read somewhere that DAB would use AAC+ giving very high quality at low bitrate, so why I read now that DAB is in MP2? I thought it was only for satellite radio station, but normal radio station would use AAC (or AAC+ when ready).

DRM - Digital Radio Mondiale = digital broadcasts below 30MHz (that's AM to you and me)
This will use AAC + SBR at bitrates around 15-30kbps. It's a world wide standard.

DAB - Digital Audio Broadcasting = digital broadcast around 200 and 1500 MHz (that's above FM)
This uses mp2 at bitrates from 32-384kbps. It's a european standard, accepted most places except North America.

HD Radio - IBOC and PAC = In Band On Channel with Peceptual Audio Coding (that's in the FM band)
This is the North American standard. I would swear that this uses AAC, but there's no mention of it on their website.
It has been suggested that it uses 96kbps AAC during transition (it's in the same space as conventional FM, so piggy-backs on the existing broadcast)
and then 128kbps AAC + other services when the analogue FM station stops broadcasting.


In the UK, with a fixed home dish or aerial, you can use DSat (digital satelite) or DTT (digital terrestrial) TV to receive radio broadcasts from the BBC and many others. The quality is higher than on the actual digital radio (=DAB) broadcasts, due to the higher bandwidth (hence bitrates) available on the TV platforms. Strange but true!

Cheers,
David.
P.S. There are two satelite radio systems too, intended for mobile reception. This is in the USA, and I have no information on these.

Imo the biggest problem would still be the credibility. Who says that certain software encoders act anywhere similar to certain hardware encoders? Or at least that's what can be said as a strong counter argument to this test.

Anyway, what comes to test samples, ff123 used nice variety of different music in his latest 64kbps test: http://ff123.net/64test/results.html

Imo the test, if started, should be done in similar style, using the ABC/HR Audio Comparison Tool.

For Layer II encoders, I would use one from QDesign.

Yes, that's my worry too. In practice, some software encoders are better than the (now quite old) hardware encoders in use in the UK DAB industry, especially at the BBC. However, it would need another listening test to prove this!

I was also considering using actual DAB recordings as one section of the test. Then there's no argument. It's possible to get the same track broadcast on DAB (mp2 128), DTT or DSat (mp2 192), FM, and CD! Of course the CD hasn't been through all the processing of the radio station, but the others would make an interesting comparison.



I had this in mind - I think "typical" rather than "stressful" would be fair - though I might put one stressful clip in there.




Yes - totally agree.


Cheers,
David.

To show differences between different encoder generations, you could do a quick comparison between ISO encoder (plain old musicam) and QDesign or TooLame.

Thanks Gabriel - I'll try that - I have a purchased version of Qdesign mp2 somewhere - it's in one of the unpacked boxes in our new house! I understand tooLame is making good progress these days.

Where can I find a binary of the ISO mp2 code?


btw, I've been in touch with ff123, and he's offered to help, which is great.


Questions 4 and 5 are still open, as are the others if there are any more comments.

I'll be back when I've made more progress.

Cheers,
David.

EDIT: typo

Get SoloH. It's a compile of the infamous dist10 sources with an usable frontend. It's slow, buggy and outputs terrible quality.

ftp://ftp.redcom.ru/pub/support/windows/u...egEnc_v007a.zip

Here's the old SoloH page:
http://www.euronet.nl/~soloh/mpegEnc/

(Binaries aren't available there anymore)

Sounds interesting. Some possible concerns:

Probably the best possible outcome would be if there are clear faults (from a group perspective) in the low bitrates which go away somewhere around (say, for the sake of argument) 192 kbit/s. Then one could definitely say that 192 kbit/s is good enough, at least from a quality, if not from an economic standpoint.

However, if that doesn't happen, and the listening group is still finding fault at high bitrates, then interpreting the test results become more problematic. Or perhaps one or two samples out of a dozen shows that there's a problem at 192. How does one decide what's good enough? Perhaps the testers, being self-selected and highly focused on listening for quality differences, are "too sensitive" and don't represent real-world listeners. A possible counter-claim: "Yeah, there are small differences even at high bitrates if you play a short sample over and over and really listen for faults, but for most people most of the time it's good enough."

From the standpoint of gathering information on differences in codec quality, I think the test should be fine. It's the subsequent interpretation which I have some concerns with.

ff123

David:

I'm still terrified by the quality of the DAB sample you once posted ("Eternal Flame"), so I think this test is a great idea. But where do you want to get the number of test listeners one would need to convince the regulators...? The 100-200 people regularly participating in HA wouldn't do, surely. (I hope I'm wrong!)

I'd also keep the number of test samples to a minimum - otherwise, you'll scare modem users like me away, and if you make this real, every listener counts.

I think the idea of using samples recorded directly off DAB isn't a good one - because of all the processing used to make the sound more "punchy", unexperienced listeners might rate the station's output higher than the samples encoded from unprocessed CD audio, despite it being more artifact-ridden.

Just my 0.02€.

ff123,

You've hit the nail on the head - it's easy to do a "good" test that stands up for what it is - but avoiding later criticism is difficult.

I'm not going to pre-judge the result - and I'm not even going to be surprised if it shows almost the opposite of what I hope! It is probably true that some people here can find faults at most bitrates with an mp2 codec. However, many won't - there are some "normal" people here too, and the test won't just be open to HA members. It would be very interesting to find how many people think each bitrate is OK. For that matter, what is OK? Acceptable? Transparent? Not annoying? On the 1-5 scale it's anything above 4, I think. There are EBU definitions that I'll use, though, interestingly, only AAC meets them for all audio samples!

As for the argument that the testers simply "listened too carefully" - well, they probably will. Sometimes I listen too carefully to radio as well! I can't counter it, but I had thought about it. Any ideas?


Volcano,

I'll get as many people as possible - if as many as 100 people take part, I'll be amazed. "Real" listening tests are often done with less than 30 people, though they are pre-screened. I'm not too worried, so long as, say, more than 30 take part.

I will certainly keep modem users in mind - I am one myself!

I'd thought about the processing problem if I get samples from DAB directly. There's one argument that says "if you don't use any samples from DAB, then you're assessing an mp2 codec that could be very different from the ones used in DAB, so the test is pointless". There's your counter argument. In fact, the processing on some UK DAB stations is very mild, and once the levels are matched, it shouldn't be a problem. I guess I won't know until I've tried it.


Let's say I find an audio clip which I can get on CD, and can also record from FM, DAB, and DTT.
Possible options include:
1. Original CD
2. mp2 encoded directly from this CD at 112 128 160 192 256
3. AAC encoded directly from this CD at 96 128 192
4. "Processed" CD (A/D > mixer desk > D/A > some dynamics processing)
5. mp2 encoded from "Processed" CD at 112 128 160 192 256
6. AAC encoded from "Processed" CD at 96 128 192
7. "Transcoded" CD (via MD of mp2 256, because many many radio stations use these for playout!)
8. mp2 encoded from "Transcoded" CD at 112 128 160 192 256
9. AAC encoded from "Transcoded" CD at 96 128 192
10. recording from FM
11. recording from DAB
12. recording from DTT


Obviously 1,2,3 is one test, 4,5,6 is a separate test, 7,8,9 is yet another test, and 10,11,12 may fit in with any, or one, or none of the above!

Then there's the question of different AAC and mp2 codecs (I've found my Qdesign one btw, so now have at least 3 mp2 encoders)

Then there's the question of source material. If it's to include10-12 it's restricted to what I can find on the radio (The album chart show on BBC Radio 2 would be an easy start - the top 40 on Radio 1 is highly processed, making comparisons with CD difficult).

If it's not to include off-air recordings, the source material could be anything, but I would probably use:
1. piece of pop music
2. piece of classical music
3. extract of speech

For 3, I'll buy a recent BBC comedy show on CD, and use that. For 1, wayitis.wav is pretty middle-of-the-road, or maybe Dido, or whatever. I specifically don't want to use "codec killers".


I'm thinking allowed here, and I really don't know where to go with that 12 point list (and various codecs!) to get (ideally) 4-8 samples per source.


I want a fair test that represents what's heard on DAB, but I don't want any bias or undefensible method to be involved. This is harder than I expected, and I haven't even started yet!


Cheers,
David.

Some questions and comments:

1. How many participants do you estimate there will be? The more, the better. Have you considered how you could drum up as much participation as possible?

2. How many different formats were you thinking of? As a participant, I probably wouldn't want to compare more than 5 or 6 formats at any given time. However, it might be possible to test many formats using a design in which different people compare different formats. I'd have to research how to do it properly, but almost certainly you need lots of people to make it work. And probably need to have some sort of automated software on the server side.

3. It's hard to say one format is better than another without listening to lots of different samples. A dozen is nice, 20 would be even better. Of course not everybody would want to listen to every sample (this worked out ok in the 64 kbit/s test).

4. If this turns out to be a large test, then automation is almost certainly necessary. When I did the 64 kbit/s test, I had people send me email. If I had to do it over, there's no way I would go that route again. Some sort of web form in which people paste their results would probably be the best way to go.

ff123

Quite OT: hey, do you think that OpenDRM (based on Vorbis) could be useful ?

Hi David, maybe you should have posted this in the AAC boards, too... These tests have already been done in a very thorough manner by the MPEG Audio Subgroup back in 1996 and 1998, and the full reports can be downloaded from their website. In my opinion they can also serve as a "role model" for other ambitious listening tests, because as usual the MPEG committee tried to cover every aspect of a valid comparison that could serve as a basis for important future decisions for radio broadcasters like the BBC and others.

The beginning was a comparison of MPEG-2 NBC (= AAC) with other codecs (also MPEG-2 Layer II) for multichannel purposes, so the bitrates in questions were quite high. The results have been published in a PDF file named "w1419.pdf" (72 pages). A shorter summary is available with "w1420.pdf" (14 pages).

The stereo setup of AAC at 96 kbps was compared to other codecs (again with MPEG-2 Layer II at 192 kbps) in "w2006.pdf". This is the official report that gets mentioned whenever the sound quality of AAC is described in comparison to MPEG-1/2 Layer III and Layer II. I don't remember if all these PDF files are also available in a HTML version from the MPEG Audio Subgroup, but you could have a look around for yourself maybe.

Some of these informations (links, excerpts, summaries) are also available from the Wiki of AudioCoding.com, e.g. the AAC page:
http://www.audiocoding.com/wiki/index.php?page=AAC

Hi Hans,

You mean you don't read all the boards on the forum? Yes, yes, I know, things fly off the front page so quickly that it's easy to miss interesting threads these days.


I have all the official tests (I think - some were also published in the AES). I often recommend w2006 in the same way as you have just done - as an excellent example of how to run a listening test. (IIRC It's on Gabriel's mp3-tech.org as a word .doc)

Given the existance of all these tests, it's quite amazing that UK broadcasters are using 128kbps JS mp2 for almost all their digital radio broadcasts. Most wouldn't dream of using any higher bitrates - the commercial broadcasters are only doing what I would expect (due to the lack of regulation in this area) but I would have expected better from the BBC.

There's a saying about "knowing the cost of everything, and the value of nothing". They know the cost of broadcasting in terms of licenses and bandwidth and uplinks, but know nothing about the value of their own radio programmes, the listeners to which are being subjected to the most appauling sound quality!


There is one way in which the official tests fail to match reality: they all used "clean" audio. No hiss, no distortion, no dynamic range compression. And certainly no transcoding. So the results from those tests, though they do suggest that 128kbps mp2 doesn't sound very good, don't actually show how terrible it sounds in real radio broadcasts! Also, they used "golden ears" - and this fact alone is enough to make the UK radio industry completely ignore them. To quote someone from GWR (a large UK radio group) "If you want CD quality, go and buy the CD! As for the audiophiles - stuff them!" (paraphrased from http://www.radioacademy.org/techcon/4audio...ualityonmpg.doc discussed in http://groups.google.com/groups?threadm=b3...ol.co.uk&rnum=1 )


I don't expect this test to help that much, but it will be useful to have some new data, gathered using a wide range of listeners, with realistic, representative test material. And beside all this, the results will be very interesting. If I can include the right things in the test, it will answer:

1. How big is the effect of "unclean" sources (transcoding/D>A>D/etc)
2. What bitrate is transparent for people in the test (i.e. not just golden ears) using undemanding material

It may even show that, under test conditions, despite intentionally degrading the source material, the audio extracts STILL don't sound as bad as UK DAB broadcasts - which would beg the question "what on earth are the broadcasters doing?!". I suspect the answer is encoding/decoding/encoding/decoding... so many times that there's barely any signal left.

Cheers,
David.

As many as possible. It sounded like question which you'd thought of an answer to? My answer would be publicise it here, and any other forum I could think of. What's yours?



As you can see, it's not just formats, it's combinations of formats plus other factors. It's encouraging that this might be possible, while keeping the number of comparisons low per test. But as I couldn't guarrentee huge numbers of people, it may be best to keep it simple. A "small test that gets a result which leads to another test" is beter than a "large test that's never finished".



You didn't have 20 samples in that test though! Again, I wonder how to balance lots of samples, each tested by a few people, or a few samples, each tested by lots of people. Maybe a few to start with. Or would that be unhelpful, because it would be difficult to get people back for a second round of "the same thing" with different samples?



This is completely beyond me. Unless I could get some help with this, it'll be emails, PMs, and a big tables of result in Excel. Which doesn't appeal to me either!


Cheers,
David.

That was my thought too. Slashdot helped out a lot for the 64 kbit/s test. Possibly you could write an article for kuro5hin.org first, and then have somebody slashdot it a little bit later to keep up momentum.



Maybe a dozen might be ok. I don't know if you'd get enough people to fill out enough data for 20 samples. I think starting with the large number would be better than going with a few at first, for exactly the reason you mention.



In this area, I probably can help. I can write a script to capture data to my server, which I can then zip up and send to you. There is a chance that people could "stuff the ballot," which is easier to catch if one uses email, but I can see no motivation for doing something like that.

I can also write programs or scripts to process the data, so you don't have to enter into Excel by hand. I hacked up several very ugly programs the last time -- I'm not going to make something like those publicly available. However, I could modify those and pass them along to you.

ff123

I've bookmarked the "Active Postings" page of HA, and it got quite "foggy" recently.



They probably are of the "never change a running system" kind...



OK, but I'm not sure if it will reveal large differences to the MPEG tests that already targeted that bitrate range. So maybe you will invest a lot of time and effort with no significantly differing results that you could use to convince those DAB people in the U.K., if they won't believe the official test. But I can also understand the reasons you mention, e.g.:



So all the best for your "MP2@128kbps stinks!" comparison...

Before I forget: if you want to use PsyTEL AACEnc for the 96 kbps AAC sample, don't forget to resample it to 32 kHz. This is also true for the -internet (~100 kbps) and -radio (~80 kbps) presets, if you decide to try them, too. And QuickTime 6 Pro can also deliver a good quality at this bitrate, see this thread on the Audiocoding.com forum:

http://www.audiocoding.com/phorum/read.php...1&i=2219&t=2219

By the way, maybe something like the Wiki could come in handy for gathering the user results? Since everyone can paste them into a Wiki page easily, it might serve as a quick platform. Of course it relies on the integrity of the people that participate in your test.

And last but not least: the recommended number of testers is at least 40 and samples at least 8 in the EBU listening tests (see e.g. http://www.audiocoding.com/wiki/index.php?...istinguishable)

Thank you for the advice ff123 and Hans. All taken.

My thoughts are taking shape now. In this test, I want to compare factors which may improve DAB audio quality. These factors are:
a) bitrate
b) processing
c) transcoding
d) choice of encoder

It may turn out that the bitrate is the least of the problems, or it may turn out that, given a sufficiently high bitrate, some of the others are less of a problem.



Now, a question. If it turns out that, say, I end up with 8 bitrate/encoder combinations, I think that's too much. 10 would be even worse, but let's not go there.

You suggested ff123 that maybe they could be split across different tests, so no listener compared them all in one test, but overall, they were all tested against each other. Maybe that would work, but how? For example, it seems to me that people adapt the quality scale to fit the codecs they are presented. So, if in test A they happened to get all the good quality codecss, and in test B the happened to get all the poor quality codecs, then in test A they may mark lower than they should, and in test B they may mark higher than they should, simply due to the lack of high and low achor points in the tests.

However, if you ensure that each test includes a low and high anchor, then you're only left with 2-4 real codecs per test. If you choose 4 from 8 at random to present to each listener, then with enough listeners you'll get enough answers for the overall result to be meaningful. But doesn't it require many more listeners than if all codecs (e.g. 6) were auditioned by each listener? And do the statistics exist to check for significance in such a test?



I must include:

CD, resampled to 48kHz, encoded to ...

mp2 @ 256, 192, 160, 128, 112 kbps.
(There are rumours of plans to lower the minimum DAB bitrate to 112kbps, so it's worth including).
(There are several european stations at 256kbps, so it's worth including).

AAC @ 128, 96 kbps.
(No relevance here in Europe, but very relevant to US digital radio - and as many potential testers will probably be from the states, I think I need to include these to get broad interest)

I would also like to include a digital simulation of a good FM channel.

I also want to include at least one transcoded sample (probably mp2 128 twice or 192 twice; once at 44.1, once at 48) because many UK stations sound like they are doing this!


Finaly, for some samples, I want to do two separate versions:
1. CD, resampled to 48k, encoded to <whatever>
2. CD, dynamic range compression, resampled to 48kHz, encoded to <whatever>
These would be in separate tests - I wouldn't ask a listener to compare the two in a single test. For the "transcode" codec(S), one encode would be at 44.1kHZ before the DRC, the other would be at 48kHz after the DRC. Just like radio stations appear to do at present!


That's it. No actual DAB or FM recordings.


I make that 9 codecs (5 mp2, 2 AAC, 1 FM, 1 transcode). That's too many! Suggestions?

Cheers,
David.

(and I haven't even thought about comparing different mp2 and aac encoders - gulp!)

Well, http://www.soundexpert.info works this way (interesting preliminary results, by the way... )



Right, that's why the EBU chose to expand their standard listening tests to the new MUSHRA method (MUlti Stimulus test with Hidden Reference and Anchors), so that the listeners would always have a low quality sample and the original within each test sequence to "rest their ears" and/or normalize their hearing ability. See http://www.ebu.ch/trev_283-kozamernik.pdf (in case you don't know it already).



Yes, see Chapter 4.9 of this Technical Review.



One or two less MP2 samples, maybe only 96 kbps AAC.

You may wish to perform a small preliminary test so that you have an idea of what's going to happen, even if that means just you sitting with headphones going through some of the permutations. A small group of trusted listeners might work too.



Let me look through my book tonight to see how it works. I don't remember seing a statistical method for the exact way you propose (always include a high and low anchor). But now that you mention it, that does sound like the way to do it. The statistical method that EBU proposes is basically a mean and 95% confidence intervals. It doesn't adjust for the fact that many different samples might be under test, which would tend to yield higher type I errors (false positives). However, for what you're doing, it might be good enough.

ff123

Edit: BTW, the EBU paper recommends sticking with one type of transducer for all participants, and keeping volume within +/- 4 dB of a reference level. It is my opinion that not following these recommendations will tend to reduce the sensitivity of the test, but at the same time, it may make it more representative of real-world listeners.

Just an Idea..

meybe you could contact a broadcasting station and ask them what kind of dymamics compression they use for their broadcasts and apply similiar filters to the original disc samples before encoding so the difference between your encodes and the FM and the Digital radio recording is less noticeable.

I'd remove 112 and 160kbps for Layer II in order to reduce the number of samples.

That's an excellent idea.

Unfortunately, they're using expensive Orban Optimod units, which I don't have access to. (It's a shame I've left uni - the radio station had one!). The other problem is that the "sound" of the station is created by the many settings which can be adjusted on an Optimod - most stations are very secretive about such things, and wouldn't let anyone know their settings.

However, if anyone is willing to send some samples through an optimod for me, I would happily take them up on their offer! It would be better than the mess I'm creating in Cool Edit Pro at the moment - though it's an authentic sounding "Independent Local Radio station with too much compression" mess!

Cheers,
David.

This is the most sensible answer, looking at it purely as a listening test. The problem is that these are the two most likely bitrates for DAB stations to change to! So I have to include them in the test.

D.

I'm sat here with my headphones on! It's quite interesting. It's quite shocking too - you have to transcode and add DRC to make it sound as bad as a typical DAB station - mp2 128kbps on its own doesn't sound that bad. Then again, I'm using the Qdesign encoder, but I have it on good authority that the BBC are using hardware encoders from 1996, while the commercial broadcasters are using dist10!!!



That's one of the distinctive points of this test - "real world" conditions and samples. Not golden ears with an excellent hi-fi system listening to harpsichord arpeggios! (OK, you and I know that the official listening tests aren't that obscure, but that's how people try to play them when they want to dismiss them). It will be much less sensative (or at least, some of the test subjects will, due to themselves of their equipment, be much less sensative), but that's real life!


How about this: Let's say each listening test is split in two - maybe everyone has to do both halves; maybe each listener only has to do one half.

In each half, I include the low and high anchors. If the anchors can be two of the test codecs, then the high anchor can be mp2 256 (not particularly high by the standards here, but it should be OK?), the low anchor can be mp2 112 (128 transcode may be even worse - is this a problem? - maybe a preliminary test to decide which is the low anchor!). This leaves:
mp2: 192, 160, 128, transcode, different encoder
AAC: 96 128 (problem: AAC 128 may be better than mp2 256 for some/all?)
FM simulation

8 to do (excluding anchors) - that's 4 in each half - or 6 in each half including anchors. That's a reasonable number. The split (which codec goes in which half) should be random.


If I remove AAC 128, then I can (hopefully) assume mp2 256 is the high anchor (though for some, maybe FM will be!) - the only problem then is that people will hear the anchor codecs twice, but the other codecs once - which may be a problem since the anchors are part of the test. Do you think this will work, or wreck it? Is there a better way? Should I make everyone do the two halves?


Maybe an alternative is to have two reference samples - not part of the test, but available for replay, basically saying "This is about as good as you can get - it's 4.9 or 5 depending on whether you can hear any problems" and "This is about as bad as you can get - it's 1.0 (or 2.0 or whatever it really is!)". Would that work? I was going to send people to your artefact examples page before the test for some training anyway!


Cheers,
David.

Hans,

Yes the soundexpert.info test seems to be working quite well. Is it the ogg results that you don't like? I think the problems of intorducing different codecs at different times will take a while to average out!



It's always good to read it again!

ff123's "ABC/HR audio comparison tool" isn't quite MUSHRA, but I think it's just what's needed. BS1116 is great when the audible problems are just detectable. MUSHRA is great when the audible problems are obvious, and the problem is ranking them. Various papers suggest that "either" can be used in the region where the two tests overlap, but in truth "neither" is very useful in this case! So, when you know that you'll probably have some "can I really hear a difference?" samples, and some "yuk - that's horrible!" samples, ff123s hybrid tool is perfect. It's especially useful when using untrained listeners, because it gives the best of both (or even 3!) worlds, and whatever level they're at, it should be able to get reliable answers out of them! (I know you know this already, but I'm thinking out loud here, and letting you know that your advice was useful, and made me think!)



Maybe - they won't be using 128kbps AAC for a while in broadcasting, and it may prove to be a high anchor, so maybe I'll scrap it.


I'm going to try dropping all the possible codecs into ABC/HR to see how fatiguing ten codecs are in a single test.
EDIT: OK, eight

Cheers,
David.

I'm wondering if a high anchor is really necessary: are we likely to get 4 or 5 codecs together which all sound really bad? Plus the original is always there on every choice.



You'll realize in about 5 seconds that I only have space for eight, which I personally think is already too much

I've looking through my book, and what I'll probably do is scan it in to allow you to read the relevant sections for yourself. The is one method of interest called a "Balanced Incomplete Block" (BIB) design:

----------------------

BIB DESIGNS
"In BIB designs the panelists evaluate only a portion of the total number of samples (notationally, each panelist evaluates k of the total of t samples, k < t). The specific set of k samples that a panelist evaluats is selected so that in a single repetition of a BIB design every sample is evaluated an equal number of times (denoted by r), and all pairs of samples are evaluated together an equal number of times (denoted by lambda). The fact that r and lambda are constant for all the samples in a BIB design ensures that each sample mean is estimated with equal precision and that all pair-wise comparisons between two sample means are equally sensitive."

---------------------

It sounds very much like the method you propose, 2B, but perhaps it requires more listeners than you suspect to work out the statistics.

ff123

Thanks for looking this up - it does sound like what I was thinking.

I can understand how to make r constant, but the possible process for making lambda constant is making my head spin. There must be a formula for the minimum number of tests in a BIB cycle, even if I can't presently figure out how to juggle the codecs so that they meet the criteria.

As you say, this could require many more subjects (or repetitions) than is reasonable. Is there a way of calculating the required number?


What I did in a previous test (different circumstance, same resulting problem) was to say that, OK, I should really have either 24 or 48 or n times 24 subjects (in that particular test, IIRC), to make sure all possible combinations are tested equally; BUT since that's just not going to happen, AND because keeping track of all the possible combinations would make life more complicated, AND because most of the things which made a combination "different" weren't that important (e.g. presentation order); INSTEAD I'll just randomise all the things I don't want to measure or worry about (e.g. presentation order), and say that, over all, any bias due to any one or other of these things should cancel out, even though I haven't been ultra careful to make sure each and every combination of them all is equally likely.

In fact, we do that all the time - ABC/HR randomises the order, without checking that every possible order is used once or "n times" during an entire test. We don't worry, because, we assume that as long as the statistical bias due to each sample always appearing in the same place is removed, then whatever effect remains is drastically smaller than the one we hope to measure.


I'm not sure I can get away with this here. I did previously, because the effect I was measuring was huge (and I was only measuring 4 samples) whereas the effects which multiplied to 24 combinations were quite small. The effect of partitioning samples into two groups, where the actual content of each group may effect the judgement, may not be small. If there were no effect, then with samples with obvious artefacts, BS1116 would give the same results as MUSHRA (though more slowly) - this clearly isn't the case, because the comparison afforded by MUSHRA is useful to test subjects.


I'd be interested to know the required subject numbers for doing it the statistically correct way.

Cheers,
David.

P.S. I agree with your first statement - a high quality anchor is probably redundant in such a high quality test. Maybe a low quality anchor is too, if the BIB method means that it'll all average out anyway.

Ah - a web search appears to answer my question about subject numbers using BIB


EDIT: I retract that statement - I don't understand a word of it!


EDIT 2: Ok, I understand some of it - it's not simple, is it?

If I've got it correct, splitting 8 codecs into blocks of 4 would require 14 blocks to complete the BIBD sequence. (Assuming I can figure out the correct sequence.)


This raises one huge problem: It's alright the server sending out the 14 blocks in the sequence, but if one person doesn't come back with answers, you're one block short, and the whole thing falls apart. A solution is to have all 8 codecs in each download, and ABC/HR contacts the server to check which 4 should be chosen for the current block. If someone never sends in the results for that block, someone else will have to be given it later. You need "14 times n" people for each sample.


This all adds an amazing amount of complexity to the task - do you think it's possible, or worth it? It's an interesting challenge, but it's easy for me to say that because it's not my program!

Unless this is something you want to do (since it's work for you rather than me), I'll sleep on it and try and think of something different! (Like fewer codecs).

Cheers,
David.

Yeah, that's a problem isn't it? My book doesn't explain how to construct the table, although it does have an example of one which uses 3 out of 7 samples.



Although I don't know how many times each pair of codecs gets tested in this particular table, I'd guess it's something like 3 or 4 times. So assuming it's 4 times, you'd need 14 * 4 = 56 people to get the equivalent of 16 people listening to all codecs at a single sitting. That's a high price to pay. Not to mention the waste that occurs if the entire table isn't completed. And the complexity on the server side. I'm liking this option less and less.

ff123

Me too.

I'll include 8 codecs per test, and that's that.

I'll have some trial runs to see if it's still too many. I know it probably is, but I can't see what else to leave out. I'm currently down to:

mp2: 112, 128, 160, 192
AAC: 96
FM simulation
mp2 128 transcoded
mp2 128 using a different encoder

I'll probably do the normal four mp2s with a good encoder, and the alternative with a poor one. Or maybe the other way round! I've been told that most UK broadcasters are using something based on dist10.


Sample selection next. Are there any killer/difficult samples for mp2? Preferably "normal" music - ie something that might actually make it onto the radio!

Cheers,
David.

Well, any Beatles recording should be catastrophic, considering the joint stereo mode (IS) of mp2.
There are 2 choices for the encoder:
*do not use joint stereo, and so full stereo at 128kbps would be catastrophic
*use joint stereo and totally destroy the not-so-subtle stereo effects of Beatles recordings

Ouch. Perhaps there is still an alternative. You can split the codecs up into two tests and treat them separately. The disadvantage, of course is that you can't really directly compare one set against another in a statistically proper way, but maybe that's the lesser of two evils when compared with listening to 8 codecs per session.



How about if you have people post or upload short samples of music they like and have heard (or think they might hear) over the radio? But I hope you have a broadband connection for getting those samples by email or by ftp or by Usenet.

ff123

I'm a big Beatles fan - I'll try some CDs.

Please - do not email me clips of what you hear on the radio! I do have a broadband connection at work, but I can easily listen to the radio myself!


I was wondering if, like mp3, there were any samples that mp2 couldn't cope with at any bitrate. I might not use them, but it would be interesting to hear them!

So far I have harpsichord.


As for 8 samples being too many - I'll try them on some unsuspecting person - but I fear you're right, because you've done this plenty of times before. I tried it and I found it easy-ish, because I could discount two as being transparent, another as obviously being the worst, which "only" left 5 to rank. Your ABC/HR program makes it a lot easier (and less biassed) than ABXing them and then playing them in Winamp!

Maybe AAC will have to go, but that removes any relevance for those in the States.

Cheers,
David.

If anyone is following this...

The Glockenspiel from the SQAM CD is another mp2 killer sample to go along side the Harpsichord from that same CD.

However, it's not quite the "normal music" mp2 killer sample I was looking for! If anyone has any ideas, I'm all ears.

Cheers,
David.

On ff123.net
Filburt's test sample, the first 30 seconds from Dave Matthews Band, "#41."

This is fairly recent popular music recording with sharp hi-hat transients that could encourage pre-echo and high frequency problems, and it's natural music.

The fatboy codec killer (Fatboy Slim's opening to "Kalifornia", track 6 on You've Come A Long Way Baby) is also popular and recent, albeit a somewhat artificially generated sample.

Or the applause sample? Live concerts are going to be played on DAB.

I don't know which of these would give problems to .mp2

What are the criteria for the sample being a legitimate test of DAB? Popular music, recent music, natural sounds (as opposed to artificial)?

Regards,

DickD

Thanks DickD!

Because some of the worst mp3 killers do nothing with mp2 (fatboy, castanets), I hadn't tried the others. But after your suggestion, it turns out that the applause sample is absolutely terrible!

I'll try and find a different, but equally challenging piece of applause, though I will use the "well known" one if I must.


I don't have a strict criteria for test samples in mind - I'm just trying to avoid anything that would never be played on mainstream radio. When normal people or radio execs read the list of samples, they should think "yeah, I play/listen to that - if digital radio can't broadcast this properly, then we have a problem on our hands!" not "who'd listen to this?!"


Thanks for the help,

Cheers,
David.

P.S. where did applaud.wav come from? I have the file, but the link from the lame samples site "with information" doesn't have the information anymore! Which CD is it taken from?

It came from the Eagles's live version of Hotel California.

On the 64 kbit/s test, I used a sample with applause in it:

http://ff123.net/samples/Layla.flac

ff123

David,

I haven't encoded with mp2 since about 1997 (using whatever encoder I had in CoolEdit 96 at 256 kbps or so) before I found FhG's l3enc for .mp3

Surmising that the original MiniDisc encoding (before LP) was rather similar to MP2, I guessed that fatboy.wav would trash MP2 like it trashed MiniDisc (loads of added hiss, but fairly pleasant, analogue-sounding hiss, so I'm still content to use MiniDisc, which only occasionally trips up and not too badly on the stuff I mostly listen too). Glad I'm wrong, as the sort of artificial audio manipulation used on that track isn't all that uncommon in today's music.

Cheers,
DickD

Thanks ff123 - I guess I'll have to buy it. Is it the 1989 CD? This is the only "Live" Eagles album at cdnow. I know someone who owns it - but they have the DTS 5.1 CD - it's very nice.

DickD - There are two types of codecs: sub-band and transform. mp2 uses a filter bank only, so it's a sub-band codec (like mpc), mp3 adds a DCT (Discrete cosine TRansform) to this, making it a transform codec. AAC and OGG don't have a filterbank, but keep the DCT (or similar?) transform, so they're pure transform codecs. MiniDisc ATRAC is a hybrid codec, which (I think) means it uses both; whether this is both in series like mp3, or both in parallel/switched somehow, I'd have to look at the MD website to find out. But whatever, it's a transform codec.

There were well known problems with sub-band codecs, and mp2 in particular. The biggest problem is that they won't work well below approx 192kbps, but there are also a few problem samples at higher bitrates. Transform codecs like mp3 (and ATRAC on MD) get the bitrate down, but the transform introduces a whole new set of problems - mainly on transients - especially tonal sounds made up from repeated transients.

Many of the "problem" clips for mp3 (and, to a lesser extent, ATRAC, OGG, and AAC) fall into this category: it's the time > frequency transform that causes the problem. These clips aren't a problem for mpc (and, to a lesser extent, mp2) because it doesn't contain a time > frequency transform.

Where the mp3 test clips are a "problem" due to Joint Stereo, they do trip up lower bitrate mp2. The "problem" mpc clips I have (from the development stage - they're not a problem anymore) are so subtle that, given the general performance of mp2 at lower bitrates, you'd hardly count it as a serious problem!


If anyone reading this is thinking "Why don't we just use mp2 at higher bitrates then" - well, mpc is like mp2 in some ways, and better in many others - so it's a great choice. If you're stuck with mp2 (like in DAB) then the simple answer to decent quality is to use the codec at the higher bitrates it was designed for. We've just got to convince the UK broadcasters of this!

Cheers,
David.

Hmm, I don't know. Someone has an mp3 of it at work, and I happened to notice the applause section at the end.

I'm downloading a version right now. The funny thing is that it's different from the one at work (at least I think it is). But the applause at the end seems to be the same, starting from where the singer says "Thank you," then there's a whistling, and then there's somebody who makes a distinctive whoohoo cheer!

They don't just graft on stock applause, do they?

ff123

I'll check to make sure the one at work is really different. Maybe I'm just misremembering it.

No, it's off the 1994 "Hell Freezes Over". It will probably not me marked as "live" because it also includes some new studio tracks.

(Just a question, I think I've missed something here - why do you need to buy the CD if you've got the sample already?)

I have a problem...


I have to choose 1 mp2 encoder to use for most of the clips/bitrates. I have Qdesign, tooLame, and SoloH. The latter two have psychoacoustic models 1 and 2 (toolame has -1, 0, 3, and 4 as well!). Qdesign has only a quality/speed switch - the quality setting doesn't give better quality that the speed setting 100% of the time!!!

To my ears, where there's little stereo image, Qdesign and tooLame psychoacoustic model 1 (p1) come out on top. Where there's lots of stereo image, these two come out worst (especially Qdesign) - tooLame p2 is best. I'd be tempted to use tooLame p2 as representative, but it gives problems in some vocals, making it sounds like the singer has a "frog in their throat". Mike Cheng (tooLame author) notes that it sounds like Davros (of Dr Who fame - never mind - you're all too young! )


My gut feeling is that it would be wrong and unrepresentative to pick the best encoder (+ encoder option) for each sample - since no one in a radio station will sit changing encoding parameters to match source material all day. So I need to pick the overall best. Can I reliably do this myself, or shall I run a pre-test test first to determine this?

It's even more difficult than taking a sample and comparing 6 encodes (3 encoders, 2 options each) - there are different bitrates, and at some bitrates, the choice of stereo or joint stereo is not obvious. An exhaustive pre-test test would be even bigger than the planned test itself!

I'm leaning towards lame p2 as the best all-round encoder, and just going with that - ideas? comments?

Cheers,
David.

I'm surprised to hear that QDesign might not be quite better than ISO code.

I think that you should perhaps ask David McIntyre, who is (was) working for QDesign.
Some of us probably remember him from old the mp3.com msg board times.

So, in real life, the radio station would first choose their bitrate, and then choose the mp2 encoder they're going to use, based on which one sounds best at that bitrate?

Why am I somewhat skeptical that it actually works this way?

The pre-test is meant to save you time, not to add time! Maybe the best advice is to use your best judgment. However, short pretests wouldn't be prohibitive.

ff123

Well, listening to the output, the radio stations probably bought the first (i.e. oldest) encoders, left them set to psy model 1 (desipte psy model 2 being better on most material), and now run them at 128kbps.

I'm half tempted to do the same in my test!

However, I want a test which accurately reflects the possibilities on DAB, not just it's current limitations. So, if it would honestly be better to buy a new encoder rather than increase the bitrates, then so be it. (I would imagine it would be best to do both, but...)


I'll just go with my best judgement on this, since it's probably as good as that of most engineers in the field - at least in the are of audio coding - I don't claim to share their knowledge in other areas.

Cheers,
David.

Where did you see about this?

I searched for OpenDRM on Google, and all I found was info about Open Digital Rights Management. :B