Planning Surveys/Tests On "General Public & Audio Professiona Options

[ezlez]

Synopsis
As a part of a research project, the goal is to find what audio format/codec is preferred by the general population, and what is audibly different amongst those files, determined by music professionals,. By general population, I mean anybody who has ever listened to music digitally through a computer or music player; By music professionals, I mean people who have studied or are ardently absorbed music and or acoustics, like teachers, or music enthusiasts such as some of the members of the community here on the Hydrogen Audio. Essentially, the purpose of this project is to find a formidable audio codec that anyone can use that can complement anyone's need or desire for a format that's apt in terms of sound, fidelity, and low or efficient bandwidth/storage size; Of course, there is no "best" audio format, as there are different formats that suit different needs.

For this project, three surveys will be conducted to figure out what audio format is preferred by the general population and what audio format is efficient by music professionals. Two surveys will be conducted on the general public, to see what digital audio format is preferred based on sound and fidelity and size of memory - One survey will be a series of structured questions asking them what they want out of digital audio files, while the other survey will be an audible, double blind survey where the subject will be exposed to a few songs expressed in all forms of the compressed formats (Eight formats; see List of Codecs), and the subjects will rank each file on preference, or what file they prefer the most based on audible perception. For the music professionals, there will be double blind trials somewhat similar to the one for the general public, except the subjects will account mostly for differences, instead of preference. The double blind testing method will be ABX testing.

List of Codecs (Digital Audio Types) being tested:
Lossy: MP3, Ogg Vorbis, AAC, Musepack
Lossless: FLAC, APE, Wav Pack
Uncompressed: WAV


Agenda
At the moment, I'm somewhat still in the planning stages of this project - I need to know:

• Is the audible double blind survey on the public really worthwhile & efficient? How can I improve it?
• How can I survey the public without losing their interest or fatiguing them?
• How can I attract different people for the tests?
• How many people I need to survey for both the general public and professional surveys?
• How many trials should there be for the ABX testing, and how exactly do I perform the tests?
• What Windows program should I use for the ABX testing? In other words, which one is the "best"?
• What encoders should I use for each specific audio codec?
• What/How many songs or sound files should I use?
• Would headphones be an efficient device to test with for the surveys?
• What bitrate should I compress the lossy formats to?
• If I am posting the surveys online, how exactly go about doing so?

I have many resources and help that I can use for these surveys; at the moment, I'm talking to an instructor at a local community college that may consent to allow me to perform the tests/surveys at a local community college, and I am seriously considering of performing the tests online. I know that some, or nearly most the aspects of this project may not seem exactly professional, but I am doing my personal best to learn more about audio formats, testing, etc. I would greatly appreciate it if any of you have any insight or advice to give on this project

[hellokeith]

Might I inquire as to why you have omitted WMA ? It is supported on ~ 99% of the hardware out there (dap's, phones, netbooks, laptops, PC's, consoles, etc).

Also, I think you are going to need to be very careful about getting skewed results. For example, at the last university I frequented, the iPod was really the only dap that anyone would ever use. So in that environment, it would be expected that apple drm protected iTunes aac files would be the dominant codec / file type. I don't know what the breakdown of drm to non-drm files on iTunes is these days, but that could cause some issues with the results.

[solive]

If the multiple lossy CODECS you are testing are lower quality with easily discernible audible differences then a MUSHRA testing protocol may be more appropriate than ABX.

Also, are you sure some of this survey information (what are the preferred CODECS) isn't already available somewhere? I think the answers to some of these questions can be found in this forum and reading various papers written on the topic in J.AES and elsewhere.

Cheers
Sean Olive
Audio Musings

This post has been edited by Frank Bicking: Mar 14 2010, 02:27

Reason for edit: Removed fullquote of the first post.

[lvqcl]

double blind survey where the subject will be exposed to a few songs expressed in all forms of the compressed formats (Eight formats ... The double blind testing method will be ABX testing

So, you want to compare FLAC vs. uncompressed WAV in ABX test?