ABX on iPod, I plan to make program to ABX on iPod Options

[sehested]

I plan to make a program to ABX music samples on an iPod.

My idea for this program is:

- Take a FLAC file with tags as input
- Convert this song using a number of predefined encodings
- Song tags should be "Title 1A", "Title 1B", "Title 2A", "Title 2B", "Title 3A", "Title 3B" in case three encoding options are selected.
- For every pair of AB one would be lossless.
- Encoder track number would be randomly selected.
- Create "mapping file with details on encoding and track mapping"
- Make play list in iTunes "Listening Test - Title" with all songs

Now this play list can be downloaded to an iPod and the user can do a ranking of the files.

After ranking the files he can view the mapping file and compare his results.

This would allow people to perform an unbiased AB test on their iPod.

I'm getting an iPod mini in two weeks and will use this program to do ABX testing to find the optimal encoding option for me.

I just hope that the iPod mini is not having the same problem as I discovered on its older sibling:

3G iPod Badly distorts high bit rate MP3

Now before I go ahead and do this program and maybe reinvent the wheel. Please let me know if such a program already exist out there.

[KikeG]

http://www.kikeg.arrakis.es/fileabx/

But it only accepts wav files as input, I'm afraid.

Edit: also, this program implements "pure" abx testing, whilst your proposal would be more like "abc/hr" testing.

This post has been edited by KikeG: Jul 19 2004, 08:38

[sehested]

I have now given the iPod ABX idea some more thought and realised that
the key feature of the program should be to mask the origin of the
samples and take care of the hassle of keeping sample files organised.

Encoding of files are better done completely outside this
program. The number of options for converted a file is simply more
than I would like to handle and besides the iTunes OLE interface only
allow you to select the encoder not to change its settings.


I anticipate the basic use of ipod-abx will be to compare a number of
encodings with a reference as follows, in this example three files:

1. Start out by preparing the files you want to compare.
2. Run ipod-abx <name> <reference> <file1> <file2> <file3>
to prepare random playlist.
The playlist would look like this:
  Artist  Song
  iPod-ABX <name> 1 A
  iPod-ABX <name> 1 B
  iPod-ABX <name> 2 A
  iPod-ABX <name> 2 B
  iPod-ABX <name> 3 A
  iPod-ABX <name> 3 B
3. Download the playlist "iPod-ABX <name>" to your iPod.
A song will appear on the iPod play screen as:
  <name> 1 A
  iPod-ABX
4. Listen to the songs and rate them 1-5, but don't rate the
references (use 0 stars). If you want decimals for your rating,
e.g. 4.2, note it down and make other notes as you please during the
tests.
5. Upload the ratings to iTunes.
6. Enter any notes you made as comments for the songs.
If a song comment contains a decimal number, e.g. 4.2 this number
will be used by ipod-abx as the rating, rather than the stars.
7. Run ipod-abx -v <name> to compile and display the results:
sample : <name> 1 A
filename : <reference>
encoder  : CD (EAC)
rating : 4.8

sample : <name> 1 B
filename : <file2>
encoder  : CD (EAC) lame aps

sample : <name> 2 A
filename : <file3>
encoder  : CD (EAC) lame api
rating : 1
note  : Unbeleivably badly distorted piano

sample : <name> 2 B
filename : <reference>
encoder  : CD (EAC)

sample : <name> 3 A
filename : <file1>
encoder  : iTunes 4.2.0.72, QuickTIme 6.5
comment  : CD (EAC) aac 128
rating : 4.2
note  : Piano lacking definition

sample : <name> 3 B
filename : <reference>
encoder  : CD (EAC)

Reference identified 2 out of 3 times.



ipod-abx can also be used to ABX two samples a number of times,
e.g. to compare LAME 3.90 with LAME 3.96 on iPod:

1. Start out by preparing the two files you want to compare.
2. Run ipod-abx -r <number> <name> <reference> <file1>
to prepare random playlist with the two same files repeated <number> times.
The playlist would look like this if <number> is 5:
  Artist  Song
  iPod-ABX <name> 1 A
  iPod-ABX <name> 1 B
  iPod-ABX <name> 2 A
  iPod-ABX <name> 2 B
  iPod-ABX <name> 3 A
  iPod-ABX <name> 3 B
  iPod-ABX <name> 4 A
  iPod-ABX <name> 4 B
  iPod-ABX <name> 5 A
  iPod-ABX <name> 5 B
3. Download the playlist "iPod-ABX <name>" to your iPod.
The songs will appear on the iPod play screen as:
  <name> 1 A
  iPod-ABX
4. Listen to the songs and rate the non-reference.
5. Upload the ratings to iTunes.
6. Run ipod-abx <name> to compile and display brief results:
Reference identified 5 out of 5 times.


<reference> or <file> can be any one of .flac, .wav, .mp3, .m4a, or .m4p.

.flac, and .wav are converted to Apple lossless.
.mp3, .m4a, and .m4p are added as they are.

The following tags are set:
Name: <name> <sample> <A/B>
Artist: iPod-ABX
Album: iPod-ABX <name>
Track Number: <tracknumber in playlist>
Track Count: <total tracks in playlist>
Discnumber: 1
Disccount: 1

The following tags are left as is:
Encoder

The following tags are cleared:
Year, Genre, Grouping, Compilation, Rating, Composer, BPM

Soundcheck must be enabled in iTunes and on the iPod to have the songs
play back at similar volume.


Usage: ipod-abx <name> <reference> <file> [file]...
    Prepares iTunes playlist for abx listening test.
    Randomly renames files, add files to iTunes, update their tags
    and create playlist.

    Use -r <number> to repeat test <number> times
    Use -s <mm:ss>  to set start time of .m4p file. Length of sample
                    will be limited to length of reference.

    ipod-abx <name>
    Compile result of listening test based on ratings and comments
    in iTunes of the tested songs.

    Use -v          for verbose output.


ipod-abx will complain about:
file type not supported
files not existing
iTunes not starting
album / playlist already in iTunes
length of songs differs by more than a few seconds
tracks not present in iTunes when compiling result
any error codes from iTunes

So that's the plan. I will probably start coding in about two weeks when
I'm back from my holidays.

[sehested]

http://www.kikeg.arrakis.es/fileabx/

But it only accepts wav files as input, I'm afraid.

Edit: also, this program implements "pure" abx testing, whilst your proposal would be more like "abc/hr" testing.

Thanks, for the input.

My main concern is that I want to perform the ABX on the iPod itself.
First I considered to convert all the test files to Apple lossless to completely mask the files origin, but I have gone away from that idea.
I want to be able to reveal defects in the iPod decoding which would not be obvious if the iPod was only to playback lossless files. E.g. some high bitrate mp3 files are distorted on the iPod, but not if the file is converted to wave or lossless first.

[sehested]

I have now completed iPod-ABX.

Please enjoy:

ipod-abx

Here's the usage notes:

Usage: ipod-abx <name> <reference> <sample> [sample]...

    Make iTunes iPod playlist for listening test.
    Randomly renames files, add files to iTunes, update their tags
    and create ABC/HR playlist with A as reference. Rate B and C. One being
    the reference and the other a sample.

    Use -x <number> to make ABX playlist with <number> sets with A as
                    sample, B as reference, and X as unknown. Rate X 5 if
                    reference or 1-4 if sample.
    Use -s <scale> set lowest rating of scale:
                          0 : 0-5 stars equals  - , 1.0, 2.0, 3.0, 4.0, 5.0
                          1 : 0-5 stars equals 1.0, 1.8, 2.6, 3.4, 4.2, 5.0
                          2 : 0-5 stars equals 2.0, 2.6, 3.2, 3.8, 4.4, 5.0
                          3 : 0-5 stars equals 3.0, 3.4, 3.8, 4.2, 4.6, 5.0
                          4 : 0-5 stars equals 4.0, 4.2, 4.4, 4.6, 4.8, 5.0
                          5 : 0-5 stars equals 4.5, 4.6, 4.7, 4.8, 4.9, 5.0
    The scale can be altered on the iPod by changing the rating of the
    reference.


    ipod-abx -u <name>

    Update song names in iPod with rating for making ranking between
    samples easier.


    ipod-abx <name>

    Prepare result file of listening test based on ratings and comments
    in iTunes of the tested songs.

    Use -i          to update playlist with rating and play info from iPod.
    Use -v          for verbose output.


    ipod-abx -c <heading>

    Compile result of all listening tests whose name start with heading.
    Songs not played will be excluded from the results.
    The folder name of each songs file name will be use to determine the
    encoder. See the example below for a suggested folder structure for
    test samples.


Hint:  Commmands in comment tag are treated as follows:
        finish( mm:ss) set finish time of song.
        start( mm:ss) to set start time of song.

Hint:  <reference>/<sample> can either be a file or a folder name. In case
    file do not exist it is assumed to be a folder name. ipod-abx will look
    for this folder in the current folder and in the previous folder used
    by ipod-abx. When specifying a folder name only part of the folder name
    needs to be given and "-" will match any part of the folder name. In
    case several folder names match the alphabetically latest will be
    choosen.

Examples:
    Assuming test samples are organised in this folder structure:

        Test Samples
          FLAC
            iTunes 4.2 aac 128
            iTunes 4.2 mp3 112 vbr
            iTunes 4.2 mp3 128 cbr
            iTunes 4.6 aac 128
            iTunes 4.6 mp3 112 vbr
            iTunes 4.6 mp3 128 cbr
            lame 3.96 mp3 128 v5

    To generate the playlist "128kbps - Waiting" with reference from
    "FLAC" and samples from "iTunes 4.6 mp3 112 vbr",
    "iTunes 4.6 mp3 128 cbr" and "lame 3.96 mp3 128 v5" folders:

ipod-abx "128kbps - Waiting" flac itunes-mp3-112 itunes-mp3-128-cbr lame-v5


    To create result file after completing iPod listening test:

    ipod-abx -iv "128kbps - Waiting"


    To compile results of a number of test all named "128kbps - <song>":

    ipod-abx -c 128kbps


    To generate the playlist "Waiting" with eight ABX tests between samples
    from folders "iTunes 4.6 mp3 112 vbr" and "lame 3.96 mp3 128 v5":

    ipod-abx -x8 Waiting itunes-mp3-112 lame-v5

[sehested]

Some design notes:

Basic use of ipod-abx will be to compare a number of
encodings with a reference as follows, in this example three files:

1. Start out by preparing the files you want to compare.
2. Run ipod-abx <name> <reference> <sample1> <sample2> <sample3>
to prepare random playlist.
The playlist would look like this:
  Artist  Song
  <name>  A (reference)
  <name>  1 - B (?)
  <name>  1 - C (?)
  <name>  A (reference)
  <name>  2 - B (?)
  <name>  2 - C (?)
  <name>  A (reference)
  <name>  3 - B (?)
  <name>  3 - C (?)

3. Download the playlist "<name>" to your iPod.
A song will appear on the iPod play screen as:
      2 - B (?)
    <name>
4. Listen to the songs and rate them 1-5, but don't rate the
references (use 0 stars). If you want decimals for your rating,
e.g. 4.2, note it down and make other notes as you please during the
tests.
5. Upload the ratings to iTunes.
6. Enter any notes you made as comments for the songs.
If a song comment contains a decimal number, e.g. 4.2 this number
will be used by ipod-abx as the rating, rather than the stars.
7. Run ipod-abx -v <name> to compile and display the results:
sample * : A (reference)
filename : <reference>
info  : Lossless, 919 kbps, 44.1 kHz
encoder  : CD (EAC)

sample * : 1 - B (?)
filename : <reference>
info  : Lossless, 919 kbps, 44.1 kHz
encoder  : CD (EAC)
rating : 4.8

sample : 1 - C (?)
filename : <sample2>
info  : MP3, 192 kbps, 44.1 kHz
encoder  : CD (EAC) lame aps

sample : 2 - B (?)
filename : <sample3>
info  : MP3, 320 kbps, 44.1 kHz
encoder  : CD (EAC) lame api
rating : 1
note  : Unbeleivably badly distorted piano

sample * : 2 - C (?)
filename : <reference>
info  : Lossless, 919 kbps, 44.1 kHz
encoder  : CD (EAC)

sample : 3 - B (?)
filename : <sample1>
info  : AAC, 128 kbps, 44.1 kHz
encoder  : iTunes 4.2.0.72, QuickTime 6.5
comment  : CD (EAC) aac 128
rating : 4.2
note  : Piano lacking definition

sample * : 3 - C (?)
filename : <reference>
info  : Lossless, 919 kbps, 44.1 kHz
encoder  : CD (EAC)

Missed 1 reference out of 3.



ipod-abx can also be used to ABX two samples a number of times,
e.g. to compare LAME 3.90 with LAME 3.96 on iPod:

1. Start out by preparing the two files you want to compare.
2. Run ipod-abx -x <number> <name> <reference> <sample>
to prepare random playlist with a refence and a sample repeated <number> times.
The playlist would look like this if <number> is 8:
  Artist  Song
  <name>  A (sample)
  <name>  B (reference)
  <name>  1 - X (?)
  <name>  A (sample)
  <name>  B (reference)
  <name>  2 - X (?)
  <name>  A (sample)
  <name>  B (reference)
  <name>  3 - X (?)
  <name>  A (sample)
  <name>  B (reference)
  <name>  4 - X (?)
  <name>  A (sample)
  <name>  B (reference)
  <name>  5 - X (?)
  <name>  A (sample)
  <name>  B (reference)
  <name>  6 - X (?)
  <name>  A (sample)
  <name>  B (reference)
  <name>  7 - X (?)
  <name>  A (sample)
  <name>  B (reference)
  <name>  8 - X (?)
3. Download the playlist "<name>" to your iPod.
The songs will appear on the iPod play screen as:
      1 - X (?)
    <name>
4. Listen to the songs and rate the non-reference.
5. Upload the ratings to iTunes.
6. Run ipod-abx <name> to compile and display brief results:
Sample/reference identified 8 out of 8 times.




The result of several individual listening tests can be combined to produce an overview of the results:

"128" listening tests compiled results:
                      A=iTunes 4.2 m4p 128
                                B=iTunes 4.6 aac 128
                                    C=iTunes 4.6 aac 160
                                              D=iTunes 4.6 mp3 112 vbr
                                                  E=iTunes 4.6 mp3 128 vbr
                                    lame 3.96 mp3 128 v5=F
                      A      B      C      D      E      F
-------------------------------------------------------------------------------
        BigYellow  -  4.9    5.0    5.0    5.0    5.0
                Echo    3.4    5.0    4.2    3.0    4.6    4.6
                Gone  -  4.8    5.0    5.0    4.9    5.0
        Hungarian  -  5.0    5.0    5.0    5.0    5.0
      ItCouldBeSweet  -  5.0    5.0    4.8    4.8    4.8
        Kraftwerk  -  4.9    5.0    4.9    5.0    5.0
    OrdinaryWorld  -  5.0    5.0    5.0    5.0    4.5
            RedHouse  -  5.0    5.0    4.5    4.8    4.9
            Rosemary  -  5.0    5.0    5.0    5.0    5.0
            Stairway  -  4.6    5.0    5.0    3.8    5.0
          Waiting  -  5.0    5.0    4.5    4.8    5.0
-------------------------------------------------------------------------------
          AVERAGE    3.4    4.9    4.9    4.7    4.8    4.9
===============================================================================
Prepared 2004-07-29 19:39 with iPod-ABX v0.4




<reference> or <file> can be any one of .flac, .wav, .mp3, .m4a, or .m4p.

.flac, and .wav are converted to Apple lossless.
.mp3, .m4a, and .m4p are added as they are.

The following tags are set:
Name: <sample>
Artist: <name>
Album: <name> [iPod-ABX]
Track Number: <tracknumber in playlist>
Track Count: <total tracks in playlist>
Discnumber: 1
Disccount: 1
Rating: 100 (five stars)
Grouping: iPod-ABX / iPod-ABC/hr

The following tags are left as is:
Encoder

The following tags are cleared:
Year, Genre, Compilation, Composer, BPM, Artwork

Soundcheck must be enabled in iTunes and on the iPod to have the songs
play back at similar volume.

[underground_soun...]

Thanks a lot sehested, this is greatly appricated. I've always wanted to be able to conduct accurate abx tests on my iPod, a lot of people did see the importance of doing abx on the iPod itself and told me to just test the formats on my computer. But it is important that such tests be done on the iPod and what you use as its output. My 3G iPod occassionaly does have distortion issues and formats/bitrates sound a lot different coming out of car audio than they do a pair of tiny computer speakers. Thanks a lot!