Foobar doesn't seem to tell you which file was which (ABXY) in each session, the log file just records your successful/failed attempts and the percentage chance that you were guessing. If you are comparing two lossy codecs, how can you know which was which and which was the better one?

You don't; the ABX protocol is generally insufficient for such a test, and a protocol such as ABC/HR is used instead.

You seem to have a misconception about ABX tests there.

ABX tests can only confirm that you hear a difference between two files, and ABX logs contain sufficient information to make that conclusion, but they cannot tell you which one is better. Of course, when you successfully test an original file against a lossily compressed one (e.g. WAV vs. MP3), you may conclude that the original one is better, since the purpose of lossy compression was to make the result sound like the original. However, when testing between two lossy files, this is not possible. Even if you were comparing them against an original file in two separate sessions, and could hear a difference between both files and the original, the decision which one sounds better is one you would have to make on your own.

There is different type of test called ABC/HR where you compare a number of lossy files against the original in multiple rounds. In each round, you compare a randomly chosen file against the reference and assign a subjective rating on a 1.0 to 5.0 scale to it. To protect you from bias, you do not know which file you are currently testing. In the end, you perform a statistical analysis on the ratings in order to find out whether there is sufficient difference between your ratings, and to conclude which (if any) file sounds better.

Further reading:
What is a blind ABX test?
Hydrogenaudio wiki article on ABC/HR

Correct me if I'm wrong, but AUIU foobarABX doesn't shuffle the files associated with of A and B during the test . It only randomizes their presentation. So whichever file was "A" at the start of the test is still "A" at the end.

I compared WMAPro and MP3 at 192 and 128kbps. I could hear a difference with the WMA at 192 and MP3 at 128, but I could not tell which file was which (the ABXY in each trial)

Is there an ABC/HR test that can play MP3 and WMApro?

krabapple is right,

A and B will remain the same during the test. The log also mentions which files were used.
A is the first track you selected in the playlist to launch the ABX test, B the second one.

Since you assign X (and Y) to A or B during the test, you should know which one is which.


ABC/HR by ff123 will only accept WAV files, so you would have to decode them first.
schnofler's ABC/HR for Java supports some audio codecs via command line decoders.
I don't know of any for WMA, so your best bet is to decode to WAV as well.

foobar2000 does the same by the way, presumably for faster seeking and switching between tracks.