I am considering the purchase of a CD based MP3 Player, either a Sony or a iRiver. Before my purchase, I would like to construct an MP3 test CD so that I can see what, if any, problems the player has with different types of MP3 files.

So far, I have this on my list:

-MP3s encoded with -preset standard and in CBR 128/192/256/320 formats.
-Different versions of ID3 tagging.
-Really large MP3 files (around 2 to 3 hundred megabytes)

Can anyone suggest something else to test for?

Most players won't have problems playing CBR MP3s, so just concentrate on -APS/APX/VBR/joint stereo mp3s.

Just to share my experience:
-Sony DC-J01: Sony's first MP3CD player. Problems playing high bitrate VBR with joint stereo (results in crackling/skipping). Having the files on stereo (with same VBR settings) gave no problems. Other bitrates are fine. Only display ID3v1, doesn't support v2.
-Sony D-NE9: Sony's first gen AtracCD/MP3CD player. Play VBR files with no problems. However, sometimes it detects VBR MP3s as 32kbps, thus when the buffer runs out, the unit spins the CD a little bit late, resulting in skipping. Also, picky in CD-R media burned at high speed (ie. unable to play some files halfway or near the end of the CD). Support ID3v2.
-Panasonic SL-SX430: Low end MP3CD player. No problems whatsoever. All MP3 that I have play fine. No problems with media compatibility either. Cheap, but cannot display ID3 tags.

Edit: Add ID3 info.

I own the SL-SX420. Exact same comments as pika2000 about it and his 430. No problems with any mp3s I've made so far but no ID3 tag display.

Try different sample rates : 22.05 kHz, 24 kHz, 32 kHz, 44.1 kHz, and 48 kHz.

EDIT : oh, and try a mono MP3 too. And dual channel stereo, and Intensity stereo.

I made a test disc by taking one song and slicing it into about 10 parts. I made 10 tracks out of one song.

When played back on my PC, it is perfectly gapless all the way though. When played back on a car CD MP3/CD player, the gaps vary depending on the manufacturer/model.

That's true. There are not many MP3 players that support gapless out there. IMO, CD based players will not play gapless since most players stop the CD to save battery life, and only spins it to fill the buffer or when the next track starts, hence resulting in a gap, waiting for the CD to spin and fill the buffer. Even in-car CD/MP3 units don't play gapless, even though they'll spin the CD all the time. It seems manufactures have not considered this feature to be important. The only MP3 player that support gapless MP3, that I know of, is the Rio Karma, which is HD based.

Note: The 2004 line Sony AtracCD walkmans can play AtracCDs gapless (called seamless play, I think). Unfortunately, this is only for AtracCDs, not MP3s.

i woud lstil ltry API as many old mp3 players does not play 320kbits mp3's

There is a funny bug with iRiver iMP-550 (MP3 CD player). I have a nonstop mix encoded with --alt-preset standard that I cut into separate tracks with MusiCutter and MP3Gained. The first track loops infinitely! That never happened with CBR MP3s cut up that way.

That said, iRiver players have a reputation for playing any kind of MP3 you throw at them.

Did you test this with the latest (1.70) firmware release ?

I usually copy the last seconds of Pink Floyd's "The Happiest Days Of Our Lives" and the first seconds of "Another Brick In The Wall, Part 2" to test gapless playback capabilities.

Yes, though IIRC, the problem was there with v1.40 as well.