Ok I was playing around with the --scale switch and notice that I would get files that were smaller if I used a setting less than one.

So I did a test and encoded the same wav twice once without the switch and once with the switch set a .1
The one with the switch and obviously the lower volume was alittle smaller. I then used mp3gain to make the files the same volume. The file sizes did not change. I now have two files whos only difference is file size.

So....

Am I some how losing quality with the --scale switch?

This issue was explored in another post, but I think the conclusion was that you are losing quality by using the scale switch, to an extreme scale the mp3 to 0 and then compress it (you'll get 32kbs CBR).

shizzle my nizzle

Yes.
--scale makes the input audio louder or quieter before it gets compressed. If you make it quieter (number < 1) then more of the already-quiet sounds in the audio will now fall below the threshold of hearing and LAME will basically throw them away, hence there's less data to compress, therefore smaller filesize. MP3gain'ing the resulting file will bring you back to the same overall volume, but you've lost a whole bunch of the very-quiet sounds that should be audible but are now completely gone.