Jst a quick question,

I do all my encoding with LAME 3.96.1 (-V 0 --vbr-new)

Sometimes the list of bitrates it uses are:
128
160
192
224
256
320

Other times it's:

32
128
160
192
224
256
320

I was just wondering why that is? Is it some files never have silence and others do?

Yes, that's it. Digital silence is encoded at 32 kbps with LAME VBR.
With 3.96.1 the bitrate floor is fixed at 128 kbps (+ 32kbps for pure silence); with 3.97 and --vbr-new, there's no limitation anymore.

Thanks for the quick answer