Ok, so this is probably me having a "Duh" moment here, but I forgot that imageScaleToFit goes both ways. What do I mean by that? If you are using the code to create a thumbnail, like so:
<cfset imageScaleToFit(myImage,255,255,"highestQuality")>
and the image is less than 255 tall or wide than ColdFusion will increase the size of the image. As I said - duh. Luckily this is simple enough to solve:
<cfif myImage.width gt 255 or myImage.height gt 255>
<cfset imageScaleToFit(myImage,255,255,"highestQuality")>
</cfif>
You could also make a simple UDF out of this as well but not sure if it is worth the trouble.