Floating spots/freckles on human skin

I’m getting good results with my upscaling, but i do have one particular issue:

If a person has freckles, birthmarks or similar spots on their face, arms or body in general, that particular area of the upscaled video becomes kinda “floaty” or a bit flickering, like AI has isolated that small area and can’t really fuze into the rest of the image.

Has anyone else experienced this and is there a solution to it?

I mainly use medium > progressive > high compression > gaia high quality > 150% scale > 0,5 grain amount > MP4