This looks like the issue we have seen with other Mac systems using the AMD GPU.
Our developers are currently in contact with Apple about an issue related to Mac systems with AMD graphics cards. We don’t have a full solution currently but you can try going into the Preferences Panel and setting the max memory to 20% and selecting low power mode.
Thanks for your response, I have 1 Mac mini M1 and 1 iMac Intel and on both I had the same exact problem, even changing the memory as I saw the suggestion in the forum.
It could sound strange but finally I could do the job realising that the problem with my video was at the very end when a kind of tale (S8mm) video fades out, so I did the encode just until this frame
and it worked!, later I did the very small tale encode with no problem so I join all and voila! The important thing is that the app rocks, is amazing how I could improve a so bad image!