If your modal dialog behaves like this:
then you have probably forced the dialog to have a specific height and/or width, and are using a Chromium-based browser like Edge or Chrome.
Also, probably you only see this when zooming out in your browser - this particular example only showed itself at 50%, other zoom levels looked OK.
And sometimes you see scrollbars appear and disappear as well, but not always.
So what is happening here? Well, that took me way too long to find out (what else is new...) and turned out to be quite easy to fix.
The culprit here is the overflow CSS property
For modal dialog this is set to "auto", which means "If overflow is clipped, a scroll-bar should be added to see the rest of the content".
And for some reason Chromium-based browsers keep changing their mind about whether or not to show the scrollbars during rendering, and they flip back and forth.
To fix this all you need is a bit of CSS - add it to the page somewhere, e.g. under "Page -> CSS -> Inline":