Fix some cases where the detached video dialog would get set to bogus sizes

At very large and very small sizes, setting the window's size with
SetSize may not actually result in the window changing to the requested
size.  Once this happened future adjustments to the size were incorrect,
and the video display would sometimes not fit in the window, and on os x
the window could get set to a negative size, with weird results.