The vidmode extension emulation in Xwayland reports the modeline based
on the current mode.
To do so, it searches for the mode using `xwl_output_find_mode(-1, -1)`
which is supposed to return the current mode, whatever that mode is.
With XRandR emulation, in rootless mode, the default value is the mode
at index 0. That assumption, however is not true when running rootful.
That means that the vidmode extension will always return the highest
mode available, which is 5120x2880, with Xwayland running rootful:
$ xwayland-run -geometry 1024x768 -- xvidtune -show
"5120x2880" 1276.50 5120 5560 6128 7136 2880 2883 2888 2982 -hsync +vsync
Luckily, when Xwayland is running rootful, we have the current mode size
conveniently stored in dedicated fields of the xwayland output struct,
so we can use that to search for the right mode being used and report
that through the vidmode extension:
$ xwayland-run -geometry 1024x768 -- xvidtune -show
"1024x768" 63.50 1024 1064 1176 1328 768 771 775 798 -hsync +vsync
That fixes legacy games using the vidmode extension and rendering at the
wrong size when running within Xwayland rootful.
Signed-off-by: Olivier Fourdan <ofourdan@redhat.com>
Part-of: <https://gitlab.freedesktop.org/xorg/xserver/-/merge_requests/1641>
e2e5842444