Rather than trying to create a gamma ramp array of the appropriate size in
drmmode_crtc_init when the GAMMA_LUT property should be used, just flag the crtc
as wanting to use the GAMMA_LUT property and then replace the gamma ramp later,
right before calling xf86HandleColormaps. This avoids a problem during initial
startup where xf86RandR12CreateObjects12 hard-codes a gamma ramp size of 256,
causing xf86RandR12CrtcSetGamma to read past the end of the DIX layer's RandR
gamma ramp array:
PreInit
drmmode_pre_init
drmmode_crtc_init
crtc->gamma_size = 1024
ScreenInit
xf86CrtcScreenInit
xf86RandR12Init
xf86RandR12Init12
xf86RandR12CreateObjects12
RRCrtcCreate
randr_crtc->gammaSize = 0
xf86RandR12InitGamma(pScrn, 256)
RRCrtcGammaSetSize
randr_crtc->gammaSize = 256
xf86RandR12InitGamma
xf86RandR12CrtcInitGamma
RRCrtcGammaSet
xf86RandR12CrtcSetGamma
// crtc->gamma_size is 1024 here, while randr_crtc->gammaRed
// is a 256-element array.
memcpy(crtc->gamma_red, randr_crtc->gammaRed, crtc->gamma_size * sizeof(crtc->gamma_red[0]));
drmmode_setup_colormap
xf86HandleColormaps
xf86RandR12InitGamma
RRCrtcGammaSetSize
randr_crtc->gammaSize = 1024
Fixes: 245b9db0 - modesetting: Use GAMMA_LUT when available
Closes: https://gitlab.freedesktop.org/xorg/xserver/-/issues/1126
Signed-off-by: Aaron Plattner <aplattner@nvidia.com>
Reviewed-by: Robert Morell <rmorell@nvidia.com>
b75d0cca28