xserver/hw/xfree86
Aaron Plattner 4226c6d032 modesetting: Check whether RandR was initialized before calling rrGetScrPriv
Calling rrGetScrPriv when RandR isn't initialized causes an assertion
failure that aborts the server:

 Xorg: ../include/privates.h:121: dixGetPrivateAddr: Assertion `key->initialized' failed.

 Thread 1 "Xorg" received signal SIGABRT, Aborted.
 0x00007ffff78a8f25 in raise () from /usr/lib/libc.so.6
 (gdb) bt
 #0  0x00007ffff78a8f25 in raise () from /usr/lib/libc.so.6
 #1  0x00007ffff7892897 in abort () from /usr/lib/libc.so.6
 #2  0x00007ffff7892767 in __assert_fail_base.cold () from /usr/lib/libc.so.6
 #3  0x00007ffff78a1526 in __assert_fail () from /usr/lib/libc.so.6
 #4  0x00007ffff7fb57c1 in dixGetPrivateAddr (privates=0x555555ab1b60, key=0x555555855720 <rrPrivKeyRec>) at ../include/privates.h:121
 #5  0x00007ffff7fb5822 in dixGetPrivate (privates=0x555555ab1b60, key=0x555555855720 <rrPrivKeyRec>) at ../include/privates.h:136
 #6  0x00007ffff7fb586a in dixLookupPrivate (privates=0x555555ab1b60, key=0x555555855720 <rrPrivKeyRec>) at ../include/privates.h:166
 #7  0x00007ffff7fb8445 in CreateScreenResources (pScreen=0x555555ab1790) at ../hw/xfree86/drivers/modesetting/driver.c:1335
 #8  0x000055555576c5e4 in xf86CrtcCreateScreenResources (screen=0x555555ab1790) at ../hw/xfree86/modes/xf86Crtc.c:744
 #9  0x00005555555d8bb6 in dix_main (argc=4, argv=0x7fffffffead8, envp=0x7fffffffeb00) at ../dix/main.c:214
 #10 0x00005555557a4f0b in main (argc=4, argv=0x7fffffffead8, envp=0x7fffffffeb00) at ../dix/stubmain.c:34

This can happen, for example, if the server is configured with Xinerama
and there is more than one X screen:

 Section "ServerLayout"
   Identifier "crash"
   Screen 0 "modesetting"
   Screen 1 "dummy" RightOf "modesetting"
   Option "Xinerama"
 EndSection

 Section "Device"
   Identifier "modesetting"
   Driver "modesetting"
 EndSection

 Section "Screen"
   Identifier "modesetting"
   Device "modesetting"
 EndSection

 Section "Device"
   Identifier "dummy"
   Driver "dummy"
 EndSection

 Section "Screen"
   Identifier "dummy"
   Device "dummy"
 EndSection

The problem does not reproduce if there is only one X screen because of
this code in xf86RandR12Init:

 #ifdef PANORAMIX
     /* XXX disable RandR when using Xinerama */
     if (!noPanoramiXExtension) {
         if (xf86NumScreens == 1)
             noPanoramiXExtension = TRUE;
         else
             return TRUE;
     }
 #endif

Fix the problem by checking dixPrivateKeyRegistered(rrPrivKey) before
calling rrGetScrPriv. This is similar to what the xf86-video-amdgpu
driver does:
fd66f5c0be/src/amdgpu_kms.c (L388)

Signed-off-by: Aaron Plattner <aplattner@nvidia.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
2020-01-03 08:39:21 -08:00
..
common modesetting: Use EGL_MESA_query_driver to select DRI driver if possible 2019-11-26 01:36:10 -08:00
ddc Add a Meson build system alongside autotools. 2017-04-26 15:25:27 -07:00
dixmods Fix NO_UNDEFINED build with statically linked fb 2019-08-04 10:28:51 -07:00
doc xfree86: Remove -flippixels 2018-09-27 16:50:22 +00:00
dri hw/xfree86: unvalidated lengths 2017-10-10 23:33:44 +02:00
dri2 dri2: Set fallback driver names for Intel and AMD chips 2019-09-30 12:57:22 -04:00
drivers modesetting: Check whether RandR was initialized before calling rrGetScrPriv 2020-01-03 08:39:21 -08:00
exa meson: Install man pages 2018-03-27 10:28:33 -04:00
fbdevhw xfree86: ensure the readlink buffer is null-terminated 2018-10-17 09:41:15 +10:00
glamor_egl glamor: Make Xv extension initialize at depth 30. 2018-02-27 10:18:06 -05:00
i2c Add a Meson build system alongside autotools. 2017-04-26 15:25:27 -07:00
int10 xfree86: Merge vbe into int10 2019-10-02 10:03:26 -04:00
loader loader: Make LoaderSymbolFromModule take a ModuleDescPtr 2019-11-21 14:20:50 -05:00
man xf86: autobind GPUs to the screen 2019-08-07 12:26:59 +02:00
modes xfree86/modes: Call xf86RotateRedisplay from xf86CrtcRotate 2019-11-25 18:46:31 +01:00
os-support xfree86: Test presence of isastream() 2019-11-22 15:22:43 -05:00
parser Fix various spelling errors 2019-10-01 17:05:28 +00:00
ramdac ramdac: Remove core ramdac code 2019-01-30 21:08:48 +00:00
shadowfb meson: Fix linkage of loadable modules for PE/COFF 2017-10-12 15:22:04 -04:00
utils gtf: Warning fix 2018-04-05 14:18:16 -04:00
vgahw xfree86: Remove -flippixels 2018-09-27 16:50:22 +00:00
x86emu Fix various spelling errors 2019-10-01 17:05:28 +00:00
xkb Move statically linked xorgxkb files from dixmods to a separate directory 2017-06-27 12:42:06 -04:00
.gitignore Xorg: Add Xorg.wrap to hw/xfree86/.gitignore 2014-04-18 11:39:16 +02:00
Makefile.am xfree86: Merge vbe into int10 2019-10-02 10:03:26 -04:00
Xorg.sh.in xfree86: rename Xorg.bin to Xorg 2015-01-05 09:53:58 +10:00
meson.build xfree86: Merge vbe into int10 2019-10-02 10:03:26 -04:00
sdksyms.sh include: Remove now-empty site.h 2019-10-30 16:17:04 +00:00
xorg-wrapper.c Fix various spelling errors 2019-10-01 17:05:28 +00:00
xorgconf.cpp xfree86: Remove NoTrapSignals 2018-09-12 20:47:15 +00:00