frederik
/
xserver
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

xwayland/shm: Avoid integer overflow on large pixmaps 

Xwayland's xwl_shm_create_pixmap() computes the size of the shared
memory pool to create using a size_t, yet the Wayland protocol uses an
integer for that size.

If the pool size becomes larger than INT32_MAX, we end up asking Wayland
to create a shared memory pool of negative size which in turn will raise
a protocol error which terminates the Wayland connection, and therefore
Xwayland.

Avoid that issue early by return a NULL pixmap in that case, which will
trigger a BadAlloc error, but leave Xwayland alive.

Signed-off-by: Olivier Fourdan <ofourdan@redhat.com>
Reviewed-by: Jonas Ådahl <jadahl@gmail.com>
This commit is contained in:
Olivier Fourdan 2021-09-16 10:18:03 +02:00 committed by Olivier Fourdan
parent 545fa90cbf
commit 079c5ccbcd
1 changed files with 9 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
hw/xwayland/xwayland-shm.c
View File

@ -234,6 +234,15 @@ xwl_shm_create_pixmap(ScreenPtr screen,
(width == 0 && height == 0) || depth < 15) (width == 0 && height == 0) || depth < 15)
return fbCreatePixmap(screen, width, height, depth, hint); return fbCreatePixmap(screen, width, height, depth, hint);
stride = PixmapBytePad(width, depth);
size = stride * height;
/* Size in the protocol is an integer, make sure we don't exceed
* INT32_MAX or else the Wayland compositor will raise an error and
* kill the Wayland connection!
*/
if (size > INT32_MAX)
return NULL;
pixmap = fbCreatePixmap(screen, 0, 0, depth, hint); pixmap = fbCreatePixmap(screen, 0, 0, depth, hint);
if (!pixmap) if (!pixmap)
return NULL; return NULL;
@ -242,8 +251,6 @@ xwl_shm_create_pixmap(ScreenPtr screen,
if (xwl_pixmap == NULL) if (xwl_pixmap == NULL)
goto err_destroy_pixmap; goto err_destroy_pixmap;
stride = PixmapBytePad(width, depth);
size = stride * height;
xwl_pixmap->buffer = NULL; xwl_pixmap->buffer = NULL;
xwl_pixmap->size = size; xwl_pixmap->size = size;
fd = os_create_anonymous_file(size); fd = os_create_anonymous_file(size);