os: Do not overflow the integer size with BigRequest

The BigRequest extension allows requests larger than the 16-bit length
limit.

It uses integers for the request length and checks for the size not to
exceed the maxBigRequestSize limit, but does so after translating the
length to integer by multiplying the given size in bytes by 4.

In doing so, it might overflow the integer size limit before actually
checking for the overflow, defeating the purpose of the test.

To avoid the issue, make sure to check that the request size does not
overflow the maxBigRequestSize limit prior to any conversion.

The caller Dispatch() function however expects the return value to be in
bytes, so we cannot just return the converted value in case of error, as
that would also overflow the integer size.

To preserve the existing API, we use a negative value for the X11 error
code BadLength as the function only return positive values, 0 or -1 and
update the caller Dispatch() function to take that case into account to
return the error code to the offending client.

CVE-2025-49176

This issue was discovered by Nils Emmerich <nemmerich@ernw.de> and
reported by Julian Suleder via ERNW Vulnerability Disclosure.

Signed-off-by: Olivier Fourdan <ofourdan@redhat.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/xorg/xserver/-/merge_requests/2024>

dix/dispatch.c

@@ -519,9 +519,10 @@ Dispatch(void)
 
                 /* now, finally, deal with client requests */
                 result = ReadRequestFromClient(client);
-                if (result <= 0) {
+                if (result == 0)
-                    if (result < 0)
+                    break;
-                        CloseDownClient(client);
+                else if (result == -1) {
+                    CloseDownClient(client);
                     break;
                 }
 
@@ -542,7 +543,7 @@ Dispatch(void)
                                           client->index,
                                           client->requestBuffer);
 #endif
-                if (result > (maxBigRequestSize << 2))
+                if (result < 0 || result > (maxBigRequestSize << 2))
                     result = BadLength;
                 else {
                     result = XaceHookDispatch(client, client->majorOp);

os/io.c

@@ -301,6 +301,10 @@ ReadRequestFromClient(ClientPtr client)
                 needed = get_big_req_len(request, client);
         }
         client->req_len = needed;
+        if (needed > MAXINT >> 2) {
+            /* Check for potential integer overflow */
+            return -(BadLength);
+        }
         needed <<= 2;           /* needed is in bytes now */
     }
     if (gotnow < needed) {