redonkable/alistair23-linux
redonkable/alistair23-linux
1
0
Fork 0
Code Releases Activity

vmwgfx: fix incorrect VRAM size check in vmw_kms_fb_create()

Browse source 
Commit e133e737 didn't correctly fix the integer overflow issue.

-	unsigned int required_size;
+	u64 required_size;
	...
	required_size = mode_cmd->pitch * mode_cmd->height;
-	if (unlikely(required_size > dev_priv->vram_size)) {
+	if (unlikely(required_size > (u64) dev_priv->vram_size)) {

Note that both pitch and height are u32.  Their product is still u32 and
would overflow before being assigned to required_size.  A correct way is
to convert pitch and height to u64 before the multiplication.

	required_size = (u64)mode_cmd->pitch * (u64)mode_cmd->height;

This patch calls the existing vmw_kms_validate_mode_vram() for
validation.

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Reviewed-and-tested-by: Thomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
This commit is contained in:
Xi Wang 2011-12-21 05:18:33 -05:00 committed by Dave Airlie
parent 77e00f2ea9
commit 8a78389651
1 changed files with 3 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
View file

@ -1093,7 +1093,6 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
struct vmw_surface *surface = NULL; struct vmw_surface *surface = NULL;
struct vmw_dma_buffer *bo = NULL; struct vmw_dma_buffer *bo = NULL;
struct ttm_base_object *user_obj; struct ttm_base_object *user_obj;
u64 required_size;
int ret; int ret;
/** /**
@ -1102,8 +1101,9 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
* requested framebuffer. * requested framebuffer.
*/ */
required_size = mode_cmd->pitch * mode_cmd->height; if (!vmw_kms_validate_mode_vram(dev_priv,
if (unlikely(required_size > (u64) dev_priv->vram_size)) { mode_cmd->pitch,
mode_cmd->height)) {
DRM_ERROR("VRAM size is too small for requested mode.\n"); DRM_ERROR("VRAM size is too small for requested mode.\n");
return ERR_PTR(-ENOMEM); return ERR_PTR(-ENOMEM);
} }