ieee1394: dv1394: fix possible deadlock in multithreaded clients

Fix a possible though highly unlikely deadlock:

Thread A:                  Thread B:
 - acquire mmap_sem         - dv1394_ioctl/read/write()
 - dv1394_mmap()            - acquire video->mtx
 - acquire video->mtx       - copy_to/from_user(), possible page fault:
                              acquire mmap_sem

The simplest fix is to use mutex_trylock() instead of mutex_lock() in
dv1394_mmap().  This changes the behavior under contention in a way
which is visible to userspace clients.  However, my guess is that no
clients exist which use mmap vs. ioctl/read/write on the dv1394
character device file interface in concurrent threads.

Reported-by: Johannes Weiner <hannes@saeurebad.de>
Signed-off-by: Stefan Richter <stefanr@s5r6.in-berlin.de>
This commit is contained in:
Stefan Richter 2008-10-26 12:02:03 +01:00
parent 638570b543
commit 8449fc3ae5

View file

@ -1270,8 +1270,14 @@ static int dv1394_mmap(struct file *file, struct vm_area_struct *vma)
struct video_card *video = file_to_video_card(file);
int retval = -EINVAL;
/* serialize mmap */
mutex_lock(&video->mtx);
/*
* We cannot use the blocking variant mutex_lock here because .mmap
* is called with mmap_sem held, while .ioctl, .read, .write acquire
* video->mtx and subsequently call copy_to/from_user which will
* grab mmap_sem in case of a page fault.
*/
if (!mutex_trylock(&video->mtx))
return -EAGAIN;
if ( ! video_card_initialized(video) ) {
retval = do_dv1394_init_default(video);