2007-07-17 19:37:06 -06:00
|
|
|
/*
|
|
|
|
* blkfront.c
|
|
|
|
*
|
|
|
|
* XenLinux virtual block device driver.
|
|
|
|
*
|
|
|
|
* Copyright (c) 2003-2004, Keir Fraser & Steve Hand
|
|
|
|
* Modifications by Mark A. Williamson are (c) Intel Research Cambridge
|
|
|
|
* Copyright (c) 2004, Christian Limpach
|
|
|
|
* Copyright (c) 2004, Andrew Warfield
|
|
|
|
* Copyright (c) 2005, Christopher Clark
|
|
|
|
* Copyright (c) 2005, XenSource Ltd
|
|
|
|
*
|
|
|
|
* This program is free software; you can redistribute it and/or
|
|
|
|
* modify it under the terms of the GNU General Public License version 2
|
|
|
|
* as published by the Free Software Foundation; or, when distributed
|
|
|
|
* separately from the Linux kernel or incorporated into other
|
|
|
|
* software packages, subject to the following license:
|
|
|
|
*
|
|
|
|
* Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
|
|
* of this source file (the "Software"), to deal in the Software without
|
|
|
|
* restriction, including without limitation the rights to use, copy, modify,
|
|
|
|
* merge, publish, distribute, sublicense, and/or sell copies of the Software,
|
|
|
|
* and to permit persons to whom the Software is furnished to do so, subject to
|
|
|
|
* the following conditions:
|
|
|
|
*
|
|
|
|
* The above copyright notice and this permission notice shall be included in
|
|
|
|
* all copies or substantial portions of the Software.
|
|
|
|
*
|
|
|
|
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
|
|
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
|
|
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
|
|
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
|
|
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
|
|
|
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
|
|
|
|
* IN THE SOFTWARE.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/interrupt.h>
|
|
|
|
#include <linux/blkdev.h>
|
2015-07-13 03:55:24 -06:00
|
|
|
#include <linux/blk-mq.h>
|
2008-02-21 14:03:45 -07:00
|
|
|
#include <linux/hdreg.h>
|
2008-06-17 02:47:08 -06:00
|
|
|
#include <linux/cdrom.h>
|
2007-07-17 19:37:06 -06:00
|
|
|
#include <linux/module.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 02:04:11 -06:00
|
|
|
#include <linux/slab.h>
|
2010-06-02 06:28:52 -06:00
|
|
|
#include <linux/mutex.h>
|
2009-02-24 00:10:09 -07:00
|
|
|
#include <linux/scatterlist.h>
|
2012-01-20 08:15:26 -07:00
|
|
|
#include <linux/bitmap.h>
|
2013-03-18 10:49:34 -06:00
|
|
|
#include <linux/list.h>
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2009-10-06 16:11:14 -06:00
|
|
|
#include <xen/xen.h>
|
2007-07-17 19:37:06 -06:00
|
|
|
#include <xen/xenbus.h>
|
|
|
|
#include <xen/grant_table.h>
|
|
|
|
#include <xen/events.h>
|
|
|
|
#include <xen/page.h>
|
2010-05-14 05:44:30 -06:00
|
|
|
#include <xen/platform_pci.h>
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
#include <xen/interface/grant_table.h>
|
|
|
|
#include <xen/interface/io/blkif.h>
|
2008-04-02 11:54:02 -06:00
|
|
|
#include <xen/interface/io/protocols.h>
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
#include <asm/xen/hypervisor.h>
|
|
|
|
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
/*
|
|
|
|
* The minimal size of segment supported by the block framework is PAGE_SIZE.
|
|
|
|
* When Linux is using a different page size than Xen, it may not be possible
|
|
|
|
* to put all the data in a single segment.
|
|
|
|
* This can happen when the backend doesn't support indirect descriptor and
|
|
|
|
* therefore the maximum amount of data that a request can carry is
|
|
|
|
* BLKIF_MAX_SEGMENTS_PER_REQUEST * XEN_PAGE_SIZE = 44KB
|
|
|
|
*
|
|
|
|
* Note that we only support one extra request. So the Linux page size
|
|
|
|
* should be <= ( 2 * BLKIF_MAX_SEGMENTS_PER_REQUEST * XEN_PAGE_SIZE) =
|
|
|
|
* 88KB.
|
|
|
|
*/
|
|
|
|
#define HAS_EXTRA_REQ (BLKIF_MAX_SEGMENTS_PER_REQUEST < XEN_PFN_PER_PAGE)
|
|
|
|
|
2007-07-17 19:37:06 -06:00
|
|
|
enum blkif_state {
|
|
|
|
BLKIF_STATE_DISCONNECTED,
|
|
|
|
BLKIF_STATE_CONNECTED,
|
|
|
|
BLKIF_STATE_SUSPENDED,
|
|
|
|
};
|
|
|
|
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
struct grant {
|
|
|
|
grant_ref_t gref;
|
2015-06-30 04:58:51 -06:00
|
|
|
struct page *page;
|
2013-03-18 10:49:34 -06:00
|
|
|
struct list_head node;
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
};
|
|
|
|
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
enum blk_req_status {
|
|
|
|
REQ_WAITING,
|
|
|
|
REQ_DONE,
|
|
|
|
REQ_ERROR,
|
|
|
|
REQ_EOPNOTSUPP,
|
|
|
|
};
|
|
|
|
|
2007-07-17 19:37:06 -06:00
|
|
|
struct blk_shadow {
|
|
|
|
struct blkif_request req;
|
2010-11-01 15:03:14 -06:00
|
|
|
struct request *request;
|
2013-04-18 08:06:54 -06:00
|
|
|
struct grant **grants_used;
|
|
|
|
struct grant **indirect_grants;
|
2013-05-02 02:58:50 -06:00
|
|
|
struct scatterlist *sg;
|
2015-07-22 09:44:54 -06:00
|
|
|
unsigned int num_sg;
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
enum blk_req_status status;
|
|
|
|
|
|
|
|
#define NO_ASSOCIATED_ID ~0UL
|
|
|
|
/*
|
|
|
|
* Id of the sibling if we ever need 2 requests when handling a
|
|
|
|
* block I/O request
|
|
|
|
*/
|
|
|
|
unsigned long associated_id;
|
2013-04-18 08:06:54 -06:00
|
|
|
};
|
|
|
|
|
|
|
|
struct split_bio {
|
|
|
|
struct bio *bio;
|
|
|
|
atomic_t pending;
|
2007-07-17 19:37:06 -06:00
|
|
|
};
|
|
|
|
|
2010-06-02 06:28:52 -06:00
|
|
|
static DEFINE_MUTEX(blkfront_mutex);
|
2009-09-21 18:01:13 -06:00
|
|
|
static const struct block_device_operations xlvbd_block_fops;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2013-04-18 08:06:54 -06:00
|
|
|
/*
|
|
|
|
* Maximum number of segments in indirect requests, the actual value used by
|
|
|
|
* the frontend driver is the minimum of this value and the value provided
|
|
|
|
* by the backend driver.
|
|
|
|
*/
|
|
|
|
|
|
|
|
static unsigned int xen_blkif_max_segments = 32;
|
2016-02-10 04:21:15 -07:00
|
|
|
module_param_named(max_indirect_segments, xen_blkif_max_segments, uint,
|
|
|
|
S_IRUGO);
|
|
|
|
MODULE_PARM_DESC(max_indirect_segments,
|
|
|
|
"Maximum amount of segments in indirect requests (default is 32)");
|
2013-04-18 08:06:54 -06:00
|
|
|
|
2015-11-13 20:12:14 -07:00
|
|
|
static unsigned int xen_blkif_max_queues = 4;
|
|
|
|
module_param_named(max_queues, xen_blkif_max_queues, uint, S_IRUGO);
|
|
|
|
MODULE_PARM_DESC(max_queues, "Maximum number of hardware queues/rings used per virtual disk");
|
|
|
|
|
2015-06-02 23:40:03 -06:00
|
|
|
/*
|
|
|
|
* Maximum order of pages to be used for the shared ring between front and
|
|
|
|
* backend, 4KB page granularity is used.
|
|
|
|
*/
|
|
|
|
static unsigned int xen_blkif_max_ring_order;
|
|
|
|
module_param_named(max_ring_page_order, xen_blkif_max_ring_order, int, S_IRUGO);
|
|
|
|
MODULE_PARM_DESC(max_ring_page_order, "Maximum order of pages to be used for the shared ring");
|
|
|
|
|
2015-07-22 09:44:54 -06:00
|
|
|
#define BLK_RING_SIZE(info) \
|
|
|
|
__CONST_RING_SIZE(blkif, XEN_PAGE_SIZE * (info)->nr_ring_pages)
|
|
|
|
|
|
|
|
#define BLK_MAX_RING_SIZE \
|
2015-10-13 10:50:11 -06:00
|
|
|
__CONST_RING_SIZE(blkif, XEN_PAGE_SIZE * XENBUS_MAX_RING_GRANTS)
|
2015-07-22 09:44:54 -06:00
|
|
|
|
2015-06-02 23:40:03 -06:00
|
|
|
/*
|
2015-11-16 13:14:41 -07:00
|
|
|
* ring-ref%u i=(-1UL) would take 11 characters + 'ring-ref' is 8, so 19
|
|
|
|
* characters are enough. Define to 20 to keep consistent with backend.
|
2015-06-02 23:40:03 -06:00
|
|
|
*/
|
|
|
|
#define RINGREF_NAME_LEN (20)
|
2015-11-13 20:12:14 -07:00
|
|
|
/*
|
|
|
|
* queue-%u would take 7 + 10(UINT_MAX) = 17 characters.
|
|
|
|
*/
|
|
|
|
#define QUEUE_NAME_LEN (17)
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
/*
|
|
|
|
* Per-ring info.
|
|
|
|
* Every blkfront device can associate with one or more blkfront_ring_info,
|
|
|
|
* depending on how many hardware queues/rings to be used.
|
|
|
|
*/
|
|
|
|
struct blkfront_ring_info {
|
2015-11-13 20:12:13 -07:00
|
|
|
/* Lock to protect data in every ring buffer. */
|
|
|
|
spinlock_t ring_lock;
|
2015-11-13 20:12:11 -07:00
|
|
|
struct blkif_front_ring ring;
|
|
|
|
unsigned int ring_ref[XENBUS_MAX_RING_GRANTS];
|
|
|
|
unsigned int evtchn, irq;
|
|
|
|
struct work_struct work;
|
|
|
|
struct gnttab_free_callback callback;
|
|
|
|
struct blk_shadow shadow[BLK_MAX_RING_SIZE];
|
|
|
|
struct list_head indirect_pages;
|
2015-11-16 14:51:39 -07:00
|
|
|
struct list_head grants;
|
|
|
|
unsigned int persistent_gnts_c;
|
2015-11-13 20:12:11 -07:00
|
|
|
unsigned long shadow_free;
|
|
|
|
struct blkfront_info *dev_info;
|
|
|
|
};
|
|
|
|
|
2007-07-17 19:37:06 -06:00
|
|
|
/*
|
|
|
|
* We have one of these per vbd, whether ide, scsi or 'other'. They
|
|
|
|
* hang in private_data off the gendisk structure. We may end up
|
|
|
|
* putting all kinds of interesting stuff here :-)
|
|
|
|
*/
|
|
|
|
struct blkfront_info
|
|
|
|
{
|
2010-04-30 16:01:19 -06:00
|
|
|
struct mutex mutex;
|
2007-07-17 19:37:06 -06:00
|
|
|
struct xenbus_device *xbdev;
|
|
|
|
struct gendisk *gd;
|
2016-07-01 15:43:39 -06:00
|
|
|
u16 sector_size;
|
|
|
|
unsigned int physical_sector_size;
|
2007-07-17 19:37:06 -06:00
|
|
|
int vdevice;
|
|
|
|
blkif_vdev_t handle;
|
|
|
|
enum blkif_state connected;
|
2015-11-13 20:12:12 -07:00
|
|
|
/* Number of pages per ring buffer. */
|
2015-06-02 23:40:03 -06:00
|
|
|
unsigned int nr_ring_pages;
|
2007-07-17 19:37:06 -06:00
|
|
|
struct request_queue *rq;
|
2010-09-03 03:56:16 -06:00
|
|
|
unsigned int feature_flush;
|
2016-06-05 13:32:24 -06:00
|
|
|
unsigned int feature_fua;
|
2011-10-12 14:23:30 -06:00
|
|
|
unsigned int feature_discard:1;
|
|
|
|
unsigned int feature_secdiscard:1;
|
2011-09-01 04:39:09 -06:00
|
|
|
unsigned int discard_granularity;
|
|
|
|
unsigned int discard_alignment;
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
unsigned int feature_persistent:1;
|
2015-07-22 09:44:54 -06:00
|
|
|
/* Number of 4KB segments handled */
|
2013-04-18 08:06:54 -06:00
|
|
|
unsigned int max_indirect_segments;
|
2008-04-02 11:54:04 -06:00
|
|
|
int is_ready;
|
2015-07-13 03:55:24 -06:00
|
|
|
struct blk_mq_tag_set tag_set;
|
2015-11-13 20:12:12 -07:00
|
|
|
struct blkfront_ring_info *rinfo;
|
|
|
|
unsigned int nr_rings;
|
2016-06-27 02:33:24 -06:00
|
|
|
/* Save uncomplete reqs and bios for migration. */
|
|
|
|
struct list_head requests;
|
|
|
|
struct bio_list bio_list;
|
2007-07-17 19:37:06 -06:00
|
|
|
};
|
|
|
|
|
2010-08-07 10:28:55 -06:00
|
|
|
static unsigned int nr_minors;
|
|
|
|
static unsigned long *minors;
|
|
|
|
static DEFINE_SPINLOCK(minor_lock);
|
|
|
|
|
2007-07-17 19:37:06 -06:00
|
|
|
#define GRANT_INVALID_REF 0
|
|
|
|
|
|
|
|
#define PARTS_PER_DISK 16
|
2008-09-17 15:30:32 -06:00
|
|
|
#define PARTS_PER_EXT_DISK 256
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
#define BLKIF_MAJOR(dev) ((dev)>>8)
|
|
|
|
#define BLKIF_MINOR(dev) ((dev) & 0xff)
|
|
|
|
|
2008-09-17 15:30:32 -06:00
|
|
|
#define EXT_SHIFT 28
|
|
|
|
#define EXTENDED (1<<EXT_SHIFT)
|
|
|
|
#define VDEV_IS_EXTENDED(dev) ((dev)&(EXTENDED))
|
|
|
|
#define BLKIF_MINOR_EXT(dev) ((dev)&(~EXTENDED))
|
2010-12-02 10:55:00 -07:00
|
|
|
#define EMULATED_HD_DISK_MINOR_OFFSET (0)
|
|
|
|
#define EMULATED_HD_DISK_NAME_OFFSET (EMULATED_HD_DISK_MINOR_OFFSET / 256)
|
2011-07-14 07:30:22 -06:00
|
|
|
#define EMULATED_SD_DISK_MINOR_OFFSET (0)
|
|
|
|
#define EMULATED_SD_DISK_NAME_OFFSET (EMULATED_SD_DISK_MINOR_OFFSET / 256)
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2008-09-17 15:30:32 -06:00
|
|
|
#define DEV_NAME "xvd" /* name in /dev */
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-07-22 09:44:54 -06:00
|
|
|
/*
|
|
|
|
* Grants are always the same size as a Xen page (i.e 4KB).
|
|
|
|
* A physical segment is always the same size as a Linux page.
|
|
|
|
* Number of grants per physical segment
|
|
|
|
*/
|
|
|
|
#define GRANTS_PER_PSEG (PAGE_SIZE / XEN_PAGE_SIZE)
|
|
|
|
|
|
|
|
#define GRANTS_PER_INDIRECT_FRAME \
|
|
|
|
(XEN_PAGE_SIZE / sizeof(struct blkif_request_segment))
|
|
|
|
|
|
|
|
#define PSEGS_PER_INDIRECT_FRAME \
|
|
|
|
(GRANTS_INDIRECT_FRAME / GRANTS_PSEGS)
|
|
|
|
|
|
|
|
#define INDIRECT_GREFS(_grants) \
|
|
|
|
DIV_ROUND_UP(_grants, GRANTS_PER_INDIRECT_FRAME)
|
|
|
|
|
|
|
|
#define GREFS(_psegs) ((_psegs) * GRANTS_PER_PSEG)
|
2013-04-18 08:06:54 -06:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
static int blkfront_setup_indirect(struct blkfront_ring_info *rinfo);
|
2015-11-13 20:12:12 -07:00
|
|
|
static void blkfront_gather_backend_features(struct blkfront_info *info);
|
2013-04-18 08:06:54 -06:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
static int get_id_from_freelist(struct blkfront_ring_info *rinfo)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
2015-11-13 20:12:11 -07:00
|
|
|
unsigned long free = rinfo->shadow_free;
|
|
|
|
|
|
|
|
BUG_ON(free >= BLK_RING_SIZE(rinfo->dev_info));
|
|
|
|
rinfo->shadow_free = rinfo->shadow[free].req.u.rw.id;
|
|
|
|
rinfo->shadow[free].req.u.rw.id = 0x0fffffee; /* debug */
|
2007-07-17 19:37:06 -06:00
|
|
|
return free;
|
|
|
|
}
|
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
static int add_id_to_freelist(struct blkfront_ring_info *rinfo,
|
2015-11-16 13:14:41 -07:00
|
|
|
unsigned long id)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
2015-11-13 20:12:11 -07:00
|
|
|
if (rinfo->shadow[id].req.u.rw.id != id)
|
2012-05-25 15:34:51 -06:00
|
|
|
return -EINVAL;
|
2015-11-13 20:12:11 -07:00
|
|
|
if (rinfo->shadow[id].request == NULL)
|
2012-05-25 15:34:51 -06:00
|
|
|
return -EINVAL;
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo->shadow[id].req.u.rw.id = rinfo->shadow_free;
|
|
|
|
rinfo->shadow[id].request = NULL;
|
|
|
|
rinfo->shadow_free = id;
|
2012-05-25 15:34:51 -06:00
|
|
|
return 0;
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
static int fill_grant_buffer(struct blkfront_ring_info *rinfo, int num)
|
2013-03-18 10:49:35 -06:00
|
|
|
{
|
2015-11-13 20:12:11 -07:00
|
|
|
struct blkfront_info *info = rinfo->dev_info;
|
2013-03-18 10:49:35 -06:00
|
|
|
struct page *granted_page;
|
|
|
|
struct grant *gnt_list_entry, *n;
|
|
|
|
int i = 0;
|
|
|
|
|
2015-11-16 13:14:41 -07:00
|
|
|
while (i < num) {
|
2013-03-18 10:49:35 -06:00
|
|
|
gnt_list_entry = kzalloc(sizeof(struct grant), GFP_NOIO);
|
|
|
|
if (!gnt_list_entry)
|
|
|
|
goto out_of_memory;
|
|
|
|
|
2013-10-29 11:31:14 -06:00
|
|
|
if (info->feature_persistent) {
|
|
|
|
granted_page = alloc_page(GFP_NOIO);
|
|
|
|
if (!granted_page) {
|
|
|
|
kfree(gnt_list_entry);
|
|
|
|
goto out_of_memory;
|
|
|
|
}
|
2015-06-30 04:58:51 -06:00
|
|
|
gnt_list_entry->page = granted_page;
|
2013-03-18 10:49:35 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
gnt_list_entry->gref = GRANT_INVALID_REF;
|
2015-11-16 14:51:39 -07:00
|
|
|
list_add(&gnt_list_entry->node, &rinfo->grants);
|
2013-03-18 10:49:35 -06:00
|
|
|
i++;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
out_of_memory:
|
|
|
|
list_for_each_entry_safe(gnt_list_entry, n,
|
2015-11-16 14:51:39 -07:00
|
|
|
&rinfo->grants, node) {
|
2013-03-18 10:49:35 -06:00
|
|
|
list_del(&gnt_list_entry->node);
|
2013-10-29 11:31:14 -06:00
|
|
|
if (info->feature_persistent)
|
2015-06-30 04:58:51 -06:00
|
|
|
__free_page(gnt_list_entry->page);
|
2013-03-18 10:49:35 -06:00
|
|
|
kfree(gnt_list_entry);
|
|
|
|
i--;
|
|
|
|
}
|
|
|
|
BUG_ON(i != 0);
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
|
2015-11-16 14:51:39 -07:00
|
|
|
static struct grant *get_free_grant(struct blkfront_ring_info *rinfo)
|
2013-03-18 10:49:35 -06:00
|
|
|
{
|
|
|
|
struct grant *gnt_list_entry;
|
|
|
|
|
2015-11-16 14:51:39 -07:00
|
|
|
BUG_ON(list_empty(&rinfo->grants));
|
|
|
|
gnt_list_entry = list_first_entry(&rinfo->grants, struct grant,
|
2015-07-01 07:10:38 -06:00
|
|
|
node);
|
2013-03-18 10:49:35 -06:00
|
|
|
list_del(&gnt_list_entry->node);
|
|
|
|
|
2015-07-01 07:10:38 -06:00
|
|
|
if (gnt_list_entry->gref != GRANT_INVALID_REF)
|
2015-11-16 14:51:39 -07:00
|
|
|
rinfo->persistent_gnts_c--;
|
2015-07-01 07:10:38 -06:00
|
|
|
|
|
|
|
return gnt_list_entry;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void grant_foreign_access(const struct grant *gnt_list_entry,
|
|
|
|
const struct blkfront_info *info)
|
|
|
|
{
|
|
|
|
gnttab_page_grant_foreign_access_ref_one(gnt_list_entry->gref,
|
|
|
|
info->xbdev->otherend_id,
|
|
|
|
gnt_list_entry->page,
|
|
|
|
0);
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct grant *get_grant(grant_ref_t *gref_head,
|
|
|
|
unsigned long gfn,
|
2015-11-16 14:51:39 -07:00
|
|
|
struct blkfront_ring_info *rinfo)
|
2015-07-01 07:10:38 -06:00
|
|
|
{
|
2015-11-16 14:51:39 -07:00
|
|
|
struct grant *gnt_list_entry = get_free_grant(rinfo);
|
|
|
|
struct blkfront_info *info = rinfo->dev_info;
|
2015-07-01 07:10:38 -06:00
|
|
|
|
|
|
|
if (gnt_list_entry->gref != GRANT_INVALID_REF)
|
2013-03-18 10:49:35 -06:00
|
|
|
return gnt_list_entry;
|
2015-07-01 07:10:38 -06:00
|
|
|
|
|
|
|
/* Assign a gref to this page */
|
|
|
|
gnt_list_entry->gref = gnttab_claim_grant_reference(gref_head);
|
|
|
|
BUG_ON(gnt_list_entry->gref == -ENOSPC);
|
|
|
|
if (info->feature_persistent)
|
|
|
|
grant_foreign_access(gnt_list_entry, info);
|
|
|
|
else {
|
|
|
|
/* Grant access to the GFN passed by the caller */
|
|
|
|
gnttab_grant_foreign_access_ref(gnt_list_entry->gref,
|
|
|
|
info->xbdev->otherend_id,
|
|
|
|
gfn, 0);
|
2013-03-18 10:49:35 -06:00
|
|
|
}
|
|
|
|
|
2015-07-01 07:10:38 -06:00
|
|
|
return gnt_list_entry;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct grant *get_indirect_grant(grant_ref_t *gref_head,
|
2015-11-16 14:51:39 -07:00
|
|
|
struct blkfront_ring_info *rinfo)
|
2015-07-01 07:10:38 -06:00
|
|
|
{
|
2015-11-16 14:51:39 -07:00
|
|
|
struct grant *gnt_list_entry = get_free_grant(rinfo);
|
|
|
|
struct blkfront_info *info = rinfo->dev_info;
|
2015-07-01 07:10:38 -06:00
|
|
|
|
|
|
|
if (gnt_list_entry->gref != GRANT_INVALID_REF)
|
|
|
|
return gnt_list_entry;
|
|
|
|
|
2013-03-18 10:49:35 -06:00
|
|
|
/* Assign a gref to this page */
|
|
|
|
gnt_list_entry->gref = gnttab_claim_grant_reference(gref_head);
|
|
|
|
BUG_ON(gnt_list_entry->gref == -ENOSPC);
|
2013-10-29 11:31:14 -06:00
|
|
|
if (!info->feature_persistent) {
|
2015-07-01 07:10:38 -06:00
|
|
|
struct page *indirect_page;
|
|
|
|
|
|
|
|
/* Fetch a pre-allocated page to use for indirect grefs */
|
2015-11-16 14:51:39 -07:00
|
|
|
BUG_ON(list_empty(&rinfo->indirect_pages));
|
|
|
|
indirect_page = list_first_entry(&rinfo->indirect_pages,
|
2015-07-01 07:10:38 -06:00
|
|
|
struct page, lru);
|
|
|
|
list_del(&indirect_page->lru);
|
|
|
|
gnt_list_entry->page = indirect_page;
|
2013-10-29 11:31:14 -06:00
|
|
|
}
|
2015-07-01 07:10:38 -06:00
|
|
|
grant_foreign_access(gnt_list_entry, info);
|
|
|
|
|
2013-03-18 10:49:35 -06:00
|
|
|
return gnt_list_entry;
|
|
|
|
}
|
|
|
|
|
2012-05-25 15:34:51 -06:00
|
|
|
static const char *op_name(int op)
|
|
|
|
{
|
|
|
|
static const char *const names[] = {
|
|
|
|
[BLKIF_OP_READ] = "read",
|
|
|
|
[BLKIF_OP_WRITE] = "write",
|
|
|
|
[BLKIF_OP_WRITE_BARRIER] = "barrier",
|
|
|
|
[BLKIF_OP_FLUSH_DISKCACHE] = "flush",
|
|
|
|
[BLKIF_OP_DISCARD] = "discard" };
|
|
|
|
|
|
|
|
if (op < 0 || op >= ARRAY_SIZE(names))
|
|
|
|
return "unknown";
|
|
|
|
|
|
|
|
if (!names[op])
|
|
|
|
return "reserved";
|
|
|
|
|
|
|
|
return names[op];
|
|
|
|
}
|
2010-08-07 10:28:55 -06:00
|
|
|
static int xlbd_reserve_minors(unsigned int minor, unsigned int nr)
|
|
|
|
{
|
|
|
|
unsigned int end = minor + nr;
|
|
|
|
int rc;
|
|
|
|
|
|
|
|
if (end > nr_minors) {
|
|
|
|
unsigned long *bitmap, *old;
|
|
|
|
|
2011-11-29 14:08:00 -07:00
|
|
|
bitmap = kcalloc(BITS_TO_LONGS(end), sizeof(*bitmap),
|
2010-08-07 10:28:55 -06:00
|
|
|
GFP_KERNEL);
|
|
|
|
if (bitmap == NULL)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
spin_lock(&minor_lock);
|
|
|
|
if (end > nr_minors) {
|
|
|
|
old = minors;
|
|
|
|
memcpy(bitmap, minors,
|
|
|
|
BITS_TO_LONGS(nr_minors) * sizeof(*bitmap));
|
|
|
|
minors = bitmap;
|
|
|
|
nr_minors = BITS_TO_LONGS(end) * BITS_PER_LONG;
|
|
|
|
} else
|
|
|
|
old = bitmap;
|
|
|
|
spin_unlock(&minor_lock);
|
|
|
|
kfree(old);
|
|
|
|
}
|
|
|
|
|
|
|
|
spin_lock(&minor_lock);
|
|
|
|
if (find_next_bit(minors, end, minor) >= end) {
|
2012-01-20 08:15:26 -07:00
|
|
|
bitmap_set(minors, minor, nr);
|
2010-08-07 10:28:55 -06:00
|
|
|
rc = 0;
|
|
|
|
} else
|
|
|
|
rc = -EBUSY;
|
|
|
|
spin_unlock(&minor_lock);
|
|
|
|
|
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void xlbd_release_minors(unsigned int minor, unsigned int nr)
|
|
|
|
{
|
|
|
|
unsigned int end = minor + nr;
|
|
|
|
|
|
|
|
BUG_ON(end > nr_minors);
|
|
|
|
spin_lock(&minor_lock);
|
2012-01-20 08:15:26 -07:00
|
|
|
bitmap_clear(minors, minor, nr);
|
2010-08-07 10:28:55 -06:00
|
|
|
spin_unlock(&minor_lock);
|
|
|
|
}
|
|
|
|
|
2007-07-17 19:37:06 -06:00
|
|
|
static void blkif_restart_queue_callback(void *arg)
|
|
|
|
{
|
2015-11-13 20:12:11 -07:00
|
|
|
struct blkfront_ring_info *rinfo = (struct blkfront_ring_info *)arg;
|
|
|
|
schedule_work(&rinfo->work);
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
|
2008-04-29 01:59:47 -06:00
|
|
|
static int blkif_getgeo(struct block_device *bd, struct hd_geometry *hg)
|
2008-02-21 14:03:45 -07:00
|
|
|
{
|
|
|
|
/* We don't have real geometry info, but let's at least return
|
|
|
|
values consistent with the size of the device */
|
|
|
|
sector_t nsect = get_capacity(bd->bd_disk);
|
|
|
|
sector_t cylinders = nsect;
|
|
|
|
|
|
|
|
hg->heads = 0xff;
|
|
|
|
hg->sectors = 0x3f;
|
|
|
|
sector_div(cylinders, hg->heads * hg->sectors);
|
|
|
|
hg->cylinders = cylinders;
|
|
|
|
if ((sector_t)(hg->cylinders + 1) * hg->heads * hg->sectors < nsect)
|
|
|
|
hg->cylinders = 0xffff;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2008-03-02 08:23:47 -07:00
|
|
|
static int blkif_ioctl(struct block_device *bdev, fmode_t mode,
|
2008-08-04 03:59:05 -06:00
|
|
|
unsigned command, unsigned long argument)
|
2008-06-17 02:47:08 -06:00
|
|
|
{
|
2008-03-02 08:23:47 -07:00
|
|
|
struct blkfront_info *info = bdev->bd_disk->private_data;
|
2008-06-17 02:47:08 -06:00
|
|
|
int i;
|
|
|
|
|
|
|
|
dev_dbg(&info->xbdev->dev, "command: 0x%x, argument: 0x%lx\n",
|
|
|
|
command, (long)argument);
|
|
|
|
|
|
|
|
switch (command) {
|
|
|
|
case CDROMMULTISESSION:
|
|
|
|
dev_dbg(&info->xbdev->dev, "FIXME: support multisession CDs later\n");
|
|
|
|
for (i = 0; i < sizeof(struct cdrom_multisession); i++)
|
|
|
|
if (put_user(0, (char __user *)(argument + i)))
|
|
|
|
return -EFAULT;
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
case CDROM_GET_CAPABILITY: {
|
|
|
|
struct gendisk *gd = info->gd;
|
|
|
|
if (gd->flags & GENHD_FL_CD)
|
|
|
|
return 0;
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
|
|
|
default:
|
|
|
|
/*printk(KERN_ALERT "ioctl %08x not supported by Xen blkdev\n",
|
|
|
|
command);*/
|
|
|
|
return -EINVAL; /* same return as native Linux */
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2015-08-13 12:23:10 -06:00
|
|
|
static unsigned long blkif_ring_get_request(struct blkfront_ring_info *rinfo,
|
|
|
|
struct request *req,
|
|
|
|
struct blkif_request **ring_req)
|
|
|
|
{
|
|
|
|
unsigned long id;
|
|
|
|
|
|
|
|
*ring_req = RING_GET_REQUEST(&rinfo->ring, rinfo->ring.req_prod_pvt);
|
|
|
|
rinfo->ring.req_prod_pvt++;
|
|
|
|
|
|
|
|
id = get_id_from_freelist(rinfo);
|
|
|
|
rinfo->shadow[id].request = req;
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
rinfo->shadow[id].status = REQ_WAITING;
|
|
|
|
rinfo->shadow[id].associated_id = NO_ASSOCIATED_ID;
|
2015-08-13 12:23:10 -06:00
|
|
|
|
|
|
|
(*ring_req)->u.rw.id = id;
|
|
|
|
|
|
|
|
return id;
|
|
|
|
}
|
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
static int blkif_queue_discard_req(struct request *req, struct blkfront_ring_info *rinfo)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
2015-11-13 20:12:11 -07:00
|
|
|
struct blkfront_info *info = rinfo->dev_info;
|
2007-07-17 19:37:06 -06:00
|
|
|
struct blkif_request *ring_req;
|
|
|
|
unsigned long id;
|
2015-06-29 10:35:24 -06:00
|
|
|
|
|
|
|
/* Fill out a communications ring structure. */
|
2015-08-13 12:23:10 -06:00
|
|
|
id = blkif_ring_get_request(rinfo, req, &ring_req);
|
2015-06-29 10:35:24 -06:00
|
|
|
|
|
|
|
ring_req->operation = BLKIF_OP_DISCARD;
|
|
|
|
ring_req->u.discard.nr_sectors = blk_rq_sectors(req);
|
|
|
|
ring_req->u.discard.id = id;
|
|
|
|
ring_req->u.discard.sector_number = (blkif_sector_t)blk_rq_pos(req);
|
2016-06-09 08:00:36 -06:00
|
|
|
if (req_op(req) == REQ_OP_SECURE_ERASE && info->feature_secdiscard)
|
2015-06-29 10:35:24 -06:00
|
|
|
ring_req->u.discard.flag = BLKIF_DISCARD_SECURE;
|
|
|
|
else
|
|
|
|
ring_req->u.discard.flag = 0;
|
|
|
|
|
|
|
|
/* Keep a private copy so we can reissue requests when recovering. */
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo->shadow[id].req = *ring_req;
|
2015-06-29 10:35:24 -06:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2015-07-22 09:44:54 -06:00
|
|
|
struct setup_rw_req {
|
|
|
|
unsigned int grant_idx;
|
|
|
|
struct blkif_request_segment *segments;
|
2015-11-13 20:12:11 -07:00
|
|
|
struct blkfront_ring_info *rinfo;
|
2015-07-22 09:44:54 -06:00
|
|
|
struct blkif_request *ring_req;
|
|
|
|
grant_ref_t gref_head;
|
|
|
|
unsigned int id;
|
|
|
|
/* Only used when persistent grant is used and it's a read request */
|
|
|
|
bool need_copy;
|
|
|
|
unsigned int bvec_off;
|
|
|
|
char *bvec_data;
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
|
|
|
|
bool require_extra_req;
|
|
|
|
struct blkif_request *extra_ring_req;
|
2015-07-22 09:44:54 -06:00
|
|
|
};
|
|
|
|
|
|
|
|
static void blkif_setup_rw_req_grant(unsigned long gfn, unsigned int offset,
|
|
|
|
unsigned int len, void *data)
|
|
|
|
{
|
|
|
|
struct setup_rw_req *setup = data;
|
|
|
|
int n, ref;
|
|
|
|
struct grant *gnt_list_entry;
|
2007-07-17 19:37:06 -06:00
|
|
|
unsigned int fsect, lsect;
|
2015-07-22 09:44:54 -06:00
|
|
|
/* Convenient aliases */
|
|
|
|
unsigned int grant_idx = setup->grant_idx;
|
|
|
|
struct blkif_request *ring_req = setup->ring_req;
|
2015-11-13 20:12:11 -07:00
|
|
|
struct blkfront_ring_info *rinfo = setup->rinfo;
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
/*
|
|
|
|
* We always use the shadow of the first request to store the list
|
|
|
|
* of grant associated to the block I/O request. This made the
|
|
|
|
* completion more easy to handle even if the block I/O request is
|
|
|
|
* split.
|
|
|
|
*/
|
2015-11-13 20:12:11 -07:00
|
|
|
struct blk_shadow *shadow = &rinfo->shadow[setup->id];
|
2015-07-22 09:44:54 -06:00
|
|
|
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
if (unlikely(setup->require_extra_req &&
|
|
|
|
grant_idx >= BLKIF_MAX_SEGMENTS_PER_REQUEST)) {
|
|
|
|
/*
|
|
|
|
* We are using the second request, setup grant_idx
|
|
|
|
* to be the index of the segment array.
|
|
|
|
*/
|
|
|
|
grant_idx -= BLKIF_MAX_SEGMENTS_PER_REQUEST;
|
|
|
|
ring_req = setup->extra_ring_req;
|
|
|
|
}
|
|
|
|
|
2015-07-22 09:44:54 -06:00
|
|
|
if ((ring_req->operation == BLKIF_OP_INDIRECT) &&
|
|
|
|
(grant_idx % GRANTS_PER_INDIRECT_FRAME == 0)) {
|
|
|
|
if (setup->segments)
|
|
|
|
kunmap_atomic(setup->segments);
|
|
|
|
|
|
|
|
n = grant_idx / GRANTS_PER_INDIRECT_FRAME;
|
2015-11-16 14:51:39 -07:00
|
|
|
gnt_list_entry = get_indirect_grant(&setup->gref_head, rinfo);
|
2015-07-22 09:44:54 -06:00
|
|
|
shadow->indirect_grants[n] = gnt_list_entry;
|
|
|
|
setup->segments = kmap_atomic(gnt_list_entry->page);
|
|
|
|
ring_req->u.indirect.indirect_grefs[n] = gnt_list_entry->gref;
|
|
|
|
}
|
|
|
|
|
2015-11-16 14:51:39 -07:00
|
|
|
gnt_list_entry = get_grant(&setup->gref_head, gfn, rinfo);
|
2015-07-22 09:44:54 -06:00
|
|
|
ref = gnt_list_entry->gref;
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
/*
|
|
|
|
* All the grants are stored in the shadow of the first
|
|
|
|
* request. Therefore we have to use the global index.
|
|
|
|
*/
|
|
|
|
shadow->grants_used[setup->grant_idx] = gnt_list_entry;
|
2015-07-22 09:44:54 -06:00
|
|
|
|
|
|
|
if (setup->need_copy) {
|
|
|
|
void *shared_data;
|
|
|
|
|
|
|
|
shared_data = kmap_atomic(gnt_list_entry->page);
|
|
|
|
/*
|
|
|
|
* this does not wipe data stored outside the
|
|
|
|
* range sg->offset..sg->offset+sg->length.
|
|
|
|
* Therefore, blkback *could* see data from
|
|
|
|
* previous requests. This is OK as long as
|
|
|
|
* persistent grants are shared with just one
|
|
|
|
* domain. It may need refactoring if this
|
|
|
|
* changes
|
|
|
|
*/
|
|
|
|
memcpy(shared_data + offset,
|
|
|
|
setup->bvec_data + setup->bvec_off,
|
|
|
|
len);
|
|
|
|
|
|
|
|
kunmap_atomic(shared_data);
|
|
|
|
setup->bvec_off += len;
|
|
|
|
}
|
|
|
|
|
|
|
|
fsect = offset >> 9;
|
|
|
|
lsect = fsect + (len >> 9) - 1;
|
|
|
|
if (ring_req->operation != BLKIF_OP_INDIRECT) {
|
|
|
|
ring_req->u.rw.seg[grant_idx] =
|
|
|
|
(struct blkif_request_segment) {
|
|
|
|
.gref = ref,
|
|
|
|
.first_sect = fsect,
|
|
|
|
.last_sect = lsect };
|
|
|
|
} else {
|
|
|
|
setup->segments[grant_idx % GRANTS_PER_INDIRECT_FRAME] =
|
|
|
|
(struct blkif_request_segment) {
|
|
|
|
.gref = ref,
|
|
|
|
.first_sect = fsect,
|
|
|
|
.last_sect = lsect };
|
|
|
|
}
|
|
|
|
|
|
|
|
(setup->grant_idx)++;
|
|
|
|
}
|
|
|
|
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
static void blkif_setup_extra_req(struct blkif_request *first,
|
|
|
|
struct blkif_request *second)
|
|
|
|
{
|
|
|
|
uint16_t nr_segments = first->u.rw.nr_segments;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The second request is only present when the first request uses
|
|
|
|
* all its segments. It's always the continuity of the first one.
|
|
|
|
*/
|
|
|
|
first->u.rw.nr_segments = BLKIF_MAX_SEGMENTS_PER_REQUEST;
|
|
|
|
|
|
|
|
second->u.rw.nr_segments = nr_segments - BLKIF_MAX_SEGMENTS_PER_REQUEST;
|
|
|
|
second->u.rw.sector_number = first->u.rw.sector_number +
|
|
|
|
(BLKIF_MAX_SEGMENTS_PER_REQUEST * XEN_PAGE_SIZE) / 512;
|
|
|
|
|
|
|
|
second->u.rw.handle = first->u.rw.handle;
|
|
|
|
second->operation = first->operation;
|
|
|
|
}
|
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
static int blkif_queue_rw_req(struct request *req, struct blkfront_ring_info *rinfo)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
2015-11-13 20:12:11 -07:00
|
|
|
struct blkfront_info *info = rinfo->dev_info;
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
struct blkif_request *ring_req, *extra_ring_req = NULL;
|
|
|
|
unsigned long id, extra_id = NO_ASSOCIATED_ID;
|
|
|
|
bool require_extra_req = false;
|
2015-07-22 09:44:54 -06:00
|
|
|
int i;
|
|
|
|
struct setup_rw_req setup = {
|
|
|
|
.grant_idx = 0,
|
|
|
|
.segments = NULL,
|
2015-11-13 20:12:11 -07:00
|
|
|
.rinfo = rinfo,
|
2015-07-22 09:44:54 -06:00
|
|
|
.need_copy = rq_data_dir(req) && info->feature_persistent,
|
|
|
|
};
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Used to store if we are able to queue the request by just using
|
|
|
|
* existing persistent grants, or if we have to get new grants,
|
|
|
|
* as there are not sufficiently many free.
|
|
|
|
*/
|
2009-02-24 00:10:09 -07:00
|
|
|
struct scatterlist *sg;
|
2015-07-22 09:44:54 -06:00
|
|
|
int num_sg, max_grefs, num_grant;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-07-22 09:44:54 -06:00
|
|
|
max_grefs = req->nr_phys_segments * GRANTS_PER_PSEG;
|
2013-08-12 04:53:43 -06:00
|
|
|
if (max_grefs > BLKIF_MAX_SEGMENTS_PER_REQUEST)
|
|
|
|
/*
|
|
|
|
* If we are using indirect segments we need to account
|
|
|
|
* for the indirect grefs used in the request.
|
|
|
|
*/
|
2015-07-22 09:44:54 -06:00
|
|
|
max_grefs += INDIRECT_GREFS(max_grefs);
|
2013-04-18 08:06:54 -06:00
|
|
|
|
2015-11-13 20:12:12 -07:00
|
|
|
/*
|
|
|
|
* We have to reserve 'max_grefs' grants because persistent
|
|
|
|
* grants are shared by all rings.
|
|
|
|
*/
|
|
|
|
if (max_grefs > 0)
|
|
|
|
if (gnttab_alloc_grant_references(max_grefs, &setup.gref_head) < 0) {
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
gnttab_request_free_callback(
|
2015-11-13 20:12:11 -07:00
|
|
|
&rinfo->callback,
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
blkif_restart_queue_callback,
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo,
|
2013-04-18 08:06:54 -06:00
|
|
|
max_grefs);
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
return 1;
|
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
/* Fill out a communications ring structure. */
|
2015-08-13 12:23:10 -06:00
|
|
|
id = blkif_ring_get_request(rinfo, req, &ring_req);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
num_sg = blk_rq_map_sg(req->q, req, rinfo->shadow[id].sg);
|
2015-07-22 09:44:54 -06:00
|
|
|
num_grant = 0;
|
|
|
|
/* Calculate the number of grant used */
|
2015-11-13 20:12:11 -07:00
|
|
|
for_each_sg(rinfo->shadow[id].sg, sg, num_sg, i)
|
2015-07-22 09:44:54 -06:00
|
|
|
num_grant += gnttab_count_grant(sg->offset, sg->length);
|
|
|
|
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
require_extra_req = info->max_indirect_segments == 0 &&
|
|
|
|
num_grant > BLKIF_MAX_SEGMENTS_PER_REQUEST;
|
|
|
|
BUG_ON(!HAS_EXTRA_REQ && require_extra_req);
|
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo->shadow[id].num_sg = num_sg;
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
if (num_grant > BLKIF_MAX_SEGMENTS_PER_REQUEST &&
|
|
|
|
likely(!require_extra_req)) {
|
2015-06-29 10:35:24 -06:00
|
|
|
/*
|
|
|
|
* The indirect operation can only be a BLKIF_OP_READ or
|
|
|
|
* BLKIF_OP_WRITE
|
|
|
|
*/
|
2016-06-05 13:32:23 -06:00
|
|
|
BUG_ON(req_op(req) == REQ_OP_FLUSH || req->cmd_flags & REQ_FUA);
|
2015-06-29 10:35:24 -06:00
|
|
|
ring_req->operation = BLKIF_OP_INDIRECT;
|
|
|
|
ring_req->u.indirect.indirect_op = rq_data_dir(req) ?
|
|
|
|
BLKIF_OP_WRITE : BLKIF_OP_READ;
|
|
|
|
ring_req->u.indirect.sector_number = (blkif_sector_t)blk_rq_pos(req);
|
|
|
|
ring_req->u.indirect.handle = info->handle;
|
2015-07-22 09:44:54 -06:00
|
|
|
ring_req->u.indirect.nr_segments = num_grant;
|
2011-09-01 04:39:09 -06:00
|
|
|
} else {
|
2015-06-29 10:35:24 -06:00
|
|
|
ring_req->u.rw.sector_number = (blkif_sector_t)blk_rq_pos(req);
|
|
|
|
ring_req->u.rw.handle = info->handle;
|
|
|
|
ring_req->operation = rq_data_dir(req) ?
|
|
|
|
BLKIF_OP_WRITE : BLKIF_OP_READ;
|
2016-06-05 13:32:23 -06:00
|
|
|
if (req_op(req) == REQ_OP_FLUSH || req->cmd_flags & REQ_FUA) {
|
2013-04-18 08:06:54 -06:00
|
|
|
/*
|
2015-06-29 10:35:24 -06:00
|
|
|
* Ideally we can do an unordered flush-to-disk.
|
|
|
|
* In case the backend onlysupports barriers, use that.
|
|
|
|
* A barrier request a superset of FUA, so we can
|
|
|
|
* implement it the same way. (It's also a FLUSH+FUA,
|
|
|
|
* since it is guaranteed ordered WRT previous writes.)
|
2013-04-18 08:06:54 -06:00
|
|
|
*/
|
2016-06-05 13:32:24 -06:00
|
|
|
if (info->feature_flush && info->feature_fua)
|
2015-06-29 10:35:24 -06:00
|
|
|
ring_req->operation =
|
|
|
|
BLKIF_OP_WRITE_BARRIER;
|
2016-06-05 13:32:24 -06:00
|
|
|
else if (info->feature_flush)
|
2015-06-29 10:35:24 -06:00
|
|
|
ring_req->operation =
|
|
|
|
BLKIF_OP_FLUSH_DISKCACHE;
|
2016-06-05 13:32:24 -06:00
|
|
|
else
|
2015-06-29 10:35:24 -06:00
|
|
|
ring_req->operation = 0;
|
2013-04-18 08:06:54 -06:00
|
|
|
}
|
2015-07-22 09:44:54 -06:00
|
|
|
ring_req->u.rw.nr_segments = num_grant;
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
if (unlikely(require_extra_req)) {
|
|
|
|
extra_id = blkif_ring_get_request(rinfo, req,
|
|
|
|
&extra_ring_req);
|
|
|
|
/*
|
|
|
|
* Only the first request contains the scatter-gather
|
|
|
|
* list.
|
|
|
|
*/
|
|
|
|
rinfo->shadow[extra_id].num_sg = 0;
|
|
|
|
|
|
|
|
blkif_setup_extra_req(ring_req, extra_ring_req);
|
|
|
|
|
|
|
|
/* Link the 2 requests together */
|
|
|
|
rinfo->shadow[extra_id].associated_id = id;
|
|
|
|
rinfo->shadow[id].associated_id = extra_id;
|
|
|
|
}
|
2015-06-29 10:35:24 -06:00
|
|
|
}
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
|
2015-07-22 09:44:54 -06:00
|
|
|
setup.ring_req = ring_req;
|
|
|
|
setup.id = id;
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
|
|
|
|
setup.require_extra_req = require_extra_req;
|
|
|
|
if (unlikely(require_extra_req))
|
|
|
|
setup.extra_ring_req = extra_ring_req;
|
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
for_each_sg(rinfo->shadow[id].sg, sg, num_sg, i) {
|
2015-07-22 09:44:54 -06:00
|
|
|
BUG_ON(sg->offset + sg->length > PAGE_SIZE);
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
|
2015-07-22 09:44:54 -06:00
|
|
|
if (setup.need_copy) {
|
|
|
|
setup.bvec_off = sg->offset;
|
|
|
|
setup.bvec_data = kmap_atomic(sg_page(sg));
|
|
|
|
}
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
|
2015-07-22 09:44:54 -06:00
|
|
|
gnttab_foreach_grant_in_range(sg_page(sg),
|
|
|
|
sg->offset,
|
|
|
|
sg->length,
|
|
|
|
blkif_setup_rw_req_grant,
|
|
|
|
&setup);
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
|
2015-07-22 09:44:54 -06:00
|
|
|
if (setup.need_copy)
|
|
|
|
kunmap_atomic(setup.bvec_data);
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
2015-07-22 09:44:54 -06:00
|
|
|
if (setup.segments)
|
|
|
|
kunmap_atomic(setup.segments);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
/* Keep a private copy so we can reissue requests when recovering. */
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo->shadow[id].req = *ring_req;
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
if (unlikely(require_extra_req))
|
|
|
|
rinfo->shadow[extra_id].req = *extra_ring_req;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-11-13 20:12:12 -07:00
|
|
|
if (max_grefs > 0)
|
2015-07-22 09:44:54 -06:00
|
|
|
gnttab_free_grant_references(setup.gref_head);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2015-06-29 10:35:24 -06:00
|
|
|
/*
|
|
|
|
* Generate a Xen blkfront IO request from a blk layer request. Reads
|
|
|
|
* and writes are handled as expected.
|
|
|
|
*
|
|
|
|
* @req: a request struct
|
|
|
|
*/
|
2015-11-13 20:12:11 -07:00
|
|
|
static int blkif_queue_request(struct request *req, struct blkfront_ring_info *rinfo)
|
2015-06-29 10:35:24 -06:00
|
|
|
{
|
2015-11-13 20:12:11 -07:00
|
|
|
if (unlikely(rinfo->dev_info->connected != BLKIF_STATE_CONNECTED))
|
2015-06-29 10:35:24 -06:00
|
|
|
return 1;
|
|
|
|
|
2016-06-05 13:32:17 -06:00
|
|
|
if (unlikely(req_op(req) == REQ_OP_DISCARD ||
|
2016-06-09 08:00:36 -06:00
|
|
|
req_op(req) == REQ_OP_SECURE_ERASE))
|
2015-11-13 20:12:11 -07:00
|
|
|
return blkif_queue_discard_req(req, rinfo);
|
2015-06-29 10:35:24 -06:00
|
|
|
else
|
2015-11-13 20:12:11 -07:00
|
|
|
return blkif_queue_rw_req(req, rinfo);
|
2015-06-29 10:35:24 -06:00
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
static inline void flush_requests(struct blkfront_ring_info *rinfo)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
|
|
|
int notify;
|
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
RING_PUSH_REQUESTS_AND_CHECK_NOTIFY(&rinfo->ring, notify);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
if (notify)
|
2015-11-13 20:12:11 -07:00
|
|
|
notify_remote_via_irq(rinfo->irq);
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
|
2014-12-01 06:01:13 -07:00
|
|
|
static inline bool blkif_request_flush_invalid(struct request *req,
|
|
|
|
struct blkfront_info *info)
|
2014-08-22 05:20:02 -06:00
|
|
|
{
|
|
|
|
return ((req->cmd_type != REQ_TYPE_FS) ||
|
2016-06-05 13:32:23 -06:00
|
|
|
((req_op(req) == REQ_OP_FLUSH) &&
|
2016-06-05 13:32:24 -06:00
|
|
|
!info->feature_flush) ||
|
2014-12-01 06:01:13 -07:00
|
|
|
((req->cmd_flags & REQ_FUA) &&
|
2016-06-05 13:32:24 -06:00
|
|
|
!info->feature_fua));
|
2014-08-22 05:20:02 -06:00
|
|
|
}
|
|
|
|
|
2015-07-13 03:55:24 -06:00
|
|
|
static int blkif_queue_rq(struct blk_mq_hw_ctx *hctx,
|
2015-11-16 13:14:41 -07:00
|
|
|
const struct blk_mq_queue_data *qd)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
2015-11-13 20:12:13 -07:00
|
|
|
unsigned long flags;
|
2016-05-31 02:59:17 -06:00
|
|
|
int qid = hctx->queue_num;
|
|
|
|
struct blkfront_info *info = hctx->queue->queuedata;
|
|
|
|
struct blkfront_ring_info *rinfo = NULL;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2016-05-31 02:59:17 -06:00
|
|
|
BUG_ON(info->nr_rings <= qid);
|
|
|
|
rinfo = &info->rinfo[qid];
|
2015-07-13 03:55:24 -06:00
|
|
|
blk_mq_start_request(qd->rq);
|
2015-11-13 20:12:13 -07:00
|
|
|
spin_lock_irqsave(&rinfo->ring_lock, flags);
|
2015-11-13 20:12:11 -07:00
|
|
|
if (RING_FULL(&rinfo->ring))
|
2015-07-13 03:55:24 -06:00
|
|
|
goto out_busy;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
if (blkif_request_flush_invalid(qd->rq, rinfo->dev_info))
|
2015-07-13 03:55:24 -06:00
|
|
|
goto out_err;
|
2009-05-07 20:54:15 -06:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
if (blkif_queue_request(qd->rq, rinfo))
|
2015-07-13 03:55:24 -06:00
|
|
|
goto out_busy;
|
2009-05-07 20:54:15 -06:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
flush_requests(rinfo);
|
2015-11-13 20:12:13 -07:00
|
|
|
spin_unlock_irqrestore(&rinfo->ring_lock, flags);
|
2015-07-13 03:55:24 -06:00
|
|
|
return BLK_MQ_RQ_QUEUE_OK;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-07-13 03:55:24 -06:00
|
|
|
out_err:
|
2015-11-13 20:12:13 -07:00
|
|
|
spin_unlock_irqrestore(&rinfo->ring_lock, flags);
|
2015-07-13 03:55:24 -06:00
|
|
|
return BLK_MQ_RQ_QUEUE_ERROR;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-07-13 03:55:24 -06:00
|
|
|
out_busy:
|
2015-11-13 20:12:13 -07:00
|
|
|
spin_unlock_irqrestore(&rinfo->ring_lock, flags);
|
2015-07-13 03:55:24 -06:00
|
|
|
blk_mq_stop_hw_queue(hctx);
|
|
|
|
return BLK_MQ_RQ_QUEUE_BUSY;
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
|
2015-07-13 03:55:24 -06:00
|
|
|
static struct blk_mq_ops blkfront_mq_ops = {
|
|
|
|
.queue_rq = blkif_queue_rq,
|
|
|
|
.map_queue = blk_mq_map_queue,
|
|
|
|
};
|
|
|
|
|
2016-07-01 15:43:39 -06:00
|
|
|
static void blkif_set_queue_limits(struct blkfront_info *info)
|
|
|
|
{
|
|
|
|
struct request_queue *rq = info->rq;
|
|
|
|
struct gendisk *gd = info->gd;
|
|
|
|
unsigned int segments = info->max_indirect_segments ? :
|
|
|
|
BLKIF_MAX_SEGMENTS_PER_REQUEST;
|
|
|
|
|
|
|
|
queue_flag_set_unlocked(QUEUE_FLAG_VIRT, rq);
|
|
|
|
|
|
|
|
if (info->feature_discard) {
|
|
|
|
queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, rq);
|
|
|
|
blk_queue_max_discard_sectors(rq, get_capacity(gd));
|
|
|
|
rq->limits.discard_granularity = info->discard_granularity;
|
|
|
|
rq->limits.discard_alignment = info->discard_alignment;
|
|
|
|
if (info->feature_secdiscard)
|
|
|
|
queue_flag_set_unlocked(QUEUE_FLAG_SECERASE, rq);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Hard sector size and max sectors impersonate the equiv. hardware. */
|
|
|
|
blk_queue_logical_block_size(rq, info->sector_size);
|
|
|
|
blk_queue_physical_block_size(rq, info->physical_sector_size);
|
|
|
|
blk_queue_max_hw_sectors(rq, (segments * XEN_PAGE_SIZE) / 512);
|
|
|
|
|
|
|
|
/* Each segment in a request is up to an aligned page in size. */
|
|
|
|
blk_queue_segment_boundary(rq, PAGE_SIZE - 1);
|
|
|
|
blk_queue_max_segment_size(rq, PAGE_SIZE);
|
|
|
|
|
|
|
|
/* Ensure a merged request will fit in a single I/O ring slot. */
|
|
|
|
blk_queue_max_segments(rq, segments / GRANTS_PER_PSEG);
|
|
|
|
|
|
|
|
/* Make sure buffer addresses are sector-aligned. */
|
|
|
|
blk_queue_dma_alignment(rq, 511);
|
|
|
|
|
|
|
|
/* Make sure we don't use bounce buffers. */
|
|
|
|
blk_queue_bounce_limit(rq, BLK_BOUNCE_ANY);
|
|
|
|
}
|
|
|
|
|
2013-04-18 08:06:54 -06:00
|
|
|
static int xlvbd_init_blk_queue(struct gendisk *gd, u16 sector_size,
|
2016-07-01 15:43:39 -06:00
|
|
|
unsigned int physical_sector_size)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
2007-07-24 01:28:11 -06:00
|
|
|
struct request_queue *rq;
|
2011-09-01 04:39:09 -06:00
|
|
|
struct blkfront_info *info = gd->private_data;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-07-13 03:55:24 -06:00
|
|
|
memset(&info->tag_set, 0, sizeof(info->tag_set));
|
|
|
|
info->tag_set.ops = &blkfront_mq_ops;
|
2015-11-13 20:12:14 -07:00
|
|
|
info->tag_set.nr_hw_queues = info->nr_rings;
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
if (HAS_EXTRA_REQ && info->max_indirect_segments == 0) {
|
|
|
|
/*
|
|
|
|
* When indirect descriptior is not supported, the I/O request
|
|
|
|
* will be split between multiple request in the ring.
|
|
|
|
* To avoid problems when sending the request, divide by
|
|
|
|
* 2 the depth of the queue.
|
|
|
|
*/
|
|
|
|
info->tag_set.queue_depth = BLK_RING_SIZE(info) / 2;
|
|
|
|
} else
|
|
|
|
info->tag_set.queue_depth = BLK_RING_SIZE(info);
|
2015-07-13 03:55:24 -06:00
|
|
|
info->tag_set.numa_node = NUMA_NO_NODE;
|
|
|
|
info->tag_set.flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_SG_MERGE;
|
|
|
|
info->tag_set.cmd_size = 0;
|
|
|
|
info->tag_set.driver_data = info;
|
|
|
|
|
|
|
|
if (blk_mq_alloc_tag_set(&info->tag_set))
|
2015-11-25 11:07:39 -07:00
|
|
|
return -EINVAL;
|
2015-07-13 03:55:24 -06:00
|
|
|
rq = blk_mq_init_queue(&info->tag_set);
|
|
|
|
if (IS_ERR(rq)) {
|
|
|
|
blk_mq_free_tag_set(&info->tag_set);
|
2015-11-25 11:07:39 -07:00
|
|
|
return PTR_ERR(rq);
|
2015-07-13 03:55:24 -06:00
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2016-05-31 02:59:17 -06:00
|
|
|
rq->queuedata = info;
|
2016-07-01 15:43:39 -06:00
|
|
|
info->rq = gd->queue = rq;
|
|
|
|
info->gd = gd;
|
|
|
|
info->sector_size = sector_size;
|
|
|
|
info->physical_sector_size = physical_sector_size;
|
|
|
|
blkif_set_queue_limits(info);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2016-06-05 13:32:24 -06:00
|
|
|
static const char *flush_info(struct blkfront_info *info)
|
2014-12-09 07:25:10 -07:00
|
|
|
{
|
2016-06-05 13:32:24 -06:00
|
|
|
if (info->feature_flush && info->feature_fua)
|
2014-12-09 07:25:10 -07:00
|
|
|
return "barrier: enabled;";
|
2016-06-05 13:32:24 -06:00
|
|
|
else if (info->feature_flush)
|
2014-12-09 07:25:10 -07:00
|
|
|
return "flush diskcache: enabled;";
|
2016-06-05 13:32:24 -06:00
|
|
|
else
|
2014-12-09 07:25:10 -07:00
|
|
|
return "barrier or flush: disabled;";
|
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2010-09-03 03:56:16 -06:00
|
|
|
static void xlvbd_flush(struct blkfront_info *info)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
2016-06-05 13:32:24 -06:00
|
|
|
blk_queue_write_cache(info->rq, info->feature_flush ? true : false,
|
|
|
|
info->feature_fua ? true : false);
|
2014-12-09 07:25:10 -07:00
|
|
|
pr_info("blkfront: %s: %s %s %s %s %s\n",
|
2016-06-05 13:32:24 -06:00
|
|
|
info->gd->disk_name, flush_info(info),
|
2014-12-09 07:25:10 -07:00
|
|
|
"persistent grants:", info->feature_persistent ?
|
|
|
|
"enabled;" : "disabled;", "indirect descriptors:",
|
|
|
|
info->max_indirect_segments ? "enabled;" : "disabled;");
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
|
2010-12-02 10:55:00 -07:00
|
|
|
static int xen_translate_vdev(int vdevice, int *minor, unsigned int *offset)
|
|
|
|
{
|
|
|
|
int major;
|
|
|
|
major = BLKIF_MAJOR(vdevice);
|
|
|
|
*minor = BLKIF_MINOR(vdevice);
|
|
|
|
switch (major) {
|
|
|
|
case XEN_IDE0_MAJOR:
|
|
|
|
*offset = (*minor / 64) + EMULATED_HD_DISK_NAME_OFFSET;
|
|
|
|
*minor = ((*minor / 64) * PARTS_PER_DISK) +
|
|
|
|
EMULATED_HD_DISK_MINOR_OFFSET;
|
|
|
|
break;
|
|
|
|
case XEN_IDE1_MAJOR:
|
|
|
|
*offset = (*minor / 64) + 2 + EMULATED_HD_DISK_NAME_OFFSET;
|
|
|
|
*minor = (((*minor / 64) + 2) * PARTS_PER_DISK) +
|
|
|
|
EMULATED_HD_DISK_MINOR_OFFSET;
|
|
|
|
break;
|
|
|
|
case XEN_SCSI_DISK0_MAJOR:
|
|
|
|
*offset = (*minor / PARTS_PER_DISK) + EMULATED_SD_DISK_NAME_OFFSET;
|
|
|
|
*minor = *minor + EMULATED_SD_DISK_MINOR_OFFSET;
|
|
|
|
break;
|
|
|
|
case XEN_SCSI_DISK1_MAJOR:
|
|
|
|
case XEN_SCSI_DISK2_MAJOR:
|
|
|
|
case XEN_SCSI_DISK3_MAJOR:
|
|
|
|
case XEN_SCSI_DISK4_MAJOR:
|
|
|
|
case XEN_SCSI_DISK5_MAJOR:
|
|
|
|
case XEN_SCSI_DISK6_MAJOR:
|
|
|
|
case XEN_SCSI_DISK7_MAJOR:
|
|
|
|
*offset = (*minor / PARTS_PER_DISK) +
|
|
|
|
((major - XEN_SCSI_DISK1_MAJOR + 1) * 16) +
|
|
|
|
EMULATED_SD_DISK_NAME_OFFSET;
|
|
|
|
*minor = *minor +
|
|
|
|
((major - XEN_SCSI_DISK1_MAJOR + 1) * 16 * PARTS_PER_DISK) +
|
|
|
|
EMULATED_SD_DISK_MINOR_OFFSET;
|
|
|
|
break;
|
|
|
|
case XEN_SCSI_DISK8_MAJOR:
|
|
|
|
case XEN_SCSI_DISK9_MAJOR:
|
|
|
|
case XEN_SCSI_DISK10_MAJOR:
|
|
|
|
case XEN_SCSI_DISK11_MAJOR:
|
|
|
|
case XEN_SCSI_DISK12_MAJOR:
|
|
|
|
case XEN_SCSI_DISK13_MAJOR:
|
|
|
|
case XEN_SCSI_DISK14_MAJOR:
|
|
|
|
case XEN_SCSI_DISK15_MAJOR:
|
|
|
|
*offset = (*minor / PARTS_PER_DISK) +
|
|
|
|
((major - XEN_SCSI_DISK8_MAJOR + 8) * 16) +
|
|
|
|
EMULATED_SD_DISK_NAME_OFFSET;
|
|
|
|
*minor = *minor +
|
|
|
|
((major - XEN_SCSI_DISK8_MAJOR + 8) * 16 * PARTS_PER_DISK) +
|
|
|
|
EMULATED_SD_DISK_MINOR_OFFSET;
|
|
|
|
break;
|
|
|
|
case XENVBD_MAJOR:
|
|
|
|
*offset = *minor / PARTS_PER_DISK;
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
printk(KERN_WARNING "blkfront: your disk configuration is "
|
|
|
|
"incorrect, please use an xvd device instead\n");
|
|
|
|
return -ENODEV;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2012-04-05 09:37:22 -06:00
|
|
|
static char *encode_disk_name(char *ptr, unsigned int n)
|
|
|
|
{
|
|
|
|
if (n >= 26)
|
|
|
|
ptr = encode_disk_name(ptr, n / 26 - 1);
|
|
|
|
*ptr = 'a' + n % 26;
|
|
|
|
return ptr + 1;
|
|
|
|
}
|
|
|
|
|
2008-09-17 15:30:32 -06:00
|
|
|
static int xlvbd_alloc_gendisk(blkif_sector_t capacity,
|
|
|
|
struct blkfront_info *info,
|
2013-05-13 08:28:15 -06:00
|
|
|
u16 vdisk_info, u16 sector_size,
|
|
|
|
unsigned int physical_sector_size)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
|
|
|
struct gendisk *gd;
|
|
|
|
int nr_minors = 1;
|
2010-12-02 10:55:00 -07:00
|
|
|
int err;
|
2008-09-17 15:30:32 -06:00
|
|
|
unsigned int offset;
|
|
|
|
int minor;
|
|
|
|
int nr_parts;
|
2012-04-05 09:37:22 -06:00
|
|
|
char *ptr;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
BUG_ON(info->gd != NULL);
|
|
|
|
BUG_ON(info->rq != NULL);
|
|
|
|
|
2008-09-17 15:30:32 -06:00
|
|
|
if ((info->vdevice>>EXT_SHIFT) > 1) {
|
|
|
|
/* this is above the extended range; something is wrong */
|
|
|
|
printk(KERN_WARNING "blkfront: vdevice 0x%x is above the extended range; ignoring\n", info->vdevice);
|
|
|
|
return -ENODEV;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!VDEV_IS_EXTENDED(info->vdevice)) {
|
2010-12-02 10:55:00 -07:00
|
|
|
err = xen_translate_vdev(info->vdevice, &minor, &offset);
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
nr_parts = PARTS_PER_DISK;
|
2008-09-17 15:30:32 -06:00
|
|
|
} else {
|
|
|
|
minor = BLKIF_MINOR_EXT(info->vdevice);
|
|
|
|
nr_parts = PARTS_PER_EXT_DISK;
|
2010-12-02 10:55:00 -07:00
|
|
|
offset = minor / nr_parts;
|
2011-07-14 07:30:37 -06:00
|
|
|
if (xen_hvm_domain() && offset < EMULATED_HD_DISK_NAME_OFFSET + 4)
|
2010-12-02 10:55:00 -07:00
|
|
|
printk(KERN_WARNING "blkfront: vdevice 0x%x might conflict with "
|
|
|
|
"emulated IDE disks,\n\t choose an xvd device name"
|
|
|
|
"from xvde on\n", info->vdevice);
|
2008-09-17 15:30:32 -06:00
|
|
|
}
|
2012-04-05 09:37:22 -06:00
|
|
|
if (minor >> MINORBITS) {
|
|
|
|
pr_warn("blkfront: %#x's minor (%#x) out of range; ignoring\n",
|
|
|
|
info->vdevice, minor);
|
|
|
|
return -ENODEV;
|
|
|
|
}
|
2008-09-17 15:30:32 -06:00
|
|
|
|
|
|
|
if ((minor % nr_parts) == 0)
|
|
|
|
nr_minors = nr_parts;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2010-08-07 10:28:55 -06:00
|
|
|
err = xlbd_reserve_minors(minor, nr_minors);
|
|
|
|
if (err)
|
|
|
|
goto out;
|
|
|
|
err = -ENODEV;
|
|
|
|
|
2007-07-17 19:37:06 -06:00
|
|
|
gd = alloc_disk(nr_minors);
|
|
|
|
if (gd == NULL)
|
2010-08-07 10:28:55 -06:00
|
|
|
goto release;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2012-04-05 09:37:22 -06:00
|
|
|
strcpy(gd->disk_name, DEV_NAME);
|
|
|
|
ptr = encode_disk_name(gd->disk_name + sizeof(DEV_NAME) - 1, offset);
|
|
|
|
BUG_ON(ptr >= gd->disk_name + DISK_NAME_LEN);
|
|
|
|
if (nr_minors > 1)
|
|
|
|
*ptr = 0;
|
|
|
|
else
|
|
|
|
snprintf(ptr, gd->disk_name + DISK_NAME_LEN - ptr,
|
|
|
|
"%d", minor & (nr_parts - 1));
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
gd->major = XENVBD_MAJOR;
|
|
|
|
gd->first_minor = minor;
|
|
|
|
gd->fops = &xlvbd_block_fops;
|
|
|
|
gd->private_data = info;
|
|
|
|
set_capacity(gd, capacity);
|
|
|
|
|
2016-07-01 15:43:39 -06:00
|
|
|
if (xlvbd_init_blk_queue(gd, sector_size, physical_sector_size)) {
|
2007-07-17 19:37:06 -06:00
|
|
|
del_gendisk(gd);
|
2010-08-07 10:28:55 -06:00
|
|
|
goto release;
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
|
2010-09-03 03:56:16 -06:00
|
|
|
xlvbd_flush(info);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
if (vdisk_info & VDISK_READONLY)
|
|
|
|
set_disk_ro(gd, 1);
|
|
|
|
|
|
|
|
if (vdisk_info & VDISK_REMOVABLE)
|
|
|
|
gd->flags |= GENHD_FL_REMOVABLE;
|
|
|
|
|
|
|
|
if (vdisk_info & VDISK_CDROM)
|
|
|
|
gd->flags |= GENHD_FL_CD;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
2010-08-07 10:28:55 -06:00
|
|
|
release:
|
|
|
|
xlbd_release_minors(minor, nr_minors);
|
2007-07-17 19:37:06 -06:00
|
|
|
out:
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2010-08-07 10:33:17 -06:00
|
|
|
static void xlvbd_release_gendisk(struct blkfront_info *info)
|
|
|
|
{
|
2015-11-13 20:12:12 -07:00
|
|
|
unsigned int minor, nr_minors, i;
|
2010-08-07 10:33:17 -06:00
|
|
|
|
|
|
|
if (info->rq == NULL)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* No more blkif_request(). */
|
2015-07-13 03:55:24 -06:00
|
|
|
blk_mq_stop_hw_queues(info->rq);
|
2010-08-07 10:33:17 -06:00
|
|
|
|
2015-11-13 20:12:12 -07:00
|
|
|
for (i = 0; i < info->nr_rings; i++) {
|
|
|
|
struct blkfront_ring_info *rinfo = &info->rinfo[i];
|
2010-08-07 10:33:17 -06:00
|
|
|
|
2015-11-13 20:12:12 -07:00
|
|
|
/* No more gnttab callback work. */
|
|
|
|
gnttab_cancel_free_callback(&rinfo->callback);
|
|
|
|
|
|
|
|
/* Flush gnttab callback work. Must be done with no locks held. */
|
|
|
|
flush_work(&rinfo->work);
|
|
|
|
}
|
2010-08-07 10:33:17 -06:00
|
|
|
|
|
|
|
del_gendisk(info->gd);
|
|
|
|
|
|
|
|
minor = info->gd->first_minor;
|
|
|
|
nr_minors = info->gd->minors;
|
|
|
|
xlbd_release_minors(minor, nr_minors);
|
|
|
|
|
|
|
|
blk_cleanup_queue(info->rq);
|
2015-07-13 03:55:24 -06:00
|
|
|
blk_mq_free_tag_set(&info->tag_set);
|
2010-08-07 10:33:17 -06:00
|
|
|
info->rq = NULL;
|
|
|
|
|
|
|
|
put_disk(info->gd);
|
|
|
|
info->gd = NULL;
|
|
|
|
}
|
|
|
|
|
2015-11-13 20:12:13 -07:00
|
|
|
/* Already hold rinfo->ring_lock. */
|
|
|
|
static inline void kick_pending_request_queues_locked(struct blkfront_ring_info *rinfo)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
2015-11-13 20:12:11 -07:00
|
|
|
if (!RING_FULL(&rinfo->ring))
|
|
|
|
blk_mq_start_stopped_hw_queues(rinfo->dev_info->rq, true);
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
|
2015-11-13 20:12:13 -07:00
|
|
|
static void kick_pending_request_queues(struct blkfront_ring_info *rinfo)
|
|
|
|
{
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&rinfo->ring_lock, flags);
|
|
|
|
kick_pending_request_queues_locked(rinfo);
|
|
|
|
spin_unlock_irqrestore(&rinfo->ring_lock, flags);
|
|
|
|
}
|
|
|
|
|
2007-07-17 19:37:06 -06:00
|
|
|
static void blkif_restart_queue(struct work_struct *work)
|
|
|
|
{
|
2015-11-13 20:12:11 -07:00
|
|
|
struct blkfront_ring_info *rinfo = container_of(work, struct blkfront_ring_info, work);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
if (rinfo->dev_info->connected == BLKIF_STATE_CONNECTED)
|
|
|
|
kick_pending_request_queues(rinfo);
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
|
2015-11-13 20:12:12 -07:00
|
|
|
static void blkif_free_ring(struct blkfront_ring_info *rinfo)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
2015-11-16 14:51:39 -07:00
|
|
|
struct grant *persistent_gnt, *n;
|
2015-11-13 20:12:12 -07:00
|
|
|
struct blkfront_info *info = rinfo->dev_info;
|
2013-04-18 08:06:54 -06:00
|
|
|
int i, j, segs;
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
|
2013-10-29 11:31:14 -06:00
|
|
|
/*
|
|
|
|
* Remove indirect pages, this only happens when using indirect
|
|
|
|
* descriptors but not persistent grants
|
|
|
|
*/
|
2015-11-13 20:12:11 -07:00
|
|
|
if (!list_empty(&rinfo->indirect_pages)) {
|
2013-10-29 11:31:14 -06:00
|
|
|
struct page *indirect_page, *n;
|
|
|
|
|
|
|
|
BUG_ON(info->feature_persistent);
|
2015-11-13 20:12:11 -07:00
|
|
|
list_for_each_entry_safe(indirect_page, n, &rinfo->indirect_pages, lru) {
|
2013-10-29 11:31:14 -06:00
|
|
|
list_del(&indirect_page->lru);
|
|
|
|
__free_page(indirect_page);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-11-16 14:51:39 -07:00
|
|
|
/* Remove all persistent grants. */
|
|
|
|
if (!list_empty(&rinfo->grants)) {
|
|
|
|
list_for_each_entry_safe(persistent_gnt, n,
|
|
|
|
&rinfo->grants, node) {
|
|
|
|
list_del(&persistent_gnt->node);
|
|
|
|
if (persistent_gnt->gref != GRANT_INVALID_REF) {
|
|
|
|
gnttab_end_foreign_access(persistent_gnt->gref,
|
|
|
|
0, 0UL);
|
|
|
|
rinfo->persistent_gnts_c--;
|
|
|
|
}
|
|
|
|
if (info->feature_persistent)
|
|
|
|
__free_page(persistent_gnt->page);
|
|
|
|
kfree(persistent_gnt);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
BUG_ON(rinfo->persistent_gnts_c != 0);
|
|
|
|
|
2015-06-02 23:40:03 -06:00
|
|
|
for (i = 0; i < BLK_RING_SIZE(info); i++) {
|
2013-04-18 08:06:54 -06:00
|
|
|
/*
|
|
|
|
* Clear persistent grants present in requests already
|
|
|
|
* on the shared ring
|
|
|
|
*/
|
2015-11-13 20:12:11 -07:00
|
|
|
if (!rinfo->shadow[i].request)
|
2013-04-18 08:06:54 -06:00
|
|
|
goto free_shadow;
|
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
segs = rinfo->shadow[i].req.operation == BLKIF_OP_INDIRECT ?
|
|
|
|
rinfo->shadow[i].req.u.indirect.nr_segments :
|
|
|
|
rinfo->shadow[i].req.u.rw.nr_segments;
|
2013-04-18 08:06:54 -06:00
|
|
|
for (j = 0; j < segs; j++) {
|
2015-11-13 20:12:11 -07:00
|
|
|
persistent_gnt = rinfo->shadow[i].grants_used[j];
|
2013-04-18 08:06:54 -06:00
|
|
|
gnttab_end_foreign_access(persistent_gnt->gref, 0, 0UL);
|
2013-10-29 11:31:14 -06:00
|
|
|
if (info->feature_persistent)
|
2015-06-30 04:58:51 -06:00
|
|
|
__free_page(persistent_gnt->page);
|
2013-04-18 08:06:54 -06:00
|
|
|
kfree(persistent_gnt);
|
|
|
|
}
|
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
if (rinfo->shadow[i].req.operation != BLKIF_OP_INDIRECT)
|
2013-04-18 08:06:54 -06:00
|
|
|
/*
|
|
|
|
* If this is not an indirect operation don't try to
|
|
|
|
* free indirect segments
|
|
|
|
*/
|
|
|
|
goto free_shadow;
|
|
|
|
|
|
|
|
for (j = 0; j < INDIRECT_GREFS(segs); j++) {
|
2015-11-13 20:12:11 -07:00
|
|
|
persistent_gnt = rinfo->shadow[i].indirect_grants[j];
|
2013-04-18 08:06:54 -06:00
|
|
|
gnttab_end_foreign_access(persistent_gnt->gref, 0, 0UL);
|
2015-06-30 04:58:51 -06:00
|
|
|
__free_page(persistent_gnt->page);
|
2013-04-18 08:06:54 -06:00
|
|
|
kfree(persistent_gnt);
|
|
|
|
}
|
|
|
|
|
|
|
|
free_shadow:
|
2015-11-13 20:12:11 -07:00
|
|
|
kfree(rinfo->shadow[i].grants_used);
|
|
|
|
rinfo->shadow[i].grants_used = NULL;
|
|
|
|
kfree(rinfo->shadow[i].indirect_grants);
|
|
|
|
rinfo->shadow[i].indirect_grants = NULL;
|
|
|
|
kfree(rinfo->shadow[i].sg);
|
|
|
|
rinfo->shadow[i].sg = NULL;
|
2013-04-18 08:06:54 -06:00
|
|
|
}
|
|
|
|
|
2007-07-17 19:37:06 -06:00
|
|
|
/* No more gnttab callback work. */
|
2015-11-13 20:12:11 -07:00
|
|
|
gnttab_cancel_free_callback(&rinfo->callback);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
/* Flush gnttab callback work. Must be done with no locks held. */
|
2015-11-13 20:12:11 -07:00
|
|
|
flush_work(&rinfo->work);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
/* Free resources associated with old device channel. */
|
2015-06-02 23:40:03 -06:00
|
|
|
for (i = 0; i < info->nr_ring_pages; i++) {
|
2015-11-13 20:12:11 -07:00
|
|
|
if (rinfo->ring_ref[i] != GRANT_INVALID_REF) {
|
|
|
|
gnttab_end_foreign_access(rinfo->ring_ref[i], 0, 0);
|
|
|
|
rinfo->ring_ref[i] = GRANT_INVALID_REF;
|
2015-06-02 23:40:03 -06:00
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
2016-07-01 13:45:57 -06:00
|
|
|
free_pages((unsigned long)rinfo->ring.sring, get_order(info->nr_ring_pages * XEN_PAGE_SIZE));
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo->ring.sring = NULL;
|
2015-06-02 23:40:03 -06:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
if (rinfo->irq)
|
|
|
|
unbind_from_irqhandler(rinfo->irq, rinfo);
|
|
|
|
rinfo->evtchn = rinfo->irq = 0;
|
2015-11-13 20:12:12 -07:00
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-11-13 20:12:12 -07:00
|
|
|
static void blkif_free(struct blkfront_info *info, int suspend)
|
|
|
|
{
|
|
|
|
unsigned int i;
|
|
|
|
|
|
|
|
/* Prevent new requests being issued until we fix things up. */
|
|
|
|
info->connected = suspend ?
|
|
|
|
BLKIF_STATE_SUSPENDED : BLKIF_STATE_DISCONNECTED;
|
|
|
|
/* No more blkif_request(). */
|
|
|
|
if (info->rq)
|
|
|
|
blk_mq_stop_hw_queues(info->rq);
|
|
|
|
|
|
|
|
for (i = 0; i < info->nr_rings; i++)
|
|
|
|
blkif_free_ring(&info->rinfo[i]);
|
|
|
|
|
|
|
|
kfree(info->rinfo);
|
|
|
|
info->rinfo = NULL;
|
|
|
|
info->nr_rings = 0;
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
|
2015-07-22 09:44:54 -06:00
|
|
|
struct copy_from_grant {
|
|
|
|
const struct blk_shadow *s;
|
|
|
|
unsigned int grant_idx;
|
|
|
|
unsigned int bvec_offset;
|
|
|
|
char *bvec_data;
|
|
|
|
};
|
|
|
|
|
|
|
|
static void blkif_copy_from_grant(unsigned long gfn, unsigned int offset,
|
|
|
|
unsigned int len, void *data)
|
|
|
|
{
|
|
|
|
struct copy_from_grant *info = data;
|
|
|
|
char *shared_data;
|
|
|
|
/* Convenient aliases */
|
|
|
|
const struct blk_shadow *s = info->s;
|
|
|
|
|
|
|
|
shared_data = kmap_atomic(s->grants_used[info->grant_idx]->page);
|
|
|
|
|
|
|
|
memcpy(info->bvec_data + info->bvec_offset,
|
|
|
|
shared_data + offset, len);
|
|
|
|
|
|
|
|
info->bvec_offset += len;
|
|
|
|
info->grant_idx++;
|
|
|
|
|
|
|
|
kunmap_atomic(shared_data);
|
|
|
|
}
|
|
|
|
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
static enum blk_req_status blkif_rsp_to_req_status(int rsp)
|
|
|
|
{
|
|
|
|
switch (rsp)
|
|
|
|
{
|
|
|
|
case BLKIF_RSP_OKAY:
|
|
|
|
return REQ_DONE;
|
|
|
|
case BLKIF_RSP_EOPNOTSUPP:
|
|
|
|
return REQ_EOPNOTSUPP;
|
|
|
|
case BLKIF_RSP_ERROR:
|
|
|
|
/* Fallthrough. */
|
|
|
|
default:
|
|
|
|
return REQ_ERROR;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Get the final status of the block request based on two ring response
|
|
|
|
*/
|
|
|
|
static int blkif_get_final_status(enum blk_req_status s1,
|
|
|
|
enum blk_req_status s2)
|
|
|
|
{
|
|
|
|
BUG_ON(s1 == REQ_WAITING);
|
|
|
|
BUG_ON(s2 == REQ_WAITING);
|
|
|
|
|
|
|
|
if (s1 == REQ_ERROR || s2 == REQ_ERROR)
|
|
|
|
return BLKIF_RSP_ERROR;
|
|
|
|
else if (s1 == REQ_EOPNOTSUPP || s2 == REQ_EOPNOTSUPP)
|
|
|
|
return BLKIF_RSP_EOPNOTSUPP;
|
|
|
|
return BLKIF_RSP_OKAY;
|
|
|
|
}
|
|
|
|
|
|
|
|
static bool blkif_completion(unsigned long *id,
|
|
|
|
struct blkfront_ring_info *rinfo,
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
struct blkif_response *bret)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
2012-12-07 11:00:31 -07:00
|
|
|
int i = 0;
|
2013-05-02 02:58:50 -06:00
|
|
|
struct scatterlist *sg;
|
2015-07-22 09:44:54 -06:00
|
|
|
int num_sg, num_grant;
|
2015-11-13 20:12:11 -07:00
|
|
|
struct blkfront_info *info = rinfo->dev_info;
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
struct blk_shadow *s = &rinfo->shadow[*id];
|
2015-07-22 09:44:54 -06:00
|
|
|
struct copy_from_grant data = {
|
|
|
|
.grant_idx = 0,
|
|
|
|
};
|
2013-04-18 08:06:54 -06:00
|
|
|
|
2015-07-22 09:44:54 -06:00
|
|
|
num_grant = s->req.operation == BLKIF_OP_INDIRECT ?
|
2013-04-18 08:06:54 -06:00
|
|
|
s->req.u.indirect.nr_segments : s->req.u.rw.nr_segments;
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
|
|
|
|
/* The I/O request may be split in two. */
|
|
|
|
if (unlikely(s->associated_id != NO_ASSOCIATED_ID)) {
|
|
|
|
struct blk_shadow *s2 = &rinfo->shadow[s->associated_id];
|
|
|
|
|
|
|
|
/* Keep the status of the current response in shadow. */
|
|
|
|
s->status = blkif_rsp_to_req_status(bret->status);
|
|
|
|
|
|
|
|
/* Wait the second response if not yet here. */
|
|
|
|
if (s2->status == REQ_WAITING)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
bret->status = blkif_get_final_status(s->status,
|
|
|
|
s2->status);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* All the grants is stored in the first shadow in order
|
|
|
|
* to make the completion code simpler.
|
|
|
|
*/
|
|
|
|
num_grant += s2->req.u.rw.nr_segments;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The two responses may not come in order. Only the
|
|
|
|
* first request will store the scatter-gather list.
|
|
|
|
*/
|
|
|
|
if (s2->num_sg != 0) {
|
|
|
|
/* Update "id" with the ID of the first response. */
|
|
|
|
*id = s->associated_id;
|
|
|
|
s = s2;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We don't need anymore the second request, so recycling
|
|
|
|
* it now.
|
|
|
|
*/
|
|
|
|
if (add_id_to_freelist(rinfo, s->associated_id))
|
|
|
|
WARN(1, "%s: can't recycle the second part (id = %ld) of the request\n",
|
|
|
|
info->gd->disk_name, s->associated_id);
|
|
|
|
}
|
|
|
|
|
|
|
|
data.s = s;
|
2015-07-22 09:44:54 -06:00
|
|
|
num_sg = s->num_sg;
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
|
2013-10-29 11:31:14 -06:00
|
|
|
if (bret->operation == BLKIF_OP_READ && info->feature_persistent) {
|
2015-07-22 09:44:54 -06:00
|
|
|
for_each_sg(s->sg, sg, num_sg, i) {
|
2013-05-02 02:58:50 -06:00
|
|
|
BUG_ON(sg->offset + sg->length > PAGE_SIZE);
|
2015-07-22 09:44:54 -06:00
|
|
|
|
|
|
|
data.bvec_offset = sg->offset;
|
|
|
|
data.bvec_data = kmap_atomic(sg_page(sg));
|
|
|
|
|
|
|
|
gnttab_foreach_grant_in_range(sg_page(sg),
|
|
|
|
sg->offset,
|
|
|
|
sg->length,
|
|
|
|
blkif_copy_from_grant,
|
|
|
|
&data);
|
|
|
|
|
|
|
|
kunmap_atomic(data.bvec_data);
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
/* Add the persistent grant into the list of free grants */
|
2015-07-22 09:44:54 -06:00
|
|
|
for (i = 0; i < num_grant; i++) {
|
2013-08-12 04:53:44 -06:00
|
|
|
if (gnttab_query_foreign_access(s->grants_used[i]->gref)) {
|
|
|
|
/*
|
|
|
|
* If the grant is still mapped by the backend (the
|
|
|
|
* backend has chosen to make this grant persistent)
|
|
|
|
* we add it at the head of the list, so it will be
|
|
|
|
* reused first.
|
|
|
|
*/
|
2013-10-29 11:31:14 -06:00
|
|
|
if (!info->feature_persistent)
|
|
|
|
pr_alert_ratelimited("backed has not unmapped grant: %u\n",
|
|
|
|
s->grants_used[i]->gref);
|
2015-11-16 14:51:39 -07:00
|
|
|
list_add(&s->grants_used[i]->node, &rinfo->grants);
|
|
|
|
rinfo->persistent_gnts_c++;
|
2013-08-12 04:53:44 -06:00
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* If the grant is not mapped by the backend we end the
|
|
|
|
* foreign access and add it to the tail of the list,
|
|
|
|
* so it will not be picked again unless we run out of
|
|
|
|
* persistent grants.
|
|
|
|
*/
|
|
|
|
gnttab_end_foreign_access(s->grants_used[i]->gref, 0, 0UL);
|
|
|
|
s->grants_used[i]->gref = GRANT_INVALID_REF;
|
2015-11-16 14:51:39 -07:00
|
|
|
list_add_tail(&s->grants_used[i]->node, &rinfo->grants);
|
2013-08-12 04:53:44 -06:00
|
|
|
}
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
}
|
2013-04-18 08:06:54 -06:00
|
|
|
if (s->req.operation == BLKIF_OP_INDIRECT) {
|
2015-07-22 09:44:54 -06:00
|
|
|
for (i = 0; i < INDIRECT_GREFS(num_grant); i++) {
|
2013-08-12 04:53:44 -06:00
|
|
|
if (gnttab_query_foreign_access(s->indirect_grants[i]->gref)) {
|
2013-10-29 11:31:14 -06:00
|
|
|
if (!info->feature_persistent)
|
|
|
|
pr_alert_ratelimited("backed has not unmapped grant: %u\n",
|
|
|
|
s->indirect_grants[i]->gref);
|
2015-11-16 14:51:39 -07:00
|
|
|
list_add(&s->indirect_grants[i]->node, &rinfo->grants);
|
|
|
|
rinfo->persistent_gnts_c++;
|
2013-08-12 04:53:44 -06:00
|
|
|
} else {
|
2013-10-29 11:31:14 -06:00
|
|
|
struct page *indirect_page;
|
|
|
|
|
2013-08-12 04:53:44 -06:00
|
|
|
gnttab_end_foreign_access(s->indirect_grants[i]->gref, 0, 0UL);
|
2013-10-29 11:31:14 -06:00
|
|
|
/*
|
|
|
|
* Add the used indirect page back to the list of
|
|
|
|
* available pages for indirect grefs.
|
|
|
|
*/
|
2015-07-22 00:40:09 -06:00
|
|
|
if (!info->feature_persistent) {
|
2015-06-30 04:58:51 -06:00
|
|
|
indirect_page = s->indirect_grants[i]->page;
|
2015-11-13 20:12:11 -07:00
|
|
|
list_add(&indirect_page->lru, &rinfo->indirect_pages);
|
2015-07-22 00:40:09 -06:00
|
|
|
}
|
2013-08-12 04:53:44 -06:00
|
|
|
s->indirect_grants[i]->gref = GRANT_INVALID_REF;
|
2015-11-16 14:51:39 -07:00
|
|
|
list_add_tail(&s->indirect_grants[i]->node, &rinfo->grants);
|
2013-08-12 04:53:44 -06:00
|
|
|
}
|
2013-04-18 08:06:54 -06:00
|
|
|
}
|
|
|
|
}
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
|
|
|
|
return 1;
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
static irqreturn_t blkif_interrupt(int irq, void *dev_id)
|
|
|
|
{
|
|
|
|
struct request *req;
|
|
|
|
struct blkif_response *bret;
|
|
|
|
RING_IDX i, rp;
|
|
|
|
unsigned long flags;
|
2015-11-13 20:12:11 -07:00
|
|
|
struct blkfront_ring_info *rinfo = (struct blkfront_ring_info *)dev_id;
|
|
|
|
struct blkfront_info *info = rinfo->dev_info;
|
2015-09-27 13:01:50 -06:00
|
|
|
int error;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-11-13 20:12:13 -07:00
|
|
|
if (unlikely(info->connected != BLKIF_STATE_CONNECTED))
|
2007-07-17 19:37:06 -06:00
|
|
|
return IRQ_HANDLED;
|
|
|
|
|
2015-11-13 20:12:13 -07:00
|
|
|
spin_lock_irqsave(&rinfo->ring_lock, flags);
|
2007-07-17 19:37:06 -06:00
|
|
|
again:
|
2015-11-13 20:12:11 -07:00
|
|
|
rp = rinfo->ring.sring->rsp_prod;
|
2007-07-17 19:37:06 -06:00
|
|
|
rmb(); /* Ensure we see queued responses up to 'rp'. */
|
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
for (i = rinfo->ring.rsp_cons; i != rp; i++) {
|
2007-07-17 19:37:06 -06:00
|
|
|
unsigned long id;
|
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
bret = RING_GET_RESPONSE(&rinfo->ring, i);
|
2007-07-17 19:37:06 -06:00
|
|
|
id = bret->id;
|
2012-05-25 15:34:51 -06:00
|
|
|
/*
|
|
|
|
* The backend has messed up and given us an id that we would
|
|
|
|
* never have given to it (we stamp it up to BLK_RING_SIZE -
|
|
|
|
* look in get_id_from_freelist.
|
|
|
|
*/
|
2015-06-02 23:40:03 -06:00
|
|
|
if (id >= BLK_RING_SIZE(info)) {
|
2012-05-25 15:34:51 -06:00
|
|
|
WARN(1, "%s: response to %s has incorrect id (%ld)\n",
|
|
|
|
info->gd->disk_name, op_name(bret->operation), id);
|
|
|
|
/* We can't safely get the 'struct request' as
|
|
|
|
* the id is busted. */
|
|
|
|
continue;
|
|
|
|
}
|
2015-11-13 20:12:11 -07:00
|
|
|
req = rinfo->shadow[id].request;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
if (bret->operation != BLKIF_OP_DISCARD) {
|
|
|
|
/*
|
|
|
|
* We may need to wait for an extra response if the
|
|
|
|
* I/O request is split in 2
|
|
|
|
*/
|
|
|
|
if (!blkif_completion(&id, rinfo, bret))
|
|
|
|
continue;
|
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
if (add_id_to_freelist(rinfo, id)) {
|
2012-05-25 15:34:51 -06:00
|
|
|
WARN(1, "%s: response to %s (id %ld) couldn't be recycled!\n",
|
|
|
|
info->gd->disk_name, op_name(bret->operation), id);
|
|
|
|
continue;
|
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-09-27 13:01:50 -06:00
|
|
|
error = (bret->status == BLKIF_RSP_OKAY) ? 0 : -EIO;
|
2007-07-17 19:37:06 -06:00
|
|
|
switch (bret->operation) {
|
2011-09-01 04:39:09 -06:00
|
|
|
case BLKIF_OP_DISCARD:
|
|
|
|
if (unlikely(bret->status == BLKIF_RSP_EOPNOTSUPP)) {
|
|
|
|
struct request_queue *rq = info->rq;
|
2012-05-25 15:34:51 -06:00
|
|
|
printk(KERN_WARNING "blkfront: %s: %s op failed\n",
|
|
|
|
info->gd->disk_name, op_name(bret->operation));
|
2015-09-27 13:01:50 -06:00
|
|
|
error = -EOPNOTSUPP;
|
2011-09-01 04:39:09 -06:00
|
|
|
info->feature_discard = 0;
|
2011-10-12 14:23:30 -06:00
|
|
|
info->feature_secdiscard = 0;
|
2011-09-01 04:39:09 -06:00
|
|
|
queue_flag_clear(QUEUE_FLAG_DISCARD, rq);
|
2016-06-09 08:00:36 -06:00
|
|
|
queue_flag_clear(QUEUE_FLAG_SECERASE, rq);
|
2011-09-01 04:39:09 -06:00
|
|
|
}
|
2015-09-27 13:01:50 -06:00
|
|
|
blk_mq_complete_request(req, error);
|
2011-09-01 04:39:09 -06:00
|
|
|
break;
|
2011-05-03 10:01:11 -06:00
|
|
|
case BLKIF_OP_FLUSH_DISKCACHE:
|
2007-07-17 19:37:06 -06:00
|
|
|
case BLKIF_OP_WRITE_BARRIER:
|
|
|
|
if (unlikely(bret->status == BLKIF_RSP_EOPNOTSUPP)) {
|
2012-05-25 15:34:51 -06:00
|
|
|
printk(KERN_WARNING "blkfront: %s: %s op failed\n",
|
|
|
|
info->gd->disk_name, op_name(bret->operation));
|
2015-09-27 13:01:50 -06:00
|
|
|
error = -EOPNOTSUPP;
|
2010-11-02 09:55:58 -06:00
|
|
|
}
|
|
|
|
if (unlikely(bret->status == BLKIF_RSP_ERROR &&
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo->shadow[id].req.u.rw.nr_segments == 0)) {
|
2012-05-25 15:34:51 -06:00
|
|
|
printk(KERN_WARNING "blkfront: %s: empty %s op failed\n",
|
|
|
|
info->gd->disk_name, op_name(bret->operation));
|
2015-09-27 13:01:50 -06:00
|
|
|
error = -EOPNOTSUPP;
|
2010-11-02 09:55:58 -06:00
|
|
|
}
|
2015-09-27 13:01:50 -06:00
|
|
|
if (unlikely(error)) {
|
|
|
|
if (error == -EOPNOTSUPP)
|
|
|
|
error = 0;
|
2016-06-05 13:32:24 -06:00
|
|
|
info->feature_fua = 0;
|
2010-09-03 03:56:16 -06:00
|
|
|
info->feature_flush = 0;
|
|
|
|
xlvbd_flush(info);
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
/* fall through */
|
|
|
|
case BLKIF_OP_READ:
|
|
|
|
case BLKIF_OP_WRITE:
|
|
|
|
if (unlikely(bret->status != BLKIF_RSP_OKAY))
|
|
|
|
dev_dbg(&info->xbdev->dev, "Bad return from blkdev data "
|
|
|
|
"request: %x\n", bret->status);
|
|
|
|
|
2015-09-27 13:01:50 -06:00
|
|
|
blk_mq_complete_request(req, error);
|
2007-07-17 19:37:06 -06:00
|
|
|
break;
|
|
|
|
default:
|
|
|
|
BUG();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo->ring.rsp_cons = i;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
if (i != rinfo->ring.req_prod_pvt) {
|
2007-07-17 19:37:06 -06:00
|
|
|
int more_to_do;
|
2015-11-13 20:12:11 -07:00
|
|
|
RING_FINAL_CHECK_FOR_RESPONSES(&rinfo->ring, more_to_do);
|
2007-07-17 19:37:06 -06:00
|
|
|
if (more_to_do)
|
|
|
|
goto again;
|
|
|
|
} else
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo->ring.sring->rsp_event = i + 1;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-11-13 20:12:13 -07:00
|
|
|
kick_pending_request_queues_locked(rinfo);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-11-13 20:12:13 -07:00
|
|
|
spin_unlock_irqrestore(&rinfo->ring_lock, flags);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
return IRQ_HANDLED;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
static int setup_blkring(struct xenbus_device *dev,
|
2015-11-13 20:12:11 -07:00
|
|
|
struct blkfront_ring_info *rinfo)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
|
|
|
struct blkif_sring *sring;
|
2015-06-02 23:40:03 -06:00
|
|
|
int err, i;
|
2015-11-13 20:12:11 -07:00
|
|
|
struct blkfront_info *info = rinfo->dev_info;
|
2015-07-22 09:44:54 -06:00
|
|
|
unsigned long ring_size = info->nr_ring_pages * XEN_PAGE_SIZE;
|
2015-10-13 10:50:11 -06:00
|
|
|
grant_ref_t gref[XENBUS_MAX_RING_GRANTS];
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-06-02 23:40:03 -06:00
|
|
|
for (i = 0; i < info->nr_ring_pages; i++)
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo->ring_ref[i] = GRANT_INVALID_REF;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-06-02 23:40:03 -06:00
|
|
|
sring = (struct blkif_sring *)__get_free_pages(GFP_NOIO | __GFP_HIGH,
|
|
|
|
get_order(ring_size));
|
2007-07-17 19:37:06 -06:00
|
|
|
if (!sring) {
|
|
|
|
xenbus_dev_fatal(dev, -ENOMEM, "allocating shared ring");
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
SHARED_RING_INIT(sring);
|
2015-11-13 20:12:11 -07:00
|
|
|
FRONT_RING_INIT(&rinfo->ring, sring, ring_size);
|
2009-02-24 00:10:09 -07:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
err = xenbus_grant_ring(dev, rinfo->ring.sring, info->nr_ring_pages, gref);
|
2007-07-17 19:37:06 -06:00
|
|
|
if (err < 0) {
|
2015-06-02 23:40:03 -06:00
|
|
|
free_pages((unsigned long)sring, get_order(ring_size));
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo->ring.sring = NULL;
|
2007-07-17 19:37:06 -06:00
|
|
|
goto fail;
|
|
|
|
}
|
2015-06-02 23:40:03 -06:00
|
|
|
for (i = 0; i < info->nr_ring_pages; i++)
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo->ring_ref[i] = gref[i];
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
err = xenbus_alloc_evtchn(dev, &rinfo->evtchn);
|
2007-07-17 19:37:06 -06:00
|
|
|
if (err)
|
|
|
|
goto fail;
|
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
err = bind_evtchn_to_irqhandler(rinfo->evtchn, blkif_interrupt, 0,
|
|
|
|
"blkif", rinfo);
|
2007-07-17 19:37:06 -06:00
|
|
|
if (err <= 0) {
|
|
|
|
xenbus_dev_fatal(dev, err,
|
|
|
|
"bind_evtchn_to_irqhandler failed");
|
|
|
|
goto fail;
|
|
|
|
}
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo->irq = err;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
fail:
|
|
|
|
blkif_free(info, 0);
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2015-11-13 20:12:14 -07:00
|
|
|
/*
|
|
|
|
* Write out per-ring/queue nodes including ring-ref and event-channel, and each
|
|
|
|
* ring buffer may have multi pages depending on ->nr_ring_pages.
|
|
|
|
*/
|
|
|
|
static int write_per_ring_nodes(struct xenbus_transaction xbt,
|
|
|
|
struct blkfront_ring_info *rinfo, const char *dir)
|
|
|
|
{
|
|
|
|
int err;
|
|
|
|
unsigned int i;
|
|
|
|
const char *message = NULL;
|
|
|
|
struct blkfront_info *info = rinfo->dev_info;
|
|
|
|
|
|
|
|
if (info->nr_ring_pages == 1) {
|
|
|
|
err = xenbus_printf(xbt, dir, "ring-ref", "%u", rinfo->ring_ref[0]);
|
|
|
|
if (err) {
|
|
|
|
message = "writing ring-ref";
|
|
|
|
goto abort_transaction;
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
for (i = 0; i < info->nr_ring_pages; i++) {
|
|
|
|
char ring_ref_name[RINGREF_NAME_LEN];
|
|
|
|
|
|
|
|
snprintf(ring_ref_name, RINGREF_NAME_LEN, "ring-ref%u", i);
|
|
|
|
err = xenbus_printf(xbt, dir, ring_ref_name,
|
|
|
|
"%u", rinfo->ring_ref[i]);
|
|
|
|
if (err) {
|
|
|
|
message = "writing ring-ref";
|
|
|
|
goto abort_transaction;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
err = xenbus_printf(xbt, dir, "event-channel", "%u", rinfo->evtchn);
|
|
|
|
if (err) {
|
|
|
|
message = "writing event-channel";
|
|
|
|
goto abort_transaction;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
abort_transaction:
|
|
|
|
xenbus_transaction_end(xbt, 1);
|
|
|
|
if (message)
|
|
|
|
xenbus_dev_fatal(info->xbdev, err, "%s", message);
|
|
|
|
|
|
|
|
return err;
|
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
/* Common code used when first setting up, and when resuming. */
|
2009-12-04 08:33:54 -07:00
|
|
|
static int talk_to_blkback(struct xenbus_device *dev,
|
2007-07-17 19:37:06 -06:00
|
|
|
struct blkfront_info *info)
|
|
|
|
{
|
|
|
|
const char *message = NULL;
|
|
|
|
struct xenbus_transaction xbt;
|
2015-11-13 20:12:14 -07:00
|
|
|
int err;
|
|
|
|
unsigned int i, max_page_order = 0;
|
2015-06-02 23:40:03 -06:00
|
|
|
unsigned int ring_page_order = 0;
|
|
|
|
|
|
|
|
err = xenbus_scanf(XBT_NIL, info->xbdev->otherend,
|
|
|
|
"max-ring-page-order", "%u", &max_page_order);
|
|
|
|
if (err != 1)
|
|
|
|
info->nr_ring_pages = 1;
|
|
|
|
else {
|
|
|
|
ring_page_order = min(xen_blkif_max_ring_order, max_page_order);
|
|
|
|
info->nr_ring_pages = 1 << ring_page_order;
|
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-11-13 20:12:12 -07:00
|
|
|
for (i = 0; i < info->nr_rings; i++) {
|
2015-11-13 20:12:14 -07:00
|
|
|
struct blkfront_ring_info *rinfo = &info->rinfo[i];
|
|
|
|
|
2015-11-13 20:12:12 -07:00
|
|
|
/* Create shared ring, alloc event channel. */
|
|
|
|
err = setup_blkring(dev, rinfo);
|
|
|
|
if (err)
|
|
|
|
goto destroy_blkring;
|
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
again:
|
|
|
|
err = xenbus_transaction_start(&xbt);
|
|
|
|
if (err) {
|
|
|
|
xenbus_dev_fatal(dev, err, "starting transaction");
|
|
|
|
goto destroy_blkring;
|
|
|
|
}
|
|
|
|
|
2015-11-13 20:12:14 -07:00
|
|
|
if (info->nr_ring_pages > 1) {
|
|
|
|
err = xenbus_printf(xbt, dev->nodename, "ring-page-order", "%u",
|
|
|
|
ring_page_order);
|
|
|
|
if (err) {
|
|
|
|
message = "writing ring-page-order";
|
|
|
|
goto abort_transaction;
|
|
|
|
}
|
|
|
|
}
|
2015-11-13 20:12:12 -07:00
|
|
|
|
2015-11-13 20:12:14 -07:00
|
|
|
/* We already got the number of queues/rings in _probe */
|
|
|
|
if (info->nr_rings == 1) {
|
|
|
|
err = write_per_ring_nodes(xbt, &info->rinfo[0], dev->nodename);
|
|
|
|
if (err)
|
|
|
|
goto destroy_blkring;
|
|
|
|
} else {
|
|
|
|
char *path;
|
|
|
|
size_t pathsize;
|
2015-11-13 20:12:12 -07:00
|
|
|
|
2015-11-13 20:12:14 -07:00
|
|
|
err = xenbus_printf(xbt, dev->nodename, "multi-queue-num-queues", "%u",
|
|
|
|
info->nr_rings);
|
2015-11-13 20:12:12 -07:00
|
|
|
if (err) {
|
2015-11-13 20:12:14 -07:00
|
|
|
message = "writing multi-queue-num-queues";
|
2015-11-13 20:12:12 -07:00
|
|
|
goto abort_transaction;
|
|
|
|
}
|
2015-11-13 20:12:14 -07:00
|
|
|
|
|
|
|
pathsize = strlen(dev->nodename) + QUEUE_NAME_LEN;
|
|
|
|
path = kmalloc(pathsize, GFP_KERNEL);
|
|
|
|
if (!path) {
|
|
|
|
err = -ENOMEM;
|
|
|
|
message = "ENOMEM while writing ring references";
|
|
|
|
goto abort_transaction;
|
|
|
|
}
|
|
|
|
|
|
|
|
for (i = 0; i < info->nr_rings; i++) {
|
|
|
|
memset(path, 0, pathsize);
|
|
|
|
snprintf(path, pathsize, "%s/queue-%u", dev->nodename, i);
|
|
|
|
err = write_per_ring_nodes(xbt, &info->rinfo[i], path);
|
|
|
|
if (err) {
|
|
|
|
kfree(path);
|
|
|
|
goto destroy_blkring;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
kfree(path);
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
2008-04-02 11:54:02 -06:00
|
|
|
err = xenbus_printf(xbt, dev->nodename, "protocol", "%s",
|
|
|
|
XEN_IO_PROTO_ABI_NATIVE);
|
|
|
|
if (err) {
|
|
|
|
message = "writing protocol";
|
|
|
|
goto abort_transaction;
|
|
|
|
}
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
err = xenbus_printf(xbt, dev->nodename,
|
2012-11-02 09:43:04 -06:00
|
|
|
"feature-persistent", "%u", 1);
|
xen/blkback: Persistent grant maps for xen blk drivers
This patch implements persistent grants for the xen-blk{front,back}
mechanism. The effect of this change is to reduce the number of unmap
operations performed, since they cause a (costly) TLB shootdown. This
allows the I/O performance to scale better when a large number of VMs
are performing I/O.
Previously, the blkfront driver was supplied a bvec[] from the request
queue. This was granted to dom0; dom0 performed the I/O and wrote
directly into the grant-mapped memory and unmapped it; blkfront then
removed foreign access for that grant. The cost of unmapping scales
badly with the number of CPUs in Dom0. An experiment showed that when
Dom0 has 24 VCPUs, and guests are performing parallel I/O to a
ramdisk, the IPIs from performing unmap's is a bottleneck at 5 guests
(at which point 650,000 IOPS are being performed in total). If more
than 5 guests are used, the performance declines. By 10 guests, only
400,000 IOPS are being performed.
This patch improves performance by only unmapping when the connection
between blkfront and back is broken.
On startup blkfront notifies blkback that it is using persistent
grants, and blkback will do the same. If blkback is not capable of
persistent mapping, blkfront will still use the same grants, since it
is compatible with the previous protocol, and simplifies the code
complexity in blkfront.
To perform a read, in persistent mode, blkfront uses a separate pool
of pages that it maps to dom0. When a request comes in, blkfront
transmutes the request so that blkback will write into one of these
free pages. Blkback keeps note of which grefs it has already
mapped. When a new ring request comes to blkback, it looks to see if
it has already mapped that page. If so, it will not map it again. If
the page hasn't been previously mapped, it is mapped now, and a record
is kept of this mapping. Blkback proceeds as usual. When blkfront is
notified that blkback has completed a request, it memcpy's from the
shared memory, into the bvec supplied. A record that the {gref, page}
tuple is mapped, and not inflight is kept.
Writes are similar, except that the memcpy is peformed from the
supplied bvecs, into the shared pages, before the request is put onto
the ring.
Blkback stores a mapping of grefs=>{page mapped to by gref} in
a red-black tree. As the grefs are not known apriori, and provide no
guarantees on their ordering, we have to perform a search
through this tree to find the page, for every gref we receive. This
operation takes O(log n) time in the worst case. In blkfront grants
are stored using a single linked list.
The maximum number of grants that blkback will persistenly map is
currently set to RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, to
prevent a malicios guest from attempting a DoS, by supplying fresh
grefs, causing the Dom0 kernel to map excessively. If a guest
is using persistent grants and exceeds the maximum number of grants to
map persistenly the newly passed grefs will be mapped and unmaped.
Using this approach, we can have requests that mix persistent and
non-persistent grants, and we need to handle them correctly.
This allows us to set the maximum number of persistent grants to a
lower value than RING_SIZE * BLKIF_MAX_SEGMENTS_PER_REQUEST, although
setting it will lead to unpredictable performance.
In writing this patch, the question arrises as to if the additional
cost of performing memcpys in the guest (to/from the pool of granted
pages) outweigh the gains of not performing TLB shootdowns. The answer
to that question is `no'. There appears to be very little, if any
additional cost to the guest of using persistent grants. There is
perhaps a small saving, from the reduced number of hypercalls
performed in granting, and ending foreign access.
Signed-off-by: Oliver Chick <oliver.chick@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v1: Fixed up the misuse of bool as int]
2012-10-24 10:58:45 -06:00
|
|
|
if (err)
|
|
|
|
dev_warn(&dev->dev,
|
|
|
|
"writing persistent grants feature to xenbus");
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
err = xenbus_transaction_end(xbt, 0);
|
|
|
|
if (err) {
|
|
|
|
if (err == -EAGAIN)
|
|
|
|
goto again;
|
|
|
|
xenbus_dev_fatal(dev, err, "completing transaction");
|
|
|
|
goto destroy_blkring;
|
|
|
|
}
|
|
|
|
|
2015-11-13 20:12:12 -07:00
|
|
|
for (i = 0; i < info->nr_rings; i++) {
|
|
|
|
unsigned int j;
|
2015-11-13 20:12:14 -07:00
|
|
|
struct blkfront_ring_info *rinfo = &info->rinfo[i];
|
2015-11-13 20:12:12 -07:00
|
|
|
|
|
|
|
for (j = 0; j < BLK_RING_SIZE(info); j++)
|
|
|
|
rinfo->shadow[j].req.u.rw.id = j + 1;
|
|
|
|
rinfo->shadow[BLK_RING_SIZE(info)-1].req.u.rw.id = 0x0fffffff;
|
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
xenbus_switch_state(dev, XenbusStateInitialised);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
abort_transaction:
|
|
|
|
xenbus_transaction_end(xbt, 1);
|
|
|
|
if (message)
|
|
|
|
xenbus_dev_fatal(dev, err, "%s", message);
|
|
|
|
destroy_blkring:
|
|
|
|
blkif_free(info, 0);
|
2015-11-13 20:12:12 -07:00
|
|
|
|
2015-12-18 14:28:53 -07:00
|
|
|
kfree(info);
|
|
|
|
dev_set_drvdata(&dev->dev, NULL);
|
|
|
|
|
2007-07-17 19:37:06 -06:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2015-11-25 15:52:55 -07:00
|
|
|
static int negotiate_mq(struct blkfront_info *info)
|
|
|
|
{
|
|
|
|
unsigned int backend_max_queues = 0;
|
|
|
|
int err;
|
|
|
|
unsigned int i;
|
|
|
|
|
|
|
|
BUG_ON(info->nr_rings);
|
|
|
|
|
|
|
|
/* Check if backend supports multiple queues. */
|
|
|
|
err = xenbus_scanf(XBT_NIL, info->xbdev->otherend,
|
|
|
|
"multi-queue-max-queues", "%u", &backend_max_queues);
|
|
|
|
if (err < 0)
|
|
|
|
backend_max_queues = 1;
|
|
|
|
|
|
|
|
info->nr_rings = min(backend_max_queues, xen_blkif_max_queues);
|
|
|
|
/* We need at least one ring. */
|
|
|
|
if (!info->nr_rings)
|
|
|
|
info->nr_rings = 1;
|
|
|
|
|
|
|
|
info->rinfo = kzalloc(sizeof(struct blkfront_ring_info) * info->nr_rings, GFP_KERNEL);
|
|
|
|
if (!info->rinfo) {
|
|
|
|
xenbus_dev_fatal(info->xbdev, -ENOMEM, "allocating ring_info structure");
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
|
|
|
|
for (i = 0; i < info->nr_rings; i++) {
|
|
|
|
struct blkfront_ring_info *rinfo;
|
|
|
|
|
|
|
|
rinfo = &info->rinfo[i];
|
|
|
|
INIT_LIST_HEAD(&rinfo->indirect_pages);
|
|
|
|
INIT_LIST_HEAD(&rinfo->grants);
|
|
|
|
rinfo->dev_info = info;
|
|
|
|
INIT_WORK(&rinfo->work, blkif_restart_queue);
|
|
|
|
spin_lock_init(&rinfo->ring_lock);
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
/**
|
|
|
|
* Entry point to this code when a new device is created. Allocate the basic
|
|
|
|
* structures and the ring buffer for communication with the backend, and
|
|
|
|
* inform the backend of the appropriate details for those. Switch to
|
|
|
|
* Initialised state.
|
|
|
|
*/
|
|
|
|
static int blkfront_probe(struct xenbus_device *dev,
|
|
|
|
const struct xenbus_device_id *id)
|
|
|
|
{
|
2015-06-02 23:40:03 -06:00
|
|
|
int err, vdevice;
|
2007-07-17 19:37:06 -06:00
|
|
|
struct blkfront_info *info;
|
|
|
|
|
|
|
|
/* FIXME: Use dynamic device id if this is not set. */
|
|
|
|
err = xenbus_scanf(XBT_NIL, dev->nodename,
|
|
|
|
"virtual-device", "%i", &vdevice);
|
|
|
|
if (err != 1) {
|
2008-09-17 15:30:32 -06:00
|
|
|
/* go looking in the extended area instead */
|
|
|
|
err = xenbus_scanf(XBT_NIL, dev->nodename, "virtual-device-ext",
|
|
|
|
"%i", &vdevice);
|
|
|
|
if (err != 1) {
|
|
|
|
xenbus_dev_fatal(dev, err, "reading virtual-device");
|
|
|
|
return err;
|
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
|
2010-07-29 07:53:16 -06:00
|
|
|
if (xen_hvm_domain()) {
|
|
|
|
char *type;
|
|
|
|
int len;
|
|
|
|
/* no unplug has been done: do not hook devices != xen vbds */
|
xen/pvhvm: If xen_platform_pci=0 is set don't blow up (v4).
The user has the option of disabling the platform driver:
00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
which is used to unplug the emulated drivers (IDE, Realtek 8169, etc)
and allow the PV drivers to take over. If the user wishes
to disable that they can set:
xen_platform_pci=0
(in the guest config file)
or
xen_emul_unplug=never
(on the Linux command line)
except it does not work properly. The PV drivers still try to
load and since the Xen platform driver is not run - and it
has not initialized the grant tables, most of the PV drivers
stumble upon:
input: Xen Virtual Keyboard as /devices/virtual/input/input5
input: Xen Virtual Pointer as /devices/virtual/input/input6M
------------[ cut here ]------------
kernel BUG at /home/konrad/ssd/konrad/linux/drivers/xen/grant-table.c:1206!
invalid opcode: 0000 [#1] SMP
Modules linked in: xen_kbdfront(+) xenfs xen_privcmd
CPU: 6 PID: 1389 Comm: modprobe Not tainted 3.13.0-rc1upstream-00021-ga6c892b-dirty #1
Hardware name: Xen HVM domU, BIOS 4.4-unstable 11/26/2013
RIP: 0010:[<ffffffff813ddc40>] [<ffffffff813ddc40>] get_free_entries+0x2e0/0x300
Call Trace:
[<ffffffff8150d9a3>] ? evdev_connect+0x1e3/0x240
[<ffffffff813ddd0e>] gnttab_grant_foreign_access+0x2e/0x70
[<ffffffffa0010081>] xenkbd_connect_backend+0x41/0x290 [xen_kbdfront]
[<ffffffffa0010a12>] xenkbd_probe+0x2f2/0x324 [xen_kbdfront]
[<ffffffff813e5757>] xenbus_dev_probe+0x77/0x130
[<ffffffff813e7217>] xenbus_frontend_dev_probe+0x47/0x50
[<ffffffff8145e9a9>] driver_probe_device+0x89/0x230
[<ffffffff8145ebeb>] __driver_attach+0x9b/0xa0
[<ffffffff8145eb50>] ? driver_probe_device+0x230/0x230
[<ffffffff8145eb50>] ? driver_probe_device+0x230/0x230
[<ffffffff8145cf1c>] bus_for_each_dev+0x8c/0xb0
[<ffffffff8145e7d9>] driver_attach+0x19/0x20
[<ffffffff8145e260>] bus_add_driver+0x1a0/0x220
[<ffffffff8145f1ff>] driver_register+0x5f/0xf0
[<ffffffff813e55c5>] xenbus_register_driver_common+0x15/0x20
[<ffffffff813e76b3>] xenbus_register_frontend+0x23/0x40
[<ffffffffa0015000>] ? 0xffffffffa0014fff
[<ffffffffa001502b>] xenkbd_init+0x2b/0x1000 [xen_kbdfront]
[<ffffffff81002049>] do_one_initcall+0x49/0x170
.. snip..
which is hardly nice. This patch fixes this by having each
PV driver check for:
- if running in PV, then it is fine to execute (as that is their
native environment).
- if running in HVM, check if user wanted 'xen_emul_unplug=never',
in which case bail out and don't load any PV drivers.
- if running in HVM, and if PCI device 5853:0001 (xen_platform_pci)
does not exist, then bail out and not load PV drivers.
- (v2) if running in HVM, and if the user wanted 'xen_emul_unplug=ide-disks',
then bail out for all PV devices _except_ the block one.
Ditto for the network one ('nics').
- (v2) if running in HVM, and if the user wanted 'xen_emul_unplug=unnecessary'
then load block PV driver, and also setup the legacy IDE paths.
In (v3) make it actually load PV drivers.
Reported-by: Sander Eikelenboom <linux@eikelenboom.it
Reported-by: Anthony PERARD <anthony.perard@citrix.com>
Reported-and-Tested-by: Fabio Fantoni <fabio.fantoni@m2r.biz>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v2: Add extra logic to handle the myrid ways 'xen_emul_unplug'
can be used per Ian and Stefano suggestion]
[v3: Make the unnecessary case work properly]
[v4: s/disks/ide-disks/ spotted by Fabio]
Reviewed-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Bjorn Helgaas <bhelgaas@google.com> [for PCI parts]
CC: stable@vger.kernel.org
2013-11-26 13:05:40 -07:00
|
|
|
if (xen_has_pv_and_legacy_disk_devices()) {
|
2010-07-29 07:53:16 -06:00
|
|
|
int major;
|
|
|
|
|
|
|
|
if (!VDEV_IS_EXTENDED(vdevice))
|
|
|
|
major = BLKIF_MAJOR(vdevice);
|
|
|
|
else
|
|
|
|
major = XENVBD_MAJOR;
|
|
|
|
|
|
|
|
if (major != XENVBD_MAJOR) {
|
|
|
|
printk(KERN_INFO
|
|
|
|
"%s: HVM does not support vbd %d as xen block device\n",
|
2015-02-12 16:01:31 -07:00
|
|
|
__func__, vdevice);
|
2010-07-29 07:53:16 -06:00
|
|
|
return -ENODEV;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
/* do not create a PV cdrom device if we are an HVM guest */
|
|
|
|
type = xenbus_read(XBT_NIL, dev->nodename, "device-type", &len);
|
|
|
|
if (IS_ERR(type))
|
|
|
|
return -ENODEV;
|
|
|
|
if (strncmp(type, "cdrom", 5) == 0) {
|
|
|
|
kfree(type);
|
2010-05-14 05:44:30 -06:00
|
|
|
return -ENODEV;
|
|
|
|
}
|
2010-07-29 07:53:16 -06:00
|
|
|
kfree(type);
|
2010-05-14 05:44:30 -06:00
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
info = kzalloc(sizeof(*info), GFP_KERNEL);
|
|
|
|
if (!info) {
|
|
|
|
xenbus_dev_fatal(dev, -ENOMEM, "allocating info structure");
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
|
2015-11-13 20:12:14 -07:00
|
|
|
info->xbdev = dev;
|
2015-11-25 15:52:55 -07:00
|
|
|
err = negotiate_mq(info);
|
|
|
|
if (err) {
|
2015-11-13 20:12:12 -07:00
|
|
|
kfree(info);
|
2015-11-25 15:52:55 -07:00
|
|
|
return err;
|
2015-11-13 20:12:12 -07:00
|
|
|
}
|
2015-11-13 20:12:11 -07:00
|
|
|
|
2010-04-30 16:01:19 -06:00
|
|
|
mutex_init(&info->mutex);
|
2007-07-17 19:37:06 -06:00
|
|
|
info->vdevice = vdevice;
|
|
|
|
info->connected = BLKIF_STATE_DISCONNECTED;
|
|
|
|
|
|
|
|
/* Front end dir is a number, which is used as the id. */
|
|
|
|
info->handle = simple_strtoul(strrchr(dev->nodename, '/')+1, NULL, 0);
|
2009-04-30 15:43:31 -06:00
|
|
|
dev_set_drvdata(&dev->dev, info);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2015-07-20 07:29:37 -06:00
|
|
|
static void split_bio_end(struct bio *bio)
|
2013-04-18 08:06:54 -06:00
|
|
|
{
|
|
|
|
struct split_bio *split_bio = bio->bi_private;
|
|
|
|
|
|
|
|
if (atomic_dec_and_test(&split_bio->pending)) {
|
|
|
|
split_bio->bio->bi_phys_segments = 0;
|
2015-07-20 07:29:37 -06:00
|
|
|
split_bio->bio->bi_error = bio->bi_error;
|
|
|
|
bio_endio(split_bio->bio);
|
2013-04-18 08:06:54 -06:00
|
|
|
kfree(split_bio);
|
|
|
|
}
|
|
|
|
bio_put(bio);
|
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
static int blkif_recover(struct blkfront_info *info)
|
|
|
|
{
|
2015-11-13 20:12:12 -07:00
|
|
|
unsigned int i, r_index;
|
2013-04-18 08:06:54 -06:00
|
|
|
struct request *req, *n;
|
|
|
|
int rc;
|
|
|
|
struct bio *bio, *cloned_bio;
|
|
|
|
unsigned int segs, offset;
|
|
|
|
int pending, size;
|
|
|
|
struct split_bio *split_bio;
|
|
|
|
|
2015-11-13 20:12:12 -07:00
|
|
|
blkfront_gather_backend_features(info);
|
2016-07-01 15:43:39 -06:00
|
|
|
/* Reset limits changed by blk_mq_update_nr_hw_queues(). */
|
|
|
|
blkif_set_queue_limits(info);
|
2013-04-18 08:06:54 -06:00
|
|
|
segs = info->max_indirect_segments ? : BLKIF_MAX_SEGMENTS_PER_REQUEST;
|
2016-07-01 13:45:57 -06:00
|
|
|
blk_queue_max_segments(info->rq, segs / GRANTS_PER_PSEG);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2015-11-13 20:12:12 -07:00
|
|
|
for (r_index = 0; r_index < info->nr_rings; r_index++) {
|
2016-06-27 02:33:24 -06:00
|
|
|
struct blkfront_ring_info *rinfo = &info->rinfo[r_index];
|
2015-11-13 20:12:12 -07:00
|
|
|
|
|
|
|
rc = blkfront_setup_indirect(rinfo);
|
2016-06-27 02:33:24 -06:00
|
|
|
if (rc)
|
2015-11-13 20:12:12 -07:00
|
|
|
return rc;
|
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
xenbus_switch_state(info->xbdev, XenbusStateConnected);
|
|
|
|
|
|
|
|
/* Now safe for us to use the shared ring */
|
|
|
|
info->connected = BLKIF_STATE_CONNECTED;
|
|
|
|
|
2015-11-13 20:12:12 -07:00
|
|
|
for (r_index = 0; r_index < info->nr_rings; r_index++) {
|
|
|
|
struct blkfront_ring_info *rinfo;
|
|
|
|
|
|
|
|
rinfo = &info->rinfo[r_index];
|
|
|
|
/* Kick any other new requests queued since we resumed */
|
|
|
|
kick_pending_request_queues(rinfo);
|
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2016-06-27 02:33:24 -06:00
|
|
|
list_for_each_entry_safe(req, n, &info->requests, queuelist) {
|
2013-04-18 08:06:54 -06:00
|
|
|
/* Requeue pending requests (flush or discard) */
|
|
|
|
list_del_init(&req->queuelist);
|
|
|
|
BUG_ON(req->nr_phys_segments > segs);
|
2015-07-13 03:55:24 -06:00
|
|
|
blk_mq_requeue_request(req);
|
2013-04-18 08:06:54 -06:00
|
|
|
}
|
2015-07-13 03:55:24 -06:00
|
|
|
blk_mq_kick_requeue_list(info->rq);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2016-06-27 02:33:24 -06:00
|
|
|
while ((bio = bio_list_pop(&info->bio_list)) != NULL) {
|
2013-04-18 08:06:54 -06:00
|
|
|
/* Traverse the list of pending bios and re-queue them */
|
|
|
|
if (bio_segments(bio) > segs) {
|
|
|
|
/*
|
|
|
|
* This bio has more segments than what we can
|
|
|
|
* handle, we have to split it.
|
|
|
|
*/
|
|
|
|
pending = (bio_segments(bio) + segs - 1) / segs;
|
|
|
|
split_bio = kzalloc(sizeof(*split_bio), GFP_NOIO);
|
|
|
|
BUG_ON(split_bio == NULL);
|
|
|
|
atomic_set(&split_bio->pending, pending);
|
|
|
|
split_bio->bio = bio;
|
|
|
|
for (i = 0; i < pending; i++) {
|
2015-07-22 09:44:54 -06:00
|
|
|
offset = (i * segs * XEN_PAGE_SIZE) >> 9;
|
|
|
|
size = min((unsigned int)(segs * XEN_PAGE_SIZE) >> 9,
|
2013-10-11 16:44:27 -06:00
|
|
|
(unsigned int)bio_sectors(bio) - offset);
|
2013-04-18 08:06:54 -06:00
|
|
|
cloned_bio = bio_clone(bio, GFP_NOIO);
|
|
|
|
BUG_ON(cloned_bio == NULL);
|
2013-08-07 12:14:32 -06:00
|
|
|
bio_trim(cloned_bio, offset, size);
|
2013-04-18 08:06:54 -06:00
|
|
|
cloned_bio->bi_private = split_bio;
|
|
|
|
cloned_bio->bi_end_io = split_bio_end;
|
2016-06-05 13:31:41 -06:00
|
|
|
submit_bio(cloned_bio);
|
2013-04-18 08:06:54 -06:00
|
|
|
}
|
|
|
|
/*
|
|
|
|
* Now we have to wait for all those smaller bios to
|
|
|
|
* end, so we can also end the "parent" bio.
|
|
|
|
*/
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
/* We don't need to split this bio */
|
2016-06-05 13:31:41 -06:00
|
|
|
submit_bio(bio);
|
2013-04-18 08:06:54 -06:00
|
|
|
}
|
|
|
|
|
2007-07-17 19:37:06 -06:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* We are reconnecting to the backend, due to a suspend/resume, or a backend
|
|
|
|
* driver restart. We tear down our blkif structure and recreate it, but
|
|
|
|
* leave the device-layer structures intact so that this is transparent to the
|
|
|
|
* rest of the kernel.
|
|
|
|
*/
|
|
|
|
static int blkfront_resume(struct xenbus_device *dev)
|
|
|
|
{
|
2009-04-30 15:43:31 -06:00
|
|
|
struct blkfront_info *info = dev_get_drvdata(&dev->dev);
|
2015-11-25 15:52:55 -07:00
|
|
|
int err = 0;
|
2016-06-27 02:33:24 -06:00
|
|
|
unsigned int i, j;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
dev_dbg(&dev->dev, "blkfront_resume: %s\n", dev->nodename);
|
|
|
|
|
2016-06-27 02:33:24 -06:00
|
|
|
bio_list_init(&info->bio_list);
|
|
|
|
INIT_LIST_HEAD(&info->requests);
|
|
|
|
for (i = 0; i < info->nr_rings; i++) {
|
|
|
|
struct blkfront_ring_info *rinfo = &info->rinfo[i];
|
|
|
|
struct bio_list merge_bio;
|
|
|
|
struct blk_shadow *shadow = rinfo->shadow;
|
|
|
|
|
|
|
|
for (j = 0; j < BLK_RING_SIZE(info); j++) {
|
|
|
|
/* Not in use? */
|
|
|
|
if (!shadow[j].request)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Get the bios in the request so we can re-queue them.
|
|
|
|
*/
|
2016-07-26 16:03:07 -06:00
|
|
|
if (req_op(shadow[i].request) == REQ_OP_FLUSH ||
|
|
|
|
req_op(shadow[i].request) == REQ_OP_DISCARD ||
|
Merge branch 'for-4.8/drivers' of git://git.kernel.dk/linux-block
Pull block driver updates from Jens Axboe:
"This branch also contains core changes. I've come to the conclusion
that from 4.9 and forward, I'll be doing just a single branch. We
often have dependencies between core and drivers, and it's hard to
always split them up appropriately without pulling core into drivers
when that happens.
That said, this contains:
- separate secure erase type for the core block layer, from
Christoph.
- set of discard fixes, from Christoph.
- bio shrinking fixes from Christoph, as a followup up to the
op/flags change in the core branch.
- map and append request fixes from Christoph.
- NVMeF (NVMe over Fabrics) code from Christoph. This is pretty
exciting!
- nvme-loop fixes from Arnd.
- removal of ->driverfs_dev from Dan, after providing a
device_add_disk() helper.
- bcache fixes from Bhaktipriya and Yijing.
- cdrom subchannel read fix from Vchannaiah.
- set of lightnvm updates from Wenwei, Matias, Johannes, and Javier.
- set of drbd updates and fixes from Fabian, Lars, and Philipp.
- mg_disk error path fix from Bart.
- user notification for failed device add for loop, from Minfei.
- NVMe in general:
+ NVMe delay quirk from Guilherme.
+ SR-IOV support and command retry limits from Keith.
+ fix for memory-less NUMA node from Masayoshi.
+ use UINT_MAX for discard sectors, from Minfei.
+ cancel IO fixes from Ming.
+ don't allocate unused major, from Neil.
+ error code fixup from Dan.
+ use constants for PSDT/FUSE from James.
+ variable init fix from Jay.
+ fabrics fixes from Ming, Sagi, and Wei.
+ various fixes"
* 'for-4.8/drivers' of git://git.kernel.dk/linux-block: (115 commits)
nvme/pci: Provide SR-IOV support
nvme: initialize variable before logical OR'ing it
block: unexport various bio mapping helpers
scsi/osd: open code blk_make_request
target: stop using blk_make_request
block: simplify and export blk_rq_append_bio
block: ensure bios return from blk_get_request are properly initialized
virtio_blk: use blk_rq_map_kern
memstick: don't allow REQ_TYPE_BLOCK_PC requests
block: shrink bio size again
block: simplify and cleanup bvec pool handling
block: get rid of bio_rw and READA
block: don't ignore -EOPNOTSUPP blkdev_issue_write_same
block: introduce BLKDEV_DISCARD_ZERO to fix zeroout
NVMe: don't allocate unused nvme_major
nvme: avoid crashes when node 0 is memoryless node.
nvme: Limit command retries
loop: Make user notify for adding loop device failed
nvme-loop: fix nvme-loop Kconfig dependencies
nvmet: fix return value check in nvmet_subsys_alloc()
...
2016-07-26 16:37:51 -06:00
|
|
|
req_op(shadow[i].request) == REQ_OP_SECURE_ERASE ||
|
|
|
|
shadow[j].request->cmd_flags & REQ_FUA) {
|
2016-06-27 02:33:24 -06:00
|
|
|
/*
|
|
|
|
* Flush operations don't contain bios, so
|
|
|
|
* we need to requeue the whole request
|
Merge branch 'for-4.8/drivers' of git://git.kernel.dk/linux-block
Pull block driver updates from Jens Axboe:
"This branch also contains core changes. I've come to the conclusion
that from 4.9 and forward, I'll be doing just a single branch. We
often have dependencies between core and drivers, and it's hard to
always split them up appropriately without pulling core into drivers
when that happens.
That said, this contains:
- separate secure erase type for the core block layer, from
Christoph.
- set of discard fixes, from Christoph.
- bio shrinking fixes from Christoph, as a followup up to the
op/flags change in the core branch.
- map and append request fixes from Christoph.
- NVMeF (NVMe over Fabrics) code from Christoph. This is pretty
exciting!
- nvme-loop fixes from Arnd.
- removal of ->driverfs_dev from Dan, after providing a
device_add_disk() helper.
- bcache fixes from Bhaktipriya and Yijing.
- cdrom subchannel read fix from Vchannaiah.
- set of lightnvm updates from Wenwei, Matias, Johannes, and Javier.
- set of drbd updates and fixes from Fabian, Lars, and Philipp.
- mg_disk error path fix from Bart.
- user notification for failed device add for loop, from Minfei.
- NVMe in general:
+ NVMe delay quirk from Guilherme.
+ SR-IOV support and command retry limits from Keith.
+ fix for memory-less NUMA node from Masayoshi.
+ use UINT_MAX for discard sectors, from Minfei.
+ cancel IO fixes from Ming.
+ don't allocate unused major, from Neil.
+ error code fixup from Dan.
+ use constants for PSDT/FUSE from James.
+ variable init fix from Jay.
+ fabrics fixes from Ming, Sagi, and Wei.
+ various fixes"
* 'for-4.8/drivers' of git://git.kernel.dk/linux-block: (115 commits)
nvme/pci: Provide SR-IOV support
nvme: initialize variable before logical OR'ing it
block: unexport various bio mapping helpers
scsi/osd: open code blk_make_request
target: stop using blk_make_request
block: simplify and export blk_rq_append_bio
block: ensure bios return from blk_get_request are properly initialized
virtio_blk: use blk_rq_map_kern
memstick: don't allow REQ_TYPE_BLOCK_PC requests
block: shrink bio size again
block: simplify and cleanup bvec pool handling
block: get rid of bio_rw and READA
block: don't ignore -EOPNOTSUPP blkdev_issue_write_same
block: introduce BLKDEV_DISCARD_ZERO to fix zeroout
NVMe: don't allocate unused nvme_major
nvme: avoid crashes when node 0 is memoryless node.
nvme: Limit command retries
loop: Make user notify for adding loop device failed
nvme-loop: fix nvme-loop Kconfig dependencies
nvmet: fix return value check in nvmet_subsys_alloc()
...
2016-07-26 16:37:51 -06:00
|
|
|
*
|
|
|
|
* XXX: but this doesn't make any sense for a
|
|
|
|
* write with the FUA flag set..
|
2016-06-27 02:33:24 -06:00
|
|
|
*/
|
|
|
|
list_add(&shadow[j].request->queuelist, &info->requests);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
merge_bio.head = shadow[j].request->bio;
|
|
|
|
merge_bio.tail = shadow[j].request->biotail;
|
|
|
|
bio_list_merge(&info->bio_list, &merge_bio);
|
|
|
|
shadow[j].request->bio = NULL;
|
|
|
|
blk_mq_end_request(shadow[j].request, 0);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2007-07-17 19:37:06 -06:00
|
|
|
blkif_free(info, info->connected == BLKIF_STATE_CONNECTED);
|
|
|
|
|
2015-11-25 15:52:55 -07:00
|
|
|
err = negotiate_mq(info);
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
|
2009-12-04 08:33:54 -07:00
|
|
|
err = talk_to_blkback(dev, info);
|
2016-05-31 02:59:17 -06:00
|
|
|
if (!err)
|
|
|
|
blk_mq_update_nr_hw_queues(&info->tag_set, info->nr_rings);
|
2013-04-18 08:06:54 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We have to wait for the backend to switch to
|
|
|
|
* connected state, since we want to read which
|
|
|
|
* features it supports.
|
|
|
|
*/
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2015-11-16 13:14:41 -07:00
|
|
|
static void blkfront_closing(struct blkfront_info *info)
|
2010-04-30 16:01:19 -06:00
|
|
|
{
|
|
|
|
struct xenbus_device *xbdev = info->xbdev;
|
|
|
|
struct block_device *bdev = NULL;
|
|
|
|
|
|
|
|
mutex_lock(&info->mutex);
|
|
|
|
|
|
|
|
if (xbdev->state == XenbusStateClosing) {
|
|
|
|
mutex_unlock(&info->mutex);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (info->gd)
|
|
|
|
bdev = bdget_disk(info->gd, 0);
|
|
|
|
|
|
|
|
mutex_unlock(&info->mutex);
|
|
|
|
|
|
|
|
if (!bdev) {
|
|
|
|
xenbus_frontend_closed(xbdev);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
mutex_lock(&bdev->bd_mutex);
|
|
|
|
|
2010-04-30 16:01:23 -06:00
|
|
|
if (bdev->bd_openers) {
|
2010-04-30 16:01:19 -06:00
|
|
|
xenbus_dev_error(xbdev, -EBUSY,
|
|
|
|
"Device in use; refusing to close");
|
|
|
|
xenbus_switch_state(xbdev, XenbusStateClosing);
|
|
|
|
} else {
|
|
|
|
xlvbd_release_gendisk(info);
|
|
|
|
xenbus_frontend_closed(xbdev);
|
|
|
|
}
|
|
|
|
|
|
|
|
mutex_unlock(&bdev->bd_mutex);
|
|
|
|
bdput(bdev);
|
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2011-09-01 04:39:09 -06:00
|
|
|
static void blkfront_setup_discard(struct blkfront_info *info)
|
|
|
|
{
|
|
|
|
int err;
|
|
|
|
unsigned int discard_granularity;
|
|
|
|
unsigned int discard_alignment;
|
2011-10-12 14:23:30 -06:00
|
|
|
unsigned int discard_secure;
|
2011-09-01 04:39:09 -06:00
|
|
|
|
2014-05-21 08:32:40 -06:00
|
|
|
info->feature_discard = 1;
|
|
|
|
err = xenbus_gather(XBT_NIL, info->xbdev->otherend,
|
|
|
|
"discard-granularity", "%u", &discard_granularity,
|
|
|
|
"discard-alignment", "%u", &discard_alignment,
|
|
|
|
NULL);
|
|
|
|
if (!err) {
|
|
|
|
info->discard_granularity = discard_granularity;
|
|
|
|
info->discard_alignment = discard_alignment;
|
|
|
|
}
|
2016-07-07 02:05:46 -06:00
|
|
|
err = xenbus_scanf(XBT_NIL, info->xbdev->otherend,
|
|
|
|
"discard-secure", "%u", &discard_secure);
|
|
|
|
if (err > 0)
|
2014-05-21 08:32:40 -06:00
|
|
|
info->feature_secdiscard = !!discard_secure;
|
2011-09-01 04:39:09 -06:00
|
|
|
}
|
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
static int blkfront_setup_indirect(struct blkfront_ring_info *rinfo)
|
2013-04-18 08:06:54 -06:00
|
|
|
{
|
2015-07-22 09:44:54 -06:00
|
|
|
unsigned int psegs, grants;
|
2013-04-18 08:06:54 -06:00
|
|
|
int err, i;
|
2015-11-13 20:12:11 -07:00
|
|
|
struct blkfront_info *info = rinfo->dev_info;
|
2013-04-18 08:06:54 -06:00
|
|
|
|
xen/blkfront: Handle non-indirect grant with 64KB pages
The minimal size of request in the block framework is always PAGE_SIZE.
It means that when 64KB guest is support, the request will at least be
64KB.
Although, if the backend doesn't support indirect descriptor (such as QDISK
in QEMU), a ring request is only able to accommodate 11 segments of 4KB
(i.e 44KB).
The current frontend is assuming that an I/O request will always fit in
a ring request. This is not true any more when using 64KB page
granularity and will therefore crash during boot.
On ARM64, the ABI is completely neutral to the page granularity used by
the domU. The guest has the choice between different page granularity
supported by the processors (for instance on ARM64: 4KB, 16KB, 64KB).
This can't be enforced by the hypervisor and therefore it's possible to
run guests using different page granularity.
So we can't mandate the block backend to support indirect descriptor
when the frontend is using 64KB page granularity and have to fix it
properly in the frontend.
The solution exposed below is based on modifying directly the frontend
guest rather than asking the block framework to support smaller size
(i.e < PAGE_SIZE). This is because the change is the block framework are
not trivial as everything seems to relying on a struct *page (see [1]).
Although, it may be possible that someone succeed to do it in the future
and we would therefore be able to use it.
Given that a block request may not fit in a single ring request, a
second request is introduced for the data that cannot fit in the first
one. This means that the second ring request should never be used on
Linux if the page size is smaller than 44KB.
To achieve the support of the extra ring request, the block queue size
is divided by two. Therefore, the ring will always contain enough space
to accommodate 2 ring requests. While this will reduce the overall
performance, it will make the implementation more contained. The way
forward to get better performance is to implement in the backend either
indirect descriptor or multiple grants ring.
Note that the parameters blk_queue_max_* helpers haven't been updated.
The block code will set the mimimum size supported and we may be able
to support directly any change in the block framework that lower down
the minimal size of a request.
[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg02200.html
Signed-off-by: Julien Grall <julien.grall@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
2015-08-13 06:13:35 -06:00
|
|
|
if (info->max_indirect_segments == 0) {
|
|
|
|
if (!HAS_EXTRA_REQ)
|
|
|
|
grants = BLKIF_MAX_SEGMENTS_PER_REQUEST;
|
|
|
|
else {
|
|
|
|
/*
|
|
|
|
* When an extra req is required, the maximum
|
|
|
|
* grants supported is related to the size of the
|
|
|
|
* Linux block segment.
|
|
|
|
*/
|
|
|
|
grants = GRANTS_PER_PSEG;
|
|
|
|
}
|
|
|
|
}
|
2015-07-22 00:40:08 -06:00
|
|
|
else
|
2015-07-22 09:44:54 -06:00
|
|
|
grants = info->max_indirect_segments;
|
|
|
|
psegs = grants / GRANTS_PER_PSEG;
|
2013-04-18 08:06:54 -06:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
err = fill_grant_buffer(rinfo,
|
2015-07-22 09:44:54 -06:00
|
|
|
(grants + INDIRECT_GREFS(grants)) * BLK_RING_SIZE(info));
|
2013-04-18 08:06:54 -06:00
|
|
|
if (err)
|
|
|
|
goto out_of_memory;
|
|
|
|
|
2013-10-29 11:31:14 -06:00
|
|
|
if (!info->feature_persistent && info->max_indirect_segments) {
|
|
|
|
/*
|
|
|
|
* We are using indirect descriptors but not persistent
|
|
|
|
* grants, we need to allocate a set of pages that can be
|
|
|
|
* used for mapping indirect grefs
|
|
|
|
*/
|
2015-07-22 09:44:54 -06:00
|
|
|
int num = INDIRECT_GREFS(grants) * BLK_RING_SIZE(info);
|
2013-10-29 11:31:14 -06:00
|
|
|
|
2015-11-13 20:12:11 -07:00
|
|
|
BUG_ON(!list_empty(&rinfo->indirect_pages));
|
2013-10-29 11:31:14 -06:00
|
|
|
for (i = 0; i < num; i++) {
|
|
|
|
struct page *indirect_page = alloc_page(GFP_NOIO);
|
|
|
|
if (!indirect_page)
|
|
|
|
goto out_of_memory;
|
2015-11-13 20:12:11 -07:00
|
|
|
list_add(&indirect_page->lru, &rinfo->indirect_pages);
|
2013-10-29 11:31:14 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-06-02 23:40:03 -06:00
|
|
|
for (i = 0; i < BLK_RING_SIZE(info); i++) {
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo->shadow[i].grants_used = kzalloc(
|
|
|
|
sizeof(rinfo->shadow[i].grants_used[0]) * grants,
|
2013-04-18 08:06:54 -06:00
|
|
|
GFP_NOIO);
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo->shadow[i].sg = kzalloc(sizeof(rinfo->shadow[i].sg[0]) * psegs, GFP_NOIO);
|
2013-04-18 08:06:54 -06:00
|
|
|
if (info->max_indirect_segments)
|
2015-11-13 20:12:11 -07:00
|
|
|
rinfo->shadow[i].indirect_grants = kzalloc(
|
|
|
|
sizeof(rinfo->shadow[i].indirect_grants[0]) *
|
2015-07-22 09:44:54 -06:00
|
|
|
INDIRECT_GREFS(grants),
|
2013-04-18 08:06:54 -06:00
|
|
|
GFP_NOIO);
|
2015-11-13 20:12:11 -07:00
|
|
|
if ((rinfo->shadow[i].grants_used == NULL) ||
|
|
|
|
(rinfo->shadow[i].sg == NULL) ||
|
2013-04-18 08:06:54 -06:00
|
|
|
(info->max_indirect_segments &&
|
2015-11-13 20:12:11 -07:00
|
|
|
(rinfo->shadow[i].indirect_grants == NULL)))
|
2013-04-18 08:06:54 -06:00
|
|
|
goto out_of_memory;
|
2015-11-13 20:12:11 -07:00
|
|
|
sg_init_table(rinfo->shadow[i].sg, psegs);
|
2013-04-18 08:06:54 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
out_of_memory:
|
2015-06-02 23:40:03 -06:00
|
|
|
for (i = 0; i < BLK_RING_SIZE(info); i++) {
|
2015-11-13 20:12:11 -07:00
|
|
|
kfree(rinfo->shadow[i].grants_used);
|
|
|
|
rinfo->shadow[i].grants_used = NULL;
|
|
|
|
kfree(rinfo->shadow[i].sg);
|
|
|
|
rinfo->shadow[i].sg = NULL;
|
|
|
|
kfree(rinfo->shadow[i].indirect_grants);
|
|
|
|
rinfo->shadow[i].indirect_grants = NULL;
|
2013-04-18 08:06:54 -06:00
|
|
|
}
|
2015-11-13 20:12:11 -07:00
|
|
|
if (!list_empty(&rinfo->indirect_pages)) {
|
2013-10-29 11:31:14 -06:00
|
|
|
struct page *indirect_page, *n;
|
2015-11-13 20:12:11 -07:00
|
|
|
list_for_each_entry_safe(indirect_page, n, &rinfo->indirect_pages, lru) {
|
2013-10-29 11:31:14 -06:00
|
|
|
list_del(&indirect_page->lru);
|
|
|
|
__free_page(indirect_page);
|
|
|
|
}
|
|
|
|
}
|
2013-04-18 08:06:54 -06:00
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
|
2015-07-22 00:40:08 -06:00
|
|
|
/*
|
|
|
|
* Gather all backend feature-*
|
|
|
|
*/
|
2015-11-13 20:12:12 -07:00
|
|
|
static void blkfront_gather_backend_features(struct blkfront_info *info)
|
2015-07-22 00:40:08 -06:00
|
|
|
{
|
|
|
|
int err;
|
|
|
|
int barrier, flush, discard, persistent;
|
|
|
|
unsigned int indirect_segments;
|
|
|
|
|
|
|
|
info->feature_flush = 0;
|
2016-06-05 13:32:24 -06:00
|
|
|
info->feature_fua = 0;
|
2015-07-22 00:40:08 -06:00
|
|
|
|
2016-07-07 02:05:46 -06:00
|
|
|
err = xenbus_scanf(XBT_NIL, info->xbdev->otherend,
|
|
|
|
"feature-barrier", "%d", &barrier);
|
2015-07-22 00:40:08 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If there's no "feature-barrier" defined, then it means
|
|
|
|
* we're dealing with a very old backend which writes
|
|
|
|
* synchronously; nothing to do.
|
|
|
|
*
|
|
|
|
* If there are barriers, then we use flush.
|
|
|
|
*/
|
2016-07-27 12:35:37 -06:00
|
|
|
if (err > 0 && barrier) {
|
2016-06-05 13:32:24 -06:00
|
|
|
info->feature_flush = 1;
|
|
|
|
info->feature_fua = 1;
|
|
|
|
}
|
|
|
|
|
2015-07-22 00:40:08 -06:00
|
|
|
/*
|
|
|
|
* And if there is "feature-flush-cache" use that above
|
|
|
|
* barriers.
|
|
|
|
*/
|
2016-07-07 02:05:46 -06:00
|
|
|
err = xenbus_scanf(XBT_NIL, info->xbdev->otherend,
|
|
|
|
"feature-flush-cache", "%d", &flush);
|
2015-07-22 00:40:08 -06:00
|
|
|
|
2016-07-27 12:35:37 -06:00
|
|
|
if (err > 0 && flush) {
|
2016-06-05 13:32:24 -06:00
|
|
|
info->feature_flush = 1;
|
|
|
|
info->feature_fua = 0;
|
|
|
|
}
|
2015-07-22 00:40:08 -06:00
|
|
|
|
2016-07-07 02:05:46 -06:00
|
|
|
err = xenbus_scanf(XBT_NIL, info->xbdev->otherend,
|
|
|
|
"feature-discard", "%d", &discard);
|
2015-07-22 00:40:08 -06:00
|
|
|
|
2016-07-07 02:05:46 -06:00
|
|
|
if (err > 0 && discard)
|
2015-07-22 00:40:08 -06:00
|
|
|
blkfront_setup_discard(info);
|
|
|
|
|
2016-07-07 02:05:46 -06:00
|
|
|
err = xenbus_scanf(XBT_NIL, info->xbdev->otherend,
|
|
|
|
"feature-persistent", "%d", &persistent);
|
|
|
|
if (err <= 0)
|
2015-07-22 00:40:08 -06:00
|
|
|
info->feature_persistent = 0;
|
|
|
|
else
|
|
|
|
info->feature_persistent = persistent;
|
|
|
|
|
2016-07-07 02:05:46 -06:00
|
|
|
err = xenbus_scanf(XBT_NIL, info->xbdev->otherend,
|
|
|
|
"feature-max-indirect-segments", "%u",
|
|
|
|
&indirect_segments);
|
|
|
|
if (err <= 0)
|
2015-07-22 00:40:08 -06:00
|
|
|
info->max_indirect_segments = 0;
|
|
|
|
else
|
|
|
|
info->max_indirect_segments = min(indirect_segments,
|
|
|
|
xen_blkif_max_segments);
|
|
|
|
}
|
|
|
|
|
2007-07-17 19:37:06 -06:00
|
|
|
/*
|
|
|
|
* Invoked when the backend is finally 'ready' (and has told produced
|
|
|
|
* the details about the physical device - #sectors, size, etc).
|
|
|
|
*/
|
|
|
|
static void blkfront_connect(struct blkfront_info *info)
|
|
|
|
{
|
|
|
|
unsigned long long sectors;
|
|
|
|
unsigned long sector_size;
|
2013-05-13 08:28:15 -06:00
|
|
|
unsigned int physical_sector_size;
|
2007-07-17 19:37:06 -06:00
|
|
|
unsigned int binfo;
|
2015-11-13 20:12:12 -07:00
|
|
|
int err, i;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2010-03-11 14:42:26 -07:00
|
|
|
switch (info->connected) {
|
|
|
|
case BLKIF_STATE_CONNECTED:
|
|
|
|
/*
|
|
|
|
* Potentially, the back-end may be signalling
|
|
|
|
* a capacity change; update the capacity.
|
|
|
|
*/
|
|
|
|
err = xenbus_scanf(XBT_NIL, info->xbdev->otherend,
|
|
|
|
"sectors", "%Lu", §ors);
|
|
|
|
if (XENBUS_EXIST_ERR(err))
|
|
|
|
return;
|
|
|
|
printk(KERN_INFO "Setting capacity to %Lu\n",
|
|
|
|
sectors);
|
|
|
|
set_capacity(info->gd, sectors);
|
2010-03-18 16:00:54 -06:00
|
|
|
revalidate_disk(info->gd);
|
2010-03-11 14:42:26 -07:00
|
|
|
|
2013-04-18 08:06:54 -06:00
|
|
|
return;
|
2010-03-11 14:42:26 -07:00
|
|
|
case BLKIF_STATE_SUSPENDED:
|
2013-04-18 08:06:54 -06:00
|
|
|
/*
|
|
|
|
* If we are recovering from suspension, we need to wait
|
|
|
|
* for the backend to announce it's features before
|
|
|
|
* reconnecting, at least we need to know if the backend
|
|
|
|
* supports indirect descriptors, and how many.
|
|
|
|
*/
|
|
|
|
blkif_recover(info);
|
2007-07-17 19:37:06 -06:00
|
|
|
return;
|
|
|
|
|
2010-03-11 16:10:40 -07:00
|
|
|
default:
|
|
|
|
break;
|
2010-03-11 14:42:26 -07:00
|
|
|
}
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
dev_dbg(&info->xbdev->dev, "%s:%s.\n",
|
|
|
|
__func__, info->xbdev->otherend);
|
|
|
|
|
|
|
|
err = xenbus_gather(XBT_NIL, info->xbdev->otherend,
|
|
|
|
"sectors", "%llu", §ors,
|
|
|
|
"info", "%u", &binfo,
|
|
|
|
"sector-size", "%lu", §or_size,
|
|
|
|
NULL);
|
|
|
|
if (err) {
|
|
|
|
xenbus_dev_fatal(info->xbdev, err,
|
|
|
|
"reading backend fields at %s",
|
|
|
|
info->xbdev->otherend);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2013-05-13 08:28:15 -06:00
|
|
|
/*
|
|
|
|
* physcial-sector-size is a newer field, so old backends may not
|
|
|
|
* provide this. Assume physical sector size to be the same as
|
|
|
|
* sector_size in that case.
|
|
|
|
*/
|
|
|
|
err = xenbus_scanf(XBT_NIL, info->xbdev->otherend,
|
|
|
|
"physical-sector-size", "%u", &physical_sector_size);
|
|
|
|
if (err != 1)
|
|
|
|
physical_sector_size = sector_size;
|
|
|
|
|
2015-11-13 20:12:12 -07:00
|
|
|
blkfront_gather_backend_features(info);
|
|
|
|
for (i = 0; i < info->nr_rings; i++) {
|
|
|
|
err = blkfront_setup_indirect(&info->rinfo[i]);
|
|
|
|
if (err) {
|
|
|
|
xenbus_dev_fatal(info->xbdev, err, "setup_indirect at %s",
|
|
|
|
info->xbdev->otherend);
|
|
|
|
blkif_free(info, 0);
|
|
|
|
break;
|
|
|
|
}
|
2013-04-18 08:06:54 -06:00
|
|
|
}
|
|
|
|
|
2013-05-13 08:28:15 -06:00
|
|
|
err = xlvbd_alloc_gendisk(sectors, info, binfo, sector_size,
|
|
|
|
physical_sector_size);
|
2007-07-17 19:37:06 -06:00
|
|
|
if (err) {
|
|
|
|
xenbus_dev_fatal(info->xbdev, err, "xlvbd_add at %s",
|
|
|
|
info->xbdev->otherend);
|
2016-07-27 03:42:04 -06:00
|
|
|
goto fail;
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
xenbus_switch_state(info->xbdev, XenbusStateConnected);
|
|
|
|
|
|
|
|
/* Kick pending requests. */
|
|
|
|
info->connected = BLKIF_STATE_CONNECTED;
|
2015-11-13 20:12:12 -07:00
|
|
|
for (i = 0; i < info->nr_rings; i++)
|
|
|
|
kick_pending_request_queues(&info->rinfo[i]);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2016-06-15 20:44:20 -06:00
|
|
|
device_add_disk(&info->xbdev->dev, info->gd);
|
2008-04-02 11:54:04 -06:00
|
|
|
|
|
|
|
info->is_ready = 1;
|
2016-07-27 03:42:04 -06:00
|
|
|
return;
|
|
|
|
|
|
|
|
fail:
|
|
|
|
blkif_free(info, 0);
|
|
|
|
return;
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* Callback received when the backend's state changes.
|
|
|
|
*/
|
2009-12-04 08:33:54 -07:00
|
|
|
static void blkback_changed(struct xenbus_device *dev,
|
2007-07-17 19:37:06 -06:00
|
|
|
enum xenbus_state backend_state)
|
|
|
|
{
|
2009-04-30 15:43:31 -06:00
|
|
|
struct blkfront_info *info = dev_get_drvdata(&dev->dev);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2009-12-04 08:33:54 -07:00
|
|
|
dev_dbg(&dev->dev, "blkfront:blkback_changed to state %d.\n", backend_state);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
switch (backend_state) {
|
|
|
|
case XenbusStateInitWait:
|
2015-06-18 22:23:00 -06:00
|
|
|
if (dev->state != XenbusStateInitialising)
|
|
|
|
break;
|
2015-12-18 14:28:53 -07:00
|
|
|
if (talk_to_blkback(dev, info))
|
2015-06-02 23:40:02 -06:00
|
|
|
break;
|
|
|
|
case XenbusStateInitialising:
|
2007-07-17 19:37:06 -06:00
|
|
|
case XenbusStateInitialised:
|
2009-10-13 15:22:29 -06:00
|
|
|
case XenbusStateReconfiguring:
|
|
|
|
case XenbusStateReconfigured:
|
2007-07-17 19:37:06 -06:00
|
|
|
case XenbusStateUnknown:
|
|
|
|
break;
|
|
|
|
|
|
|
|
case XenbusStateConnected:
|
2016-06-07 08:43:15 -06:00
|
|
|
/*
|
|
|
|
* talk_to_blkback sets state to XenbusStateInitialised
|
|
|
|
* and blkfront_connect sets it to XenbusStateConnected
|
|
|
|
* (if connection went OK).
|
|
|
|
*
|
|
|
|
* If the backend (or toolstack) decides to poke at backend
|
|
|
|
* state (and re-trigger the watch by setting the state repeatedly
|
|
|
|
* to XenbusStateConnected (4)) we need to deal with this.
|
|
|
|
* This is allowed as this is used to communicate to the guest
|
|
|
|
* that the size of disk has changed!
|
|
|
|
*/
|
|
|
|
if ((dev->state != XenbusStateInitialised) &&
|
|
|
|
(dev->state != XenbusStateConnected)) {
|
2015-12-18 14:28:53 -07:00
|
|
|
if (talk_to_blkback(dev, info))
|
|
|
|
break;
|
|
|
|
}
|
2016-06-07 08:43:15 -06:00
|
|
|
|
2007-07-17 19:37:06 -06:00
|
|
|
blkfront_connect(info);
|
|
|
|
break;
|
|
|
|
|
2014-02-04 11:53:56 -07:00
|
|
|
case XenbusStateClosed:
|
|
|
|
if (dev->state == XenbusStateClosed)
|
|
|
|
break;
|
|
|
|
/* Missed the backend's Closing state -- fallthrough */
|
2007-07-17 19:37:06 -06:00
|
|
|
case XenbusStateClosing:
|
2015-10-02 07:35:01 -06:00
|
|
|
if (info)
|
|
|
|
blkfront_closing(info);
|
2007-07-17 19:37:06 -06:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2010-04-30 16:01:22 -06:00
|
|
|
static int blkfront_remove(struct xenbus_device *xbdev)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
2010-04-30 16:01:22 -06:00
|
|
|
struct blkfront_info *info = dev_get_drvdata(&xbdev->dev);
|
|
|
|
struct block_device *bdev = NULL;
|
|
|
|
struct gendisk *disk;
|
2007-07-17 19:37:06 -06:00
|
|
|
|
2010-04-30 16:01:22 -06:00
|
|
|
dev_dbg(&xbdev->dev, "%s removed", xbdev->nodename);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
blkif_free(info, 0);
|
|
|
|
|
2010-04-30 16:01:22 -06:00
|
|
|
mutex_lock(&info->mutex);
|
|
|
|
|
|
|
|
disk = info->gd;
|
|
|
|
if (disk)
|
|
|
|
bdev = bdget_disk(disk, 0);
|
|
|
|
|
|
|
|
info->xbdev = NULL;
|
|
|
|
mutex_unlock(&info->mutex);
|
|
|
|
|
|
|
|
if (!bdev) {
|
|
|
|
kfree(info);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The xbdev was removed before we reached the Closed
|
|
|
|
* state. See if it's safe to remove the disk. If the bdev
|
|
|
|
* isn't closed yet, we let release take care of it.
|
|
|
|
*/
|
|
|
|
|
|
|
|
mutex_lock(&bdev->bd_mutex);
|
|
|
|
info = disk->private_data;
|
|
|
|
|
2010-08-07 10:51:21 -06:00
|
|
|
dev_warn(disk_to_dev(disk),
|
|
|
|
"%s was hot-unplugged, %d stale handles\n",
|
|
|
|
xbdev->nodename, bdev->bd_openers);
|
|
|
|
|
2010-04-30 16:01:23 -06:00
|
|
|
if (info && !bdev->bd_openers) {
|
2010-04-30 16:01:22 -06:00
|
|
|
xlvbd_release_gendisk(info);
|
|
|
|
disk->private_data = NULL;
|
2010-08-07 10:28:55 -06:00
|
|
|
kfree(info);
|
2010-04-30 16:01:22 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
mutex_unlock(&bdev->bd_mutex);
|
|
|
|
bdput(bdev);
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2008-04-02 11:54:04 -06:00
|
|
|
static int blkfront_is_ready(struct xenbus_device *dev)
|
|
|
|
{
|
2009-04-30 15:43:31 -06:00
|
|
|
struct blkfront_info *info = dev_get_drvdata(&dev->dev);
|
2008-04-02 11:54:04 -06:00
|
|
|
|
2010-08-07 10:31:12 -06:00
|
|
|
return info->is_ready && info->xbdev;
|
2008-04-02 11:54:04 -06:00
|
|
|
}
|
|
|
|
|
2008-03-02 08:23:47 -07:00
|
|
|
static int blkif_open(struct block_device *bdev, fmode_t mode)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
2010-08-07 10:36:53 -06:00
|
|
|
struct gendisk *disk = bdev->bd_disk;
|
|
|
|
struct blkfront_info *info;
|
|
|
|
int err = 0;
|
2010-08-07 10:25:34 -06:00
|
|
|
|
2010-06-02 06:28:52 -06:00
|
|
|
mutex_lock(&blkfront_mutex);
|
2010-08-07 10:25:34 -06:00
|
|
|
|
2010-08-07 10:36:53 -06:00
|
|
|
info = disk->private_data;
|
|
|
|
if (!info) {
|
|
|
|
/* xbdev gone */
|
|
|
|
err = -ERESTARTSYS;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
mutex_lock(&info->mutex);
|
|
|
|
|
|
|
|
if (!info->gd)
|
|
|
|
/* xbdev is closed */
|
|
|
|
err = -ERESTARTSYS;
|
|
|
|
|
|
|
|
mutex_unlock(&info->mutex);
|
|
|
|
|
|
|
|
out:
|
2010-06-02 06:28:52 -06:00
|
|
|
mutex_unlock(&blkfront_mutex);
|
2010-08-07 10:36:53 -06:00
|
|
|
return err;
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
|
2013-05-05 19:52:57 -06:00
|
|
|
static void blkif_release(struct gendisk *disk, fmode_t mode)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
2008-03-02 08:23:47 -07:00
|
|
|
struct blkfront_info *info = disk->private_data;
|
2010-08-07 10:45:12 -06:00
|
|
|
struct block_device *bdev;
|
|
|
|
struct xenbus_device *xbdev;
|
|
|
|
|
2010-06-02 06:28:52 -06:00
|
|
|
mutex_lock(&blkfront_mutex);
|
2010-08-07 10:45:12 -06:00
|
|
|
|
|
|
|
bdev = bdget_disk(disk, 0);
|
|
|
|
|
2013-11-09 08:36:09 -07:00
|
|
|
if (!bdev) {
|
|
|
|
WARN(1, "Block device %s yanked out from us!\n", disk->disk_name);
|
|
|
|
goto out_mutex;
|
|
|
|
}
|
2010-08-07 10:47:26 -06:00
|
|
|
if (bdev->bd_openers)
|
|
|
|
goto out;
|
|
|
|
|
2010-08-07 10:45:12 -06:00
|
|
|
/*
|
|
|
|
* Check if we have been instructed to close. We will have
|
|
|
|
* deferred this request, because the bdev was still open.
|
|
|
|
*/
|
|
|
|
|
|
|
|
mutex_lock(&info->mutex);
|
|
|
|
xbdev = info->xbdev;
|
|
|
|
|
|
|
|
if (xbdev && xbdev->state == XenbusStateClosing) {
|
|
|
|
/* pending switch to state closed */
|
2010-08-07 10:51:21 -06:00
|
|
|
dev_info(disk_to_dev(bdev->bd_disk), "releasing disk\n");
|
2010-08-07 10:45:12 -06:00
|
|
|
xlvbd_release_gendisk(info);
|
|
|
|
xenbus_frontend_closed(info->xbdev);
|
|
|
|
}
|
|
|
|
|
|
|
|
mutex_unlock(&info->mutex);
|
|
|
|
|
|
|
|
if (!xbdev) {
|
|
|
|
/* sudden device removal */
|
2010-08-07 10:51:21 -06:00
|
|
|
dev_info(disk_to_dev(bdev->bd_disk), "releasing disk\n");
|
2010-08-07 10:45:12 -06:00
|
|
|
xlvbd_release_gendisk(info);
|
|
|
|
disk->private_data = NULL;
|
|
|
|
kfree(info);
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
2010-08-07 10:45:12 -06:00
|
|
|
|
2010-08-08 19:50:05 -06:00
|
|
|
out:
|
2012-02-16 05:16:25 -07:00
|
|
|
bdput(bdev);
|
2013-11-09 08:36:09 -07:00
|
|
|
out_mutex:
|
2010-06-02 06:28:52 -06:00
|
|
|
mutex_unlock(&blkfront_mutex);
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
|
2009-09-21 18:01:13 -06:00
|
|
|
static const struct block_device_operations xlvbd_block_fops =
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
|
|
|
.owner = THIS_MODULE,
|
2008-03-02 08:23:47 -07:00
|
|
|
.open = blkif_open,
|
|
|
|
.release = blkif_release,
|
2008-02-21 14:03:45 -07:00
|
|
|
.getgeo = blkif_getgeo,
|
2010-07-08 02:18:46 -06:00
|
|
|
.ioctl = blkif_ioctl,
|
2007-07-17 19:37:06 -06:00
|
|
|
};
|
|
|
|
|
|
|
|
|
2010-01-10 05:39:52 -07:00
|
|
|
static const struct xenbus_device_id blkfront_ids[] = {
|
2007-07-17 19:37:06 -06:00
|
|
|
{ "vbd" },
|
|
|
|
{ "" }
|
|
|
|
};
|
|
|
|
|
2014-09-08 10:30:41 -06:00
|
|
|
static struct xenbus_driver blkfront_driver = {
|
|
|
|
.ids = blkfront_ids,
|
2007-07-17 19:37:06 -06:00
|
|
|
.probe = blkfront_probe,
|
|
|
|
.remove = blkfront_remove,
|
|
|
|
.resume = blkfront_resume,
|
2009-12-04 08:33:54 -07:00
|
|
|
.otherend_changed = blkback_changed,
|
2008-04-02 11:54:04 -06:00
|
|
|
.is_ready = blkfront_is_ready,
|
2014-09-08 10:30:41 -06:00
|
|
|
};
|
2007-07-17 19:37:06 -06:00
|
|
|
|
|
|
|
static int __init xlblk_init(void)
|
|
|
|
{
|
2011-10-07 13:34:38 -06:00
|
|
|
int ret;
|
2015-11-13 20:12:14 -07:00
|
|
|
int nr_cpus = num_online_cpus();
|
2011-10-07 13:34:38 -06:00
|
|
|
|
2008-08-19 14:16:17 -06:00
|
|
|
if (!xen_domain())
|
2007-07-17 19:37:06 -06:00
|
|
|
return -ENODEV;
|
|
|
|
|
2015-10-13 10:50:11 -06:00
|
|
|
if (xen_blkif_max_ring_order > XENBUS_MAX_RING_GRANT_ORDER) {
|
2015-06-02 23:40:03 -06:00
|
|
|
pr_info("Invalid max_ring_order (%d), will use default max: %d.\n",
|
2015-10-13 10:50:11 -06:00
|
|
|
xen_blkif_max_ring_order, XENBUS_MAX_RING_GRANT_ORDER);
|
2015-11-25 03:26:01 -07:00
|
|
|
xen_blkif_max_ring_order = XENBUS_MAX_RING_GRANT_ORDER;
|
2015-06-02 23:40:03 -06:00
|
|
|
}
|
|
|
|
|
2015-11-13 20:12:14 -07:00
|
|
|
if (xen_blkif_max_queues > nr_cpus) {
|
|
|
|
pr_info("Invalid max_queues (%d), will use default max: %d.\n",
|
|
|
|
xen_blkif_max_queues, nr_cpus);
|
|
|
|
xen_blkif_max_queues = nr_cpus;
|
|
|
|
}
|
|
|
|
|
xen/pvhvm: If xen_platform_pci=0 is set don't blow up (v4).
The user has the option of disabling the platform driver:
00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
which is used to unplug the emulated drivers (IDE, Realtek 8169, etc)
and allow the PV drivers to take over. If the user wishes
to disable that they can set:
xen_platform_pci=0
(in the guest config file)
or
xen_emul_unplug=never
(on the Linux command line)
except it does not work properly. The PV drivers still try to
load and since the Xen platform driver is not run - and it
has not initialized the grant tables, most of the PV drivers
stumble upon:
input: Xen Virtual Keyboard as /devices/virtual/input/input5
input: Xen Virtual Pointer as /devices/virtual/input/input6M
------------[ cut here ]------------
kernel BUG at /home/konrad/ssd/konrad/linux/drivers/xen/grant-table.c:1206!
invalid opcode: 0000 [#1] SMP
Modules linked in: xen_kbdfront(+) xenfs xen_privcmd
CPU: 6 PID: 1389 Comm: modprobe Not tainted 3.13.0-rc1upstream-00021-ga6c892b-dirty #1
Hardware name: Xen HVM domU, BIOS 4.4-unstable 11/26/2013
RIP: 0010:[<ffffffff813ddc40>] [<ffffffff813ddc40>] get_free_entries+0x2e0/0x300
Call Trace:
[<ffffffff8150d9a3>] ? evdev_connect+0x1e3/0x240
[<ffffffff813ddd0e>] gnttab_grant_foreign_access+0x2e/0x70
[<ffffffffa0010081>] xenkbd_connect_backend+0x41/0x290 [xen_kbdfront]
[<ffffffffa0010a12>] xenkbd_probe+0x2f2/0x324 [xen_kbdfront]
[<ffffffff813e5757>] xenbus_dev_probe+0x77/0x130
[<ffffffff813e7217>] xenbus_frontend_dev_probe+0x47/0x50
[<ffffffff8145e9a9>] driver_probe_device+0x89/0x230
[<ffffffff8145ebeb>] __driver_attach+0x9b/0xa0
[<ffffffff8145eb50>] ? driver_probe_device+0x230/0x230
[<ffffffff8145eb50>] ? driver_probe_device+0x230/0x230
[<ffffffff8145cf1c>] bus_for_each_dev+0x8c/0xb0
[<ffffffff8145e7d9>] driver_attach+0x19/0x20
[<ffffffff8145e260>] bus_add_driver+0x1a0/0x220
[<ffffffff8145f1ff>] driver_register+0x5f/0xf0
[<ffffffff813e55c5>] xenbus_register_driver_common+0x15/0x20
[<ffffffff813e76b3>] xenbus_register_frontend+0x23/0x40
[<ffffffffa0015000>] ? 0xffffffffa0014fff
[<ffffffffa001502b>] xenkbd_init+0x2b/0x1000 [xen_kbdfront]
[<ffffffff81002049>] do_one_initcall+0x49/0x170
.. snip..
which is hardly nice. This patch fixes this by having each
PV driver check for:
- if running in PV, then it is fine to execute (as that is their
native environment).
- if running in HVM, check if user wanted 'xen_emul_unplug=never',
in which case bail out and don't load any PV drivers.
- if running in HVM, and if PCI device 5853:0001 (xen_platform_pci)
does not exist, then bail out and not load PV drivers.
- (v2) if running in HVM, and if the user wanted 'xen_emul_unplug=ide-disks',
then bail out for all PV devices _except_ the block one.
Ditto for the network one ('nics').
- (v2) if running in HVM, and if the user wanted 'xen_emul_unplug=unnecessary'
then load block PV driver, and also setup the legacy IDE paths.
In (v3) make it actually load PV drivers.
Reported-by: Sander Eikelenboom <linux@eikelenboom.it
Reported-by: Anthony PERARD <anthony.perard@citrix.com>
Reported-and-Tested-by: Fabio Fantoni <fabio.fantoni@m2r.biz>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
[v2: Add extra logic to handle the myrid ways 'xen_emul_unplug'
can be used per Ian and Stefano suggestion]
[v3: Make the unnecessary case work properly]
[v4: s/disks/ide-disks/ spotted by Fabio]
Reviewed-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Bjorn Helgaas <bhelgaas@google.com> [for PCI parts]
CC: stable@vger.kernel.org
2013-11-26 13:05:40 -07:00
|
|
|
if (!xen_has_pv_disk_devices())
|
2012-03-21 08:08:38 -06:00
|
|
|
return -ENODEV;
|
|
|
|
|
2007-07-17 19:37:06 -06:00
|
|
|
if (register_blkdev(XENVBD_MAJOR, DEV_NAME)) {
|
|
|
|
printk(KERN_WARNING "xen_blk: can't get major %d with name %s\n",
|
|
|
|
XENVBD_MAJOR, DEV_NAME);
|
|
|
|
return -ENODEV;
|
|
|
|
}
|
|
|
|
|
2011-12-22 02:08:13 -07:00
|
|
|
ret = xenbus_register_frontend(&blkfront_driver);
|
2011-10-07 13:34:38 -06:00
|
|
|
if (ret) {
|
|
|
|
unregister_blkdev(XENVBD_MAJOR, DEV_NAME);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
module_init(xlblk_init);
|
|
|
|
|
|
|
|
|
2008-06-17 02:47:08 -06:00
|
|
|
static void __exit xlblk_exit(void)
|
2007-07-17 19:37:06 -06:00
|
|
|
{
|
2012-04-05 09:04:52 -06:00
|
|
|
xenbus_unregister_driver(&blkfront_driver);
|
|
|
|
unregister_blkdev(XENVBD_MAJOR, DEV_NAME);
|
|
|
|
kfree(minors);
|
2007-07-17 19:37:06 -06:00
|
|
|
}
|
|
|
|
module_exit(xlblk_exit);
|
|
|
|
|
|
|
|
MODULE_DESCRIPTION("Xen virtual block device frontend");
|
|
|
|
MODULE_LICENSE("GPL");
|
|
|
|
MODULE_ALIAS_BLOCKDEV_MAJOR(XENVBD_MAJOR);
|
2008-04-02 11:54:05 -06:00
|
|
|
MODULE_ALIAS("xen:vbd");
|
2008-04-02 11:54:06 -06:00
|
|
|
MODULE_ALIAS("xenblk");
|