2011-11-03 02:20:04 -06:00
|
|
|
/*
|
|
|
|
* Virtio balloon implementation, inspired by Dor Laor and Marcelo
|
2008-02-04 21:50:12 -07:00
|
|
|
* Tosatti's implementations.
|
|
|
|
*
|
|
|
|
* Copyright 2008 Rusty Russell IBM Corporation
|
|
|
|
*
|
|
|
|
* This program is free software; you can redistribute it and/or modify
|
|
|
|
* it under the terms of the GNU General Public License as published by
|
|
|
|
* the Free Software Foundation; either version 2 of the License, or
|
|
|
|
* (at your option) any later version.
|
|
|
|
*
|
|
|
|
* This program is distributed in the hope that it will be useful,
|
|
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
* GNU General Public License for more details.
|
|
|
|
*
|
|
|
|
* You should have received a copy of the GNU General Public License
|
|
|
|
* along with this program; if not, write to the Free Software
|
|
|
|
* Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
|
|
|
|
*/
|
2011-11-03 02:20:04 -06:00
|
|
|
|
2008-02-04 21:50:12 -07:00
|
|
|
#include <linux/virtio.h>
|
|
|
|
#include <linux/virtio_balloon.h>
|
|
|
|
#include <linux/swap.h>
|
|
|
|
#include <linux/kthread.h>
|
|
|
|
#include <linux/freezer.h>
|
2008-02-06 02:40:22 -07:00
|
|
|
#include <linux/delay.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 02:04:11 -06:00
|
|
|
#include <linux/slab.h>
|
2011-07-03 14:20:30 -06:00
|
|
|
#include <linux/module.h>
|
2012-12-11 17:02:45 -07:00
|
|
|
#include <linux/balloon_compaction.h>
|
2008-02-04 21:50:12 -07:00
|
|
|
|
2012-04-12 07:38:00 -06:00
|
|
|
/*
|
|
|
|
* Balloon device works in 4K page units. So each page is pointed to by
|
|
|
|
* multiple balloon pages. All memory counters in this driver are in balloon
|
|
|
|
* page units.
|
|
|
|
*/
|
2012-12-11 17:02:45 -07:00
|
|
|
#define VIRTIO_BALLOON_PAGES_PER_PAGE (unsigned)(PAGE_SIZE >> VIRTIO_BALLOON_PFN_SHIFT)
|
|
|
|
#define VIRTIO_BALLOON_ARRAY_PFNS_MAX 256
|
2012-04-12 07:38:00 -06:00
|
|
|
|
2008-02-04 21:50:12 -07:00
|
|
|
struct virtio_balloon
|
|
|
|
{
|
|
|
|
struct virtio_device *vdev;
|
2009-11-30 09:14:15 -07:00
|
|
|
struct virtqueue *inflate_vq, *deflate_vq, *stats_vq;
|
2008-02-04 21:50:12 -07:00
|
|
|
|
|
|
|
/* Where the ballooning thread waits for config to change. */
|
|
|
|
wait_queue_head_t config_change;
|
|
|
|
|
|
|
|
/* The thread servicing the balloon. */
|
|
|
|
struct task_struct *thread;
|
|
|
|
|
|
|
|
/* Waiting for host to ack the pages we released. */
|
2012-07-02 01:33:08 -06:00
|
|
|
wait_queue_head_t acked;
|
2008-02-04 21:50:12 -07:00
|
|
|
|
2012-04-12 07:38:00 -06:00
|
|
|
/* Number of balloon pages we've told the Host we're not using. */
|
2008-02-04 21:50:12 -07:00
|
|
|
unsigned int num_pages;
|
2012-04-12 07:38:00 -06:00
|
|
|
/*
|
2012-12-11 17:02:45 -07:00
|
|
|
* The pages we've told the Host we're not using are enqueued
|
|
|
|
* at vb_dev_info->pages list.
|
2012-04-12 07:38:00 -06:00
|
|
|
* Each page on this list adds VIRTIO_BALLOON_PAGES_PER_PAGE
|
|
|
|
* to num_pages above.
|
|
|
|
*/
|
2012-12-11 17:02:45 -07:00
|
|
|
struct balloon_dev_info *vb_dev_info;
|
|
|
|
|
|
|
|
/* Synchronize access/update to this struct virtio_balloon elements */
|
|
|
|
struct mutex balloon_lock;
|
2008-02-04 21:50:12 -07:00
|
|
|
|
|
|
|
/* The array of pfns we tell the Host about. */
|
|
|
|
unsigned int num_pfns;
|
2012-12-11 17:02:45 -07:00
|
|
|
u32 pfns[VIRTIO_BALLOON_ARRAY_PFNS_MAX];
|
2009-11-30 09:14:15 -07:00
|
|
|
|
|
|
|
/* Memory statistics */
|
2009-12-10 15:35:15 -07:00
|
|
|
int need_stats_update;
|
2009-11-30 09:14:15 -07:00
|
|
|
struct virtio_balloon_stat stats[VIRTIO_BALLOON_S_NR];
|
2008-02-04 21:50:12 -07:00
|
|
|
};
|
|
|
|
|
|
|
|
static struct virtio_device_id id_table[] = {
|
|
|
|
{ VIRTIO_ID_BALLOON, VIRTIO_DEV_ANY_ID },
|
|
|
|
{ 0 },
|
|
|
|
};
|
|
|
|
|
2008-11-13 14:48:33 -07:00
|
|
|
static u32 page_to_balloon_pfn(struct page *page)
|
|
|
|
{
|
|
|
|
unsigned long pfn = page_to_pfn(page);
|
|
|
|
|
|
|
|
BUILD_BUG_ON(PAGE_SHIFT < VIRTIO_BALLOON_PFN_SHIFT);
|
|
|
|
/* Convert pfn from Linux page size to balloon page size. */
|
2012-04-12 07:38:00 -06:00
|
|
|
return pfn * VIRTIO_BALLOON_PAGES_PER_PAGE;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct page *balloon_pfn_to_page(u32 pfn)
|
|
|
|
{
|
|
|
|
BUG_ON(pfn % VIRTIO_BALLOON_PAGES_PER_PAGE);
|
|
|
|
return pfn_to_page(pfn / VIRTIO_BALLOON_PAGES_PER_PAGE);
|
2008-11-13 14:48:33 -07:00
|
|
|
}
|
|
|
|
|
2008-02-04 21:50:12 -07:00
|
|
|
static void balloon_ack(struct virtqueue *vq)
|
|
|
|
{
|
2012-07-02 01:33:08 -06:00
|
|
|
struct virtio_balloon *vb = vq->vdev->priv;
|
2008-02-04 21:50:12 -07:00
|
|
|
|
2012-07-02 01:33:08 -06:00
|
|
|
wake_up(&vb->acked);
|
2008-02-04 21:50:12 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq)
|
|
|
|
{
|
|
|
|
struct scatterlist sg;
|
2012-07-02 01:33:08 -06:00
|
|
|
unsigned int len;
|
2008-02-04 21:50:12 -07:00
|
|
|
|
|
|
|
sg_init_one(&sg, vb->pfns, sizeof(vb->pfns[0]) * vb->num_pfns);
|
|
|
|
|
|
|
|
/* We should always be able to add one buffer to an empty queue. */
|
2012-01-11 22:14:42 -07:00
|
|
|
if (virtqueue_add_buf(vq, &sg, 1, 0, vb, GFP_KERNEL) < 0)
|
2008-02-04 21:50:12 -07:00
|
|
|
BUG();
|
2010-04-12 07:18:28 -06:00
|
|
|
virtqueue_kick(vq);
|
2008-02-04 21:50:12 -07:00
|
|
|
|
|
|
|
/* When host has read buffer, this completes via balloon_ack */
|
2012-07-02 01:33:08 -06:00
|
|
|
wait_event(vb->acked, virtqueue_get_buf(vq, &len));
|
2008-02-04 21:50:12 -07:00
|
|
|
}
|
|
|
|
|
2012-04-12 07:38:00 -06:00
|
|
|
static void set_page_pfns(u32 pfns[], struct page *page)
|
|
|
|
{
|
|
|
|
unsigned int i;
|
|
|
|
|
|
|
|
/* Set balloon pfns pointing at this page.
|
|
|
|
* Note that the first pfn points at start of the page. */
|
|
|
|
for (i = 0; i < VIRTIO_BALLOON_PAGES_PER_PAGE; i++)
|
|
|
|
pfns[i] = page_to_balloon_pfn(page) + i;
|
|
|
|
}
|
|
|
|
|
2008-02-04 21:50:12 -07:00
|
|
|
static void fill_balloon(struct virtio_balloon *vb, size_t num)
|
|
|
|
{
|
2012-12-11 17:02:45 -07:00
|
|
|
struct balloon_dev_info *vb_dev_info = vb->vb_dev_info;
|
|
|
|
|
2008-02-04 21:50:12 -07:00
|
|
|
/* We can only do one array worth at a time. */
|
|
|
|
num = min(num, ARRAY_SIZE(vb->pfns));
|
|
|
|
|
2012-12-11 17:02:45 -07:00
|
|
|
mutex_lock(&vb->balloon_lock);
|
2012-04-12 07:38:00 -06:00
|
|
|
for (vb->num_pfns = 0; vb->num_pfns < num;
|
|
|
|
vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE) {
|
2012-12-11 17:02:45 -07:00
|
|
|
struct page *page = balloon_page_enqueue(vb_dev_info);
|
|
|
|
|
2008-02-04 21:50:12 -07:00
|
|
|
if (!page) {
|
2012-10-30 16:57:22 -06:00
|
|
|
dev_info_ratelimited(&vb->vdev->dev,
|
2012-12-20 09:37:04 -07:00
|
|
|
"Out of puff! Can't get %u pages\n",
|
|
|
|
VIRTIO_BALLOON_PAGES_PER_PAGE);
|
2008-02-04 21:50:12 -07:00
|
|
|
/* Sleep for at least 1/5 of a second before retry. */
|
|
|
|
msleep(200);
|
|
|
|
break;
|
|
|
|
}
|
2012-04-12 07:38:00 -06:00
|
|
|
set_page_pfns(vb->pfns + vb->num_pfns, page);
|
|
|
|
vb->num_pages += VIRTIO_BALLOON_PAGES_PER_PAGE;
|
2008-02-04 21:50:12 -07:00
|
|
|
totalram_pages--;
|
|
|
|
}
|
|
|
|
|
2012-12-11 17:02:45 -07:00
|
|
|
/* Did we get any? */
|
|
|
|
if (vb->num_pfns != 0)
|
|
|
|
tell_host(vb, vb->inflate_vq);
|
|
|
|
mutex_unlock(&vb->balloon_lock);
|
2008-02-04 21:50:12 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
static void release_pages_by_pfn(const u32 pfns[], unsigned int num)
|
|
|
|
{
|
|
|
|
unsigned int i;
|
|
|
|
|
2012-04-12 07:38:00 -06:00
|
|
|
/* Find pfns pointing at start of each page, get pages and free them. */
|
|
|
|
for (i = 0; i < num; i += VIRTIO_BALLOON_PAGES_PER_PAGE) {
|
2012-12-11 17:02:45 -07:00
|
|
|
balloon_page_free(balloon_pfn_to_page(pfns[i]));
|
2008-02-04 21:50:12 -07:00
|
|
|
totalram_pages++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void leak_balloon(struct virtio_balloon *vb, size_t num)
|
|
|
|
{
|
|
|
|
struct page *page;
|
2012-12-11 17:02:45 -07:00
|
|
|
struct balloon_dev_info *vb_dev_info = vb->vb_dev_info;
|
2008-02-04 21:50:12 -07:00
|
|
|
|
|
|
|
/* We can only do one array worth at a time. */
|
|
|
|
num = min(num, ARRAY_SIZE(vb->pfns));
|
|
|
|
|
2012-12-11 17:02:45 -07:00
|
|
|
mutex_lock(&vb->balloon_lock);
|
2012-04-12 07:38:00 -06:00
|
|
|
for (vb->num_pfns = 0; vb->num_pfns < num;
|
|
|
|
vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE) {
|
2012-12-11 17:02:45 -07:00
|
|
|
page = balloon_page_dequeue(vb_dev_info);
|
|
|
|
if (!page)
|
|
|
|
break;
|
2012-04-12 07:38:00 -06:00
|
|
|
set_page_pfns(vb->pfns + vb->num_pfns, page);
|
|
|
|
vb->num_pages -= VIRTIO_BALLOON_PAGES_PER_PAGE;
|
2008-02-04 21:50:12 -07:00
|
|
|
}
|
|
|
|
|
2011-04-07 11:43:25 -06:00
|
|
|
/*
|
|
|
|
* Note that if
|
|
|
|
* virtio_has_feature(vdev, VIRTIO_BALLOON_F_MUST_TELL_HOST);
|
|
|
|
* is true, we *have* to do it in this order
|
|
|
|
*/
|
|
|
|
tell_host(vb, vb->deflate_vq);
|
2012-12-11 17:02:45 -07:00
|
|
|
mutex_unlock(&vb->balloon_lock);
|
2011-04-07 11:43:25 -06:00
|
|
|
release_pages_by_pfn(vb->pfns, vb->num_pfns);
|
2008-02-04 21:50:12 -07:00
|
|
|
}
|
|
|
|
|
2009-11-30 09:14:15 -07:00
|
|
|
static inline void update_stat(struct virtio_balloon *vb, int idx,
|
|
|
|
u16 tag, u64 val)
|
|
|
|
{
|
|
|
|
BUG_ON(idx >= VIRTIO_BALLOON_S_NR);
|
|
|
|
vb->stats[idx].tag = tag;
|
|
|
|
vb->stats[idx].val = val;
|
|
|
|
}
|
|
|
|
|
|
|
|
#define pages_to_bytes(x) ((u64)(x) << PAGE_SHIFT)
|
|
|
|
|
|
|
|
static void update_balloon_stats(struct virtio_balloon *vb)
|
|
|
|
{
|
|
|
|
unsigned long events[NR_VM_EVENT_ITEMS];
|
|
|
|
struct sysinfo i;
|
|
|
|
int idx = 0;
|
|
|
|
|
|
|
|
all_vm_events(events);
|
|
|
|
si_meminfo(&i);
|
|
|
|
|
|
|
|
update_stat(vb, idx++, VIRTIO_BALLOON_S_SWAP_IN,
|
|
|
|
pages_to_bytes(events[PSWPIN]));
|
|
|
|
update_stat(vb, idx++, VIRTIO_BALLOON_S_SWAP_OUT,
|
|
|
|
pages_to_bytes(events[PSWPOUT]));
|
|
|
|
update_stat(vb, idx++, VIRTIO_BALLOON_S_MAJFLT, events[PGMAJFAULT]);
|
|
|
|
update_stat(vb, idx++, VIRTIO_BALLOON_S_MINFLT, events[PGFAULT]);
|
|
|
|
update_stat(vb, idx++, VIRTIO_BALLOON_S_MEMFREE,
|
|
|
|
pages_to_bytes(i.freeram));
|
|
|
|
update_stat(vb, idx++, VIRTIO_BALLOON_S_MEMTOT,
|
|
|
|
pages_to_bytes(i.totalram));
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* While most virtqueues communicate guest-initiated requests to the hypervisor,
|
|
|
|
* the stats queue operates in reverse. The driver initializes the virtqueue
|
|
|
|
* with a single buffer. From that point forward, all conversations consist of
|
|
|
|
* a hypervisor request (a call to this function) which directs us to refill
|
2009-12-10 15:35:15 -07:00
|
|
|
* the virtqueue with a fresh stats buffer. Since stats collection can sleep,
|
|
|
|
* we notify our kthread which does the actual work via stats_handle_request().
|
2009-11-30 09:14:15 -07:00
|
|
|
*/
|
2009-12-10 15:35:15 -07:00
|
|
|
static void stats_request(struct virtqueue *vq)
|
2009-11-30 09:14:15 -07:00
|
|
|
{
|
2012-07-02 01:33:08 -06:00
|
|
|
struct virtio_balloon *vb = vq->vdev->priv;
|
2009-11-30 09:14:15 -07:00
|
|
|
|
2009-12-10 15:35:15 -07:00
|
|
|
vb->need_stats_update = 1;
|
|
|
|
wake_up(&vb->config_change);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void stats_handle_request(struct virtio_balloon *vb)
|
|
|
|
{
|
|
|
|
struct virtqueue *vq;
|
|
|
|
struct scatterlist sg;
|
2012-07-02 01:33:08 -06:00
|
|
|
unsigned int len;
|
2009-11-30 09:14:15 -07:00
|
|
|
|
2009-12-10 15:35:15 -07:00
|
|
|
vb->need_stats_update = 0;
|
2009-11-30 09:14:15 -07:00
|
|
|
update_balloon_stats(vb);
|
|
|
|
|
2009-12-10 15:35:15 -07:00
|
|
|
vq = vb->stats_vq;
|
2012-07-02 01:33:08 -06:00
|
|
|
if (!virtqueue_get_buf(vq, &len))
|
|
|
|
return;
|
2009-11-30 09:14:15 -07:00
|
|
|
sg_init_one(&sg, vb->stats, sizeof(vb->stats));
|
2012-01-11 22:14:42 -07:00
|
|
|
if (virtqueue_add_buf(vq, &sg, 1, 0, vb, GFP_KERNEL) < 0)
|
2009-11-30 09:14:15 -07:00
|
|
|
BUG();
|
2010-04-12 07:18:28 -06:00
|
|
|
virtqueue_kick(vq);
|
2009-11-30 09:14:15 -07:00
|
|
|
}
|
|
|
|
|
2008-02-04 21:50:12 -07:00
|
|
|
static void virtballoon_changed(struct virtio_device *vdev)
|
|
|
|
{
|
|
|
|
struct virtio_balloon *vb = vdev->priv;
|
|
|
|
|
|
|
|
wake_up(&vb->config_change);
|
|
|
|
}
|
|
|
|
|
2008-03-17 21:58:15 -06:00
|
|
|
static inline s64 towards_target(struct virtio_balloon *vb)
|
2008-02-04 21:50:12 -07:00
|
|
|
{
|
2012-04-11 23:36:34 -06:00
|
|
|
__le32 v;
|
|
|
|
s64 target;
|
|
|
|
|
2008-05-02 20:50:49 -06:00
|
|
|
vb->vdev->config->get(vb->vdev,
|
|
|
|
offsetof(struct virtio_balloon_config, num_pages),
|
|
|
|
&v, sizeof(v));
|
2012-04-11 23:36:34 -06:00
|
|
|
target = le32_to_cpu(v);
|
|
|
|
return target - vb->num_pages;
|
2008-02-04 21:50:12 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
static void update_balloon_size(struct virtio_balloon *vb)
|
|
|
|
{
|
|
|
|
__le32 actual = cpu_to_le32(vb->num_pages);
|
|
|
|
|
|
|
|
vb->vdev->config->set(vb->vdev,
|
|
|
|
offsetof(struct virtio_balloon_config, actual),
|
|
|
|
&actual, sizeof(actual));
|
|
|
|
}
|
|
|
|
|
|
|
|
static int balloon(void *_vballoon)
|
|
|
|
{
|
|
|
|
struct virtio_balloon *vb = _vballoon;
|
|
|
|
|
|
|
|
set_freezable();
|
|
|
|
while (!kthread_should_stop()) {
|
2008-03-17 21:58:15 -06:00
|
|
|
s64 diff;
|
2008-02-04 21:50:12 -07:00
|
|
|
|
|
|
|
try_to_freeze();
|
|
|
|
wait_event_interruptible(vb->config_change,
|
|
|
|
(diff = towards_target(vb)) != 0
|
2009-12-10 15:35:15 -07:00
|
|
|
|| vb->need_stats_update
|
2009-04-16 18:14:04 -06:00
|
|
|
|| kthread_should_stop()
|
|
|
|
|| freezing(current));
|
2009-12-10 15:35:15 -07:00
|
|
|
if (vb->need_stats_update)
|
|
|
|
stats_handle_request(vb);
|
2008-02-04 21:50:12 -07:00
|
|
|
if (diff > 0)
|
|
|
|
fill_balloon(vb, diff);
|
|
|
|
else if (diff < 0)
|
|
|
|
leak_balloon(vb, -diff);
|
|
|
|
update_balloon_size(vb);
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2011-12-22 04:28:34 -07:00
|
|
|
static int init_vqs(struct virtio_balloon *vb)
|
2008-02-04 21:50:12 -07:00
|
|
|
{
|
2009-11-30 09:14:15 -07:00
|
|
|
struct virtqueue *vqs[3];
|
2009-12-10 15:35:15 -07:00
|
|
|
vq_callback_t *callbacks[] = { balloon_ack, balloon_ack, stats_request };
|
2009-11-30 09:14:15 -07:00
|
|
|
const char *names[] = { "inflate", "deflate", "stats" };
|
|
|
|
int err, nvqs;
|
2008-02-04 21:50:12 -07:00
|
|
|
|
2011-12-22 04:28:34 -07:00
|
|
|
/*
|
|
|
|
* We expect two virtqueues: inflate and deflate, and
|
|
|
|
* optionally stat.
|
|
|
|
*/
|
2009-11-30 09:14:15 -07:00
|
|
|
nvqs = virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ) ? 3 : 2;
|
2011-12-22 04:28:34 -07:00
|
|
|
err = vb->vdev->config->find_vqs(vb->vdev, nvqs, vqs, callbacks, names);
|
2009-06-12 22:16:36 -06:00
|
|
|
if (err)
|
2011-12-22 04:28:34 -07:00
|
|
|
return err;
|
2008-02-04 21:50:12 -07:00
|
|
|
|
2009-06-12 22:16:36 -06:00
|
|
|
vb->inflate_vq = vqs[0];
|
|
|
|
vb->deflate_vq = vqs[1];
|
2009-11-30 09:14:15 -07:00
|
|
|
if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ)) {
|
|
|
|
struct scatterlist sg;
|
|
|
|
vb->stats_vq = vqs[2];
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Prime this virtqueue with one buffer so the hypervisor can
|
|
|
|
* use it to signal us later.
|
|
|
|
*/
|
|
|
|
sg_init_one(&sg, vb->stats, sizeof vb->stats);
|
2012-01-11 22:14:42 -07:00
|
|
|
if (virtqueue_add_buf(vb->stats_vq, &sg, 1, 0, vb, GFP_KERNEL)
|
|
|
|
< 0)
|
2009-11-30 09:14:15 -07:00
|
|
|
BUG();
|
2010-04-12 07:18:28 -06:00
|
|
|
virtqueue_kick(vb->stats_vq);
|
2009-11-30 09:14:15 -07:00
|
|
|
}
|
2011-12-22 04:28:34 -07:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2012-12-11 17:02:45 -07:00
|
|
|
static const struct address_space_operations virtio_balloon_aops;
|
|
|
|
#ifdef CONFIG_BALLOON_COMPACTION
|
|
|
|
/*
|
|
|
|
* virtballoon_migratepage - perform the balloon page migration on behalf of
|
|
|
|
* a compation thread. (called under page lock)
|
|
|
|
* @mapping: the page->mapping which will be assigned to the new migrated page.
|
|
|
|
* @newpage: page that will replace the isolated page after migration finishes.
|
|
|
|
* @page : the isolated (old) page that is about to be migrated to newpage.
|
|
|
|
* @mode : compaction mode -- not used for balloon page migration.
|
|
|
|
*
|
|
|
|
* After a ballooned page gets isolated by compaction procedures, this is the
|
|
|
|
* function that performs the page migration on behalf of a compaction thread
|
|
|
|
* The page migration for virtio balloon is done in a simple swap fashion which
|
|
|
|
* follows these two macro steps:
|
|
|
|
* 1) insert newpage into vb->pages list and update the host about it;
|
|
|
|
* 2) update the host about the old page removed from vb->pages list;
|
|
|
|
*
|
|
|
|
* This function preforms the balloon page migration task.
|
|
|
|
* Called through balloon_mapping->a_ops->migratepage
|
|
|
|
*/
|
|
|
|
int virtballoon_migratepage(struct address_space *mapping,
|
|
|
|
struct page *newpage, struct page *page, enum migrate_mode mode)
|
|
|
|
{
|
|
|
|
struct balloon_dev_info *vb_dev_info = balloon_page_device(page);
|
|
|
|
struct virtio_balloon *vb;
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
BUG_ON(!vb_dev_info);
|
|
|
|
|
|
|
|
vb = vb_dev_info->balloon_device;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* In order to avoid lock contention while migrating pages concurrently
|
|
|
|
* to leak_balloon() or fill_balloon() we just give up the balloon_lock
|
|
|
|
* this turn, as it is easier to retry the page migration later.
|
|
|
|
* This also prevents fill_balloon() getting stuck into a mutex
|
|
|
|
* recursion in the case it ends up triggering memory compaction
|
|
|
|
* while it is attempting to inflate the ballon.
|
|
|
|
*/
|
|
|
|
if (!mutex_trylock(&vb->balloon_lock))
|
|
|
|
return -EAGAIN;
|
|
|
|
|
|
|
|
/* balloon's page migration 1st step -- inflate "newpage" */
|
|
|
|
spin_lock_irqsave(&vb_dev_info->pages_lock, flags);
|
|
|
|
balloon_page_insert(newpage, mapping, &vb_dev_info->pages);
|
|
|
|
vb_dev_info->isolated_pages--;
|
|
|
|
spin_unlock_irqrestore(&vb_dev_info->pages_lock, flags);
|
|
|
|
vb->num_pfns = VIRTIO_BALLOON_PAGES_PER_PAGE;
|
|
|
|
set_page_pfns(vb->pfns, newpage);
|
|
|
|
tell_host(vb, vb->inflate_vq);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* balloon's page migration 2nd step -- deflate "page"
|
|
|
|
*
|
|
|
|
* It's safe to delete page->lru here because this page is at
|
|
|
|
* an isolated migration list, and this step is expected to happen here
|
|
|
|
*/
|
|
|
|
balloon_page_delete(page);
|
|
|
|
vb->num_pfns = VIRTIO_BALLOON_PAGES_PER_PAGE;
|
|
|
|
set_page_pfns(vb->pfns, page);
|
|
|
|
tell_host(vb, vb->deflate_vq);
|
|
|
|
|
|
|
|
mutex_unlock(&vb->balloon_lock);
|
|
|
|
|
|
|
|
return MIGRATEPAGE_BALLOON_SUCCESS;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* define the balloon_mapping->a_ops callback to allow balloon page migration */
|
|
|
|
static const struct address_space_operations virtio_balloon_aops = {
|
|
|
|
.migratepage = virtballoon_migratepage,
|
|
|
|
};
|
|
|
|
#endif /* CONFIG_BALLOON_COMPACTION */
|
|
|
|
|
2011-12-22 04:28:34 -07:00
|
|
|
static int virtballoon_probe(struct virtio_device *vdev)
|
|
|
|
{
|
|
|
|
struct virtio_balloon *vb;
|
2012-12-11 17:02:45 -07:00
|
|
|
struct address_space *vb_mapping;
|
|
|
|
struct balloon_dev_info *vb_devinfo;
|
2011-12-22 04:28:34 -07:00
|
|
|
int err;
|
|
|
|
|
|
|
|
vdev->priv = vb = kmalloc(sizeof(*vb), GFP_KERNEL);
|
|
|
|
if (!vb) {
|
|
|
|
err = -ENOMEM;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
vb->num_pages = 0;
|
2012-12-11 17:02:45 -07:00
|
|
|
mutex_init(&vb->balloon_lock);
|
2011-12-22 04:28:34 -07:00
|
|
|
init_waitqueue_head(&vb->config_change);
|
2012-07-02 01:33:08 -06:00
|
|
|
init_waitqueue_head(&vb->acked);
|
2011-12-22 04:28:34 -07:00
|
|
|
vb->vdev = vdev;
|
|
|
|
vb->need_stats_update = 0;
|
|
|
|
|
2012-12-11 17:02:45 -07:00
|
|
|
vb_devinfo = balloon_devinfo_alloc(vb);
|
|
|
|
if (IS_ERR(vb_devinfo)) {
|
|
|
|
err = PTR_ERR(vb_devinfo);
|
|
|
|
goto out_free_vb;
|
|
|
|
}
|
|
|
|
|
|
|
|
vb_mapping = balloon_mapping_alloc(vb_devinfo,
|
|
|
|
(balloon_compaction_check()) ?
|
|
|
|
&virtio_balloon_aops : NULL);
|
|
|
|
if (IS_ERR(vb_mapping)) {
|
|
|
|
/*
|
|
|
|
* IS_ERR(vb_mapping) && PTR_ERR(vb_mapping) == -EOPNOTSUPP
|
|
|
|
* This means !CONFIG_BALLOON_COMPACTION, otherwise we get off.
|
|
|
|
*/
|
|
|
|
err = PTR_ERR(vb_mapping);
|
|
|
|
if (err != -EOPNOTSUPP)
|
|
|
|
goto out_free_vb_devinfo;
|
|
|
|
}
|
|
|
|
|
|
|
|
vb->vb_dev_info = vb_devinfo;
|
|
|
|
|
2011-12-22 04:28:34 -07:00
|
|
|
err = init_vqs(vb);
|
|
|
|
if (err)
|
2012-12-11 17:02:45 -07:00
|
|
|
goto out_free_vb_mapping;
|
2008-02-04 21:50:12 -07:00
|
|
|
|
|
|
|
vb->thread = kthread_run(balloon, vb, "vballoon");
|
|
|
|
if (IS_ERR(vb->thread)) {
|
|
|
|
err = PTR_ERR(vb->thread);
|
2009-06-12 22:16:36 -06:00
|
|
|
goto out_del_vqs;
|
2008-02-04 21:50:12 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
2009-06-12 22:16:36 -06:00
|
|
|
out_del_vqs:
|
|
|
|
vdev->config->del_vqs(vdev);
|
2012-12-11 17:02:45 -07:00
|
|
|
out_free_vb_mapping:
|
|
|
|
balloon_mapping_free(vb_mapping);
|
|
|
|
out_free_vb_devinfo:
|
|
|
|
balloon_devinfo_free(vb_devinfo);
|
2008-02-04 21:50:12 -07:00
|
|
|
out_free_vb:
|
|
|
|
kfree(vb);
|
|
|
|
out:
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2012-04-26 13:15:57 -06:00
|
|
|
static void remove_common(struct virtio_balloon *vb)
|
2008-02-04 21:50:12 -07:00
|
|
|
{
|
|
|
|
/* There might be pages left in the balloon: free them. */
|
|
|
|
while (vb->num_pages)
|
|
|
|
leak_balloon(vb, vb->num_pages);
|
2012-04-26 13:15:56 -06:00
|
|
|
update_balloon_size(vb);
|
2008-02-04 21:50:12 -07:00
|
|
|
|
|
|
|
/* Now we reset the device so we can clean up the queues. */
|
2012-04-26 13:15:57 -06:00
|
|
|
vb->vdev->config->reset(vb->vdev);
|
2008-02-04 21:50:12 -07:00
|
|
|
|
2012-04-26 13:15:57 -06:00
|
|
|
vb->vdev->config->del_vqs(vb->vdev);
|
|
|
|
}
|
|
|
|
|
2012-12-21 14:05:30 -07:00
|
|
|
static void virtballoon_remove(struct virtio_device *vdev)
|
2012-04-26 13:15:57 -06:00
|
|
|
{
|
|
|
|
struct virtio_balloon *vb = vdev->priv;
|
|
|
|
|
|
|
|
kthread_stop(vb->thread);
|
|
|
|
remove_common(vb);
|
2012-12-11 17:02:45 -07:00
|
|
|
balloon_mapping_free(vb->vb_dev_info->mapping);
|
|
|
|
balloon_devinfo_free(vb->vb_dev_info);
|
2008-02-04 21:50:12 -07:00
|
|
|
kfree(vb);
|
|
|
|
}
|
|
|
|
|
virtio: balloon: Add freeze, restore handlers to support S4
Handling balloon hibernate / restore is tricky. If the balloon was
inflated before going into the hibernation state, upon resume, the host
will not have any memory of that. Any pages that were passed on to the
host earlier would most likely be invalid, and the host will have to
re-balloon to the previous value to get in the pre-hibernate state.
So the only sane thing for the guest to do here is to discard all the
pages that were put in the balloon. When to discard the pages is the
next question.
One solution is to deflate the balloon just before writing the image to
the disk (in the freeze() PM callback). However, asking for pages from
the host just to discard them immediately after seems wasteful of
resources. Hence, it makes sense to do this by just fudging our
counters soon after wakeup. This means we don't deflate the balloon
before sleep, and also don't put unnecessary pressure on the host.
This also helps in the thaw case: if the freeze fails for whatever
reason, the balloon should continue to remain in the inflated state.
This was tested by issuing 'swapoff -a' and trying to go into the S4
state. That fails, and the balloon stays inflated, as expected. Both
the host and the guest are happy.
Finally, in the restore() callback, we empty the list of pages that were
previously given off to the host, add the appropriate number of pages to
the totalram_pages counter, reset the num_pages counter to 0, and
all is fine.
As a last step, delete the vqs on the freeze callback to prepare for
hibernation, and re-create them in the restore and thaw callbacks to
resume normal operation.
The kthread doesn't race with any operations here, since it's frozen
before the freeze() call and is thawed after the thaw() and restore()
callbacks, so we're safe with that.
Signed-off-by: Amit Shah <amit.shah@redhat.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2011-12-22 04:28:35 -07:00
|
|
|
#ifdef CONFIG_PM
|
|
|
|
static int virtballoon_freeze(struct virtio_device *vdev)
|
|
|
|
{
|
2012-02-29 05:12:51 -07:00
|
|
|
struct virtio_balloon *vb = vdev->priv;
|
|
|
|
|
virtio: balloon: Add freeze, restore handlers to support S4
Handling balloon hibernate / restore is tricky. If the balloon was
inflated before going into the hibernation state, upon resume, the host
will not have any memory of that. Any pages that were passed on to the
host earlier would most likely be invalid, and the host will have to
re-balloon to the previous value to get in the pre-hibernate state.
So the only sane thing for the guest to do here is to discard all the
pages that were put in the balloon. When to discard the pages is the
next question.
One solution is to deflate the balloon just before writing the image to
the disk (in the freeze() PM callback). However, asking for pages from
the host just to discard them immediately after seems wasteful of
resources. Hence, it makes sense to do this by just fudging our
counters soon after wakeup. This means we don't deflate the balloon
before sleep, and also don't put unnecessary pressure on the host.
This also helps in the thaw case: if the freeze fails for whatever
reason, the balloon should continue to remain in the inflated state.
This was tested by issuing 'swapoff -a' and trying to go into the S4
state. That fails, and the balloon stays inflated, as expected. Both
the host and the guest are happy.
Finally, in the restore() callback, we empty the list of pages that were
previously given off to the host, add the appropriate number of pages to
the totalram_pages counter, reset the num_pages counter to 0, and
all is fine.
As a last step, delete the vqs on the freeze callback to prepare for
hibernation, and re-create them in the restore and thaw callbacks to
resume normal operation.
The kthread doesn't race with any operations here, since it's frozen
before the freeze() call and is thawed after the thaw() and restore()
callbacks, so we're safe with that.
Signed-off-by: Amit Shah <amit.shah@redhat.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2011-12-22 04:28:35 -07:00
|
|
|
/*
|
|
|
|
* The kthread is already frozen by the PM core before this
|
|
|
|
* function is called.
|
|
|
|
*/
|
|
|
|
|
2012-04-26 13:15:57 -06:00
|
|
|
remove_common(vb);
|
virtio: balloon: Add freeze, restore handlers to support S4
Handling balloon hibernate / restore is tricky. If the balloon was
inflated before going into the hibernation state, upon resume, the host
will not have any memory of that. Any pages that were passed on to the
host earlier would most likely be invalid, and the host will have to
re-balloon to the previous value to get in the pre-hibernate state.
So the only sane thing for the guest to do here is to discard all the
pages that were put in the balloon. When to discard the pages is the
next question.
One solution is to deflate the balloon just before writing the image to
the disk (in the freeze() PM callback). However, asking for pages from
the host just to discard them immediately after seems wasteful of
resources. Hence, it makes sense to do this by just fudging our
counters soon after wakeup. This means we don't deflate the balloon
before sleep, and also don't put unnecessary pressure on the host.
This also helps in the thaw case: if the freeze fails for whatever
reason, the balloon should continue to remain in the inflated state.
This was tested by issuing 'swapoff -a' and trying to go into the S4
state. That fails, and the balloon stays inflated, as expected. Both
the host and the guest are happy.
Finally, in the restore() callback, we empty the list of pages that were
previously given off to the host, add the appropriate number of pages to
the totalram_pages counter, reset the num_pages counter to 0, and
all is fine.
As a last step, delete the vqs on the freeze callback to prepare for
hibernation, and re-create them in the restore and thaw callbacks to
resume normal operation.
The kthread doesn't race with any operations here, since it's frozen
before the freeze() call and is thawed after the thaw() and restore()
callbacks, so we're safe with that.
Signed-off-by: Amit Shah <amit.shah@redhat.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2011-12-22 04:28:35 -07:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2012-04-26 13:15:55 -06:00
|
|
|
static int virtballoon_restore(struct virtio_device *vdev)
|
2012-02-29 05:12:51 -07:00
|
|
|
{
|
|
|
|
struct virtio_balloon *vb = vdev->priv;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
ret = init_vqs(vdev->priv);
|
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
fill_balloon(vb, towards_target(vb));
|
|
|
|
update_balloon_size(vb);
|
|
|
|
return 0;
|
|
|
|
}
|
virtio: balloon: Add freeze, restore handlers to support S4
Handling balloon hibernate / restore is tricky. If the balloon was
inflated before going into the hibernation state, upon resume, the host
will not have any memory of that. Any pages that were passed on to the
host earlier would most likely be invalid, and the host will have to
re-balloon to the previous value to get in the pre-hibernate state.
So the only sane thing for the guest to do here is to discard all the
pages that were put in the balloon. When to discard the pages is the
next question.
One solution is to deflate the balloon just before writing the image to
the disk (in the freeze() PM callback). However, asking for pages from
the host just to discard them immediately after seems wasteful of
resources. Hence, it makes sense to do this by just fudging our
counters soon after wakeup. This means we don't deflate the balloon
before sleep, and also don't put unnecessary pressure on the host.
This also helps in the thaw case: if the freeze fails for whatever
reason, the balloon should continue to remain in the inflated state.
This was tested by issuing 'swapoff -a' and trying to go into the S4
state. That fails, and the balloon stays inflated, as expected. Both
the host and the guest are happy.
Finally, in the restore() callback, we empty the list of pages that were
previously given off to the host, add the appropriate number of pages to
the totalram_pages counter, reset the num_pages counter to 0, and
all is fine.
As a last step, delete the vqs on the freeze callback to prepare for
hibernation, and re-create them in the restore and thaw callbacks to
resume normal operation.
The kthread doesn't race with any operations here, since it's frozen
before the freeze() call and is thawed after the thaw() and restore()
callbacks, so we're safe with that.
Signed-off-by: Amit Shah <amit.shah@redhat.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2011-12-22 04:28:35 -07:00
|
|
|
#endif
|
|
|
|
|
2009-11-30 09:14:15 -07:00
|
|
|
static unsigned int features[] = {
|
|
|
|
VIRTIO_BALLOON_F_MUST_TELL_HOST,
|
|
|
|
VIRTIO_BALLOON_F_STATS_VQ,
|
|
|
|
};
|
2008-05-02 20:50:50 -06:00
|
|
|
|
2010-01-15 18:01:26 -07:00
|
|
|
static struct virtio_driver virtio_balloon_driver = {
|
2008-05-02 20:50:50 -06:00
|
|
|
.feature_table = features,
|
|
|
|
.feature_table_size = ARRAY_SIZE(features),
|
2008-02-04 21:50:12 -07:00
|
|
|
.driver.name = KBUILD_MODNAME,
|
|
|
|
.driver.owner = THIS_MODULE,
|
|
|
|
.id_table = id_table,
|
|
|
|
.probe = virtballoon_probe,
|
2012-12-21 14:05:30 -07:00
|
|
|
.remove = virtballoon_remove,
|
2008-02-04 21:50:12 -07:00
|
|
|
.config_changed = virtballoon_changed,
|
virtio: balloon: Add freeze, restore handlers to support S4
Handling balloon hibernate / restore is tricky. If the balloon was
inflated before going into the hibernation state, upon resume, the host
will not have any memory of that. Any pages that were passed on to the
host earlier would most likely be invalid, and the host will have to
re-balloon to the previous value to get in the pre-hibernate state.
So the only sane thing for the guest to do here is to discard all the
pages that were put in the balloon. When to discard the pages is the
next question.
One solution is to deflate the balloon just before writing the image to
the disk (in the freeze() PM callback). However, asking for pages from
the host just to discard them immediately after seems wasteful of
resources. Hence, it makes sense to do this by just fudging our
counters soon after wakeup. This means we don't deflate the balloon
before sleep, and also don't put unnecessary pressure on the host.
This also helps in the thaw case: if the freeze fails for whatever
reason, the balloon should continue to remain in the inflated state.
This was tested by issuing 'swapoff -a' and trying to go into the S4
state. That fails, and the balloon stays inflated, as expected. Both
the host and the guest are happy.
Finally, in the restore() callback, we empty the list of pages that were
previously given off to the host, add the appropriate number of pages to
the totalram_pages counter, reset the num_pages counter to 0, and
all is fine.
As a last step, delete the vqs on the freeze callback to prepare for
hibernation, and re-create them in the restore and thaw callbacks to
resume normal operation.
The kthread doesn't race with any operations here, since it's frozen
before the freeze() call and is thawed after the thaw() and restore()
callbacks, so we're safe with that.
Signed-off-by: Amit Shah <amit.shah@redhat.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2011-12-22 04:28:35 -07:00
|
|
|
#ifdef CONFIG_PM
|
|
|
|
.freeze = virtballoon_freeze,
|
|
|
|
.restore = virtballoon_restore,
|
|
|
|
#endif
|
2008-02-04 21:50:12 -07:00
|
|
|
};
|
|
|
|
|
2013-02-12 23:29:28 -07:00
|
|
|
module_virtio_driver(virtio_balloon_driver);
|
2008-02-04 21:50:12 -07:00
|
|
|
MODULE_DEVICE_TABLE(virtio, id_table);
|
|
|
|
MODULE_DESCRIPTION("Virtio balloon driver");
|
|
|
|
MODULE_LICENSE("GPL");
|