2007-10-21 17:41:41 -06:00
|
|
|
/*
|
|
|
|
* Copyright (c) 2006, Intel Corporation.
|
|
|
|
*
|
|
|
|
* This program is free software; you can redistribute it and/or modify it
|
|
|
|
* under the terms and conditions of the GNU General Public License,
|
|
|
|
* version 2, as published by the Free Software Foundation.
|
|
|
|
*
|
|
|
|
* This program is distributed in the hope it will be useful, but WITHOUT
|
|
|
|
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
|
|
|
|
* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
|
|
|
|
* more details.
|
|
|
|
*
|
|
|
|
* You should have received a copy of the GNU General Public License along with
|
|
|
|
* this program; if not, write to the Free Software Foundation, Inc., 59 Temple
|
|
|
|
* Place - Suite 330, Boston, MA 02111-1307 USA.
|
|
|
|
*
|
2008-02-23 16:23:35 -07:00
|
|
|
* Copyright (C) 2006-2008 Intel Corporation
|
|
|
|
* Author: Ashok Raj <ashok.raj@intel.com>
|
|
|
|
* Author: Shaohua Li <shaohua.li@intel.com>
|
|
|
|
* Author: Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>
|
2007-10-21 17:41:41 -06:00
|
|
|
*
|
2008-07-10 12:16:35 -06:00
|
|
|
* This file implements early detection/parsing of Remapping Devices
|
2007-10-21 17:41:41 -06:00
|
|
|
* reported to OS through BIOS via DMA remapping reporting (DMAR) ACPI
|
|
|
|
* tables.
|
2008-07-10 12:16:35 -06:00
|
|
|
*
|
|
|
|
* These routines are used by both DMA-remapping and Interrupt-remapping
|
2007-10-21 17:41:41 -06:00
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/pci.h>
|
|
|
|
#include <linux/dmar.h>
|
2008-09-09 09:37:29 -06:00
|
|
|
#include <linux/iova.h>
|
|
|
|
#include <linux/intel-iommu.h>
|
2008-07-10 12:16:42 -06:00
|
|
|
#include <linux/timer.h>
|
2009-03-16 18:04:54 -06:00
|
|
|
#include <linux/irq.h>
|
|
|
|
#include <linux/interrupt.h>
|
2009-09-01 19:25:07 -06:00
|
|
|
#include <linux/tboot.h>
|
2009-07-06 21:40:19 -06:00
|
|
|
#include <linux/dmi.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 02:04:11 -06:00
|
|
|
#include <linux/slab.h>
|
2007-10-21 17:41:41 -06:00
|
|
|
|
2009-07-28 14:45:54 -06:00
|
|
|
#define PREFIX "DMAR: "
|
2007-10-21 17:41:41 -06:00
|
|
|
|
|
|
|
/* No locks are needed as DMA remapping hardware unit
|
|
|
|
* list is constructed at boot time and hotplug of
|
|
|
|
* these units are not supported by the architecture.
|
|
|
|
*/
|
|
|
|
LIST_HEAD(dmar_drhd_units);
|
|
|
|
|
|
|
|
static struct acpi_table_header * __initdata dmar_tbl;
|
2009-02-11 02:06:59 -07:00
|
|
|
static acpi_size dmar_tbl_size;
|
2007-10-21 17:41:41 -06:00
|
|
|
|
|
|
|
static void __init dmar_register_drhd_unit(struct dmar_drhd_unit *drhd)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* add INCLUDE_ALL at the tail, so scan the list will find it at
|
|
|
|
* the very end.
|
|
|
|
*/
|
|
|
|
if (drhd->include_all)
|
|
|
|
list_add_tail(&drhd->list, &dmar_drhd_units);
|
|
|
|
else
|
|
|
|
list_add(&drhd->list, &dmar_drhd_units);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int __init dmar_parse_one_dev_scope(struct acpi_dmar_device_scope *scope,
|
|
|
|
struct pci_dev **dev, u16 segment)
|
|
|
|
{
|
|
|
|
struct pci_bus *bus;
|
|
|
|
struct pci_dev *pdev = NULL;
|
|
|
|
struct acpi_dmar_pci_path *path;
|
|
|
|
int count;
|
|
|
|
|
|
|
|
bus = pci_find_bus(segment, scope->bus);
|
|
|
|
path = (struct acpi_dmar_pci_path *)(scope + 1);
|
|
|
|
count = (scope->length - sizeof(struct acpi_dmar_device_scope))
|
|
|
|
/ sizeof(struct acpi_dmar_pci_path);
|
|
|
|
|
|
|
|
while (count) {
|
|
|
|
if (pdev)
|
|
|
|
pci_dev_put(pdev);
|
|
|
|
/*
|
|
|
|
* Some BIOSes list non-exist devices in DMAR table, just
|
|
|
|
* ignore it
|
|
|
|
*/
|
|
|
|
if (!bus) {
|
|
|
|
printk(KERN_WARNING
|
|
|
|
PREFIX "Device scope bus [%d] not found\n",
|
|
|
|
scope->bus);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
pdev = pci_get_slot(bus, PCI_DEVFN(path->dev, path->fn));
|
|
|
|
if (!pdev) {
|
|
|
|
printk(KERN_WARNING PREFIX
|
|
|
|
"Device scope device [%04x:%02x:%02x.%02x] not found\n",
|
|
|
|
segment, bus->number, path->dev, path->fn);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
path ++;
|
|
|
|
count --;
|
|
|
|
bus = pdev->subordinate;
|
|
|
|
}
|
|
|
|
if (!pdev) {
|
|
|
|
printk(KERN_WARNING PREFIX
|
|
|
|
"Device scope device [%04x:%02x:%02x.%02x] not found\n",
|
|
|
|
segment, scope->bus, path->dev, path->fn);
|
|
|
|
*dev = NULL;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
if ((scope->entry_type == ACPI_DMAR_SCOPE_TYPE_ENDPOINT && \
|
|
|
|
pdev->subordinate) || (scope->entry_type == \
|
|
|
|
ACPI_DMAR_SCOPE_TYPE_BRIDGE && !pdev->subordinate)) {
|
|
|
|
pci_dev_put(pdev);
|
|
|
|
printk(KERN_WARNING PREFIX
|
|
|
|
"Device scope type does not match for %s\n",
|
|
|
|
pci_name(pdev));
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
*dev = pdev;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int __init dmar_parse_dev_scope(void *start, void *end, int *cnt,
|
|
|
|
struct pci_dev ***devices, u16 segment)
|
|
|
|
{
|
|
|
|
struct acpi_dmar_device_scope *scope;
|
|
|
|
void * tmp = start;
|
|
|
|
int index;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
*cnt = 0;
|
|
|
|
while (start < end) {
|
|
|
|
scope = start;
|
|
|
|
if (scope->entry_type == ACPI_DMAR_SCOPE_TYPE_ENDPOINT ||
|
|
|
|
scope->entry_type == ACPI_DMAR_SCOPE_TYPE_BRIDGE)
|
|
|
|
(*cnt)++;
|
|
|
|
else
|
|
|
|
printk(KERN_WARNING PREFIX
|
|
|
|
"Unsupported device scope\n");
|
|
|
|
start += scope->length;
|
|
|
|
}
|
|
|
|
if (*cnt == 0)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
*devices = kcalloc(*cnt, sizeof(struct pci_dev *), GFP_KERNEL);
|
|
|
|
if (!*devices)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
start = tmp;
|
|
|
|
index = 0;
|
|
|
|
while (start < end) {
|
|
|
|
scope = start;
|
|
|
|
if (scope->entry_type == ACPI_DMAR_SCOPE_TYPE_ENDPOINT ||
|
|
|
|
scope->entry_type == ACPI_DMAR_SCOPE_TYPE_BRIDGE) {
|
|
|
|
ret = dmar_parse_one_dev_scope(scope,
|
|
|
|
&(*devices)[index], segment);
|
|
|
|
if (ret) {
|
|
|
|
kfree(*devices);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
index ++;
|
|
|
|
}
|
|
|
|
start += scope->length;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* dmar_parse_one_drhd - parses exactly one DMA remapping hardware definition
|
|
|
|
* structure which uniquely represent one DMA remapping hardware unit
|
|
|
|
* present in the platform
|
|
|
|
*/
|
|
|
|
static int __init
|
|
|
|
dmar_parse_one_drhd(struct acpi_dmar_header *header)
|
|
|
|
{
|
|
|
|
struct acpi_dmar_hardware_unit *drhd;
|
|
|
|
struct dmar_drhd_unit *dmaru;
|
|
|
|
int ret = 0;
|
|
|
|
|
2009-04-10 23:27:48 -06:00
|
|
|
drhd = (struct acpi_dmar_hardware_unit *)header;
|
2007-10-21 17:41:41 -06:00
|
|
|
dmaru = kzalloc(sizeof(*dmaru), GFP_KERNEL);
|
|
|
|
if (!dmaru)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
2008-07-10 12:16:37 -06:00
|
|
|
dmaru->hdr = header;
|
2007-10-21 17:41:41 -06:00
|
|
|
dmaru->reg_base_addr = drhd->address;
|
2009-04-03 18:45:37 -06:00
|
|
|
dmaru->segment = drhd->segment;
|
2007-10-21 17:41:41 -06:00
|
|
|
dmaru->include_all = drhd->flags & 0x1; /* BIT0: INCLUDE_ALL */
|
|
|
|
|
2008-07-10 12:16:37 -06:00
|
|
|
ret = alloc_iommu(dmaru);
|
|
|
|
if (ret) {
|
|
|
|
kfree(dmaru);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
dmar_register_drhd_unit(dmaru);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2008-10-18 08:43:14 -06:00
|
|
|
static int __init dmar_parse_dev(struct dmar_drhd_unit *dmaru)
|
2008-07-10 12:16:37 -06:00
|
|
|
{
|
|
|
|
struct acpi_dmar_hardware_unit *drhd;
|
2008-10-18 08:43:14 -06:00
|
|
|
int ret = 0;
|
2008-07-10 12:16:37 -06:00
|
|
|
|
|
|
|
drhd = (struct acpi_dmar_hardware_unit *) dmaru->hdr;
|
|
|
|
|
2008-12-22 01:54:58 -07:00
|
|
|
if (dmaru->include_all)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
ret = dmar_parse_dev_scope((void *)(drhd + 1),
|
2008-07-10 12:16:37 -06:00
|
|
|
((void *)drhd) + drhd->header.length,
|
2007-10-21 17:41:41 -06:00
|
|
|
&dmaru->devices_cnt, &dmaru->devices,
|
|
|
|
drhd->segment);
|
2008-09-03 17:58:35 -06:00
|
|
|
if (ret) {
|
2008-07-10 12:16:37 -06:00
|
|
|
list_del(&dmaru->list);
|
2007-10-21 17:41:41 -06:00
|
|
|
kfree(dmaru);
|
2008-07-10 12:16:37 -06:00
|
|
|
}
|
2007-10-21 17:41:41 -06:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2008-07-10 12:16:38 -06:00
|
|
|
#ifdef CONFIG_DMAR
|
|
|
|
LIST_HEAD(dmar_rmrr_units);
|
|
|
|
|
|
|
|
static void __init dmar_register_rmrr_unit(struct dmar_rmrr_unit *rmrr)
|
|
|
|
{
|
|
|
|
list_add(&rmrr->list, &dmar_rmrr_units);
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2007-10-21 17:41:41 -06:00
|
|
|
static int __init
|
|
|
|
dmar_parse_one_rmrr(struct acpi_dmar_header *header)
|
|
|
|
{
|
|
|
|
struct acpi_dmar_reserved_memory *rmrr;
|
|
|
|
struct dmar_rmrr_unit *rmrru;
|
|
|
|
|
|
|
|
rmrru = kzalloc(sizeof(*rmrru), GFP_KERNEL);
|
|
|
|
if (!rmrru)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
2008-07-10 12:16:37 -06:00
|
|
|
rmrru->hdr = header;
|
2007-10-21 17:41:41 -06:00
|
|
|
rmrr = (struct acpi_dmar_reserved_memory *)header;
|
|
|
|
rmrru->base_address = rmrr->base_address;
|
|
|
|
rmrru->end_address = rmrr->end_address;
|
2008-07-10 12:16:37 -06:00
|
|
|
|
|
|
|
dmar_register_rmrr_unit(rmrru);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int __init
|
|
|
|
rmrr_parse_dev(struct dmar_rmrr_unit *rmrru)
|
|
|
|
{
|
|
|
|
struct acpi_dmar_reserved_memory *rmrr;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
rmrr = (struct acpi_dmar_reserved_memory *) rmrru->hdr;
|
2007-10-21 17:41:41 -06:00
|
|
|
ret = dmar_parse_dev_scope((void *)(rmrr + 1),
|
2008-07-10 12:16:37 -06:00
|
|
|
((void *)rmrr) + rmrr->header.length,
|
2007-10-21 17:41:41 -06:00
|
|
|
&rmrru->devices_cnt, &rmrru->devices, rmrr->segment);
|
|
|
|
|
2008-07-10 12:16:37 -06:00
|
|
|
if (ret || (rmrru->devices_cnt == 0)) {
|
|
|
|
list_del(&rmrru->list);
|
2007-10-21 17:41:41 -06:00
|
|
|
kfree(rmrru);
|
2008-07-10 12:16:37 -06:00
|
|
|
}
|
2007-10-21 17:41:41 -06:00
|
|
|
return ret;
|
|
|
|
}
|
2009-05-17 23:51:34 -06:00
|
|
|
|
|
|
|
static LIST_HEAD(dmar_atsr_units);
|
|
|
|
|
|
|
|
static int __init dmar_parse_one_atsr(struct acpi_dmar_header *hdr)
|
|
|
|
{
|
|
|
|
struct acpi_dmar_atsr *atsr;
|
|
|
|
struct dmar_atsr_unit *atsru;
|
|
|
|
|
|
|
|
atsr = container_of(hdr, struct acpi_dmar_atsr, header);
|
|
|
|
atsru = kzalloc(sizeof(*atsru), GFP_KERNEL);
|
|
|
|
if (!atsru)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
atsru->hdr = hdr;
|
|
|
|
atsru->include_all = atsr->flags & 0x1;
|
|
|
|
|
|
|
|
list_add(&atsru->list, &dmar_atsr_units);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int __init atsr_parse_dev(struct dmar_atsr_unit *atsru)
|
|
|
|
{
|
|
|
|
int rc;
|
|
|
|
struct acpi_dmar_atsr *atsr;
|
|
|
|
|
|
|
|
if (atsru->include_all)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
atsr = container_of(atsru->hdr, struct acpi_dmar_atsr, header);
|
|
|
|
rc = dmar_parse_dev_scope((void *)(atsr + 1),
|
|
|
|
(void *)atsr + atsr->header.length,
|
|
|
|
&atsru->devices_cnt, &atsru->devices,
|
|
|
|
atsr->segment);
|
|
|
|
if (rc || !atsru->devices_cnt) {
|
|
|
|
list_del(&atsru->list);
|
|
|
|
kfree(atsru);
|
|
|
|
}
|
|
|
|
|
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
|
|
|
|
int dmar_find_matched_atsr_unit(struct pci_dev *dev)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
struct pci_bus *bus;
|
|
|
|
struct acpi_dmar_atsr *atsr;
|
|
|
|
struct dmar_atsr_unit *atsru;
|
|
|
|
|
|
|
|
list_for_each_entry(atsru, &dmar_atsr_units, list) {
|
|
|
|
atsr = container_of(atsru->hdr, struct acpi_dmar_atsr, header);
|
|
|
|
if (atsr->segment == pci_domain_nr(dev->bus))
|
|
|
|
goto found;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
found:
|
|
|
|
for (bus = dev->bus; bus; bus = bus->parent) {
|
|
|
|
struct pci_dev *bridge = bus->self;
|
|
|
|
|
2009-11-10 22:36:17 -07:00
|
|
|
if (!bridge || !pci_is_pcie(bridge) ||
|
2009-05-17 23:51:34 -06:00
|
|
|
bridge->pcie_type == PCI_EXP_TYPE_PCI_BRIDGE)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
if (bridge->pcie_type == PCI_EXP_TYPE_ROOT_PORT) {
|
|
|
|
for (i = 0; i < atsru->devices_cnt; i++)
|
|
|
|
if (atsru->devices[i] == bridge)
|
|
|
|
return 1;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (atsru->include_all)
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
2008-07-10 12:16:38 -06:00
|
|
|
#endif
|
2007-10-21 17:41:41 -06:00
|
|
|
|
2009-10-07 05:18:00 -06:00
|
|
|
#ifdef CONFIG_ACPI_NUMA
|
2009-10-02 12:01:21 -06:00
|
|
|
static int __init
|
|
|
|
dmar_parse_one_rhsa(struct acpi_dmar_header *header)
|
|
|
|
{
|
|
|
|
struct acpi_dmar_rhsa *rhsa;
|
|
|
|
struct dmar_drhd_unit *drhd;
|
|
|
|
|
|
|
|
rhsa = (struct acpi_dmar_rhsa *)header;
|
2009-10-07 05:18:00 -06:00
|
|
|
for_each_drhd_unit(drhd) {
|
2009-10-02 12:01:21 -06:00
|
|
|
if (drhd->reg_base_addr == rhsa->base_address) {
|
|
|
|
int node = acpi_map_pxm_to_node(rhsa->proximity_domain);
|
|
|
|
|
|
|
|
if (!node_online(node))
|
|
|
|
node = -1;
|
|
|
|
drhd->iommu->node = node;
|
2009-10-07 05:18:00 -06:00
|
|
|
return 0;
|
|
|
|
}
|
2009-10-02 12:01:21 -06:00
|
|
|
}
|
2009-10-07 05:18:00 -06:00
|
|
|
WARN(1, "Your BIOS is broken; RHSA refers to non-existent DMAR unit at %llx\n"
|
|
|
|
"BIOS vendor: %s; Ver: %s; Product Version: %s\n",
|
|
|
|
drhd->reg_base_addr,
|
|
|
|
dmi_get_system_info(DMI_BIOS_VENDOR),
|
|
|
|
dmi_get_system_info(DMI_BIOS_VERSION),
|
|
|
|
dmi_get_system_info(DMI_PRODUCT_VERSION));
|
2009-10-02 12:01:21 -06:00
|
|
|
|
2009-10-07 05:18:00 -06:00
|
|
|
return 0;
|
2009-10-02 12:01:21 -06:00
|
|
|
}
|
2009-10-07 05:18:00 -06:00
|
|
|
#endif
|
2009-10-02 12:01:21 -06:00
|
|
|
|
2007-10-21 17:41:41 -06:00
|
|
|
static void __init
|
|
|
|
dmar_table_print_dmar_entry(struct acpi_dmar_header *header)
|
|
|
|
{
|
|
|
|
struct acpi_dmar_hardware_unit *drhd;
|
|
|
|
struct acpi_dmar_reserved_memory *rmrr;
|
2009-05-17 23:51:34 -06:00
|
|
|
struct acpi_dmar_atsr *atsr;
|
2009-09-24 13:14:00 -06:00
|
|
|
struct acpi_dmar_rhsa *rhsa;
|
2007-10-21 17:41:41 -06:00
|
|
|
|
|
|
|
switch (header->type) {
|
|
|
|
case ACPI_DMAR_TYPE_HARDWARE_UNIT:
|
2009-05-17 23:51:34 -06:00
|
|
|
drhd = container_of(header, struct acpi_dmar_hardware_unit,
|
|
|
|
header);
|
2007-10-21 17:41:41 -06:00
|
|
|
printk (KERN_INFO PREFIX
|
2009-05-17 23:51:34 -06:00
|
|
|
"DRHD base: %#016Lx flags: %#x\n",
|
|
|
|
(unsigned long long)drhd->address, drhd->flags);
|
2007-10-21 17:41:41 -06:00
|
|
|
break;
|
|
|
|
case ACPI_DMAR_TYPE_RESERVED_MEMORY:
|
2009-05-17 23:51:34 -06:00
|
|
|
rmrr = container_of(header, struct acpi_dmar_reserved_memory,
|
|
|
|
header);
|
2007-10-21 17:41:41 -06:00
|
|
|
printk (KERN_INFO PREFIX
|
2009-05-17 23:51:34 -06:00
|
|
|
"RMRR base: %#016Lx end: %#016Lx\n",
|
2008-10-16 19:02:32 -06:00
|
|
|
(unsigned long long)rmrr->base_address,
|
|
|
|
(unsigned long long)rmrr->end_address);
|
2007-10-21 17:41:41 -06:00
|
|
|
break;
|
2009-05-17 23:51:34 -06:00
|
|
|
case ACPI_DMAR_TYPE_ATSR:
|
|
|
|
atsr = container_of(header, struct acpi_dmar_atsr, header);
|
|
|
|
printk(KERN_INFO PREFIX "ATSR flags: %#x\n", atsr->flags);
|
|
|
|
break;
|
2009-09-24 13:14:00 -06:00
|
|
|
case ACPI_DMAR_HARDWARE_AFFINITY:
|
|
|
|
rhsa = container_of(header, struct acpi_dmar_rhsa, header);
|
|
|
|
printk(KERN_INFO PREFIX "RHSA base: %#016Lx proximity domain: %#x\n",
|
|
|
|
(unsigned long long)rhsa->base_address,
|
|
|
|
rhsa->proximity_domain);
|
|
|
|
break;
|
2007-10-21 17:41:41 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-09-03 17:58:32 -06:00
|
|
|
/**
|
|
|
|
* dmar_table_detect - checks to see if the platform supports DMAR devices
|
|
|
|
*/
|
|
|
|
static int __init dmar_table_detect(void)
|
|
|
|
{
|
|
|
|
acpi_status status = AE_OK;
|
|
|
|
|
|
|
|
/* if we could find DMAR table, then there are DMAR devices */
|
2009-02-11 02:06:59 -07:00
|
|
|
status = acpi_get_table_with_size(ACPI_SIG_DMAR, 0,
|
|
|
|
(struct acpi_table_header **)&dmar_tbl,
|
|
|
|
&dmar_tbl_size);
|
2008-09-03 17:58:32 -06:00
|
|
|
|
|
|
|
if (ACPI_SUCCESS(status) && !dmar_tbl) {
|
|
|
|
printk (KERN_WARNING PREFIX "Unable to map DMAR\n");
|
|
|
|
status = AE_NOT_FOUND;
|
|
|
|
}
|
|
|
|
|
|
|
|
return (ACPI_SUCCESS(status) ? 1 : 0);
|
|
|
|
}
|
2008-07-10 12:16:38 -06:00
|
|
|
|
2007-10-21 17:41:41 -06:00
|
|
|
/**
|
|
|
|
* parse_dmar_table - parses the DMA reporting table
|
|
|
|
*/
|
|
|
|
static int __init
|
|
|
|
parse_dmar_table(void)
|
|
|
|
{
|
|
|
|
struct acpi_table_dmar *dmar;
|
|
|
|
struct acpi_dmar_header *entry_header;
|
|
|
|
int ret = 0;
|
|
|
|
|
2008-09-03 17:58:32 -06:00
|
|
|
/*
|
|
|
|
* Do it again, earlier dmar_tbl mapping could be mapped with
|
|
|
|
* fixed map.
|
|
|
|
*/
|
|
|
|
dmar_table_detect();
|
|
|
|
|
2009-06-30 20:31:10 -06:00
|
|
|
/*
|
|
|
|
* ACPI tables may not be DMA protected by tboot, so use DMAR copy
|
|
|
|
* SINIT saved in SinitMleData in TXT heap (which is DMA protected)
|
|
|
|
*/
|
|
|
|
dmar_tbl = tboot_get_dmar_table(dmar_tbl);
|
|
|
|
|
2007-10-21 17:41:41 -06:00
|
|
|
dmar = (struct acpi_table_dmar *)dmar_tbl;
|
|
|
|
if (!dmar)
|
|
|
|
return -ENODEV;
|
|
|
|
|
2008-10-16 19:02:32 -06:00
|
|
|
if (dmar->width < PAGE_SHIFT - 1) {
|
2007-11-21 16:07:14 -07:00
|
|
|
printk(KERN_WARNING PREFIX "Invalid DMAR haw\n");
|
2007-10-21 17:41:41 -06:00
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
|
|
|
printk (KERN_INFO PREFIX "Host address width %d\n",
|
|
|
|
dmar->width + 1);
|
|
|
|
|
|
|
|
entry_header = (struct acpi_dmar_header *)(dmar + 1);
|
|
|
|
while (((unsigned long)entry_header) <
|
|
|
|
(((unsigned long)dmar) + dmar_tbl->length)) {
|
2009-02-11 14:24:19 -07:00
|
|
|
/* Avoid looping forever on bad ACPI tables */
|
|
|
|
if (entry_header->length == 0) {
|
|
|
|
printk(KERN_WARNING PREFIX
|
|
|
|
"Invalid 0-length structure\n");
|
|
|
|
ret = -EINVAL;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2007-10-21 17:41:41 -06:00
|
|
|
dmar_table_print_dmar_entry(entry_header);
|
|
|
|
|
|
|
|
switch (entry_header->type) {
|
|
|
|
case ACPI_DMAR_TYPE_HARDWARE_UNIT:
|
|
|
|
ret = dmar_parse_one_drhd(entry_header);
|
|
|
|
break;
|
|
|
|
case ACPI_DMAR_TYPE_RESERVED_MEMORY:
|
2008-07-10 12:16:38 -06:00
|
|
|
#ifdef CONFIG_DMAR
|
2007-10-21 17:41:41 -06:00
|
|
|
ret = dmar_parse_one_rmrr(entry_header);
|
2009-05-17 23:51:34 -06:00
|
|
|
#endif
|
|
|
|
break;
|
|
|
|
case ACPI_DMAR_TYPE_ATSR:
|
|
|
|
#ifdef CONFIG_DMAR
|
|
|
|
ret = dmar_parse_one_atsr(entry_header);
|
2008-07-10 12:16:38 -06:00
|
|
|
#endif
|
2007-10-21 17:41:41 -06:00
|
|
|
break;
|
2009-09-24 13:14:00 -06:00
|
|
|
case ACPI_DMAR_HARDWARE_AFFINITY:
|
2009-10-07 05:18:00 -06:00
|
|
|
#ifdef CONFIG_ACPI_NUMA
|
2009-10-02 12:01:21 -06:00
|
|
|
ret = dmar_parse_one_rhsa(entry_header);
|
2009-10-07 05:18:00 -06:00
|
|
|
#endif
|
2009-09-24 13:14:00 -06:00
|
|
|
break;
|
2007-10-21 17:41:41 -06:00
|
|
|
default:
|
|
|
|
printk(KERN_WARNING PREFIX
|
2009-09-23 18:01:29 -06:00
|
|
|
"Unknown DMAR structure type %d\n",
|
|
|
|
entry_header->type);
|
2007-10-21 17:41:41 -06:00
|
|
|
ret = 0; /* for forward compatibility */
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
if (ret)
|
|
|
|
break;
|
|
|
|
|
|
|
|
entry_header = ((void *)entry_header + entry_header->length);
|
|
|
|
}
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2008-07-10 12:16:35 -06:00
|
|
|
int dmar_pci_device_match(struct pci_dev *devices[], int cnt,
|
|
|
|
struct pci_dev *dev)
|
|
|
|
{
|
|
|
|
int index;
|
|
|
|
|
|
|
|
while (dev) {
|
|
|
|
for (index = 0; index < cnt; index++)
|
|
|
|
if (dev == devices[index])
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
/* Check our parent */
|
|
|
|
dev = dev->bus->self;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
struct dmar_drhd_unit *
|
|
|
|
dmar_find_matched_drhd_unit(struct pci_dev *dev)
|
|
|
|
{
|
2008-12-22 01:54:58 -07:00
|
|
|
struct dmar_drhd_unit *dmaru = NULL;
|
|
|
|
struct acpi_dmar_hardware_unit *drhd;
|
|
|
|
|
|
|
|
list_for_each_entry(dmaru, &dmar_drhd_units, list) {
|
|
|
|
drhd = container_of(dmaru->hdr,
|
|
|
|
struct acpi_dmar_hardware_unit,
|
|
|
|
header);
|
|
|
|
|
|
|
|
if (dmaru->include_all &&
|
|
|
|
drhd->segment == pci_domain_nr(dev->bus))
|
|
|
|
return dmaru;
|
2008-07-10 12:16:35 -06:00
|
|
|
|
2008-12-22 01:54:58 -07:00
|
|
|
if (dmar_pci_device_match(dmaru->devices,
|
|
|
|
dmaru->devices_cnt, dev))
|
|
|
|
return dmaru;
|
2008-07-10 12:16:35 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2008-07-10 12:16:37 -06:00
|
|
|
int __init dmar_dev_scope_init(void)
|
|
|
|
{
|
2008-09-03 17:58:34 -06:00
|
|
|
struct dmar_drhd_unit *drhd, *drhd_n;
|
2008-07-10 12:16:37 -06:00
|
|
|
int ret = -ENODEV;
|
|
|
|
|
2008-09-03 17:58:34 -06:00
|
|
|
list_for_each_entry_safe(drhd, drhd_n, &dmar_drhd_units, list) {
|
2008-07-10 12:16:37 -06:00
|
|
|
ret = dmar_parse_dev(drhd);
|
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2008-07-10 12:16:38 -06:00
|
|
|
#ifdef CONFIG_DMAR
|
|
|
|
{
|
2008-09-03 17:58:34 -06:00
|
|
|
struct dmar_rmrr_unit *rmrr, *rmrr_n;
|
2009-05-17 23:51:34 -06:00
|
|
|
struct dmar_atsr_unit *atsr, *atsr_n;
|
|
|
|
|
2008-09-03 17:58:34 -06:00
|
|
|
list_for_each_entry_safe(rmrr, rmrr_n, &dmar_rmrr_units, list) {
|
2008-07-10 12:16:38 -06:00
|
|
|
ret = rmrr_parse_dev(rmrr);
|
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
}
|
2009-05-17 23:51:34 -06:00
|
|
|
|
|
|
|
list_for_each_entry_safe(atsr, atsr_n, &dmar_atsr_units, list) {
|
|
|
|
ret = atsr_parse_dev(atsr);
|
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
}
|
2008-07-10 12:16:37 -06:00
|
|
|
}
|
2008-07-10 12:16:38 -06:00
|
|
|
#endif
|
2008-07-10 12:16:37 -06:00
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2007-10-21 17:41:41 -06:00
|
|
|
|
|
|
|
int __init dmar_table_init(void)
|
|
|
|
{
|
2008-07-10 12:16:37 -06:00
|
|
|
static int dmar_table_initialized;
|
2007-11-21 16:07:14 -07:00
|
|
|
int ret;
|
|
|
|
|
2008-07-10 12:16:37 -06:00
|
|
|
if (dmar_table_initialized)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
dmar_table_initialized = 1;
|
|
|
|
|
2007-11-21 16:07:14 -07:00
|
|
|
ret = parse_dmar_table();
|
|
|
|
if (ret) {
|
2008-07-10 12:16:37 -06:00
|
|
|
if (ret != -ENODEV)
|
|
|
|
printk(KERN_INFO PREFIX "parse DMAR table failure.\n");
|
2007-11-21 16:07:14 -07:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2007-10-21 17:41:41 -06:00
|
|
|
if (list_empty(&dmar_drhd_units)) {
|
|
|
|
printk(KERN_INFO PREFIX "No DMAR devices found\n");
|
|
|
|
return -ENODEV;
|
|
|
|
}
|
2007-11-21 16:07:14 -07:00
|
|
|
|
2008-07-10 12:16:38 -06:00
|
|
|
#ifdef CONFIG_DMAR
|
2008-07-10 12:16:39 -06:00
|
|
|
if (list_empty(&dmar_rmrr_units))
|
2007-11-21 16:07:14 -07:00
|
|
|
printk(KERN_INFO PREFIX "No RMRR found\n");
|
2009-05-17 23:51:34 -06:00
|
|
|
|
|
|
|
if (list_empty(&dmar_atsr_units))
|
|
|
|
printk(KERN_INFO PREFIX "No ATSR found\n");
|
2008-07-10 12:16:38 -06:00
|
|
|
#endif
|
2007-11-21 16:07:14 -07:00
|
|
|
|
2007-10-21 17:41:41 -06:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2009-12-02 02:20:27 -07:00
|
|
|
static int bios_warned;
|
|
|
|
|
2009-11-09 15:15:15 -07:00
|
|
|
int __init check_zero_address(void)
|
|
|
|
{
|
|
|
|
struct acpi_table_dmar *dmar;
|
|
|
|
struct acpi_dmar_header *entry_header;
|
|
|
|
struct acpi_dmar_hardware_unit *drhd;
|
|
|
|
|
|
|
|
dmar = (struct acpi_table_dmar *)dmar_tbl;
|
|
|
|
entry_header = (struct acpi_dmar_header *)(dmar + 1);
|
|
|
|
|
|
|
|
while (((unsigned long)entry_header) <
|
|
|
|
(((unsigned long)dmar) + dmar_tbl->length)) {
|
|
|
|
/* Avoid looping forever on bad ACPI tables */
|
|
|
|
if (entry_header->length == 0) {
|
|
|
|
printk(KERN_WARNING PREFIX
|
|
|
|
"Invalid 0-length structure\n");
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (entry_header->type == ACPI_DMAR_TYPE_HARDWARE_UNIT) {
|
2009-12-02 02:17:13 -07:00
|
|
|
void __iomem *addr;
|
|
|
|
u64 cap, ecap;
|
|
|
|
|
2009-11-09 15:15:15 -07:00
|
|
|
drhd = (void *)entry_header;
|
|
|
|
if (!drhd->address) {
|
|
|
|
/* Promote an attitude of violence to a BIOS engineer today */
|
|
|
|
WARN(1, "Your BIOS is broken; DMAR reported at address zero!\n"
|
|
|
|
"BIOS vendor: %s; Ver: %s; Product Version: %s\n",
|
|
|
|
dmi_get_system_info(DMI_BIOS_VENDOR),
|
|
|
|
dmi_get_system_info(DMI_BIOS_VERSION),
|
|
|
|
dmi_get_system_info(DMI_PRODUCT_VERSION));
|
2009-12-02 02:20:27 -07:00
|
|
|
bios_warned = 1;
|
2009-12-02 02:17:13 -07:00
|
|
|
goto failed;
|
|
|
|
}
|
|
|
|
|
|
|
|
addr = early_ioremap(drhd->address, VTD_PAGE_SIZE);
|
|
|
|
if (!addr ) {
|
|
|
|
printk("IOMMU: can't validate: %llx\n", drhd->address);
|
|
|
|
goto failed;
|
|
|
|
}
|
|
|
|
cap = dmar_readq(addr + DMAR_CAP_REG);
|
|
|
|
ecap = dmar_readq(addr + DMAR_ECAP_REG);
|
|
|
|
early_iounmap(addr, VTD_PAGE_SIZE);
|
|
|
|
if (cap == (uint64_t)-1 && ecap == (uint64_t)-1) {
|
|
|
|
/* Promote an attitude of violence to a BIOS engineer today */
|
|
|
|
WARN(1, "Your BIOS is broken; DMAR reported at address %llx returns all ones!\n"
|
|
|
|
"BIOS vendor: %s; Ver: %s; Product Version: %s\n",
|
|
|
|
drhd->address,
|
|
|
|
dmi_get_system_info(DMI_BIOS_VENDOR),
|
|
|
|
dmi_get_system_info(DMI_BIOS_VERSION),
|
|
|
|
dmi_get_system_info(DMI_PRODUCT_VERSION));
|
2009-12-02 02:20:27 -07:00
|
|
|
bios_warned = 1;
|
2009-12-02 02:17:13 -07:00
|
|
|
goto failed;
|
2009-11-09 15:15:15 -07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
entry_header = ((void *)entry_header + entry_header->length);
|
|
|
|
}
|
|
|
|
return 1;
|
2009-12-02 02:17:13 -07:00
|
|
|
|
|
|
|
failed:
|
|
|
|
#ifdef CONFIG_DMAR
|
|
|
|
dmar_disabled = 1;
|
|
|
|
#endif
|
|
|
|
return 0;
|
2009-11-09 15:15:15 -07:00
|
|
|
}
|
|
|
|
|
2008-07-10 12:16:43 -06:00
|
|
|
void __init detect_intel_iommu(void)
|
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
2008-09-03 17:58:32 -06:00
|
|
|
ret = dmar_table_detect();
|
2009-11-09 15:15:15 -07:00
|
|
|
if (ret)
|
|
|
|
ret = check_zero_address();
|
2008-07-10 12:16:43 -06:00
|
|
|
{
|
2008-10-16 17:31:57 -06:00
|
|
|
#ifdef CONFIG_INTR_REMAP
|
2008-07-10 12:16:51 -06:00
|
|
|
struct acpi_table_dmar *dmar;
|
|
|
|
/*
|
|
|
|
* for now we will disable dma-remapping when interrupt
|
|
|
|
* remapping is enabled.
|
|
|
|
* When support for queued invalidation for IOTLB invalidation
|
|
|
|
* is added, we will not need this any more.
|
|
|
|
*/
|
|
|
|
dmar = (struct acpi_table_dmar *) dmar_tbl;
|
2008-10-16 17:31:57 -06:00
|
|
|
if (ret && cpu_has_x2apic && dmar->flags & 0x1)
|
2008-07-10 12:16:51 -06:00
|
|
|
printk(KERN_INFO
|
|
|
|
"Queued invalidation will be enabled to support "
|
|
|
|
"x2apic and Intr-remapping.\n");
|
2008-10-16 17:31:57 -06:00
|
|
|
#endif
|
|
|
|
#ifdef CONFIG_DMAR
|
2009-12-11 13:18:16 -07:00
|
|
|
if (ret && !no_iommu && !iommu_detected && !dmar_disabled) {
|
2008-07-10 12:16:43 -06:00
|
|
|
iommu_detected = 1;
|
2009-12-04 13:15:21 -07:00
|
|
|
/* Make sure ACS will be enabled */
|
|
|
|
pci_request_acs();
|
|
|
|
}
|
2009-11-10 03:46:16 -07:00
|
|
|
#endif
|
|
|
|
#ifdef CONFIG_X86
|
|
|
|
if (ret)
|
|
|
|
x86_init.iommu.iommu_init = intel_iommu_init;
|
2008-07-10 12:16:43 -06:00
|
|
|
#endif
|
2008-10-16 17:31:57 -06:00
|
|
|
}
|
2009-02-11 02:06:59 -07:00
|
|
|
early_acpi_os_unmap_memory(dmar_tbl, dmar_tbl_size);
|
2008-09-03 17:58:32 -06:00
|
|
|
dmar_tbl = NULL;
|
2008-07-10 12:16:43 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
|
2008-07-10 12:16:37 -06:00
|
|
|
int alloc_iommu(struct dmar_drhd_unit *drhd)
|
2008-07-10 12:16:35 -06:00
|
|
|
{
|
2008-07-10 12:16:36 -06:00
|
|
|
struct intel_iommu *iommu;
|
2008-07-10 12:16:35 -06:00
|
|
|
int map_size;
|
|
|
|
u32 ver;
|
2008-07-10 12:16:36 -06:00
|
|
|
static int iommu_allocated = 0;
|
2009-01-03 15:56:27 -07:00
|
|
|
int agaw = 0;
|
2009-04-24 18:30:20 -06:00
|
|
|
int msagaw = 0;
|
2008-07-10 12:16:36 -06:00
|
|
|
|
2009-12-02 02:20:27 -07:00
|
|
|
if (!drhd->reg_base_addr) {
|
|
|
|
if (!bios_warned) {
|
|
|
|
WARN(1, "Your BIOS is broken; DMAR reported at address zero!\n"
|
|
|
|
"BIOS vendor: %s; Ver: %s; Product Version: %s\n",
|
|
|
|
dmi_get_system_info(DMI_BIOS_VENDOR),
|
|
|
|
dmi_get_system_info(DMI_BIOS_VERSION),
|
|
|
|
dmi_get_system_info(DMI_PRODUCT_VERSION));
|
|
|
|
bios_warned = 1;
|
|
|
|
}
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
2008-07-10 12:16:36 -06:00
|
|
|
iommu = kzalloc(sizeof(*iommu), GFP_KERNEL);
|
|
|
|
if (!iommu)
|
2008-07-10 12:16:37 -06:00
|
|
|
return -ENOMEM;
|
2008-07-10 12:16:36 -06:00
|
|
|
|
|
|
|
iommu->seq_id = iommu_allocated++;
|
2009-03-16 18:04:55 -06:00
|
|
|
sprintf (iommu->name, "dmar%d", iommu->seq_id);
|
2008-07-10 12:16:35 -06:00
|
|
|
|
2008-10-16 19:02:32 -06:00
|
|
|
iommu->reg = ioremap(drhd->reg_base_addr, VTD_PAGE_SIZE);
|
2008-07-10 12:16:35 -06:00
|
|
|
if (!iommu->reg) {
|
|
|
|
printk(KERN_ERR "IOMMU: can't map the region\n");
|
|
|
|
goto error;
|
|
|
|
}
|
|
|
|
iommu->cap = dmar_readq(iommu->reg + DMAR_CAP_REG);
|
|
|
|
iommu->ecap = dmar_readq(iommu->reg + DMAR_ECAP_REG);
|
|
|
|
|
2009-08-04 02:17:20 -06:00
|
|
|
if (iommu->cap == (uint64_t)-1 && iommu->ecap == (uint64_t)-1) {
|
2009-12-02 02:20:27 -07:00
|
|
|
if (!bios_warned) {
|
|
|
|
/* Promote an attitude of violence to a BIOS engineer today */
|
|
|
|
WARN(1, "Your BIOS is broken; DMAR reported at address %llx returns all ones!\n"
|
|
|
|
"BIOS vendor: %s; Ver: %s; Product Version: %s\n",
|
|
|
|
drhd->reg_base_addr,
|
|
|
|
dmi_get_system_info(DMI_BIOS_VENDOR),
|
|
|
|
dmi_get_system_info(DMI_BIOS_VERSION),
|
|
|
|
dmi_get_system_info(DMI_PRODUCT_VERSION));
|
|
|
|
bios_warned = 1;
|
|
|
|
}
|
2009-08-04 02:17:20 -06:00
|
|
|
goto err_unmap;
|
|
|
|
}
|
|
|
|
|
2009-01-03 15:56:27 -07:00
|
|
|
#ifdef CONFIG_DMAR
|
2008-12-08 00:34:06 -07:00
|
|
|
agaw = iommu_calculate_agaw(iommu);
|
|
|
|
if (agaw < 0) {
|
|
|
|
printk(KERN_ERR
|
2009-04-24 18:30:20 -06:00
|
|
|
"Cannot get a valid agaw for iommu (seq_id = %d)\n",
|
|
|
|
iommu->seq_id);
|
2009-08-04 02:17:20 -06:00
|
|
|
goto err_unmap;
|
2009-04-24 18:30:20 -06:00
|
|
|
}
|
|
|
|
msagaw = iommu_calculate_max_sagaw(iommu);
|
|
|
|
if (msagaw < 0) {
|
|
|
|
printk(KERN_ERR
|
|
|
|
"Cannot get a valid max agaw for iommu (seq_id = %d)\n",
|
2008-12-08 00:34:06 -07:00
|
|
|
iommu->seq_id);
|
2009-08-04 02:17:20 -06:00
|
|
|
goto err_unmap;
|
2008-12-08 00:34:06 -07:00
|
|
|
}
|
2009-01-03 15:56:27 -07:00
|
|
|
#endif
|
2008-12-08 00:34:06 -07:00
|
|
|
iommu->agaw = agaw;
|
2009-04-24 18:30:20 -06:00
|
|
|
iommu->msagaw = msagaw;
|
2008-12-08 00:34:06 -07:00
|
|
|
|
2009-10-02 12:01:21 -06:00
|
|
|
iommu->node = -1;
|
|
|
|
|
2008-07-10 12:16:35 -06:00
|
|
|
/* the registers might be more than one page */
|
|
|
|
map_size = max_t(int, ecap_max_iotlb_offset(iommu->ecap),
|
|
|
|
cap_max_fault_reg_offset(iommu->cap));
|
2008-10-16 19:02:32 -06:00
|
|
|
map_size = VTD_PAGE_ALIGN(map_size);
|
|
|
|
if (map_size > VTD_PAGE_SIZE) {
|
2008-07-10 12:16:35 -06:00
|
|
|
iounmap(iommu->reg);
|
|
|
|
iommu->reg = ioremap(drhd->reg_base_addr, map_size);
|
|
|
|
if (!iommu->reg) {
|
|
|
|
printk(KERN_ERR "IOMMU: can't map the region\n");
|
|
|
|
goto error;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
ver = readl(iommu->reg + DMAR_VER_REG);
|
2009-08-04 02:17:20 -06:00
|
|
|
pr_info("IOMMU %llx: ver %d:%d cap %llx ecap %llx\n",
|
2008-10-16 19:02:32 -06:00
|
|
|
(unsigned long long)drhd->reg_base_addr,
|
|
|
|
DMAR_VER_MAJOR(ver), DMAR_VER_MINOR(ver),
|
|
|
|
(unsigned long long)iommu->cap,
|
|
|
|
(unsigned long long)iommu->ecap);
|
2008-07-10 12:16:35 -06:00
|
|
|
|
|
|
|
spin_lock_init(&iommu->register_lock);
|
|
|
|
|
|
|
|
drhd->iommu = iommu;
|
2008-07-10 12:16:37 -06:00
|
|
|
return 0;
|
2009-08-04 02:17:20 -06:00
|
|
|
|
|
|
|
err_unmap:
|
|
|
|
iounmap(iommu->reg);
|
|
|
|
error:
|
2008-07-10 12:16:35 -06:00
|
|
|
kfree(iommu);
|
2008-07-10 12:16:37 -06:00
|
|
|
return -1;
|
2008-07-10 12:16:35 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
void free_iommu(struct intel_iommu *iommu)
|
|
|
|
{
|
|
|
|
if (!iommu)
|
|
|
|
return;
|
|
|
|
|
|
|
|
#ifdef CONFIG_DMAR
|
|
|
|
free_dmar_iommu(iommu);
|
|
|
|
#endif
|
|
|
|
|
|
|
|
if (iommu->reg)
|
|
|
|
iounmap(iommu->reg);
|
|
|
|
kfree(iommu);
|
|
|
|
}
|
2008-07-10 12:16:42 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Reclaim all the submitted descriptors which have completed its work.
|
|
|
|
*/
|
|
|
|
static inline void reclaim_free_desc(struct q_inval *qi)
|
|
|
|
{
|
2009-05-17 23:51:35 -06:00
|
|
|
while (qi->desc_status[qi->free_tail] == QI_DONE ||
|
|
|
|
qi->desc_status[qi->free_tail] == QI_ABORT) {
|
2008-07-10 12:16:42 -06:00
|
|
|
qi->desc_status[qi->free_tail] = QI_FREE;
|
|
|
|
qi->free_tail = (qi->free_tail + 1) % QI_LENGTH;
|
|
|
|
qi->free_cnt++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2009-01-04 01:28:52 -07:00
|
|
|
static int qi_check_fault(struct intel_iommu *iommu, int index)
|
|
|
|
{
|
|
|
|
u32 fault;
|
2009-05-17 23:51:35 -06:00
|
|
|
int head, tail;
|
2009-01-04 01:28:52 -07:00
|
|
|
struct q_inval *qi = iommu->qi;
|
|
|
|
int wait_index = (index + 1) % QI_LENGTH;
|
|
|
|
|
2009-05-17 23:51:35 -06:00
|
|
|
if (qi->desc_status[wait_index] == QI_ABORT)
|
|
|
|
return -EAGAIN;
|
|
|
|
|
2009-01-04 01:28:52 -07:00
|
|
|
fault = readl(iommu->reg + DMAR_FSTS_REG);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If IQE happens, the head points to the descriptor associated
|
|
|
|
* with the error. No new descriptors are fetched until the IQE
|
|
|
|
* is cleared.
|
|
|
|
*/
|
|
|
|
if (fault & DMA_FSTS_IQE) {
|
|
|
|
head = readl(iommu->reg + DMAR_IQH_REG);
|
2009-05-17 23:51:35 -06:00
|
|
|
if ((head >> DMAR_IQ_SHIFT) == index) {
|
|
|
|
printk(KERN_ERR "VT-d detected invalid descriptor: "
|
|
|
|
"low=%llx, high=%llx\n",
|
|
|
|
(unsigned long long)qi->desc[index].low,
|
|
|
|
(unsigned long long)qi->desc[index].high);
|
2009-01-04 01:28:52 -07:00
|
|
|
memcpy(&qi->desc[index], &qi->desc[wait_index],
|
|
|
|
sizeof(struct qi_desc));
|
|
|
|
__iommu_flush_cache(iommu, &qi->desc[index],
|
|
|
|
sizeof(struct qi_desc));
|
|
|
|
writel(DMA_FSTS_IQE, iommu->reg + DMAR_FSTS_REG);
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2009-05-17 23:51:35 -06:00
|
|
|
/*
|
|
|
|
* If ITE happens, all pending wait_desc commands are aborted.
|
|
|
|
* No new descriptors are fetched until the ITE is cleared.
|
|
|
|
*/
|
|
|
|
if (fault & DMA_FSTS_ITE) {
|
|
|
|
head = readl(iommu->reg + DMAR_IQH_REG);
|
|
|
|
head = ((head >> DMAR_IQ_SHIFT) - 1 + QI_LENGTH) % QI_LENGTH;
|
|
|
|
head |= 1;
|
|
|
|
tail = readl(iommu->reg + DMAR_IQT_REG);
|
|
|
|
tail = ((tail >> DMAR_IQ_SHIFT) - 1 + QI_LENGTH) % QI_LENGTH;
|
|
|
|
|
|
|
|
writel(DMA_FSTS_ITE, iommu->reg + DMAR_FSTS_REG);
|
|
|
|
|
|
|
|
do {
|
|
|
|
if (qi->desc_status[head] == QI_IN_USE)
|
|
|
|
qi->desc_status[head] = QI_ABORT;
|
|
|
|
head = (head - 2 + QI_LENGTH) % QI_LENGTH;
|
|
|
|
} while (head != tail);
|
|
|
|
|
|
|
|
if (qi->desc_status[wait_index] == QI_ABORT)
|
|
|
|
return -EAGAIN;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (fault & DMA_FSTS_ICE)
|
|
|
|
writel(DMA_FSTS_ICE, iommu->reg + DMAR_FSTS_REG);
|
|
|
|
|
2009-01-04 01:28:52 -07:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2008-07-10 12:16:42 -06:00
|
|
|
/*
|
|
|
|
* Submit the queued invalidation descriptor to the remapping
|
|
|
|
* hardware unit and wait for its completion.
|
|
|
|
*/
|
2009-01-04 01:28:52 -07:00
|
|
|
int qi_submit_sync(struct qi_desc *desc, struct intel_iommu *iommu)
|
2008-07-10 12:16:42 -06:00
|
|
|
{
|
2009-05-17 23:51:35 -06:00
|
|
|
int rc;
|
2008-07-10 12:16:42 -06:00
|
|
|
struct q_inval *qi = iommu->qi;
|
|
|
|
struct qi_desc *hw, wait_desc;
|
|
|
|
int wait_index, index;
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
if (!qi)
|
2009-01-04 01:28:52 -07:00
|
|
|
return 0;
|
2008-07-10 12:16:42 -06:00
|
|
|
|
|
|
|
hw = qi->desc;
|
|
|
|
|
2009-05-17 23:51:35 -06:00
|
|
|
restart:
|
|
|
|
rc = 0;
|
|
|
|
|
2008-10-16 17:31:54 -06:00
|
|
|
spin_lock_irqsave(&qi->q_lock, flags);
|
2008-07-10 12:16:42 -06:00
|
|
|
while (qi->free_cnt < 3) {
|
2008-10-16 17:31:54 -06:00
|
|
|
spin_unlock_irqrestore(&qi->q_lock, flags);
|
2008-07-10 12:16:42 -06:00
|
|
|
cpu_relax();
|
2008-10-16 17:31:54 -06:00
|
|
|
spin_lock_irqsave(&qi->q_lock, flags);
|
2008-07-10 12:16:42 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
index = qi->free_head;
|
|
|
|
wait_index = (index + 1) % QI_LENGTH;
|
|
|
|
|
|
|
|
qi->desc_status[index] = qi->desc_status[wait_index] = QI_IN_USE;
|
|
|
|
|
|
|
|
hw[index] = *desc;
|
|
|
|
|
2009-01-04 01:28:52 -07:00
|
|
|
wait_desc.low = QI_IWD_STATUS_DATA(QI_DONE) |
|
|
|
|
QI_IWD_STATUS_WRITE | QI_IWD_TYPE;
|
2008-07-10 12:16:42 -06:00
|
|
|
wait_desc.high = virt_to_phys(&qi->desc_status[wait_index]);
|
|
|
|
|
|
|
|
hw[wait_index] = wait_desc;
|
|
|
|
|
|
|
|
__iommu_flush_cache(iommu, &hw[index], sizeof(struct qi_desc));
|
|
|
|
__iommu_flush_cache(iommu, &hw[wait_index], sizeof(struct qi_desc));
|
|
|
|
|
|
|
|
qi->free_head = (qi->free_head + 2) % QI_LENGTH;
|
|
|
|
qi->free_cnt -= 2;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* update the HW tail register indicating the presence of
|
|
|
|
* new descriptors.
|
|
|
|
*/
|
2009-05-17 23:51:35 -06:00
|
|
|
writel(qi->free_head << DMAR_IQ_SHIFT, iommu->reg + DMAR_IQT_REG);
|
2008-07-10 12:16:42 -06:00
|
|
|
|
|
|
|
while (qi->desc_status[wait_index] != QI_DONE) {
|
2008-10-16 17:31:54 -06:00
|
|
|
/*
|
|
|
|
* We will leave the interrupts disabled, to prevent interrupt
|
|
|
|
* context to queue another cmd while a cmd is already submitted
|
|
|
|
* and waiting for completion on this cpu. This is to avoid
|
|
|
|
* a deadlock where the interrupt context can wait indefinitely
|
|
|
|
* for free slots in the queue.
|
|
|
|
*/
|
2009-01-04 01:28:52 -07:00
|
|
|
rc = qi_check_fault(iommu, index);
|
|
|
|
if (rc)
|
2009-05-17 23:51:35 -06:00
|
|
|
break;
|
2009-01-04 01:28:52 -07:00
|
|
|
|
2008-07-10 12:16:42 -06:00
|
|
|
spin_unlock(&qi->q_lock);
|
|
|
|
cpu_relax();
|
|
|
|
spin_lock(&qi->q_lock);
|
|
|
|
}
|
2009-05-17 23:51:35 -06:00
|
|
|
|
|
|
|
qi->desc_status[index] = QI_DONE;
|
2008-07-10 12:16:42 -06:00
|
|
|
|
|
|
|
reclaim_free_desc(qi);
|
2008-10-16 17:31:54 -06:00
|
|
|
spin_unlock_irqrestore(&qi->q_lock, flags);
|
2009-01-04 01:28:52 -07:00
|
|
|
|
2009-05-17 23:51:35 -06:00
|
|
|
if (rc == -EAGAIN)
|
|
|
|
goto restart;
|
|
|
|
|
2009-01-04 01:28:52 -07:00
|
|
|
return rc;
|
2008-07-10 12:16:42 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Flush the global interrupt entry cache.
|
|
|
|
*/
|
|
|
|
void qi_global_iec(struct intel_iommu *iommu)
|
|
|
|
{
|
|
|
|
struct qi_desc desc;
|
|
|
|
|
|
|
|
desc.low = QI_IEC_TYPE;
|
|
|
|
desc.high = 0;
|
|
|
|
|
2009-01-04 01:28:52 -07:00
|
|
|
/* should never fail */
|
2008-07-10 12:16:42 -06:00
|
|
|
qi_submit_sync(&desc, iommu);
|
|
|
|
}
|
|
|
|
|
2009-05-10 10:16:06 -06:00
|
|
|
void qi_flush_context(struct intel_iommu *iommu, u16 did, u16 sid, u8 fm,
|
|
|
|
u64 type)
|
2008-10-16 17:31:55 -06:00
|
|
|
{
|
|
|
|
struct qi_desc desc;
|
|
|
|
|
|
|
|
desc.low = QI_CC_FM(fm) | QI_CC_SID(sid) | QI_CC_DID(did)
|
|
|
|
| QI_CC_GRAN(type) | QI_CC_TYPE;
|
|
|
|
desc.high = 0;
|
|
|
|
|
2009-05-10 10:16:06 -06:00
|
|
|
qi_submit_sync(&desc, iommu);
|
2008-10-16 17:31:55 -06:00
|
|
|
}
|
|
|
|
|
2009-05-10 12:58:49 -06:00
|
|
|
void qi_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,
|
|
|
|
unsigned int size_order, u64 type)
|
2008-10-16 17:31:55 -06:00
|
|
|
{
|
|
|
|
u8 dw = 0, dr = 0;
|
|
|
|
|
|
|
|
struct qi_desc desc;
|
|
|
|
int ih = 0;
|
|
|
|
|
|
|
|
if (cap_write_drain(iommu->cap))
|
|
|
|
dw = 1;
|
|
|
|
|
|
|
|
if (cap_read_drain(iommu->cap))
|
|
|
|
dr = 1;
|
|
|
|
|
|
|
|
desc.low = QI_IOTLB_DID(did) | QI_IOTLB_DR(dr) | QI_IOTLB_DW(dw)
|
|
|
|
| QI_IOTLB_GRAN(type) | QI_IOTLB_TYPE;
|
|
|
|
desc.high = QI_IOTLB_ADDR(addr) | QI_IOTLB_IH(ih)
|
|
|
|
| QI_IOTLB_AM(size_order);
|
|
|
|
|
2009-05-10 12:58:49 -06:00
|
|
|
qi_submit_sync(&desc, iommu);
|
2008-10-16 17:31:55 -06:00
|
|
|
}
|
|
|
|
|
2009-05-17 23:51:35 -06:00
|
|
|
void qi_flush_dev_iotlb(struct intel_iommu *iommu, u16 sid, u16 qdep,
|
|
|
|
u64 addr, unsigned mask)
|
|
|
|
{
|
|
|
|
struct qi_desc desc;
|
|
|
|
|
|
|
|
if (mask) {
|
|
|
|
BUG_ON(addr & ((1 << (VTD_PAGE_SHIFT + mask)) - 1));
|
|
|
|
addr |= (1 << (VTD_PAGE_SHIFT + mask - 1)) - 1;
|
|
|
|
desc.high = QI_DEV_IOTLB_ADDR(addr) | QI_DEV_IOTLB_SIZE;
|
|
|
|
} else
|
|
|
|
desc.high = QI_DEV_IOTLB_ADDR(addr);
|
|
|
|
|
|
|
|
if (qdep >= QI_DEV_IOTLB_MAX_INVS)
|
|
|
|
qdep = 0;
|
|
|
|
|
|
|
|
desc.low = QI_DEV_IOTLB_SID(sid) | QI_DEV_IOTLB_QDEP(qdep) |
|
|
|
|
QI_DIOTLB_TYPE;
|
|
|
|
|
|
|
|
qi_submit_sync(&desc, iommu);
|
|
|
|
}
|
|
|
|
|
2009-03-16 18:04:56 -06:00
|
|
|
/*
|
|
|
|
* Disable Queued Invalidation interface.
|
|
|
|
*/
|
|
|
|
void dmar_disable_qi(struct intel_iommu *iommu)
|
|
|
|
{
|
|
|
|
unsigned long flags;
|
|
|
|
u32 sts;
|
|
|
|
cycles_t start_time = get_cycles();
|
|
|
|
|
|
|
|
if (!ecap_qis(iommu->ecap))
|
|
|
|
return;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&iommu->register_lock, flags);
|
|
|
|
|
|
|
|
sts = dmar_readq(iommu->reg + DMAR_GSTS_REG);
|
|
|
|
if (!(sts & DMA_GSTS_QIES))
|
|
|
|
goto end;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Give a chance to HW to complete the pending invalidation requests.
|
|
|
|
*/
|
|
|
|
while ((readl(iommu->reg + DMAR_IQT_REG) !=
|
|
|
|
readl(iommu->reg + DMAR_IQH_REG)) &&
|
|
|
|
(DMAR_OPERATION_TIMEOUT > (get_cycles() - start_time)))
|
|
|
|
cpu_relax();
|
|
|
|
|
|
|
|
iommu->gcmd &= ~DMA_GCMD_QIE;
|
|
|
|
writel(iommu->gcmd, iommu->reg + DMAR_GCMD_REG);
|
|
|
|
|
|
|
|
IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG, readl,
|
|
|
|
!(sts & DMA_GSTS_QIES), sts);
|
|
|
|
end:
|
|
|
|
spin_unlock_irqrestore(&iommu->register_lock, flags);
|
|
|
|
}
|
|
|
|
|
2009-03-27 15:22:43 -06:00
|
|
|
/*
|
|
|
|
* Enable queued invalidation.
|
|
|
|
*/
|
|
|
|
static void __dmar_enable_qi(struct intel_iommu *iommu)
|
|
|
|
{
|
2009-05-10 13:30:58 -06:00
|
|
|
u32 sts;
|
2009-03-27 15:22:43 -06:00
|
|
|
unsigned long flags;
|
|
|
|
struct q_inval *qi = iommu->qi;
|
|
|
|
|
|
|
|
qi->free_head = qi->free_tail = 0;
|
|
|
|
qi->free_cnt = QI_LENGTH;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&iommu->register_lock, flags);
|
|
|
|
|
|
|
|
/* write zero to the tail reg */
|
|
|
|
writel(0, iommu->reg + DMAR_IQT_REG);
|
|
|
|
|
|
|
|
dmar_writeq(iommu->reg + DMAR_IQA_REG, virt_to_phys(qi->desc));
|
|
|
|
|
|
|
|
iommu->gcmd |= DMA_GCMD_QIE;
|
2009-05-10 13:30:58 -06:00
|
|
|
writel(iommu->gcmd, iommu->reg + DMAR_GCMD_REG);
|
2009-03-27 15:22:43 -06:00
|
|
|
|
|
|
|
/* Make sure hardware complete it */
|
|
|
|
IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG, readl, (sts & DMA_GSTS_QIES), sts);
|
|
|
|
|
|
|
|
spin_unlock_irqrestore(&iommu->register_lock, flags);
|
|
|
|
}
|
|
|
|
|
2008-07-10 12:16:42 -06:00
|
|
|
/*
|
|
|
|
* Enable Queued Invalidation interface. This is a must to support
|
|
|
|
* interrupt-remapping. Also used by DMA-remapping, which replaces
|
|
|
|
* register based IOTLB invalidation.
|
|
|
|
*/
|
|
|
|
int dmar_enable_qi(struct intel_iommu *iommu)
|
|
|
|
{
|
|
|
|
struct q_inval *qi;
|
2009-10-02 12:01:22 -06:00
|
|
|
struct page *desc_page;
|
2008-07-10 12:16:42 -06:00
|
|
|
|
|
|
|
if (!ecap_qis(iommu->ecap))
|
|
|
|
return -ENOENT;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* queued invalidation is already setup and enabled.
|
|
|
|
*/
|
|
|
|
if (iommu->qi)
|
|
|
|
return 0;
|
|
|
|
|
2009-03-16 18:05:05 -06:00
|
|
|
iommu->qi = kmalloc(sizeof(*qi), GFP_ATOMIC);
|
2008-07-10 12:16:42 -06:00
|
|
|
if (!iommu->qi)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
qi = iommu->qi;
|
|
|
|
|
2009-10-02 12:01:22 -06:00
|
|
|
|
|
|
|
desc_page = alloc_pages_node(iommu->node, GFP_ATOMIC | __GFP_ZERO, 0);
|
|
|
|
if (!desc_page) {
|
2008-07-10 12:16:42 -06:00
|
|
|
kfree(qi);
|
|
|
|
iommu->qi = 0;
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
|
2009-10-02 12:01:22 -06:00
|
|
|
qi->desc = page_address(desc_page);
|
|
|
|
|
2009-03-16 18:05:05 -06:00
|
|
|
qi->desc_status = kmalloc(QI_LENGTH * sizeof(int), GFP_ATOMIC);
|
2008-07-10 12:16:42 -06:00
|
|
|
if (!qi->desc_status) {
|
|
|
|
free_page((unsigned long) qi->desc);
|
|
|
|
kfree(qi);
|
|
|
|
iommu->qi = 0;
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
|
|
|
|
qi->free_head = qi->free_tail = 0;
|
|
|
|
qi->free_cnt = QI_LENGTH;
|
|
|
|
|
|
|
|
spin_lock_init(&qi->q_lock);
|
|
|
|
|
2009-03-27 15:22:43 -06:00
|
|
|
__dmar_enable_qi(iommu);
|
2008-07-10 12:16:42 -06:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
2009-03-16 18:04:54 -06:00
|
|
|
|
|
|
|
/* iommu interrupt handling. Most stuff are MSI-like. */
|
|
|
|
|
2009-03-16 18:04:55 -06:00
|
|
|
enum faulttype {
|
|
|
|
DMA_REMAP,
|
|
|
|
INTR_REMAP,
|
|
|
|
UNKNOWN,
|
|
|
|
};
|
|
|
|
|
|
|
|
static const char *dma_remap_fault_reasons[] =
|
2009-03-16 18:04:54 -06:00
|
|
|
{
|
|
|
|
"Software",
|
|
|
|
"Present bit in root entry is clear",
|
|
|
|
"Present bit in context entry is clear",
|
|
|
|
"Invalid context entry",
|
|
|
|
"Access beyond MGAW",
|
|
|
|
"PTE Write access is not set",
|
|
|
|
"PTE Read access is not set",
|
|
|
|
"Next page table ptr is invalid",
|
|
|
|
"Root table address invalid",
|
|
|
|
"Context table ptr is invalid",
|
|
|
|
"non-zero reserved fields in RTP",
|
|
|
|
"non-zero reserved fields in CTP",
|
|
|
|
"non-zero reserved fields in PTE",
|
|
|
|
};
|
2009-03-16 18:04:55 -06:00
|
|
|
|
|
|
|
static const char *intr_remap_fault_reasons[] =
|
|
|
|
{
|
|
|
|
"Detected reserved fields in the decoded interrupt-remapped request",
|
|
|
|
"Interrupt index exceeded the interrupt-remapping table size",
|
|
|
|
"Present field in the IRTE entry is clear",
|
|
|
|
"Error accessing interrupt-remapping table pointed by IRTA_REG",
|
|
|
|
"Detected reserved fields in the IRTE entry",
|
|
|
|
"Blocked a compatibility format interrupt request",
|
|
|
|
"Blocked an interrupt request due to source-id verification failure",
|
|
|
|
};
|
|
|
|
|
2009-03-16 18:04:54 -06:00
|
|
|
#define MAX_FAULT_REASON_IDX (ARRAY_SIZE(fault_reason_strings) - 1)
|
|
|
|
|
2009-03-16 18:04:55 -06:00
|
|
|
const char *dmar_get_fault_reason(u8 fault_reason, int *fault_type)
|
2009-03-16 18:04:54 -06:00
|
|
|
{
|
2009-03-16 18:04:55 -06:00
|
|
|
if (fault_reason >= 0x20 && (fault_reason <= 0x20 +
|
|
|
|
ARRAY_SIZE(intr_remap_fault_reasons))) {
|
|
|
|
*fault_type = INTR_REMAP;
|
|
|
|
return intr_remap_fault_reasons[fault_reason - 0x20];
|
|
|
|
} else if (fault_reason < ARRAY_SIZE(dma_remap_fault_reasons)) {
|
|
|
|
*fault_type = DMA_REMAP;
|
|
|
|
return dma_remap_fault_reasons[fault_reason];
|
|
|
|
} else {
|
|
|
|
*fault_type = UNKNOWN;
|
2009-03-16 18:04:54 -06:00
|
|
|
return "Unknown";
|
2009-03-16 18:04:55 -06:00
|
|
|
}
|
2009-03-16 18:04:54 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
void dmar_msi_unmask(unsigned int irq)
|
|
|
|
{
|
|
|
|
struct intel_iommu *iommu = get_irq_data(irq);
|
|
|
|
unsigned long flag;
|
|
|
|
|
|
|
|
/* unmask it */
|
|
|
|
spin_lock_irqsave(&iommu->register_lock, flag);
|
|
|
|
writel(0, iommu->reg + DMAR_FECTL_REG);
|
|
|
|
/* Read a reg to force flush the post write */
|
|
|
|
readl(iommu->reg + DMAR_FECTL_REG);
|
|
|
|
spin_unlock_irqrestore(&iommu->register_lock, flag);
|
|
|
|
}
|
|
|
|
|
|
|
|
void dmar_msi_mask(unsigned int irq)
|
|
|
|
{
|
|
|
|
unsigned long flag;
|
|
|
|
struct intel_iommu *iommu = get_irq_data(irq);
|
|
|
|
|
|
|
|
/* mask it */
|
|
|
|
spin_lock_irqsave(&iommu->register_lock, flag);
|
|
|
|
writel(DMA_FECTL_IM, iommu->reg + DMAR_FECTL_REG);
|
|
|
|
/* Read a reg to force flush the post write */
|
|
|
|
readl(iommu->reg + DMAR_FECTL_REG);
|
|
|
|
spin_unlock_irqrestore(&iommu->register_lock, flag);
|
|
|
|
}
|
|
|
|
|
|
|
|
void dmar_msi_write(int irq, struct msi_msg *msg)
|
|
|
|
{
|
|
|
|
struct intel_iommu *iommu = get_irq_data(irq);
|
|
|
|
unsigned long flag;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&iommu->register_lock, flag);
|
|
|
|
writel(msg->data, iommu->reg + DMAR_FEDATA_REG);
|
|
|
|
writel(msg->address_lo, iommu->reg + DMAR_FEADDR_REG);
|
|
|
|
writel(msg->address_hi, iommu->reg + DMAR_FEUADDR_REG);
|
|
|
|
spin_unlock_irqrestore(&iommu->register_lock, flag);
|
|
|
|
}
|
|
|
|
|
|
|
|
void dmar_msi_read(int irq, struct msi_msg *msg)
|
|
|
|
{
|
|
|
|
struct intel_iommu *iommu = get_irq_data(irq);
|
|
|
|
unsigned long flag;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&iommu->register_lock, flag);
|
|
|
|
msg->data = readl(iommu->reg + DMAR_FEDATA_REG);
|
|
|
|
msg->address_lo = readl(iommu->reg + DMAR_FEADDR_REG);
|
|
|
|
msg->address_hi = readl(iommu->reg + DMAR_FEUADDR_REG);
|
|
|
|
spin_unlock_irqrestore(&iommu->register_lock, flag);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int dmar_fault_do_one(struct intel_iommu *iommu, int type,
|
|
|
|
u8 fault_reason, u16 source_id, unsigned long long addr)
|
|
|
|
{
|
|
|
|
const char *reason;
|
2009-03-16 18:04:55 -06:00
|
|
|
int fault_type;
|
2009-03-16 18:04:54 -06:00
|
|
|
|
2009-03-16 18:04:55 -06:00
|
|
|
reason = dmar_get_fault_reason(fault_reason, &fault_type);
|
2009-03-16 18:04:54 -06:00
|
|
|
|
2009-03-16 18:04:55 -06:00
|
|
|
if (fault_type == INTR_REMAP)
|
|
|
|
printk(KERN_ERR "INTR-REMAP: Request device [[%02x:%02x.%d] "
|
|
|
|
"fault index %llx\n"
|
|
|
|
"INTR-REMAP:[fault reason %02d] %s\n",
|
|
|
|
(source_id >> 8), PCI_SLOT(source_id & 0xFF),
|
|
|
|
PCI_FUNC(source_id & 0xFF), addr >> 48,
|
|
|
|
fault_reason, reason);
|
|
|
|
else
|
|
|
|
printk(KERN_ERR
|
|
|
|
"DMAR:[%s] Request device [%02x:%02x.%d] "
|
|
|
|
"fault addr %llx \n"
|
|
|
|
"DMAR:[fault reason %02d] %s\n",
|
|
|
|
(type ? "DMA Read" : "DMA Write"),
|
|
|
|
(source_id >> 8), PCI_SLOT(source_id & 0xFF),
|
|
|
|
PCI_FUNC(source_id & 0xFF), addr, fault_reason, reason);
|
2009-03-16 18:04:54 -06:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
#define PRIMARY_FAULT_REG_LEN (16)
|
2009-03-16 18:04:57 -06:00
|
|
|
irqreturn_t dmar_fault(int irq, void *dev_id)
|
2009-03-16 18:04:54 -06:00
|
|
|
{
|
|
|
|
struct intel_iommu *iommu = dev_id;
|
|
|
|
int reg, fault_index;
|
|
|
|
u32 fault_status;
|
|
|
|
unsigned long flag;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&iommu->register_lock, flag);
|
|
|
|
fault_status = readl(iommu->reg + DMAR_FSTS_REG);
|
2009-03-16 18:04:55 -06:00
|
|
|
if (fault_status)
|
|
|
|
printk(KERN_ERR "DRHD: handling fault status reg %x\n",
|
|
|
|
fault_status);
|
2009-03-16 18:04:54 -06:00
|
|
|
|
|
|
|
/* TBD: ignore advanced fault log currently */
|
|
|
|
if (!(fault_status & DMA_FSTS_PPF))
|
2009-03-16 18:04:55 -06:00
|
|
|
goto clear_rest;
|
2009-03-16 18:04:54 -06:00
|
|
|
|
|
|
|
fault_index = dma_fsts_fault_record_index(fault_status);
|
|
|
|
reg = cap_fault_reg_offset(iommu->cap);
|
|
|
|
while (1) {
|
|
|
|
u8 fault_reason;
|
|
|
|
u16 source_id;
|
|
|
|
u64 guest_addr;
|
|
|
|
int type;
|
|
|
|
u32 data;
|
|
|
|
|
|
|
|
/* highest 32 bits */
|
|
|
|
data = readl(iommu->reg + reg +
|
|
|
|
fault_index * PRIMARY_FAULT_REG_LEN + 12);
|
|
|
|
if (!(data & DMA_FRCD_F))
|
|
|
|
break;
|
|
|
|
|
|
|
|
fault_reason = dma_frcd_fault_reason(data);
|
|
|
|
type = dma_frcd_type(data);
|
|
|
|
|
|
|
|
data = readl(iommu->reg + reg +
|
|
|
|
fault_index * PRIMARY_FAULT_REG_LEN + 8);
|
|
|
|
source_id = dma_frcd_source_id(data);
|
|
|
|
|
|
|
|
guest_addr = dmar_readq(iommu->reg + reg +
|
|
|
|
fault_index * PRIMARY_FAULT_REG_LEN);
|
|
|
|
guest_addr = dma_frcd_page_addr(guest_addr);
|
|
|
|
/* clear the fault */
|
|
|
|
writel(DMA_FRCD_F, iommu->reg + reg +
|
|
|
|
fault_index * PRIMARY_FAULT_REG_LEN + 12);
|
|
|
|
|
|
|
|
spin_unlock_irqrestore(&iommu->register_lock, flag);
|
|
|
|
|
|
|
|
dmar_fault_do_one(iommu, type, fault_reason,
|
|
|
|
source_id, guest_addr);
|
|
|
|
|
|
|
|
fault_index++;
|
2009-08-19 15:26:11 -06:00
|
|
|
if (fault_index >= cap_num_fault_regs(iommu->cap))
|
2009-03-16 18:04:54 -06:00
|
|
|
fault_index = 0;
|
|
|
|
spin_lock_irqsave(&iommu->register_lock, flag);
|
|
|
|
}
|
2009-03-16 18:04:55 -06:00
|
|
|
clear_rest:
|
|
|
|
/* clear all the other faults */
|
2009-03-16 18:04:54 -06:00
|
|
|
fault_status = readl(iommu->reg + DMAR_FSTS_REG);
|
2009-03-16 18:04:55 -06:00
|
|
|
writel(fault_status, iommu->reg + DMAR_FSTS_REG);
|
2009-03-16 18:04:54 -06:00
|
|
|
|
|
|
|
spin_unlock_irqrestore(&iommu->register_lock, flag);
|
|
|
|
return IRQ_HANDLED;
|
|
|
|
}
|
|
|
|
|
|
|
|
int dmar_set_interrupt(struct intel_iommu *iommu)
|
|
|
|
{
|
|
|
|
int irq, ret;
|
|
|
|
|
2009-03-16 18:04:55 -06:00
|
|
|
/*
|
|
|
|
* Check if the fault interrupt is already initialized.
|
|
|
|
*/
|
|
|
|
if (iommu->irq)
|
|
|
|
return 0;
|
|
|
|
|
2009-03-16 18:04:54 -06:00
|
|
|
irq = create_irq();
|
|
|
|
if (!irq) {
|
|
|
|
printk(KERN_ERR "IOMMU: no free vectors\n");
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
|
|
|
set_irq_data(irq, iommu);
|
|
|
|
iommu->irq = irq;
|
|
|
|
|
|
|
|
ret = arch_setup_dmar_msi(irq);
|
|
|
|
if (ret) {
|
|
|
|
set_irq_data(irq, NULL);
|
|
|
|
iommu->irq = 0;
|
|
|
|
destroy_irq(irq);
|
2009-05-13 16:55:52 -06:00
|
|
|
return ret;
|
2009-03-16 18:04:54 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
ret = request_irq(irq, dmar_fault, 0, iommu->name, iommu);
|
|
|
|
if (ret)
|
|
|
|
printk(KERN_ERR "IOMMU: can't request irq\n");
|
|
|
|
return ret;
|
|
|
|
}
|
2009-03-16 18:04:55 -06:00
|
|
|
|
|
|
|
int __init enable_drhd_fault_handling(void)
|
|
|
|
{
|
|
|
|
struct dmar_drhd_unit *drhd;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Enable fault control interrupt.
|
|
|
|
*/
|
|
|
|
for_each_drhd_unit(drhd) {
|
|
|
|
int ret;
|
|
|
|
struct intel_iommu *iommu = drhd->iommu;
|
|
|
|
ret = dmar_set_interrupt(iommu);
|
|
|
|
|
|
|
|
if (ret) {
|
|
|
|
printk(KERN_ERR "DRHD %Lx: failed to enable fault, "
|
|
|
|
" interrupt, ret %d\n",
|
|
|
|
(unsigned long long)drhd->reg_base_addr, ret);
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
2009-03-27 15:22:43 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Re-enable Queued Invalidation interface.
|
|
|
|
*/
|
|
|
|
int dmar_reenable_qi(struct intel_iommu *iommu)
|
|
|
|
{
|
|
|
|
if (!ecap_qis(iommu->ecap))
|
|
|
|
return -ENOENT;
|
|
|
|
|
|
|
|
if (!iommu->qi)
|
|
|
|
return -ENOENT;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* First disable queued invalidation.
|
|
|
|
*/
|
|
|
|
dmar_disable_qi(iommu);
|
|
|
|
/*
|
|
|
|
* Then enable queued invalidation again. Since there is no pending
|
|
|
|
* invalidation requests now, it's safe to re-enable queued
|
|
|
|
* invalidation.
|
|
|
|
*/
|
|
|
|
__dmar_enable_qi(iommu);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
2009-09-09 10:05:39 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Check interrupt remapping support in DMAR table description.
|
|
|
|
*/
|
|
|
|
int dmar_ir_support(void)
|
|
|
|
{
|
|
|
|
struct acpi_table_dmar *dmar;
|
|
|
|
dmar = (struct acpi_table_dmar *)dmar_tbl;
|
|
|
|
return dmar->flags & 0x1;
|
|
|
|
}
|