redonkable
/
alistair23-linux
1
0
Fork 0
Code Releases Activity

Round two of 4.6 merge window patches 

- A few minor core fixups needed for the next patch series
 - The IB SRIOV series.  This has bounced around for several versions.
   Of note is the fact that the first patch in this series effects
   the net core.  It was directed to netdev and DaveM for each iteration
   of the series (three versions total).  Dave did not object, but did
   not respond either.  I've taken this as permission to move forward
   with the series.
 - The new Intel X722 iWARP driver
 - A huge set of updates to the Intel hfi1 driver.  Of particular interest
   here is that we have left the driver in staging since it still has an
   API that people object to.  Intel is working on a fix, but getting
   these patches in now helps keep me sane as the upstream and Intel's
   trees were over 300 patches apart.
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1
 
 iQIcBAABAgAGBQJW8HR9AAoJELgmozMOVy/dDYMP+wSBALhIdV/pqVzdLCGfIUbK
 H5agonm/3b/Oj74W30w2JYqXBFfZC2LGVJy6OwocJ3wK04v/KfZbA9G+QsOuh2hQ
 Db+tFn1eoltvzrcx3k/a7x6zHGC4YyxyH9OX2B3QfRsNHeE7PG9KGp5dfEs2OH1r
 WGp3jMLAsHf7o8uKpa0jyTEUEErATaTlG+YoaJ+BGHwurgCNy8ni+wAn+EAFiJ3w
 iEJhcXB6KY69vkLsrLYuT9xxJn4udFJ3QEk8xdPkpLKsu+6Ue5i/eNQ19VfbpZgR
 c6fTc8genfIv5S+fis+0P44u1oA7Kl2JT6IZYLi35gJ60ZmxTD+7GruWP3xX/wJ2
 zuR3sTj5fjcFWenk087RSIU/EK87ONPD4g9QPdZpf3FtgleTVKk3YDlqwjqf8pgv
 cO6gQ1BcOBnixJvhjNFiX1c2hvNhb3CkgObly1JBwhcCzZhLkV7BNFPbZuDHAeAx
 VqzNEUse4hupkgiiuiGgudcJ4fsSxMW37kyfX9QC/qyk6YVuUDbrekcWI+MAKot7
 5e5dHqFExpbn1Zgvc8yfvh88H2MUQAgaYwjanWF/qpppOPRd01nTisVQIOJn7s5C
 arcWzvocpQe0GL2UsvDoWwAABXznL3bnnAoCyTWOES2RhOOcw0Ibw46Jl8FQ8gnl
 2IRxQ+ltNEscb2cwi5wE
 =t2Ko
 -----END PGP SIGNATURE-----

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma

Pull more rdma updates from Doug Ledford:
 "Round two of 4.6 merge window patches.

  This is a monster pull request.  I held off on the hfi1 driver updates
  (the hfi1 driver is intimately tied to the qib driver and the new
  rdmavt software library that was created to help both of them) in my
  first pull request.  The hfi1/qib/rdmavt update is probably 90% of
  this pull request.  The hfi1 driver is being left in staging so that
  it can be fixed up in regards to the API that Al and yourself didn't
  like.  Intel has agreed to do the work, but in the meantime, this
  clears out 300+ patches in the backlog queue and brings my tree and
  their tree closer to sync.

  This also includes about 10 patches to the core and a few to mlx5 to
  create an infrastructure for configuring SRIOV ports on IB devices.
  That series includes one patch to the net core that we sent to netdev@
  and Dave Miller with each of the three revisions to the series.  We
  didn't get any response to the patch, so we took that as implicit
  approval.

  Finally, this series includes Intel's new iWARP driver for their x722
  cards.  It's not nearly the beast as the hfi1 driver.  It also has a
  linux-next merge issue, but that has been resolved and it now passes
  just fine.

  Summary:

   - A few minor core fixups needed for the next patch series

   - The IB SRIOV series.  This has bounced around for several versions.
     Of note is the fact that the first patch in this series effects the
     net core.  It was directed to netdev and DaveM for each iteration
     of the series (three versions total).  Dave did not object, but did
     not respond either.  I've taken this as permission to move forward
     with the series.

   - The new Intel X722 iWARP driver

   - A huge set of updates to the Intel hfi1 driver.  Of particular
     interest here is that we have left the driver in staging since it
     still has an API that people object to.  Intel is working on a fix,
     but getting these patches in now helps keep me sane as the upstream
     and Intel's trees were over 300 patches apart"

* tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma: (362 commits)
  IB/ipoib: Allow mcast packets from other VFs
  IB/mlx5: Implement callbacks for manipulating VFs
  net/mlx5_core: Implement modify HCA vport command
  net/mlx5_core: Add VF param when querying vport counter
  IB/ipoib: Add ndo operations for configuring VFs
  IB/core: Add interfaces to control VF attributes
  IB/core: Support accessing SA in virtualized environment
  IB/core: Add subnet prefix to port info
  IB/mlx5: Fix decision on using MAD_IFC
  net/core: Add support for configuring VF GUIDs
  IB/{core, ulp} Support above 32 possible device capability flags
  IB/core: Replace setting the zero values in ib_uverbs_ex_query_device
  net/mlx5_core: Introduce offload arithmetic hardware capabilities
  net/mlx5_core: Refactor device capability function
  net/mlx5_core: Fix caching ATOMIC endian mode capability
  ib_srpt: fix a WARN_ON() message
  i40iw: Replace the obsolete crypto hash interface with shash
  IB/hfi1: Add SDMA cache eviction algorithm
  IB/hfi1: Switch to using the pin query function
  IB/hfi1: Specify mm when releasing pages
  ...
hifive-unleashed-5.1
Linus Torvalds 2016-03-22 15:48:44 -07:00
parent 01cde1538e 520a07bff6
commit b8ba452683
192 changed files with 49090 additions and 15229 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
Documentation/infiniband/sysfs.txt
View File

@ -78,9 +78,10 @@ HFI1
chip_reset - diagnostic (root only)
boardversion - board version
ports/1/
CMgtA/
CCMgtA/
cc_settings_bin - CCA tables used by PSM2
cc_table_bin
cc_prescan - enable prescaning for faster BECN response
sc2v/ - 32 files (0 - 31) used to translate sl->vl
sl2sc/ - 32 files (0 - 31) used to translate sl->sc
vl2mtu/ - 16 (0 - 15) files used to determine MTU for vl

16
MAINTAINERS
View File

@ -5770,6 +5770,16 @@ F: Documentation/networking/i40evf.txt
F: drivers/net/ethernet/intel/
F: drivers/net/ethernet/intel/*/
INTEL RDMA RNIC DRIVER
M: Faisal Latif <faisal.latif@intel.com>
R: Chien Tin Tung <chien.tin.tung@intel.com>
R: Mustafa Ismail <mustafa.ismail@intel.com>
R: Shiraz Saleem <shiraz.saleem@intel.com>
R: Tatyana Nikolova <tatyana.e.nikolova@intel.com>
L: linux-rdma@vger.kernel.org
S: Supported
F: drivers/infiniband/hw/i40iw/
INTEL-MID GPIO DRIVER
M: David Cohen <david.a.cohen@linux.intel.com>
L: linux-gpio@vger.kernel.org
@ -9224,6 +9234,12 @@ S: Supported
F: net/rds/
F: Documentation/networking/rds.txt
RDMAVT - RDMA verbs software
M: Dennis Dalessandro <dennis.dalessandro@intel.com>
L: linux-rdma@vger.kernel.org
S: Supported
F: drivers/infiniband/sw/rdmavt
READ-COPY UPDATE (RCU)
M: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
M: Josh Triplett <josh@joshtriplett.org>

3
drivers/infiniband/Kconfig
View File

@ -68,6 +68,7 @@ source "drivers/infiniband/hw/mthca/Kconfig"
source "drivers/infiniband/hw/qib/Kconfig"
source "drivers/infiniband/hw/cxgb3/Kconfig"
source "drivers/infiniband/hw/cxgb4/Kconfig"
source "drivers/infiniband/hw/i40iw/Kconfig"
source "drivers/infiniband/hw/mlx4/Kconfig"
source "drivers/infiniband/hw/mlx5/Kconfig"
source "drivers/infiniband/hw/nes/Kconfig"
@ -82,4 +83,6 @@ source "drivers/infiniband/ulp/srpt/Kconfig"
source "drivers/infiniband/ulp/iser/Kconfig"
source "drivers/infiniband/ulp/isert/Kconfig"
source "drivers/infiniband/sw/rdmavt/Kconfig"
endif # INFINIBAND

1
drivers/infiniband/Makefile
View File

@ -1,3 +1,4 @@
obj-$(CONFIG_INFINIBAND) += core/
obj-$(CONFIG_INFINIBAND) += hw/
obj-$(CONFIG_INFINIBAND) += ulp/
obj-$(CONFIG_INFINIBAND) += sw/

15
drivers/infiniband/core/device.c
View File

@ -650,10 +650,23 @@ int ib_query_port(struct ib_device *device,
u8 port_num,
struct ib_port_attr *port_attr)
{
union ib_gid gid;
int err;
if (port_num < rdma_start_port(device) || port_num > rdma_end_port(device))
return -EINVAL;
return device->query_port(device, port_num, port_attr);
memset(port_attr, 0, sizeof(*port_attr));
err = device->query_port(device, port_num, port_attr);
if (err || port_attr->subnet_prefix)
return err;
err = ib_query_gid(device, port_num, 0, &gid, NULL);
if (err)
return err;
port_attr->subnet_prefix = be64_to_cpu(gid.global.subnet_prefix);
return 0;
}
EXPORT_SYMBOL(ib_query_port);

5
drivers/infiniband/core/sa_query.c
View File

@ -885,6 +885,11 @@ static void update_sm_ah(struct work_struct *work)
ah_attr.dlid = port_attr.sm_lid;
ah_attr.sl = port_attr.sm_sl;
ah_attr.port_num = port->port_num;
if (port_attr.grh_required) {
ah_attr.ah_flags = IB_AH_GRH;
ah_attr.grh.dgid.global.subnet_prefix = cpu_to_be64(port_attr.subnet_prefix);
ah_attr.grh.dgid.global.interface_id = cpu_to_be64(IB_SA_WELL_KNOWN_GUID);
}
new_ah->ah = ib_create_ah(port->agent->qp->pd, &ah_attr);
if (IS_ERR(new_ah->ah)) {

17
drivers/infiniband/core/uverbs_cmd.c
View File

@ -402,7 +402,7 @@ static void copy_query_dev_fields(struct ib_uverbs_file *file,
resp->hw_ver = attr->hw_ver;
resp->max_qp = attr->max_qp;
resp->max_qp_wr = attr->max_qp_wr;
resp->device_cap_flags = attr->device_cap_flags;
resp->device_cap_flags = lower_32_bits(attr->device_cap_flags);
resp->max_sge = attr->max_sge;
resp->max_sge_rd = attr->max_sge_rd;
resp->max_cq = attr->max_cq;
@ -3600,9 +3600,9 @@ int ib_uverbs_ex_query_device(struct ib_uverbs_file *file,
struct ib_udata *ucore,
struct ib_udata *uhw)
{
struct ib_uverbs_ex_query_device_resp resp;
struct ib_uverbs_ex_query_device_resp resp = { {0} };
struct ib_uverbs_ex_query_device cmd;
struct ib_device_attr attr;
struct ib_device_attr attr = {0};
int err;
if (ucore->inlen < sizeof(cmd))
@ -3623,14 +3623,11 @@ int ib_uverbs_ex_query_device(struct ib_uverbs_file *file,
if (ucore->outlen < resp.response_length)
return -ENOSPC;
memset(&attr, 0, sizeof(attr));
err = ib_dev->query_device(ib_dev, &attr, uhw);
if (err)
return err;
copy_query_dev_fields(file, ib_dev, &resp.base, &attr);
resp.comp_mask = 0;
if (ucore->outlen < resp.response_length + sizeof(resp.odp_caps))
goto end;
@ -3643,9 +3640,6 @@ int ib_uverbs_ex_query_device(struct ib_uverbs_file *file,
attr.odp_caps.per_transport_caps.uc_odp_caps;
resp.odp_caps.per_transport_caps.ud_odp_caps =
attr.odp_caps.per_transport_caps.ud_odp_caps;
resp.odp_caps.reserved = 0;
#else
memset(&resp.odp_caps, 0, sizeof(resp.odp_caps));
#endif
resp.response_length += sizeof(resp.odp_caps);
@ -3663,8 +3657,5 @@ int ib_uverbs_ex_query_device(struct ib_uverbs_file *file,
end:
err = ib_copy_to_udata(ucore, &resp, resp.response_length);
if (err)
return err;
return 0;
return err;
}

40
drivers/infiniband/core/verbs.c
View File

@ -1551,6 +1551,46 @@ int ib_check_mr_status(struct ib_mr *mr, u32 check_mask,
}
EXPORT_SYMBOL(ib_check_mr_status);
int ib_set_vf_link_state(struct ib_device *device, int vf, u8 port,
int state)
{
if (!device->set_vf_link_state)
return -ENOSYS;
return device->set_vf_link_state(device, vf, port, state);
}
EXPORT_SYMBOL(ib_set_vf_link_state);
int ib_get_vf_config(struct ib_device *device, int vf, u8 port,
struct ifla_vf_info *info)
{
if (!device->get_vf_config)
return -ENOSYS;
return device->get_vf_config(device, vf, port, info);
}
EXPORT_SYMBOL(ib_get_vf_config);
int ib_get_vf_stats(struct ib_device *device, int vf, u8 port,
struct ifla_vf_stats *stats)
{
if (!device->get_vf_stats)
return -ENOSYS;
return device->get_vf_stats(device, vf, port, stats);
}
EXPORT_SYMBOL(ib_get_vf_stats);
int ib_set_vf_guid(struct ib_device *device, int vf, u8 port, u64 guid,
int type)
{
if (!device->set_vf_guid)
return -ENOSYS;
return device->set_vf_guid(device, vf, port, guid, type);
}
EXPORT_SYMBOL(ib_set_vf_guid);
/**
* ib_map_mr_sg() - Map the largest prefix of a dma mapped SG list
* and set it the memory region.

1
drivers/infiniband/hw/Makefile
View File

@ -2,6 +2,7 @@ obj-$(CONFIG_INFINIBAND_MTHCA) += mthca/
obj-$(CONFIG_INFINIBAND_QIB) += qib/
obj-$(CONFIG_INFINIBAND_CXGB3) += cxgb3/
obj-$(CONFIG_INFINIBAND_CXGB4) += cxgb4/
obj-$(CONFIG_INFINIBAND_I40IW) += i40iw/
obj-$(CONFIG_MLX4_INFINIBAND) += mlx4/
obj-$(CONFIG_MLX5_INFINIBAND) += mlx5/
obj-$(CONFIG_INFINIBAND_NES) += nes/

7
drivers/infiniband/hw/i40iw/Kconfig 100644
View File

@ -0,0 +1,7 @@
config INFINIBAND_I40IW
tristate "Intel(R) Ethernet X722 iWARP Driver"
depends on INET && I40E
select GENERIC_ALLOCATOR
---help---
Intel(R) Ethernet X722 iWARP Driver
INET && I40IW && INFINIBAND && I40E

9
drivers/infiniband/hw/i40iw/Makefile 100644
View File

@ -0,0 +1,9 @@
ccflags-y := -Idrivers/net/ethernet/intel/i40e
obj-$(CONFIG_INFINIBAND_I40IW) += i40iw.o
i40iw-objs :=\
i40iw_cm.o i40iw_ctrl.o \
i40iw_hmc.o i40iw_hw.o i40iw_main.o \
i40iw_pble.o i40iw_puda.o i40iw_uk.o i40iw_utils.o \
i40iw_verbs.o i40iw_virtchnl.o i40iw_vf.o

570
drivers/infiniband/hw/i40iw/i40iw.h 100644
View File

@ -0,0 +1,570 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#ifndef I40IW_IW_H
#define I40IW_IW_H
#include <linux/netdevice.h>
#include <linux/inetdevice.h>
#include <linux/spinlock.h>
#include <linux/kernel.h>
#include <linux/delay.h>
#include <linux/pci.h>
#include <linux/dma-mapping.h>
#include <linux/workqueue.h>
#include <linux/slab.h>
#include <linux/io.h>
#include <linux/crc32c.h>
#include <rdma/ib_smi.h>
#include <rdma/ib_verbs.h>
#include <rdma/ib_pack.h>
#include <rdma/rdma_cm.h>
#include <rdma/iw_cm.h>
#include <rdma/iw_portmap.h>
#include <rdma/rdma_netlink.h>
#include <crypto/hash.h>
#include "i40iw_status.h"
#include "i40iw_osdep.h"
#include "i40iw_d.h"
#include "i40iw_hmc.h"
#include <i40e_client.h>
#include "i40iw_type.h"
#include "i40iw_p.h"
#include "i40iw_ucontext.h"
#include "i40iw_pble.h"
#include "i40iw_verbs.h"
#include "i40iw_cm.h"
#include "i40iw_user.h"
#include "i40iw_puda.h"
#define I40IW_FW_VERSION 2
#define I40IW_HW_VERSION 2
#define I40IW_ARP_ADD 1
#define I40IW_ARP_DELETE 2
#define I40IW_ARP_RESOLVE 3
#define I40IW_MACIP_ADD 1
#define I40IW_MACIP_DELETE 2
#define IW_CCQ_SIZE (I40IW_CQP_SW_SQSIZE_2048 + 1)
#define IW_CEQ_SIZE 2048
#define IW_AEQ_SIZE 2048
#define RX_BUF_SIZE (1536 + 8)
#define IW_REG0_SIZE (4 * 1024)
#define IW_TX_TIMEOUT (6 * HZ)
#define IW_FIRST_QPN 1
#define IW_SW_CONTEXT_ALIGN 1024
#define MAX_DPC_ITERATIONS 128
#define I40IW_EVENT_TIMEOUT 100000
#define I40IW_VCHNL_EVENT_TIMEOUT 100000
#define I40IW_NO_VLAN 0xffff
#define I40IW_NO_QSET 0xffff
/* access to mcast filter list */
#define IW_ADD_MCAST false
#define IW_DEL_MCAST true
#define I40IW_DRV_OPT_ENABLE_MPA_VER_0 0x00000001
#define I40IW_DRV_OPT_DISABLE_MPA_CRC 0x00000002
#define I40IW_DRV_OPT_DISABLE_FIRST_WRITE 0x00000004
#define I40IW_DRV_OPT_DISABLE_INTF 0x00000008
#define I40IW_DRV_OPT_ENABLE_MSI 0x00000010
#define I40IW_DRV_OPT_DUAL_LOGICAL_PORT 0x00000020
#define I40IW_DRV_OPT_NO_INLINE_DATA 0x00000080
#define I40IW_DRV_OPT_DISABLE_INT_MOD 0x00000100
#define I40IW_DRV_OPT_DISABLE_VIRT_WQ 0x00000200
#define I40IW_DRV_OPT_ENABLE_PAU 0x00000400
#define I40IW_DRV_OPT_MCAST_LOGPORT_MAP 0x00000800
#define IW_HMC_OBJ_TYPE_NUM ARRAY_SIZE(iw_hmc_obj_types)
#define IW_CFG_FPM_QP_COUNT 32768
#define I40IW_MTU_TO_MSS 40
#define I40IW_DEFAULT_MSS 1460
struct i40iw_cqp_compl_info {
u32 op_ret_val;
u16 maj_err_code;
u16 min_err_code;
bool error;
u8 op_code;
};
#define i40iw_pr_err(fmt, args ...) pr_err("%s: "fmt, __func__, ## args)
#define i40iw_pr_info(fmt, args ...) pr_info("%s: " fmt, __func__, ## args)
#define i40iw_pr_warn(fmt, args ...) pr_warn("%s: " fmt, __func__, ## args)
struct i40iw_cqp_request {
struct cqp_commands_info info;
wait_queue_head_t waitq;
struct list_head list;
atomic_t refcount;
void (*callback_fcn)(struct i40iw_cqp_request*, u32);
void *param;
struct i40iw_cqp_compl_info compl_info;
bool waiting;
bool request_done;
bool dynamic;
};
struct i40iw_cqp {
struct i40iw_sc_cqp sc_cqp;
spinlock_t req_lock; /*cqp request list */
wait_queue_head_t waitq;
struct i40iw_dma_mem sq;
struct i40iw_dma_mem host_ctx;
u64 *scratch_array;
struct i40iw_cqp_request *cqp_requests;
struct list_head cqp_avail_reqs;
struct list_head cqp_pending_reqs;
};
struct i40iw_device;
struct i40iw_ccq {
struct i40iw_sc_cq sc_cq;
spinlock_t lock; /* ccq control */
wait_queue_head_t waitq;
struct i40iw_dma_mem mem_cq;
struct i40iw_dma_mem shadow_area;
};
struct i40iw_ceq {
struct i40iw_sc_ceq sc_ceq;
struct i40iw_dma_mem mem;
u32 irq;
u32 msix_idx;
struct i40iw_device *iwdev;
struct tasklet_struct dpc_tasklet;
};
struct i40iw_aeq {
struct i40iw_sc_aeq sc_aeq;
struct i40iw_dma_mem mem;
};
struct i40iw_arp_entry {
u32 ip_addr[4];
u8 mac_addr[ETH_ALEN];
};
enum init_completion_state {
INVALID_STATE = 0,
INITIAL_STATE,
CQP_CREATED,
HMC_OBJS_CREATED,
PBLE_CHUNK_MEM,
CCQ_CREATED,
AEQ_CREATED,
CEQ_CREATED,
ILQ_CREATED,
IEQ_CREATED,
INET_NOTIFIER,
IP_ADDR_REGISTERED,
RDMA_DEV_REGISTERED
};
struct i40iw_msix_vector {
u32 idx;
u32 irq;
u32 cpu_affinity;
u32 ceq_id;
};
#define I40IW_MSIX_TABLE_SIZE 65
struct virtchnl_work {
struct work_struct work;
union {
struct i40iw_cqp_request *cqp_request;
struct i40iw_virtchnl_work_info work_info;
};
};
struct i40e_qvlist_info;
struct i40iw_device {
struct i40iw_ib_device *iwibdev;
struct net_device *netdev;
wait_queue_head_t vchnl_waitq;
struct i40iw_sc_dev sc_dev;
struct i40iw_handler *hdl;
struct i40e_info *ldev;
struct i40e_client *client;
struct i40iw_hw hw;
struct i40iw_cm_core cm_core;
unsigned long *mem_resources;
unsigned long *allocated_qps;
unsigned long *allocated_cqs;
unsigned long *allocated_mrs;
unsigned long *allocated_pds;
unsigned long *allocated_arps;
struct i40iw_qp **qp_table;
bool msix_shared;
u32 msix_count;
struct i40iw_msix_vector *iw_msixtbl;
struct i40e_qvlist_info *iw_qvlist;
struct i40iw_hmc_pble_rsrc *pble_rsrc;
struct i40iw_arp_entry *arp_table;
struct i40iw_cqp cqp;
struct i40iw_ccq ccq;
u32 ceqs_count;
struct i40iw_ceq *ceqlist;
struct i40iw_aeq aeq;
u32 arp_table_size;
u32 next_arp_index;
spinlock_t resource_lock; /* hw resource access */
u32 vendor_id;
u32 vendor_part_id;
u32 of_device_registered;
u32 device_cap_flags;
unsigned long db_start;
u8 resource_profile;
u8 max_rdma_vfs;
u8 max_enabled_vfs;
u8 max_sge;
u8 iw_status;
u8 send_term_ok;
bool push_mode; /* Initialized from parameter passed to driver */
/* x710 specific */
struct mutex pbl_mutex;
struct tasklet_struct dpc_tasklet;
struct workqueue_struct *virtchnl_wq;
struct virtchnl_work virtchnl_w[I40IW_MAX_PE_ENABLED_VF_COUNT];
struct i40iw_dma_mem obj_mem;
struct i40iw_dma_mem obj_next;
u8 *hmc_info_mem;
u32 sd_type;
struct workqueue_struct *param_wq;
atomic_t params_busy;
u32 mss;
enum init_completion_state init_state;
u16 mac_ip_table_idx;
atomic_t vchnl_msgs;
u32 max_mr;
u32 max_qp;
u32 max_cq;
u32 max_pd;
u32 next_qp;
u32 next_cq;
u32 next_pd;
u32 max_mr_size;
u32 max_qp_wr;
u32 max_cqe;
u32 mr_stagmask;
u32 mpa_version;
bool dcb;
};
struct i40iw_ib_device {
struct ib_device ibdev;
struct i40iw_device *iwdev;
};
struct i40iw_handler {
struct list_head list;
struct i40e_client *client;
struct i40iw_device device;
struct i40e_info ldev;
};
/**
* to_iwdev - get device
* @ibdev: ib device
**/
static inline struct i40iw_device *to_iwdev(struct ib_device *ibdev)
{
return container_of(ibdev, struct i40iw_ib_device, ibdev)->iwdev;
}
/**
* to_ucontext - get user context
* @ibucontext: ib user context
**/
static inline struct i40iw_ucontext *to_ucontext(struct ib_ucontext *ibucontext)
{
return container_of(ibucontext, struct i40iw_ucontext, ibucontext);
}
/**
* to_iwpd - get protection domain
* @ibpd: ib pd
**/
static inline struct i40iw_pd *to_iwpd(struct ib_pd *ibpd)
{
return container_of(ibpd, struct i40iw_pd, ibpd);
}
/**
* to_iwmr - get device memory region
* @ibdev: ib memory region
**/
static inline struct i40iw_mr *to_iwmr(struct ib_mr *ibmr)
{
return container_of(ibmr, struct i40iw_mr, ibmr);
}
/**
* to_iwmr_from_ibfmr - get device memory region
* @ibfmr: ib fmr
**/
static inline struct i40iw_mr *to_iwmr_from_ibfmr(struct ib_fmr *ibfmr)
{
return container_of(ibfmr, struct i40iw_mr, ibfmr);
}
/**
* to_iwmw - get device memory window
* @ibmw: ib memory window
**/
static inline struct i40iw_mr *to_iwmw(struct ib_mw *ibmw)
{
return container_of(ibmw, struct i40iw_mr, ibmw);
}
/**
* to_iwcq - get completion queue
* @ibcq: ib cqdevice
**/
static inline struct i40iw_cq *to_iwcq(struct ib_cq *ibcq)
{
return container_of(ibcq, struct i40iw_cq, ibcq);
}
/**
* to_iwqp - get device qp
* @ibqp: ib qp
**/
static inline struct i40iw_qp *to_iwqp(struct ib_qp *ibqp)
{
return container_of(ibqp, struct i40iw_qp, ibqp);
}
/* i40iw.c */
void i40iw_add_ref(struct ib_qp *);
void i40iw_rem_ref(struct ib_qp *);
struct ib_qp *i40iw_get_qp(struct ib_device *, int);
void i40iw_flush_wqes(struct i40iw_device *iwdev,
struct i40iw_qp *qp);
void i40iw_manage_arp_cache(struct i40iw_device *iwdev,
unsigned char *mac_addr,
__be32 *ip_addr,
bool ipv4,
u32 action);
int i40iw_manage_apbvt(struct i40iw_device *iwdev,
u16 accel_local_port,
bool add_port);
struct i40iw_cqp_request *i40iw_get_cqp_request(struct i40iw_cqp *cqp, bool wait);
void i40iw_free_cqp_request(struct i40iw_cqp *cqp, struct i40iw_cqp_request *cqp_request);
void i40iw_put_cqp_request(struct i40iw_cqp *cqp, struct i40iw_cqp_request *cqp_request);
/**
* i40iw_alloc_resource - allocate a resource
* @iwdev: device pointer
* @resource_array: resource bit array:
* @max_resources: maximum resource number
* @req_resources_num: Allocated resource number
* @next: next free id
**/
static inline int i40iw_alloc_resource(struct i40iw_device *iwdev,
unsigned long *resource_array,
u32 max_resources,
u32 *req_resource_num,
u32 *next)
{
u32 resource_num;
unsigned long flags;
spin_lock_irqsave(&iwdev->resource_lock, flags);
resource_num = find_next_zero_bit(resource_array, max_resources, *next);
if (resource_num >= max_resources) {
resource_num = find_first_zero_bit(resource_array, max_resources);
if (resource_num >= max_resources) {
spin_unlock_irqrestore(&iwdev->resource_lock, flags);
return -EOVERFLOW;
}
}
set_bit(resource_num, resource_array);
*next = resource_num + 1;
if (*next == max_resources)
*next = 0;
spin_unlock_irqrestore(&iwdev->resource_lock, flags);
*req_resource_num = resource_num;
return 0;
}
/**
* i40iw_is_resource_allocated - detrmine if resource is
* allocated
* @iwdev: device pointer
* @resource_array: resource array for the resource_num
* @resource_num: resource number to check
**/
static inline bool i40iw_is_resource_allocated(struct i40iw_device *iwdev,
unsigned long *resource_array,
u32 resource_num)
{
bool bit_is_set;
unsigned long flags;
spin_lock_irqsave(&iwdev->resource_lock, flags);
bit_is_set = test_bit(resource_num, resource_array);
spin_unlock_irqrestore(&iwdev->resource_lock, flags);
return bit_is_set;
}
/**
* i40iw_free_resource - free a resource
* @iwdev: device pointer
* @resource_array: resource array for the resource_num
* @resource_num: resource number to free
**/
static inline void i40iw_free_resource(struct i40iw_device *iwdev,
unsigned long *resource_array,
u32 resource_num)
{
unsigned long flags;
spin_lock_irqsave(&iwdev->resource_lock, flags);
clear_bit(resource_num, resource_array);
spin_unlock_irqrestore(&iwdev->resource_lock, flags);
}
/**
* to_iwhdl - Get the handler from the device pointer
* @iwdev: device pointer
**/
static inline struct i40iw_handler *to_iwhdl(struct i40iw_device *iw_dev)
{
return container_of(iw_dev, struct i40iw_handler, device);
}
struct i40iw_handler *i40iw_find_netdev(struct net_device *netdev);
/**
* iw_init_resources -
*/
u32 i40iw_initialize_hw_resources(struct i40iw_device *iwdev);
int i40iw_register_rdma_device(struct i40iw_device *iwdev);
void i40iw_port_ibevent(struct i40iw_device *iwdev);
int i40iw_cm_disconn(struct i40iw_qp *);
void i40iw_cm_disconn_worker(void *);
int mini_cm_recv_pkt(struct i40iw_cm_core *, struct i40iw_device *,
struct sk_buff *);
enum i40iw_status_code i40iw_handle_cqp_op(struct i40iw_device *iwdev,
struct i40iw_cqp_request *cqp_request);
enum i40iw_status_code i40iw_add_mac_addr(struct i40iw_device *iwdev,
u8 *mac_addr, u8 *mac_index);
int i40iw_modify_qp(struct ib_qp *, struct ib_qp_attr *, int, struct ib_udata *);
void i40iw_rem_pdusecount(struct i40iw_pd *iwpd, struct i40iw_device *iwdev);
void i40iw_add_pdusecount(struct i40iw_pd *iwpd);
void i40iw_hw_modify_qp(struct i40iw_device *iwdev, struct i40iw_qp *iwqp,
struct i40iw_modify_qp_info *info, bool wait);
enum i40iw_status_code i40iw_manage_qhash(struct i40iw_device *iwdev,
struct i40iw_cm_info *cminfo,
enum i40iw_quad_entry_type etype,
enum i40iw_quad_hash_manage_type mtype,
void *cmnode,
bool wait);
void i40iw_receive_ilq(struct i40iw_sc_dev *dev, struct i40iw_puda_buf *rbuf);
void i40iw_free_sqbuf(struct i40iw_sc_dev *dev, void *bufp);
void i40iw_free_qp_resources(struct i40iw_device *iwdev,
struct i40iw_qp *iwqp,
u32 qp_num);
enum i40iw_status_code i40iw_obj_aligned_mem(struct i40iw_device *iwdev,
struct i40iw_dma_mem *memptr,
u32 size, u32 mask);
void i40iw_request_reset(struct i40iw_device *iwdev);
void i40iw_destroy_rdma_device(struct i40iw_ib_device *iwibdev);
void i40iw_setup_cm_core(struct i40iw_device *iwdev);
void i40iw_cleanup_cm_core(struct i40iw_cm_core *cm_core);
void i40iw_process_ceq(struct i40iw_device *, struct i40iw_ceq *iwceq);
void i40iw_process_aeq(struct i40iw_device *);
void i40iw_next_iw_state(struct i40iw_qp *iwqp,
u8 state, u8 del_hash,
u8 term, u8 term_len);
int i40iw_send_syn(struct i40iw_cm_node *cm_node, u32 sendack);
struct i40iw_cm_node *i40iw_find_node(struct i40iw_cm_core *cm_core,
u16 rem_port,
u32 *rem_addr,
u16 loc_port,
u32 *loc_addr,
bool add_refcnt);
enum i40iw_status_code i40iw_hw_flush_wqes(struct i40iw_device *iwdev,
struct i40iw_sc_qp *qp,
struct i40iw_qp_flush_info *info,
bool wait);
void i40iw_copy_ip_ntohl(u32 *dst, u32 *src);
struct ib_mr *i40iw_reg_phys_mr(struct ib_pd *ib_pd,
u64 addr,
u64 size,
int acc,
u64 *iova_start);
int i40iw_inetaddr_event(struct notifier_block *notifier,
unsigned long event,
void *ptr);
int i40iw_inet6addr_event(struct notifier_block *notifier,
unsigned long event,
void *ptr);
int i40iw_net_event(struct notifier_block *notifier,
unsigned long event,
void *ptr);
#endif

4141
drivers/infiniband/hw/i40iw/i40iw_cm.c 100644
View File

File diff suppressed because it is too large Load Diff

456
drivers/infiniband/hw/i40iw/i40iw_cm.h 100644
View File

@ -0,0 +1,456 @@
/*******************************************************************************
*
* Copyright (c) 2015 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#ifndef I40IW_CM_H
#define I40IW_CM_H
#define QUEUE_EVENTS
#define I40IW_MANAGE_APBVT_DEL 0
#define I40IW_MANAGE_APBVT_ADD 1
#define I40IW_MPA_REQUEST_ACCEPT 1
#define I40IW_MPA_REQUEST_REJECT 2
/* IETF MPA -- defines, enums, structs */
#define IEFT_MPA_KEY_REQ "MPA ID Req Frame"
#define IEFT_MPA_KEY_REP "MPA ID Rep Frame"
#define IETF_MPA_KEY_SIZE 16
#define IETF_MPA_VERSION 1
#define IETF_MAX_PRIV_DATA_LEN 512
#define IETF_MPA_FRAME_SIZE 20
#define IETF_RTR_MSG_SIZE 4
#define IETF_MPA_V2_FLAG 0x10
#define SNDMARKER_SEQNMASK 0x000001FF
#define I40IW_MAX_IETF_SIZE 32
#define MPA_ZERO_PAD_LEN 4
/* IETF RTR MSG Fields */
#define IETF_PEER_TO_PEER 0x8000
#define IETF_FLPDU_ZERO_LEN 0x4000
#define IETF_RDMA0_WRITE 0x8000
#define IETF_RDMA0_READ 0x4000
#define IETF_NO_IRD_ORD 0x3FFF
/* HW-supported IRD sizes*/
#define I40IW_HW_IRD_SETTING_2 2
#define I40IW_HW_IRD_SETTING_4 4
#define I40IW_HW_IRD_SETTING_8 8
#define I40IW_HW_IRD_SETTING_16 16
#define I40IW_HW_IRD_SETTING_32 32
#define I40IW_HW_IRD_SETTING_64 64
enum ietf_mpa_flags {
IETF_MPA_FLAGS_MARKERS = 0x80, /* receive Markers */
IETF_MPA_FLAGS_CRC = 0x40, /* receive Markers */
IETF_MPA_FLAGS_REJECT = 0x20, /* Reject */
};
struct ietf_mpa_v1 {
u8 key[IETF_MPA_KEY_SIZE];
u8 flags;
u8 rev;
__be16 priv_data_len;
u8 priv_data[0];
};
#define ietf_mpa_req_resp_frame ietf_mpa_frame
struct ietf_rtr_msg {
__be16 ctrl_ird;
__be16 ctrl_ord;
};
struct ietf_mpa_v2 {
u8 key[IETF_MPA_KEY_SIZE];
u8 flags;
u8 rev;
__be16 priv_data_len;
struct ietf_rtr_msg rtr_msg;
u8 priv_data[0];
};
struct i40iw_cm_node;
enum i40iw_timer_type {
I40IW_TIMER_TYPE_SEND,
I40IW_TIMER_TYPE_RECV,
I40IW_TIMER_NODE_CLEANUP,
I40IW_TIMER_TYPE_CLOSE,
};
#define I40IW_PASSIVE_STATE_INDICATED 0
#define I40IW_DO_NOT_SEND_RESET_EVENT 1
#define I40IW_SEND_RESET_EVENT 2
#define MAX_I40IW_IFS 4
#define SET_ACK 0x1
#define SET_SYN 0x2
#define SET_FIN 0x4
#define SET_RST 0x8
#define TCP_OPTIONS_PADDING 3
struct option_base {
u8 optionnum;
u8 length;
};
enum option_numbers {
OPTION_NUMBER_END,
OPTION_NUMBER_NONE,
OPTION_NUMBER_MSS,
OPTION_NUMBER_WINDOW_SCALE,
OPTION_NUMBER_SACK_PERM,
OPTION_NUMBER_SACK,
OPTION_NUMBER_WRITE0 = 0xbc
};
struct option_mss {
u8 optionnum;
u8 length;
__be16 mss;
};
struct option_windowscale {
u8 optionnum;
u8 length;
u8 shiftcount;
};
union all_known_options {
char as_end;
struct option_base as_base;
struct option_mss as_mss;
struct option_windowscale as_windowscale;
};
struct i40iw_timer_entry {
struct list_head list;
unsigned long timetosend; /* jiffies */
struct i40iw_puda_buf *sqbuf;
u32 type;
u32 retrycount;
u32 retranscount;
u32 context;
u32 send_retrans;
int close_when_complete;
};
#define I40IW_DEFAULT_RETRYS 64
#define I40IW_DEFAULT_RETRANS 8
#define I40IW_DEFAULT_TTL 0x40
#define I40IW_DEFAULT_RTT_VAR 0x6
#define I40IW_DEFAULT_SS_THRESH 0x3FFFFFFF
#define I40IW_DEFAULT_REXMIT_THRESH 8
#define I40IW_RETRY_TIMEOUT HZ
#define I40IW_SHORT_TIME 10
#define I40IW_LONG_TIME (2 * HZ)
#define I40IW_MAX_TIMEOUT ((unsigned long)(12 * HZ))
#define I40IW_CM_HASHTABLE_SIZE 1024
#define I40IW_CM_TCP_TIMER_INTERVAL 3000
#define I40IW_CM_DEFAULT_MTU 1540
#define I40IW_CM_DEFAULT_FRAME_CNT 10
#define I40IW_CM_THREAD_STACK_SIZE 256
#define I40IW_CM_DEFAULT_RCV_WND 64240
#define I40IW_CM_DEFAULT_RCV_WND_SCALED 0x3fffc
#define I40IW_CM_DEFAULT_RCV_WND_SCALE 2
#define I40IW_CM_DEFAULT_FREE_PKTS 0x000A
#define I40IW_CM_FREE_PKT_LO_WATERMARK 2
#define I40IW_CM_DEFAULT_MSS 536
#define I40IW_CM_DEF_SEQ 0x159bf75f
#define I40IW_CM_DEF_LOCAL_ID 0x3b47
#define I40IW_CM_DEF_SEQ2 0x18ed5740
#define I40IW_CM_DEF_LOCAL_ID2 0xb807
#define MAX_CM_BUFFER (I40IW_MAX_IETF_SIZE + IETF_MAX_PRIV_DATA_LEN)
typedef u32 i40iw_addr_t;
#define i40iw_cm_tsa_context i40iw_qp_context
struct i40iw_qp;
/* cm node transition states */
enum i40iw_cm_node_state {
I40IW_CM_STATE_UNKNOWN,
I40IW_CM_STATE_INITED,
I40IW_CM_STATE_LISTENING,
I40IW_CM_STATE_SYN_RCVD,
I40IW_CM_STATE_SYN_SENT,
I40IW_CM_STATE_ONE_SIDE_ESTABLISHED,
I40IW_CM_STATE_ESTABLISHED,
I40IW_CM_STATE_ACCEPTING,
I40IW_CM_STATE_MPAREQ_SENT,
I40IW_CM_STATE_MPAREQ_RCVD,
I40IW_CM_STATE_MPAREJ_RCVD,
I40IW_CM_STATE_OFFLOADED,
I40IW_CM_STATE_FIN_WAIT1,
I40IW_CM_STATE_FIN_WAIT2,
I40IW_CM_STATE_CLOSE_WAIT,
I40IW_CM_STATE_TIME_WAIT,
I40IW_CM_STATE_LAST_ACK,
I40IW_CM_STATE_CLOSING,
I40IW_CM_STATE_LISTENER_DESTROYED,
I40IW_CM_STATE_CLOSED
};
enum mpa_frame_version {
IETF_MPA_V1 = 1,
IETF_MPA_V2 = 2
};
enum mpa_frame_key {
MPA_KEY_REQUEST,
MPA_KEY_REPLY
};
enum send_rdma0 {
SEND_RDMA_READ_ZERO = 1,
SEND_RDMA_WRITE_ZERO = 2
};
enum i40iw_tcpip_pkt_type {
I40IW_PKT_TYPE_UNKNOWN,
I40IW_PKT_TYPE_SYN,
I40IW_PKT_TYPE_SYNACK,
I40IW_PKT_TYPE_ACK,
I40IW_PKT_TYPE_FIN,
I40IW_PKT_TYPE_RST
};
/* CM context params */
struct i40iw_cm_tcp_context {
u8 client;
u32 loc_seq_num;
u32 loc_ack_num;
u32 rem_ack_num;
u32 rcv_nxt;
u32 loc_id;
u32 rem_id;
u32 snd_wnd;
u32 max_snd_wnd;
u32 rcv_wnd;
u32 mss;
u8 snd_wscale;
u8 rcv_wscale;
struct timeval sent_ts;
};
enum i40iw_cm_listener_state {
I40IW_CM_LISTENER_PASSIVE_STATE = 1,
I40IW_CM_LISTENER_ACTIVE_STATE = 2,
I40IW_CM_LISTENER_EITHER_STATE = 3
};
struct i40iw_cm_listener {
struct list_head list;
struct i40iw_cm_core *cm_core;
u8 loc_mac[ETH_ALEN];
u32 loc_addr[4];
u16 loc_port;
u32 map_loc_addr[4];
u16 map_loc_port;
struct iw_cm_id *cm_id;
atomic_t ref_count;
struct i40iw_device *iwdev;
atomic_t pend_accepts_cnt;
int backlog;
enum i40iw_cm_listener_state listener_state;
u32 reused_node;
u8 user_pri;
u16 vlan_id;
bool qhash_set;
bool ipv4;
struct list_head child_listen_list;
};
struct i40iw_kmem_info {
void *addr;
u32 size;
};
/* per connection node and node state information */
struct i40iw_cm_node {
u32 loc_addr[4], rem_addr[4];
u16 loc_port, rem_port;
u32 map_loc_addr[4], map_rem_addr[4];
u16 map_loc_port, map_rem_port;
u16 vlan_id;
enum i40iw_cm_node_state state;
u8 loc_mac[ETH_ALEN];
u8 rem_mac[ETH_ALEN];
atomic_t ref_count;
struct i40iw_qp *iwqp;
struct i40iw_device *iwdev;
struct i40iw_sc_dev *dev;
struct i40iw_cm_tcp_context tcp_cntxt;
struct i40iw_cm_core *cm_core;
struct i40iw_cm_node *loopbackpartner;
struct i40iw_timer_entry *send_entry;
struct i40iw_timer_entry *close_entry;
spinlock_t retrans_list_lock; /* cm transmit packet */
enum send_rdma0 send_rdma0_op;
u16 ird_size;
u16 ord_size;
u16 mpav2_ird_ord;
struct iw_cm_id *cm_id;
struct list_head list;
int accelerated;
struct i40iw_cm_listener *listener;
int apbvt_set;
int accept_pend;
struct list_head timer_entry;
struct list_head reset_entry;
atomic_t passive_state;
bool qhash_set;
u8 user_pri;
bool ipv4;
bool snd_mark_en;
u16 lsmm_size;
enum mpa_frame_version mpa_frame_rev;
struct i40iw_kmem_info pdata;
union {
struct ietf_mpa_v1 mpa_frame;
struct ietf_mpa_v2 mpa_v2_frame;
};
u8 pdata_buf[IETF_MAX_PRIV_DATA_LEN];
struct i40iw_kmem_info mpa_hdr;
};
/* structure for client or CM to fill when making CM api calls. */
/* - only need to set relevant data, based on op. */
struct i40iw_cm_info {
struct iw_cm_id *cm_id;
u16 loc_port;
u16 rem_port;
u32 loc_addr[4];
u32 rem_addr[4];
u16 map_loc_port;
u16 map_rem_port;
u32 map_loc_addr[4];
u32 map_rem_addr[4];
u16 vlan_id;
int backlog;
u16 user_pri;
bool ipv4;
};
/* CM event codes */
enum i40iw_cm_event_type {
I40IW_CM_EVENT_UNKNOWN,
I40IW_CM_EVENT_ESTABLISHED,
I40IW_CM_EVENT_MPA_REQ,
I40IW_CM_EVENT_MPA_CONNECT,
I40IW_CM_EVENT_MPA_ACCEPT,
I40IW_CM_EVENT_MPA_REJECT,
I40IW_CM_EVENT_MPA_ESTABLISHED,
I40IW_CM_EVENT_CONNECTED,
I40IW_CM_EVENT_RESET,
I40IW_CM_EVENT_ABORTED
};
/* event to post to CM event handler */
struct i40iw_cm_event {
enum i40iw_cm_event_type type;
struct i40iw_cm_info cm_info;
struct work_struct event_work;
struct i40iw_cm_node *cm_node;
};
struct i40iw_cm_core {
struct i40iw_device *iwdev;
struct i40iw_sc_dev *dev;
struct list_head listen_nodes;
struct list_head connected_nodes;
struct timer_list tcp_timer;
struct workqueue_struct *event_wq;
struct workqueue_struct *disconn_wq;
spinlock_t ht_lock; /* manage hash table */
spinlock_t listen_list_lock; /* listen list */
u64 stats_nodes_created;
u64 stats_nodes_destroyed;
u64 stats_listen_created;
u64 stats_listen_destroyed;
u64 stats_listen_nodes_created;
u64 stats_listen_nodes_destroyed;
u64 stats_loopbacks;
u64 stats_accepts;
u64 stats_rejects;
u64 stats_connect_errs;
u64 stats_passive_errs;
u64 stats_pkt_retrans;
u64 stats_backlog_drops;
};
int i40iw_schedule_cm_timer(struct i40iw_cm_node *cm_node,
struct i40iw_puda_buf *sqbuf,
enum i40iw_timer_type type,
int send_retrans,
int close_when_complete);
int i40iw_accept(struct iw_cm_id *, struct iw_cm_conn_param *);
int i40iw_reject(struct iw_cm_id *, const void *, u8);
int i40iw_connect(struct iw_cm_id *, struct iw_cm_conn_param *);
int i40iw_create_listen(struct iw_cm_id *, int);
int i40iw_destroy_listen(struct iw_cm_id *);
int i40iw_cm_start(struct i40iw_device *);
int i40iw_cm_stop(struct i40iw_device *);
int i40iw_arp_table(struct i40iw_device *iwdev,
u32 *ip_addr,
bool ipv4,
u8 *mac_addr,
u32 action);
#endif /* I40IW_CM_H */

4743
drivers/infiniband/hw/i40iw/i40iw_ctrl.c 100644
View File

File diff suppressed because it is too large Load Diff

1713
drivers/infiniband/hw/i40iw/i40iw_d.h 100644
View File

File diff suppressed because it is too large Load Diff

821
drivers/infiniband/hw/i40iw/i40iw_hmc.c 100644
View File

@ -0,0 +1,821 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#include "i40iw_osdep.h"
#include "i40iw_register.h"
#include "i40iw_status.h"
#include "i40iw_hmc.h"
#include "i40iw_d.h"
#include "i40iw_type.h"
#include "i40iw_p.h"
#include "i40iw_vf.h"
#include "i40iw_virtchnl.h"
/**
* i40iw_find_sd_index_limit - finds segment descriptor index limit
* @hmc_info: pointer to the HMC configuration information structure
* @type: type of HMC resources we're searching
* @index: starting index for the object
* @cnt: number of objects we're trying to create
* @sd_idx: pointer to return index of the segment descriptor in question
* @sd_limit: pointer to return the maximum number of segment descriptors
*
* This function calculates the segment descriptor index and index limit
* for the resource defined by i40iw_hmc_rsrc_type.
*/
static inline void i40iw_find_sd_index_limit(struct i40iw_hmc_info *hmc_info,
u32 type,
u32 idx,
u32 cnt,
u32 *sd_idx,
u32 *sd_limit)
{
u64 fpm_addr, fpm_limit;
fpm_addr = hmc_info->hmc_obj[(type)].base +
hmc_info->hmc_obj[type].size * idx;
fpm_limit = fpm_addr + hmc_info->hmc_obj[type].size * cnt;
*sd_idx = (u32)(fpm_addr / I40IW_HMC_DIRECT_BP_SIZE);
*sd_limit = (u32)((fpm_limit - 1) / I40IW_HMC_DIRECT_BP_SIZE);
*sd_limit += 1;
}
/**
* i40iw_find_pd_index_limit - finds page descriptor index limit
* @hmc_info: pointer to the HMC configuration information struct
* @type: HMC resource type we're examining
* @idx: starting index for the object
* @cnt: number of objects we're trying to create
* @pd_index: pointer to return page descriptor index
* @pd_limit: pointer to return page descriptor index limit
*
* Calculates the page descriptor index and index limit for the resource
* defined by i40iw_hmc_rsrc_type.
*/
static inline void i40iw_find_pd_index_limit(struct i40iw_hmc_info *hmc_info,
u32 type,
u32 idx,
u32 cnt,
u32 *pd_idx,
u32 *pd_limit)
{
u64 fpm_adr, fpm_limit;
fpm_adr = hmc_info->hmc_obj[type].base +
hmc_info->hmc_obj[type].size * idx;
fpm_limit = fpm_adr + (hmc_info)->hmc_obj[(type)].size * (cnt);
*(pd_idx) = (u32)(fpm_adr / I40IW_HMC_PAGED_BP_SIZE);
*(pd_limit) = (u32)((fpm_limit - 1) / I40IW_HMC_PAGED_BP_SIZE);
*(pd_limit) += 1;
}
/**
* i40iw_set_sd_entry - setup entry for sd programming
* @pa: physical addr
* @idx: sd index
* @type: paged or direct sd
* @entry: sd entry ptr
*/
static inline void i40iw_set_sd_entry(u64 pa,
u32 idx,
enum i40iw_sd_entry_type type,
struct update_sd_entry *entry)
{
entry->data = pa | (I40IW_HMC_MAX_BP_COUNT << I40E_PFHMC_SDDATALOW_PMSDBPCOUNT_SHIFT) |
(((type == I40IW_SD_TYPE_PAGED) ? 0 : 1) <<
I40E_PFHMC_SDDATALOW_PMSDTYPE_SHIFT) |
(1 << I40E_PFHMC_SDDATALOW_PMSDVALID_SHIFT);
entry->cmd = (idx | (1 << I40E_PFHMC_SDCMD_PMSDWR_SHIFT) | (1 << 15));
}
/**
* i40iw_clr_sd_entry - setup entry for sd clear
* @idx: sd index
* @type: paged or direct sd
* @entry: sd entry ptr
*/
static inline void i40iw_clr_sd_entry(u32 idx, enum i40iw_sd_entry_type type,
struct update_sd_entry *entry)
{
entry->data = (I40IW_HMC_MAX_BP_COUNT <<
I40E_PFHMC_SDDATALOW_PMSDBPCOUNT_SHIFT) |
(((type == I40IW_SD_TYPE_PAGED) ? 0 : 1) <<
I40E_PFHMC_SDDATALOW_PMSDTYPE_SHIFT);
entry->cmd = (idx | (1 << I40E_PFHMC_SDCMD_PMSDWR_SHIFT) | (1 << 15));
}
/**
* i40iw_hmc_sd_one - setup 1 sd entry for cqp
* @dev: pointer to the device structure
* @hmc_fn_id: hmc's function id
* @pa: physical addr
* @sd_idx: sd index
* @type: paged or direct sd
* @setsd: flag to set or clear sd
*/
enum i40iw_status_code i40iw_hmc_sd_one(struct i40iw_sc_dev *dev,
u8 hmc_fn_id,
u64 pa, u32 sd_idx,
enum i40iw_sd_entry_type type,
bool setsd)
{
struct i40iw_update_sds_info sdinfo;
sdinfo.cnt = 1;
sdinfo.hmc_fn_id = hmc_fn_id;
if (setsd)
i40iw_set_sd_entry(pa, sd_idx, type, sdinfo.entry);
else
i40iw_clr_sd_entry(sd_idx, type, sdinfo.entry);
return dev->cqp->process_cqp_sds(dev, &sdinfo);
}
/**
* i40iw_hmc_sd_grp - setup group od sd entries for cqp
* @dev: pointer to the device structure
* @hmc_info: pointer to the HMC configuration information struct
* @sd_index: sd index
* @sd_cnt: number of sd entries
* @setsd: flag to set or clear sd
*/
static enum i40iw_status_code i40iw_hmc_sd_grp(struct i40iw_sc_dev *dev,
struct i40iw_hmc_info *hmc_info,
u32 sd_index,
u32 sd_cnt,
bool setsd)
{
struct i40iw_hmc_sd_entry *sd_entry;
struct i40iw_update_sds_info sdinfo;
u64 pa;
u32 i;
enum i40iw_status_code ret_code = 0;
memset(&sdinfo, 0, sizeof(sdinfo));
sdinfo.hmc_fn_id = hmc_info->hmc_fn_id;
for (i = sd_index; i < sd_index + sd_cnt; i++) {
sd_entry = &hmc_info->sd_table.sd_entry[i];
if (!sd_entry ||
(!sd_entry->valid && setsd) ||
(sd_entry->valid && !setsd))
continue;
if (setsd) {
pa = (sd_entry->entry_type == I40IW_SD_TYPE_PAGED) ?
sd_entry->u.pd_table.pd_page_addr.pa :
sd_entry->u.bp.addr.pa;
i40iw_set_sd_entry(pa, i, sd_entry->entry_type,
&sdinfo.entry[sdinfo.cnt]);
} else {
i40iw_clr_sd_entry(i, sd_entry->entry_type,
&sdinfo.entry[sdinfo.cnt]);
}
sdinfo.cnt++;
if (sdinfo.cnt == I40IW_MAX_SD_ENTRIES) {
ret_code = dev->cqp->process_cqp_sds(dev, &sdinfo);
if (ret_code) {
i40iw_debug(dev, I40IW_DEBUG_HMC,
"i40iw_hmc_sd_grp: sd_programming failed err=%d\n",
ret_code);
return ret_code;
}
sdinfo.cnt = 0;
}
}
if (sdinfo.cnt)
ret_code = dev->cqp->process_cqp_sds(dev, &sdinfo);
return ret_code;
}
/**
* i40iw_vfdev_from_fpm - return vf dev ptr for hmc function id
* @dev: pointer to the device structure
* @hmc_fn_id: hmc's function id
*/
struct i40iw_vfdev *i40iw_vfdev_from_fpm(struct i40iw_sc_dev *dev, u8 hmc_fn_id)
{
struct i40iw_vfdev *vf_dev = NULL;
u16 idx;
for (idx = 0; idx < I40IW_MAX_PE_ENABLED_VF_COUNT; idx++) {
if (dev->vf_dev[idx] &&
((u8)dev->vf_dev[idx]->pmf_index == hmc_fn_id)) {
vf_dev = dev->vf_dev[idx];
break;
}
}
return vf_dev;
}
/**
* i40iw_vf_hmcinfo_from_fpm - get ptr to hmc for func_id
* @dev: pointer to the device structure
* @hmc_fn_id: hmc's function id
*/
struct i40iw_hmc_info *i40iw_vf_hmcinfo_from_fpm(struct i40iw_sc_dev *dev,
u8 hmc_fn_id)
{
struct i40iw_hmc_info *hmc_info = NULL;
u16 idx;
for (idx = 0; idx < I40IW_MAX_PE_ENABLED_VF_COUNT; idx++) {
if (dev->vf_dev[idx] &&
((u8)dev->vf_dev[idx]->pmf_index == hmc_fn_id)) {
hmc_info = &dev->vf_dev[idx]->hmc_info;
break;
}
}
return hmc_info;
}
/**
* i40iw_hmc_finish_add_sd_reg - program sd entries for objects
* @dev: pointer to the device structure
* @info: create obj info
*/
static enum i40iw_status_code i40iw_hmc_finish_add_sd_reg(struct i40iw_sc_dev *dev,
struct i40iw_hmc_create_obj_info *info)
{
if (info->start_idx >= info->hmc_info->hmc_obj[info->rsrc_type].cnt)
return I40IW_ERR_INVALID_HMC_OBJ_INDEX;
if ((info->start_idx + info->count) >
info->hmc_info->hmc_obj[info->rsrc_type].cnt)
return I40IW_ERR_INVALID_HMC_OBJ_COUNT;
if (!info->add_sd_cnt)
return 0;
return i40iw_hmc_sd_grp(dev, info->hmc_info,
info->hmc_info->sd_indexes[0],
info->add_sd_cnt, true);
}
/**
* i40iw_create_iw_hmc_obj - allocate backing store for hmc objects
* @dev: pointer to the device structure
* @info: pointer to i40iw_hmc_iw_create_obj_info struct
*
* This will allocate memory for PDs and backing pages and populate
* the sd and pd entries.
*/
enum i40iw_status_code i40iw_sc_create_hmc_obj(struct i40iw_sc_dev *dev,
struct i40iw_hmc_create_obj_info *info)
{
struct i40iw_hmc_sd_entry *sd_entry;
u32 sd_idx, sd_lmt;
u32 pd_idx = 0, pd_lmt = 0;
u32 pd_idx1 = 0, pd_lmt1 = 0;
u32 i, j;
bool pd_error = false;
enum i40iw_status_code ret_code = 0;
if (info->start_idx >= info->hmc_info->hmc_obj[info->rsrc_type].cnt)
return I40IW_ERR_INVALID_HMC_OBJ_INDEX;
if ((info->start_idx + info->count) >
info->hmc_info->hmc_obj[info->rsrc_type].cnt) {
i40iw_debug(dev, I40IW_DEBUG_HMC,
"%s: error type %u, start = %u, req cnt %u, cnt = %u\n",
__func__, info->rsrc_type, info->start_idx, info->count,
info->hmc_info->hmc_obj[info->rsrc_type].cnt);
return I40IW_ERR_INVALID_HMC_OBJ_COUNT;
}
if (!dev->is_pf)
return i40iw_vchnl_vf_add_hmc_objs(dev, info->rsrc_type, 0, info->count);
i40iw_find_sd_index_limit(info->hmc_info, info->rsrc_type,
info->start_idx, info->count,
&sd_idx, &sd_lmt);
if (sd_idx >= info->hmc_info->sd_table.sd_cnt ||
sd_lmt > info->hmc_info->sd_table.sd_cnt) {
return I40IW_ERR_INVALID_SD_INDEX;
}
i40iw_find_pd_index_limit(info->hmc_info, info->rsrc_type,
info->start_idx, info->count, &pd_idx, &pd_lmt);
for (j = sd_idx; j < sd_lmt; j++) {
ret_code = i40iw_add_sd_table_entry(dev->hw, info->hmc_info,
j,
info->entry_type,
I40IW_HMC_DIRECT_BP_SIZE);
if (ret_code)
goto exit_sd_error;
sd_entry = &info->hmc_info->sd_table.sd_entry[j];
if ((sd_entry->entry_type == I40IW_SD_TYPE_PAGED) &&
((dev->hmc_info == info->hmc_info) &&
(info->rsrc_type != I40IW_HMC_IW_PBLE))) {
pd_idx1 = max(pd_idx, (j * I40IW_HMC_MAX_BP_COUNT));
pd_lmt1 = min(pd_lmt,
(j + 1) * I40IW_HMC_MAX_BP_COUNT);
for (i = pd_idx1; i < pd_lmt1; i++) {
/* update the pd table entry */
ret_code = i40iw_add_pd_table_entry(dev->hw, info->hmc_info,
i, NULL);
if (ret_code) {
pd_error = true;
break;
}
}
if (pd_error) {
while (i && (i > pd_idx1)) {
i40iw_remove_pd_bp(dev->hw, info->hmc_info, (i - 1),
info->is_pf);
i--;
}
}
}
if (sd_entry->valid)
continue;
info->hmc_info->sd_indexes[info->add_sd_cnt] = (u16)j;
info->add_sd_cnt++;
sd_entry->valid = true;
}
return i40iw_hmc_finish_add_sd_reg(dev, info);
exit_sd_error:
while (j && (j > sd_idx)) {
sd_entry = &info->hmc_info->sd_table.sd_entry[j - 1];
switch (sd_entry->entry_type) {
case I40IW_SD_TYPE_PAGED:
pd_idx1 = max(pd_idx,
(j - 1) * I40IW_HMC_MAX_BP_COUNT);
pd_lmt1 = min(pd_lmt, (j * I40IW_HMC_MAX_BP_COUNT));
for (i = pd_idx1; i < pd_lmt1; i++)
i40iw_prep_remove_pd_page(info->hmc_info, i);
break;
case I40IW_SD_TYPE_DIRECT:
i40iw_prep_remove_pd_page(info->hmc_info, (j - 1));
break;
default:
ret_code = I40IW_ERR_INVALID_SD_TYPE;
break;
}
j--;
}
return ret_code;
}
/**
* i40iw_finish_del_sd_reg - delete sd entries for objects
* @dev: pointer to the device structure
* @info: dele obj info
* @reset: true if called before reset
*/
static enum i40iw_status_code i40iw_finish_del_sd_reg(struct i40iw_sc_dev *dev,
struct i40iw_hmc_del_obj_info *info,
bool reset)
{
struct i40iw_hmc_sd_entry *sd_entry;
enum i40iw_status_code ret_code = 0;
u32 i, sd_idx;
struct i40iw_dma_mem *mem;
if (dev->is_pf && !reset)
ret_code = i40iw_hmc_sd_grp(dev, info->hmc_info,
info->hmc_info->sd_indexes[0],
info->del_sd_cnt, false);
if (ret_code)
i40iw_debug(dev, I40IW_DEBUG_HMC, "%s: error cqp sd sd_grp\n", __func__);
for (i = 0; i < info->del_sd_cnt; i++) {
sd_idx = info->hmc_info->sd_indexes[i];
sd_entry = &info->hmc_info->sd_table.sd_entry[sd_idx];
if (!sd_entry)
continue;
mem = (sd_entry->entry_type == I40IW_SD_TYPE_PAGED) ?
&sd_entry->u.pd_table.pd_page_addr :
&sd_entry->u.bp.addr;
if (!mem || !mem->va)
i40iw_debug(dev, I40IW_DEBUG_HMC, "%s: error cqp sd mem\n", __func__);
else
i40iw_free_dma_mem(dev->hw, mem);
}
return ret_code;
}
/**
* i40iw_del_iw_hmc_obj - remove pe hmc objects
* @dev: pointer to the device structure
* @info: pointer to i40iw_hmc_del_obj_info struct
* @reset: true if called before reset
*
* This will de-populate the SDs and PDs. It frees
* the memory for PDS and backing storage. After this function is returned,
* caller should deallocate memory allocated previously for
* book-keeping information about PDs and backing storage.
*/
enum i40iw_status_code i40iw_sc_del_hmc_obj(struct i40iw_sc_dev *dev,
struct i40iw_hmc_del_obj_info *info,
bool reset)
{
struct i40iw_hmc_pd_table *pd_table;
u32 sd_idx, sd_lmt;
u32 pd_idx, pd_lmt, rel_pd_idx;
u32 i, j;
enum i40iw_status_code ret_code = 0;
if (info->start_idx >= info->hmc_info->hmc_obj[info->rsrc_type].cnt) {
i40iw_debug(dev, I40IW_DEBUG_HMC,
"%s: error start_idx[%04d] >= [type %04d].cnt[%04d]\n",
__func__, info->start_idx, info->rsrc_type,
info->hmc_info->hmc_obj[info->rsrc_type].cnt);
return I40IW_ERR_INVALID_HMC_OBJ_INDEX;
}
if ((info->start_idx + info->count) >
info->hmc_info->hmc_obj[info->rsrc_type].cnt) {
i40iw_debug(dev, I40IW_DEBUG_HMC,
"%s: error start_idx[%04d] + count %04d >= [type %04d].cnt[%04d]\n",
__func__, info->start_idx, info->count,
info->rsrc_type,
info->hmc_info->hmc_obj[info->rsrc_type].cnt);
return I40IW_ERR_INVALID_HMC_OBJ_COUNT;
}
if (!dev->is_pf) {
ret_code = i40iw_vchnl_vf_del_hmc_obj(dev, info->rsrc_type, 0,
info->count);
if (info->rsrc_type != I40IW_HMC_IW_PBLE)
return ret_code;
}
i40iw_find_pd_index_limit(info->hmc_info, info->rsrc_type,
info->start_idx, info->count, &pd_idx, &pd_lmt);
for (j = pd_idx; j < pd_lmt; j++) {
sd_idx = j / I40IW_HMC_PD_CNT_IN_SD;
if (info->hmc_info->sd_table.sd_entry[sd_idx].entry_type !=
I40IW_SD_TYPE_PAGED)
continue;
rel_pd_idx = j % I40IW_HMC_PD_CNT_IN_SD;
pd_table = &info->hmc_info->sd_table.sd_entry[sd_idx].u.pd_table;
if (pd_table->pd_entry[rel_pd_idx].valid) {
ret_code = i40iw_remove_pd_bp(dev->hw, info->hmc_info, j,
info->is_pf);
if (ret_code) {
i40iw_debug(dev, I40IW_DEBUG_HMC, "%s: error\n", __func__);
return ret_code;
}
}
}
i40iw_find_sd_index_limit(info->hmc_info, info->rsrc_type,
info->start_idx, info->count, &sd_idx, &sd_lmt);
if (sd_idx >= info->hmc_info->sd_table.sd_cnt ||
sd_lmt > info->hmc_info->sd_table.sd_cnt) {
i40iw_debug(dev, I40IW_DEBUG_HMC, "%s: error invalid sd_idx\n", __func__);
return I40IW_ERR_INVALID_SD_INDEX;
}
for (i = sd_idx; i < sd_lmt; i++) {
if (!info->hmc_info->sd_table.sd_entry[i].valid)
continue;
switch (info->hmc_info->sd_table.sd_entry[i].entry_type) {
case I40IW_SD_TYPE_DIRECT:
ret_code = i40iw_prep_remove_sd_bp(info->hmc_info, i);
if (!ret_code) {
info->hmc_info->sd_indexes[info->del_sd_cnt] = (u16)i;
info->del_sd_cnt++;
}
break;
case I40IW_SD_TYPE_PAGED:
ret_code = i40iw_prep_remove_pd_page(info->hmc_info, i);
if (!ret_code) {
info->hmc_info->sd_indexes[info->del_sd_cnt] = (u16)i;
info->del_sd_cnt++;
}
break;
default:
break;
}
}
return i40iw_finish_del_sd_reg(dev, info, reset);
}
/**
* i40iw_add_sd_table_entry - Adds a segment descriptor to the table
* @hw: pointer to our hw struct
* @hmc_info: pointer to the HMC configuration information struct
* @sd_index: segment descriptor index to manipulate
* @type: what type of segment descriptor we're manipulating
* @direct_mode_sz: size to alloc in direct mode
*/
enum i40iw_status_code i40iw_add_sd_table_entry(struct i40iw_hw *hw,
struct i40iw_hmc_info *hmc_info,
u32 sd_index,
enum i40iw_sd_entry_type type,
u64 direct_mode_sz)
{
enum i40iw_status_code ret_code = 0;
struct i40iw_hmc_sd_entry *sd_entry;
bool dma_mem_alloc_done = false;
struct i40iw_dma_mem mem;
u64 alloc_len;
sd_entry = &hmc_info->sd_table.sd_entry[sd_index];
if (!sd_entry->valid) {
if (type == I40IW_SD_TYPE_PAGED)
alloc_len = I40IW_HMC_PAGED_BP_SIZE;
else
alloc_len = direct_mode_sz;
/* allocate a 4K pd page or 2M backing page */
ret_code = i40iw_allocate_dma_mem(hw, &mem, alloc_len,
I40IW_HMC_PD_BP_BUF_ALIGNMENT);
if (ret_code)
goto exit;
dma_mem_alloc_done = true;
if (type == I40IW_SD_TYPE_PAGED) {
ret_code = i40iw_allocate_virt_mem(hw,
&sd_entry->u.pd_table.pd_entry_virt_mem,
sizeof(struct i40iw_hmc_pd_entry) * 512);
if (ret_code)
goto exit;
sd_entry->u.pd_table.pd_entry = (struct i40iw_hmc_pd_entry *)
sd_entry->u.pd_table.pd_entry_virt_mem.va;
memcpy(&sd_entry->u.pd_table.pd_page_addr, &mem, sizeof(struct i40iw_dma_mem));
} else {
memcpy(&sd_entry->u.bp.addr, &mem, sizeof(struct i40iw_dma_mem));
sd_entry->u.bp.sd_pd_index = sd_index;
}
hmc_info->sd_table.sd_entry[sd_index].entry_type = type;
I40IW_INC_SD_REFCNT(&hmc_info->sd_table);
}
if (sd_entry->entry_type == I40IW_SD_TYPE_DIRECT)
I40IW_INC_BP_REFCNT(&sd_entry->u.bp);
exit:
if (ret_code)
if (dma_mem_alloc_done)
i40iw_free_dma_mem(hw, &mem);
return ret_code;
}
/**
* i40iw_add_pd_table_entry - Adds page descriptor to the specified table
* @hw: pointer to our HW structure
* @hmc_info: pointer to the HMC configuration information structure
* @pd_index: which page descriptor index to manipulate
* @rsrc_pg: if not NULL, use preallocated page instead of allocating new one.
*
* This function:
* 1. Initializes the pd entry
* 2. Adds pd_entry in the pd_table
* 3. Mark the entry valid in i40iw_hmc_pd_entry structure
* 4. Initializes the pd_entry's ref count to 1
* assumptions:
* 1. The memory for pd should be pinned down, physically contiguous and
* aligned on 4K boundary and zeroed memory.
* 2. It should be 4K in size.
*/
enum i40iw_status_code i40iw_add_pd_table_entry(struct i40iw_hw *hw,
struct i40iw_hmc_info *hmc_info,
u32 pd_index,
struct i40iw_dma_mem *rsrc_pg)
{
enum i40iw_status_code ret_code = 0;
struct i40iw_hmc_pd_table *pd_table;
struct i40iw_hmc_pd_entry *pd_entry;
struct i40iw_dma_mem mem;
struct i40iw_dma_mem *page = &mem;
u32 sd_idx, rel_pd_idx;
u64 *pd_addr;
u64 page_desc;
if (pd_index / I40IW_HMC_PD_CNT_IN_SD >= hmc_info->sd_table.sd_cnt)
return I40IW_ERR_INVALID_PAGE_DESC_INDEX;
sd_idx = (pd_index / I40IW_HMC_PD_CNT_IN_SD);
if (hmc_info->sd_table.sd_entry[sd_idx].entry_type != I40IW_SD_TYPE_PAGED)
return 0;
rel_pd_idx = (pd_index % I40IW_HMC_PD_CNT_IN_SD);
pd_table = &hmc_info->sd_table.sd_entry[sd_idx].u.pd_table;
pd_entry = &pd_table->pd_entry[rel_pd_idx];
if (!pd_entry->valid) {
if (rsrc_pg) {
pd_entry->rsrc_pg = true;
page = rsrc_pg;
} else {
ret_code = i40iw_allocate_dma_mem(hw, page,
I40IW_HMC_PAGED_BP_SIZE,
I40IW_HMC_PD_BP_BUF_ALIGNMENT);
if (ret_code)
return ret_code;
pd_entry->rsrc_pg = false;
}
memcpy(&pd_entry->bp.addr, page, sizeof(struct i40iw_dma_mem));
pd_entry->bp.sd_pd_index = pd_index;
pd_entry->bp.entry_type = I40IW_SD_TYPE_PAGED;
page_desc = page->pa | 0x1;
pd_addr = (u64 *)pd_table->pd_page_addr.va;
pd_addr += rel_pd_idx;
memcpy(pd_addr, &page_desc, sizeof(*pd_addr));
pd_entry->sd_index = sd_idx;
pd_entry->valid = true;
I40IW_INC_PD_REFCNT(pd_table);
if (hmc_info->hmc_fn_id < I40IW_FIRST_VF_FPM_ID)
I40IW_INVALIDATE_PF_HMC_PD(hw, sd_idx, rel_pd_idx);
else if (hw->hmc.hmc_fn_id != hmc_info->hmc_fn_id)
I40IW_INVALIDATE_VF_HMC_PD(hw, sd_idx, rel_pd_idx,
hmc_info->hmc_fn_id);
}
I40IW_INC_BP_REFCNT(&pd_entry->bp);
return 0;
}
/**
* i40iw_remove_pd_bp - remove a backing page from a page descriptor
* @hw: pointer to our HW structure
* @hmc_info: pointer to the HMC configuration information structure
* @idx: the page index
* @is_pf: distinguishes a VF from a PF
*
* This function:
* 1. Marks the entry in pd table (for paged address mode) or in sd table
* (for direct address mode) invalid.
* 2. Write to register PMPDINV to invalidate the backing page in FV cache
* 3. Decrement the ref count for the pd _entry
* assumptions:
* 1. Caller can deallocate the memory used by backing storage after this
* function returns.
*/
enum i40iw_status_code i40iw_remove_pd_bp(struct i40iw_hw *hw,
struct i40iw_hmc_info *hmc_info,
u32 idx,
bool is_pf)
{
struct i40iw_hmc_pd_entry *pd_entry;
struct i40iw_hmc_pd_table *pd_table;
struct i40iw_hmc_sd_entry *sd_entry;
u32 sd_idx, rel_pd_idx;
struct i40iw_dma_mem *mem;
u64 *pd_addr;
sd_idx = idx / I40IW_HMC_PD_CNT_IN_SD;
rel_pd_idx = idx % I40IW_HMC_PD_CNT_IN_SD;
if (sd_idx >= hmc_info->sd_table.sd_cnt)
return I40IW_ERR_INVALID_PAGE_DESC_INDEX;
sd_entry = &hmc_info->sd_table.sd_entry[sd_idx];
if (sd_entry->entry_type != I40IW_SD_TYPE_PAGED)
return I40IW_ERR_INVALID_SD_TYPE;
pd_table = &hmc_info->sd_table.sd_entry[sd_idx].u.pd_table;
pd_entry = &pd_table->pd_entry[rel_pd_idx];
I40IW_DEC_BP_REFCNT(&pd_entry->bp);
if (pd_entry->bp.ref_cnt)
return 0;
pd_entry->valid = false;
I40IW_DEC_PD_REFCNT(pd_table);
pd_addr = (u64 *)pd_table->pd_page_addr.va;
pd_addr += rel_pd_idx;
memset(pd_addr, 0, sizeof(u64));
if (is_pf)
I40IW_INVALIDATE_PF_HMC_PD(hw, sd_idx, idx);
else
I40IW_INVALIDATE_VF_HMC_PD(hw, sd_idx, idx,
hmc_info->hmc_fn_id);
if (!pd_entry->rsrc_pg) {
mem = &pd_entry->bp.addr;
if (!mem || !mem->va)
return I40IW_ERR_PARAM;
i40iw_free_dma_mem(hw, mem);
}
if (!pd_table->ref_cnt)
i40iw_free_virt_mem(hw, &pd_table->pd_entry_virt_mem);
return 0;
}
/**
* i40iw_prep_remove_sd_bp - Prepares to remove a backing page from a sd entry
* @hmc_info: pointer to the HMC configuration information structure
* @idx: the page index
*/
enum i40iw_status_code i40iw_prep_remove_sd_bp(struct i40iw_hmc_info *hmc_info, u32 idx)
{
struct i40iw_hmc_sd_entry *sd_entry;
sd_entry = &hmc_info->sd_table.sd_entry[idx];
I40IW_DEC_BP_REFCNT(&sd_entry->u.bp);
if (sd_entry->u.bp.ref_cnt)
return I40IW_ERR_NOT_READY;
I40IW_DEC_SD_REFCNT(&hmc_info->sd_table);
sd_entry->valid = false;
return 0;
}
/**
* i40iw_prep_remove_pd_page - Prepares to remove a PD page from sd entry.
* @hmc_info: pointer to the HMC configuration information structure
* @idx: segment descriptor index to find the relevant page descriptor
*/
enum i40iw_status_code i40iw_prep_remove_pd_page(struct i40iw_hmc_info *hmc_info,
u32 idx)
{
struct i40iw_hmc_sd_entry *sd_entry;
sd_entry = &hmc_info->sd_table.sd_entry[idx];
if (sd_entry->u.pd_table.ref_cnt)
return I40IW_ERR_NOT_READY;
sd_entry->valid = false;
I40IW_DEC_SD_REFCNT(&hmc_info->sd_table);
return 0;
}
/**
* i40iw_pf_init_vfhmc -
* @vf_cnt_array: array of cnt values of iwarp hmc objects
* @vf_hmc_fn_id: hmc function id ofr vf driver
* @dev: pointer to i40iw_dev struct
*
* Called by pf driver to initialize hmc_info for vf driver instance.
*/
enum i40iw_status_code i40iw_pf_init_vfhmc(struct i40iw_sc_dev *dev,
u8 vf_hmc_fn_id,
u32 *vf_cnt_array)
{
struct i40iw_hmc_info *hmc_info;
enum i40iw_status_code ret_code = 0;
u32 i;
if ((vf_hmc_fn_id < I40IW_FIRST_VF_FPM_ID) ||
(vf_hmc_fn_id >= I40IW_FIRST_VF_FPM_ID +
I40IW_MAX_PE_ENABLED_VF_COUNT)) {
i40iw_debug(dev, I40IW_DEBUG_HMC, "%s: invalid vf_hmc_fn_id 0x%x\n",
__func__, vf_hmc_fn_id);
return I40IW_ERR_INVALID_HMCFN_ID;
}
ret_code = i40iw_sc_init_iw_hmc(dev, vf_hmc_fn_id);
if (ret_code)
return ret_code;
hmc_info = i40iw_vf_hmcinfo_from_fpm(dev, vf_hmc_fn_id);
for (i = I40IW_HMC_IW_QP; i < I40IW_HMC_IW_MAX; i++)
if (vf_cnt_array)
hmc_info->hmc_obj[i].cnt =
vf_cnt_array[i - I40IW_HMC_IW_QP];
else
hmc_info->hmc_obj[i].cnt = hmc_info->hmc_obj[i].max_cnt;
return 0;
}

241
drivers/infiniband/hw/i40iw/i40iw_hmc.h 100644
View File

@ -0,0 +1,241 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#ifndef I40IW_HMC_H
#define I40IW_HMC_H
#include "i40iw_d.h"
struct i40iw_hw;
enum i40iw_status_code;
#define I40IW_HMC_MAX_BP_COUNT 512
#define I40IW_MAX_SD_ENTRIES 11
#define I40IW_HW_DBG_HMC_INVALID_BP_MARK 0xCA
#define I40IW_HMC_INFO_SIGNATURE 0x484D5347
#define I40IW_HMC_PD_CNT_IN_SD 512
#define I40IW_HMC_DIRECT_BP_SIZE 0x200000
#define I40IW_HMC_MAX_SD_COUNT 4096
#define I40IW_HMC_PAGED_BP_SIZE 4096
#define I40IW_HMC_PD_BP_BUF_ALIGNMENT 4096
#define I40IW_FIRST_VF_FPM_ID 16
#define FPM_MULTIPLIER 1024
#define I40IW_INC_SD_REFCNT(sd_table) ((sd_table)->ref_cnt++)
#define I40IW_INC_PD_REFCNT(pd_table) ((pd_table)->ref_cnt++)
#define I40IW_INC_BP_REFCNT(bp) ((bp)->ref_cnt++)
#define I40IW_DEC_SD_REFCNT(sd_table) ((sd_table)->ref_cnt--)
#define I40IW_DEC_PD_REFCNT(pd_table) ((pd_table)->ref_cnt--)
#define I40IW_DEC_BP_REFCNT(bp) ((bp)->ref_cnt--)
/**
* I40IW_INVALIDATE_PF_HMC_PD - Invalidates the pd cache in the hardware
* @hw: pointer to our hw struct
* @sd_idx: segment descriptor index
* @pd_idx: page descriptor index
*/
#define I40IW_INVALIDATE_PF_HMC_PD(hw, sd_idx, pd_idx) \
i40iw_wr32((hw), I40E_PFHMC_PDINV, \
(((sd_idx) << I40E_PFHMC_PDINV_PMSDIDX_SHIFT) | \
(0x1 << I40E_PFHMC_PDINV_PMSDPARTSEL_SHIFT) | \
((pd_idx) << I40E_PFHMC_PDINV_PMPDIDX_SHIFT)))
/**
* I40IW_INVALIDATE_VF_HMC_PD - Invalidates the pd cache in the hardware
* @hw: pointer to our hw struct
* @sd_idx: segment descriptor index
* @pd_idx: page descriptor index
* @hmc_fn_id: VF's function id
*/
#define I40IW_INVALIDATE_VF_HMC_PD(hw, sd_idx, pd_idx, hmc_fn_id) \
i40iw_wr32(hw, I40E_GLHMC_VFPDINV(hmc_fn_id - I40IW_FIRST_VF_FPM_ID), \
((sd_idx << I40E_PFHMC_PDINV_PMSDIDX_SHIFT) | \
(pd_idx << I40E_PFHMC_PDINV_PMPDIDX_SHIFT)))
struct i40iw_hmc_obj_info {
u64 base;
u32 max_cnt;
u32 cnt;
u64 size;
};
enum i40iw_sd_entry_type {
I40IW_SD_TYPE_INVALID = 0,
I40IW_SD_TYPE_PAGED = 1,
I40IW_SD_TYPE_DIRECT = 2
};
struct i40iw_hmc_bp {
enum i40iw_sd_entry_type entry_type;
struct i40iw_dma_mem addr;
u32 sd_pd_index;
u32 ref_cnt;
};
struct i40iw_hmc_pd_entry {
struct i40iw_hmc_bp bp;
u32 sd_index;
bool rsrc_pg;
bool valid;
};
struct i40iw_hmc_pd_table {
struct i40iw_dma_mem pd_page_addr;
struct i40iw_hmc_pd_entry *pd_entry;
struct i40iw_virt_mem pd_entry_virt_mem;
u32 ref_cnt;
u32 sd_index;
};
struct i40iw_hmc_sd_entry {
enum i40iw_sd_entry_type entry_type;
bool valid;
union {
struct i40iw_hmc_pd_table pd_table;
struct i40iw_hmc_bp bp;
} u;
};
struct i40iw_hmc_sd_table {
struct i40iw_virt_mem addr;
u32 sd_cnt;
u32 ref_cnt;
struct i40iw_hmc_sd_entry *sd_entry;
};
struct i40iw_hmc_info {
u32 signature;
u8 hmc_fn_id;
u16 first_sd_index;
struct i40iw_hmc_obj_info *hmc_obj;
struct i40iw_virt_mem hmc_obj_virt_mem;
struct i40iw_hmc_sd_table sd_table;
u16 sd_indexes[I40IW_HMC_MAX_SD_COUNT];
};
struct update_sd_entry {
u64 cmd;
u64 data;
};
struct i40iw_update_sds_info {
u32 cnt;
u8 hmc_fn_id;
struct update_sd_entry entry[I40IW_MAX_SD_ENTRIES];
};
struct i40iw_ccq_cqe_info;
struct i40iw_hmc_fcn_info {
void (*callback_fcn)(struct i40iw_sc_dev *, void *,
struct i40iw_ccq_cqe_info *);
void *cqp_callback_param;
u32 vf_id;
u16 iw_vf_idx;
bool free_fcn;
};
enum i40iw_hmc_rsrc_type {
I40IW_HMC_IW_QP = 0,
I40IW_HMC_IW_CQ = 1,
I40IW_HMC_IW_SRQ = 2,
I40IW_HMC_IW_HTE = 3,
I40IW_HMC_IW_ARP = 4,
I40IW_HMC_IW_APBVT_ENTRY = 5,
I40IW_HMC_IW_MR = 6,
I40IW_HMC_IW_XF = 7,
I40IW_HMC_IW_XFFL = 8,
I40IW_HMC_IW_Q1 = 9,
I40IW_HMC_IW_Q1FL = 10,
I40IW_HMC_IW_TIMER = 11,
I40IW_HMC_IW_FSIMC = 12,
I40IW_HMC_IW_FSIAV = 13,
I40IW_HMC_IW_PBLE = 14,
I40IW_HMC_IW_MAX = 15,
};
struct i40iw_hmc_create_obj_info {
struct i40iw_hmc_info *hmc_info;
struct i40iw_virt_mem add_sd_virt_mem;
u32 rsrc_type;
u32 start_idx;
u32 count;
u32 add_sd_cnt;
enum i40iw_sd_entry_type entry_type;
bool is_pf;
};
struct i40iw_hmc_del_obj_info {
struct i40iw_hmc_info *hmc_info;
struct i40iw_virt_mem del_sd_virt_mem;
u32 rsrc_type;
u32 start_idx;
u32 count;
u32 del_sd_cnt;
bool is_pf;
};
enum i40iw_status_code i40iw_copy_dma_mem(struct i40iw_hw *hw, void *dest_buf,
struct i40iw_dma_mem *src_mem, u64 src_offset, u64 size);
enum i40iw_status_code i40iw_sc_create_hmc_obj(struct i40iw_sc_dev *dev,
struct i40iw_hmc_create_obj_info *info);
enum i40iw_status_code i40iw_sc_del_hmc_obj(struct i40iw_sc_dev *dev,
struct i40iw_hmc_del_obj_info *info,
bool reset);
enum i40iw_status_code i40iw_hmc_sd_one(struct i40iw_sc_dev *dev, u8 hmc_fn_id,
u64 pa, u32 sd_idx, enum i40iw_sd_entry_type type,
bool setsd);
enum i40iw_status_code i40iw_update_sds_noccq(struct i40iw_sc_dev *dev,
struct i40iw_update_sds_info *info);
struct i40iw_vfdev *i40iw_vfdev_from_fpm(struct i40iw_sc_dev *dev, u8 hmc_fn_id);
struct i40iw_hmc_info *i40iw_vf_hmcinfo_from_fpm(struct i40iw_sc_dev *dev,
u8 hmc_fn_id);
enum i40iw_status_code i40iw_add_sd_table_entry(struct i40iw_hw *hw,
struct i40iw_hmc_info *hmc_info, u32 sd_index,
enum i40iw_sd_entry_type type, u64 direct_mode_sz);
enum i40iw_status_code i40iw_add_pd_table_entry(struct i40iw_hw *hw,
struct i40iw_hmc_info *hmc_info, u32 pd_index,
struct i40iw_dma_mem *rsrc_pg);
enum i40iw_status_code i40iw_remove_pd_bp(struct i40iw_hw *hw,
struct i40iw_hmc_info *hmc_info, u32 idx, bool is_pf);
enum i40iw_status_code i40iw_prep_remove_sd_bp(struct i40iw_hmc_info *hmc_info, u32 idx);
enum i40iw_status_code i40iw_prep_remove_pd_page(struct i40iw_hmc_info *hmc_info, u32 idx);
#define ENTER_SHARED_FUNCTION()
#define EXIT_SHARED_FUNCTION()
#endif /* I40IW_HMC_H */

730
drivers/infiniband/hw/i40iw/i40iw_hw.c 100644
View File

@ -0,0 +1,730 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#include <linux/module.h>
#include <linux/moduleparam.h>
#include <linux/netdevice.h>
#include <linux/etherdevice.h>
#include <linux/ip.h>
#include <linux/tcp.h>
#include <linux/if_vlan.h>
#include "i40iw.h"
/**
* i40iw_initialize_hw_resources - initialize hw resource during open
* @iwdev: iwarp device
*/
u32 i40iw_initialize_hw_resources(struct i40iw_device *iwdev)
{
unsigned long num_pds;
u32 resources_size;
u32 max_mr;
u32 max_qp;
u32 max_cq;
u32 arp_table_size;
u32 mrdrvbits;
void *resource_ptr;
max_qp = iwdev->sc_dev.hmc_info->hmc_obj[I40IW_HMC_IW_QP].cnt;
max_cq = iwdev->sc_dev.hmc_info->hmc_obj[I40IW_HMC_IW_CQ].cnt;
max_mr = iwdev->sc_dev.hmc_info->hmc_obj[I40IW_HMC_IW_MR].cnt;
arp_table_size = iwdev->sc_dev.hmc_info->hmc_obj[I40IW_HMC_IW_ARP].cnt;
iwdev->max_cqe = 0xFFFFF;
num_pds = max_qp * 4;
resources_size = sizeof(struct i40iw_arp_entry) * arp_table_size;
resources_size += sizeof(unsigned long) * BITS_TO_LONGS(max_qp);
resources_size += sizeof(unsigned long) * BITS_TO_LONGS(max_mr);
resources_size += sizeof(unsigned long) * BITS_TO_LONGS(max_cq);
resources_size += sizeof(unsigned long) * BITS_TO_LONGS(num_pds);
resources_size += sizeof(unsigned long) * BITS_TO_LONGS(arp_table_size);
resources_size += sizeof(struct i40iw_qp **) * max_qp;
iwdev->mem_resources = kzalloc(resources_size, GFP_KERNEL);
if (!iwdev->mem_resources)
return -ENOMEM;
iwdev->max_qp = max_qp;
iwdev->max_mr = max_mr;
iwdev->max_cq = max_cq;
iwdev->max_pd = num_pds;
iwdev->arp_table_size = arp_table_size;
iwdev->arp_table = (struct i40iw_arp_entry *)iwdev->mem_resources;
resource_ptr = iwdev->mem_resources + (sizeof(struct i40iw_arp_entry) * arp_table_size);
iwdev->device_cap_flags = IB_DEVICE_LOCAL_DMA_LKEY |
IB_DEVICE_MEM_WINDOW | IB_DEVICE_MEM_MGT_EXTENSIONS;
iwdev->allocated_qps = resource_ptr;
iwdev->allocated_cqs = &iwdev->allocated_qps[BITS_TO_LONGS(max_qp)];
iwdev->allocated_mrs = &iwdev->allocated_cqs[BITS_TO_LONGS(max_cq)];
iwdev->allocated_pds = &iwdev->allocated_mrs[BITS_TO_LONGS(max_mr)];
iwdev->allocated_arps = &iwdev->allocated_pds[BITS_TO_LONGS(num_pds)];
iwdev->qp_table = (struct i40iw_qp **)(&iwdev->allocated_arps[BITS_TO_LONGS(arp_table_size)]);
set_bit(0, iwdev->allocated_mrs);
set_bit(0, iwdev->allocated_qps);
set_bit(0, iwdev->allocated_cqs);
set_bit(0, iwdev->allocated_pds);
set_bit(0, iwdev->allocated_arps);
/* Following for ILQ/IEQ */
set_bit(1, iwdev->allocated_qps);
set_bit(1, iwdev->allocated_cqs);
set_bit(1, iwdev->allocated_pds);
set_bit(2, iwdev->allocated_cqs);
set_bit(2, iwdev->allocated_pds);
spin_lock_init(&iwdev->resource_lock);
mrdrvbits = 24 - get_count_order(iwdev->max_mr);
iwdev->mr_stagmask = ~(((1 << mrdrvbits) - 1) << (32 - mrdrvbits));
return 0;
}
/**
* i40iw_cqp_ce_handler - handle cqp completions
* @iwdev: iwarp device
* @arm: flag to arm after completions
* @cq: cq for cqp completions
*/
static void i40iw_cqp_ce_handler(struct i40iw_device *iwdev, struct i40iw_sc_cq *cq, bool arm)
{
struct i40iw_cqp_request *cqp_request;
struct i40iw_sc_dev *dev = &iwdev->sc_dev;
u32 cqe_count = 0;
struct i40iw_ccq_cqe_info info;
int ret;
do {
memset(&info, 0, sizeof(info));
ret = dev->ccq_ops->ccq_get_cqe_info(cq, &info);
if (ret)
break;
cqp_request = (struct i40iw_cqp_request *)(unsigned long)info.scratch;
if (info.error)
i40iw_pr_err("opcode = 0x%x maj_err_code = 0x%x min_err_code = 0x%x\n",
info.op_code, info.maj_err_code, info.min_err_code);
if (cqp_request) {
cqp_request->compl_info.maj_err_code = info.maj_err_code;
cqp_request->compl_info.min_err_code = info.min_err_code;
cqp_request->compl_info.op_ret_val = info.op_ret_val;
cqp_request->compl_info.error = info.error;
if (cqp_request->waiting) {
cqp_request->request_done = true;
wake_up(&cqp_request->waitq);
i40iw_put_cqp_request(&iwdev->cqp, cqp_request);
} else {
if (cqp_request->callback_fcn)
cqp_request->callback_fcn(cqp_request, 1);
i40iw_put_cqp_request(&iwdev->cqp, cqp_request);
}
}
cqe_count++;
} while (1);
if (arm && cqe_count) {
i40iw_process_bh(dev);
dev->ccq_ops->ccq_arm(cq);
}
}
/**
* i40iw_iwarp_ce_handler - handle iwarp completions
* @iwdev: iwarp device
* @iwcp: iwarp cq receiving event
*/
static void i40iw_iwarp_ce_handler(struct i40iw_device *iwdev,
struct i40iw_sc_cq *iwcq)
{
struct i40iw_cq *i40iwcq = iwcq->back_cq;
if (i40iwcq->ibcq.comp_handler)
i40iwcq->ibcq.comp_handler(&i40iwcq->ibcq,
i40iwcq->ibcq.cq_context);
}
/**
* i40iw_puda_ce_handler - handle puda completion events
* @iwdev: iwarp device
* @cq: puda completion q for event
*/
static void i40iw_puda_ce_handler(struct i40iw_device *iwdev,
struct i40iw_sc_cq *cq)
{
struct i40iw_sc_dev *dev = (struct i40iw_sc_dev *)&iwdev->sc_dev;
enum i40iw_status_code status;
u32 compl_error;
do {
status = i40iw_puda_poll_completion(dev, cq, &compl_error);
if (status == I40IW_ERR_QUEUE_EMPTY)
break;
if (status) {
i40iw_pr_err("puda status = %d\n", status);
break;
}
if (compl_error) {
i40iw_pr_err("puda compl_err =0x%x\n", compl_error);
break;
}
} while (1);
dev->ccq_ops->ccq_arm(cq);
}
/**
* i40iw_process_ceq - handle ceq for completions
* @iwdev: iwarp device
* @ceq: ceq having cq for completion
*/
void i40iw_process_ceq(struct i40iw_device *iwdev, struct i40iw_ceq *ceq)
{
struct i40iw_sc_dev *dev = &iwdev->sc_dev;
struct i40iw_sc_ceq *sc_ceq;
struct i40iw_sc_cq *cq;
bool arm = true;
sc_ceq = &ceq->sc_ceq;
do {
cq = dev->ceq_ops->process_ceq(dev, sc_ceq);
if (!cq)
break;
if (cq->cq_type == I40IW_CQ_TYPE_CQP)
i40iw_cqp_ce_handler(iwdev, cq, arm);
else if (cq->cq_type == I40IW_CQ_TYPE_IWARP)
i40iw_iwarp_ce_handler(iwdev, cq);
else if ((cq->cq_type == I40IW_CQ_TYPE_ILQ) ||
(cq->cq_type == I40IW_CQ_TYPE_IEQ))
i40iw_puda_ce_handler(iwdev, cq);
} while (1);
}
/**
* i40iw_next_iw_state - modify qp state
* @iwqp: iwarp qp to modify
* @state: next state for qp
* @del_hash: del hash
* @term: term message
* @termlen: length of term message
*/
void i40iw_next_iw_state(struct i40iw_qp *iwqp,
u8 state,
u8 del_hash,
u8 term,
u8 termlen)
{
struct i40iw_modify_qp_info info;
memset(&info, 0, sizeof(info));
info.next_iwarp_state = state;
info.remove_hash_idx = del_hash;
info.cq_num_valid = true;
info.arp_cache_idx_valid = true;
info.dont_send_term = true;
info.dont_send_fin = true;
info.termlen = termlen;
if (term & I40IWQP_TERM_SEND_TERM_ONLY)
info.dont_send_term = false;
if (term & I40IWQP_TERM_SEND_FIN_ONLY)
info.dont_send_fin = false;
if (iwqp->sc_qp.term_flags && (state == I40IW_QP_STATE_ERROR))
info.reset_tcp_conn = true;
i40iw_hw_modify_qp(iwqp->iwdev, iwqp, &info, 0);
}
/**
* i40iw_process_aeq - handle aeq events
* @iwdev: iwarp device
*/
void i40iw_process_aeq(struct i40iw_device *iwdev)
{
struct i40iw_sc_dev *dev = &iwdev->sc_dev;
struct i40iw_aeq *aeq = &iwdev->aeq;
struct i40iw_sc_aeq *sc_aeq = &aeq->sc_aeq;
struct i40iw_aeqe_info aeinfo;
struct i40iw_aeqe_info *info = &aeinfo;
int ret;
struct i40iw_qp *iwqp = NULL;
struct i40iw_sc_cq *cq = NULL;
struct i40iw_cq *iwcq = NULL;
struct i40iw_sc_qp *qp = NULL;
struct i40iw_qp_host_ctx_info *ctx_info = NULL;
unsigned long flags;
u32 aeqcnt = 0;
if (!sc_aeq->size)
return;
do {
memset(info, 0, sizeof(*info));
ret = dev->aeq_ops->get_next_aeqe(sc_aeq, info);
if (ret)
break;
aeqcnt++;
i40iw_debug(dev, I40IW_DEBUG_AEQ,
"%s ae_id = 0x%x bool qp=%d qp_id = %d\n",
__func__, info->ae_id, info->qp, info->qp_cq_id);
if (info->qp) {
iwqp = iwdev->qp_table[info->qp_cq_id];
if (!iwqp) {
i40iw_pr_err("qp_id %d is already freed\n", info->qp_cq_id);
continue;
}
qp = &iwqp->sc_qp;
spin_lock_irqsave(&iwqp->lock, flags);
iwqp->hw_tcp_state = info->tcp_state;
iwqp->hw_iwarp_state = info->iwarp_state;
iwqp->last_aeq = info->ae_id;
spin_unlock_irqrestore(&iwqp->lock, flags);
ctx_info = &iwqp->ctx_info;
ctx_info->err_rq_idx_valid = true;
} else {
if (info->ae_id != I40IW_AE_CQ_OPERATION_ERROR)
continue;
}
switch (info->ae_id) {
case I40IW_AE_LLP_FIN_RECEIVED:
if (qp->term_flags)
continue;
if (atomic_inc_return(&iwqp->close_timer_started) == 1) {
iwqp->hw_tcp_state = I40IW_TCP_STATE_CLOSE_WAIT;
if ((iwqp->hw_tcp_state == I40IW_TCP_STATE_CLOSE_WAIT) &&
(iwqp->ibqp_state == IB_QPS_RTS)) {
i40iw_next_iw_state(iwqp,
I40IW_QP_STATE_CLOSING, 0, 0, 0);
i40iw_cm_disconn(iwqp);
}
iwqp->cm_id->add_ref(iwqp->cm_id);
i40iw_schedule_cm_timer(iwqp->cm_node,
(struct i40iw_puda_buf *)iwqp,
I40IW_TIMER_TYPE_CLOSE, 1, 0);
}
break;
case I40IW_AE_LLP_CLOSE_COMPLETE:
if (qp->term_flags)
i40iw_terminate_done(qp, 0);
else
i40iw_cm_disconn(iwqp);
break;
case I40IW_AE_RESET_SENT:
i40iw_next_iw_state(iwqp, I40IW_QP_STATE_ERROR, 1, 0, 0);
i40iw_cm_disconn(iwqp);
break;
case I40IW_AE_LLP_CONNECTION_RESET:
if (atomic_read(&iwqp->close_timer_started))
continue;
i40iw_cm_disconn(iwqp);
break;
case I40IW_AE_TERMINATE_SENT:
i40iw_terminate_send_fin(qp);
break;
case I40IW_AE_LLP_TERMINATE_RECEIVED:
i40iw_terminate_received(qp, info);
break;
case I40IW_AE_CQ_OPERATION_ERROR:
i40iw_pr_err("Processing an iWARP related AE for CQ misc = 0x%04X\n",
info->ae_id);
cq = (struct i40iw_sc_cq *)(unsigned long)info->compl_ctx;
iwcq = (struct i40iw_cq *)cq->back_cq;
if (iwcq->ibcq.event_handler) {
struct ib_event ibevent;
ibevent.device = iwcq->ibcq.device;
ibevent.event = IB_EVENT_CQ_ERR;
ibevent.element.cq = &iwcq->ibcq;
iwcq->ibcq.event_handler(&ibevent, iwcq->ibcq.cq_context);
}
break;
case I40IW_AE_PRIV_OPERATION_DENIED:
case I40IW_AE_STAG_ZERO_INVALID:
case I40IW_AE_IB_RREQ_AND_Q1_FULL:
case I40IW_AE_DDP_UBE_INVALID_DDP_VERSION:
case I40IW_AE_DDP_UBE_INVALID_MO:
case I40IW_AE_DDP_UBE_INVALID_QN:
case I40IW_AE_DDP_NO_L_BIT:
case I40IW_AE_RDMAP_ROE_INVALID_RDMAP_VERSION:
case I40IW_AE_RDMAP_ROE_UNEXPECTED_OPCODE:
case I40IW_AE_ROE_INVALID_RDMA_READ_REQUEST:
case I40IW_AE_ROE_INVALID_RDMA_WRITE_OR_READ_RESP:
case I40IW_AE_INVALID_ARP_ENTRY:
case I40IW_AE_INVALID_TCP_OPTION_RCVD:
case I40IW_AE_STALE_ARP_ENTRY:
case I40IW_AE_LLP_RECEIVED_MPA_CRC_ERROR:
case I40IW_AE_LLP_SEGMENT_TOO_SMALL:
case I40IW_AE_LLP_SYN_RECEIVED:
case I40IW_AE_LLP_TOO_MANY_RETRIES:
case I40IW_AE_LLP_DOUBT_REACHABILITY:
case I40IW_AE_LCE_QP_CATASTROPHIC:
case I40IW_AE_LCE_FUNCTION_CATASTROPHIC:
case I40IW_AE_LCE_CQ_CATASTROPHIC:
case I40IW_AE_UDA_XMIT_DGRAM_TOO_LONG:
case I40IW_AE_UDA_XMIT_IPADDR_MISMATCH:
case I40IW_AE_QP_SUSPEND_COMPLETE:
ctx_info->err_rq_idx_valid = false;
default:
if (!info->sq && ctx_info->err_rq_idx_valid) {
ctx_info->err_rq_idx = info->wqe_idx;
ctx_info->tcp_info_valid = false;
ctx_info->iwarp_info_valid = false;
ret = dev->iw_priv_qp_ops->qp_setctx(&iwqp->sc_qp,
iwqp->host_ctx.va,
ctx_info);
}
i40iw_terminate_connection(qp, info);
break;
}
} while (1);
if (aeqcnt)
dev->aeq_ops->repost_aeq_entries(dev, aeqcnt);
}
/**
* i40iw_manage_apbvt - add or delete tcp port
* @iwdev: iwarp device
* @accel_local_port: port for apbvt
* @add_port: add or delete port
*/
int i40iw_manage_apbvt(struct i40iw_device *iwdev, u16 accel_local_port, bool add_port)
{
struct i40iw_apbvt_info *info;
enum i40iw_status_code status;
struct i40iw_cqp_request *cqp_request;
struct cqp_commands_info *cqp_info;
cqp_request = i40iw_get_cqp_request(&iwdev->cqp, add_port);
if (!cqp_request)
return -ENOMEM;
cqp_info = &cqp_request->info;
info = &cqp_info->in.u.manage_apbvt_entry.info;
memset(info, 0, sizeof(*info));
info->add = add_port;
info->port = cpu_to_le16(accel_local_port);
cqp_info->cqp_cmd = OP_MANAGE_APBVT_ENTRY;
cqp_info->post_sq = 1;
cqp_info->in.u.manage_apbvt_entry.cqp = &iwdev->cqp.sc_cqp;
cqp_info->in.u.manage_apbvt_entry.scratch = (uintptr_t)cqp_request;
status = i40iw_handle_cqp_op(iwdev, cqp_request);
if (status)
i40iw_pr_err("CQP-OP Manage APBVT entry fail");
return status;
}
/**
* i40iw_manage_arp_cache - manage hw arp cache
* @iwdev: iwarp device
* @mac_addr: mac address ptr
* @ip_addr: ip addr for arp cache
* @action: add, delete or modify
*/
void i40iw_manage_arp_cache(struct i40iw_device *iwdev,
unsigned char *mac_addr,
__be32 *ip_addr,
bool ipv4,
u32 action)
{
struct i40iw_add_arp_cache_entry_info *info;
struct i40iw_cqp_request *cqp_request;
struct cqp_commands_info *cqp_info;
int arp_index;
arp_index = i40iw_arp_table(iwdev, ip_addr, ipv4, mac_addr, action);
if (arp_index == -1)
return;
cqp_request = i40iw_get_cqp_request(&iwdev->cqp, false);
if (!cqp_request)
return;
cqp_info = &cqp_request->info;
if (action == I40IW_ARP_ADD) {
cqp_info->cqp_cmd = OP_ADD_ARP_CACHE_ENTRY;
info = &cqp_info->in.u.add_arp_cache_entry.info;
memset(info, 0, sizeof(*info));
info->arp_index = cpu_to_le32(arp_index);
info->permanent = true;
ether_addr_copy(info->mac_addr, mac_addr);
cqp_info->in.u.add_arp_cache_entry.scratch = (uintptr_t)cqp_request;
cqp_info->in.u.add_arp_cache_entry.cqp = &iwdev->cqp.sc_cqp;
} else {
cqp_info->cqp_cmd = OP_DELETE_ARP_CACHE_ENTRY;
cqp_info->in.u.del_arp_cache_entry.scratch = (uintptr_t)cqp_request;
cqp_info->in.u.del_arp_cache_entry.cqp = &iwdev->cqp.sc_cqp;
cqp_info->in.u.del_arp_cache_entry.arp_index = arp_index;
}
cqp_info->in.u.add_arp_cache_entry.cqp = &iwdev->cqp.sc_cqp;
cqp_info->in.u.add_arp_cache_entry.scratch = (uintptr_t)cqp_request;
cqp_info->post_sq = 1;
if (i40iw_handle_cqp_op(iwdev, cqp_request))
i40iw_pr_err("CQP-OP Add/Del Arp Cache entry fail");
}
/**
* i40iw_send_syn_cqp_callback - do syn/ack after qhash
* @cqp_request: qhash cqp completion
* @send_ack: flag send ack
*/
static void i40iw_send_syn_cqp_callback(struct i40iw_cqp_request *cqp_request, u32 send_ack)
{
i40iw_send_syn(cqp_request->param, send_ack);
}
/**
* i40iw_manage_qhash - add or modify qhash
* @iwdev: iwarp device
* @cminfo: cm info for qhash
* @etype: type (syn or quad)
* @mtype: type of qhash
* @cmnode: cmnode associated with connection
* @wait: wait for completion
* @user_pri:user pri of the connection
*/
enum i40iw_status_code i40iw_manage_qhash(struct i40iw_device *iwdev,
struct i40iw_cm_info *cminfo,
enum i40iw_quad_entry_type etype,
enum i40iw_quad_hash_manage_type mtype,
void *cmnode,
bool wait)
{
struct i40iw_qhash_table_info *info;
struct i40iw_sc_dev *dev = &iwdev->sc_dev;
enum i40iw_status_code status;
struct i40iw_cqp *iwcqp = &iwdev->cqp;
struct i40iw_cqp_request *cqp_request;
struct cqp_commands_info *cqp_info;
cqp_request = i40iw_get_cqp_request(iwcqp, wait);
if (!cqp_request)
return I40IW_ERR_NO_MEMORY;
cqp_info = &cqp_request->info;
info = &cqp_info->in.u.manage_qhash_table_entry.info;
memset(info, 0, sizeof(*info));
info->manage = mtype;
info->entry_type = etype;
if (cminfo->vlan_id != 0xFFFF) {
info->vlan_valid = true;
info->vlan_id = cpu_to_le16(cminfo->vlan_id);
} else {
info->vlan_valid = false;
}
info->ipv4_valid = cminfo->ipv4;
ether_addr_copy(info->mac_addr, iwdev->netdev->dev_addr);
info->qp_num = cpu_to_le32(dev->ilq->qp_id);
info->dest_port = cpu_to_le16(cminfo->loc_port);
info->dest_ip[0] = cpu_to_le32(cminfo->loc_addr[0]);
info->dest_ip[1] = cpu_to_le32(cminfo->loc_addr[1]);
info->dest_ip[2] = cpu_to_le32(cminfo->loc_addr[2]);
info->dest_ip[3] = cpu_to_le32(cminfo->loc_addr[3]);
if (etype == I40IW_QHASH_TYPE_TCP_ESTABLISHED) {
info->src_port = cpu_to_le16(cminfo->rem_port);
info->src_ip[0] = cpu_to_le32(cminfo->rem_addr[0]);
info->src_ip[1] = cpu_to_le32(cminfo->rem_addr[1]);
info->src_ip[2] = cpu_to_le32(cminfo->rem_addr[2]);
info->src_ip[3] = cpu_to_le32(cminfo->rem_addr[3]);
}
if (cmnode) {
cqp_request->callback_fcn = i40iw_send_syn_cqp_callback;
cqp_request->param = (void *)cmnode;
}
if (info->ipv4_valid)
i40iw_debug(dev, I40IW_DEBUG_CM,
"%s:%s IP=%pI4, port=%d, mac=%pM, vlan_id=%d\n",
__func__, (!mtype) ? "DELETE" : "ADD",
info->dest_ip,
info->dest_port, info->mac_addr, cminfo->vlan_id);
else
i40iw_debug(dev, I40IW_DEBUG_CM,
"%s:%s IP=%pI6, port=%d, mac=%pM, vlan_id=%d\n",
__func__, (!mtype) ? "DELETE" : "ADD",
info->dest_ip,
info->dest_port, info->mac_addr, cminfo->vlan_id);
cqp_info->in.u.manage_qhash_table_entry.cqp = &iwdev->cqp.sc_cqp;
cqp_info->in.u.manage_qhash_table_entry.scratch = (uintptr_t)cqp_request;
cqp_info->cqp_cmd = OP_MANAGE_QHASH_TABLE_ENTRY;
cqp_info->post_sq = 1;
status = i40iw_handle_cqp_op(iwdev, cqp_request);
if (status)
i40iw_pr_err("CQP-OP Manage Qhash Entry fail");
return status;
}
/**
* i40iw_hw_flush_wqes - flush qp's wqe
* @iwdev: iwarp device
* @qp: hardware control qp
* @info: info for flush
* @wait: flag wait for completion
*/
enum i40iw_status_code i40iw_hw_flush_wqes(struct i40iw_device *iwdev,
struct i40iw_sc_qp *qp,
struct i40iw_qp_flush_info *info,
bool wait)
{
enum i40iw_status_code status;
struct i40iw_qp_flush_info *hw_info;
struct i40iw_cqp_request *cqp_request;
struct cqp_commands_info *cqp_info;
cqp_request = i40iw_get_cqp_request(&iwdev->cqp, wait);
if (!cqp_request)
return I40IW_ERR_NO_MEMORY;
cqp_info = &cqp_request->info;
hw_info = &cqp_request->info.in.u.qp_flush_wqes.info;
memcpy(hw_info, info, sizeof(*hw_info));
cqp_info->cqp_cmd = OP_QP_FLUSH_WQES;
cqp_info->post_sq = 1;
cqp_info->in.u.qp_flush_wqes.qp = qp;
cqp_info->in.u.qp_flush_wqes.scratch = (uintptr_t)cqp_request;
status = i40iw_handle_cqp_op(iwdev, cqp_request);
if (status)
i40iw_pr_err("CQP-OP Flush WQE's fail");
return status;
}
/**
* i40iw_hw_manage_vf_pble_bp - manage vf pbles
* @iwdev: iwarp device
* @info: info for managing pble
* @wait: flag wait for completion
*/
enum i40iw_status_code i40iw_hw_manage_vf_pble_bp(struct i40iw_device *iwdev,
struct i40iw_manage_vf_pble_info *info,
bool wait)
{
enum i40iw_status_code status;
struct i40iw_manage_vf_pble_info *hw_info;
struct i40iw_cqp_request *cqp_request;
struct cqp_commands_info *cqp_info;
if ((iwdev->init_state < CCQ_CREATED) && wait)
wait = false;
cqp_request = i40iw_get_cqp_request(&iwdev->cqp, wait);
if (!cqp_request)
return I40IW_ERR_NO_MEMORY;
cqp_info = &cqp_request->info;
hw_info = &cqp_request->info.in.u.manage_vf_pble_bp.info;
memcpy(hw_info, info, sizeof(*hw_info));
cqp_info->cqp_cmd = OP_MANAGE_VF_PBLE_BP;
cqp_info->post_sq = 1;
cqp_info->in.u.manage_vf_pble_bp.cqp = &iwdev->cqp.sc_cqp;
cqp_info->in.u.manage_vf_pble_bp.scratch = (uintptr_t)cqp_request;
status = i40iw_handle_cqp_op(iwdev, cqp_request);
if (status)
i40iw_pr_err("CQP-OP Manage VF pble_bp fail");
return status;
}
/**
* i40iw_get_ib_wc - return change flush code to IB's
* @opcode: iwarp flush code
*/
static enum ib_wc_status i40iw_get_ib_wc(enum i40iw_flush_opcode opcode)
{
switch (opcode) {
case FLUSH_PROT_ERR:
return IB_WC_LOC_PROT_ERR;
case FLUSH_REM_ACCESS_ERR:
return IB_WC_REM_ACCESS_ERR;
case FLUSH_LOC_QP_OP_ERR:
return IB_WC_LOC_QP_OP_ERR;
case FLUSH_REM_OP_ERR:
return IB_WC_REM_OP_ERR;
case FLUSH_LOC_LEN_ERR:
return IB_WC_LOC_LEN_ERR;
case FLUSH_GENERAL_ERR:
return IB_WC_GENERAL_ERR;
case FLUSH_FATAL_ERR:
default:
return IB_WC_FATAL_ERR;
}
}
/**
* i40iw_set_flush_info - set flush info
* @pinfo: set flush info
* @min: minor err
* @maj: major err
* @opcode: flush error code
*/
static void i40iw_set_flush_info(struct i40iw_qp_flush_info *pinfo,
u16 *min,
u16 *maj,
enum i40iw_flush_opcode opcode)
{
*min = (u16)i40iw_get_ib_wc(opcode);
*maj = CQE_MAJOR_DRV;
pinfo->userflushcode = true;
}
/**
* i40iw_flush_wqes - flush wqe for qp
* @iwdev: iwarp device
* @iwqp: qp to flush wqes
*/
void i40iw_flush_wqes(struct i40iw_device *iwdev, struct i40iw_qp *iwqp)
{
struct i40iw_qp_flush_info info;
struct i40iw_qp_flush_info *pinfo = &info;
struct i40iw_sc_qp *qp = &iwqp->sc_qp;
memset(pinfo, 0, sizeof(*pinfo));
info.sq = true;
info.rq = true;
if (qp->term_flags) {
i40iw_set_flush_info(pinfo, &pinfo->sq_minor_code,
&pinfo->sq_major_code, qp->flush_code);
i40iw_set_flush_info(pinfo, &pinfo->rq_minor_code,
&pinfo->rq_major_code, qp->flush_code);
}
(void)i40iw_hw_flush_wqes(iwdev, &iwqp->sc_qp, &info, true);
}

1910
drivers/infiniband/hw/i40iw/i40iw_main.c 100644
View File

File diff suppressed because it is too large Load Diff

215
drivers/infiniband/hw/i40iw/i40iw_osdep.h 100644
View File

@ -0,0 +1,215 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#ifndef I40IW_OSDEP_H
#define I40IW_OSDEP_H
#include <linux/kernel.h>
#include <linux/string.h>
#include <linux/bitops.h>
#include <net/tcp.h>
#include <crypto/hash.h>
/* get readq/writeq support for 32 bit kernels, use the low-first version */
#include <linux/io-64-nonatomic-lo-hi.h>
#define STATS_TIMER_DELAY 1000
static inline void set_64bit_val(u64 *wqe_words, u32 byte_index, u64 value)
{
wqe_words[byte_index >> 3] = value;
}
/**
* set_32bit_val - set 32 value to hw wqe
* @wqe_words: wqe addr to write
* @byte_index: index in wqe
* @value: value to write
**/
static inline void set_32bit_val(u32 *wqe_words, u32 byte_index, u32 value)
{
wqe_words[byte_index >> 2] = value;
}
/**
* get_64bit_val - read 64 bit value from wqe
* @wqe_words: wqe addr
* @byte_index: index to read from
* @value: read value
**/
static inline void get_64bit_val(u64 *wqe_words, u32 byte_index, u64 *value)
{
*value = wqe_words[byte_index >> 3];
}
/**
* get_32bit_val - read 32 bit value from wqe
* @wqe_words: wqe addr
* @byte_index: index to reaad from
* @value: return 32 bit value
**/
static inline void get_32bit_val(u32 *wqe_words, u32 byte_index, u32 *value)
{
*value = wqe_words[byte_index >> 2];
}
struct i40iw_dma_mem {
void *va;
dma_addr_t pa;
u32 size;
} __packed;
struct i40iw_virt_mem {
void *va;
u32 size;
} __packed;
#define i40iw_debug(h, m, s, ...) \
do { \
if (((m) & (h)->debug_mask)) \
pr_info("i40iw " s, ##__VA_ARGS__); \
} while (0)
#define i40iw_flush(a) readl((a)->hw_addr + I40E_GLGEN_STAT)
#define I40E_GLHMC_VFSDCMD(_i) (0x000C8000 + ((_i) * 4)) \
/* _i=0...31 */
#define I40E_GLHMC_VFSDCMD_MAX_INDEX 31
#define I40E_GLHMC_VFSDCMD_PMSDIDX_SHIFT 0
#define I40E_GLHMC_VFSDCMD_PMSDIDX_MASK (0xFFF \
<< I40E_GLHMC_VFSDCMD_PMSDIDX_SHIFT)
#define I40E_GLHMC_VFSDCMD_PF_SHIFT 16
#define I40E_GLHMC_VFSDCMD_PF_MASK (0xF << I40E_GLHMC_VFSDCMD_PF_SHIFT)
#define I40E_GLHMC_VFSDCMD_VF_SHIFT 20
#define I40E_GLHMC_VFSDCMD_VF_MASK (0x1FF << I40E_GLHMC_VFSDCMD_VF_SHIFT)
#define I40E_GLHMC_VFSDCMD_PMF_TYPE_SHIFT 29
#define I40E_GLHMC_VFSDCMD_PMF_TYPE_MASK (0x3 \
<< I40E_GLHMC_VFSDCMD_PMF_TYPE_SHIFT)
#define I40E_GLHMC_VFSDCMD_PMSDWR_SHIFT 31
#define I40E_GLHMC_VFSDCMD_PMSDWR_MASK (0x1 << I40E_GLHMC_VFSDCMD_PMSDWR_SHIFT)
#define I40E_GLHMC_VFSDDATAHIGH(_i) (0x000C8200 + ((_i) * 4)) \
/* _i=0...31 */
#define I40E_GLHMC_VFSDDATAHIGH_MAX_INDEX 31
#define I40E_GLHMC_VFSDDATAHIGH_PMSDDATAHIGH_SHIFT 0
#define I40E_GLHMC_VFSDDATAHIGH_PMSDDATAHIGH_MASK (0xFFFFFFFF \
<< I40E_GLHMC_VFSDDATAHIGH_PMSDDATAHIGH_SHIFT)
#define I40E_GLHMC_VFSDDATALOW(_i) (0x000C8100 + ((_i) * 4)) \
/* _i=0...31 */
#define I40E_GLHMC_VFSDDATALOW_MAX_INDEX 31
#define I40E_GLHMC_VFSDDATALOW_PMSDVALID_SHIFT 0
#define I40E_GLHMC_VFSDDATALOW_PMSDVALID_MASK (0x1 \
<< I40E_GLHMC_VFSDDATALOW_PMSDVALID_SHIFT)
#define I40E_GLHMC_VFSDDATALOW_PMSDTYPE_SHIFT 1
#define I40E_GLHMC_VFSDDATALOW_PMSDTYPE_MASK (0x1 \
<< I40E_GLHMC_VFSDDATALOW_PMSDTYPE_SHIFT)
#define I40E_GLHMC_VFSDDATALOW_PMSDBPCOUNT_SHIFT 2
#define I40E_GLHMC_VFSDDATALOW_PMSDBPCOUNT_MASK (0x3FF \
<< I40E_GLHMC_VFSDDATALOW_PMSDBPCOUNT_SHIFT)
#define I40E_GLHMC_VFSDDATALOW_PMSDDATALOW_SHIFT 12
#define I40E_GLHMC_VFSDDATALOW_PMSDDATALOW_MASK (0xFFFFF \
<< I40E_GLHMC_VFSDDATALOW_PMSDDATALOW_SHIFT)
#define I40E_GLPE_FWLDSTATUS 0x0000D200
#define I40E_GLPE_FWLDSTATUS_LOAD_REQUESTED_SHIFT 0
#define I40E_GLPE_FWLDSTATUS_LOAD_REQUESTED_MASK (0x1 \
<< I40E_GLPE_FWLDSTATUS_LOAD_REQUESTED_SHIFT)
#define I40E_GLPE_FWLDSTATUS_DONE_SHIFT 1
#define I40E_GLPE_FWLDSTATUS_DONE_MASK (0x1 << I40E_GLPE_FWLDSTATUS_DONE_SHIFT)
#define I40E_GLPE_FWLDSTATUS_CQP_FAIL_SHIFT 2
#define I40E_GLPE_FWLDSTATUS_CQP_FAIL_MASK (0x1 \
<< I40E_GLPE_FWLDSTATUS_CQP_FAIL_SHIFT)
#define I40E_GLPE_FWLDSTATUS_TEP_FAIL_SHIFT 3
#define I40E_GLPE_FWLDSTATUS_TEP_FAIL_MASK (0x1 \
<< I40E_GLPE_FWLDSTATUS_TEP_FAIL_SHIFT)
#define I40E_GLPE_FWLDSTATUS_OOP_FAIL_SHIFT 4
#define I40E_GLPE_FWLDSTATUS_OOP_FAIL_MASK (0x1 \
<< I40E_GLPE_FWLDSTATUS_OOP_FAIL_SHIFT)
struct i40iw_sc_dev;
struct i40iw_sc_qp;
struct i40iw_puda_buf;
struct i40iw_puda_completion_info;
struct i40iw_update_sds_info;
struct i40iw_hmc_fcn_info;
struct i40iw_virtchnl_work_info;
struct i40iw_manage_vf_pble_info;
struct i40iw_device;
struct i40iw_hmc_info;
struct i40iw_hw;
u8 __iomem *i40iw_get_hw_addr(void *dev);
void i40iw_ieq_mpa_crc_ae(struct i40iw_sc_dev *dev, struct i40iw_sc_qp *qp);
enum i40iw_status_code i40iw_vf_wait_vchnl_resp(struct i40iw_sc_dev *dev);
enum i40iw_status_code i40iw_ieq_check_mpacrc(struct shash_desc *desc, void *addr,
u32 length, u32 value);
struct i40iw_sc_qp *i40iw_ieq_get_qp(struct i40iw_sc_dev *dev, struct i40iw_puda_buf *buf);
void i40iw_ieq_update_tcpip_info(struct i40iw_puda_buf *buf, u16 length, u32 seqnum);
void i40iw_free_hash_desc(struct shash_desc *);
enum i40iw_status_code i40iw_init_hash_desc(struct shash_desc **);
enum i40iw_status_code i40iw_puda_get_tcpip_info(struct i40iw_puda_completion_info *info,
struct i40iw_puda_buf *buf);
enum i40iw_status_code i40iw_cqp_sds_cmd(struct i40iw_sc_dev *dev,
struct i40iw_update_sds_info *info);
enum i40iw_status_code i40iw_cqp_manage_hmc_fcn_cmd(struct i40iw_sc_dev *dev,
struct i40iw_hmc_fcn_info *hmcfcninfo);
enum i40iw_status_code i40iw_cqp_query_fpm_values_cmd(struct i40iw_sc_dev *dev,
struct i40iw_dma_mem *values_mem,
u8 hmc_fn_id);
enum i40iw_status_code i40iw_cqp_commit_fpm_values_cmd(struct i40iw_sc_dev *dev,
struct i40iw_dma_mem *values_mem,
u8 hmc_fn_id);
enum i40iw_status_code i40iw_alloc_query_fpm_buf(struct i40iw_sc_dev *dev,
struct i40iw_dma_mem *mem);
enum i40iw_status_code i40iw_cqp_manage_vf_pble_bp(struct i40iw_sc_dev *dev,
struct i40iw_manage_vf_pble_info *info);
void i40iw_cqp_spawn_worker(struct i40iw_sc_dev *dev,
struct i40iw_virtchnl_work_info *work_info, u32 iw_vf_idx);
void *i40iw_remove_head(struct list_head *list);
void i40iw_term_modify_qp(struct i40iw_sc_qp *qp, u8 next_state, u8 term, u8 term_len);
void i40iw_terminate_done(struct i40iw_sc_qp *qp, int timeout_occurred);
void i40iw_terminate_start_timer(struct i40iw_sc_qp *qp);
void i40iw_terminate_del_timer(struct i40iw_sc_qp *qp);
enum i40iw_status_code i40iw_hw_manage_vf_pble_bp(struct i40iw_device *iwdev,
struct i40iw_manage_vf_pble_info *info,
bool wait);
struct i40iw_dev_pestat;
void i40iw_hw_stats_start_timer(struct i40iw_sc_dev *);
void i40iw_hw_stats_del_timer(struct i40iw_sc_dev *);
#define i40iw_mmiowb() mmiowb()
void i40iw_wr32(struct i40iw_hw *hw, u32 reg, u32 value);
u32 i40iw_rd32(struct i40iw_hw *hw, u32 reg);
#endif /* _I40IW_OSDEP_H_ */

106
drivers/infiniband/hw/i40iw/i40iw_p.h 100644
View File

@ -0,0 +1,106 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#ifndef I40IW_P_H
#define I40IW_P_H
#define PAUSE_TIMER_VALUE 0xFFFF
#define REFRESH_THRESHOLD 0x7FFF
#define HIGH_THRESHOLD 0x800
#define LOW_THRESHOLD 0x200
#define ALL_TC2PFC 0xFF
void i40iw_debug_buf(struct i40iw_sc_dev *dev, enum i40iw_debug_flag mask,
char *desc, u64 *buf, u32 size);
/* init operations */
enum i40iw_status_code i40iw_device_init(struct i40iw_sc_dev *dev,
struct i40iw_device_init_info *info);
enum i40iw_status_code i40iw_device_init_pestat(struct i40iw_dev_pestat *);
void i40iw_sc_cqp_post_sq(struct i40iw_sc_cqp *cqp);
u64 *i40iw_sc_cqp_get_next_send_wqe(struct i40iw_sc_cqp *cqp, u64 scratch);
enum i40iw_status_code i40iw_sc_mr_fast_register(struct i40iw_sc_qp *qp,
struct i40iw_fast_reg_stag_info *info,
bool post_sq);
/* HMC/FPM functions */
enum i40iw_status_code i40iw_sc_init_iw_hmc(struct i40iw_sc_dev *dev,
u8 hmc_fn_id);
enum i40iw_status_code i40iw_pf_init_vfhmc(struct i40iw_sc_dev *dev, u8 vf_hmc_fn_id,
u32 *vf_cnt_array);
/* cqp misc functions */
void i40iw_terminate_send_fin(struct i40iw_sc_qp *qp);
void i40iw_terminate_connection(struct i40iw_sc_qp *qp, struct i40iw_aeqe_info *info);
void i40iw_terminate_received(struct i40iw_sc_qp *qp, struct i40iw_aeqe_info *info);
enum i40iw_status_code i40iw_sc_suspend_qp(struct i40iw_sc_cqp *cqp,
struct i40iw_sc_qp *qp, u64 scratch);
enum i40iw_status_code i40iw_sc_resume_qp(struct i40iw_sc_cqp *cqp,
struct i40iw_sc_qp *qp, u64 scratch);
enum i40iw_status_code i40iw_sc_static_hmc_pages_allocated(struct i40iw_sc_cqp *cqp,
u64 scratch, u8 hmc_fn_id,
bool post_sq,
bool poll_registers);
enum i40iw_status_code i40iw_config_fpm_values(struct i40iw_sc_dev *dev, u32 qp_count);
void free_sd_mem(struct i40iw_sc_dev *dev);
enum i40iw_status_code i40iw_process_cqp_cmd(struct i40iw_sc_dev *dev,
struct cqp_commands_info *pcmdinfo);
enum i40iw_status_code i40iw_process_bh(struct i40iw_sc_dev *dev);
/* prototype for functions used for dynamic memory allocation */
enum i40iw_status_code i40iw_allocate_dma_mem(struct i40iw_hw *hw,
struct i40iw_dma_mem *mem, u64 size,
u32 alignment);
void i40iw_free_dma_mem(struct i40iw_hw *hw, struct i40iw_dma_mem *mem);
enum i40iw_status_code i40iw_allocate_virt_mem(struct i40iw_hw *hw,
struct i40iw_virt_mem *mem, u32 size);
enum i40iw_status_code i40iw_free_virt_mem(struct i40iw_hw *hw,
struct i40iw_virt_mem *mem);
u8 i40iw_get_encoded_wqe_size(u32 wqsize, bool cqpsq);
#endif

618
drivers/infiniband/hw/i40iw/i40iw_pble.c 100644
View File

@ -0,0 +1,618 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#include "i40iw_status.h"
#include "i40iw_osdep.h"
#include "i40iw_register.h"
#include "i40iw_hmc.h"
#include "i40iw_d.h"
#include "i40iw_type.h"
#include "i40iw_p.h"
#include <linux/pci.h>
#include <linux/genalloc.h>
#include <linux/vmalloc.h>
#include "i40iw_pble.h"
#include "i40iw.h"
struct i40iw_device;
static enum i40iw_status_code add_pble_pool(struct i40iw_sc_dev *dev,
struct i40iw_hmc_pble_rsrc *pble_rsrc);
static void i40iw_free_vmalloc_mem(struct i40iw_hw *hw, struct i40iw_chunk *chunk);
/**
* i40iw_destroy_pble_pool - destroy pool during module unload
* @pble_rsrc: pble resources
*/
void i40iw_destroy_pble_pool(struct i40iw_sc_dev *dev, struct i40iw_hmc_pble_rsrc *pble_rsrc)
{
struct list_head *clist;
struct list_head *tlist;
struct i40iw_chunk *chunk;
struct i40iw_pble_pool *pinfo = &pble_rsrc->pinfo;
if (pinfo->pool) {
list_for_each_safe(clist, tlist, &pinfo->clist) {
chunk = list_entry(clist, struct i40iw_chunk, list);
if (chunk->type == I40IW_VMALLOC)
i40iw_free_vmalloc_mem(dev->hw, chunk);
kfree(chunk);
}
gen_pool_destroy(pinfo->pool);
}
}
/**
* i40iw_hmc_init_pble - Initialize pble resources during module load
* @dev: i40iw_sc_dev struct
* @pble_rsrc: pble resources
*/
enum i40iw_status_code i40iw_hmc_init_pble(struct i40iw_sc_dev *dev,
struct i40iw_hmc_pble_rsrc *pble_rsrc)
{
struct i40iw_hmc_info *hmc_info;
u32 fpm_idx = 0;
hmc_info = dev->hmc_info;
pble_rsrc->fpm_base_addr = hmc_info->hmc_obj[I40IW_HMC_IW_PBLE].base;
/* Now start the pble' on 4k boundary */
if (pble_rsrc->fpm_base_addr & 0xfff)
fpm_idx = (PAGE_SIZE - (pble_rsrc->fpm_base_addr & 0xfff)) >> 3;
pble_rsrc->unallocated_pble =
hmc_info->hmc_obj[I40IW_HMC_IW_PBLE].cnt - fpm_idx;
pble_rsrc->next_fpm_addr = pble_rsrc->fpm_base_addr + (fpm_idx << 3);
pble_rsrc->pinfo.pool_shift = POOL_SHIFT;
pble_rsrc->pinfo.pool = gen_pool_create(pble_rsrc->pinfo.pool_shift, -1);
INIT_LIST_HEAD(&pble_rsrc->pinfo.clist);
if (!pble_rsrc->pinfo.pool)
goto error;
if (add_pble_pool(dev, pble_rsrc))
goto error;
return 0;
error:i40iw_destroy_pble_pool(dev, pble_rsrc);
return I40IW_ERR_NO_MEMORY;
}
/**
* get_sd_pd_idx - Returns sd index, pd index and rel_pd_idx from fpm address
* @ pble_rsrc: structure containing fpm address
* @ idx: where to return indexes
*/
static inline void get_sd_pd_idx(struct i40iw_hmc_pble_rsrc *pble_rsrc,
struct sd_pd_idx *idx)
{
idx->sd_idx = (u32)(pble_rsrc->next_fpm_addr) / I40IW_HMC_DIRECT_BP_SIZE;
idx->pd_idx = (u32)(pble_rsrc->next_fpm_addr) / I40IW_HMC_PAGED_BP_SIZE;
idx->rel_pd_idx = (idx->pd_idx % I40IW_HMC_PD_CNT_IN_SD);
}
/**
* add_sd_direct - add sd direct for pble
* @dev: hardware control device structure
* @pble_rsrc: pble resource ptr
* @info: page info for sd
*/
static enum i40iw_status_code add_sd_direct(struct i40iw_sc_dev *dev,
struct i40iw_hmc_pble_rsrc *pble_rsrc,
struct i40iw_add_page_info *info)
{
enum i40iw_status_code ret_code = 0;
struct sd_pd_idx *idx = &info->idx;
struct i40iw_chunk *chunk = info->chunk;
struct i40iw_hmc_info *hmc_info = info->hmc_info;
struct i40iw_hmc_sd_entry *sd_entry = info->sd_entry;
u32 offset = 0;
if (!sd_entry->valid) {
if (dev->is_pf) {
ret_code = i40iw_add_sd_table_entry(dev->hw, hmc_info,
info->idx.sd_idx,
I40IW_SD_TYPE_DIRECT,
I40IW_HMC_DIRECT_BP_SIZE);
if (ret_code)
return ret_code;
chunk->type = I40IW_DMA_COHERENT;
}
}
offset = idx->rel_pd_idx << I40IW_HMC_PAGED_BP_SHIFT;
chunk->size = info->pages << I40IW_HMC_PAGED_BP_SHIFT;
chunk->vaddr = ((u8 *)sd_entry->u.bp.addr.va + offset);
chunk->fpm_addr = pble_rsrc->next_fpm_addr;
i40iw_debug(dev, I40IW_DEBUG_PBLE, "chunk_size[%d] = 0x%x vaddr=%p fpm_addr = %llx\n",
chunk->size, chunk->size, chunk->vaddr, chunk->fpm_addr);
return 0;
}
/**
* i40iw_free_vmalloc_mem - free vmalloc during close
* @hw: hw struct
* @chunk: chunk information for vmalloc
*/
static void i40iw_free_vmalloc_mem(struct i40iw_hw *hw, struct i40iw_chunk *chunk)
{
struct pci_dev *pcidev = (struct pci_dev *)hw->dev_context;
int i;
if (!chunk->pg_cnt)
goto done;
for (i = 0; i < chunk->pg_cnt; i++)
dma_unmap_page(&pcidev->dev, chunk->dmaaddrs[i], PAGE_SIZE, DMA_BIDIRECTIONAL);
done:
kfree(chunk->dmaaddrs);
chunk->dmaaddrs = NULL;
vfree(chunk->vaddr);
chunk->vaddr = NULL;
chunk->type = 0;
}
/**
* i40iw_get_vmalloc_mem - get 2M page for sd
* @hw: hardware address
* @chunk: chunk to adf
* @pg_cnt: #of 4 K pages
*/
static enum i40iw_status_code i40iw_get_vmalloc_mem(struct i40iw_hw *hw,
struct i40iw_chunk *chunk,
int pg_cnt)
{
struct pci_dev *pcidev = (struct pci_dev *)hw->dev_context;
struct page *page;
u8 *addr;
u32 size;
int i;
chunk->dmaaddrs = kzalloc(pg_cnt << 3, GFP_KERNEL);
if (!chunk->dmaaddrs)
return I40IW_ERR_NO_MEMORY;
size = PAGE_SIZE * pg_cnt;
chunk->vaddr = vmalloc(size);
if (!chunk->vaddr) {
kfree(chunk->dmaaddrs);
chunk->dmaaddrs = NULL;
return I40IW_ERR_NO_MEMORY;
}
chunk->size = size;
addr = (u8 *)chunk->vaddr;
for (i = 0; i < pg_cnt; i++) {
page = vmalloc_to_page((void *)addr);
if (!page)
break;
chunk->dmaaddrs[i] = dma_map_page(&pcidev->dev, page, 0,
PAGE_SIZE, DMA_BIDIRECTIONAL);
if (dma_mapping_error(&pcidev->dev, chunk->dmaaddrs[i]))
break;
addr += PAGE_SIZE;
}
chunk->pg_cnt = i;
chunk->type = I40IW_VMALLOC;
if (i == pg_cnt)
return 0;
i40iw_free_vmalloc_mem(hw, chunk);
return I40IW_ERR_NO_MEMORY;
}
/**
* fpm_to_idx - given fpm address, get pble index
* @pble_rsrc: pble resource management
* @addr: fpm address for index
*/
static inline u32 fpm_to_idx(struct i40iw_hmc_pble_rsrc *pble_rsrc, u64 addr)
{
return (addr - (pble_rsrc->fpm_base_addr)) >> 3;
}
/**
* add_bp_pages - add backing pages for sd
* @dev: hardware control device structure
* @pble_rsrc: pble resource management
* @info: page info for sd
*/
static enum i40iw_status_code add_bp_pages(struct i40iw_sc_dev *dev,
struct i40iw_hmc_pble_rsrc *pble_rsrc,
struct i40iw_add_page_info *info)
{
u8 *addr;
struct i40iw_dma_mem mem;
struct i40iw_hmc_pd_entry *pd_entry;
struct i40iw_hmc_sd_entry *sd_entry = info->sd_entry;
struct i40iw_hmc_info *hmc_info = info->hmc_info;
struct i40iw_chunk *chunk = info->chunk;
struct i40iw_manage_vf_pble_info vf_pble_info;
enum i40iw_status_code status = 0;
u32 rel_pd_idx = info->idx.rel_pd_idx;
u32 pd_idx = info->idx.pd_idx;
u32 i;
status = i40iw_get_vmalloc_mem(dev->hw, chunk, info->pages);
if (status)
return I40IW_ERR_NO_MEMORY;
status = i40iw_add_sd_table_entry(dev->hw, hmc_info,
info->idx.sd_idx, I40IW_SD_TYPE_PAGED,
I40IW_HMC_DIRECT_BP_SIZE);
if (status) {
i40iw_free_vmalloc_mem(dev->hw, chunk);
return status;
}
if (!dev->is_pf) {
status = i40iw_vchnl_vf_add_hmc_objs(dev, I40IW_HMC_IW_PBLE,
fpm_to_idx(pble_rsrc,
pble_rsrc->next_fpm_addr),
(info->pages << PBLE_512_SHIFT));
if (status) {
i40iw_pr_err("allocate PBLEs in the PF. Error %i\n", status);
i40iw_free_vmalloc_mem(dev->hw, chunk);
return status;
}
}
addr = chunk->vaddr;
for (i = 0; i < info->pages; i++) {
mem.pa = chunk->dmaaddrs[i];
mem.size = PAGE_SIZE;
mem.va = (void *)(addr);
pd_entry = &sd_entry->u.pd_table.pd_entry[rel_pd_idx++];
if (!pd_entry->valid) {
status = i40iw_add_pd_table_entry(dev->hw, hmc_info, pd_idx++, &mem);
if (status)
goto error;
addr += PAGE_SIZE;
} else {
i40iw_pr_err("pd entry is valid expecting to be invalid\n");
}
}
if (!dev->is_pf) {
vf_pble_info.first_pd_index = info->idx.rel_pd_idx;
vf_pble_info.inv_pd_ent = false;
vf_pble_info.pd_entry_cnt = PBLE_PER_PAGE;
vf_pble_info.pd_pl_pba = sd_entry->u.pd_table.pd_page_addr.pa;
vf_pble_info.sd_index = info->idx.sd_idx;
status = i40iw_hw_manage_vf_pble_bp(dev->back_dev,
&vf_pble_info, true);
if (status) {
i40iw_pr_err("CQP manage VF PBLE BP failed. %i\n", status);
goto error;
}
}
chunk->fpm_addr = pble_rsrc->next_fpm_addr;
return 0;
error:
i40iw_free_vmalloc_mem(dev->hw, chunk);
return status;
}
/**
* add_pble_pool - add a sd entry for pble resoure
* @dev: hardware control device structure
* @pble_rsrc: pble resource management
*/
static enum i40iw_status_code add_pble_pool(struct i40iw_sc_dev *dev,
struct i40iw_hmc_pble_rsrc *pble_rsrc)
{
struct i40iw_hmc_sd_entry *sd_entry;
struct i40iw_hmc_info *hmc_info;
struct i40iw_chunk *chunk;
struct i40iw_add_page_info info;
struct sd_pd_idx *idx = &info.idx;
enum i40iw_status_code ret_code = 0;
enum i40iw_sd_entry_type sd_entry_type;
u64 sd_reg_val = 0;
u32 pages;
if (pble_rsrc->unallocated_pble < PBLE_PER_PAGE)
return I40IW_ERR_NO_MEMORY;
if (pble_rsrc->next_fpm_addr & 0xfff) {
i40iw_pr_err("next fpm_addr %llx\n", pble_rsrc->next_fpm_addr);
return I40IW_ERR_INVALID_PAGE_DESC_INDEX;
}
chunk = kzalloc(sizeof(*chunk), GFP_KERNEL);
if (!chunk)
return I40IW_ERR_NO_MEMORY;
hmc_info = dev->hmc_info;
chunk->fpm_addr = pble_rsrc->next_fpm_addr;
get_sd_pd_idx(pble_rsrc, idx);
sd_entry = &hmc_info->sd_table.sd_entry[idx->sd_idx];
pages = (idx->rel_pd_idx) ? (I40IW_HMC_PD_CNT_IN_SD -
idx->rel_pd_idx) : I40IW_HMC_PD_CNT_IN_SD;
pages = min(pages, pble_rsrc->unallocated_pble >> PBLE_512_SHIFT);
if (!pages) {
ret_code = I40IW_ERR_NO_PBLCHUNKS_AVAILABLE;
goto error;
}
info.chunk = chunk;
info.hmc_info = hmc_info;
info.pages = pages;
info.sd_entry = sd_entry;
if (!sd_entry->valid) {
sd_entry_type = (!idx->rel_pd_idx &&
(pages == I40IW_HMC_PD_CNT_IN_SD) &&
dev->is_pf) ? I40IW_SD_TYPE_DIRECT : I40IW_SD_TYPE_PAGED;
} else {
sd_entry_type = sd_entry->entry_type;
}
i40iw_debug(dev, I40IW_DEBUG_PBLE,
"pages = %d, unallocated_pble[%u] current_fpm_addr = %llx\n",
pages, pble_rsrc->unallocated_pble, pble_rsrc->next_fpm_addr);
i40iw_debug(dev, I40IW_DEBUG_PBLE, "sd_entry_type = %d sd_entry valid = %d\n",
sd_entry_type, sd_entry->valid);
if (sd_entry_type == I40IW_SD_TYPE_DIRECT)
ret_code = add_sd_direct(dev, pble_rsrc, &info);
if (ret_code)
sd_entry_type = I40IW_SD_TYPE_PAGED;
else
pble_rsrc->stats_direct_sds++;
if (sd_entry_type == I40IW_SD_TYPE_PAGED) {
ret_code = add_bp_pages(dev, pble_rsrc, &info);
if (ret_code)
goto error;
else
pble_rsrc->stats_paged_sds++;
}
if (gen_pool_add_virt(pble_rsrc->pinfo.pool, (unsigned long)chunk->vaddr,
(phys_addr_t)chunk->fpm_addr, chunk->size, -1)) {
i40iw_pr_err("could not allocate memory by gen_pool_addr_virt()\n");
ret_code = I40IW_ERR_NO_MEMORY;
goto error;
}
pble_rsrc->next_fpm_addr += chunk->size;
i40iw_debug(dev, I40IW_DEBUG_PBLE, "next_fpm_addr = %llx chunk_size[%u] = 0x%x\n",
pble_rsrc->next_fpm_addr, chunk->size, chunk->size);
pble_rsrc->unallocated_pble -= (chunk->size >> 3);
list_add(&chunk->list, &pble_rsrc->pinfo.clist);
sd_reg_val = (sd_entry_type == I40IW_SD_TYPE_PAGED) ?
sd_entry->u.pd_table.pd_page_addr.pa : sd_entry->u.bp.addr.pa;
if (sd_entry->valid)
return 0;
if (dev->is_pf)
ret_code = i40iw_hmc_sd_one(dev, hmc_info->hmc_fn_id,
sd_reg_val, idx->sd_idx,
sd_entry->entry_type, true);
if (ret_code) {
i40iw_pr_err("cqp cmd failed for sd (pbles)\n");
goto error;
}
sd_entry->valid = true;
return 0;
error:
kfree(chunk);
return ret_code;
}
/**
* free_lvl2 - fee level 2 pble
* @pble_rsrc: pble resource management
* @palloc: level 2 pble allocation
*/
static void free_lvl2(struct i40iw_hmc_pble_rsrc *pble_rsrc,
struct i40iw_pble_alloc *palloc)
{
u32 i;
struct gen_pool *pool;
struct i40iw_pble_level2 *lvl2 = &palloc->level2;
struct i40iw_pble_info *root = &lvl2->root;
struct i40iw_pble_info *leaf = lvl2->leaf;
pool = pble_rsrc->pinfo.pool;
for (i = 0; i < lvl2->leaf_cnt; i++, leaf++) {
if (leaf->addr)
gen_pool_free(pool, leaf->addr, (leaf->cnt << 3));
else
break;
}
if (root->addr)
gen_pool_free(pool, root->addr, (root->cnt << 3));
kfree(lvl2->leaf);
lvl2->leaf = NULL;
}
/**
* get_lvl2_pble - get level 2 pble resource
* @pble_rsrc: pble resource management
* @palloc: level 2 pble allocation
* @pool: pool pointer
*/
static enum i40iw_status_code get_lvl2_pble(struct i40iw_hmc_pble_rsrc *pble_rsrc,
struct i40iw_pble_alloc *palloc,
struct gen_pool *pool)
{
u32 lf4k, lflast, total, i;
u32 pblcnt = PBLE_PER_PAGE;
u64 *addr;
struct i40iw_pble_level2 *lvl2 = &palloc->level2;
struct i40iw_pble_info *root = &lvl2->root;
struct i40iw_pble_info *leaf;
/* number of full 512 (4K) leafs) */
lf4k = palloc->total_cnt >> 9;
lflast = palloc->total_cnt % PBLE_PER_PAGE;
total = (lflast == 0) ? lf4k : lf4k + 1;
lvl2->leaf_cnt = total;
leaf = kzalloc((sizeof(*leaf) * total), GFP_ATOMIC);
if (!leaf)
return I40IW_ERR_NO_MEMORY;
lvl2->leaf = leaf;
/* allocate pbles for the root */
root->addr = gen_pool_alloc(pool, (total << 3));
if (!root->addr) {
kfree(lvl2->leaf);
lvl2->leaf = NULL;
return I40IW_ERR_NO_MEMORY;
}
root->idx = fpm_to_idx(pble_rsrc,
(u64)gen_pool_virt_to_phys(pool, root->addr));
root->cnt = total;
addr = (u64 *)root->addr;
for (i = 0; i < total; i++, leaf++) {
pblcnt = (lflast && ((i + 1) == total)) ? lflast : PBLE_PER_PAGE;
leaf->addr = gen_pool_alloc(pool, (pblcnt << 3));
if (!leaf->addr)
goto error;
leaf->idx = fpm_to_idx(pble_rsrc, (u64)gen_pool_virt_to_phys(pool, leaf->addr));
leaf->cnt = pblcnt;
*addr = (u64)leaf->idx;
addr++;
}
palloc->level = I40IW_LEVEL_2;
pble_rsrc->stats_lvl2++;
return 0;
error:
free_lvl2(pble_rsrc, palloc);
return I40IW_ERR_NO_MEMORY;
}
/**
* get_lvl1_pble - get level 1 pble resource
* @dev: hardware control device structure
* @pble_rsrc: pble resource management
* @palloc: level 1 pble allocation
*/
static enum i40iw_status_code get_lvl1_pble(struct i40iw_sc_dev *dev,
struct i40iw_hmc_pble_rsrc *pble_rsrc,
struct i40iw_pble_alloc *palloc)
{
u64 *addr;
struct gen_pool *pool;
struct i40iw_pble_info *lvl1 = &palloc->level1;
pool = pble_rsrc->pinfo.pool;
addr = (u64 *)gen_pool_alloc(pool, (palloc->total_cnt << 3));
if (!addr)
return I40IW_ERR_NO_MEMORY;
palloc->level = I40IW_LEVEL_1;
lvl1->addr = (unsigned long)addr;
lvl1->idx = fpm_to_idx(pble_rsrc, (u64)gen_pool_virt_to_phys(pool,
(unsigned long)addr));
lvl1->cnt = palloc->total_cnt;
pble_rsrc->stats_lvl1++;
return 0;
}
/**
* get_lvl1_lvl2_pble - calls get_lvl1 and get_lvl2 pble routine
* @dev: i40iw_sc_dev struct
* @pble_rsrc: pble resources
* @palloc: contains all inforamtion regarding pble (idx + pble addr)
* @pool: pointer to general purpose special memory pool descriptor
*/
static inline enum i40iw_status_code get_lvl1_lvl2_pble(struct i40iw_sc_dev *dev,
struct i40iw_hmc_pble_rsrc *pble_rsrc,
struct i40iw_pble_alloc *palloc,
struct gen_pool *pool)
{
enum i40iw_status_code status = 0;
status = get_lvl1_pble(dev, pble_rsrc, palloc);
if (status && (palloc->total_cnt > PBLE_PER_PAGE))
status = get_lvl2_pble(pble_rsrc, palloc, pool);
return status;
}
/**
* i40iw_get_pble - allocate pbles from the pool
* @dev: i40iw_sc_dev struct
* @pble_rsrc: pble resources
* @palloc: contains all inforamtion regarding pble (idx + pble addr)
* @pble_cnt: #of pbles requested
*/
enum i40iw_status_code i40iw_get_pble(struct i40iw_sc_dev *dev,
struct i40iw_hmc_pble_rsrc *pble_rsrc,
struct i40iw_pble_alloc *palloc,
u32 pble_cnt)
{
struct gen_pool *pool;
enum i40iw_status_code status = 0;
u32 max_sds = 0;
int i;
pool = pble_rsrc->pinfo.pool;
palloc->total_cnt = pble_cnt;
palloc->level = I40IW_LEVEL_0;
/*check first to see if we can get pble's without acquiring additional sd's */
status = get_lvl1_lvl2_pble(dev, pble_rsrc, palloc, pool);
if (!status)
goto exit;
max_sds = (palloc->total_cnt >> 18) + 1;
for (i = 0; i < max_sds; i++) {
status = add_pble_pool(dev, pble_rsrc);
if (status)
break;
status = get_lvl1_lvl2_pble(dev, pble_rsrc, palloc, pool);
if (!status)
break;
}
exit:
if (!status)
pble_rsrc->stats_alloc_ok++;
else
pble_rsrc->stats_alloc_fail++;
return status;
}
/**
* i40iw_free_pble - put pbles back into pool
* @pble_rsrc: pble resources
* @palloc: contains all inforamtion regarding pble resource being freed
*/
void i40iw_free_pble(struct i40iw_hmc_pble_rsrc *pble_rsrc,
struct i40iw_pble_alloc *palloc)
{
struct gen_pool *pool;
pool = pble_rsrc->pinfo.pool;
if (palloc->level == I40IW_LEVEL_2)
free_lvl2(pble_rsrc, palloc);
else
gen_pool_free(pool, palloc->level1.addr,
(palloc->level1.cnt << 3));
pble_rsrc->stats_alloc_freed++;
}

131
drivers/infiniband/hw/i40iw/i40iw_pble.h 100644
View File

@ -0,0 +1,131 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#ifndef I40IW_PBLE_H
#define I40IW_PBLE_H
#define POOL_SHIFT 6
#define PBLE_PER_PAGE 512
#define I40IW_HMC_PAGED_BP_SHIFT 12
#define PBLE_512_SHIFT 9
enum i40iw_pble_level {
I40IW_LEVEL_0 = 0,
I40IW_LEVEL_1 = 1,
I40IW_LEVEL_2 = 2
};
enum i40iw_alloc_type {
I40IW_NO_ALLOC = 0,
I40IW_DMA_COHERENT = 1,
I40IW_VMALLOC = 2
};
struct i40iw_pble_info {
unsigned long addr;
u32 idx;
u32 cnt;
};
struct i40iw_pble_level2 {
struct i40iw_pble_info root;
struct i40iw_pble_info *leaf;
u32 leaf_cnt;
};
struct i40iw_pble_alloc {
u32 total_cnt;
enum i40iw_pble_level level;
union {
struct i40iw_pble_info level1;
struct i40iw_pble_level2 level2;
};
};
struct sd_pd_idx {
u32 sd_idx;
u32 pd_idx;
u32 rel_pd_idx;
};
struct i40iw_add_page_info {
struct i40iw_chunk *chunk;
struct i40iw_hmc_sd_entry *sd_entry;
struct i40iw_hmc_info *hmc_info;
struct sd_pd_idx idx;
u32 pages;
};
struct i40iw_chunk {
struct list_head list;
u32 size;
void *vaddr;
u64 fpm_addr;
u32 pg_cnt;
dma_addr_t *dmaaddrs;
enum i40iw_alloc_type type;
};
struct i40iw_pble_pool {
struct gen_pool *pool;
struct list_head clist;
u32 total_pble_alloc;
u32 free_pble_cnt;
u32 pool_shift;
};
struct i40iw_hmc_pble_rsrc {
u32 unallocated_pble;
u64 fpm_base_addr;
u64 next_fpm_addr;
struct i40iw_pble_pool pinfo;
u32 stats_direct_sds;
u32 stats_paged_sds;
u64 stats_alloc_ok;
u64 stats_alloc_fail;
u64 stats_alloc_freed;
u64 stats_lvl1;
u64 stats_lvl2;
};
void i40iw_destroy_pble_pool(struct i40iw_sc_dev *dev, struct i40iw_hmc_pble_rsrc *pble_rsrc);
enum i40iw_status_code i40iw_hmc_init_pble(struct i40iw_sc_dev *dev,
struct i40iw_hmc_pble_rsrc *pble_rsrc);
void i40iw_free_pble(struct i40iw_hmc_pble_rsrc *pble_rsrc, struct i40iw_pble_alloc *palloc);
enum i40iw_status_code i40iw_get_pble(struct i40iw_sc_dev *dev,
struct i40iw_hmc_pble_rsrc *pble_rsrc,
struct i40iw_pble_alloc *palloc,
u32 pble_cnt);
#endif

1436
drivers/infiniband/hw/i40iw/i40iw_puda.c 100644
View File

File diff suppressed because it is too large Load Diff

183
drivers/infiniband/hw/i40iw/i40iw_puda.h 100644
View File

@ -0,0 +1,183 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#ifndef I40IW_PUDA_H
#define I40IW_PUDA_H
#define I40IW_IEQ_MPA_FRAMING 6
struct i40iw_sc_dev;
struct i40iw_sc_qp;
struct i40iw_sc_cq;
enum puda_resource_type {
I40IW_PUDA_RSRC_TYPE_ILQ = 1,
I40IW_PUDA_RSRC_TYPE_IEQ
};
enum puda_rsrc_complete {
PUDA_CQ_CREATED = 1,
PUDA_QP_CREATED,
PUDA_TX_COMPLETE,
PUDA_RX_COMPLETE,
PUDA_HASH_CRC_COMPLETE
};
struct i40iw_puda_completion_info {
struct i40iw_qp_uk *qp;
u8 q_type;
u8 vlan_valid;
u8 l3proto;
u8 l4proto;
u16 payload_len;
u32 compl_error; /* No_err=0, else major and minor err code */
u32 qp_id;
u32 wqe_idx;
};
struct i40iw_puda_send_info {
u64 paddr; /* Physical address */
u32 len;
u8 tcplen;
u8 maclen;
bool ipv4;
bool doloopback;
void *scratch;
};
struct i40iw_puda_buf {
struct list_head list; /* MUST be first entry */
struct i40iw_dma_mem mem; /* DMA memory for the buffer */
struct i40iw_puda_buf *next; /* for alloclist in rsrc struct */
struct i40iw_virt_mem buf_mem; /* Buffer memory for this buffer */
void *scratch;
u8 *iph;
u8 *tcph;
u8 *data;
u16 datalen;
u16 vlan_id;
u8 tcphlen; /* tcp length in bytes */
u8 maclen; /* mac length in bytes */
u32 totallen; /* machlen+iphlen+tcphlen+datalen */
atomic_t refcount;
u8 hdrlen;
bool ipv4;
u32 seqnum;
};
struct i40iw_puda_rsrc_info {
enum puda_resource_type type; /* ILQ or IEQ */
u32 count;
u16 pd_id;
u32 cq_id;
u32 qp_id;
u32 sq_size;
u32 rq_size;
u16 buf_size;
u16 mss;
u32 tx_buf_cnt; /* total bufs allocated will be rq_size + tx_buf_cnt */
void (*receive)(struct i40iw_sc_dev *, struct i40iw_puda_buf *);
void (*xmit_complete)(struct i40iw_sc_dev *, void *);
};
struct i40iw_puda_rsrc {
struct i40iw_sc_cq cq;
struct i40iw_sc_qp qp;
struct i40iw_sc_pd sc_pd;
struct i40iw_sc_dev *dev;
struct i40iw_dma_mem cqmem;
struct i40iw_dma_mem qpmem;
struct i40iw_virt_mem ilq_mem;
enum puda_rsrc_complete completion;
enum puda_resource_type type;
u16 buf_size; /*buffer must be max datalen + tcpip hdr + mac */
u16 mss;
u32 cq_id;
u32 qp_id;
u32 sq_size;
u32 rq_size;
u32 cq_size;
struct i40iw_sq_uk_wr_trk_info *sq_wrtrk_array;
u64 *rq_wrid_array;
u32 compl_rxwqe_idx;
u32 rx_wqe_idx;
u32 rxq_invalid_cnt;
u32 tx_wqe_avail_cnt;
bool check_crc;
struct shash_desc *hash_desc;
struct list_head txpend;
struct list_head bufpool; /* free buffers pool list for recv and xmit */
u32 alloc_buf_count;
u32 avail_buf_count; /* snapshot of currently available buffers */
spinlock_t bufpool_lock;
struct i40iw_puda_buf *alloclist;
void (*receive)(struct i40iw_sc_dev *, struct i40iw_puda_buf *);
void (*xmit_complete)(struct i40iw_sc_dev *, void *);
/* puda stats */
u64 stats_buf_alloc_fail;
u64 stats_pkt_rcvd;
u64 stats_pkt_sent;
u64 stats_rcvd_pkt_err;
u64 stats_sent_pkt_q;
u64 stats_bad_qp_id;
};
struct i40iw_puda_buf *i40iw_puda_get_bufpool(struct i40iw_puda_rsrc *rsrc);
void i40iw_puda_ret_bufpool(struct i40iw_puda_rsrc *rsrc,
struct i40iw_puda_buf *buf);
void i40iw_puda_send_buf(struct i40iw_puda_rsrc *rsrc,
struct i40iw_puda_buf *buf);
enum i40iw_status_code i40iw_puda_send(struct i40iw_sc_qp *qp,
struct i40iw_puda_send_info *info);
enum i40iw_status_code i40iw_puda_create_rsrc(struct i40iw_sc_dev *dev,
struct i40iw_puda_rsrc_info *info);
void i40iw_puda_dele_resources(struct i40iw_sc_dev *dev,
enum puda_resource_type type,
bool reset);
enum i40iw_status_code i40iw_puda_poll_completion(struct i40iw_sc_dev *dev,
struct i40iw_sc_cq *cq, u32 *compl_err);
void i40iw_ieq_cleanup_qp(struct i40iw_sc_dev *dev, struct i40iw_sc_qp *qp);
struct i40iw_sc_qp *i40iw_ieq_get_qp(struct i40iw_sc_dev *dev,
struct i40iw_puda_buf *buf);
enum i40iw_status_code i40iw_puda_get_tcpip_info(struct i40iw_puda_completion_info *info,
struct i40iw_puda_buf *buf);
enum i40iw_status_code i40iw_ieq_check_mpacrc(struct shash_desc *desc,
void *addr, u32 length, u32 value);
enum i40iw_status_code i40iw_init_hash_desc(struct shash_desc **desc);
void i40iw_ieq_mpa_crc_ae(struct i40iw_sc_dev *dev, struct i40iw_sc_qp *qp);
void i40iw_free_hash_desc(struct shash_desc *desc);
void i40iw_ieq_update_tcpip_info(struct i40iw_puda_buf *buf, u16 length,
u32 seqnum);
#endif

1030
drivers/infiniband/hw/i40iw/i40iw_register.h 100644
View File

File diff suppressed because it is too large Load Diff

100
drivers/infiniband/hw/i40iw/i40iw_status.h 100644
View File

@ -0,0 +1,100 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#ifndef I40IW_STATUS_H
#define I40IW_STATUS_H
/* Error Codes */
enum i40iw_status_code {
I40IW_SUCCESS = 0,
I40IW_ERR_NVM = -1,
I40IW_ERR_NVM_CHECKSUM = -2,
I40IW_ERR_CONFIG = -4,
I40IW_ERR_PARAM = -5,
I40IW_ERR_DEVICE_NOT_SUPPORTED = -6,
I40IW_ERR_RESET_FAILED = -7,
I40IW_ERR_SWFW_SYNC = -8,
I40IW_ERR_NO_MEMORY = -9,
I40IW_ERR_BAD_PTR = -10,
I40IW_ERR_INVALID_PD_ID = -11,
I40IW_ERR_INVALID_QP_ID = -12,
I40IW_ERR_INVALID_CQ_ID = -13,
I40IW_ERR_INVALID_CEQ_ID = -14,
I40IW_ERR_INVALID_AEQ_ID = -15,
I40IW_ERR_INVALID_SIZE = -16,
I40IW_ERR_INVALID_ARP_INDEX = -17,
I40IW_ERR_INVALID_FPM_FUNC_ID = -18,
I40IW_ERR_QP_INVALID_MSG_SIZE = -19,
I40IW_ERR_QP_TOOMANY_WRS_POSTED = -20,
I40IW_ERR_INVALID_FRAG_COUNT = -21,
I40IW_ERR_QUEUE_EMPTY = -22,
I40IW_ERR_INVALID_ALIGNMENT = -23,
I40IW_ERR_FLUSHED_QUEUE = -24,
I40IW_ERR_INVALID_PUSH_PAGE_INDEX = -25,
I40IW_ERR_INVALID_IMM_DATA_SIZE = -26,
I40IW_ERR_TIMEOUT = -27,
I40IW_ERR_OPCODE_MISMATCH = -28,
I40IW_ERR_CQP_COMPL_ERROR = -29,
I40IW_ERR_INVALID_VF_ID = -30,
I40IW_ERR_INVALID_HMCFN_ID = -31,
I40IW_ERR_BACKING_PAGE_ERROR = -32,
I40IW_ERR_NO_PBLCHUNKS_AVAILABLE = -33,
I40IW_ERR_INVALID_PBLE_INDEX = -34,
I40IW_ERR_INVALID_SD_INDEX = -35,
I40IW_ERR_INVALID_PAGE_DESC_INDEX = -36,
I40IW_ERR_INVALID_SD_TYPE = -37,
I40IW_ERR_MEMCPY_FAILED = -38,
I40IW_ERR_INVALID_HMC_OBJ_INDEX = -39,
I40IW_ERR_INVALID_HMC_OBJ_COUNT = -40,
I40IW_ERR_INVALID_SRQ_ARM_LIMIT = -41,
I40IW_ERR_SRQ_ENABLED = -42,
I40IW_ERR_BUF_TOO_SHORT = -43,
I40IW_ERR_BAD_IWARP_CQE = -44,
I40IW_ERR_NVM_BLANK_MODE = -45,
I40IW_ERR_NOT_IMPLEMENTED = -46,
I40IW_ERR_PE_DOORBELL_NOT_ENABLED = -47,
I40IW_ERR_NOT_READY = -48,
I40IW_NOT_SUPPORTED = -49,
I40IW_ERR_FIRMWARE_API_VERSION = -50,
I40IW_ERR_RING_FULL = -51,
I40IW_ERR_MPA_CRC = -61,
I40IW_ERR_NO_TXBUFS = -62,
I40IW_ERR_SEQ_NUM = -63,
I40IW_ERR_list_empty = -64,
I40IW_ERR_INVALID_MAC_ADDR = -65,
I40IW_ERR_BAD_STAG = -66,
I40IW_ERR_CQ_COMPL_ERROR = -67,
};
#endif

1312
drivers/infiniband/hw/i40iw/i40iw_type.h 100644
View File

File diff suppressed because it is too large Load Diff

107
drivers/infiniband/hw/i40iw/i40iw_ucontext.h 100644
View File

@ -0,0 +1,107 @@
/*
* Copyright (c) 2006 - 2016 Intel Corporation. All rights reserved.
* Copyright (c) 2005 Topspin Communications. All rights reserved.
* Copyright (c) 2005 Cisco Systems. All rights reserved.
* Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*/
#ifndef I40IW_USER_CONTEXT_H
#define I40IW_USER_CONTEXT_H
#include <linux/types.h>
#define I40IW_ABI_USERSPACE_VER 4
#define I40IW_ABI_KERNEL_VER 4
struct i40iw_alloc_ucontext_req {
__u32 reserved32;
__u8 userspace_ver;
__u8 reserved8[3];
};
struct i40iw_alloc_ucontext_resp {
__u32 max_pds; /* maximum pds allowed for this user process */
__u32 max_qps; /* maximum qps allowed for this user process */
__u32 wq_size; /* size of the WQs (sq+rq) allocated to the mmaped area */
__u8 kernel_ver;
__u8 reserved[3];
};
struct i40iw_alloc_pd_resp {
__u32 pd_id;
__u8 reserved[4];
};
struct i40iw_create_cq_req {
__u64 user_cq_buffer;
__u64 user_shadow_area;
};
struct i40iw_create_qp_req {
__u64 user_wqe_buffers;
__u64 user_compl_ctx;
/* UDA QP PHB */
__u64 user_sq_phb; /* place for VA of the sq phb buff */
__u64 user_rq_phb; /* place for VA of the rq phb buff */
};
enum i40iw_memreg_type {
IW_MEMREG_TYPE_MEM = 0x0000,
IW_MEMREG_TYPE_QP = 0x0001,
IW_MEMREG_TYPE_CQ = 0x0002,
};
struct i40iw_mem_reg_req {
__u16 reg_type; /* Memory, QP or CQ */
__u16 cq_pages;
__u16 rq_pages;
__u16 sq_pages;
};
struct i40iw_create_cq_resp {
__u32 cq_id;
__u32 cq_size;
__u32 mmap_db_index;
__u32 reserved;
};
struct i40iw_create_qp_resp {
__u32 qp_id;
__u32 actual_sq_size;
__u32 actual_rq_size;
__u32 i40iw_drv_opt;
__u16 push_idx;
__u8 lsmm;
__u8 rsvd2;
};
#endif

1204
drivers/infiniband/hw/i40iw/i40iw_uk.c 100644
View File

File diff suppressed because it is too large Load Diff

442
drivers/infiniband/hw/i40iw/i40iw_user.h 100644
View File

@ -0,0 +1,442 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#ifndef I40IW_USER_H
#define I40IW_USER_H
enum i40iw_device_capabilities_const {
I40IW_WQE_SIZE = 4,
I40IW_CQP_WQE_SIZE = 8,
I40IW_CQE_SIZE = 4,
I40IW_EXTENDED_CQE_SIZE = 8,
I40IW_AEQE_SIZE = 2,
I40IW_CEQE_SIZE = 1,
I40IW_CQP_CTX_SIZE = 8,
I40IW_SHADOW_AREA_SIZE = 8,
I40IW_CEQ_MAX_COUNT = 256,
I40IW_QUERY_FPM_BUF_SIZE = 128,
I40IW_COMMIT_FPM_BUF_SIZE = 128,
I40IW_MIN_IW_QP_ID = 1,
I40IW_MAX_IW_QP_ID = 262143,
I40IW_MIN_CEQID = 0,
I40IW_MAX_CEQID = 256,
I40IW_MIN_CQID = 0,
I40IW_MAX_CQID = 131071,
I40IW_MIN_AEQ_ENTRIES = 1,
I40IW_MAX_AEQ_ENTRIES = 524287,
I40IW_MIN_CEQ_ENTRIES = 1,
I40IW_MAX_CEQ_ENTRIES = 131071,
I40IW_MIN_CQ_SIZE = 1,
I40IW_MAX_CQ_SIZE = 1048575,
I40IW_MAX_AEQ_ALLOCATE_COUNT = 255,
I40IW_DB_ID_ZERO = 0,
I40IW_MAX_WQ_FRAGMENT_COUNT = 6,
I40IW_MAX_SGE_RD = 1,
I40IW_MAX_OUTBOUND_MESSAGE_SIZE = 2147483647,
I40IW_MAX_INBOUND_MESSAGE_SIZE = 2147483647,
I40IW_MAX_PUSH_PAGE_COUNT = 4096,
I40IW_MAX_PE_ENABLED_VF_COUNT = 32,
I40IW_MAX_VF_FPM_ID = 47,
I40IW_MAX_VF_PER_PF = 127,
I40IW_MAX_SQ_PAYLOAD_SIZE = 2145386496,
I40IW_MAX_INLINE_DATA_SIZE = 112,
I40IW_MAX_PUSHMODE_INLINE_DATA_SIZE = 112,
I40IW_MAX_IRD_SIZE = 32,
I40IW_QPCTX_ENCD_MAXIRD = 3,
I40IW_MAX_WQ_ENTRIES = 2048,
I40IW_MAX_ORD_SIZE = 32,
I40IW_Q2_BUFFER_SIZE = (248 + 100),
I40IW_QP_CTX_SIZE = 248
};
#define i40iw_handle void *
#define i40iw_adapter_handle i40iw_handle
#define i40iw_qp_handle i40iw_handle
#define i40iw_cq_handle i40iw_handle
#define i40iw_srq_handle i40iw_handle
#define i40iw_pd_id i40iw_handle
#define i40iw_stag_handle i40iw_handle
#define i40iw_stag_index u32
#define i40iw_stag u32
#define i40iw_stag_key u8
#define i40iw_tagged_offset u64
#define i40iw_access_privileges u32
#define i40iw_physical_fragment u64
#define i40iw_address_list u64 *
#define I40IW_CREATE_STAG(index, key) (((index) << 8) + (key))
#define I40IW_STAG_KEY_FROM_STAG(stag) ((stag) && 0x000000FF)
#define I40IW_STAG_INDEX_FROM_STAG(stag) (((stag) && 0xFFFFFF00) >> 8)
struct i40iw_qp_uk;
struct i40iw_cq_uk;
struct i40iw_srq_uk;
struct i40iw_qp_uk_init_info;
struct i40iw_cq_uk_init_info;
struct i40iw_srq_uk_init_info;
struct i40iw_sge {
i40iw_tagged_offset tag_off;
u32 len;
i40iw_stag stag;
};
#define i40iw_sgl struct i40iw_sge *
struct i40iw_ring {
u32 head;
u32 tail;
u32 size;
};
struct i40iw_cqe {
u64 buf[I40IW_CQE_SIZE];
};
struct i40iw_extended_cqe {
u64 buf[I40IW_EXTENDED_CQE_SIZE];
};
struct i40iw_wqe {
u64 buf[I40IW_WQE_SIZE];
};
struct i40iw_qp_uk_ops;
enum i40iw_addressing_type {
I40IW_ADDR_TYPE_ZERO_BASED = 0,
I40IW_ADDR_TYPE_VA_BASED = 1,
};
#define I40IW_ACCESS_FLAGS_LOCALREAD 0x01
#define I40IW_ACCESS_FLAGS_LOCALWRITE 0x02
#define I40IW_ACCESS_FLAGS_REMOTEREAD_ONLY 0x04
#define I40IW_ACCESS_FLAGS_REMOTEREAD 0x05
#define I40IW_ACCESS_FLAGS_REMOTEWRITE_ONLY 0x08
#define I40IW_ACCESS_FLAGS_REMOTEWRITE 0x0a
#define I40IW_ACCESS_FLAGS_BIND_WINDOW 0x10
#define I40IW_ACCESS_FLAGS_ALL 0x1F
#define I40IW_OP_TYPE_RDMA_WRITE 0
#define I40IW_OP_TYPE_RDMA_READ 1
#define I40IW_OP_TYPE_SEND 3
#define I40IW_OP_TYPE_SEND_INV 4
#define I40IW_OP_TYPE_SEND_SOL 5
#define I40IW_OP_TYPE_SEND_SOL_INV 6
#define I40IW_OP_TYPE_REC 7
#define I40IW_OP_TYPE_BIND_MW 8
#define I40IW_OP_TYPE_FAST_REG_NSMR 9
#define I40IW_OP_TYPE_INV_STAG 10
#define I40IW_OP_TYPE_RDMA_READ_INV_STAG 11
#define I40IW_OP_TYPE_NOP 12
enum i40iw_completion_status {
I40IW_COMPL_STATUS_SUCCESS = 0,
I40IW_COMPL_STATUS_FLUSHED,
I40IW_COMPL_STATUS_INVALID_WQE,
I40IW_COMPL_STATUS_QP_CATASTROPHIC,
I40IW_COMPL_STATUS_REMOTE_TERMINATION,
I40IW_COMPL_STATUS_INVALID_STAG,
I40IW_COMPL_STATUS_BASE_BOUND_VIOLATION,
I40IW_COMPL_STATUS_ACCESS_VIOLATION,
I40IW_COMPL_STATUS_INVALID_PD_ID,
I40IW_COMPL_STATUS_WRAP_ERROR,
I40IW_COMPL_STATUS_STAG_INVALID_PDID,
I40IW_COMPL_STATUS_RDMA_READ_ZERO_ORD,
I40IW_COMPL_STATUS_QP_NOT_PRIVLEDGED,
I40IW_COMPL_STATUS_STAG_NOT_INVALID,
I40IW_COMPL_STATUS_INVALID_PHYS_BUFFER_SIZE,
I40IW_COMPL_STATUS_INVALID_PHYS_BUFFER_ENTRY,
I40IW_COMPL_STATUS_INVALID_FBO,
I40IW_COMPL_STATUS_INVALID_LENGTH,
I40IW_COMPL_STATUS_INVALID_ACCESS,
I40IW_COMPL_STATUS_PHYS_BUFFER_LIST_TOO_LONG,
I40IW_COMPL_STATUS_INVALID_VIRT_ADDRESS,
I40IW_COMPL_STATUS_INVALID_REGION,
I40IW_COMPL_STATUS_INVALID_WINDOW,
I40IW_COMPL_STATUS_INVALID_TOTAL_LENGTH
};
enum i40iw_completion_notify {
IW_CQ_COMPL_EVENT = 0,
IW_CQ_COMPL_SOLICITED = 1
};
struct i40iw_post_send {
i40iw_sgl sg_list;
u8 num_sges;
};
struct i40iw_post_inline_send {
void *data;
u32 len;
};
struct i40iw_post_send_w_inv {
i40iw_sgl sg_list;
u32 num_sges;
i40iw_stag remote_stag_to_inv;
};
struct i40iw_post_inline_send_w_inv {
void *data;
u32 len;
i40iw_stag remote_stag_to_inv;
};
struct i40iw_rdma_write {
i40iw_sgl lo_sg_list;
u8 num_lo_sges;
struct i40iw_sge rem_addr;
};
struct i40iw_inline_rdma_write {
void *data;
u32 len;
struct i40iw_sge rem_addr;
};
struct i40iw_rdma_read {
struct i40iw_sge lo_addr;
struct i40iw_sge rem_addr;
};
struct i40iw_bind_window {
i40iw_stag mr_stag;
u64 bind_length;
void *va;
enum i40iw_addressing_type addressing_type;
bool enable_reads;
bool enable_writes;
i40iw_stag mw_stag;
};
struct i40iw_inv_local_stag {
i40iw_stag target_stag;
};
struct i40iw_post_sq_info {
u64 wr_id;
u8 op_type;
bool signaled;
bool read_fence;
bool local_fence;
bool inline_data;
bool defer_flag;
union {
struct i40iw_post_send send;
struct i40iw_post_send send_w_sol;
struct i40iw_post_send_w_inv send_w_inv;
struct i40iw_post_send_w_inv send_w_sol_inv;
struct i40iw_rdma_write rdma_write;
struct i40iw_rdma_read rdma_read;
struct i40iw_rdma_read rdma_read_inv;
struct i40iw_bind_window bind_window;
struct i40iw_inv_local_stag inv_local_stag;
struct i40iw_inline_rdma_write inline_rdma_write;
struct i40iw_post_inline_send inline_send;
struct i40iw_post_inline_send inline_send_w_sol;
struct i40iw_post_inline_send_w_inv inline_send_w_inv;
struct i40iw_post_inline_send_w_inv inline_send_w_sol_inv;
} op;
};
struct i40iw_post_rq_info {
u64 wr_id;
i40iw_sgl sg_list;
u32 num_sges;
};
struct i40iw_cq_poll_info {
u64 wr_id;
i40iw_qp_handle qp_handle;
u32 bytes_xfered;
u32 tcp_seq_num;
u32 qp_id;
i40iw_stag inv_stag;
enum i40iw_completion_status comp_status;
u16 major_err;
u16 minor_err;
u8 op_type;
bool stag_invalid_set;
bool push_dropped;
bool error;
bool is_srq;
bool solicited_event;
};
struct i40iw_qp_uk_ops {
void (*iw_qp_post_wr)(struct i40iw_qp_uk *);
void (*iw_qp_ring_push_db)(struct i40iw_qp_uk *, u32);
enum i40iw_status_code (*iw_rdma_write)(struct i40iw_qp_uk *,
struct i40iw_post_sq_info *, bool);
enum i40iw_status_code (*iw_rdma_read)(struct i40iw_qp_uk *,
struct i40iw_post_sq_info *, bool, bool);
enum i40iw_status_code (*iw_send)(struct i40iw_qp_uk *,
struct i40iw_post_sq_info *, u32, bool);
enum i40iw_status_code (*iw_inline_rdma_write)(struct i40iw_qp_uk *,
struct i40iw_post_sq_info *, bool);
enum i40iw_status_code (*iw_inline_send)(struct i40iw_qp_uk *,
struct i40iw_post_sq_info *, u32, bool);
enum i40iw_status_code (*iw_stag_local_invalidate)(struct i40iw_qp_uk *,
struct i40iw_post_sq_info *, bool);
enum i40iw_status_code (*iw_mw_bind)(struct i40iw_qp_uk *,
struct i40iw_post_sq_info *, bool);
enum i40iw_status_code (*iw_post_receive)(struct i40iw_qp_uk *,
struct i40iw_post_rq_info *);
enum i40iw_status_code (*iw_post_nop)(struct i40iw_qp_uk *, u64, bool, bool);
};
struct i40iw_cq_ops {
void (*iw_cq_request_notification)(struct i40iw_cq_uk *,
enum i40iw_completion_notify);
enum i40iw_status_code (*iw_cq_poll_completion)(struct i40iw_cq_uk *,
struct i40iw_cq_poll_info *, bool);
enum i40iw_status_code (*iw_cq_post_entries)(struct i40iw_cq_uk *, u8 count);
void (*iw_cq_clean)(void *, struct i40iw_cq_uk *);
};
struct i40iw_dev_uk;
struct i40iw_device_uk_ops {
enum i40iw_status_code (*iwarp_cq_uk_init)(struct i40iw_cq_uk *,
struct i40iw_cq_uk_init_info *);
enum i40iw_status_code (*iwarp_qp_uk_init)(struct i40iw_qp_uk *,
struct i40iw_qp_uk_init_info *);
};
struct i40iw_dev_uk {
struct i40iw_device_uk_ops ops_uk;
};
struct i40iw_sq_uk_wr_trk_info {
u64 wrid;
u64 wr_len;
};
struct i40iw_qp_quanta {
u64 elem[I40IW_WQE_SIZE];
};
struct i40iw_qp_uk {
struct i40iw_qp_quanta *sq_base;
struct i40iw_qp_quanta *rq_base;
u32 __iomem *wqe_alloc_reg;
struct i40iw_sq_uk_wr_trk_info *sq_wrtrk_array;
u64 *rq_wrid_array;
u64 *shadow_area;
u32 *push_db;
u64 *push_wqe;
struct i40iw_ring sq_ring;
struct i40iw_ring rq_ring;
struct i40iw_ring initial_ring;
u32 qp_id;
u32 sq_size;
u32 rq_size;
struct i40iw_qp_uk_ops ops;
bool use_srq;
u8 swqe_polarity;
u8 swqe_polarity_deferred;
u8 rwqe_polarity;
u8 rq_wqe_size;
u8 rq_wqe_size_multiplier;
u8 max_sq_frag_cnt;
u8 max_rq_frag_cnt;
bool deferred_flag;
};
struct i40iw_cq_uk {
struct i40iw_cqe *cq_base;
u32 __iomem *cqe_alloc_reg;
u64 *shadow_area;
u32 cq_id;
u32 cq_size;
struct i40iw_ring cq_ring;
u8 polarity;
bool avoid_mem_cflct;
struct i40iw_cq_ops ops;
};
struct i40iw_qp_uk_init_info {
struct i40iw_qp_quanta *sq;
struct i40iw_qp_quanta *rq;
u32 __iomem *wqe_alloc_reg;
u64 *shadow_area;
struct i40iw_sq_uk_wr_trk_info *sq_wrtrk_array;
u64 *rq_wrid_array;
u32 *push_db;
u64 *push_wqe;
u32 qp_id;
u32 sq_size;
u32 rq_size;
u8 max_sq_frag_cnt;
u8 max_rq_frag_cnt;
};
struct i40iw_cq_uk_init_info {
u32 __iomem *cqe_alloc_reg;
struct i40iw_cqe *cq_base;
u64 *shadow_area;
u32 cq_size;
u32 cq_id;
bool avoid_mem_cflct;
};
void i40iw_device_init_uk(struct i40iw_dev_uk *dev);
void i40iw_qp_post_wr(struct i40iw_qp_uk *qp);
u64 *i40iw_qp_get_next_send_wqe(struct i40iw_qp_uk *qp, u32 *wqe_idx,
u8 wqe_size);
u64 *i40iw_qp_get_next_recv_wqe(struct i40iw_qp_uk *qp, u32 *wqe_idx);
u64 *i40iw_qp_get_next_srq_wqe(struct i40iw_srq_uk *srq, u32 *wqe_idx);
enum i40iw_status_code i40iw_cq_uk_init(struct i40iw_cq_uk *cq,
struct i40iw_cq_uk_init_info *info);
enum i40iw_status_code i40iw_qp_uk_init(struct i40iw_qp_uk *qp,
struct i40iw_qp_uk_init_info *info);
void i40iw_clean_cq(void *queue, struct i40iw_cq_uk *cq);
enum i40iw_status_code i40iw_nop(struct i40iw_qp_uk *qp, u64 wr_id,
bool signaled, bool post_sq);
enum i40iw_status_code i40iw_fragcnt_to_wqesize_sq(u8 frag_cnt, u8 *wqe_size);
enum i40iw_status_code i40iw_fragcnt_to_wqesize_rq(u8 frag_cnt, u8 *wqe_size);
enum i40iw_status_code i40iw_inline_data_size_to_wqesize(u32 data_size,
u8 *wqe_size);
enum i40iw_status_code i40iw_get_wqe_shift(u32 wqdepth, u8 sge, u8 *shift);
#endif

1270
drivers/infiniband/hw/i40iw/i40iw_utils.c 100644
View File

File diff suppressed because it is too large Load Diff

2437
drivers/infiniband/hw/i40iw/i40iw_verbs.c 100644
View File

File diff suppressed because it is too large Load Diff

173
drivers/infiniband/hw/i40iw/i40iw_verbs.h 100644
View File

@ -0,0 +1,173 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#ifndef I40IW_VERBS_H
#define I40IW_VERBS_H
struct i40iw_ucontext {
struct ib_ucontext ibucontext;
struct i40iw_device *iwdev;
struct list_head cq_reg_mem_list;
spinlock_t cq_reg_mem_list_lock; /* memory list for cq's */
struct list_head qp_reg_mem_list;
spinlock_t qp_reg_mem_list_lock; /* memory list for qp's */
};
struct i40iw_pd {
struct ib_pd ibpd;
struct i40iw_sc_pd sc_pd;
atomic_t usecount;
};
struct i40iw_hmc_pble {
union {
u32 idx;
dma_addr_t addr;
};
};
struct i40iw_cq_mr {
struct i40iw_hmc_pble cq_pbl;
dma_addr_t shadow;
};
struct i40iw_qp_mr {
struct i40iw_hmc_pble sq_pbl;
struct i40iw_hmc_pble rq_pbl;
dma_addr_t shadow;
struct page *sq_page;
};
struct i40iw_pbl {
struct list_head list;
union {
struct i40iw_qp_mr qp_mr;
struct i40iw_cq_mr cq_mr;
};
bool pbl_allocated;
u64 user_base;
struct i40iw_pble_alloc pble_alloc;
struct i40iw_mr *iwmr;
};
#define MAX_SAVE_PAGE_ADDRS 4
struct i40iw_mr {
union {
struct ib_mr ibmr;
struct ib_mw ibmw;
struct ib_fmr ibfmr;
};
struct ib_umem *region;
u16 type;
u32 page_cnt;
u32 stag;
u64 length;
u64 pgaddrmem[MAX_SAVE_PAGE_ADDRS];
struct i40iw_pbl iwpbl;
};
struct i40iw_cq {
struct ib_cq ibcq;
struct i40iw_sc_cq sc_cq;
u16 cq_head;
u16 cq_size;
u16 cq_number;
bool user_mode;
u32 polled_completions;
u32 cq_mem_size;
struct i40iw_dma_mem kmem;
spinlock_t lock; /* for poll cq */
struct i40iw_pbl *iwpbl;
};
struct disconn_work {
struct work_struct work;
struct i40iw_qp *iwqp;
};
struct iw_cm_id;
struct ietf_mpa_frame;
struct i40iw_ud_file;
struct i40iw_qp_kmode {
struct i40iw_dma_mem dma_mem;
u64 *wrid_mem;
};
struct i40iw_qp {
struct ib_qp ibqp;
struct i40iw_sc_qp sc_qp;
struct i40iw_device *iwdev;
struct i40iw_cq *iwscq;
struct i40iw_cq *iwrcq;
struct i40iw_pd *iwpd;
struct i40iw_qp_host_ctx_info ctx_info;
struct i40iwarp_offload_info iwarp_info;
void *allocated_buffer;
atomic_t refcount;
struct iw_cm_id *cm_id;
void *cm_node;
struct ib_mr *lsmm_mr;
struct work_struct work;
enum ib_qp_state ibqp_state;
u32 iwarp_state;
u32 qp_mem_size;
u32 last_aeq;
atomic_t close_timer_started;
spinlock_t lock; /* for post work requests */
struct i40iw_qp_context *iwqp_context;
void *pbl_vbase;
dma_addr_t pbl_pbase;
struct page *page;
u8 active_conn:1;
u8 user_mode:1;
u8 hte_added:1;
u8 flush_issued:1;
u8 destroyed:1;
u8 sig_all:1;
u8 pau_mode:1;
u8 rsvd:1;
u16 term_sq_flush_code;
u16 term_rq_flush_code;
u8 hw_iwarp_state;
u8 hw_tcp_state;
struct i40iw_qp_kmode kqp;
struct i40iw_dma_mem host_ctx;
struct timer_list terminate_timer;
struct i40iw_pbl *iwpbl;
struct i40iw_dma_mem q2_ctx_mem;
struct i40iw_dma_mem ietf_mem;
};
#endif

85
drivers/infiniband/hw/i40iw/i40iw_vf.c 100644
View File

@ -0,0 +1,85 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#include "i40iw_osdep.h"
#include "i40iw_register.h"
#include "i40iw_status.h"
#include "i40iw_hmc.h"
#include "i40iw_d.h"
#include "i40iw_type.h"
#include "i40iw_p.h"
#include "i40iw_vf.h"
/**
* i40iw_manage_vf_pble_bp - manage vf pble
* @cqp: cqp for cqp' sq wqe
* @info: pble info
* @scratch: pointer for completion
* @post_sq: to post and ring
*/
enum i40iw_status_code i40iw_manage_vf_pble_bp(struct i40iw_sc_cqp *cqp,
struct i40iw_manage_vf_pble_info *info,
u64 scratch,
bool post_sq)
{
u64 *wqe;
u64 temp, header, pd_pl_pba = 0;
wqe = i40iw_sc_cqp_get_next_send_wqe(cqp, scratch);
if (!wqe)
return I40IW_ERR_RING_FULL;
temp = LS_64(info->pd_entry_cnt, I40IW_CQPSQ_MVPBP_PD_ENTRY_CNT) |
LS_64(info->first_pd_index, I40IW_CQPSQ_MVPBP_FIRST_PD_INX) |
LS_64(info->sd_index, I40IW_CQPSQ_MVPBP_SD_INX);
set_64bit_val(wqe, 16, temp);
header = LS_64((info->inv_pd_ent ? 1 : 0), I40IW_CQPSQ_MVPBP_INV_PD_ENT) |
LS_64(I40IW_CQP_OP_MANAGE_VF_PBLE_BP, I40IW_CQPSQ_OPCODE) |
LS_64(cqp->polarity, I40IW_CQPSQ_WQEVALID);
set_64bit_val(wqe, 24, header);
pd_pl_pba = LS_64(info->pd_pl_pba >> 3, I40IW_CQPSQ_MVPBP_PD_PLPBA);
set_64bit_val(wqe, 32, pd_pl_pba);
i40iw_debug_buf(cqp->dev, I40IW_DEBUG_WQE, "MANAGE VF_PBLE_BP WQE", wqe, I40IW_CQP_WQE_SIZE * 8);
if (post_sq)
i40iw_sc_cqp_post_sq(cqp);
return 0;
}
struct i40iw_vf_cqp_ops iw_vf_cqp_ops = {
i40iw_manage_vf_pble_bp
};

62
drivers/infiniband/hw/i40iw/i40iw_vf.h 100644
View File

@ -0,0 +1,62 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#ifndef I40IW_VF_H
#define I40IW_VF_H
struct i40iw_sc_cqp;
struct i40iw_manage_vf_pble_info {
u32 sd_index;
u16 first_pd_index;
u16 pd_entry_cnt;
u8 inv_pd_ent;
u64 pd_pl_pba;
};
struct i40iw_vf_cqp_ops {
enum i40iw_status_code (*manage_vf_pble_bp)(struct i40iw_sc_cqp *,
struct i40iw_manage_vf_pble_info *,
u64,
bool);
};
enum i40iw_status_code i40iw_manage_vf_pble_bp(struct i40iw_sc_cqp *cqp,
struct i40iw_manage_vf_pble_info *info,
u64 scratch,
bool post_sq);
extern struct i40iw_vf_cqp_ops iw_vf_cqp_ops;
#endif

748
drivers/infiniband/hw/i40iw/i40iw_virtchnl.c 100644
View File

@ -0,0 +1,748 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#include "i40iw_osdep.h"
#include "i40iw_register.h"
#include "i40iw_status.h"
#include "i40iw_hmc.h"
#include "i40iw_d.h"
#include "i40iw_type.h"
#include "i40iw_p.h"
#include "i40iw_virtchnl.h"
/**
* vchnl_vf_send_get_ver_req - Request Channel version
* @dev: IWARP device pointer
* @vchnl_req: Virtual channel message request pointer
*/
static enum i40iw_status_code vchnl_vf_send_get_ver_req(struct i40iw_sc_dev *dev,
struct i40iw_virtchnl_req *vchnl_req)
{
enum i40iw_status_code ret_code = I40IW_ERR_NOT_READY;
struct i40iw_virtchnl_op_buf *vchnl_msg = vchnl_req->vchnl_msg;
if (!dev->vchnl_up)
return ret_code;
memset(vchnl_msg, 0, sizeof(*vchnl_msg));
vchnl_msg->iw_chnl_op_ctx = (uintptr_t)vchnl_req;
vchnl_msg->iw_chnl_buf_len = sizeof(*vchnl_msg);
vchnl_msg->iw_op_code = I40IW_VCHNL_OP_GET_VER;
vchnl_msg->iw_op_ver = I40IW_VCHNL_OP_GET_VER_V0;
ret_code = dev->vchnl_if.vchnl_send(dev, 0, (u8 *)vchnl_msg, vchnl_msg->iw_chnl_buf_len);
if (ret_code)
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s: virt channel send failed 0x%x\n", __func__, ret_code);
return ret_code;
}
/**
* vchnl_vf_send_get_hmc_fcn_req - Request HMC Function from VF
* @dev: IWARP device pointer
* @vchnl_req: Virtual channel message request pointer
*/
static enum i40iw_status_code vchnl_vf_send_get_hmc_fcn_req(struct i40iw_sc_dev *dev,
struct i40iw_virtchnl_req *vchnl_req)
{
enum i40iw_status_code ret_code = I40IW_ERR_NOT_READY;
struct i40iw_virtchnl_op_buf *vchnl_msg = vchnl_req->vchnl_msg;
if (!dev->vchnl_up)
return ret_code;
memset(vchnl_msg, 0, sizeof(*vchnl_msg));
vchnl_msg->iw_chnl_op_ctx = (uintptr_t)vchnl_req;
vchnl_msg->iw_chnl_buf_len = sizeof(*vchnl_msg);
vchnl_msg->iw_op_code = I40IW_VCHNL_OP_GET_HMC_FCN;
vchnl_msg->iw_op_ver = I40IW_VCHNL_OP_GET_HMC_FCN_V0;
ret_code = dev->vchnl_if.vchnl_send(dev, 0, (u8 *)vchnl_msg, vchnl_msg->iw_chnl_buf_len);
if (ret_code)
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s: virt channel send failed 0x%x\n", __func__, ret_code);
return ret_code;
}
/**
* vchnl_vf_send_get_pe_stats_req - Request PE stats from VF
* @dev: IWARP device pointer
* @vchnl_req: Virtual channel message request pointer
*/
static enum i40iw_status_code vchnl_vf_send_get_pe_stats_req(struct i40iw_sc_dev *dev,
struct i40iw_virtchnl_req *vchnl_req)
{
enum i40iw_status_code ret_code = I40IW_ERR_NOT_READY;
struct i40iw_virtchnl_op_buf *vchnl_msg = vchnl_req->vchnl_msg;
if (!dev->vchnl_up)
return ret_code;
memset(vchnl_msg, 0, sizeof(*vchnl_msg));
vchnl_msg->iw_chnl_op_ctx = (uintptr_t)vchnl_req;
vchnl_msg->iw_chnl_buf_len = sizeof(*vchnl_msg) + sizeof(struct i40iw_dev_hw_stats) - 1;
vchnl_msg->iw_op_code = I40IW_VCHNL_OP_GET_STATS;
vchnl_msg->iw_op_ver = I40IW_VCHNL_OP_GET_STATS_V0;
ret_code = dev->vchnl_if.vchnl_send(dev, 0, (u8 *)vchnl_msg, vchnl_msg->iw_chnl_buf_len);
if (ret_code)
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s: virt channel send failed 0x%x\n", __func__, ret_code);
return ret_code;
}
/**
* vchnl_vf_send_add_hmc_objs_req - Add HMC objects
* @dev: IWARP device pointer
* @vchnl_req: Virtual channel message request pointer
*/
static enum i40iw_status_code vchnl_vf_send_add_hmc_objs_req(struct i40iw_sc_dev *dev,
struct i40iw_virtchnl_req *vchnl_req,
enum i40iw_hmc_rsrc_type rsrc_type,
u32 start_index,
u32 rsrc_count)
{
enum i40iw_status_code ret_code = I40IW_ERR_NOT_READY;
struct i40iw_virtchnl_op_buf *vchnl_msg = vchnl_req->vchnl_msg;
struct i40iw_virtchnl_hmc_obj_range *add_hmc_obj;
if (!dev->vchnl_up)
return ret_code;
add_hmc_obj = (struct i40iw_virtchnl_hmc_obj_range *)vchnl_msg->iw_chnl_buf;
memset(vchnl_msg, 0, sizeof(*vchnl_msg));
memset(add_hmc_obj, 0, sizeof(*add_hmc_obj));
vchnl_msg->iw_chnl_op_ctx = (uintptr_t)vchnl_req;
vchnl_msg->iw_chnl_buf_len = sizeof(*vchnl_msg) + sizeof(struct i40iw_virtchnl_hmc_obj_range) - 1;
vchnl_msg->iw_op_code = I40IW_VCHNL_OP_ADD_HMC_OBJ_RANGE;
vchnl_msg->iw_op_ver = I40IW_VCHNL_OP_ADD_HMC_OBJ_RANGE_V0;
add_hmc_obj->obj_type = (u16)rsrc_type;
add_hmc_obj->start_index = start_index;
add_hmc_obj->obj_count = rsrc_count;
ret_code = dev->vchnl_if.vchnl_send(dev, 0, (u8 *)vchnl_msg, vchnl_msg->iw_chnl_buf_len);
if (ret_code)
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s: virt channel send failed 0x%x\n", __func__, ret_code);
return ret_code;
}
/**
* vchnl_vf_send_del_hmc_objs_req - del HMC objects
* @dev: IWARP device pointer
* @vchnl_req: Virtual channel message request pointer
* @ rsrc_type - resource type to delete
* @ start_index - starting index for resource
* @ rsrc_count - number of resource type to delete
*/
static enum i40iw_status_code vchnl_vf_send_del_hmc_objs_req(struct i40iw_sc_dev *dev,
struct i40iw_virtchnl_req *vchnl_req,
enum i40iw_hmc_rsrc_type rsrc_type,
u32 start_index,
u32 rsrc_count)
{
enum i40iw_status_code ret_code = I40IW_ERR_NOT_READY;
struct i40iw_virtchnl_op_buf *vchnl_msg = vchnl_req->vchnl_msg;
struct i40iw_virtchnl_hmc_obj_range *add_hmc_obj;
if (!dev->vchnl_up)
return ret_code;
add_hmc_obj = (struct i40iw_virtchnl_hmc_obj_range *)vchnl_msg->iw_chnl_buf;
memset(vchnl_msg, 0, sizeof(*vchnl_msg));
memset(add_hmc_obj, 0, sizeof(*add_hmc_obj));
vchnl_msg->iw_chnl_op_ctx = (uintptr_t)vchnl_req;
vchnl_msg->iw_chnl_buf_len = sizeof(*vchnl_msg) + sizeof(struct i40iw_virtchnl_hmc_obj_range) - 1;
vchnl_msg->iw_op_code = I40IW_VCHNL_OP_DEL_HMC_OBJ_RANGE;
vchnl_msg->iw_op_ver = I40IW_VCHNL_OP_DEL_HMC_OBJ_RANGE_V0;
add_hmc_obj->obj_type = (u16)rsrc_type;
add_hmc_obj->start_index = start_index;
add_hmc_obj->obj_count = rsrc_count;
ret_code = dev->vchnl_if.vchnl_send(dev, 0, (u8 *)vchnl_msg, vchnl_msg->iw_chnl_buf_len);
if (ret_code)
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s: virt channel send failed 0x%x\n", __func__, ret_code);
return ret_code;
}
/**
* vchnl_pf_send_get_ver_resp - Send channel version to VF
* @dev: IWARP device pointer
* @vf_id: Virtual function ID associated with the message
* @vchnl_msg: Virtual channel message buffer pointer
*/
static void vchnl_pf_send_get_ver_resp(struct i40iw_sc_dev *dev,
u32 vf_id,
struct i40iw_virtchnl_op_buf *vchnl_msg)
{
enum i40iw_status_code ret_code;
u8 resp_buffer[sizeof(struct i40iw_virtchnl_resp_buf) + sizeof(u32) - 1];
struct i40iw_virtchnl_resp_buf *vchnl_msg_resp = (struct i40iw_virtchnl_resp_buf *)resp_buffer;
memset(resp_buffer, 0, sizeof(*resp_buffer));
vchnl_msg_resp->iw_chnl_op_ctx = vchnl_msg->iw_chnl_op_ctx;
vchnl_msg_resp->iw_chnl_buf_len = sizeof(resp_buffer);
vchnl_msg_resp->iw_op_ret_code = I40IW_SUCCESS;
*((u32 *)vchnl_msg_resp->iw_chnl_buf) = I40IW_VCHNL_CHNL_VER_V0;
ret_code = dev->vchnl_if.vchnl_send(dev, vf_id, resp_buffer, sizeof(resp_buffer));
if (ret_code)
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s: virt channel send failed 0x%x\n", __func__, ret_code);
}
/**
* vchnl_pf_send_get_hmc_fcn_resp - Send HMC Function to VF
* @dev: IWARP device pointer
* @vf_id: Virtual function ID associated with the message
* @vchnl_msg: Virtual channel message buffer pointer
*/
static void vchnl_pf_send_get_hmc_fcn_resp(struct i40iw_sc_dev *dev,
u32 vf_id,
struct i40iw_virtchnl_op_buf *vchnl_msg,
u16 hmc_fcn)
{
enum i40iw_status_code ret_code;
u8 resp_buffer[sizeof(struct i40iw_virtchnl_resp_buf) + sizeof(u16) - 1];
struct i40iw_virtchnl_resp_buf *vchnl_msg_resp = (struct i40iw_virtchnl_resp_buf *)resp_buffer;
memset(resp_buffer, 0, sizeof(*resp_buffer));
vchnl_msg_resp->iw_chnl_op_ctx = vchnl_msg->iw_chnl_op_ctx;
vchnl_msg_resp->iw_chnl_buf_len = sizeof(resp_buffer);
vchnl_msg_resp->iw_op_ret_code = I40IW_SUCCESS;
*((u16 *)vchnl_msg_resp->iw_chnl_buf) = hmc_fcn;
ret_code = dev->vchnl_if.vchnl_send(dev, vf_id, resp_buffer, sizeof(resp_buffer));
if (ret_code)
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s: virt channel send failed 0x%x\n", __func__, ret_code);
}
/**
* vchnl_pf_send_get_pe_stats_resp - Send PE Stats to VF
* @dev: IWARP device pointer
* @vf_id: Virtual function ID associated with the message
* @vchnl_msg: Virtual channel message buffer pointer
* @hw_stats: HW Stats struct
*/
static void vchnl_pf_send_get_pe_stats_resp(struct i40iw_sc_dev *dev,
u32 vf_id,
struct i40iw_virtchnl_op_buf *vchnl_msg,
struct i40iw_dev_hw_stats hw_stats)
{
enum i40iw_status_code ret_code;
u8 resp_buffer[sizeof(struct i40iw_virtchnl_resp_buf) + sizeof(struct i40iw_dev_hw_stats) - 1];
struct i40iw_virtchnl_resp_buf *vchnl_msg_resp = (struct i40iw_virtchnl_resp_buf *)resp_buffer;
memset(resp_buffer, 0, sizeof(*resp_buffer));
vchnl_msg_resp->iw_chnl_op_ctx = vchnl_msg->iw_chnl_op_ctx;
vchnl_msg_resp->iw_chnl_buf_len = sizeof(resp_buffer);
vchnl_msg_resp->iw_op_ret_code = I40IW_SUCCESS;
*((struct i40iw_dev_hw_stats *)vchnl_msg_resp->iw_chnl_buf) = hw_stats;
ret_code = dev->vchnl_if.vchnl_send(dev, vf_id, resp_buffer, sizeof(resp_buffer));
if (ret_code)
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s: virt channel send failed 0x%x\n", __func__, ret_code);
}
/**
* vchnl_pf_send_error_resp - Send an error response to VF
* @dev: IWARP device pointer
* @vf_id: Virtual function ID associated with the message
* @vchnl_msg: Virtual channel message buffer pointer
*/
static void vchnl_pf_send_error_resp(struct i40iw_sc_dev *dev, u32 vf_id,
struct i40iw_virtchnl_op_buf *vchnl_msg,
u16 op_ret_code)
{
enum i40iw_status_code ret_code;
u8 resp_buffer[sizeof(struct i40iw_virtchnl_resp_buf)];
struct i40iw_virtchnl_resp_buf *vchnl_msg_resp = (struct i40iw_virtchnl_resp_buf *)resp_buffer;
memset(resp_buffer, 0, sizeof(resp_buffer));
vchnl_msg_resp->iw_chnl_op_ctx = vchnl_msg->iw_chnl_op_ctx;
vchnl_msg_resp->iw_chnl_buf_len = sizeof(resp_buffer);
vchnl_msg_resp->iw_op_ret_code = (u16)op_ret_code;
ret_code = dev->vchnl_if.vchnl_send(dev, vf_id, resp_buffer, sizeof(resp_buffer));
if (ret_code)
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s: virt channel send failed 0x%x\n", __func__, ret_code);
}
/**
* pf_cqp_get_hmc_fcn_callback - Callback for Get HMC Fcn
* @cqp_req_param: CQP Request param value
* @not_used: unused CQP callback parameter
*/
static void pf_cqp_get_hmc_fcn_callback(struct i40iw_sc_dev *dev, void *callback_param,
struct i40iw_ccq_cqe_info *cqe_info)
{
struct i40iw_vfdev *vf_dev = callback_param;
struct i40iw_virt_mem vf_dev_mem;
if (cqe_info->error) {
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"CQP Completion Error on Get HMC Function. Maj = 0x%04x, Minor = 0x%04x\n",
cqe_info->maj_err_code, cqe_info->min_err_code);
dev->vf_dev[vf_dev->iw_vf_idx] = NULL;
vchnl_pf_send_error_resp(dev, vf_dev->vf_id, &vf_dev->vf_msg_buffer.vchnl_msg,
(u16)I40IW_ERR_CQP_COMPL_ERROR);
vf_dev_mem.va = vf_dev;
vf_dev_mem.size = sizeof(*vf_dev);
i40iw_free_virt_mem(dev->hw, &vf_dev_mem);
} else {
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"CQP Completion Operation Return information = 0x%08x\n",
cqe_info->op_ret_val);
vf_dev->pmf_index = (u16)cqe_info->op_ret_val;
vf_dev->msg_count--;
vchnl_pf_send_get_hmc_fcn_resp(dev,
vf_dev->vf_id,
&vf_dev->vf_msg_buffer.vchnl_msg,
vf_dev->pmf_index);
}
}
/**
* pf_add_hmc_obj - Callback for Add HMC Object
* @vf_dev: pointer to the VF Device
*/
static void pf_add_hmc_obj_callback(void *work_vf_dev)
{
struct i40iw_vfdev *vf_dev = (struct i40iw_vfdev *)work_vf_dev;
struct i40iw_hmc_info *hmc_info = &vf_dev->hmc_info;
struct i40iw_virtchnl_op_buf *vchnl_msg = &vf_dev->vf_msg_buffer.vchnl_msg;
struct i40iw_hmc_create_obj_info info;
struct i40iw_virtchnl_hmc_obj_range *add_hmc_obj;
enum i40iw_status_code ret_code;
if (!vf_dev->pf_hmc_initialized) {
ret_code = i40iw_pf_init_vfhmc(vf_dev->pf_dev, (u8)vf_dev->pmf_index, NULL);
if (ret_code)
goto add_out;
vf_dev->pf_hmc_initialized = true;
}
add_hmc_obj = (struct i40iw_virtchnl_hmc_obj_range *)vchnl_msg->iw_chnl_buf;
memset(&info, 0, sizeof(info));
info.hmc_info = hmc_info;
info.is_pf = false;
info.rsrc_type = (u32)add_hmc_obj->obj_type;
info.entry_type = (info.rsrc_type == I40IW_HMC_IW_PBLE) ? I40IW_SD_TYPE_PAGED : I40IW_SD_TYPE_DIRECT;
info.start_idx = add_hmc_obj->start_index;
info.count = add_hmc_obj->obj_count;
i40iw_debug(vf_dev->pf_dev, I40IW_DEBUG_VIRT,
"I40IW_VCHNL_OP_ADD_HMC_OBJ_RANGE. Add %u type %u objects\n",
info.count, info.rsrc_type);
ret_code = i40iw_sc_create_hmc_obj(vf_dev->pf_dev, &info);
if (!ret_code)
vf_dev->hmc_info.hmc_obj[add_hmc_obj->obj_type].cnt = add_hmc_obj->obj_count;
add_out:
vf_dev->msg_count--;
vchnl_pf_send_error_resp(vf_dev->pf_dev, vf_dev->vf_id, vchnl_msg, (u16)ret_code);
}
/**
* pf_del_hmc_obj_callback - Callback for delete HMC Object
* @work_vf_dev: pointer to the VF Device
*/
static void pf_del_hmc_obj_callback(void *work_vf_dev)
{
struct i40iw_vfdev *vf_dev = (struct i40iw_vfdev *)work_vf_dev;
struct i40iw_hmc_info *hmc_info = &vf_dev->hmc_info;
struct i40iw_virtchnl_op_buf *vchnl_msg = &vf_dev->vf_msg_buffer.vchnl_msg;
struct i40iw_hmc_del_obj_info info;
struct i40iw_virtchnl_hmc_obj_range *del_hmc_obj;
enum i40iw_status_code ret_code = I40IW_SUCCESS;
if (!vf_dev->pf_hmc_initialized)
goto del_out;
del_hmc_obj = (struct i40iw_virtchnl_hmc_obj_range *)vchnl_msg->iw_chnl_buf;
memset(&info, 0, sizeof(info));
info.hmc_info = hmc_info;
info.is_pf = false;
info.rsrc_type = (u32)del_hmc_obj->obj_type;
info.start_idx = del_hmc_obj->start_index;
info.count = del_hmc_obj->obj_count;
i40iw_debug(vf_dev->pf_dev, I40IW_DEBUG_VIRT,
"I40IW_VCHNL_OP_DEL_HMC_OBJ_RANGE. Delete %u type %u objects\n",
info.count, info.rsrc_type);
ret_code = i40iw_sc_del_hmc_obj(vf_dev->pf_dev, &info, false);
del_out:
vf_dev->msg_count--;
vchnl_pf_send_error_resp(vf_dev->pf_dev, vf_dev->vf_id, vchnl_msg, (u16)ret_code);
}
/**
* i40iw_vchnl_recv_pf - Receive PF virtual channel messages
* @dev: IWARP device pointer
* @vf_id: Virtual function ID associated with the message
* @msg: Virtual channel message buffer pointer
* @len: Length of the virtual channels message
*/
enum i40iw_status_code i40iw_vchnl_recv_pf(struct i40iw_sc_dev *dev,
u32 vf_id,
u8 *msg,
u16 len)
{
struct i40iw_virtchnl_op_buf *vchnl_msg = (struct i40iw_virtchnl_op_buf *)msg;
struct i40iw_vfdev *vf_dev = NULL;
struct i40iw_hmc_fcn_info hmc_fcn_info;
u16 iw_vf_idx;
u16 first_avail_iw_vf = I40IW_MAX_PE_ENABLED_VF_COUNT;
struct i40iw_virt_mem vf_dev_mem;
struct i40iw_virtchnl_work_info work_info;
struct i40iw_dev_pestat *devstat;
enum i40iw_status_code ret_code;
unsigned long flags;
if (!dev || !msg || !len)
return I40IW_ERR_PARAM;
if (!dev->vchnl_up)
return I40IW_ERR_NOT_READY;
if (vchnl_msg->iw_op_code == I40IW_VCHNL_OP_GET_VER) {
if (vchnl_msg->iw_op_ver != I40IW_VCHNL_OP_GET_VER_V0)
vchnl_pf_send_get_ver_resp(dev, vf_id, vchnl_msg);
else
vchnl_pf_send_get_ver_resp(dev, vf_id, vchnl_msg);
return I40IW_SUCCESS;
}
for (iw_vf_idx = 0; iw_vf_idx < I40IW_MAX_PE_ENABLED_VF_COUNT;
iw_vf_idx++) {
if (!dev->vf_dev[iw_vf_idx]) {
if (first_avail_iw_vf ==
I40IW_MAX_PE_ENABLED_VF_COUNT)
first_avail_iw_vf = iw_vf_idx;
continue;
}
if (dev->vf_dev[iw_vf_idx]->vf_id == vf_id) {
vf_dev = dev->vf_dev[iw_vf_idx];
break;
}
}
if (vf_dev) {
if (!vf_dev->msg_count) {
vf_dev->msg_count++;
} else {
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"VF%u already has a channel message in progress.\n",
vf_id);
return I40IW_SUCCESS;
}
}
switch (vchnl_msg->iw_op_code) {
case I40IW_VCHNL_OP_GET_HMC_FCN:
if (!vf_dev &&
(first_avail_iw_vf != I40IW_MAX_PE_ENABLED_VF_COUNT)) {
ret_code = i40iw_allocate_virt_mem(dev->hw, &vf_dev_mem, sizeof(struct i40iw_vfdev) +
(sizeof(struct i40iw_hmc_obj_info) * I40IW_HMC_IW_MAX));
if (!ret_code) {
vf_dev = vf_dev_mem.va;
vf_dev->stats_initialized = false;
vf_dev->pf_dev = dev;
vf_dev->msg_count = 1;
vf_dev->vf_id = vf_id;
vf_dev->iw_vf_idx = first_avail_iw_vf;
vf_dev->pf_hmc_initialized = false;
vf_dev->hmc_info.hmc_obj = (struct i40iw_hmc_obj_info *)(&vf_dev[1]);
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"vf_dev %p, hmc_info %p, hmc_obj %p\n",
vf_dev, &vf_dev->hmc_info, vf_dev->hmc_info.hmc_obj);
dev->vf_dev[first_avail_iw_vf] = vf_dev;
iw_vf_idx = first_avail_iw_vf;
} else {
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"VF%u Unable to allocate a VF device structure.\n",
vf_id);
vchnl_pf_send_error_resp(dev, vf_id, vchnl_msg, (u16)I40IW_ERR_NO_MEMORY);
return I40IW_SUCCESS;
}
memcpy(&vf_dev->vf_msg_buffer.vchnl_msg, vchnl_msg, len);
hmc_fcn_info.callback_fcn = pf_cqp_get_hmc_fcn_callback;
hmc_fcn_info.vf_id = vf_id;
hmc_fcn_info.iw_vf_idx = vf_dev->iw_vf_idx;
hmc_fcn_info.cqp_callback_param = vf_dev;
hmc_fcn_info.free_fcn = false;
ret_code = i40iw_cqp_manage_hmc_fcn_cmd(dev, &hmc_fcn_info);
if (ret_code)
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"VF%u error CQP HMC Function operation.\n",
vf_id);
ret_code = i40iw_device_init_pestat(&vf_dev->dev_pestat);
if (ret_code)
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"VF%u - i40iw_device_init_pestat failed\n",
vf_id);
vf_dev->dev_pestat.ops.iw_hw_stat_init(&vf_dev->dev_pestat,
(u8)vf_dev->pmf_index,
dev->hw, false);
vf_dev->stats_initialized = true;
} else {
if (vf_dev) {
vf_dev->msg_count--;
vchnl_pf_send_get_hmc_fcn_resp(dev, vf_id, vchnl_msg, vf_dev->pmf_index);
} else {
vchnl_pf_send_error_resp(dev, vf_id, vchnl_msg,
(u16)I40IW_ERR_NO_MEMORY);
}
}
break;
case I40IW_VCHNL_OP_ADD_HMC_OBJ_RANGE:
if (!vf_dev)
return I40IW_ERR_BAD_PTR;
work_info.worker_vf_dev = vf_dev;
work_info.callback_fcn = pf_add_hmc_obj_callback;
memcpy(&vf_dev->vf_msg_buffer.vchnl_msg, vchnl_msg, len);
i40iw_cqp_spawn_worker(dev, &work_info, vf_dev->iw_vf_idx);
break;
case I40IW_VCHNL_OP_DEL_HMC_OBJ_RANGE:
if (!vf_dev)
return I40IW_ERR_BAD_PTR;
work_info.worker_vf_dev = vf_dev;
work_info.callback_fcn = pf_del_hmc_obj_callback;
memcpy(&vf_dev->vf_msg_buffer.vchnl_msg, vchnl_msg, len);
i40iw_cqp_spawn_worker(dev, &work_info, vf_dev->iw_vf_idx);
break;
case I40IW_VCHNL_OP_GET_STATS:
if (!vf_dev)
return I40IW_ERR_BAD_PTR;
devstat = &vf_dev->dev_pestat;
spin_lock_irqsave(&dev->dev_pestat.stats_lock, flags);
devstat->ops.iw_hw_stat_read_all(devstat, &devstat->hw_stats);
spin_unlock_irqrestore(&dev->dev_pestat.stats_lock, flags);
vf_dev->msg_count--;
vchnl_pf_send_get_pe_stats_resp(dev, vf_id, vchnl_msg, devstat->hw_stats);
break;
default:
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"40iw_vchnl_recv_pf: Invalid OpCode 0x%x\n",
vchnl_msg->iw_op_code);
vchnl_pf_send_error_resp(dev, vf_id,
vchnl_msg, (u16)I40IW_ERR_NOT_IMPLEMENTED);
}
return I40IW_SUCCESS;
}
/**
* i40iw_vchnl_recv_vf - Receive VF virtual channel messages
* @dev: IWARP device pointer
* @vf_id: Virtual function ID associated with the message
* @msg: Virtual channel message buffer pointer
* @len: Length of the virtual channels message
*/
enum i40iw_status_code i40iw_vchnl_recv_vf(struct i40iw_sc_dev *dev,
u32 vf_id,
u8 *msg,
u16 len)
{
struct i40iw_virtchnl_resp_buf *vchnl_msg_resp = (struct i40iw_virtchnl_resp_buf *)msg;
struct i40iw_virtchnl_req *vchnl_req;
vchnl_req = (struct i40iw_virtchnl_req *)(uintptr_t)vchnl_msg_resp->iw_chnl_op_ctx;
vchnl_req->ret_code = (enum i40iw_status_code)vchnl_msg_resp->iw_op_ret_code;
if (len == (sizeof(*vchnl_msg_resp) + vchnl_req->parm_len - 1)) {
if (vchnl_req->parm_len && vchnl_req->parm)
memcpy(vchnl_req->parm, vchnl_msg_resp->iw_chnl_buf, vchnl_req->parm_len);
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s: Got response, data size %u\n", __func__,
vchnl_req->parm_len);
} else {
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s: error length on response, Got %u, expected %u\n", __func__,
len, (u32)(sizeof(*vchnl_msg_resp) + vchnl_req->parm_len - 1));
}
return I40IW_SUCCESS;
}
/**
* i40iw_vchnl_vf_get_ver - Request Channel version
* @dev: IWARP device pointer
* @vchnl_ver: Virtual channel message version pointer
*/
enum i40iw_status_code i40iw_vchnl_vf_get_ver(struct i40iw_sc_dev *dev,
u32 *vchnl_ver)
{
struct i40iw_virtchnl_req vchnl_req;
enum i40iw_status_code ret_code;
memset(&vchnl_req, 0, sizeof(vchnl_req));
vchnl_req.dev = dev;
vchnl_req.parm = vchnl_ver;
vchnl_req.parm_len = sizeof(*vchnl_ver);
vchnl_req.vchnl_msg = &dev->vchnl_vf_msg_buf.vchnl_msg;
ret_code = vchnl_vf_send_get_ver_req(dev, &vchnl_req);
if (!ret_code) {
ret_code = i40iw_vf_wait_vchnl_resp(dev);
if (!ret_code)
ret_code = vchnl_req.ret_code;
else
dev->vchnl_up = false;
} else {
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s Send message failed 0x%0x\n", __func__, ret_code);
}
return ret_code;
}
/**
* i40iw_vchnl_vf_get_hmc_fcn - Request HMC Function
* @dev: IWARP device pointer
* @hmc_fcn: HMC function index pointer
*/
enum i40iw_status_code i40iw_vchnl_vf_get_hmc_fcn(struct i40iw_sc_dev *dev,
u16 *hmc_fcn)
{
struct i40iw_virtchnl_req vchnl_req;
enum i40iw_status_code ret_code;
memset(&vchnl_req, 0, sizeof(vchnl_req));
vchnl_req.dev = dev;
vchnl_req.parm = hmc_fcn;
vchnl_req.parm_len = sizeof(*hmc_fcn);
vchnl_req.vchnl_msg = &dev->vchnl_vf_msg_buf.vchnl_msg;
ret_code = vchnl_vf_send_get_hmc_fcn_req(dev, &vchnl_req);
if (!ret_code) {
ret_code = i40iw_vf_wait_vchnl_resp(dev);
if (!ret_code)
ret_code = vchnl_req.ret_code;
else
dev->vchnl_up = false;
} else {
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s Send message failed 0x%0x\n", __func__, ret_code);
}
return ret_code;
}
/**
* i40iw_vchnl_vf_add_hmc_objs - Add HMC Object
* @dev: IWARP device pointer
* @rsrc_type: HMC Resource type
* @start_index: Starting index of the objects to be added
* @rsrc_count: Number of resources to be added
*/
enum i40iw_status_code i40iw_vchnl_vf_add_hmc_objs(struct i40iw_sc_dev *dev,
enum i40iw_hmc_rsrc_type rsrc_type,
u32 start_index,
u32 rsrc_count)
{
struct i40iw_virtchnl_req vchnl_req;
enum i40iw_status_code ret_code;
memset(&vchnl_req, 0, sizeof(vchnl_req));
vchnl_req.dev = dev;
vchnl_req.vchnl_msg = &dev->vchnl_vf_msg_buf.vchnl_msg;
ret_code = vchnl_vf_send_add_hmc_objs_req(dev,
&vchnl_req,
rsrc_type,
start_index,
rsrc_count);
if (!ret_code) {
ret_code = i40iw_vf_wait_vchnl_resp(dev);
if (!ret_code)
ret_code = vchnl_req.ret_code;
else
dev->vchnl_up = false;
} else {
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s Send message failed 0x%0x\n", __func__, ret_code);
}
return ret_code;
}
/**
* i40iw_vchnl_vf_del_hmc_obj - del HMC obj
* @dev: IWARP device pointer
* @rsrc_type: HMC Resource type
* @start_index: Starting index of the object to delete
* @rsrc_count: Number of resources to be delete
*/
enum i40iw_status_code i40iw_vchnl_vf_del_hmc_obj(struct i40iw_sc_dev *dev,
enum i40iw_hmc_rsrc_type rsrc_type,
u32 start_index,
u32 rsrc_count)
{
struct i40iw_virtchnl_req vchnl_req;
enum i40iw_status_code ret_code;
memset(&vchnl_req, 0, sizeof(vchnl_req));
vchnl_req.dev = dev;
vchnl_req.vchnl_msg = &dev->vchnl_vf_msg_buf.vchnl_msg;
ret_code = vchnl_vf_send_del_hmc_objs_req(dev,
&vchnl_req,
rsrc_type,
start_index,
rsrc_count);
if (!ret_code) {
ret_code = i40iw_vf_wait_vchnl_resp(dev);
if (!ret_code)
ret_code = vchnl_req.ret_code;
else
dev->vchnl_up = false;
} else {
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s Send message failed 0x%0x\n", __func__, ret_code);
}
return ret_code;
}
/**
* i40iw_vchnl_vf_get_pe_stats - Get PE stats
* @dev: IWARP device pointer
* @hw_stats: HW stats struct
*/
enum i40iw_status_code i40iw_vchnl_vf_get_pe_stats(struct i40iw_sc_dev *dev,
struct i40iw_dev_hw_stats *hw_stats)
{
struct i40iw_virtchnl_req vchnl_req;
enum i40iw_status_code ret_code;
memset(&vchnl_req, 0, sizeof(vchnl_req));
vchnl_req.dev = dev;
vchnl_req.parm = hw_stats;
vchnl_req.parm_len = sizeof(*hw_stats);
vchnl_req.vchnl_msg = &dev->vchnl_vf_msg_buf.vchnl_msg;
ret_code = vchnl_vf_send_get_pe_stats_req(dev, &vchnl_req);
if (!ret_code) {
ret_code = i40iw_vf_wait_vchnl_resp(dev);
if (!ret_code)
ret_code = vchnl_req.ret_code;
else
dev->vchnl_up = false;
} else {
i40iw_debug(dev, I40IW_DEBUG_VIRT,
"%s Send message failed 0x%0x\n", __func__, ret_code);
}
return ret_code;
}

124
drivers/infiniband/hw/i40iw/i40iw_virtchnl.h 100644
View File

@ -0,0 +1,124 @@
/*******************************************************************************
*
* Copyright (c) 2015-2016 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenFabrics.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
*******************************************************************************/
#ifndef I40IW_VIRTCHNL_H
#define I40IW_VIRTCHNL_H
#include "i40iw_hmc.h"
#pragma pack(push, 1)
struct i40iw_virtchnl_op_buf {
u16 iw_op_code;
u16 iw_op_ver;
u16 iw_chnl_buf_len;
u16 rsvd;
u64 iw_chnl_op_ctx;
/* Member alignment MUST be maintained above this location */
u8 iw_chnl_buf[1];
};
struct i40iw_virtchnl_resp_buf {
u64 iw_chnl_op_ctx;
u16 iw_chnl_buf_len;
s16 iw_op_ret_code;
/* Member alignment MUST be maintained above this location */
u16 rsvd[2];
u8 iw_chnl_buf[1];
};
enum i40iw_virtchnl_ops {
I40IW_VCHNL_OP_GET_VER = 0,
I40IW_VCHNL_OP_GET_HMC_FCN,
I40IW_VCHNL_OP_ADD_HMC_OBJ_RANGE,
I40IW_VCHNL_OP_DEL_HMC_OBJ_RANGE,
I40IW_VCHNL_OP_GET_STATS
};
#define I40IW_VCHNL_OP_GET_VER_V0 0
#define I40IW_VCHNL_OP_GET_HMC_FCN_V0 0
#define I40IW_VCHNL_OP_ADD_HMC_OBJ_RANGE_V0 0
#define I40IW_VCHNL_OP_DEL_HMC_OBJ_RANGE_V0 0
#define I40IW_VCHNL_OP_GET_STATS_V0 0
#define I40IW_VCHNL_CHNL_VER_V0 0
struct i40iw_dev_hw_stats;
struct i40iw_virtchnl_hmc_obj_range {
u16 obj_type;
u16 rsvd;
u32 start_index;
u32 obj_count;
};
enum i40iw_status_code i40iw_vchnl_recv_pf(struct i40iw_sc_dev *dev,
u32 vf_id,
u8 *msg,
u16 len);
enum i40iw_status_code i40iw_vchnl_recv_vf(struct i40iw_sc_dev *dev,
u32 vf_id,
u8 *msg,
u16 len);
struct i40iw_virtchnl_req {
struct i40iw_sc_dev *dev;
struct i40iw_virtchnl_op_buf *vchnl_msg;
void *parm;
u32 vf_id;
u16 parm_len;
s16 ret_code;
};
#pragma pack(pop)
enum i40iw_status_code i40iw_vchnl_vf_get_ver(struct i40iw_sc_dev *dev,
u32 *vchnl_ver);
enum i40iw_status_code i40iw_vchnl_vf_get_hmc_fcn(struct i40iw_sc_dev *dev,
u16 *hmc_fcn);
enum i40iw_status_code i40iw_vchnl_vf_add_hmc_objs(struct i40iw_sc_dev *dev,
enum i40iw_hmc_rsrc_type rsrc_type,
u32 start_index,
u32 rsrc_count);
enum i40iw_status_code i40iw_vchnl_vf_del_hmc_obj(struct i40iw_sc_dev *dev,
enum i40iw_hmc_rsrc_type rsrc_type,
u32 start_index,
u32 rsrc_count);
enum i40iw_status_code i40iw_vchnl_vf_get_pe_stats(struct i40iw_sc_dev *dev,
struct i40iw_dev_hw_stats *hw_stats);
#endif

2
drivers/infiniband/hw/mlx5/Makefile
View File

@ -1,4 +1,4 @@
obj-$(CONFIG_MLX5_INFINIBAND) += mlx5_ib.o
mlx5_ib-y := main.o cq.o doorbell.o qp.o mem.o srq.o mr.o ah.o mad.o gsi.o
mlx5_ib-y := main.o cq.o doorbell.o qp.o mem.o srq.o mr.o ah.o mad.o gsi.o ib_virt.o
mlx5_ib-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += odp.o

194
drivers/infiniband/hw/mlx5/ib_virt.c 100644
View File

@ -0,0 +1,194 @@
/*
* Copyright (c) 2016, Mellanox Technologies. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include <linux/module.h>
#include <linux/mlx5/vport.h>
#include "mlx5_ib.h"
static inline u32 mlx_to_net_policy(enum port_state_policy mlx_policy)
{
switch (mlx_policy) {
case MLX5_POLICY_DOWN:
return IFLA_VF_LINK_STATE_DISABLE;
case MLX5_POLICY_UP:
return IFLA_VF_LINK_STATE_ENABLE;
case MLX5_POLICY_FOLLOW:
return IFLA_VF_LINK_STATE_AUTO;
default:
return __IFLA_VF_LINK_STATE_MAX;
}
}
int mlx5_ib_get_vf_config(struct ib_device *device, int vf, u8 port,
struct ifla_vf_info *info)
{
struct mlx5_ib_dev *dev = to_mdev(device);
struct mlx5_core_dev *mdev = dev->mdev;
struct mlx5_hca_vport_context *rep;
int err;
rep = kzalloc(sizeof(*rep), GFP_KERNEL);
if (!rep)
return -ENOMEM;
err = mlx5_query_hca_vport_context(mdev, 1, 1, vf + 1, rep);
if (err) {
mlx5_ib_warn(dev, "failed to query port policy for vf %d (%d)\n",
vf, err);
goto free;
}
memset(info, 0, sizeof(*info));
info->linkstate = mlx_to_net_policy(rep->policy);
if (info->linkstate == __IFLA_VF_LINK_STATE_MAX)
err = -EINVAL;
free:
kfree(rep);
return err;
}
static inline enum port_state_policy net_to_mlx_policy(int policy)
{
switch (policy) {
case IFLA_VF_LINK_STATE_DISABLE:
return MLX5_POLICY_DOWN;
case IFLA_VF_LINK_STATE_ENABLE:
return MLX5_POLICY_UP;
case IFLA_VF_LINK_STATE_AUTO:
return MLX5_POLICY_FOLLOW;
default:
return MLX5_POLICY_INVALID;
}
}
int mlx5_ib_set_vf_link_state(struct ib_device *device, int vf,
u8 port, int state)
{
struct mlx5_ib_dev *dev = to_mdev(device);
struct mlx5_core_dev *mdev = dev->mdev;
struct mlx5_hca_vport_context *in;
int err;
in = kzalloc(sizeof(*in), GFP_KERNEL);
if (!in)
return -ENOMEM;
in->policy = net_to_mlx_policy(state);
if (in->policy == MLX5_POLICY_INVALID) {
err = -EINVAL;
goto out;
}
in->field_select = MLX5_HCA_VPORT_SEL_STATE_POLICY;
err = mlx5_core_modify_hca_vport_context(mdev, 1, 1, vf + 1, in);
out:
kfree(in);
return err;
}
int mlx5_ib_get_vf_stats(struct ib_device *device, int vf,
u8 port, struct ifla_vf_stats *stats)
{
int out_sz = MLX5_ST_SZ_BYTES(query_vport_counter_out);
struct mlx5_core_dev *mdev;
struct mlx5_ib_dev *dev;
void *out;
int err;
dev = to_mdev(device);
mdev = dev->mdev;
out = kzalloc(out_sz, GFP_KERNEL);
if (!out)
return -ENOMEM;
err = mlx5_core_query_vport_counter(mdev, true, vf, port, out, out_sz);
if (err)
goto ex;
stats->rx_packets = MLX5_GET64_PR(query_vport_counter_out, out, received_ib_unicast.packets);
stats->tx_packets = MLX5_GET64_PR(query_vport_counter_out, out, transmitted_ib_unicast.packets);
stats->rx_bytes = MLX5_GET64_PR(query_vport_counter_out, out, received_ib_unicast.octets);
stats->tx_bytes = MLX5_GET64_PR(query_vport_counter_out, out, transmitted_ib_unicast.octets);
stats->multicast = MLX5_GET64_PR(query_vport_counter_out, out, received_ib_multicast.packets);
ex:
kfree(out);
return err;
}
static int set_vf_node_guid(struct ib_device *device, int vf, u8 port, u64 guid)
{
struct mlx5_ib_dev *dev = to_mdev(device);
struct mlx5_core_dev *mdev = dev->mdev;
struct mlx5_hca_vport_context *in;
int err;
in = kzalloc(sizeof(*in), GFP_KERNEL);
if (!in)
return -ENOMEM;
in->field_select = MLX5_HCA_VPORT_SEL_NODE_GUID;
in->node_guid = guid;
err = mlx5_core_modify_hca_vport_context(mdev, 1, 1, vf + 1, in);
kfree(in);
return err;
}
static int set_vf_port_guid(struct ib_device *device, int vf, u8 port, u64 guid)
{
struct mlx5_ib_dev *dev = to_mdev(device);
struct mlx5_core_dev *mdev = dev->mdev;
struct mlx5_hca_vport_context *in;
int err;
in = kzalloc(sizeof(*in), GFP_KERNEL);
if (!in)
return -ENOMEM;
in->field_select = MLX5_HCA_VPORT_SEL_PORT_GUID;
in->port_guid = guid;
err = mlx5_core_modify_hca_vport_context(mdev, 1, 1, vf + 1, in);
kfree(in);
return err;
}
int mlx5_ib_set_vf_guid(struct ib_device *device, int vf, u8 port,
u64 guid, int type)
{
if (type == IFLA_VF_IB_NODE_GUID)
return set_vf_node_guid(device, vf, port, guid);
else if (type == IFLA_VF_IB_PORT_GUID)
return set_vf_port_guid(device, vf, port, guid);
return -EINVAL;
}

2
drivers/infiniband/hw/mlx5/mad.c
View File

@ -208,7 +208,7 @@ static int process_pma_cmd(struct ib_device *ibdev, u8 port_num,
if (!out_cnt)
return IB_MAD_RESULT_FAILURE;
err = mlx5_core_query_vport_counter(dev->mdev, 0,
err = mlx5_core_query_vport_counter(dev->mdev, 0, 0,
port_num, out_cnt, sz);
if (!err)
pma_cnt_ext_assign(pma_cnt_ext, out_cnt);

12
drivers/infiniband/hw/mlx5/main.c
View File

@ -284,7 +284,7 @@ __be16 mlx5_get_roce_udp_sport(struct mlx5_ib_dev *dev, u8 port_num,
static int mlx5_use_mad_ifc(struct mlx5_ib_dev *dev)
{
return !dev->mdev->issi;
return !MLX5_CAP_GEN(dev->mdev, ib_virt);
}
enum {
@ -563,6 +563,9 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
if (MLX5_CAP_GEN(mdev, cd))
props->device_cap_flags |= IB_DEVICE_CROSS_CHANNEL;
if (!mlx5_core_is_pf(mdev))
props->device_cap_flags |= IB_DEVICE_VIRTUAL_FUNCTION;
return 0;
}
@ -700,6 +703,7 @@ static int mlx5_query_hca_port(struct ib_device *ibdev, u8 port,
props->qkey_viol_cntr = rep->qkey_violation_counter;
props->subnet_timeout = rep->subnet_timeout;
props->init_type_reply = rep->init_type_reply;
props->grh_required = rep->grh_required;
err = mlx5_query_port_link_width_oper(mdev, &ib_link_width_oper, port);
if (err)
@ -2350,6 +2354,12 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev)
dev->ib_dev.map_mr_sg = mlx5_ib_map_mr_sg;
dev->ib_dev.check_mr_status = mlx5_ib_check_mr_status;
dev->ib_dev.get_port_immutable = mlx5_port_immutable;
if (mlx5_core_is_pf(mdev)) {
dev->ib_dev.get_vf_config = mlx5_ib_get_vf_config;
dev->ib_dev.set_vf_link_state = mlx5_ib_set_vf_link_state;
dev->ib_dev.get_vf_stats = mlx5_ib_get_vf_stats;
dev->ib_dev.set_vf_guid = mlx5_ib_set_vf_guid;
}
mlx5_ib_internal_fill_odp_caps(dev);

8
drivers/infiniband/hw/mlx5/mlx5_ib.h
View File

@ -776,6 +776,14 @@ void mlx5_ib_qp_disable_pagefaults(struct mlx5_ib_qp *qp);
void mlx5_ib_qp_enable_pagefaults(struct mlx5_ib_qp *qp);
void mlx5_ib_invalidate_range(struct ib_umem *umem, unsigned long start,
unsigned long end);
int mlx5_ib_get_vf_config(struct ib_device *device, int vf,
u8 port, struct ifla_vf_info *info);
int mlx5_ib_set_vf_link_state(struct ib_device *device, int vf,
u8 port, int state);
int mlx5_ib_get_vf_stats(struct ib_device *device, int vf,
u8 port, struct ifla_vf_stats *stats);
int mlx5_ib_set_vf_guid(struct ib_device *device, int vf, u8 port,
u64 guid, int type);
#else /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */
static inline void mlx5_ib_internal_fill_odp_caps(struct mlx5_ib_dev *dev)

2
drivers/infiniband/hw/qib/Kconfig
View File

@ -1,6 +1,6 @@
config INFINIBAND_QIB
tristate "Intel PCIe HCA support"
depends on 64BIT
depends on 64BIT && INFINIBAND_RDMAVT
---help---
This is a low-level driver for Intel PCIe QLE InfiniBand host
channel adapters. This driver does not support the Intel

10
drivers/infiniband/hw/qib/Makefile
View File

@ -1,11 +1,11 @@
obj-$(CONFIG_INFINIBAND_QIB) += ib_qib.o
ib_qib-y := qib_cq.o qib_diag.o qib_dma.o qib_driver.o qib_eeprom.o \
qib_file_ops.o qib_fs.o qib_init.o qib_intr.o qib_keys.o \
qib_mad.o qib_mmap.o qib_mr.o qib_pcie.o qib_pio_copy.o \
qib_qp.o qib_qsfp.o qib_rc.o qib_ruc.o qib_sdma.o qib_srq.o \
ib_qib-y := qib_diag.o qib_driver.o qib_eeprom.o \
qib_file_ops.o qib_fs.o qib_init.o qib_intr.o \
qib_mad.o qib_pcie.o qib_pio_copy.o \
qib_qp.o qib_qsfp.o qib_rc.o qib_ruc.o qib_sdma.o \
qib_sysfs.o qib_twsi.o qib_tx.o qib_uc.o qib_ud.o \
qib_user_pages.o qib_user_sdma.o qib_verbs_mcast.o qib_iba7220.o \
qib_user_pages.o qib_user_sdma.o qib_iba7220.o \
qib_sd7220.o qib_iba7322.o qib_verbs.o
# 6120 has no fallback if no MSI interrupts, others can do INTx

33
drivers/infiniband/hw/qib/qib.h
View File

@ -52,6 +52,7 @@
#include <linux/kref.h>
#include <linux/sched.h>
#include <linux/kthread.h>
#include <rdma/rdma_vt.h>
#include "qib_common.h"
#include "qib_verbs.h"
@ -229,9 +230,6 @@ struct qib_ctxtdata {
u8 redirect_seq_cnt;
/* ctxt rcvhdrq head offset */
u32 head;
/* lookaside fields */
struct qib_qp *lookaside_qp;
u32 lookaside_qpn;
/* QPs waiting for context processing */
struct list_head qp_wait_list;
#ifdef CONFIG_DEBUG_FS
@ -240,7 +238,7 @@ struct qib_ctxtdata {
#endif
};
struct qib_sge_state;
struct rvt_sge_state;
struct qib_sdma_txreq {
int flags;
@ -258,14 +256,14 @@ struct qib_sdma_desc {
struct qib_verbs_txreq {
struct qib_sdma_txreq txreq;
struct qib_qp *qp;
struct qib_swqe *wqe;
struct rvt_qp *qp;
struct rvt_swqe *wqe;
u32 dwords;
u16 hdr_dwords;
u16 hdr_inx;
struct qib_pio_header *align_buf;
struct qib_mregion *mr;
struct qib_sge_state *ss;
struct rvt_mregion *mr;
struct rvt_sge_state *ss;
};
#define QIB_SDMA_TXREQ_F_USELARGEBUF 0x1
@ -1096,8 +1094,6 @@ struct qib_devdata {
u16 psxmitwait_check_rate;
/* high volume overflow errors defered to tasklet */
struct tasklet_struct error_tasklet;
/* per device cq worker */
struct kthread_worker *worker;
int assigned_node_id; /* NUMA node closest to HCA */
};
@ -1135,8 +1131,9 @@ extern spinlock_t qib_devs_lock;
extern struct qib_devdata *qib_lookup(int unit);
extern u32 qib_cpulist_count;
extern unsigned long *qib_cpulist;
extern u16 qpt_mask;
extern unsigned qib_cc_table_size;
int qib_init(struct qib_devdata *, int);
int init_chip_wc_pat(struct qib_devdata *dd, u32);
int qib_enable_wc(struct qib_devdata *dd);
@ -1323,7 +1320,7 @@ void __qib_sdma_intr(struct qib_pportdata *);
void qib_sdma_intr(struct qib_pportdata *);
void qib_user_sdma_send_desc(struct qib_pportdata *dd,
struct list_head *pktlist);
int qib_sdma_verbs_send(struct qib_pportdata *, struct qib_sge_state *,
int qib_sdma_verbs_send(struct qib_pportdata *, struct rvt_sge_state *,
u32, struct qib_verbs_txreq *);
/* ppd->sdma_lock should be locked before calling this. */
int qib_sdma_make_progress(struct qib_pportdata *dd);
@ -1454,6 +1451,8 @@ u64 qib_sps_ints(void);
dma_addr_t qib_map_page(struct pci_dev *, struct page *, unsigned long,
size_t, int);
const char *qib_get_unit_name(int unit);
const char *qib_get_card_name(struct rvt_dev_info *rdi);
struct pci_dev *qib_get_pci_dev(struct rvt_dev_info *rdi);
/*
* Flush write combining store buffers (if present) and perform a write
@ -1540,4 +1539,14 @@ struct qib_hwerror_msgs {
void qib_format_hwerrors(u64 hwerrs,
const struct qib_hwerror_msgs *hwerrmsgs,
size_t nhwerrmsgs, char *msg, size_t lmsg);
void qib_stop_send_queue(struct rvt_qp *qp);
void qib_quiesce_qp(struct rvt_qp *qp);
void qib_flush_qp_waiters(struct rvt_qp *qp);
int qib_mtu_to_path_mtu(u32 mtu);
u32 qib_mtu_from_qp(struct rvt_dev_info *rdi, struct rvt_qp *qp, u32 pmtu);
void qib_notify_error_qp(struct rvt_qp *qp);
int qib_get_pmtu_from_attr(struct rvt_dev_info *rdi, struct rvt_qp *qp,
struct ib_qp_attr *attr);
#endif /* _QIB_KERNEL_H */

3
drivers/infiniband/hw/qib/qib_common.h
View File

@ -742,14 +742,11 @@ struct qib_tid_session_member {
#define SIZE_OF_CRC 1
#define QIB_DEFAULT_P_KEY 0xFFFF
#define QIB_PERMISSIVE_LID 0xFFFF
#define QIB_AETH_CREDIT_SHIFT 24
#define QIB_AETH_CREDIT_MASK 0x1F
#define QIB_AETH_CREDIT_INVAL 0x1F
#define QIB_PSN_MASK 0xFFFFFF
#define QIB_MSN_MASK 0xFFFFFF
#define QIB_QPN_MASK 0xFFFFFF
#define QIB_MULTICAST_LID_BASE 0xC000
#define QIB_EAGER_TID_ID QLOGIC_IB_I_TID_MASK
#define QIB_MULTICAST_QPN 0xFFFFFF

545
drivers/infiniband/hw/qib/qib_cq.c
View File

@ -1,545 +0,0 @@
/*
* Copyright (c) 2013 Intel Corporation. All rights reserved.
* Copyright (c) 2006, 2007, 2008, 2010 QLogic Corporation. All rights reserved.
* Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include <linux/err.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>
#include <linux/kthread.h>
#include "qib_verbs.h"
#include "qib.h"
/**
* qib_cq_enter - add a new entry to the completion queue
* @cq: completion queue
* @entry: work completion entry to add
* @sig: true if @entry is a solicitated entry
*
* This may be called with qp->s_lock held.
*/
void qib_cq_enter(struct qib_cq *cq, struct ib_wc *entry, int solicited)
{
struct qib_cq_wc *wc;
unsigned long flags;
u32 head;
u32 next;
spin_lock_irqsave(&cq->lock, flags);
/*
* Note that the head pointer might be writable by user processes.
* Take care to verify it is a sane value.
*/
wc = cq->queue;
head = wc->head;
if (head >= (unsigned) cq->ibcq.cqe) {
head = cq->ibcq.cqe;
next = 0;
} else
next = head + 1;
if (unlikely(next == wc->tail)) {
spin_unlock_irqrestore(&cq->lock, flags);
if (cq->ibcq.event_handler) {
struct ib_event ev;
ev.device = cq->ibcq.device;
ev.element.cq = &cq->ibcq;
ev.event = IB_EVENT_CQ_ERR;
cq->ibcq.event_handler(&ev, cq->ibcq.cq_context);
}
return;
}
if (cq->ip) {
wc->uqueue[head].wr_id = entry->wr_id;
wc->uqueue[head].status = entry->status;
wc->uqueue[head].opcode = entry->opcode;
wc->uqueue[head].vendor_err = entry->vendor_err;
wc->uqueue[head].byte_len = entry->byte_len;
wc->uqueue[head].ex.imm_data =
(__u32 __force)entry->ex.imm_data;
wc->uqueue[head].qp_num = entry->qp->qp_num;
wc->uqueue[head].src_qp = entry->src_qp;
wc->uqueue[head].wc_flags = entry->wc_flags;
wc->uqueue[head].pkey_index = entry->pkey_index;
wc->uqueue[head].slid = entry->slid;
wc->uqueue[head].sl = entry->sl;
wc->uqueue[head].dlid_path_bits = entry->dlid_path_bits;
wc->uqueue[head].port_num = entry->port_num;
/* Make sure entry is written before the head index. */
smp_wmb();
} else
wc->kqueue[head] = *entry;
wc->head = next;
if (cq->notify == IB_CQ_NEXT_COMP ||
(cq->notify == IB_CQ_SOLICITED &&
(solicited || entry->status != IB_WC_SUCCESS))) {
struct kthread_worker *worker;
/*
* This will cause send_complete() to be called in
* another thread.
*/
smp_rmb();
worker = cq->dd->worker;
if (likely(worker)) {
cq->notify = IB_CQ_NONE;
cq->triggered++;
queue_kthread_work(worker, &cq->comptask);
}
}
spin_unlock_irqrestore(&cq->lock, flags);
}
/**
* qib_poll_cq - poll for work completion entries
* @ibcq: the completion queue to poll
* @num_entries: the maximum number of entries to return
* @entry: pointer to array where work completions are placed
*
* Returns the number of completion entries polled.
*
* This may be called from interrupt context. Also called by ib_poll_cq()
* in the generic verbs code.
*/
int qib_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *entry)
{
struct qib_cq *cq = to_icq(ibcq);
struct qib_cq_wc *wc;
unsigned long flags;
int npolled;
u32 tail;
/* The kernel can only poll a kernel completion queue */
if (cq->ip) {
npolled = -EINVAL;
goto bail;
}
spin_lock_irqsave(&cq->lock, flags);
wc = cq->queue;
tail = wc->tail;
if (tail > (u32) cq->ibcq.cqe)
tail = (u32) cq->ibcq.cqe;
for (npolled = 0; npolled < num_entries; ++npolled, ++entry) {
if (tail == wc->head)
break;
/* The kernel doesn't need a RMB since it has the lock. */
*entry = wc->kqueue[tail];
if (tail >= cq->ibcq.cqe)
tail = 0;
else
tail++;
}
wc->tail = tail;
spin_unlock_irqrestore(&cq->lock, flags);
bail:
return npolled;
}
static void send_complete(struct kthread_work *work)
{
struct qib_cq *cq = container_of(work, struct qib_cq, comptask);
/*
* The completion handler will most likely rearm the notification
* and poll for all pending entries. If a new completion entry
* is added while we are in this routine, queue_work()
* won't call us again until we return so we check triggered to
* see if we need to call the handler again.
*/
for (;;) {
u8 triggered = cq->triggered;
/*
* IPoIB connected mode assumes the callback is from a
* soft IRQ. We simulate this by blocking "bottom halves".
* See the implementation for ipoib_cm_handle_tx_wc(),
* netif_tx_lock_bh() and netif_tx_lock().
*/
local_bh_disable();
cq->ibcq.comp_handler(&cq->ibcq, cq->ibcq.cq_context);
local_bh_enable();
if (cq->triggered == triggered)
return;
}
}
/**
* qib_create_cq - create a completion queue
* @ibdev: the device this completion queue is attached to
* @attr: creation attributes
* @context: unused by the QLogic_IB driver
* @udata: user data for libibverbs.so
*
* Returns a pointer to the completion queue or negative errno values
* for failure.
*
* Called by ib_create_cq() in the generic verbs code.
*/
struct ib_cq *qib_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
struct ib_ucontext *context,
struct ib_udata *udata)
{
int entries = attr->cqe;
struct qib_ibdev *dev = to_idev(ibdev);
struct qib_cq *cq;
struct qib_cq_wc *wc;
struct ib_cq *ret;
u32 sz;
if (attr->flags)
return ERR_PTR(-EINVAL);
if (entries < 1 || entries > ib_qib_max_cqes) {
ret = ERR_PTR(-EINVAL);
goto done;
}
/* Allocate the completion queue structure. */
cq = kmalloc(sizeof(*cq), GFP_KERNEL);
if (!cq) {
ret = ERR_PTR(-ENOMEM);
goto done;
}
/*
* Allocate the completion queue entries and head/tail pointers.
* This is allocated separately so that it can be resized and
* also mapped into user space.
* We need to use vmalloc() in order to support mmap and large
* numbers of entries.
*/
sz = sizeof(*wc);
if (udata && udata->outlen >= sizeof(__u64))
sz += sizeof(struct ib_uverbs_wc) * (entries + 1);
else
sz += sizeof(struct ib_wc) * (entries + 1);
wc = vmalloc_user(sz);
if (!wc) {
ret = ERR_PTR(-ENOMEM);
goto bail_cq;
}
/*
* Return the address of the WC as the offset to mmap.
* See qib_mmap() for details.
*/
if (udata && udata->outlen >= sizeof(__u64)) {
int err;
cq->ip = qib_create_mmap_info(dev, sz, context, wc);
if (!cq->ip) {
ret = ERR_PTR(-ENOMEM);
goto bail_wc;
}
err = ib_copy_to_udata(udata, &cq->ip->offset,
sizeof(cq->ip->offset));
if (err) {
ret = ERR_PTR(err);
goto bail_ip;
}
} else
cq->ip = NULL;
spin_lock(&dev->n_cqs_lock);
if (dev->n_cqs_allocated == ib_qib_max_cqs) {
spin_unlock(&dev->n_cqs_lock);
ret = ERR_PTR(-ENOMEM);
goto bail_ip;
}
dev->n_cqs_allocated++;
spin_unlock(&dev->n_cqs_lock);
if (cq->ip) {
spin_lock_irq(&dev->pending_lock);
list_add(&cq->ip->pending_mmaps, &dev->pending_mmaps);
spin_unlock_irq(&dev->pending_lock);
}
/*
* ib_create_cq() will initialize cq->ibcq except for cq->ibcq.cqe.
* The number of entries should be >= the number requested or return
* an error.
*/
cq->dd = dd_from_dev(dev);
cq->ibcq.cqe = entries;
cq->notify = IB_CQ_NONE;
cq->triggered = 0;
spin_lock_init(&cq->lock);
init_kthread_work(&cq->comptask, send_complete);
wc->head = 0;
wc->tail = 0;
cq->queue = wc;
ret = &cq->ibcq;
goto done;
bail_ip:
kfree(cq->ip);
bail_wc:
vfree(wc);
bail_cq:
kfree(cq);
done:
return ret;
}
/**
* qib_destroy_cq - destroy a completion queue
* @ibcq: the completion queue to destroy.
*
* Returns 0 for success.
*
* Called by ib_destroy_cq() in the generic verbs code.
*/
int qib_destroy_cq(struct ib_cq *ibcq)
{
struct qib_ibdev *dev = to_idev(ibcq->device);
struct qib_cq *cq = to_icq(ibcq);
flush_kthread_work(&cq->comptask);
spin_lock(&dev->n_cqs_lock);
dev->n_cqs_allocated--;
spin_unlock(&dev->n_cqs_lock);
if (cq->ip)
kref_put(&cq->ip->ref, qib_release_mmap_info);
else
vfree(cq->queue);
kfree(cq);
return 0;
}
/**
* qib_req_notify_cq - change the notification type for a completion queue
* @ibcq: the completion queue
* @notify_flags: the type of notification to request
*
* Returns 0 for success.
*
* This may be called from interrupt context. Also called by
* ib_req_notify_cq() in the generic verbs code.
*/
int qib_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags notify_flags)
{
struct qib_cq *cq = to_icq(ibcq);
unsigned long flags;
int ret = 0;
spin_lock_irqsave(&cq->lock, flags);
/*
* Don't change IB_CQ_NEXT_COMP to IB_CQ_SOLICITED but allow
* any other transitions (see C11-31 and C11-32 in ch. 11.4.2.2).
*/
if (cq->notify != IB_CQ_NEXT_COMP)
cq->notify = notify_flags & IB_CQ_SOLICITED_MASK;
if ((notify_flags & IB_CQ_REPORT_MISSED_EVENTS) &&
cq->queue->head != cq->queue->tail)
ret = 1;
spin_unlock_irqrestore(&cq->lock, flags);
return ret;
}
/**
* qib_resize_cq - change the size of the CQ
* @ibcq: the completion queue
*
* Returns 0 for success.
*/
int qib_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata)
{
struct qib_cq *cq = to_icq(ibcq);
struct qib_cq_wc *old_wc;
struct qib_cq_wc *wc;
u32 head, tail, n;
int ret;
u32 sz;
if (cqe < 1 || cqe > ib_qib_max_cqes) {
ret = -EINVAL;
goto bail;
}
/*
* Need to use vmalloc() if we want to support large #s of entries.
*/
sz = sizeof(*wc);
if (udata && udata->outlen >= sizeof(__u64))
sz += sizeof(struct ib_uverbs_wc) * (cqe + 1);
else
sz += sizeof(struct ib_wc) * (cqe + 1);
wc = vmalloc_user(sz);
if (!wc) {
ret = -ENOMEM;
goto bail;
}
/* Check that we can write the offset to mmap. */
if (udata && udata->outlen >= sizeof(__u64)) {
__u64 offset = 0;
ret = ib_copy_to_udata(udata, &offset, sizeof(offset));
if (ret)
goto bail_free;
}
spin_lock_irq(&cq->lock);
/*
* Make sure head and tail are sane since they
* might be user writable.
*/
old_wc = cq->queue;
head = old_wc->head;
if (head > (u32) cq->ibcq.cqe)
head = (u32) cq->ibcq.cqe;
tail = old_wc->tail;
if (tail > (u32) cq->ibcq.cqe)
tail = (u32) cq->ibcq.cqe;
if (head < tail)
n = cq->ibcq.cqe + 1 + head - tail;
else
n = head - tail;
if (unlikely((u32)cqe < n)) {
ret = -EINVAL;
goto bail_unlock;
}
for (n = 0; tail != head; n++) {
if (cq->ip)
wc->uqueue[n] = old_wc->uqueue[tail];
else
wc->kqueue[n] = old_wc->kqueue[tail];
if (tail == (u32) cq->ibcq.cqe)
tail = 0;
else
tail++;
}
cq->ibcq.cqe = cqe;
wc->head = n;
wc->tail = 0;
cq->queue = wc;
spin_unlock_irq(&cq->lock);
vfree(old_wc);
if (cq->ip) {
struct qib_ibdev *dev = to_idev(ibcq->device);
struct qib_mmap_info *ip = cq->ip;
qib_update_mmap_info(dev, ip, sz, wc);
/*
* Return the offset to mmap.
* See qib_mmap() for details.
*/
if (udata && udata->outlen >= sizeof(__u64)) {
ret = ib_copy_to_udata(udata, &ip->offset,
sizeof(ip->offset));
if (ret)
goto bail;
}
spin_lock_irq(&dev->pending_lock);
if (list_empty(&ip->pending_mmaps))
list_add(&ip->pending_mmaps, &dev->pending_mmaps);
spin_unlock_irq(&dev->pending_lock);
}
ret = 0;
goto bail;
bail_unlock:
spin_unlock_irq(&cq->lock);
bail_free:
vfree(wc);
bail:
return ret;
}
int qib_cq_init(struct qib_devdata *dd)
{
int ret = 0;
int cpu;
struct task_struct *task;
if (dd->worker)
return 0;
dd->worker = kzalloc(sizeof(*dd->worker), GFP_KERNEL);
if (!dd->worker)
return -ENOMEM;
init_kthread_worker(dd->worker);
task = kthread_create_on_node(
kthread_worker_fn,
dd->worker,
dd->assigned_node_id,
"qib_cq%d", dd->unit);
if (IS_ERR(task))
goto task_fail;
cpu = cpumask_first(cpumask_of_node(dd->assigned_node_id));
kthread_bind(task, cpu);
wake_up_process(task);
out:
return ret;
task_fail:
ret = PTR_ERR(task);
kfree(dd->worker);
dd->worker = NULL;
goto out;
}
void qib_cq_exit(struct qib_devdata *dd)
{
struct kthread_worker *worker;
worker = dd->worker;
if (!worker)
return;
/* blocks future queuing from send_complete() */
dd->worker = NULL;
smp_wmb();
flush_kthread_worker(worker);
kthread_stop(worker->task);
kfree(worker);
}

71
drivers/infiniband/hw/qib/qib_driver.c
View File

@ -90,6 +90,22 @@ const char *qib_get_unit_name(int unit)
return iname;
}
const char *qib_get_card_name(struct rvt_dev_info *rdi)
{
struct qib_ibdev *ibdev = container_of(rdi, struct qib_ibdev, rdi);
struct qib_devdata *dd = container_of(ibdev,
struct qib_devdata, verbs_dev);
return qib_get_unit_name(dd->unit);
}
struct pci_dev *qib_get_pci_dev(struct rvt_dev_info *rdi)
{
struct qib_ibdev *ibdev = container_of(rdi, struct qib_ibdev, rdi);
struct qib_devdata *dd = container_of(ibdev,
struct qib_devdata, verbs_dev);
return dd->pcidev;
}
/*
* Return count of units with at least one port ACTIVE.
*/
@ -306,7 +322,9 @@ static u32 qib_rcv_hdrerr(struct qib_ctxtdata *rcd, struct qib_pportdata *ppd,
struct qib_ib_header *hdr = (struct qib_ib_header *) rhdr;
struct qib_other_headers *ohdr = NULL;
struct qib_ibport *ibp = &ppd->ibport_data;
struct qib_qp *qp = NULL;
struct qib_devdata *dd = ppd->dd;
struct rvt_dev_info *rdi = &dd->verbs_dev.rdi;
struct rvt_qp *qp = NULL;
u32 tlen = qib_hdrget_length_in_bytes(rhf_addr);
u16 lid = be16_to_cpu(hdr->lrh[1]);
int lnh = be16_to_cpu(hdr->lrh[0]) & 3;
@ -319,7 +337,7 @@ static u32 qib_rcv_hdrerr(struct qib_ctxtdata *rcd, struct qib_pportdata *ppd,
if (tlen < 24)
goto drop;
if (lid < QIB_MULTICAST_LID_BASE) {
if (lid < be16_to_cpu(IB_MULTICAST_LID_BASE)) {
lid &= ~((1 << ppd->lmc) - 1);
if (unlikely(lid != ppd->lid))
goto drop;
@ -346,13 +364,16 @@ static u32 qib_rcv_hdrerr(struct qib_ctxtdata *rcd, struct qib_pportdata *ppd,
psn = be32_to_cpu(ohdr->bth[2]);
/* Get the destination QP number. */
qp_num = be32_to_cpu(ohdr->bth[1]) & QIB_QPN_MASK;
qp_num = be32_to_cpu(ohdr->bth[1]) & RVT_QPN_MASK;
if (qp_num != QIB_MULTICAST_QPN) {
int ruc_res;
qp = qib_lookup_qpn(ibp, qp_num);
if (!qp)
rcu_read_lock();
qp = rvt_lookup_qpn(rdi, &ibp->rvp, qp_num);
if (!qp) {
rcu_read_unlock();
goto drop;
}
/*
* Handle only RC QPs - for other QP types drop error
@ -361,9 +382,9 @@ static u32 qib_rcv_hdrerr(struct qib_ctxtdata *rcd, struct qib_pportdata *ppd,
spin_lock(&qp->r_lock);
/* Check for valid receive state. */
if (!(ib_qib_state_ops[qp->state] &
QIB_PROCESS_RECV_OK)) {
ibp->n_pkt_drops++;
if (!(ib_rvt_state_ops[qp->state] &
RVT_PROCESS_RECV_OK)) {
ibp->rvp.n_pkt_drops++;
goto unlock;
}
@ -383,7 +404,7 @@ static u32 qib_rcv_hdrerr(struct qib_ctxtdata *rcd, struct qib_pportdata *ppd,
IB_OPCODE_RC_RDMA_READ_RESPONSE_FIRST) {
diff = qib_cmp24(psn, qp->r_psn);
if (!qp->r_nak_state && diff >= 0) {
ibp->n_rc_seqnak++;
ibp->rvp.n_rc_seqnak++;
qp->r_nak_state =
IB_NAK_PSN_ERROR;
/* Use the expected PSN. */
@ -398,7 +419,7 @@ static u32 qib_rcv_hdrerr(struct qib_ctxtdata *rcd, struct qib_pportdata *ppd,
*/
if (list_empty(&qp->rspwait)) {
qp->r_flags |=
QIB_R_RSP_NAK;
RVT_R_RSP_NAK;
atomic_inc(
&qp->refcount);
list_add_tail(
@ -419,12 +440,7 @@ static u32 qib_rcv_hdrerr(struct qib_ctxtdata *rcd, struct qib_pportdata *ppd,
unlock:
spin_unlock(&qp->r_lock);
/*
* Notify qib_destroy_qp() if it is waiting
* for us to finish.
*/
if (atomic_dec_and_test(&qp->refcount))
wake_up(&qp->wait);
rcu_read_unlock();
} /* Unicast QP */
} /* Valid packet with TIDErr */
@ -456,7 +472,7 @@ u32 qib_kreceive(struct qib_ctxtdata *rcd, u32 *llic, u32 *npkts)
u32 eflags, etype, tlen, i = 0, updegr = 0, crcs = 0;
int last;
u64 lval;
struct qib_qp *qp, *nqp;
struct rvt_qp *qp, *nqp;
l = rcd->head;
rhf_addr = (__le32 *) rcd->rcvhdrq + l + dd->rhf_offset;
@ -549,15 +565,6 @@ move_along:
updegr = 0;
}
}
/*
* Notify qib_destroy_qp() if it is waiting
* for lookaside_qp to finish.
*/
if (rcd->lookaside_qp) {
if (atomic_dec_and_test(&rcd->lookaside_qp->refcount))
wake_up(&rcd->lookaside_qp->wait);
rcd->lookaside_qp = NULL;
}
rcd->head = l;
@ -567,17 +574,17 @@ move_along:
*/
list_for_each_entry_safe(qp, nqp, &rcd->qp_wait_list, rspwait) {
list_del_init(&qp->rspwait);
if (qp->r_flags & QIB_R_RSP_NAK) {
qp->r_flags &= ~QIB_R_RSP_NAK;
if (qp->r_flags & RVT_R_RSP_NAK) {
qp->r_flags &= ~RVT_R_RSP_NAK;
qib_send_rc_ack(qp);
}
if (qp->r_flags & QIB_R_RSP_SEND) {
if (qp->r_flags & RVT_R_RSP_SEND) {
unsigned long flags;
qp->r_flags &= ~QIB_R_RSP_SEND;
qp->r_flags &= ~RVT_R_RSP_SEND;
spin_lock_irqsave(&qp->s_lock, flags);
if (ib_qib_state_ops[qp->state] &
QIB_PROCESS_OR_FLUSH_SEND)
if (ib_rvt_state_ops[qp->state] &
RVT_PROCESS_OR_FLUSH_SEND)
qib_schedule_send(qp);
spin_unlock_irqrestore(&qp->s_lock, flags);
}

8
drivers/infiniband/hw/qib/qib_iba6120.c
View File

@ -2956,13 +2956,13 @@ static void pma_6120_timer(unsigned long data)
struct qib_ibport *ibp = &ppd->ibport_data;
unsigned long flags;
spin_lock_irqsave(&ibp->lock, flags);
spin_lock_irqsave(&ibp->rvp.lock, flags);
if (cs->pma_sample_status == IB_PMA_SAMPLE_STATUS_STARTED) {
cs->pma_sample_status = IB_PMA_SAMPLE_STATUS_RUNNING;
qib_snapshot_counters(ppd, &cs->sword, &cs->rword,
&cs->spkts, &cs->rpkts, &cs->xmit_wait);
mod_timer(&cs->pma_timer,
jiffies + usecs_to_jiffies(ibp->pma_sample_interval));
jiffies + usecs_to_jiffies(ibp->rvp.pma_sample_interval));
} else if (cs->pma_sample_status == IB_PMA_SAMPLE_STATUS_RUNNING) {
u64 ta, tb, tc, td, te;
@ -2975,11 +2975,11 @@ static void pma_6120_timer(unsigned long data)
cs->rpkts = td - cs->rpkts;
cs->xmit_wait = te - cs->xmit_wait;
}
spin_unlock_irqrestore(&ibp->lock, flags);
spin_unlock_irqrestore(&ibp->rvp.lock, flags);
}
/*
* Note that the caller has the ibp->lock held.
* Note that the caller has the ibp->rvp.lock held.
*/
static void qib_set_cntr_6120_sample(struct qib_pportdata *ppd, u32 intv,
u32 start)

6
drivers/infiniband/hw/qib/qib_iba7322.c
View File

@ -2910,8 +2910,6 @@ static void qib_setup_7322_cleanup(struct qib_devdata *dd)
spin_unlock_irqrestore(&dd->cspec->gpio_lock, flags);
qib_qsfp_deinit(&dd->pport[i].cpspec->qsfp_data);
}
if (dd->pport[i].ibport_data.smi_ah)
ib_destroy_ah(&dd->pport[i].ibport_data.smi_ah->ibah);
}
}
@ -5497,7 +5495,7 @@ static void try_7322_ipg(struct qib_pportdata *ppd)
unsigned delay;
int ret;
agent = ibp->send_agent;
agent = ibp->rvp.send_agent;
if (!agent)
goto retry;
@ -5515,7 +5513,7 @@ static void try_7322_ipg(struct qib_pportdata *ppd)
ret = PTR_ERR(ah);
else {
send_buf->ah = ah;
ibp->smi_ah = to_iah(ah);
ibp->smi_ah = ibah_to_rvtah(ah);
ret = 0;
}
} else {

25
drivers/infiniband/hw/qib/qib_init.c
View File

@ -42,6 +42,7 @@
#ifdef CONFIG_INFINIBAND_QIB_DCA
#include <linux/dca.h>
#endif
#include <rdma/rdma_vt.h>
#include "qib.h"
#include "qib_common.h"
@ -244,6 +245,13 @@ int qib_init_pportdata(struct qib_pportdata *ppd, struct qib_devdata *dd,
alloc_percpu(struct qib_pma_counters);
if (!ppd->ibport_data.pmastats)
return -ENOMEM;
ppd->ibport_data.rvp.rc_acks = alloc_percpu(u64);
ppd->ibport_data.rvp.rc_qacks = alloc_percpu(u64);
ppd->ibport_data.rvp.rc_delayed_comp = alloc_percpu(u64);
if (!(ppd->ibport_data.rvp.rc_acks) ||
!(ppd->ibport_data.rvp.rc_qacks) ||
!(ppd->ibport_data.rvp.rc_delayed_comp))
return -ENOMEM;
if (qib_cc_table_size < IB_CCT_MIN_ENTRIES)
goto bail;
@ -449,8 +457,6 @@ static int loadtime_init(struct qib_devdata *dd)
init_timer(&dd->intrchk_timer);
dd->intrchk_timer.function = verify_interrupt;
dd->intrchk_timer.data = (unsigned long) dd;
ret = qib_cq_init(dd);
done:
return ret;
}
@ -631,6 +637,9 @@ wq_error:
static void qib_free_pportdata(struct qib_pportdata *ppd)
{
free_percpu(ppd->ibport_data.pmastats);
free_percpu(ppd->ibport_data.rvp.rc_acks);
free_percpu(ppd->ibport_data.rvp.rc_qacks);
free_percpu(ppd->ibport_data.rvp.rc_delayed_comp);
ppd->ibport_data.pmastats = NULL;
}
@ -1081,7 +1090,7 @@ void qib_free_devdata(struct qib_devdata *dd)
qib_dbg_ibdev_exit(&dd->verbs_dev);
#endif
free_percpu(dd->int_counter);
ib_dealloc_device(&dd->verbs_dev.ibdev);
ib_dealloc_device(&dd->verbs_dev.rdi.ibdev);
}
u64 qib_int_counter(struct qib_devdata *dd)
@ -1120,9 +1129,12 @@ struct qib_devdata *qib_alloc_devdata(struct pci_dev *pdev, size_t extra)
{
unsigned long flags;
struct qib_devdata *dd;
int ret;
int ret, nports;
dd = (struct qib_devdata *) ib_alloc_device(sizeof(*dd) + extra);
/* extra is * number of ports */
nports = extra / sizeof(struct qib_pportdata);
dd = (struct qib_devdata *)rvt_alloc_device(sizeof(*dd) + extra,
nports);
if (!dd)
return ERR_PTR(-ENOMEM);
@ -1171,7 +1183,7 @@ struct qib_devdata *qib_alloc_devdata(struct pci_dev *pdev, size_t extra)
bail:
if (!list_empty(&dd->list))
list_del_init(&dd->list);
ib_dealloc_device(&dd->verbs_dev.ibdev);
ib_dealloc_device(&dd->verbs_dev.rdi.ibdev);
return ERR_PTR(ret);
}
@ -1421,7 +1433,6 @@ static void cleanup_device_data(struct qib_devdata *dd)
}
kfree(tmp);
kfree(dd->boardname);
qib_cq_exit(dd);
}
/*

2
drivers/infiniband/hw/qib/qib_intr.c
View File

@ -74,7 +74,7 @@ static void signal_ib_event(struct qib_pportdata *ppd, enum ib_event_type ev)
struct ib_event event;
struct qib_devdata *dd = ppd->dd;
event.device = &dd->verbs_dev.ibdev;
event.device = &dd->verbs_dev.rdi.ibdev;
event.element.port_num = ppd->port;
event.event = ev;
ib_dispatch_event(&event);

186
drivers/infiniband/hw/qib/qib_keys.c
View File

@ -46,20 +46,20 @@
*
*/
int qib_alloc_lkey(struct qib_mregion *mr, int dma_region)
int qib_alloc_lkey(struct rvt_mregion *mr, int dma_region)
{
unsigned long flags;
u32 r;
u32 n;
int ret = 0;
struct qib_ibdev *dev = to_idev(mr->pd->device);
struct qib_lkey_table *rkt = &dev->lk_table;
struct rvt_lkey_table *rkt = &dev->lk_table;
spin_lock_irqsave(&rkt->lock, flags);
/* special case for dma_mr lkey == 0 */
if (dma_region) {
struct qib_mregion *tmr;
struct rvt_mregion *tmr;
tmr = rcu_access_pointer(dev->dma_mr);
if (!tmr) {
@ -90,8 +90,8 @@ int qib_alloc_lkey(struct qib_mregion *mr, int dma_region)
* bits are capped in qib_verbs.c to insure enough bits
* for generation number
*/
mr->lkey = (r << (32 - ib_qib_lkey_table_size)) |
((((1 << (24 - ib_qib_lkey_table_size)) - 1) & rkt->gen)
mr->lkey = (r << (32 - ib_rvt_lkey_table_size)) |
((((1 << (24 - ib_rvt_lkey_table_size)) - 1) & rkt->gen)
<< 8);
if (mr->lkey == 0) {
mr->lkey |= 1 << 8;
@ -114,13 +114,13 @@ bail:
* qib_free_lkey - free an lkey
* @mr: mr to free from tables
*/
void qib_free_lkey(struct qib_mregion *mr)
void qib_free_lkey(struct rvt_mregion *mr)
{
unsigned long flags;
u32 lkey = mr->lkey;
u32 r;
struct qib_ibdev *dev = to_idev(mr->pd->device);
struct qib_lkey_table *rkt = &dev->lk_table;
struct rvt_lkey_table *rkt = &dev->lk_table;
spin_lock_irqsave(&rkt->lock, flags);
if (!mr->lkey_published)
@ -128,7 +128,7 @@ void qib_free_lkey(struct qib_mregion *mr)
if (lkey == 0)
RCU_INIT_POINTER(dev->dma_mr, NULL);
else {
r = lkey >> (32 - ib_qib_lkey_table_size);
r = lkey >> (32 - ib_rvt_lkey_table_size);
RCU_INIT_POINTER(rkt->table[r], NULL);
}
qib_put_mr(mr);
@ -137,105 +137,6 @@ out:
spin_unlock_irqrestore(&rkt->lock, flags);
}
/**
* qib_lkey_ok - check IB SGE for validity and initialize
* @rkt: table containing lkey to check SGE against
* @pd: protection domain
* @isge: outgoing internal SGE
* @sge: SGE to check
* @acc: access flags
*
* Return 1 if valid and successful, otherwise returns 0.
*
* increments the reference count upon success
*
* Check the IB SGE for validity and initialize our internal version
* of it.
*/
int qib_lkey_ok(struct qib_lkey_table *rkt, struct qib_pd *pd,
struct qib_sge *isge, struct ib_sge *sge, int acc)
{
struct qib_mregion *mr;
unsigned n, m;
size_t off;
/*
* We use LKEY == zero for kernel virtual addresses
* (see qib_get_dma_mr and qib_dma.c).
*/
rcu_read_lock();
if (sge->lkey == 0) {
struct qib_ibdev *dev = to_idev(pd->ibpd.device);
if (pd->user)
goto bail;
mr = rcu_dereference(dev->dma_mr);
if (!mr)
goto bail;
if (unlikely(!atomic_inc_not_zero(&mr->refcount)))
goto bail;
rcu_read_unlock();
isge->mr = mr;
isge->vaddr = (void *) sge->addr;
isge->length = sge->length;
isge->sge_length = sge->length;
isge->m = 0;
isge->n = 0;
goto ok;
}
mr = rcu_dereference(
rkt->table[(sge->lkey >> (32 - ib_qib_lkey_table_size))]);
if (unlikely(!mr || mr->lkey != sge->lkey || mr->pd != &pd->ibpd))
goto bail;
off = sge->addr - mr->user_base;
if (unlikely(sge->addr < mr->user_base ||
off + sge->length > mr->length ||
(mr->access_flags & acc) != acc))
goto bail;
if (unlikely(!atomic_inc_not_zero(&mr->refcount)))
goto bail;
rcu_read_unlock();
off += mr->offset;
if (mr->page_shift) {
/*
page sizes are uniform power of 2 so no loop is necessary
entries_spanned_by_off is the number of times the loop below
would have executed.
*/
size_t entries_spanned_by_off;
entries_spanned_by_off = off >> mr->page_shift;
off -= (entries_spanned_by_off << mr->page_shift);
m = entries_spanned_by_off/QIB_SEGSZ;
n = entries_spanned_by_off%QIB_SEGSZ;
} else {
m = 0;
n = 0;
while (off >= mr->map[m]->segs[n].length) {
off -= mr->map[m]->segs[n].length;
n++;
if (n >= QIB_SEGSZ) {
m++;
n = 0;
}
}
}
isge->mr = mr;
isge->vaddr = mr->map[m]->segs[n].vaddr + off;
isge->length = mr->map[m]->segs[n].length - off;
isge->sge_length = sge->length;
isge->m = m;
isge->n = n;
ok:
return 1;
bail:
rcu_read_unlock();
return 0;
}
/**
* qib_rkey_ok - check the IB virtual address, length, and RKEY
* @qp: qp for validation
@ -249,11 +150,11 @@ bail:
*
* increments the reference count upon success
*/
int qib_rkey_ok(struct qib_qp *qp, struct qib_sge *sge,
int qib_rkey_ok(struct rvt_qp *qp, struct rvt_sge *sge,
u32 len, u64 vaddr, u32 rkey, int acc)
{
struct qib_lkey_table *rkt = &to_idev(qp->ibqp.device)->lk_table;
struct qib_mregion *mr;
struct rvt_lkey_table *rkt = &to_idev(qp->ibqp.device)->lk_table;
struct rvt_mregion *mr;
unsigned n, m;
size_t off;
@ -263,7 +164,7 @@ int qib_rkey_ok(struct qib_qp *qp, struct qib_sge *sge,
*/
rcu_read_lock();
if (rkey == 0) {
struct qib_pd *pd = to_ipd(qp->ibqp.pd);
struct rvt_pd *pd = ibpd_to_rvtpd(qp->ibqp.pd);
struct qib_ibdev *dev = to_idev(pd->ibpd.device);
if (pd->user)
@ -285,7 +186,7 @@ int qib_rkey_ok(struct qib_qp *qp, struct qib_sge *sge,
}
mr = rcu_dereference(
rkt->table[(rkey >> (32 - ib_qib_lkey_table_size))]);
rkt->table[(rkey >> (32 - ib_rvt_lkey_table_size))]);
if (unlikely(!mr || mr->lkey != rkey || qp->ibqp.pd != mr->pd))
goto bail;
@ -308,15 +209,15 @@ int qib_rkey_ok(struct qib_qp *qp, struct qib_sge *sge,
entries_spanned_by_off = off >> mr->page_shift;
off -= (entries_spanned_by_off << mr->page_shift);
m = entries_spanned_by_off/QIB_SEGSZ;
n = entries_spanned_by_off%QIB_SEGSZ;
m = entries_spanned_by_off / RVT_SEGSZ;
n = entries_spanned_by_off % RVT_SEGSZ;
} else {
m = 0;
n = 0;
while (off >= mr->map[m]->segs[n].length) {
off -= mr->map[m]->segs[n].length;
n++;
if (n >= QIB_SEGSZ) {
if (n >= RVT_SEGSZ) {
m++;
n = 0;
}
@ -335,58 +236,3 @@ bail:
return 0;
}
/*
* Initialize the memory region specified by the work request.
*/
int qib_reg_mr(struct qib_qp *qp, struct ib_reg_wr *wr)
{
struct qib_lkey_table *rkt = &to_idev(qp->ibqp.device)->lk_table;
struct qib_pd *pd = to_ipd(qp->ibqp.pd);
struct qib_mr *mr = to_imr(wr->mr);
struct qib_mregion *mrg;
u32 key = wr->key;
unsigned i, n, m;
int ret = -EINVAL;
unsigned long flags;
u64 *page_list;
size_t ps;
spin_lock_irqsave(&rkt->lock, flags);
if (pd->user || key == 0)
goto bail;
mrg = rcu_dereference_protected(
rkt->table[(key >> (32 - ib_qib_lkey_table_size))],
lockdep_is_held(&rkt->lock));
if (unlikely(mrg == NULL || qp->ibqp.pd != mrg->pd))
goto bail;
if (mr->npages > mrg->max_segs)
goto bail;
ps = mr->ibmr.page_size;
if (mr->ibmr.length > ps * mr->npages)
goto bail;
mrg->user_base = mr->ibmr.iova;
mrg->iova = mr->ibmr.iova;
mrg->lkey = key;
mrg->length = mr->ibmr.length;
mrg->access_flags = wr->access;
page_list = mr->pages;
m = 0;
n = 0;
for (i = 0; i < mr->npages; i++) {
mrg->map[m]->segs[n].vaddr = (void *) page_list[i];
mrg->map[m]->segs[n].length = ps;
if (++n == QIB_SEGSZ) {
m++;
n = 0;
}
}
ret = 0;
bail:
spin_unlock_irqrestore(&rkt->lock, flags);
return ret;
}

336
drivers/infiniband/hw/qib/qib_mad.c
View File

@ -70,7 +70,7 @@ static void qib_send_trap(struct qib_ibport *ibp, void *data, unsigned len)
unsigned long flags;
unsigned long timeout;
agent = ibp->send_agent;
agent = ibp->rvp.send_agent;
if (!agent)
return;
@ -79,7 +79,8 @@ static void qib_send_trap(struct qib_ibport *ibp, void *data, unsigned len)
return;
/* o14-2 */
if (ibp->trap_timeout && time_before(jiffies, ibp->trap_timeout))
if (ibp->rvp.trap_timeout &&
time_before(jiffies, ibp->rvp.trap_timeout))
return;
send_buf = ib_create_send_mad(agent, 0, 0, 0, IB_MGMT_MAD_HDR,
@ -93,42 +94,42 @@ static void qib_send_trap(struct qib_ibport *ibp, void *data, unsigned len)
smp->mgmt_class = IB_MGMT_CLASS_SUBN_LID_ROUTED;
smp->class_version = 1;
smp->method = IB_MGMT_METHOD_TRAP;
ibp->tid++;
smp->tid = cpu_to_be64(ibp->tid);
ibp->rvp.tid++;
smp->tid = cpu_to_be64(ibp->rvp.tid);
smp->attr_id = IB_SMP_ATTR_NOTICE;
/* o14-1: smp->mkey = 0; */
memcpy(smp->data, data, len);
spin_lock_irqsave(&ibp->lock, flags);
if (!ibp->sm_ah) {
if (ibp->sm_lid != be16_to_cpu(IB_LID_PERMISSIVE)) {
spin_lock_irqsave(&ibp->rvp.lock, flags);
if (!ibp->rvp.sm_ah) {
if (ibp->rvp.sm_lid != be16_to_cpu(IB_LID_PERMISSIVE)) {
struct ib_ah *ah;
ah = qib_create_qp0_ah(ibp, ibp->sm_lid);
ah = qib_create_qp0_ah(ibp, ibp->rvp.sm_lid);
if (IS_ERR(ah))
ret = PTR_ERR(ah);
else {
send_buf->ah = ah;
ibp->sm_ah = to_iah(ah);
ibp->rvp.sm_ah = ibah_to_rvtah(ah);
ret = 0;
}
} else
ret = -EINVAL;
} else {
send_buf->ah = &ibp->sm_ah->ibah;
send_buf->ah = &ibp->rvp.sm_ah->ibah;
ret = 0;
}
spin_unlock_irqrestore(&ibp->lock, flags);
spin_unlock_irqrestore(&ibp->rvp.lock, flags);
if (!ret)
ret = ib_post_send_mad(send_buf, NULL);
if (!ret) {
/* 4.096 usec. */
timeout = (4096 * (1UL << ibp->subnet_timeout)) / 1000;
ibp->trap_timeout = jiffies + usecs_to_jiffies(timeout);
timeout = (4096 * (1UL << ibp->rvp.subnet_timeout)) / 1000;
ibp->rvp.trap_timeout = jiffies + usecs_to_jiffies(timeout);
} else {
ib_free_send_mad(send_buf);
ibp->trap_timeout = 0;
ibp->rvp.trap_timeout = 0;
}
}
@ -141,10 +142,10 @@ void qib_bad_pqkey(struct qib_ibport *ibp, __be16 trap_num, u32 key, u32 sl,
struct ib_mad_notice_attr data;
if (trap_num == IB_NOTICE_TRAP_BAD_PKEY)
ibp->pkey_violations++;
ibp->rvp.pkey_violations++;
else
ibp->qkey_violations++;
ibp->n_pkt_drops++;
ibp->rvp.qkey_violations++;
ibp->rvp.n_pkt_drops++;
/* Send violation trap */
data.generic_type = IB_NOTICE_TYPE_SECURITY;
@ -205,8 +206,11 @@ static void qib_bad_mkey(struct qib_ibport *ibp, struct ib_smp *smp)
/*
* Send a Port Capability Mask Changed trap (ch. 14.3.11).
*/
void qib_cap_mask_chg(struct qib_ibport *ibp)
void qib_cap_mask_chg(struct rvt_dev_info *rdi, u8 port_num)
{
struct qib_ibdev *ibdev = container_of(rdi, struct qib_ibdev, rdi);
struct qib_devdata *dd = dd_from_dev(ibdev);
struct qib_ibport *ibp = &dd->pport[port_num - 1].ibport_data;
struct ib_mad_notice_attr data;
data.generic_type = IB_NOTICE_TYPE_INFO;
@ -217,8 +221,8 @@ void qib_cap_mask_chg(struct qib_ibport *ibp)
data.toggle_count = 0;
memset(&data.details, 0, sizeof(data.details));
data.details.ntc_144.lid = data.issuer_lid;
data.details.ntc_144.new_cap_mask = cpu_to_be32(ibp->port_cap_flags);
data.details.ntc_144.new_cap_mask =
cpu_to_be32(ibp->rvp.port_cap_flags);
qib_send_trap(ibp, &data, sizeof(data));
}
@ -409,37 +413,38 @@ static int check_mkey(struct qib_ibport *ibp, struct ib_smp *smp, int mad_flags)
int ret = 0;
/* Is the mkey in the process of expiring? */
if (ibp->mkey_lease_timeout &&
time_after_eq(jiffies, ibp->mkey_lease_timeout)) {
if (ibp->rvp.mkey_lease_timeout &&
time_after_eq(jiffies, ibp->rvp.mkey_lease_timeout)) {
/* Clear timeout and mkey protection field. */
ibp->mkey_lease_timeout = 0;
ibp->mkeyprot = 0;
ibp->rvp.mkey_lease_timeout = 0;
ibp->rvp.mkeyprot = 0;
}
if ((mad_flags & IB_MAD_IGNORE_MKEY) || ibp->mkey == 0 ||
ibp->mkey == smp->mkey)
if ((mad_flags & IB_MAD_IGNORE_MKEY) || ibp->rvp.mkey == 0 ||
ibp->rvp.mkey == smp->mkey)
valid_mkey = 1;
/* Unset lease timeout on any valid Get/Set/TrapRepress */
if (valid_mkey && ibp->mkey_lease_timeout &&
if (valid_mkey && ibp->rvp.mkey_lease_timeout &&
(smp->method == IB_MGMT_METHOD_GET ||
smp->method == IB_MGMT_METHOD_SET ||
smp->method == IB_MGMT_METHOD_TRAP_REPRESS))
ibp->mkey_lease_timeout = 0;
ibp->rvp.mkey_lease_timeout = 0;
if (!valid_mkey) {
switch (smp->method) {
case IB_MGMT_METHOD_GET:
/* Bad mkey not a violation below level 2 */
if (ibp->mkeyprot < 2)
if (ibp->rvp.mkeyprot < 2)
break;
case IB_MGMT_METHOD_SET:
case IB_MGMT_METHOD_TRAP_REPRESS:
if (ibp->mkey_violations != 0xFFFF)
++ibp->mkey_violations;
if (!ibp->mkey_lease_timeout && ibp->mkey_lease_period)
ibp->mkey_lease_timeout = jiffies +
ibp->mkey_lease_period * HZ;
if (ibp->rvp.mkey_violations != 0xFFFF)
++ibp->rvp.mkey_violations;
if (!ibp->rvp.mkey_lease_timeout &&
ibp->rvp.mkey_lease_period)
ibp->rvp.mkey_lease_timeout = jiffies +
ibp->rvp.mkey_lease_period * HZ;
/* Generate a trap notice. */
qib_bad_mkey(ibp, smp);
ret = 1;
@ -489,15 +494,15 @@ static int subn_get_portinfo(struct ib_smp *smp, struct ib_device *ibdev,
/* Only return the mkey if the protection field allows it. */
if (!(smp->method == IB_MGMT_METHOD_GET &&
ibp->mkey != smp->mkey &&
ibp->mkeyprot == 1))
pip->mkey = ibp->mkey;
pip->gid_prefix = ibp->gid_prefix;
ibp->rvp.mkey != smp->mkey &&
ibp->rvp.mkeyprot == 1))
pip->mkey = ibp->rvp.mkey;
pip->gid_prefix = ibp->rvp.gid_prefix;
pip->lid = cpu_to_be16(ppd->lid);
pip->sm_lid = cpu_to_be16(ibp->sm_lid);
pip->cap_mask = cpu_to_be32(ibp->port_cap_flags);
pip->sm_lid = cpu_to_be16(ibp->rvp.sm_lid);
pip->cap_mask = cpu_to_be32(ibp->rvp.port_cap_flags);
/* pip->diag_code; */
pip->mkey_lease_period = cpu_to_be16(ibp->mkey_lease_period);
pip->mkey_lease_period = cpu_to_be16(ibp->rvp.mkey_lease_period);
pip->local_port_num = port;
pip->link_width_enabled = ppd->link_width_enabled;
pip->link_width_supported = ppd->link_width_supported;
@ -508,7 +513,7 @@ static int subn_get_portinfo(struct ib_smp *smp, struct ib_device *ibdev,
pip->portphysstate_linkdown =
(dd->f_ibphys_portstate(ppd->lastibcstat) << 4) |
(get_linkdowndefaultstate(ppd) ? 1 : 2);
pip->mkeyprot_resv_lmc = (ibp->mkeyprot << 6) | ppd->lmc;
pip->mkeyprot_resv_lmc = (ibp->rvp.mkeyprot << 6) | ppd->lmc;
pip->linkspeedactive_enabled = (ppd->link_speed_active << 4) |
ppd->link_speed_enabled;
switch (ppd->ibmtu) {
@ -529,9 +534,9 @@ static int subn_get_portinfo(struct ib_smp *smp, struct ib_device *ibdev,
mtu = IB_MTU_256;
break;
}
pip->neighbormtu_mastersmsl = (mtu << 4) | ibp->sm_sl;
pip->neighbormtu_mastersmsl = (mtu << 4) | ibp->rvp.sm_sl;
pip->vlcap_inittype = ppd->vls_supported << 4; /* InitType = 0 */
pip->vl_high_limit = ibp->vl_high_limit;
pip->vl_high_limit = ibp->rvp.vl_high_limit;
pip->vl_arb_high_cap =
dd->f_get_ib_cfg(ppd, QIB_IB_CFG_VL_HIGH_CAP);
pip->vl_arb_low_cap =
@ -542,20 +547,20 @@ static int subn_get_portinfo(struct ib_smp *smp, struct ib_device *ibdev,
/* pip->vlstallcnt_hoqlife; */
pip->operationalvl_pei_peo_fpi_fpo =
dd->f_get_ib_cfg(ppd, QIB_IB_CFG_OP_VLS) << 4;
pip->mkey_violations = cpu_to_be16(ibp->mkey_violations);
pip->mkey_violations = cpu_to_be16(ibp->rvp.mkey_violations);
/* P_KeyViolations are counted by hardware. */
pip->pkey_violations = cpu_to_be16(ibp->pkey_violations);
pip->qkey_violations = cpu_to_be16(ibp->qkey_violations);
pip->pkey_violations = cpu_to_be16(ibp->rvp.pkey_violations);
pip->qkey_violations = cpu_to_be16(ibp->rvp.qkey_violations);
/* Only the hardware GUID is supported for now */
pip->guid_cap = QIB_GUIDS_PER_PORT;
pip->clientrereg_resv_subnetto = ibp->subnet_timeout;
pip->clientrereg_resv_subnetto = ibp->rvp.subnet_timeout;
/* 32.768 usec. response time (guessing) */
pip->resv_resptimevalue = 3;
pip->localphyerrors_overrunerrors =
(get_phyerrthreshold(ppd) << 4) |
get_overrunthreshold(ppd);
/* pip->max_credit_hint; */
if (ibp->port_cap_flags & IB_PORT_LINK_LATENCY_SUP) {
if (ibp->rvp.port_cap_flags & IB_PORT_LINK_LATENCY_SUP) {
u32 v;
v = dd->f_get_ib_cfg(ppd, QIB_IB_CFG_LINKLATENCY);
@ -685,13 +690,13 @@ static int subn_set_portinfo(struct ib_smp *smp, struct ib_device *ibdev,
event.device = ibdev;
event.element.port_num = port;
ibp->mkey = pip->mkey;
ibp->gid_prefix = pip->gid_prefix;
ibp->mkey_lease_period = be16_to_cpu(pip->mkey_lease_period);
ibp->rvp.mkey = pip->mkey;
ibp->rvp.gid_prefix = pip->gid_prefix;
ibp->rvp.mkey_lease_period = be16_to_cpu(pip->mkey_lease_period);
lid = be16_to_cpu(pip->lid);
/* Must be a valid unicast LID address. */
if (lid == 0 || lid >= QIB_MULTICAST_LID_BASE)
if (lid == 0 || lid >= be16_to_cpu(IB_MULTICAST_LID_BASE))
smp->status |= IB_SMP_INVALID_FIELD;
else if (ppd->lid != lid || ppd->lmc != (pip->mkeyprot_resv_lmc & 7)) {
if (ppd->lid != lid)
@ -706,21 +711,21 @@ static int subn_set_portinfo(struct ib_smp *smp, struct ib_device *ibdev,
smlid = be16_to_cpu(pip->sm_lid);
msl = pip->neighbormtu_mastersmsl & 0xF;
/* Must be a valid unicast LID address. */
if (smlid == 0 || smlid >= QIB_MULTICAST_LID_BASE)
if (smlid == 0 || smlid >= be16_to_cpu(IB_MULTICAST_LID_BASE))
smp->status |= IB_SMP_INVALID_FIELD;
else if (smlid != ibp->sm_lid || msl != ibp->sm_sl) {
spin_lock_irqsave(&ibp->lock, flags);
if (ibp->sm_ah) {
if (smlid != ibp->sm_lid)
ibp->sm_ah->attr.dlid = smlid;
if (msl != ibp->sm_sl)
ibp->sm_ah->attr.sl = msl;
else if (smlid != ibp->rvp.sm_lid || msl != ibp->rvp.sm_sl) {
spin_lock_irqsave(&ibp->rvp.lock, flags);
if (ibp->rvp.sm_ah) {
if (smlid != ibp->rvp.sm_lid)
ibp->rvp.sm_ah->attr.dlid = smlid;
if (msl != ibp->rvp.sm_sl)
ibp->rvp.sm_ah->attr.sl = msl;
}
spin_unlock_irqrestore(&ibp->lock, flags);
if (smlid != ibp->sm_lid)
ibp->sm_lid = smlid;
if (msl != ibp->sm_sl)
ibp->sm_sl = msl;
spin_unlock_irqrestore(&ibp->rvp.lock, flags);
if (smlid != ibp->rvp.sm_lid)
ibp->rvp.sm_lid = smlid;
if (msl != ibp->rvp.sm_sl)
ibp->rvp.sm_sl = msl;
event.event = IB_EVENT_SM_CHANGE;
ib_dispatch_event(&event);
}
@ -768,10 +773,10 @@ static int subn_set_portinfo(struct ib_smp *smp, struct ib_device *ibdev,
smp->status |= IB_SMP_INVALID_FIELD;
}
ibp->mkeyprot = pip->mkeyprot_resv_lmc >> 6;
ibp->vl_high_limit = pip->vl_high_limit;
ibp->rvp.mkeyprot = pip->mkeyprot_resv_lmc >> 6;
ibp->rvp.vl_high_limit = pip->vl_high_limit;
(void) dd->f_set_ib_cfg(ppd, QIB_IB_CFG_VL_HIGH_LIMIT,
ibp->vl_high_limit);
ibp->rvp.vl_high_limit);
mtu = ib_mtu_enum_to_int((pip->neighbormtu_mastersmsl >> 4) & 0xF);
if (mtu == -1)
@ -789,13 +794,13 @@ static int subn_set_portinfo(struct ib_smp *smp, struct ib_device *ibdev,
}
if (pip->mkey_violations == 0)
ibp->mkey_violations = 0;
ibp->rvp.mkey_violations = 0;
if (pip->pkey_violations == 0)
ibp->pkey_violations = 0;
ibp->rvp.pkey_violations = 0;
if (pip->qkey_violations == 0)
ibp->qkey_violations = 0;
ibp->rvp.qkey_violations = 0;
ore = pip->localphyerrors_overrunerrors;
if (set_phyerrthreshold(ppd, (ore >> 4) & 0xF))
@ -804,7 +809,7 @@ static int subn_set_portinfo(struct ib_smp *smp, struct ib_device *ibdev,
if (set_overrunthreshold(ppd, (ore & 0xF)))
smp->status |= IB_SMP_INVALID_FIELD;
ibp->subnet_timeout = pip->clientrereg_resv_subnetto & 0x1F;
ibp->rvp.subnet_timeout = pip->clientrereg_resv_subnetto & 0x1F;
/*
* Do the port state change now that the other link parameters
@ -1028,7 +1033,7 @@ static int set_pkeys(struct qib_devdata *dd, u8 port, u16 *pkeys)
(void) dd->f_set_ib_cfg(ppd, QIB_IB_CFG_PKEYS, 0);
event.event = IB_EVENT_PKEY_CHANGE;
event.device = &dd->verbs_dev.ibdev;
event.device = &dd->verbs_dev.rdi.ibdev;
event.element.port_num = port;
ib_dispatch_event(&event);
}
@ -1062,7 +1067,7 @@ static int subn_get_sl_to_vl(struct ib_smp *smp, struct ib_device *ibdev,
memset(smp->data, 0, sizeof(smp->data));
if (!(ibp->port_cap_flags & IB_PORT_SL_MAP_SUP))
if (!(ibp->rvp.port_cap_flags & IB_PORT_SL_MAP_SUP))
smp->status |= IB_SMP_UNSUP_METHOD;
else
for (i = 0; i < ARRAY_SIZE(ibp->sl_to_vl); i += 2)
@ -1078,7 +1083,7 @@ static int subn_set_sl_to_vl(struct ib_smp *smp, struct ib_device *ibdev,
u8 *p = (u8 *) smp->data;
unsigned i;
if (!(ibp->port_cap_flags & IB_PORT_SL_MAP_SUP)) {
if (!(ibp->rvp.port_cap_flags & IB_PORT_SL_MAP_SUP)) {
smp->status |= IB_SMP_UNSUP_METHOD;
return reply(smp);
}
@ -1195,20 +1200,20 @@ static int pma_get_portsamplescontrol(struct ib_pma_mad *pmp,
pmp->mad_hdr.status |= IB_SMP_INVALID_FIELD;
goto bail;
}
spin_lock_irqsave(&ibp->lock, flags);
spin_lock_irqsave(&ibp->rvp.lock, flags);
p->tick = dd->f_get_ib_cfg(ppd, QIB_IB_CFG_PMA_TICKS);
p->sample_status = dd->f_portcntr(ppd, QIBPORTCNTR_PSSTAT);
p->counter_width = 4; /* 32 bit counters */
p->counter_mask0_9 = COUNTER_MASK0_9;
p->sample_start = cpu_to_be32(ibp->pma_sample_start);
p->sample_interval = cpu_to_be32(ibp->pma_sample_interval);
p->tag = cpu_to_be16(ibp->pma_tag);
p->counter_select[0] = ibp->pma_counter_select[0];
p->counter_select[1] = ibp->pma_counter_select[1];
p->counter_select[2] = ibp->pma_counter_select[2];
p->counter_select[3] = ibp->pma_counter_select[3];
p->counter_select[4] = ibp->pma_counter_select[4];
spin_unlock_irqrestore(&ibp->lock, flags);
p->sample_start = cpu_to_be32(ibp->rvp.pma_sample_start);
p->sample_interval = cpu_to_be32(ibp->rvp.pma_sample_interval);
p->tag = cpu_to_be16(ibp->rvp.pma_tag);
p->counter_select[0] = ibp->rvp.pma_counter_select[0];
p->counter_select[1] = ibp->rvp.pma_counter_select[1];
p->counter_select[2] = ibp->rvp.pma_counter_select[2];
p->counter_select[3] = ibp->rvp.pma_counter_select[3];
p->counter_select[4] = ibp->rvp.pma_counter_select[4];
spin_unlock_irqrestore(&ibp->rvp.lock, flags);
bail:
return reply((struct ib_smp *) pmp);
@ -1233,7 +1238,7 @@ static int pma_set_portsamplescontrol(struct ib_pma_mad *pmp,
goto bail;
}
spin_lock_irqsave(&ibp->lock, flags);
spin_lock_irqsave(&ibp->rvp.lock, flags);
/* Port Sampling code owns the PS* HW counters */
xmit_flags = ppd->cong_stats.flags;
@ -1242,18 +1247,18 @@ static int pma_set_portsamplescontrol(struct ib_pma_mad *pmp,
if (status == IB_PMA_SAMPLE_STATUS_DONE ||
(status == IB_PMA_SAMPLE_STATUS_RUNNING &&
xmit_flags == IB_PMA_CONG_HW_CONTROL_TIMER)) {
ibp->pma_sample_start = be32_to_cpu(p->sample_start);
ibp->pma_sample_interval = be32_to_cpu(p->sample_interval);
ibp->pma_tag = be16_to_cpu(p->tag);
ibp->pma_counter_select[0] = p->counter_select[0];
ibp->pma_counter_select[1] = p->counter_select[1];
ibp->pma_counter_select[2] = p->counter_select[2];
ibp->pma_counter_select[3] = p->counter_select[3];
ibp->pma_counter_select[4] = p->counter_select[4];
dd->f_set_cntr_sample(ppd, ibp->pma_sample_interval,
ibp->pma_sample_start);
ibp->rvp.pma_sample_start = be32_to_cpu(p->sample_start);
ibp->rvp.pma_sample_interval = be32_to_cpu(p->sample_interval);
ibp->rvp.pma_tag = be16_to_cpu(p->tag);
ibp->rvp.pma_counter_select[0] = p->counter_select[0];
ibp->rvp.pma_counter_select[1] = p->counter_select[1];
ibp->rvp.pma_counter_select[2] = p->counter_select[2];
ibp->rvp.pma_counter_select[3] = p->counter_select[3];
ibp->rvp.pma_counter_select[4] = p->counter_select[4];
dd->f_set_cntr_sample(ppd, ibp->rvp.pma_sample_interval,
ibp->rvp.pma_sample_start);
}
spin_unlock_irqrestore(&ibp->lock, flags);
spin_unlock_irqrestore(&ibp->rvp.lock, flags);
ret = pma_get_portsamplescontrol(pmp, ibdev, port);
@ -1357,8 +1362,8 @@ static int pma_get_portsamplesresult(struct ib_pma_mad *pmp,
int i;
memset(pmp->data, 0, sizeof(pmp->data));
spin_lock_irqsave(&ibp->lock, flags);
p->tag = cpu_to_be16(ibp->pma_tag);
spin_lock_irqsave(&ibp->rvp.lock, flags);
p->tag = cpu_to_be16(ibp->rvp.pma_tag);
if (ppd->cong_stats.flags == IB_PMA_CONG_HW_CONTROL_TIMER)
p->sample_status = IB_PMA_SAMPLE_STATUS_DONE;
else {
@ -1373,11 +1378,11 @@ static int pma_get_portsamplesresult(struct ib_pma_mad *pmp,
ppd->cong_stats.flags = IB_PMA_CONG_HW_CONTROL_TIMER;
}
}
for (i = 0; i < ARRAY_SIZE(ibp->pma_counter_select); i++)
for (i = 0; i < ARRAY_SIZE(ibp->rvp.pma_counter_select); i++)
p->counter[i] = cpu_to_be32(
get_cache_hw_sample_counters(
ppd, ibp->pma_counter_select[i]));
spin_unlock_irqrestore(&ibp->lock, flags);
ppd, ibp->rvp.pma_counter_select[i]));
spin_unlock_irqrestore(&ibp->rvp.lock, flags);
return reply((struct ib_smp *) pmp);
}
@ -1397,8 +1402,8 @@ static int pma_get_portsamplesresult_ext(struct ib_pma_mad *pmp,
/* Port Sampling code owns the PS* HW counters */
memset(pmp->data, 0, sizeof(pmp->data));
spin_lock_irqsave(&ibp->lock, flags);
p->tag = cpu_to_be16(ibp->pma_tag);
spin_lock_irqsave(&ibp->rvp.lock, flags);
p->tag = cpu_to_be16(ibp->rvp.pma_tag);
if (ppd->cong_stats.flags == IB_PMA_CONG_HW_CONTROL_TIMER)
p->sample_status = IB_PMA_SAMPLE_STATUS_DONE;
else {
@ -1415,11 +1420,11 @@ static int pma_get_portsamplesresult_ext(struct ib_pma_mad *pmp,
ppd->cong_stats.flags = IB_PMA_CONG_HW_CONTROL_TIMER;
}
}
for (i = 0; i < ARRAY_SIZE(ibp->pma_counter_select); i++)
for (i = 0; i < ARRAY_SIZE(ibp->rvp.pma_counter_select); i++)
p->counter[i] = cpu_to_be64(
get_cache_hw_sample_counters(
ppd, ibp->pma_counter_select[i]));
spin_unlock_irqrestore(&ibp->lock, flags);
ppd, ibp->rvp.pma_counter_select[i]));
spin_unlock_irqrestore(&ibp->rvp.lock, flags);
return reply((struct ib_smp *) pmp);
}
@ -1453,7 +1458,7 @@ static int pma_get_portcounters(struct ib_pma_mad *pmp,
cntrs.excessive_buffer_overrun_errors -=
ibp->z_excessive_buffer_overrun_errors;
cntrs.vl15_dropped -= ibp->z_vl15_dropped;
cntrs.vl15_dropped += ibp->n_vl15_dropped;
cntrs.vl15_dropped += ibp->rvp.n_vl15_dropped;
memset(pmp->data, 0, sizeof(pmp->data));
@ -1546,9 +1551,9 @@ static int pma_get_portcounters_cong(struct ib_pma_mad *pmp,
pmp->mad_hdr.status |= IB_SMP_INVALID_FIELD;
qib_get_counters(ppd, &cntrs);
spin_lock_irqsave(&ppd->ibport_data.lock, flags);
spin_lock_irqsave(&ppd->ibport_data.rvp.lock, flags);
xmit_wait_counter = xmit_wait_get_value_delta(ppd);
spin_unlock_irqrestore(&ppd->ibport_data.lock, flags);
spin_unlock_irqrestore(&ppd->ibport_data.rvp.lock, flags);
/* Adjust counters for any resets done. */
cntrs.symbol_error_counter -= ibp->z_symbol_error_counter;
@ -1564,7 +1569,7 @@ static int pma_get_portcounters_cong(struct ib_pma_mad *pmp,
cntrs.excessive_buffer_overrun_errors -=
ibp->z_excessive_buffer_overrun_errors;
cntrs.vl15_dropped -= ibp->z_vl15_dropped;
cntrs.vl15_dropped += ibp->n_vl15_dropped;
cntrs.vl15_dropped += ibp->rvp.n_vl15_dropped;
cntrs.port_xmit_data -= ibp->z_port_xmit_data;
cntrs.port_rcv_data -= ibp->z_port_rcv_data;
cntrs.port_xmit_packets -= ibp->z_port_xmit_packets;
@ -1743,7 +1748,7 @@ static int pma_set_portcounters(struct ib_pma_mad *pmp,
cntrs.excessive_buffer_overrun_errors;
if (p->counter_select & IB_PMA_SEL_PORT_VL15_DROPPED) {
ibp->n_vl15_dropped = 0;
ibp->rvp.n_vl15_dropped = 0;
ibp->z_vl15_dropped = cntrs.vl15_dropped;
}
@ -1778,11 +1783,11 @@ static int pma_set_portcounters_cong(struct ib_pma_mad *pmp,
ret = pma_get_portcounters_cong(pmp, ibdev, port);
if (counter_select & IB_PMA_SEL_CONG_XMIT) {
spin_lock_irqsave(&ppd->ibport_data.lock, flags);
spin_lock_irqsave(&ppd->ibport_data.rvp.lock, flags);
ppd->cong_stats.counter = 0;
dd->f_set_cntr_sample(ppd, QIB_CONG_TIMER_PSINTERVAL,
0x0);
spin_unlock_irqrestore(&ppd->ibport_data.lock, flags);
spin_unlock_irqrestore(&ppd->ibport_data.rvp.lock, flags);
}
if (counter_select & IB_PMA_SEL_CONG_PORT_DATA) {
ibp->z_port_xmit_data = cntrs.port_xmit_data;
@ -1806,7 +1811,7 @@ static int pma_set_portcounters_cong(struct ib_pma_mad *pmp,
cntrs.local_link_integrity_errors;
ibp->z_excessive_buffer_overrun_errors =
cntrs.excessive_buffer_overrun_errors;
ibp->n_vl15_dropped = 0;
ibp->rvp.n_vl15_dropped = 0;
ibp->z_vl15_dropped = cntrs.vl15_dropped;
}
@ -1916,12 +1921,12 @@ static int process_subn(struct ib_device *ibdev, int mad_flags,
ret = subn_get_vl_arb(smp, ibdev, port);
goto bail;
case IB_SMP_ATTR_SM_INFO:
if (ibp->port_cap_flags & IB_PORT_SM_DISABLED) {
if (ibp->rvp.port_cap_flags & IB_PORT_SM_DISABLED) {
ret = IB_MAD_RESULT_SUCCESS |
IB_MAD_RESULT_CONSUMED;
goto bail;
}
if (ibp->port_cap_flags & IB_PORT_SM) {
if (ibp->rvp.port_cap_flags & IB_PORT_SM) {
ret = IB_MAD_RESULT_SUCCESS;
goto bail;
}
@ -1950,12 +1955,12 @@ static int process_subn(struct ib_device *ibdev, int mad_flags,
ret = subn_set_vl_arb(smp, ibdev, port);
goto bail;
case IB_SMP_ATTR_SM_INFO:
if (ibp->port_cap_flags & IB_PORT_SM_DISABLED) {
if (ibp->rvp.port_cap_flags & IB_PORT_SM_DISABLED) {
ret = IB_MAD_RESULT_SUCCESS |
IB_MAD_RESULT_CONSUMED;
goto bail;
}
if (ibp->port_cap_flags & IB_PORT_SM) {
if (ibp->rvp.port_cap_flags & IB_PORT_SM) {
ret = IB_MAD_RESULT_SUCCESS;
goto bail;
}
@ -2443,12 +2448,6 @@ bail:
return ret;
}
static void send_handler(struct ib_mad_agent *agent,
struct ib_mad_send_wc *mad_send_wc)
{
ib_free_send_mad(mad_send_wc->send_buf);
}
static void xmit_wait_timer_func(unsigned long opaque)
{
struct qib_pportdata *ppd = (struct qib_pportdata *)opaque;
@ -2456,7 +2455,7 @@ static void xmit_wait_timer_func(unsigned long opaque)
unsigned long flags;
u8 status;
spin_lock_irqsave(&ppd->ibport_data.lock, flags);
spin_lock_irqsave(&ppd->ibport_data.rvp.lock, flags);
if (ppd->cong_stats.flags == IB_PMA_CONG_HW_CONTROL_SAMPLE) {
status = dd->f_portcntr(ppd, QIBPORTCNTR_PSSTAT);
if (status == IB_PMA_SAMPLE_STATUS_DONE) {
@ -2469,74 +2468,35 @@ static void xmit_wait_timer_func(unsigned long opaque)
ppd->cong_stats.counter = xmit_wait_get_value_delta(ppd);
dd->f_set_cntr_sample(ppd, QIB_CONG_TIMER_PSINTERVAL, 0x0);
done:
spin_unlock_irqrestore(&ppd->ibport_data.lock, flags);
spin_unlock_irqrestore(&ppd->ibport_data.rvp.lock, flags);
mod_timer(&ppd->cong_stats.timer, jiffies + HZ);
}
int qib_create_agents(struct qib_ibdev *dev)
void qib_notify_create_mad_agent(struct rvt_dev_info *rdi, int port_idx)
{
struct qib_devdata *dd = dd_from_dev(dev);
struct ib_mad_agent *agent;
struct qib_ibport *ibp;
int p;
int ret;
struct qib_ibdev *ibdev = container_of(rdi, struct qib_ibdev, rdi);
struct qib_devdata *dd = container_of(ibdev,
struct qib_devdata, verbs_dev);
for (p = 0; p < dd->num_pports; p++) {
ibp = &dd->pport[p].ibport_data;
agent = ib_register_mad_agent(&dev->ibdev, p + 1, IB_QPT_SMI,
NULL, 0, send_handler,
NULL, NULL, 0);
if (IS_ERR(agent)) {
ret = PTR_ERR(agent);
goto err;
}
/* Initialize xmit_wait structure */
dd->pport[p].cong_stats.counter = 0;
init_timer(&dd->pport[p].cong_stats.timer);
dd->pport[p].cong_stats.timer.function = xmit_wait_timer_func;
dd->pport[p].cong_stats.timer.data =
(unsigned long)(&dd->pport[p]);
dd->pport[p].cong_stats.timer.expires = 0;
add_timer(&dd->pport[p].cong_stats.timer);
ibp->send_agent = agent;
}
return 0;
err:
for (p = 0; p < dd->num_pports; p++) {
ibp = &dd->pport[p].ibport_data;
if (ibp->send_agent) {
agent = ibp->send_agent;
ibp->send_agent = NULL;
ib_unregister_mad_agent(agent);
}
}
return ret;
/* Initialize xmit_wait structure */
dd->pport[port_idx].cong_stats.counter = 0;
init_timer(&dd->pport[port_idx].cong_stats.timer);
dd->pport[port_idx].cong_stats.timer.function = xmit_wait_timer_func;
dd->pport[port_idx].cong_stats.timer.data =
(unsigned long)(&dd->pport[port_idx]);
dd->pport[port_idx].cong_stats.timer.expires = 0;
add_timer(&dd->pport[port_idx].cong_stats.timer);
}
void qib_free_agents(struct qib_ibdev *dev)
void qib_notify_free_mad_agent(struct rvt_dev_info *rdi, int port_idx)
{
struct qib_devdata *dd = dd_from_dev(dev);
struct ib_mad_agent *agent;
struct qib_ibport *ibp;
int p;
struct qib_ibdev *ibdev = container_of(rdi, struct qib_ibdev, rdi);
struct qib_devdata *dd = container_of(ibdev,
struct qib_devdata, verbs_dev);
for (p = 0; p < dd->num_pports; p++) {
ibp = &dd->pport[p].ibport_data;
if (ibp->send_agent) {
agent = ibp->send_agent;
ibp->send_agent = NULL;
ib_unregister_mad_agent(agent);
}
if (ibp->sm_ah) {
ib_destroy_ah(&ibp->sm_ah->ibah);
ibp->sm_ah = NULL;
}
if (dd->pport[p].cong_stats.timer.data)
del_timer_sync(&dd->pport[p].cong_stats.timer);
}
if (dd->pport[port_idx].cong_stats.timer.data)
del_timer_sync(&dd->pport[port_idx].cong_stats.timer);
if (dd->pport[port_idx].ibport_data.smi_ah)
ib_destroy_ah(&dd->pport[port_idx].ibport_data.smi_ah->ibah);
}

174
drivers/infiniband/hw/qib/qib_mmap.c
View File

@ -1,174 +0,0 @@
/*
* Copyright (c) 2006, 2007, 2008, 2009 QLogic Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include <linux/module.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>
#include <linux/mm.h>
#include <linux/errno.h>
#include <asm/pgtable.h>
#include "qib_verbs.h"
/**
* qib_release_mmap_info - free mmap info structure
* @ref: a pointer to the kref within struct qib_mmap_info
*/
void qib_release_mmap_info(struct kref *ref)
{
struct qib_mmap_info *ip =
container_of(ref, struct qib_mmap_info, ref);
struct qib_ibdev *dev = to_idev(ip->context->device);
spin_lock_irq(&dev->pending_lock);
list_del(&ip->pending_mmaps);
spin_unlock_irq(&dev->pending_lock);
vfree(ip->obj);
kfree(ip);
}
/*
* open and close keep track of how many times the CQ is mapped,
* to avoid releasing it.
*/
static void qib_vma_open(struct vm_area_struct *vma)
{
struct qib_mmap_info *ip = vma->vm_private_data;
kref_get(&ip->ref);
}
static void qib_vma_close(struct vm_area_struct *vma)
{
struct qib_mmap_info *ip = vma->vm_private_data;
kref_put(&ip->ref, qib_release_mmap_info);
}
static const struct vm_operations_struct qib_vm_ops = {
.open = qib_vma_open,
.close = qib_vma_close,
};
/**
* qib_mmap - create a new mmap region
* @context: the IB user context of the process making the mmap() call
* @vma: the VMA to be initialized
* Return zero if the mmap is OK. Otherwise, return an errno.
*/
int qib_mmap(struct ib_ucontext *context, struct vm_area_struct *vma)
{
struct qib_ibdev *dev = to_idev(context->device);
unsigned long offset = vma->vm_pgoff << PAGE_SHIFT;
unsigned long size = vma->vm_end - vma->vm_start;
struct qib_mmap_info *ip, *pp;
int ret = -EINVAL;
/*
* Search the device's list of objects waiting for a mmap call.
* Normally, this list is very short since a call to create a
* CQ, QP, or SRQ is soon followed by a call to mmap().
*/
spin_lock_irq(&dev->pending_lock);
list_for_each_entry_safe(ip, pp, &dev->pending_mmaps,
pending_mmaps) {
/* Only the creator is allowed to mmap the object */
if (context != ip->context || (__u64) offset != ip->offset)
continue;
/* Don't allow a mmap larger than the object. */
if (size > ip->size)
break;
list_del_init(&ip->pending_mmaps);
spin_unlock_irq(&dev->pending_lock);
ret = remap_vmalloc_range(vma, ip->obj, 0);
if (ret)
goto done;
vma->vm_ops = &qib_vm_ops;
vma->vm_private_data = ip;
qib_vma_open(vma);
goto done;
}
spin_unlock_irq(&dev->pending_lock);
done:
return ret;
}
/*
* Allocate information for qib_mmap
*/
struct qib_mmap_info *qib_create_mmap_info(struct qib_ibdev *dev,
u32 size,
struct ib_ucontext *context,
void *obj) {
struct qib_mmap_info *ip;
ip = kmalloc(sizeof(*ip), GFP_KERNEL);
if (!ip)
goto bail;
size = PAGE_ALIGN(size);
spin_lock_irq(&dev->mmap_offset_lock);
if (dev->mmap_offset == 0)
dev->mmap_offset = PAGE_SIZE;
ip->offset = dev->mmap_offset;
dev->mmap_offset += size;
spin_unlock_irq(&dev->mmap_offset_lock);
INIT_LIST_HEAD(&ip->pending_mmaps);
ip->size = size;
ip->context = context;
ip->obj = obj;
kref_init(&ip->ref);
bail:
return ip;
}
void qib_update_mmap_info(struct qib_ibdev *dev, struct qib_mmap_info *ip,
u32 size, void *obj)
{
size = PAGE_ALIGN(size);
spin_lock_irq(&dev->mmap_offset_lock);
if (dev->mmap_offset == 0)
dev->mmap_offset = PAGE_SIZE;
ip->offset = dev->mmap_offset;
dev->mmap_offset += size;
spin_unlock_irq(&dev->mmap_offset_lock);
ip->size = size;
ip->obj = obj;
}

490
drivers/infiniband/hw/qib/qib_mr.c
View File

@ -1,490 +0,0 @@
/*
* Copyright (c) 2006, 2007, 2008, 2009 QLogic Corporation. All rights reserved.
* Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include <rdma/ib_umem.h>
#include <rdma/ib_smi.h>
#include "qib.h"
/* Fast memory region */
struct qib_fmr {
struct ib_fmr ibfmr;
struct qib_mregion mr; /* must be last */
};
static inline struct qib_fmr *to_ifmr(struct ib_fmr *ibfmr)
{
return container_of(ibfmr, struct qib_fmr, ibfmr);
}
static int init_qib_mregion(struct qib_mregion *mr, struct ib_pd *pd,
int count)
{
int m, i = 0;
int rval = 0;
m = (count + QIB_SEGSZ - 1) / QIB_SEGSZ;
for (; i < m; i++) {
mr->map[i] = kzalloc(sizeof(*mr->map[0]), GFP_KERNEL);
if (!mr->map[i])
goto bail;
}
mr->mapsz = m;
init_completion(&mr->comp);
/* count returning the ptr to user */
atomic_set(&mr->refcount, 1);
mr->pd = pd;
mr->max_segs = count;
out:
return rval;
bail:
while (i)
kfree(mr->map[--i]);
rval = -ENOMEM;
goto out;
}
static void deinit_qib_mregion(struct qib_mregion *mr)
{
int i = mr->mapsz;
mr->mapsz = 0;
while (i)
kfree(mr->map[--i]);
}
/**
* qib_get_dma_mr - get a DMA memory region
* @pd: protection domain for this memory region
* @acc: access flags
*
* Returns the memory region on success, otherwise returns an errno.
* Note that all DMA addresses should be created via the
* struct ib_dma_mapping_ops functions (see qib_dma.c).
*/
struct ib_mr *qib_get_dma_mr(struct ib_pd *pd, int acc)
{
struct qib_mr *mr = NULL;
struct ib_mr *ret;
int rval;
if (to_ipd(pd)->user) {
ret = ERR_PTR(-EPERM);
goto bail;
}
mr = kzalloc(sizeof(*mr), GFP_KERNEL);
if (!mr) {
ret = ERR_PTR(-ENOMEM);
goto bail;
}
rval = init_qib_mregion(&mr->mr, pd, 0);
if (rval) {
ret = ERR_PTR(rval);
goto bail;
}
rval = qib_alloc_lkey(&mr->mr, 1);
if (rval) {
ret = ERR_PTR(rval);
goto bail_mregion;
}
mr->mr.access_flags = acc;
ret = &mr->ibmr;
done:
return ret;
bail_mregion:
deinit_qib_mregion(&mr->mr);
bail:
kfree(mr);
goto done;
}
static struct qib_mr *alloc_mr(int count, struct ib_pd *pd)
{
struct qib_mr *mr;
int rval = -ENOMEM;
int m;
/* Allocate struct plus pointers to first level page tables. */
m = (count + QIB_SEGSZ - 1) / QIB_SEGSZ;
mr = kzalloc(sizeof(*mr) + m * sizeof(mr->mr.map[0]), GFP_KERNEL);
if (!mr)
goto bail;
rval = init_qib_mregion(&mr->mr, pd, count);
if (rval)
goto bail;
rval = qib_alloc_lkey(&mr->mr, 0);
if (rval)
goto bail_mregion;
mr->ibmr.lkey = mr->mr.lkey;
mr->ibmr.rkey = mr->mr.lkey;
done:
return mr;
bail_mregion:
deinit_qib_mregion(&mr->mr);
bail:
kfree(mr);
mr = ERR_PTR(rval);
goto done;
}
/**
* qib_reg_user_mr - register a userspace memory region
* @pd: protection domain for this memory region
* @start: starting userspace address
* @length: length of region to register
* @mr_access_flags: access flags for this memory region
* @udata: unused by the QLogic_IB driver
*
* Returns the memory region on success, otherwise returns an errno.
*/
struct ib_mr *qib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
u64 virt_addr, int mr_access_flags,
struct ib_udata *udata)
{
struct qib_mr *mr;
struct ib_umem *umem;
struct scatterlist *sg;
int n, m, entry;
struct ib_mr *ret;
if (length == 0) {
ret = ERR_PTR(-EINVAL);
goto bail;
}
umem = ib_umem_get(pd->uobject->context, start, length,
mr_access_flags, 0);
if (IS_ERR(umem))
return (void *) umem;
n = umem->nmap;
mr = alloc_mr(n, pd);
if (IS_ERR(mr)) {
ret = (struct ib_mr *)mr;
ib_umem_release(umem);
goto bail;
}
mr->mr.user_base = start;
mr->mr.iova = virt_addr;
mr->mr.length = length;
mr->mr.offset = ib_umem_offset(umem);
mr->mr.access_flags = mr_access_flags;
mr->umem = umem;
if (is_power_of_2(umem->page_size))
mr->mr.page_shift = ilog2(umem->page_size);
m = 0;
n = 0;
for_each_sg(umem->sg_head.sgl, sg, umem->nmap, entry) {
void *vaddr;
vaddr = page_address(sg_page(sg));
if (!vaddr) {
ret = ERR_PTR(-EINVAL);
goto bail;
}
mr->mr.map[m]->segs[n].vaddr = vaddr;
mr->mr.map[m]->segs[n].length = umem->page_size;
n++;
if (n == QIB_SEGSZ) {
m++;
n = 0;
}
}
ret = &mr->ibmr;
bail:
return ret;
}
/**
* qib_dereg_mr - unregister and free a memory region
* @ibmr: the memory region to free
*
* Returns 0 on success.
*
* Note that this is called to free MRs created by qib_get_dma_mr()
* or qib_reg_user_mr().
*/
int qib_dereg_mr(struct ib_mr *ibmr)
{
struct qib_mr *mr = to_imr(ibmr);
int ret = 0;
unsigned long timeout;
kfree(mr->pages);
qib_free_lkey(&mr->mr);
qib_put_mr(&mr->mr); /* will set completion if last */
timeout = wait_for_completion_timeout(&mr->mr.comp,
5 * HZ);
if (!timeout) {
qib_get_mr(&mr->mr);
ret = -EBUSY;
goto out;
}
deinit_qib_mregion(&mr->mr);
if (mr->umem)
ib_umem_release(mr->umem);
kfree(mr);
out:
return ret;
}
/*
* Allocate a memory region usable with the
* IB_WR_REG_MR send work request.
*
* Return the memory region on success, otherwise return an errno.
*/
struct ib_mr *qib_alloc_mr(struct ib_pd *pd,
enum ib_mr_type mr_type,
u32 max_num_sg)
{
struct qib_mr *mr;
if (mr_type != IB_MR_TYPE_MEM_REG)
return ERR_PTR(-EINVAL);
mr = alloc_mr(max_num_sg, pd);
if (IS_ERR(mr))
return (struct ib_mr *)mr;
mr->pages = kcalloc(max_num_sg, sizeof(u64), GFP_KERNEL);
if (!mr->pages)
goto err;
return &mr->ibmr;
err:
qib_dereg_mr(&mr->ibmr);
return ERR_PTR(-ENOMEM);
}
static int qib_set_page(struct ib_mr *ibmr, u64 addr)
{
struct qib_mr *mr = to_imr(ibmr);
if (unlikely(mr->npages == mr->mr.max_segs))
return -ENOMEM;
mr->pages[mr->npages++] = addr;
return 0;
}
int qib_map_mr_sg(struct ib_mr *ibmr,
struct scatterlist *sg,
int sg_nents)
{
struct qib_mr *mr = to_imr(ibmr);
mr->npages = 0;
return ib_sg_to_pages(ibmr, sg, sg_nents, qib_set_page);
}
/**
* qib_alloc_fmr - allocate a fast memory region
* @pd: the protection domain for this memory region
* @mr_access_flags: access flags for this memory region
* @fmr_attr: fast memory region attributes
*
* Returns the memory region on success, otherwise returns an errno.
*/
struct ib_fmr *qib_alloc_fmr(struct ib_pd *pd, int mr_access_flags,
struct ib_fmr_attr *fmr_attr)
{
struct qib_fmr *fmr;
int m;
struct ib_fmr *ret;
int rval = -ENOMEM;
/* Allocate struct plus pointers to first level page tables. */
m = (fmr_attr->max_pages + QIB_SEGSZ - 1) / QIB_SEGSZ;
fmr = kzalloc(sizeof(*fmr) + m * sizeof(fmr->mr.map[0]), GFP_KERNEL);
if (!fmr)
goto bail;
rval = init_qib_mregion(&fmr->mr, pd, fmr_attr->max_pages);
if (rval)
goto bail;
/*
* ib_alloc_fmr() will initialize fmr->ibfmr except for lkey &
* rkey.
*/
rval = qib_alloc_lkey(&fmr->mr, 0);
if (rval)
goto bail_mregion;
fmr->ibfmr.rkey = fmr->mr.lkey;
fmr->ibfmr.lkey = fmr->mr.lkey;
/*
* Resources are allocated but no valid mapping (RKEY can't be
* used).
*/
fmr->mr.access_flags = mr_access_flags;
fmr->mr.max_segs = fmr_attr->max_pages;
fmr->mr.page_shift = fmr_attr->page_shift;
ret = &fmr->ibfmr;
done:
return ret;
bail_mregion:
deinit_qib_mregion(&fmr->mr);
bail:
kfree(fmr);
ret = ERR_PTR(rval);
goto done;
}
/**
* qib_map_phys_fmr - set up a fast memory region
* @ibmfr: the fast memory region to set up
* @page_list: the list of pages to associate with the fast memory region
* @list_len: the number of pages to associate with the fast memory region
* @iova: the virtual address of the start of the fast memory region
*
* This may be called from interrupt context.
*/
int qib_map_phys_fmr(struct ib_fmr *ibfmr, u64 *page_list,
int list_len, u64 iova)
{
struct qib_fmr *fmr = to_ifmr(ibfmr);
struct qib_lkey_table *rkt;
unsigned long flags;
int m, n, i;
u32 ps;
int ret;
i = atomic_read(&fmr->mr.refcount);
if (i > 2)
return -EBUSY;
if (list_len > fmr->mr.max_segs) {
ret = -EINVAL;
goto bail;
}
rkt = &to_idev(ibfmr->device)->lk_table;
spin_lock_irqsave(&rkt->lock, flags);
fmr->mr.user_base = iova;
fmr->mr.iova = iova;
ps = 1 << fmr->mr.page_shift;
fmr->mr.length = list_len * ps;
m = 0;
n = 0;
for (i = 0; i < list_len; i++) {
fmr->mr.map[m]->segs[n].vaddr = (void *) page_list[i];
fmr->mr.map[m]->segs[n].length = ps;
if (++n == QIB_SEGSZ) {
m++;
n = 0;
}
}
spin_unlock_irqrestore(&rkt->lock, flags);
ret = 0;
bail:
return ret;
}
/**
* qib_unmap_fmr - unmap fast memory regions
* @fmr_list: the list of fast memory regions to unmap
*
* Returns 0 on success.
*/
int qib_unmap_fmr(struct list_head *fmr_list)
{
struct qib_fmr *fmr;
struct qib_lkey_table *rkt;
unsigned long flags;
list_for_each_entry(fmr, fmr_list, ibfmr.list) {
rkt = &to_idev(fmr->ibfmr.device)->lk_table;
spin_lock_irqsave(&rkt->lock, flags);
fmr->mr.user_base = 0;
fmr->mr.iova = 0;
fmr->mr.length = 0;
spin_unlock_irqrestore(&rkt->lock, flags);
}
return 0;
}
/**
* qib_dealloc_fmr - deallocate a fast memory region
* @ibfmr: the fast memory region to deallocate
*
* Returns 0 on success.
*/
int qib_dealloc_fmr(struct ib_fmr *ibfmr)
{
struct qib_fmr *fmr = to_ifmr(ibfmr);
int ret = 0;
unsigned long timeout;
qib_free_lkey(&fmr->mr);
qib_put_mr(&fmr->mr); /* will set completion if last */
timeout = wait_for_completion_timeout(&fmr->mr.comp,
5 * HZ);
if (!timeout) {
qib_get_mr(&fmr->mr);
ret = -EBUSY;
goto out;
}
deinit_qib_mregion(&fmr->mr);
kfree(fmr);
out:
return ret;
}
void mr_rcu_callback(struct rcu_head *list)
{
struct qib_mregion *mr = container_of(list, struct qib_mregion, list);
complete(&mr->comp);
}

1182
drivers/infiniband/hw/qib/qib_qp.c
View File

File diff suppressed because it is too large Load Diff

411
drivers/infiniband/hw/qib/qib_rc.c
View File

File diff suppressed because it is too large Load Diff

191
drivers/infiniband/hw/qib/qib_ruc.c
View File

@ -79,16 +79,16 @@ const u32 ib_qib_rnr_table[32] = {
* Validate a RWQE and fill in the SGE state.
* Return 1 if OK.
*/
static int qib_init_sge(struct qib_qp *qp, struct qib_rwqe *wqe)
static int qib_init_sge(struct rvt_qp *qp, struct rvt_rwqe *wqe)
{
int i, j, ret;
struct ib_wc wc;
struct qib_lkey_table *rkt;
struct qib_pd *pd;
struct qib_sge_state *ss;
struct rvt_lkey_table *rkt;
struct rvt_pd *pd;
struct rvt_sge_state *ss;
rkt = &to_idev(qp->ibqp.device)->lk_table;
pd = to_ipd(qp->ibqp.srq ? qp->ibqp.srq->pd : qp->ibqp.pd);
rkt = &to_idev(qp->ibqp.device)->rdi.lkey_table;
pd = ibpd_to_rvtpd(qp->ibqp.srq ? qp->ibqp.srq->pd : qp->ibqp.pd);
ss = &qp->r_sge;
ss->sg_list = qp->r_sg_list;
qp->r_len = 0;
@ -96,7 +96,7 @@ static int qib_init_sge(struct qib_qp *qp, struct qib_rwqe *wqe)
if (wqe->sg_list[i].length == 0)
continue;
/* Check LKEY */
if (!qib_lkey_ok(rkt, pd, j ? &ss->sg_list[j - 1] : &ss->sge,
if (!rvt_lkey_ok(rkt, pd, j ? &ss->sg_list[j - 1] : &ss->sge,
&wqe->sg_list[i], IB_ACCESS_LOCAL_WRITE))
goto bad_lkey;
qp->r_len += wqe->sg_list[i].length;
@ -109,9 +109,9 @@ static int qib_init_sge(struct qib_qp *qp, struct qib_rwqe *wqe)
bad_lkey:
while (j) {
struct qib_sge *sge = --j ? &ss->sg_list[j - 1] : &ss->sge;
struct rvt_sge *sge = --j ? &ss->sg_list[j - 1] : &ss->sge;
qib_put_mr(sge->mr);
rvt_put_mr(sge->mr);
}
ss->num_sge = 0;
memset(&wc, 0, sizeof(wc));
@ -120,7 +120,7 @@ bad_lkey:
wc.opcode = IB_WC_RECV;
wc.qp = &qp->ibqp;
/* Signal solicited completion event. */
qib_cq_enter(to_icq(qp->ibqp.recv_cq), &wc, 1);
rvt_cq_enter(ibcq_to_rvtcq(qp->ibqp.recv_cq), &wc, 1);
ret = 0;
bail:
return ret;
@ -136,19 +136,19 @@ bail:
*
* Can be called from interrupt level.
*/
int qib_get_rwqe(struct qib_qp *qp, int wr_id_only)
int qib_get_rwqe(struct rvt_qp *qp, int wr_id_only)
{
unsigned long flags;
struct qib_rq *rq;
struct qib_rwq *wq;
struct qib_srq *srq;
struct qib_rwqe *wqe;
struct rvt_rq *rq;
struct rvt_rwq *wq;
struct rvt_srq *srq;
struct rvt_rwqe *wqe;
void (*handler)(struct ib_event *, void *);
u32 tail;
int ret;
if (qp->ibqp.srq) {
srq = to_isrq(qp->ibqp.srq);
srq = ibsrq_to_rvtsrq(qp->ibqp.srq);
handler = srq->ibsrq.event_handler;
rq = &srq->rq;
} else {
@ -158,7 +158,7 @@ int qib_get_rwqe(struct qib_qp *qp, int wr_id_only)
}
spin_lock_irqsave(&rq->lock, flags);
if (!(ib_qib_state_ops[qp->state] & QIB_PROCESS_RECV_OK)) {
if (!(ib_rvt_state_ops[qp->state] & RVT_PROCESS_RECV_OK)) {
ret = 0;
goto unlock;
}
@ -174,7 +174,7 @@ int qib_get_rwqe(struct qib_qp *qp, int wr_id_only)
}
/* Make sure entry is read after head index is read. */
smp_rmb();
wqe = get_rwqe_ptr(rq, tail);
wqe = rvt_get_rwqe_ptr(rq, tail);
/*
* Even though we update the tail index in memory, the verbs
* consumer is not supposed to post more entries until a
@ -190,7 +190,7 @@ int qib_get_rwqe(struct qib_qp *qp, int wr_id_only)
qp->r_wr_id = wqe->wr_id;
ret = 1;
set_bit(QIB_R_WRID_VALID, &qp->r_aflags);
set_bit(RVT_R_WRID_VALID, &qp->r_aflags);
if (handler) {
u32 n;
@ -227,7 +227,7 @@ bail:
* Switch to alternate path.
* The QP s_lock should be held and interrupts disabled.
*/
void qib_migrate_qp(struct qib_qp *qp)
void qib_migrate_qp(struct rvt_qp *qp)
{
struct ib_event ev;
@ -266,7 +266,7 @@ static int gid_ok(union ib_gid *gid, __be64 gid_prefix, __be64 id)
* The s_lock will be acquired around the qib_migrate_qp() call.
*/
int qib_ruc_check_hdr(struct qib_ibport *ibp, struct qib_ib_header *hdr,
int has_grh, struct qib_qp *qp, u32 bth0)
int has_grh, struct rvt_qp *qp, u32 bth0)
{
__be64 guid;
unsigned long flags;
@ -279,7 +279,8 @@ int qib_ruc_check_hdr(struct qib_ibport *ibp, struct qib_ib_header *hdr,
if (!(qp->alt_ah_attr.ah_flags & IB_AH_GRH))
goto err;
guid = get_sguid(ibp, qp->alt_ah_attr.grh.sgid_index);
if (!gid_ok(&hdr->u.l.grh.dgid, ibp->gid_prefix, guid))
if (!gid_ok(&hdr->u.l.grh.dgid,
ibp->rvp.gid_prefix, guid))
goto err;
if (!gid_ok(&hdr->u.l.grh.sgid,
qp->alt_ah_attr.grh.dgid.global.subnet_prefix,
@ -311,7 +312,8 @@ int qib_ruc_check_hdr(struct qib_ibport *ibp, struct qib_ib_header *hdr,
goto err;
guid = get_sguid(ibp,
qp->remote_ah_attr.grh.sgid_index);
if (!gid_ok(&hdr->u.l.grh.dgid, ibp->gid_prefix, guid))
if (!gid_ok(&hdr->u.l.grh.dgid,
ibp->rvp.gid_prefix, guid))
goto err;
if (!gid_ok(&hdr->u.l.grh.sgid,
qp->remote_ah_attr.grh.dgid.global.subnet_prefix,
@ -353,12 +355,15 @@ err:
* receive interrupts since this is a connected protocol and all packets
* will pass through here.
*/
static void qib_ruc_loopback(struct qib_qp *sqp)
static void qib_ruc_loopback(struct rvt_qp *sqp)
{
struct qib_ibport *ibp = to_iport(sqp->ibqp.device, sqp->port_num);
struct qib_qp *qp;
struct qib_swqe *wqe;
struct qib_sge *sge;
struct qib_pportdata *ppd = ppd_from_ibp(ibp);
struct qib_devdata *dd = ppd->dd;
struct rvt_dev_info *rdi = &dd->verbs_dev.rdi;
struct rvt_qp *qp;
struct rvt_swqe *wqe;
struct rvt_sge *sge;
unsigned long flags;
struct ib_wc wc;
u64 sdata;
@ -367,29 +372,33 @@ static void qib_ruc_loopback(struct qib_qp *sqp)
int release;
int ret;
rcu_read_lock();
/*
* Note that we check the responder QP state after
* checking the requester's state.
*/
qp = qib_lookup_qpn(ibp, sqp->remote_qpn);
qp = rvt_lookup_qpn(rdi, &ibp->rvp, sqp->remote_qpn);
if (!qp)
goto done;
spin_lock_irqsave(&sqp->s_lock, flags);
/* Return if we are already busy processing a work request. */
if ((sqp->s_flags & (QIB_S_BUSY | QIB_S_ANY_WAIT)) ||
!(ib_qib_state_ops[sqp->state] & QIB_PROCESS_OR_FLUSH_SEND))
if ((sqp->s_flags & (RVT_S_BUSY | RVT_S_ANY_WAIT)) ||
!(ib_rvt_state_ops[sqp->state] & RVT_PROCESS_OR_FLUSH_SEND))
goto unlock;
sqp->s_flags |= QIB_S_BUSY;
sqp->s_flags |= RVT_S_BUSY;
again:
if (sqp->s_last == sqp->s_head)
smp_read_barrier_depends(); /* see post_one_send() */
if (sqp->s_last == ACCESS_ONCE(sqp->s_head))
goto clr_busy;
wqe = get_swqe_ptr(sqp, sqp->s_last);
wqe = rvt_get_swqe_ptr(sqp, sqp->s_last);
/* Return if it is not OK to start a new work reqeust. */
if (!(ib_qib_state_ops[sqp->state] & QIB_PROCESS_NEXT_SEND_OK)) {
if (!(ib_qib_state_ops[sqp->state] & QIB_FLUSH_SEND))
if (!(ib_rvt_state_ops[sqp->state] & RVT_PROCESS_NEXT_SEND_OK)) {
if (!(ib_rvt_state_ops[sqp->state] & RVT_FLUSH_SEND))
goto clr_busy;
/* We are in the error state, flush the work request. */
send_status = IB_WC_WR_FLUSH_ERR;
@ -407,9 +416,9 @@ again:
}
spin_unlock_irqrestore(&sqp->s_lock, flags);
if (!qp || !(ib_qib_state_ops[qp->state] & QIB_PROCESS_RECV_OK) ||
if (!qp || !(ib_rvt_state_ops[qp->state] & RVT_PROCESS_RECV_OK) ||
qp->ibqp.qp_type != sqp->ibqp.qp_type) {
ibp->n_pkt_drops++;
ibp->rvp.n_pkt_drops++;
/*
* For RC, the requester would timeout and retry so
* shortcut the timeouts and just signal too many retries.
@ -458,7 +467,7 @@ again:
goto inv_err;
if (wqe->length == 0)
break;
if (unlikely(!qib_rkey_ok(qp, &qp->r_sge.sge, wqe->length,
if (unlikely(!rvt_rkey_ok(qp, &qp->r_sge.sge, wqe->length,
wqe->rdma_wr.remote_addr,
wqe->rdma_wr.rkey,
IB_ACCESS_REMOTE_WRITE)))
@ -471,7 +480,7 @@ again:
case IB_WR_RDMA_READ:
if (unlikely(!(qp->qp_access_flags & IB_ACCESS_REMOTE_READ)))
goto inv_err;
if (unlikely(!qib_rkey_ok(qp, &sqp->s_sge.sge, wqe->length,
if (unlikely(!rvt_rkey_ok(qp, &sqp->s_sge.sge, wqe->length,
wqe->rdma_wr.remote_addr,
wqe->rdma_wr.rkey,
IB_ACCESS_REMOTE_READ)))
@ -489,7 +498,7 @@ again:
case IB_WR_ATOMIC_FETCH_AND_ADD:
if (unlikely(!(qp->qp_access_flags & IB_ACCESS_REMOTE_ATOMIC)))
goto inv_err;
if (unlikely(!qib_rkey_ok(qp, &qp->r_sge.sge, sizeof(u64),
if (unlikely(!rvt_rkey_ok(qp, &qp->r_sge.sge, sizeof(u64),
wqe->atomic_wr.remote_addr,
wqe->atomic_wr.rkey,
IB_ACCESS_REMOTE_ATOMIC)))
@ -502,7 +511,7 @@ again:
(u64) atomic64_add_return(sdata, maddr) - sdata :
(u64) cmpxchg((u64 *) qp->r_sge.sge.vaddr,
sdata, wqe->atomic_wr.swap);
qib_put_mr(qp->r_sge.sge.mr);
rvt_put_mr(qp->r_sge.sge.mr);
qp->r_sge.num_sge = 0;
goto send_comp;
@ -526,11 +535,11 @@ again:
sge->sge_length -= len;
if (sge->sge_length == 0) {
if (!release)
qib_put_mr(sge->mr);
rvt_put_mr(sge->mr);
if (--sqp->s_sge.num_sge)
*sge = *sqp->s_sge.sg_list++;
} else if (sge->length == 0 && sge->mr->lkey) {
if (++sge->n >= QIB_SEGSZ) {
if (++sge->n >= RVT_SEGSZ) {
if (++sge->m >= sge->mr->mapsz)
break;
sge->n = 0;
@ -543,9 +552,9 @@ again:
sqp->s_len -= len;
}
if (release)
qib_put_ss(&qp->r_sge);
rvt_put_ss(&qp->r_sge);
if (!test_and_clear_bit(QIB_R_WRID_VALID, &qp->r_aflags))
if (!test_and_clear_bit(RVT_R_WRID_VALID, &qp->r_aflags))
goto send_comp;
if (wqe->wr.opcode == IB_WR_RDMA_WRITE_WITH_IMM)
@ -561,12 +570,12 @@ again:
wc.sl = qp->remote_ah_attr.sl;
wc.port_num = 1;
/* Signal completion event if the solicited bit is set. */
qib_cq_enter(to_icq(qp->ibqp.recv_cq), &wc,
wqe->wr.send_flags & IB_SEND_SOLICITED);
rvt_cq_enter(ibcq_to_rvtcq(qp->ibqp.recv_cq), &wc,
wqe->wr.send_flags & IB_SEND_SOLICITED);
send_comp:
spin_lock_irqsave(&sqp->s_lock, flags);
ibp->n_loop_pkts++;
ibp->rvp.n_loop_pkts++;
flush_send:
sqp->s_rnr_retry = sqp->s_rnr_retry_cnt;
qib_send_complete(sqp, wqe, send_status);
@ -576,7 +585,7 @@ rnr_nak:
/* Handle RNR NAK */
if (qp->ibqp.qp_type == IB_QPT_UC)
goto send_comp;
ibp->n_rnr_naks++;
ibp->rvp.n_rnr_naks++;
/*
* Note: we don't need the s_lock held since the BUSY flag
* makes this single threaded.
@ -588,9 +597,9 @@ rnr_nak:
if (sqp->s_rnr_retry_cnt < 7)
sqp->s_rnr_retry--;
spin_lock_irqsave(&sqp->s_lock, flags);
if (!(ib_qib_state_ops[sqp->state] & QIB_PROCESS_RECV_OK))
if (!(ib_rvt_state_ops[sqp->state] & RVT_PROCESS_RECV_OK))
goto clr_busy;
sqp->s_flags |= QIB_S_WAIT_RNR;
sqp->s_flags |= RVT_S_WAIT_RNR;
sqp->s_timer.function = qib_rc_rnr_retry;
sqp->s_timer.expires = jiffies +
usecs_to_jiffies(ib_qib_rnr_table[qp->r_min_rnr_timer]);
@ -618,9 +627,9 @@ serr:
spin_lock_irqsave(&sqp->s_lock, flags);
qib_send_complete(sqp, wqe, send_status);
if (sqp->ibqp.qp_type == IB_QPT_RC) {
int lastwqe = qib_error_qp(sqp, IB_WC_WR_FLUSH_ERR);
int lastwqe = rvt_error_qp(sqp, IB_WC_WR_FLUSH_ERR);
sqp->s_flags &= ~QIB_S_BUSY;
sqp->s_flags &= ~RVT_S_BUSY;
spin_unlock_irqrestore(&sqp->s_lock, flags);
if (lastwqe) {
struct ib_event ev;
@ -633,12 +642,11 @@ serr:
goto done;
}
clr_busy:
sqp->s_flags &= ~QIB_S_BUSY;
sqp->s_flags &= ~RVT_S_BUSY;
unlock:
spin_unlock_irqrestore(&sqp->s_lock, flags);
done:
if (qp && atomic_dec_and_test(&qp->refcount))
wake_up(&qp->wait);
rcu_read_unlock();
}
/**
@ -663,7 +671,7 @@ u32 qib_make_grh(struct qib_ibport *ibp, struct ib_grh *hdr,
hdr->next_hdr = IB_GRH_NEXT_HDR;
hdr->hop_limit = grh->hop_limit;
/* The SGID is 32-bit aligned. */
hdr->sgid.global.subnet_prefix = ibp->gid_prefix;
hdr->sgid.global.subnet_prefix = ibp->rvp.gid_prefix;
hdr->sgid.global.interface_id = grh->sgid_index ?
ibp->guids[grh->sgid_index - 1] : ppd_from_ibp(ibp)->guid;
hdr->dgid = grh->dgid;
@ -672,9 +680,10 @@ u32 qib_make_grh(struct qib_ibport *ibp, struct ib_grh *hdr,
return sizeof(struct ib_grh) / sizeof(u32);
}
void qib_make_ruc_header(struct qib_qp *qp, struct qib_other_headers *ohdr,
void qib_make_ruc_header(struct rvt_qp *qp, struct qib_other_headers *ohdr,
u32 bth0, u32 bth2)
{
struct qib_qp_priv *priv = qp->priv;
struct qib_ibport *ibp = to_iport(qp->ibqp.device, qp->port_num);
u16 lrh0;
u32 nwords;
@ -685,17 +694,18 @@ void qib_make_ruc_header(struct qib_qp *qp, struct qib_other_headers *ohdr,
nwords = (qp->s_cur_size + extra_bytes) >> 2;
lrh0 = QIB_LRH_BTH;
if (unlikely(qp->remote_ah_attr.ah_flags & IB_AH_GRH)) {
qp->s_hdrwords += qib_make_grh(ibp, &qp->s_hdr->u.l.grh,
qp->s_hdrwords += qib_make_grh(ibp, &priv->s_hdr->u.l.grh,
&qp->remote_ah_attr.grh,
qp->s_hdrwords, nwords);
lrh0 = QIB_LRH_GRH;
}
lrh0 |= ibp->sl_to_vl[qp->remote_ah_attr.sl] << 12 |
qp->remote_ah_attr.sl << 4;
qp->s_hdr->lrh[0] = cpu_to_be16(lrh0);
qp->s_hdr->lrh[1] = cpu_to_be16(qp->remote_ah_attr.dlid);
qp->s_hdr->lrh[2] = cpu_to_be16(qp->s_hdrwords + nwords + SIZE_OF_CRC);
qp->s_hdr->lrh[3] = cpu_to_be16(ppd_from_ibp(ibp)->lid |
priv->s_hdr->lrh[0] = cpu_to_be16(lrh0);
priv->s_hdr->lrh[1] = cpu_to_be16(qp->remote_ah_attr.dlid);
priv->s_hdr->lrh[2] =
cpu_to_be16(qp->s_hdrwords + nwords + SIZE_OF_CRC);
priv->s_hdr->lrh[3] = cpu_to_be16(ppd_from_ibp(ibp)->lid |
qp->remote_ah_attr.src_path_bits);
bth0 |= qib_get_pkey(ibp, qp->s_pkey_index);
bth0 |= extra_bytes << 20;
@ -707,20 +717,29 @@ void qib_make_ruc_header(struct qib_qp *qp, struct qib_other_headers *ohdr,
this_cpu_inc(ibp->pmastats->n_unicast_xmit);
}
void _qib_do_send(struct work_struct *work)
{
struct qib_qp_priv *priv = container_of(work, struct qib_qp_priv,
s_work);
struct rvt_qp *qp = priv->owner;
qib_do_send(qp);
}
/**
* qib_do_send - perform a send on a QP
* @work: contains a pointer to the QP
* @qp: pointer to the QP
*
* Process entries in the send work queue until credit or queue is
* exhausted. Only allow one CPU to send a packet per QP (tasklet).
* Otherwise, two threads could send packets out of order.
*/
void qib_do_send(struct work_struct *work)
void qib_do_send(struct rvt_qp *qp)
{
struct qib_qp *qp = container_of(work, struct qib_qp, s_work);
struct qib_qp_priv *priv = qp->priv;
struct qib_ibport *ibp = to_iport(qp->ibqp.device, qp->port_num);
struct qib_pportdata *ppd = ppd_from_ibp(ibp);
int (*make_req)(struct qib_qp *qp);
int (*make_req)(struct rvt_qp *qp);
unsigned long flags;
if ((qp->ibqp.qp_type == IB_QPT_RC ||
@ -745,50 +764,59 @@ void qib_do_send(struct work_struct *work)
return;
}
qp->s_flags |= QIB_S_BUSY;
spin_unlock_irqrestore(&qp->s_lock, flags);
qp->s_flags |= RVT_S_BUSY;
do {
/* Check for a constructed packet to be sent. */
if (qp->s_hdrwords != 0) {
spin_unlock_irqrestore(&qp->s_lock, flags);
/*
* If the packet cannot be sent now, return and
* the send tasklet will be woken up later.
*/
if (qib_verbs_send(qp, qp->s_hdr, qp->s_hdrwords,
if (qib_verbs_send(qp, priv->s_hdr, qp->s_hdrwords,
qp->s_cur_sge, qp->s_cur_size))
break;
return;
/* Record that s_hdr is empty. */
qp->s_hdrwords = 0;
spin_lock_irqsave(&qp->s_lock, flags);
}
} while (make_req(qp));
spin_unlock_irqrestore(&qp->s_lock, flags);
}
/*
* This should be called with s_lock held.
*/
void qib_send_complete(struct qib_qp *qp, struct qib_swqe *wqe,
void qib_send_complete(struct rvt_qp *qp, struct rvt_swqe *wqe,
enum ib_wc_status status)
{
u32 old_last, last;
unsigned i;
if (!(ib_qib_state_ops[qp->state] & QIB_PROCESS_OR_FLUSH_SEND))
if (!(ib_rvt_state_ops[qp->state] & RVT_PROCESS_OR_FLUSH_SEND))
return;
last = qp->s_last;
old_last = last;
if (++last >= qp->s_size)
last = 0;
qp->s_last = last;
/* See post_send() */
barrier();
for (i = 0; i < wqe->wr.num_sge; i++) {
struct qib_sge *sge = &wqe->sg_list[i];
struct rvt_sge *sge = &wqe->sg_list[i];
qib_put_mr(sge->mr);
rvt_put_mr(sge->mr);
}
if (qp->ibqp.qp_type == IB_QPT_UD ||
qp->ibqp.qp_type == IB_QPT_SMI ||
qp->ibqp.qp_type == IB_QPT_GSI)
atomic_dec(&to_iah(wqe->ud_wr.ah)->refcount);
atomic_dec(&ibah_to_rvtah(wqe->ud_wr.ah)->refcount);
/* See ch. 11.2.4.1 and 10.7.3.1 */
if (!(qp->s_flags & QIB_S_SIGNAL_REQ_WR) ||
if (!(qp->s_flags & RVT_S_SIGNAL_REQ_WR) ||
(wqe->wr.send_flags & IB_SEND_SIGNALED) ||
status != IB_WC_SUCCESS) {
struct ib_wc wc;
@ -800,15 +828,10 @@ void qib_send_complete(struct qib_qp *qp, struct qib_swqe *wqe,
wc.qp = &qp->ibqp;
if (status == IB_WC_SUCCESS)
wc.byte_len = wqe->length;
qib_cq_enter(to_icq(qp->ibqp.send_cq), &wc,
rvt_cq_enter(ibcq_to_rvtcq(qp->ibqp.send_cq), &wc,
status != IB_WC_SUCCESS);
}
last = qp->s_last;
old_last = last;
if (++last >= qp->s_size)
last = 0;
qp->s_last = last;
if (qp->s_acked == old_last)
qp->s_acked = last;
if (qp->s_cur == old_last)

41
drivers/infiniband/hw/qib/qib_sdma.c
View File

@ -513,7 +513,9 @@ int qib_sdma_running(struct qib_pportdata *ppd)
static void complete_sdma_err_req(struct qib_pportdata *ppd,
struct qib_verbs_txreq *tx)
{
atomic_inc(&tx->qp->s_dma_busy);
struct qib_qp_priv *priv = tx->qp->priv;
atomic_inc(&priv->s_dma_busy);
/* no sdma descriptors, so no unmap_desc */
tx->txreq.start_idx = 0;
tx->txreq.next_descq_idx = 0;
@ -531,18 +533,19 @@ static void complete_sdma_err_req(struct qib_pportdata *ppd,
* 3) The SGE addresses are suitable for passing to dma_map_single().
*/
int qib_sdma_verbs_send(struct qib_pportdata *ppd,
struct qib_sge_state *ss, u32 dwords,
struct rvt_sge_state *ss, u32 dwords,
struct qib_verbs_txreq *tx)
{
unsigned long flags;
struct qib_sge *sge;
struct qib_qp *qp;
struct rvt_sge *sge;
struct rvt_qp *qp;
int ret = 0;
u16 tail;
__le64 *descqp;
u64 sdmadesc[2];
u32 dwoffset;
dma_addr_t addr;
struct qib_qp_priv *priv;
spin_lock_irqsave(&ppd->sdma_lock, flags);
@ -621,7 +624,7 @@ retry:
if (--ss->num_sge)
*sge = *ss->sg_list++;
} else if (sge->length == 0 && sge->mr->lkey) {
if (++sge->n >= QIB_SEGSZ) {
if (++sge->n >= RVT_SEGSZ) {
if (++sge->m >= sge->mr->mapsz)
break;
sge->n = 0;
@ -644,8 +647,8 @@ retry:
descqp[0] |= cpu_to_le64(SDMA_DESC_DMA_HEAD);
if (tx->txreq.flags & QIB_SDMA_TXREQ_F_INTREQ)
descqp[0] |= cpu_to_le64(SDMA_DESC_INTR);
atomic_inc(&tx->qp->s_dma_busy);
priv = tx->qp->priv;
atomic_inc(&priv->s_dma_busy);
tx->txreq.next_descq_idx = tail;
ppd->dd->f_sdma_update_tail(ppd, tail);
ppd->sdma_descq_added += tx->txreq.sg_count;
@ -663,13 +666,14 @@ unmap:
unmap_desc(ppd, tail);
}
qp = tx->qp;
priv = qp->priv;
qib_put_txreq(tx);
spin_lock(&qp->r_lock);
spin_lock(&qp->s_lock);
if (qp->ibqp.qp_type == IB_QPT_RC) {
/* XXX what about error sending RDMA read responses? */
if (ib_qib_state_ops[qp->state] & QIB_PROCESS_RECV_OK)
qib_error_qp(qp, IB_WC_GENERAL_ERR);
if (ib_rvt_state_ops[qp->state] & RVT_PROCESS_RECV_OK)
rvt_error_qp(qp, IB_WC_GENERAL_ERR);
} else if (qp->s_wqe)
qib_send_complete(qp, qp->s_wqe, IB_WC_GENERAL_ERR);
spin_unlock(&qp->s_lock);
@ -679,8 +683,9 @@ unmap:
busy:
qp = tx->qp;
priv = qp->priv;
spin_lock(&qp->s_lock);
if (ib_qib_state_ops[qp->state] & QIB_PROCESS_RECV_OK) {
if (ib_rvt_state_ops[qp->state] & RVT_PROCESS_RECV_OK) {
struct qib_ibdev *dev;
/*
@ -690,19 +695,19 @@ busy:
*/
tx->ss = ss;
tx->dwords = dwords;
qp->s_tx = tx;
priv->s_tx = tx;
dev = &ppd->dd->verbs_dev;
spin_lock(&dev->pending_lock);
if (list_empty(&qp->iowait)) {
spin_lock(&dev->rdi.pending_lock);
if (list_empty(&priv->iowait)) {
struct qib_ibport *ibp;
ibp = &ppd->ibport_data;
ibp->n_dmawait++;
qp->s_flags |= QIB_S_WAIT_DMA_DESC;
list_add_tail(&qp->iowait, &dev->dmawait);
ibp->rvp.n_dmawait++;
qp->s_flags |= RVT_S_WAIT_DMA_DESC;
list_add_tail(&priv->iowait, &dev->dmawait);
}
spin_unlock(&dev->pending_lock);
qp->s_flags &= ~QIB_S_BUSY;
spin_unlock(&dev->rdi.pending_lock);
qp->s_flags &= ~RVT_S_BUSY;
spin_unlock(&qp->s_lock);
ret = -EBUSY;
} else {

380
drivers/infiniband/hw/qib/qib_srq.c
View File

@ -1,380 +0,0 @@
/*
* Copyright (c) 2006, 2007, 2008, 2009 QLogic Corporation. All rights reserved.
* Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include <linux/err.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>
#include "qib_verbs.h"
/**
* qib_post_srq_receive - post a receive on a shared receive queue
* @ibsrq: the SRQ to post the receive on
* @wr: the list of work requests to post
* @bad_wr: A pointer to the first WR to cause a problem is put here
*
* This may be called from interrupt context.
*/
int qib_post_srq_receive(struct ib_srq *ibsrq, struct ib_recv_wr *wr,
struct ib_recv_wr **bad_wr)
{
struct qib_srq *srq = to_isrq(ibsrq);
struct qib_rwq *wq;
unsigned long flags;
int ret;
for (; wr; wr = wr->next) {
struct qib_rwqe *wqe;
u32 next;
int i;
if ((unsigned) wr->num_sge > srq->rq.max_sge) {
*bad_wr = wr;
ret = -EINVAL;
goto bail;
}
spin_lock_irqsave(&srq->rq.lock, flags);
wq = srq->rq.wq;
next = wq->head + 1;
if (next >= srq->rq.size)
next = 0;
if (next == wq->tail) {
spin_unlock_irqrestore(&srq->rq.lock, flags);
*bad_wr = wr;
ret = -ENOMEM;
goto bail;
}
wqe = get_rwqe_ptr(&srq->rq, wq->head);
wqe->wr_id = wr->wr_id;
wqe->num_sge = wr->num_sge;
for (i = 0; i < wr->num_sge; i++)
wqe->sg_list[i] = wr->sg_list[i];
/* Make sure queue entry is written before the head index. */
smp_wmb();
wq->head = next;
spin_unlock_irqrestore(&srq->rq.lock, flags);
}
ret = 0;
bail:
return ret;
}
/**
* qib_create_srq - create a shared receive queue
* @ibpd: the protection domain of the SRQ to create
* @srq_init_attr: the attributes of the SRQ
* @udata: data from libibverbs when creating a user SRQ
*/
struct ib_srq *qib_create_srq(struct ib_pd *ibpd,
struct ib_srq_init_attr *srq_init_attr,
struct ib_udata *udata)
{
struct qib_ibdev *dev = to_idev(ibpd->device);
struct qib_srq *srq;
u32 sz;
struct ib_srq *ret;
if (srq_init_attr->srq_type != IB_SRQT_BASIC) {
ret = ERR_PTR(-ENOSYS);
goto done;
}
if (srq_init_attr->attr.max_sge == 0 ||
srq_init_attr->attr.max_sge > ib_qib_max_srq_sges ||
srq_init_attr->attr.max_wr == 0 ||
srq_init_attr->attr.max_wr > ib_qib_max_srq_wrs) {
ret = ERR_PTR(-EINVAL);
goto done;
}
srq = kmalloc(sizeof(*srq), GFP_KERNEL);
if (!srq) {
ret = ERR_PTR(-ENOMEM);
goto done;
}
/*
* Need to use vmalloc() if we want to support large #s of entries.
*/
srq->rq.size = srq_init_attr->attr.max_wr + 1;
srq->rq.max_sge = srq_init_attr->attr.max_sge;
sz = sizeof(struct ib_sge) * srq->rq.max_sge +
sizeof(struct qib_rwqe);
srq->rq.wq = vmalloc_user(sizeof(struct qib_rwq) + srq->rq.size * sz);
if (!srq->rq.wq) {
ret = ERR_PTR(-ENOMEM);
goto bail_srq;
}
/*
* Return the address of the RWQ as the offset to mmap.
* See qib_mmap() for details.
*/
if (udata && udata->outlen >= sizeof(__u64)) {
int err;
u32 s = sizeof(struct qib_rwq) + srq->rq.size * sz;
srq->ip =
qib_create_mmap_info(dev, s, ibpd->uobject->context,
srq->rq.wq);
if (!srq->ip) {
ret = ERR_PTR(-ENOMEM);
goto bail_wq;
}
err = ib_copy_to_udata(udata, &srq->ip->offset,
sizeof(srq->ip->offset));
if (err) {
ret = ERR_PTR(err);
goto bail_ip;
}
} else
srq->ip = NULL;
/*
* ib_create_srq() will initialize srq->ibsrq.
*/
spin_lock_init(&srq->rq.lock);
srq->rq.wq->head = 0;
srq->rq.wq->tail = 0;
srq->limit = srq_init_attr->attr.srq_limit;
spin_lock(&dev->n_srqs_lock);
if (dev->n_srqs_allocated == ib_qib_max_srqs) {
spin_unlock(&dev->n_srqs_lock);
ret = ERR_PTR(-ENOMEM);
goto bail_ip;
}
dev->n_srqs_allocated++;
spin_unlock(&dev->n_srqs_lock);
if (srq->ip) {
spin_lock_irq(&dev->pending_lock);
list_add(&srq->ip->pending_mmaps, &dev->pending_mmaps);
spin_unlock_irq(&dev->pending_lock);
}
ret = &srq->ibsrq;
goto done;
bail_ip:
kfree(srq->ip);
bail_wq:
vfree(srq->rq.wq);
bail_srq:
kfree(srq);
done:
return ret;
}
/**
* qib_modify_srq - modify a shared receive queue
* @ibsrq: the SRQ to modify
* @attr: the new attributes of the SRQ
* @attr_mask: indicates which attributes to modify
* @udata: user data for libibverbs.so
*/
int qib_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
enum ib_srq_attr_mask attr_mask,
struct ib_udata *udata)
{
struct qib_srq *srq = to_isrq(ibsrq);
struct qib_rwq *wq;
int ret = 0;
if (attr_mask & IB_SRQ_MAX_WR) {
struct qib_rwq *owq;
struct qib_rwqe *p;
u32 sz, size, n, head, tail;
/* Check that the requested sizes are below the limits. */
if ((attr->max_wr > ib_qib_max_srq_wrs) ||
((attr_mask & IB_SRQ_LIMIT) ?
attr->srq_limit : srq->limit) > attr->max_wr) {
ret = -EINVAL;
goto bail;
}
sz = sizeof(struct qib_rwqe) +
srq->rq.max_sge * sizeof(struct ib_sge);
size = attr->max_wr + 1;
wq = vmalloc_user(sizeof(struct qib_rwq) + size * sz);
if (!wq) {
ret = -ENOMEM;
goto bail;
}
/* Check that we can write the offset to mmap. */
if (udata && udata->inlen >= sizeof(__u64)) {
__u64 offset_addr;
__u64 offset = 0;
ret = ib_copy_from_udata(&offset_addr, udata,
sizeof(offset_addr));
if (ret)
goto bail_free;
udata->outbuf =
(void __user *) (unsigned long) offset_addr;
ret = ib_copy_to_udata(udata, &offset,
sizeof(offset));
if (ret)
goto bail_free;
}
spin_lock_irq(&srq->rq.lock);
/*
* validate head and tail pointer values and compute
* the number of remaining WQEs.
*/
owq = srq->rq.wq;
head = owq->head;
tail = owq->tail;
if (head >= srq->rq.size || tail >= srq->rq.size) {
ret = -EINVAL;
goto bail_unlock;
}
n = head;
if (n < tail)
n += srq->rq.size - tail;
else
n -= tail;
if (size <= n) {
ret = -EINVAL;
goto bail_unlock;
}
n = 0;
p = wq->wq;
while (tail != head) {
struct qib_rwqe *wqe;
int i;
wqe = get_rwqe_ptr(&srq->rq, tail);
p->wr_id = wqe->wr_id;
p->num_sge = wqe->num_sge;
for (i = 0; i < wqe->num_sge; i++)
p->sg_list[i] = wqe->sg_list[i];
n++;
p = (struct qib_rwqe *)((char *) p + sz);
if (++tail >= srq->rq.size)
tail = 0;
}
srq->rq.wq = wq;
srq->rq.size = size;
wq->head = n;
wq->tail = 0;
if (attr_mask & IB_SRQ_LIMIT)
srq->limit = attr->srq_limit;
spin_unlock_irq(&srq->rq.lock);
vfree(owq);
if (srq->ip) {
struct qib_mmap_info *ip = srq->ip;
struct qib_ibdev *dev = to_idev(srq->ibsrq.device);
u32 s = sizeof(struct qib_rwq) + size * sz;
qib_update_mmap_info(dev, ip, s, wq);
/*
* Return the offset to mmap.
* See qib_mmap() for details.
*/
if (udata && udata->inlen >= sizeof(__u64)) {
ret = ib_copy_to_udata(udata, &ip->offset,
sizeof(ip->offset));
if (ret)
goto bail;
}
/*
* Put user mapping info onto the pending list
* unless it already is on the list.
*/
spin_lock_irq(&dev->pending_lock);
if (list_empty(&ip->pending_mmaps))
list_add(&ip->pending_mmaps,
&dev->pending_mmaps);
spin_unlock_irq(&dev->pending_lock);
}
} else if (attr_mask & IB_SRQ_LIMIT) {
spin_lock_irq(&srq->rq.lock);
if (attr->srq_limit >= srq->rq.size)
ret = -EINVAL;
else
srq->limit = attr->srq_limit;
spin_unlock_irq(&srq->rq.lock);
}
goto bail;
bail_unlock:
spin_unlock_irq(&srq->rq.lock);
bail_free:
vfree(wq);
bail:
return ret;
}
int qib_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr)
{
struct qib_srq *srq = to_isrq(ibsrq);
attr->max_wr = srq->rq.size - 1;
attr->max_sge = srq->rq.max_sge;
attr->srq_limit = srq->limit;
return 0;
}
/**
* qib_destroy_srq - destroy a shared receive queue
* @ibsrq: the SRQ to destroy
*/
int qib_destroy_srq(struct ib_srq *ibsrq)
{
struct qib_srq *srq = to_isrq(ibsrq);
struct qib_ibdev *dev = to_idev(ibsrq->device);
spin_lock(&dev->n_srqs_lock);
dev->n_srqs_allocated--;
spin_unlock(&dev->n_srqs_lock);
if (srq->ip)
kref_put(&srq->ip->ref, qib_release_mmap_info);
else
vfree(srq->rq.wq);
kfree(srq);
return 0;
}

85
drivers/infiniband/hw/qib/qib_sysfs.c
View File

@ -406,7 +406,13 @@ static struct kobj_type qib_sl2vl_ktype = {
#define QIB_DIAGC_ATTR(N) \
static struct qib_diagc_attr qib_diagc_attr_##N = { \
.attr = { .name = __stringify(N), .mode = 0664 }, \
.counter = offsetof(struct qib_ibport, n_##N) \
.counter = offsetof(struct qib_ibport, rvp.n_##N) \
}
#define QIB_DIAGC_ATTR_PER_CPU(N) \
static struct qib_diagc_attr qib_diagc_attr_##N = { \
.attr = { .name = __stringify(N), .mode = 0664 }, \
.counter = offsetof(struct qib_ibport, rvp.z_##N) \
}
struct qib_diagc_attr {
@ -414,10 +420,11 @@ struct qib_diagc_attr {
size_t counter;
};
QIB_DIAGC_ATTR_PER_CPU(rc_acks);
QIB_DIAGC_ATTR_PER_CPU(rc_qacks);
QIB_DIAGC_ATTR_PER_CPU(rc_delayed_comp);
QIB_DIAGC_ATTR(rc_resends);
QIB_DIAGC_ATTR(rc_acks);
QIB_DIAGC_ATTR(rc_qacks);
QIB_DIAGC_ATTR(rc_delayed_comp);
QIB_DIAGC_ATTR(seq_naks);
QIB_DIAGC_ATTR(rdma_seq);
QIB_DIAGC_ATTR(rnr_naks);
@ -449,6 +456,35 @@ static struct attribute *diagc_default_attributes[] = {
NULL
};
static u64 get_all_cpu_total(u64 __percpu *cntr)
{
int cpu;
u64 counter = 0;
for_each_possible_cpu(cpu)
counter += *per_cpu_ptr(cntr, cpu);
return counter;
}
#define def_write_per_cpu(cntr) \
static void write_per_cpu_##cntr(struct qib_pportdata *ppd, u32 data) \
{ \
struct qib_devdata *dd = ppd->dd; \
struct qib_ibport *qibp = &ppd->ibport_data; \
/* A write can only zero the counter */ \
if (data == 0) \
qibp->rvp.z_##cntr = get_all_cpu_total(qibp->rvp.cntr); \
else \
qib_dev_err(dd, "Per CPU cntrs can only be zeroed"); \
}
def_write_per_cpu(rc_acks)
def_write_per_cpu(rc_qacks)
def_write_per_cpu(rc_delayed_comp)
#define READ_PER_CPU_CNTR(cntr) (get_all_cpu_total(qibp->rvp.cntr) - \
qibp->rvp.z_##cntr)
static ssize_t diagc_attr_show(struct kobject *kobj, struct attribute *attr,
char *buf)
{
@ -458,7 +494,16 @@ static ssize_t diagc_attr_show(struct kobject *kobj, struct attribute *attr,
container_of(kobj, struct qib_pportdata, diagc_kobj);
struct qib_ibport *qibp = &ppd->ibport_data;
return sprintf(buf, "%u\n", *(u32 *)((char *)qibp + dattr->counter));
if (!strncmp(dattr->attr.name, "rc_acks", 7))
return sprintf(buf, "%llu\n", READ_PER_CPU_CNTR(rc_acks));
else if (!strncmp(dattr->attr.name, "rc_qacks", 8))
return sprintf(buf, "%llu\n", READ_PER_CPU_CNTR(rc_qacks));
else if (!strncmp(dattr->attr.name, "rc_delayed_comp", 15))
return sprintf(buf, "%llu\n",
READ_PER_CPU_CNTR(rc_delayed_comp));
else
return sprintf(buf, "%u\n",
*(u32 *)((char *)qibp + dattr->counter));
}
static ssize_t diagc_attr_store(struct kobject *kobj, struct attribute *attr,
@ -475,7 +520,15 @@ static ssize_t diagc_attr_store(struct kobject *kobj, struct attribute *attr,
ret = kstrtou32(buf, 0, &val);
if (ret)
return ret;
*(u32 *)((char *) qibp + dattr->counter) = val;
if (!strncmp(dattr->attr.name, "rc_acks", 7))
write_per_cpu_rc_acks(ppd, val);
else if (!strncmp(dattr->attr.name, "rc_qacks", 8))
write_per_cpu_rc_qacks(ppd, val);
else if (!strncmp(dattr->attr.name, "rc_delayed_comp", 15))
write_per_cpu_rc_delayed_comp(ppd, val);
else
*(u32 *)((char *)qibp + dattr->counter) = val;
return size;
}
@ -502,7 +555,7 @@ static ssize_t show_rev(struct device *device, struct device_attribute *attr,
char *buf)
{
struct qib_ibdev *dev =
container_of(device, struct qib_ibdev, ibdev.dev);
container_of(device, struct qib_ibdev, rdi.ibdev.dev);
return sprintf(buf, "%x\n", dd_from_dev(dev)->minrev);
}
@ -511,7 +564,7 @@ static ssize_t show_hca(struct device *device, struct device_attribute *attr,
char *buf)
{
struct qib_ibdev *dev =
container_of(device, struct qib_ibdev, ibdev.dev);
container_of(device, struct qib_ibdev, rdi.ibdev.dev);
struct qib_devdata *dd = dd_from_dev(dev);
int ret;
@ -533,7 +586,7 @@ static ssize_t show_boardversion(struct device *device,
struct device_attribute *attr, char *buf)
{
struct qib_ibdev *dev =
container_of(device, struct qib_ibdev, ibdev.dev);
container_of(device, struct qib_ibdev, rdi.ibdev.dev);
struct qib_devdata *dd = dd_from_dev(dev);
/* The string printed here is already newline-terminated. */
@ -545,7 +598,7 @@ static ssize_t show_localbus_info(struct device *device,
struct device_attribute *attr, char *buf)
{
struct qib_ibdev *dev =
container_of(device, struct qib_ibdev, ibdev.dev);
container_of(device, struct qib_ibdev, rdi.ibdev.dev);
struct qib_devdata *dd = dd_from_dev(dev);
/* The string printed here is already newline-terminated. */
@ -557,7 +610,7 @@ static ssize_t show_nctxts(struct device *device,
struct device_attribute *attr, char *buf)
{
struct qib_ibdev *dev =
container_of(device, struct qib_ibdev, ibdev.dev);
container_of(device, struct qib_ibdev, rdi.ibdev.dev);
struct qib_devdata *dd = dd_from_dev(dev);
/* Return the number of user ports (contexts) available. */
@ -572,7 +625,7 @@ static ssize_t show_nfreectxts(struct device *device,
struct device_attribute *attr, char *buf)
{
struct qib_ibdev *dev =
container_of(device, struct qib_ibdev, ibdev.dev);
container_of(device, struct qib_ibdev, rdi.ibdev.dev);
struct qib_devdata *dd = dd_from_dev(dev);
/* Return the number of free user ports (contexts) available. */
@ -583,7 +636,7 @@ static ssize_t show_serial(struct device *device,
struct device_attribute *attr, char *buf)
{
struct qib_ibdev *dev =
container_of(device, struct qib_ibdev, ibdev.dev);
container_of(device, struct qib_ibdev, rdi.ibdev.dev);
struct qib_devdata *dd = dd_from_dev(dev);
buf[sizeof(dd->serial)] = '\0';
@ -597,7 +650,7 @@ static ssize_t store_chip_reset(struct device *device,
size_t count)
{
struct qib_ibdev *dev =
container_of(device, struct qib_ibdev, ibdev.dev);
container_of(device, struct qib_ibdev, rdi.ibdev.dev);
struct qib_devdata *dd = dd_from_dev(dev);
int ret;
@ -618,7 +671,7 @@ static ssize_t show_tempsense(struct device *device,
struct device_attribute *attr, char *buf)
{
struct qib_ibdev *dev =
container_of(device, struct qib_ibdev, ibdev.dev);
container_of(device, struct qib_ibdev, rdi.ibdev.dev);
struct qib_devdata *dd = dd_from_dev(dev);
int ret;
int idx;
@ -778,7 +831,7 @@ bail:
*/
int qib_verbs_register_sysfs(struct qib_devdata *dd)
{
struct ib_device *dev = &dd->verbs_dev.ibdev;
struct ib_device *dev = &dd->verbs_dev.rdi.ibdev;
int i, ret;
for (i = 0; i < ARRAY_SIZE(qib_attributes); ++i) {

79
drivers/infiniband/hw/qib/qib_uc.c
View File

@ -41,61 +41,62 @@
* qib_make_uc_req - construct a request packet (SEND, RDMA write)
* @qp: a pointer to the QP
*
* Assumes the s_lock is held.
*
* Return 1 if constructed; otherwise, return 0.
*/
int qib_make_uc_req(struct qib_qp *qp)
int qib_make_uc_req(struct rvt_qp *qp)
{
struct qib_qp_priv *priv = qp->priv;
struct qib_other_headers *ohdr;
struct qib_swqe *wqe;
unsigned long flags;
struct rvt_swqe *wqe;
u32 hwords;
u32 bth0;
u32 len;
u32 pmtu = qp->pmtu;
int ret = 0;
spin_lock_irqsave(&qp->s_lock, flags);
if (!(ib_qib_state_ops[qp->state] & QIB_PROCESS_SEND_OK)) {
if (!(ib_qib_state_ops[qp->state] & QIB_FLUSH_SEND))
if (!(ib_rvt_state_ops[qp->state] & RVT_PROCESS_SEND_OK)) {
if (!(ib_rvt_state_ops[qp->state] & RVT_FLUSH_SEND))
goto bail;
/* We are in the error state, flush the work request. */
if (qp->s_last == qp->s_head)
smp_read_barrier_depends(); /* see post_one_send() */
if (qp->s_last == ACCESS_ONCE(qp->s_head))
goto bail;
/* If DMAs are in progress, we can't flush immediately. */
if (atomic_read(&qp->s_dma_busy)) {
qp->s_flags |= QIB_S_WAIT_DMA;
if (atomic_read(&priv->s_dma_busy)) {
qp->s_flags |= RVT_S_WAIT_DMA;
goto bail;
}
wqe = get_swqe_ptr(qp, qp->s_last);
wqe = rvt_get_swqe_ptr(qp, qp->s_last);
qib_send_complete(qp, wqe, IB_WC_WR_FLUSH_ERR);
goto done;
}
ohdr = &qp->s_hdr->u.oth;
ohdr = &priv->s_hdr->u.oth;
if (qp->remote_ah_attr.ah_flags & IB_AH_GRH)
ohdr = &qp->s_hdr->u.l.oth;
ohdr = &priv->s_hdr->u.l.oth;
/* header size in 32-bit words LRH+BTH = (8+12)/4. */
hwords = 5;
bth0 = 0;
/* Get the next send request. */
wqe = get_swqe_ptr(qp, qp->s_cur);
wqe = rvt_get_swqe_ptr(qp, qp->s_cur);
qp->s_wqe = NULL;
switch (qp->s_state) {
default:
if (!(ib_qib_state_ops[qp->state] &
QIB_PROCESS_NEXT_SEND_OK))
if (!(ib_rvt_state_ops[qp->state] &
RVT_PROCESS_NEXT_SEND_OK))
goto bail;
/* Check if send work queue is empty. */
if (qp->s_cur == qp->s_head)
smp_read_barrier_depends(); /* see post_one_send() */
if (qp->s_cur == ACCESS_ONCE(qp->s_head))
goto bail;
/*
* Start a new request.
*/
wqe->psn = qp->s_next_psn;
qp->s_psn = qp->s_next_psn;
qp->s_psn = wqe->psn;
qp->s_sge.sge = wqe->sg_list[0];
qp->s_sge.sg_list = wqe->sg_list + 1;
qp->s_sge.num_sge = wqe->wr.num_sge;
@ -214,15 +215,11 @@ int qib_make_uc_req(struct qib_qp *qp)
qp->s_cur_sge = &qp->s_sge;
qp->s_cur_size = len;
qib_make_ruc_header(qp, ohdr, bth0 | (qp->s_state << 24),
qp->s_next_psn++ & QIB_PSN_MASK);
qp->s_psn++ & QIB_PSN_MASK);
done:
ret = 1;
goto unlock;
return 1;
bail:
qp->s_flags &= ~QIB_S_BUSY;
unlock:
spin_unlock_irqrestore(&qp->s_lock, flags);
qp->s_flags &= ~RVT_S_BUSY;
return ret;
}
@ -240,7 +237,7 @@ unlock:
* Called at interrupt level.
*/
void qib_uc_rcv(struct qib_ibport *ibp, struct qib_ib_header *hdr,
int has_grh, void *data, u32 tlen, struct qib_qp *qp)
int has_grh, void *data, u32 tlen, struct rvt_qp *qp)
{
struct qib_other_headers *ohdr;
u32 opcode;
@ -278,10 +275,10 @@ void qib_uc_rcv(struct qib_ibport *ibp, struct qib_ib_header *hdr,
inv:
if (qp->r_state == OP(SEND_FIRST) ||
qp->r_state == OP(SEND_MIDDLE)) {
set_bit(QIB_R_REWIND_SGE, &qp->r_aflags);
set_bit(RVT_R_REWIND_SGE, &qp->r_aflags);
qp->r_sge.num_sge = 0;
} else
qib_put_ss(&qp->r_sge);
rvt_put_ss(&qp->r_sge);
qp->r_state = OP(SEND_LAST);
switch (opcode) {
case OP(SEND_FIRST):
@ -328,8 +325,8 @@ inv:
goto inv;
}
if (qp->state == IB_QPS_RTR && !(qp->r_flags & QIB_R_COMM_EST)) {
qp->r_flags |= QIB_R_COMM_EST;
if (qp->state == IB_QPS_RTR && !(qp->r_flags & RVT_R_COMM_EST)) {
qp->r_flags |= RVT_R_COMM_EST;
if (qp->ibqp.event_handler) {
struct ib_event ev;
@ -346,7 +343,7 @@ inv:
case OP(SEND_ONLY):
case OP(SEND_ONLY_WITH_IMMEDIATE):
send_first:
if (test_and_clear_bit(QIB_R_REWIND_SGE, &qp->r_aflags))
if (test_and_clear_bit(RVT_R_REWIND_SGE, &qp->r_aflags))
qp->r_sge = qp->s_rdma_read_sge;
else {
ret = qib_get_rwqe(qp, 0);
@ -400,7 +397,7 @@ send_last:
goto rewind;
wc.opcode = IB_WC_RECV;
qib_copy_sge(&qp->r_sge, data, tlen, 0);
qib_put_ss(&qp->s_rdma_read_sge);
rvt_put_ss(&qp->s_rdma_read_sge);
last_imm:
wc.wr_id = qp->r_wr_id;
wc.status = IB_WC_SUCCESS;
@ -414,7 +411,7 @@ last_imm:
wc.dlid_path_bits = 0;
wc.port_num = 0;
/* Signal completion event if the solicited bit is set. */
qib_cq_enter(to_icq(qp->ibqp.recv_cq), &wc,
rvt_cq_enter(ibcq_to_rvtcq(qp->ibqp.recv_cq), &wc,
(ohdr->bth[0] &
cpu_to_be32(IB_BTH_SOLICITED)) != 0);
break;
@ -438,7 +435,7 @@ rdma_first:
int ok;
/* Check rkey */
ok = qib_rkey_ok(qp, &qp->r_sge.sge, qp->r_len,
ok = rvt_rkey_ok(qp, &qp->r_sge.sge, qp->r_len,
vaddr, rkey, IB_ACCESS_REMOTE_WRITE);
if (unlikely(!ok))
goto drop;
@ -483,8 +480,8 @@ rdma_last_imm:
tlen -= (hdrsize + pad + 4);
if (unlikely(tlen + qp->r_rcv_len != qp->r_len))
goto drop;
if (test_and_clear_bit(QIB_R_REWIND_SGE, &qp->r_aflags))
qib_put_ss(&qp->s_rdma_read_sge);
if (test_and_clear_bit(RVT_R_REWIND_SGE, &qp->r_aflags))
rvt_put_ss(&qp->s_rdma_read_sge);
else {
ret = qib_get_rwqe(qp, 1);
if (ret < 0)
@ -495,7 +492,7 @@ rdma_last_imm:
wc.byte_len = qp->r_len;
wc.opcode = IB_WC_RECV_RDMA_WITH_IMM;
qib_copy_sge(&qp->r_sge, data, tlen, 1);
qib_put_ss(&qp->r_sge);
rvt_put_ss(&qp->r_sge);
goto last_imm;
case OP(RDMA_WRITE_LAST):
@ -511,7 +508,7 @@ rdma_last:
if (unlikely(tlen + qp->r_rcv_len != qp->r_len))
goto drop;
qib_copy_sge(&qp->r_sge, data, tlen, 1);
qib_put_ss(&qp->r_sge);
rvt_put_ss(&qp->r_sge);
break;
default:
@ -523,10 +520,10 @@ rdma_last:
return;
rewind:
set_bit(QIB_R_REWIND_SGE, &qp->r_aflags);
set_bit(RVT_R_REWIND_SGE, &qp->r_aflags);
qp->r_sge.num_sge = 0;
drop:
ibp->n_pkt_drops++;
ibp->rvp.n_pkt_drops++;
return;
op_err:

142
drivers/infiniband/hw/qib/qib_ud.c
View File

@ -32,6 +32,7 @@
*/
#include <rdma/ib_smi.h>
#include <rdma/ib_verbs.h>
#include "qib.h"
#include "qib_mad.h"
@ -46,22 +47,26 @@
* Note that the receive interrupt handler may be calling qib_ud_rcv()
* while this is being called.
*/
static void qib_ud_loopback(struct qib_qp *sqp, struct qib_swqe *swqe)
static void qib_ud_loopback(struct rvt_qp *sqp, struct rvt_swqe *swqe)
{
struct qib_ibport *ibp = to_iport(sqp->ibqp.device, sqp->port_num);
struct qib_pportdata *ppd;
struct qib_qp *qp;
struct qib_pportdata *ppd = ppd_from_ibp(ibp);
struct qib_devdata *dd = ppd->dd;
struct rvt_dev_info *rdi = &dd->verbs_dev.rdi;
struct rvt_qp *qp;
struct ib_ah_attr *ah_attr;
unsigned long flags;
struct qib_sge_state ssge;
struct qib_sge *sge;
struct rvt_sge_state ssge;
struct rvt_sge *sge;
struct ib_wc wc;
u32 length;
enum ib_qp_type sqptype, dqptype;
qp = qib_lookup_qpn(ibp, swqe->ud_wr.remote_qpn);
rcu_read_lock();
qp = rvt_lookup_qpn(rdi, &ibp->rvp, swqe->ud_wr.remote_qpn);
if (!qp) {
ibp->n_pkt_drops++;
ibp->rvp.n_pkt_drops++;
rcu_read_unlock();
return;
}
@ -71,12 +76,12 @@ static void qib_ud_loopback(struct qib_qp *sqp, struct qib_swqe *swqe)
IB_QPT_UD : qp->ibqp.qp_type;
if (dqptype != sqptype ||
!(ib_qib_state_ops[qp->state] & QIB_PROCESS_RECV_OK)) {
ibp->n_pkt_drops++;
!(ib_rvt_state_ops[qp->state] & RVT_PROCESS_RECV_OK)) {
ibp->rvp.n_pkt_drops++;
goto drop;
}
ah_attr = &to_iah(swqe->ud_wr.ah)->attr;
ah_attr = &ibah_to_rvtah(swqe->ud_wr.ah)->attr;
ppd = ppd_from_ibp(ibp);
if (qp->ibqp.qp_num > 1) {
@ -140,8 +145,8 @@ static void qib_ud_loopback(struct qib_qp *sqp, struct qib_swqe *swqe)
/*
* Get the next work request entry to find where to put the data.
*/
if (qp->r_flags & QIB_R_REUSE_SGE)
qp->r_flags &= ~QIB_R_REUSE_SGE;
if (qp->r_flags & RVT_R_REUSE_SGE)
qp->r_flags &= ~RVT_R_REUSE_SGE;
else {
int ret;
@ -152,14 +157,14 @@ static void qib_ud_loopback(struct qib_qp *sqp, struct qib_swqe *swqe)
}
if (!ret) {
if (qp->ibqp.qp_num == 0)
ibp->n_vl15_dropped++;
ibp->rvp.n_vl15_dropped++;
goto bail_unlock;
}
}
/* Silently drop packets which are too big. */
if (unlikely(wc.byte_len > qp->r_len)) {
qp->r_flags |= QIB_R_REUSE_SGE;
ibp->n_pkt_drops++;
qp->r_flags |= RVT_R_REUSE_SGE;
ibp->rvp.n_pkt_drops++;
goto bail_unlock;
}
@ -189,7 +194,7 @@ static void qib_ud_loopback(struct qib_qp *sqp, struct qib_swqe *swqe)
if (--ssge.num_sge)
*sge = *ssge.sg_list++;
} else if (sge->length == 0 && sge->mr->lkey) {
if (++sge->n >= QIB_SEGSZ) {
if (++sge->n >= RVT_SEGSZ) {
if (++sge->m >= sge->mr->mapsz)
break;
sge->n = 0;
@ -201,8 +206,8 @@ static void qib_ud_loopback(struct qib_qp *sqp, struct qib_swqe *swqe)
}
length -= len;
}
qib_put_ss(&qp->r_sge);
if (!test_and_clear_bit(QIB_R_WRID_VALID, &qp->r_aflags))
rvt_put_ss(&qp->r_sge);
if (!test_and_clear_bit(RVT_R_WRID_VALID, &qp->r_aflags))
goto bail_unlock;
wc.wr_id = qp->r_wr_id;
wc.status = IB_WC_SUCCESS;
@ -216,30 +221,31 @@ static void qib_ud_loopback(struct qib_qp *sqp, struct qib_swqe *swqe)
wc.dlid_path_bits = ah_attr->dlid & ((1 << ppd->lmc) - 1);
wc.port_num = qp->port_num;
/* Signal completion event if the solicited bit is set. */
qib_cq_enter(to_icq(qp->ibqp.recv_cq), &wc,
rvt_cq_enter(ibcq_to_rvtcq(qp->ibqp.recv_cq), &wc,
swqe->wr.send_flags & IB_SEND_SOLICITED);
ibp->n_loop_pkts++;
ibp->rvp.n_loop_pkts++;
bail_unlock:
spin_unlock_irqrestore(&qp->r_lock, flags);
drop:
if (atomic_dec_and_test(&qp->refcount))
wake_up(&qp->wait);
rcu_read_unlock();
}
/**
* qib_make_ud_req - construct a UD request packet
* @qp: the QP
*
* Assumes the s_lock is held.
*
* Return 1 if constructed; otherwise, return 0.
*/
int qib_make_ud_req(struct qib_qp *qp)
int qib_make_ud_req(struct rvt_qp *qp)
{
struct qib_qp_priv *priv = qp->priv;
struct qib_other_headers *ohdr;
struct ib_ah_attr *ah_attr;
struct qib_pportdata *ppd;
struct qib_ibport *ibp;
struct qib_swqe *wqe;
unsigned long flags;
struct rvt_swqe *wqe;
u32 nwords;
u32 extra_bytes;
u32 bth0;
@ -248,28 +254,29 @@ int qib_make_ud_req(struct qib_qp *qp)
int ret = 0;
int next_cur;
spin_lock_irqsave(&qp->s_lock, flags);
if (!(ib_qib_state_ops[qp->state] & QIB_PROCESS_NEXT_SEND_OK)) {
if (!(ib_qib_state_ops[qp->state] & QIB_FLUSH_SEND))
if (!(ib_rvt_state_ops[qp->state] & RVT_PROCESS_NEXT_SEND_OK)) {
if (!(ib_rvt_state_ops[qp->state] & RVT_FLUSH_SEND))
goto bail;
/* We are in the error state, flush the work request. */
if (qp->s_last == qp->s_head)
smp_read_barrier_depends(); /* see post_one_send */
if (qp->s_last == ACCESS_ONCE(qp->s_head))
goto bail;
/* If DMAs are in progress, we can't flush immediately. */
if (atomic_read(&qp->s_dma_busy)) {
qp->s_flags |= QIB_S_WAIT_DMA;
if (atomic_read(&priv->s_dma_busy)) {
qp->s_flags |= RVT_S_WAIT_DMA;
goto bail;
}
wqe = get_swqe_ptr(qp, qp->s_last);
wqe = rvt_get_swqe_ptr(qp, qp->s_last);
qib_send_complete(qp, wqe, IB_WC_WR_FLUSH_ERR);
goto done;
}
if (qp->s_cur == qp->s_head)
/* see post_one_send() */
smp_read_barrier_depends();
if (qp->s_cur == ACCESS_ONCE(qp->s_head))
goto bail;
wqe = get_swqe_ptr(qp, qp->s_cur);
wqe = rvt_get_swqe_ptr(qp, qp->s_cur);
next_cur = qp->s_cur + 1;
if (next_cur >= qp->s_size)
next_cur = 0;
@ -277,9 +284,9 @@ int qib_make_ud_req(struct qib_qp *qp)
/* Construct the header. */
ibp = to_iport(qp->ibqp.device, qp->port_num);
ppd = ppd_from_ibp(ibp);
ah_attr = &to_iah(wqe->ud_wr.ah)->attr;
if (ah_attr->dlid >= QIB_MULTICAST_LID_BASE) {
if (ah_attr->dlid != QIB_PERMISSIVE_LID)
ah_attr = &ibah_to_rvtah(wqe->ud_wr.ah)->attr;
if (ah_attr->dlid >= be16_to_cpu(IB_MULTICAST_LID_BASE)) {
if (ah_attr->dlid != be16_to_cpu(IB_LID_PERMISSIVE))
this_cpu_inc(ibp->pmastats->n_multicast_xmit);
else
this_cpu_inc(ibp->pmastats->n_unicast_xmit);
@ -287,6 +294,7 @@ int qib_make_ud_req(struct qib_qp *qp)
this_cpu_inc(ibp->pmastats->n_unicast_xmit);
lid = ah_attr->dlid & ~((1 << ppd->lmc) - 1);
if (unlikely(lid == ppd->lid)) {
unsigned long flags;
/*
* If DMAs are in progress, we can't generate
* a completion for the loopback packet since
@ -294,11 +302,12 @@ int qib_make_ud_req(struct qib_qp *qp)
* XXX Instead of waiting, we could queue a
* zero length descriptor so we get a callback.
*/
if (atomic_read(&qp->s_dma_busy)) {
qp->s_flags |= QIB_S_WAIT_DMA;
if (atomic_read(&priv->s_dma_busy)) {
qp->s_flags |= RVT_S_WAIT_DMA;
goto bail;
}
qp->s_cur = next_cur;
local_irq_save(flags);
spin_unlock_irqrestore(&qp->s_lock, flags);
qib_ud_loopback(qp, wqe);
spin_lock_irqsave(&qp->s_lock, flags);
@ -324,11 +333,11 @@ int qib_make_ud_req(struct qib_qp *qp)
if (ah_attr->ah_flags & IB_AH_GRH) {
/* Header size in 32-bit words. */
qp->s_hdrwords += qib_make_grh(ibp, &qp->s_hdr->u.l.grh,
qp->s_hdrwords += qib_make_grh(ibp, &priv->s_hdr->u.l.grh,
&ah_attr->grh,
qp->s_hdrwords, nwords);
lrh0 = QIB_LRH_GRH;
ohdr = &qp->s_hdr->u.l.oth;
ohdr = &priv->s_hdr->u.l.oth;
/*
* Don't worry about sending to locally attached multicast
* QPs. It is unspecified by the spec. what happens.
@ -336,7 +345,7 @@ int qib_make_ud_req(struct qib_qp *qp)
} else {
/* Header size in 32-bit words. */
lrh0 = QIB_LRH_BTH;
ohdr = &qp->s_hdr->u.oth;
ohdr = &priv->s_hdr->u.oth;
}
if (wqe->wr.opcode == IB_WR_SEND_WITH_IMM) {
qp->s_hdrwords++;
@ -349,15 +358,16 @@ int qib_make_ud_req(struct qib_qp *qp)
lrh0 |= 0xF000; /* Set VL (see ch. 13.5.3.1) */
else
lrh0 |= ibp->sl_to_vl[ah_attr->sl] << 12;
qp->s_hdr->lrh[0] = cpu_to_be16(lrh0);
qp->s_hdr->lrh[1] = cpu_to_be16(ah_attr->dlid); /* DEST LID */
qp->s_hdr->lrh[2] = cpu_to_be16(qp->s_hdrwords + nwords + SIZE_OF_CRC);
priv->s_hdr->lrh[0] = cpu_to_be16(lrh0);
priv->s_hdr->lrh[1] = cpu_to_be16(ah_attr->dlid); /* DEST LID */
priv->s_hdr->lrh[2] =
cpu_to_be16(qp->s_hdrwords + nwords + SIZE_OF_CRC);
lid = ppd->lid;
if (lid) {
lid |= ah_attr->src_path_bits & ((1 << ppd->lmc) - 1);
qp->s_hdr->lrh[3] = cpu_to_be16(lid);
priv->s_hdr->lrh[3] = cpu_to_be16(lid);
} else
qp->s_hdr->lrh[3] = IB_LID_PERMISSIVE;
priv->s_hdr->lrh[3] = IB_LID_PERMISSIVE;
if (wqe->wr.send_flags & IB_SEND_SOLICITED)
bth0 |= IB_BTH_SOLICITED;
bth0 |= extra_bytes << 20;
@ -368,11 +378,11 @@ int qib_make_ud_req(struct qib_qp *qp)
/*
* Use the multicast QP if the destination LID is a multicast LID.
*/
ohdr->bth[1] = ah_attr->dlid >= QIB_MULTICAST_LID_BASE &&
ah_attr->dlid != QIB_PERMISSIVE_LID ?
ohdr->bth[1] = ah_attr->dlid >= be16_to_cpu(IB_MULTICAST_LID_BASE) &&
ah_attr->dlid != be16_to_cpu(IB_LID_PERMISSIVE) ?
cpu_to_be32(QIB_MULTICAST_QPN) :
cpu_to_be32(wqe->ud_wr.remote_qpn);
ohdr->bth[2] = cpu_to_be32(qp->s_next_psn++ & QIB_PSN_MASK);
ohdr->bth[2] = cpu_to_be32(wqe->psn & QIB_PSN_MASK);
/*
* Qkeys with the high order bit set mean use the
* qkey from the QP context instead of the WR (see 10.2.5).
@ -382,13 +392,9 @@ int qib_make_ud_req(struct qib_qp *qp)
ohdr->u.ud.deth[1] = cpu_to_be32(qp->ibqp.qp_num);
done:
ret = 1;
goto unlock;
return 1;
bail:
qp->s_flags &= ~QIB_S_BUSY;
unlock:
spin_unlock_irqrestore(&qp->s_lock, flags);
qp->s_flags &= ~RVT_S_BUSY;
return ret;
}
@ -426,7 +432,7 @@ static unsigned qib_lookup_pkey(struct qib_ibport *ibp, u16 pkey)
* Called at interrupt level.
*/
void qib_ud_rcv(struct qib_ibport *ibp, struct qib_ib_header *hdr,
int has_grh, void *data, u32 tlen, struct qib_qp *qp)
int has_grh, void *data, u32 tlen, struct rvt_qp *qp)
{
struct qib_other_headers *ohdr;
int opcode;
@ -446,7 +452,7 @@ void qib_ud_rcv(struct qib_ibport *ibp, struct qib_ib_header *hdr,
hdrsize = 8 + 40 + 12 + 8; /* LRH + GRH + BTH + DETH */
}
qkey = be32_to_cpu(ohdr->u.ud.deth[0]);
src_qp = be32_to_cpu(ohdr->u.ud.deth[1]) & QIB_QPN_MASK;
src_qp = be32_to_cpu(ohdr->u.ud.deth[1]) & RVT_QPN_MASK;
/*
* Get the number of bytes the message was padded by
@ -531,8 +537,8 @@ void qib_ud_rcv(struct qib_ibport *ibp, struct qib_ib_header *hdr,
/*
* Get the next work request entry to find where to put the data.
*/
if (qp->r_flags & QIB_R_REUSE_SGE)
qp->r_flags &= ~QIB_R_REUSE_SGE;
if (qp->r_flags & RVT_R_REUSE_SGE)
qp->r_flags &= ~RVT_R_REUSE_SGE;
else {
int ret;
@ -543,13 +549,13 @@ void qib_ud_rcv(struct qib_ibport *ibp, struct qib_ib_header *hdr,
}
if (!ret) {
if (qp->ibqp.qp_num == 0)
ibp->n_vl15_dropped++;
ibp->rvp.n_vl15_dropped++;
return;
}
}
/* Silently drop packets which are too big. */
if (unlikely(wc.byte_len > qp->r_len)) {
qp->r_flags |= QIB_R_REUSE_SGE;
qp->r_flags |= RVT_R_REUSE_SGE;
goto drop;
}
if (has_grh) {
@ -559,8 +565,8 @@ void qib_ud_rcv(struct qib_ibport *ibp, struct qib_ib_header *hdr,
} else
qib_skip_sge(&qp->r_sge, sizeof(struct ib_grh), 1);
qib_copy_sge(&qp->r_sge, data, wc.byte_len - sizeof(struct ib_grh), 1);
qib_put_ss(&qp->r_sge);
if (!test_and_clear_bit(QIB_R_WRID_VALID, &qp->r_aflags))
rvt_put_ss(&qp->r_sge);
if (!test_and_clear_bit(RVT_R_WRID_VALID, &qp->r_aflags))
return;
wc.wr_id = qp->r_wr_id;
wc.status = IB_WC_SUCCESS;
@ -576,15 +582,15 @@ void qib_ud_rcv(struct qib_ibport *ibp, struct qib_ib_header *hdr,
/*
* Save the LMC lower bits if the destination LID is a unicast LID.
*/
wc.dlid_path_bits = dlid >= QIB_MULTICAST_LID_BASE ? 0 :
wc.dlid_path_bits = dlid >= be16_to_cpu(IB_MULTICAST_LID_BASE) ? 0 :
dlid & ((1 << ppd_from_ibp(ibp)->lmc) - 1);
wc.port_num = qp->port_num;
/* Signal completion event if the solicited bit is set. */
qib_cq_enter(to_icq(qp->ibqp.recv_cq), &wc,
rvt_cq_enter(ibcq_to_rvtcq(qp->ibqp.recv_cq), &wc,
(ohdr->bth[0] &
cpu_to_be32(IB_BTH_SOLICITED)) != 0);
return;
drop:
ibp->n_pkt_drops++;
ibp->rvp.n_pkt_drops++;
}

1221
drivers/infiniband/hw/qib/qib_verbs.c
View File

File diff suppressed because it is too large Load Diff

810
drivers/infiniband/hw/qib/qib_verbs.h
View File

@ -45,6 +45,8 @@
#include <linux/completion.h>
#include <rdma/ib_pack.h>
#include <rdma/ib_user_verbs.h>
#include <rdma/rdma_vt.h>
#include <rdma/rdmavt_cq.h>
struct qib_ctxtdata;
struct qib_pportdata;
@ -53,9 +55,7 @@ struct qib_verbs_txreq;
#define QIB_MAX_RDMA_ATOMIC 16
#define QIB_GUIDS_PER_PORT 5
#define QPN_MAX (1 << 24)
#define QPNMAP_ENTRIES (QPN_MAX / PAGE_SIZE / BITS_PER_BYTE)
#define QIB_PSN_SHIFT 8
/*
* Increment this value if any changes that break userspace ABI
@ -63,12 +63,6 @@ struct qib_verbs_txreq;
*/
#define QIB_UVERBS_ABI_VERSION 2
/*
* Define an ib_cq_notify value that is not valid so we know when CQ
* notifications are armed.
*/
#define IB_CQ_NONE (IB_CQ_NEXT_COMP + 1)
#define IB_SEQ_NAK (3 << 29)
/* AETH NAK opcode values */
@ -79,17 +73,6 @@ struct qib_verbs_txreq;
#define IB_NAK_REMOTE_OPERATIONAL_ERROR 0x63
#define IB_NAK_INVALID_RD_REQUEST 0x64
/* Flags for checking QP state (see ib_qib_state_ops[]) */
#define QIB_POST_SEND_OK 0x01
#define QIB_POST_RECV_OK 0x02
#define QIB_PROCESS_RECV_OK 0x04
#define QIB_PROCESS_SEND_OK 0x08
#define QIB_PROCESS_NEXT_SEND_OK 0x10
#define QIB_FLUSH_SEND 0x20
#define QIB_FLUSH_RECV 0x40
#define QIB_PROCESS_OR_FLUSH_SEND \
(QIB_PROCESS_SEND_OK | QIB_FLUSH_SEND)
/* IB Performance Manager status values */
#define IB_PMA_SAMPLE_STATUS_DONE 0x00
#define IB_PMA_SAMPLE_STATUS_STARTED 0x01
@ -203,468 +186,21 @@ struct qib_pio_header {
} __packed;
/*
* There is one struct qib_mcast for each multicast GID.
* All attached QPs are then stored as a list of
* struct qib_mcast_qp.
* qib specific data structure that will be hidden from rvt after the queue pair
* is made common.
*/
struct qib_mcast_qp {
struct list_head list;
struct qib_qp *qp;
};
struct qib_mcast {
struct rb_node rb_node;
union ib_gid mgid;
struct list_head qp_list;
wait_queue_head_t wait;
atomic_t refcount;
int n_attached;
};
/* Protection domain */
struct qib_pd {
struct ib_pd ibpd;
int user; /* non-zero if created from user space */
};
/* Address Handle */
struct qib_ah {
struct ib_ah ibah;
struct ib_ah_attr attr;
atomic_t refcount;
};
/*
* This structure is used by qib_mmap() to validate an offset
* when an mmap() request is made. The vm_area_struct then uses
* this as its vm_private_data.
*/
struct qib_mmap_info {
struct list_head pending_mmaps;
struct ib_ucontext *context;
void *obj;
__u64 offset;
struct kref ref;
unsigned size;
};
/*
* This structure is used to contain the head pointer, tail pointer,
* and completion queue entries as a single memory allocation so
* it can be mmap'ed into user space.
*/
struct qib_cq_wc {
u32 head; /* index of next entry to fill */
u32 tail; /* index of next ib_poll_cq() entry */
union {
/* these are actually size ibcq.cqe + 1 */
struct ib_uverbs_wc uqueue[0];
struct ib_wc kqueue[0];
};
};
/*
* The completion queue structure.
*/
struct qib_cq {
struct ib_cq ibcq;
struct kthread_work comptask;
struct qib_devdata *dd;
spinlock_t lock; /* protect changes in this struct */
u8 notify;
u8 triggered;
struct qib_cq_wc *queue;
struct qib_mmap_info *ip;
};
/*
* A segment is a linear region of low physical memory.
* XXX Maybe we should use phys addr here and kmap()/kunmap().
* Used by the verbs layer.
*/
struct qib_seg {
void *vaddr;
size_t length;
};
/* The number of qib_segs that fit in a page. */
#define QIB_SEGSZ (PAGE_SIZE / sizeof(struct qib_seg))
struct qib_segarray {
struct qib_seg segs[QIB_SEGSZ];
};
struct qib_mregion {
struct ib_pd *pd; /* shares refcnt of ibmr.pd */
u64 user_base; /* User's address for this region */
u64 iova; /* IB start address of this region */
size_t length;
u32 lkey;
u32 offset; /* offset (bytes) to start of region */
int access_flags;
u32 max_segs; /* number of qib_segs in all the arrays */
u32 mapsz; /* size of the map array */
u8 page_shift; /* 0 - non unform/non powerof2 sizes */
u8 lkey_published; /* in global table */
struct completion comp; /* complete when refcount goes to zero */
struct rcu_head list;
atomic_t refcount;
struct qib_segarray *map[0]; /* the segments */
};
/*
* These keep track of the copy progress within a memory region.
* Used by the verbs layer.
*/
struct qib_sge {
struct qib_mregion *mr;
void *vaddr; /* kernel virtual address of segment */
u32 sge_length; /* length of the SGE */
u32 length; /* remaining length of the segment */
u16 m; /* current index: mr->map[m] */
u16 n; /* current index: mr->map[m]->segs[n] */
};
/* Memory region */
struct qib_mr {
struct ib_mr ibmr;
struct ib_umem *umem;
u64 *pages;
u32 npages;
struct qib_mregion mr; /* must be last */
};
/*
* Send work request queue entry.
* The size of the sg_list is determined when the QP is created and stored
* in qp->s_max_sge.
*/
struct qib_swqe {
union {
struct ib_send_wr wr; /* don't use wr.sg_list */
struct ib_ud_wr ud_wr;
struct ib_reg_wr reg_wr;
struct ib_rdma_wr rdma_wr;
struct ib_atomic_wr atomic_wr;
};
u32 psn; /* first packet sequence number */
u32 lpsn; /* last packet sequence number */
u32 ssn; /* send sequence number */
u32 length; /* total length of data in sg_list */
struct qib_sge sg_list[0];
};
/*
* Receive work request queue entry.
* The size of the sg_list is determined when the QP (or SRQ) is created
* and stored in qp->r_rq.max_sge (or srq->rq.max_sge).
*/
struct qib_rwqe {
u64 wr_id;
u8 num_sge;
struct ib_sge sg_list[0];
};
/*
* This structure is used to contain the head pointer, tail pointer,
* and receive work queue entries as a single memory allocation so
* it can be mmap'ed into user space.
* Note that the wq array elements are variable size so you can't
* just index into the array to get the N'th element;
* use get_rwqe_ptr() instead.
*/
struct qib_rwq {
u32 head; /* new work requests posted to the head */
u32 tail; /* receives pull requests from here. */
struct qib_rwqe wq[0];
};
struct qib_rq {
struct qib_rwq *wq;
u32 size; /* size of RWQE array */
u8 max_sge;
spinlock_t lock /* protect changes in this struct */
____cacheline_aligned_in_smp;
};
struct qib_srq {
struct ib_srq ibsrq;
struct qib_rq rq;
struct qib_mmap_info *ip;
/* send signal when number of RWQEs < limit */
u32 limit;
};
struct qib_sge_state {
struct qib_sge *sg_list; /* next SGE to be used if any */
struct qib_sge sge; /* progress state for the current SGE */
u32 total_len;
u8 num_sge;
};
/*
* This structure holds the information that the send tasklet needs
* to send a RDMA read response or atomic operation.
*/
struct qib_ack_entry {
u8 opcode;
u8 sent;
u32 psn;
u32 lpsn;
union {
struct qib_sge rdma_sge;
u64 atomic_data;
};
};
/*
* Variables prefixed with s_ are for the requester (sender).
* Variables prefixed with r_ are for the responder (receiver).
* Variables prefixed with ack_ are for responder replies.
*
* Common variables are protected by both r_rq.lock and s_lock in that order
* which only happens in modify_qp() or changing the QP 'state'.
*/
struct qib_qp {
struct ib_qp ibqp;
/* read mostly fields above and below */
struct ib_ah_attr remote_ah_attr;
struct ib_ah_attr alt_ah_attr;
struct qib_qp __rcu *next; /* link list for QPN hash table */
struct qib_swqe *s_wq; /* send work queue */
struct qib_mmap_info *ip;
struct qib_ib_header *s_hdr; /* next packet header to send */
unsigned long timeout_jiffies; /* computed from timeout */
enum ib_mtu path_mtu;
u32 remote_qpn;
u32 pmtu; /* decoded from path_mtu */
u32 qkey; /* QKEY for this QP (for UD or RD) */
u32 s_size; /* send work queue size */
u32 s_rnr_timeout; /* number of milliseconds for RNR timeout */
u8 state; /* QP state */
u8 qp_access_flags;
u8 alt_timeout; /* Alternate path timeout for this QP */
u8 timeout; /* Timeout for this QP */
u8 s_srate;
u8 s_mig_state;
u8 port_num;
u8 s_pkey_index; /* PKEY index to use */
u8 s_alt_pkey_index; /* Alternate path PKEY index to use */
u8 r_max_rd_atomic; /* max number of RDMA read/atomic to receive */
u8 s_max_rd_atomic; /* max number of RDMA read/atomic to send */
u8 s_retry_cnt; /* number of times to retry */
u8 s_rnr_retry_cnt;
u8 r_min_rnr_timer; /* retry timeout value for RNR NAKs */
u8 s_max_sge; /* size of s_wq->sg_list */
u8 s_draining;
/* start of read/write fields */
atomic_t refcount ____cacheline_aligned_in_smp;
wait_queue_head_t wait;
struct qib_ack_entry s_ack_queue[QIB_MAX_RDMA_ATOMIC + 1]
____cacheline_aligned_in_smp;
struct qib_sge_state s_rdma_read_sge;
spinlock_t r_lock ____cacheline_aligned_in_smp; /* used for APM */
unsigned long r_aflags;
u64 r_wr_id; /* ID for current receive WQE */
u32 r_ack_psn; /* PSN for next ACK or atomic ACK */
u32 r_len; /* total length of r_sge */
u32 r_rcv_len; /* receive data len processed */
u32 r_psn; /* expected rcv packet sequence number */
u32 r_msn; /* message sequence number */
u8 r_state; /* opcode of last packet received */
u8 r_flags;
u8 r_head_ack_queue; /* index into s_ack_queue[] */
struct list_head rspwait; /* link for waititing to respond */
struct qib_sge_state r_sge; /* current receive data */
struct qib_rq r_rq; /* receive work queue */
spinlock_t s_lock ____cacheline_aligned_in_smp;
struct qib_sge_state *s_cur_sge;
u32 s_flags;
struct qib_verbs_txreq *s_tx;
struct qib_swqe *s_wqe;
struct qib_sge_state s_sge; /* current send request data */
struct qib_mregion *s_rdma_mr;
atomic_t s_dma_busy;
u32 s_cur_size; /* size of send packet in bytes */
u32 s_len; /* total length of s_sge */
u32 s_rdma_read_len; /* total length of s_rdma_read_sge */
u32 s_next_psn; /* PSN for next request */
u32 s_last_psn; /* last response PSN processed */
u32 s_sending_psn; /* lowest PSN that is being sent */
u32 s_sending_hpsn; /* highest PSN that is being sent */
u32 s_psn; /* current packet sequence number */
u32 s_ack_rdma_psn; /* PSN for sending RDMA read responses */
u32 s_ack_psn; /* PSN for acking sends and RDMA writes */
u32 s_head; /* new entries added here */
u32 s_tail; /* next entry to process */
u32 s_cur; /* current work queue entry */
u32 s_acked; /* last un-ACK'ed entry */
u32 s_last; /* last completed entry */
u32 s_ssn; /* SSN of tail entry */
u32 s_lsn; /* limit sequence number (credit) */
u16 s_hdrwords; /* size of s_hdr in 32 bit words */
u16 s_rdma_ack_cnt;
u8 s_state; /* opcode of last packet sent */
u8 s_ack_state; /* opcode of packet to ACK */
u8 s_nak_state; /* non-zero if NAK is pending */
u8 r_nak_state; /* non-zero if NAK is pending */
u8 s_retry; /* requester retry counter */
u8 s_rnr_retry; /* requester RNR retry counter */
u8 s_num_rd_atomic; /* number of RDMA read/atomic pending */
u8 s_tail_ack_queue; /* index into s_ack_queue[] */
struct qib_sge_state s_ack_rdma_sge;
struct timer_list s_timer;
struct qib_qp_priv {
struct qib_ib_header *s_hdr; /* next packet header to send */
struct list_head iowait; /* link for wait PIO buf */
atomic_t s_dma_busy;
struct qib_verbs_txreq *s_tx;
struct work_struct s_work;
wait_queue_head_t wait_dma;
struct qib_sge r_sg_list[0] /* verified SGEs */
____cacheline_aligned_in_smp;
struct rvt_qp *owner;
};
/*
* Atomic bit definitions for r_aflags.
*/
#define QIB_R_WRID_VALID 0
#define QIB_R_REWIND_SGE 1
/*
* Bit definitions for r_flags.
*/
#define QIB_R_REUSE_SGE 0x01
#define QIB_R_RDMAR_SEQ 0x02
#define QIB_R_RSP_NAK 0x04
#define QIB_R_RSP_SEND 0x08
#define QIB_R_COMM_EST 0x10
/*
* Bit definitions for s_flags.
*
* QIB_S_SIGNAL_REQ_WR - set if QP send WRs contain completion signaled
* QIB_S_BUSY - send tasklet is processing the QP
* QIB_S_TIMER - the RC retry timer is active
* QIB_S_ACK_PENDING - an ACK is waiting to be sent after RDMA read/atomics
* QIB_S_WAIT_FENCE - waiting for all prior RDMA read or atomic SWQEs
* before processing the next SWQE
* QIB_S_WAIT_RDMAR - waiting for a RDMA read or atomic SWQE to complete
* before processing the next SWQE
* QIB_S_WAIT_RNR - waiting for RNR timeout
* QIB_S_WAIT_SSN_CREDIT - waiting for RC credits to process next SWQE
* QIB_S_WAIT_DMA - waiting for send DMA queue to drain before generating
* next send completion entry not via send DMA
* QIB_S_WAIT_PIO - waiting for a send buffer to be available
* QIB_S_WAIT_TX - waiting for a struct qib_verbs_txreq to be available
* QIB_S_WAIT_DMA_DESC - waiting for DMA descriptors to be available
* QIB_S_WAIT_KMEM - waiting for kernel memory to be available
* QIB_S_WAIT_PSN - waiting for a packet to exit the send DMA queue
* QIB_S_WAIT_ACK - waiting for an ACK packet before sending more requests
* QIB_S_SEND_ONE - send one packet, request ACK, then wait for ACK
*/
#define QIB_S_SIGNAL_REQ_WR 0x0001
#define QIB_S_BUSY 0x0002
#define QIB_S_TIMER 0x0004
#define QIB_S_RESP_PENDING 0x0008
#define QIB_S_ACK_PENDING 0x0010
#define QIB_S_WAIT_FENCE 0x0020
#define QIB_S_WAIT_RDMAR 0x0040
#define QIB_S_WAIT_RNR 0x0080
#define QIB_S_WAIT_SSN_CREDIT 0x0100
#define QIB_S_WAIT_DMA 0x0200
#define QIB_S_WAIT_PIO 0x0400
#define QIB_S_WAIT_TX 0x0800
#define QIB_S_WAIT_DMA_DESC 0x1000
#define QIB_S_WAIT_KMEM 0x2000
#define QIB_S_WAIT_PSN 0x4000
#define QIB_S_WAIT_ACK 0x8000
#define QIB_S_SEND_ONE 0x10000
#define QIB_S_UNLIMITED_CREDIT 0x20000
/*
* Wait flags that would prevent any packet type from being sent.
*/
#define QIB_S_ANY_WAIT_IO (QIB_S_WAIT_PIO | QIB_S_WAIT_TX | \
QIB_S_WAIT_DMA_DESC | QIB_S_WAIT_KMEM)
/*
* Wait flags that would prevent send work requests from making progress.
*/
#define QIB_S_ANY_WAIT_SEND (QIB_S_WAIT_FENCE | QIB_S_WAIT_RDMAR | \
QIB_S_WAIT_RNR | QIB_S_WAIT_SSN_CREDIT | QIB_S_WAIT_DMA | \
QIB_S_WAIT_PSN | QIB_S_WAIT_ACK)
#define QIB_S_ANY_WAIT (QIB_S_ANY_WAIT_IO | QIB_S_ANY_WAIT_SEND)
#define QIB_PSN_CREDIT 16
/*
* Since struct qib_swqe is not a fixed size, we can't simply index into
* struct qib_qp.s_wq. This function does the array index computation.
*/
static inline struct qib_swqe *get_swqe_ptr(struct qib_qp *qp,
unsigned n)
{
return (struct qib_swqe *)((char *)qp->s_wq +
(sizeof(struct qib_swqe) +
qp->s_max_sge *
sizeof(struct qib_sge)) * n);
}
/*
* Since struct qib_rwqe is not a fixed size, we can't simply index into
* struct qib_rwq.wq. This function does the array index computation.
*/
static inline struct qib_rwqe *get_rwqe_ptr(struct qib_rq *rq, unsigned n)
{
return (struct qib_rwqe *)
((char *) rq->wq->wq +
(sizeof(struct qib_rwqe) +
rq->max_sge * sizeof(struct ib_sge)) * n);
}
/*
* QPN-map pages start out as NULL, they get allocated upon
* first use and are never deallocated. This way,
* large bitmaps are not allocated unless large numbers of QPs are used.
*/
struct qpn_map {
void *page;
};
struct qib_qpn_table {
spinlock_t lock; /* protect changes in this struct */
unsigned flags; /* flags for QP0/1 allocated for each port */
u32 last; /* last QP number allocated */
u32 nmaps; /* size of the map table */
u16 limit;
u16 mask;
/* bit map of free QP numbers other than 0/1 */
struct qpn_map map[QPNMAP_ENTRIES];
};
#define MAX_LKEY_TABLE_BITS 23
struct qib_lkey_table {
spinlock_t lock; /* protect changes in this struct */
u32 next; /* next unused index (speeds search) */
u32 gen; /* generation count */
u32 max; /* size of the table */
struct qib_mregion __rcu **table;
};
struct qib_opcode_stats {
u64 n_packets; /* number of packets */
u64 n_bytes; /* total number of bytes */
@ -682,21 +218,9 @@ struct qib_pma_counters {
};
struct qib_ibport {
struct qib_qp __rcu *qp0;
struct qib_qp __rcu *qp1;
struct ib_mad_agent *send_agent; /* agent for SMI (traps) */
struct qib_ah *sm_ah;
struct qib_ah *smi_ah;
struct rb_root mcast_tree;
spinlock_t lock; /* protect changes in this struct */
/* non-zero when timer is set */
unsigned long mkey_lease_timeout;
unsigned long trap_timeout;
__be64 gid_prefix; /* in network order */
__be64 mkey;
struct rvt_ibport rvp;
struct rvt_ah *smi_ah;
__be64 guids[QIB_GUIDS_PER_PORT - 1]; /* writable GUIDs */
u64 tid; /* TID for traps */
struct qib_pma_counters __percpu *pmastats;
u64 z_unicast_xmit; /* starting count for PMA */
u64 z_unicast_rcv; /* starting count for PMA */
@ -715,82 +239,25 @@ struct qib_ibport {
u32 z_local_link_integrity_errors; /* starting count for PMA */
u32 z_excessive_buffer_overrun_errors; /* starting count for PMA */
u32 z_vl15_dropped; /* starting count for PMA */
u32 n_rc_resends;
u32 n_rc_acks;
u32 n_rc_qacks;
u32 n_rc_delayed_comp;
u32 n_seq_naks;
u32 n_rdma_seq;
u32 n_rnr_naks;
u32 n_other_naks;
u32 n_loop_pkts;
u32 n_pkt_drops;
u32 n_vl15_dropped;
u32 n_rc_timeouts;
u32 n_dmawait;
u32 n_unaligned;
u32 n_rc_dupreq;
u32 n_rc_seqnak;
u32 port_cap_flags;
u32 pma_sample_start;
u32 pma_sample_interval;
__be16 pma_counter_select[5];
u16 pma_tag;
u16 pkey_violations;
u16 qkey_violations;
u16 mkey_violations;
u16 mkey_lease_period;
u16 sm_lid;
u16 repress_traps;
u8 sm_sl;
u8 mkeyprot;
u8 subnet_timeout;
u8 vl_high_limit;
u8 sl_to_vl[16];
};
struct qib_ibdev {
struct ib_device ibdev;
struct list_head pending_mmaps;
spinlock_t mmap_offset_lock; /* protect mmap_offset */
u32 mmap_offset;
struct qib_mregion __rcu *dma_mr;
struct rvt_dev_info rdi;
/* QP numbers are shared by all IB ports */
struct qib_qpn_table qpn_table;
struct qib_lkey_table lk_table;
struct list_head piowait; /* list for wait PIO buf */
struct list_head dmawait; /* list for wait DMA */
struct list_head txwait; /* list for wait qib_verbs_txreq */
struct list_head memwait; /* list for wait kernel memory */
struct list_head txreq_free;
struct timer_list mem_timer;
struct qib_qp __rcu **qp_table;
struct qib_pio_header *pio_hdrs;
dma_addr_t pio_hdrs_phys;
/* list of QPs waiting for RNR timer */
spinlock_t pending_lock; /* protect wait lists, PMA counters, etc. */
u32 qp_table_size; /* size of the hash table */
u32 qp_rnd; /* random bytes for hash */
spinlock_t qpt_lock;
u32 n_piowait;
u32 n_txwait;
u32 n_pds_allocated; /* number of PDs allocated for device */
spinlock_t n_pds_lock;
u32 n_ahs_allocated; /* number of AHs allocated for device */
spinlock_t n_ahs_lock;
u32 n_cqs_allocated; /* number of CQs allocated for device */
spinlock_t n_cqs_lock;
u32 n_qps_allocated; /* number of QPs allocated for device */
spinlock_t n_qps_lock;
u32 n_srqs_allocated; /* number of SRQs allocated for device */
spinlock_t n_srqs_lock;
u32 n_mcast_grps_allocated; /* number of mcast groups allocated */
spinlock_t n_mcast_grps_lock;
#ifdef CONFIG_DEBUG_FS
/* per HCA debugfs */
struct dentry *qib_ibdev_dbg;
@ -813,56 +280,27 @@ struct qib_verbs_counters {
u32 vl15_dropped;
};
static inline struct qib_mr *to_imr(struct ib_mr *ibmr)
{
return container_of(ibmr, struct qib_mr, ibmr);
}
static inline struct qib_pd *to_ipd(struct ib_pd *ibpd)
{
return container_of(ibpd, struct qib_pd, ibpd);
}
static inline struct qib_ah *to_iah(struct ib_ah *ibah)
{
return container_of(ibah, struct qib_ah, ibah);
}
static inline struct qib_cq *to_icq(struct ib_cq *ibcq)
{
return container_of(ibcq, struct qib_cq, ibcq);
}
static inline struct qib_srq *to_isrq(struct ib_srq *ibsrq)
{
return container_of(ibsrq, struct qib_srq, ibsrq);
}
static inline struct qib_qp *to_iqp(struct ib_qp *ibqp)
{
return container_of(ibqp, struct qib_qp, ibqp);
}
static inline struct qib_ibdev *to_idev(struct ib_device *ibdev)
{
return container_of(ibdev, struct qib_ibdev, ibdev);
struct rvt_dev_info *rdi;
rdi = container_of(ibdev, struct rvt_dev_info, ibdev);
return container_of(rdi, struct qib_ibdev, rdi);
}
/*
* Send if not busy or waiting for I/O and either
* a RC response is pending or we can process send work requests.
*/
static inline int qib_send_ok(struct qib_qp *qp)
static inline int qib_send_ok(struct rvt_qp *qp)
{
return !(qp->s_flags & (QIB_S_BUSY | QIB_S_ANY_WAIT_IO)) &&
(qp->s_hdrwords || (qp->s_flags & QIB_S_RESP_PENDING) ||
!(qp->s_flags & QIB_S_ANY_WAIT_SEND));
return !(qp->s_flags & (RVT_S_BUSY | RVT_S_ANY_WAIT_IO)) &&
(qp->s_hdrwords || (qp->s_flags & RVT_S_RESP_PENDING) ||
!(qp->s_flags & RVT_S_ANY_WAIT_SEND));
}
/*
* This must be called with s_lock held.
*/
void qib_schedule_send(struct qib_qp *qp);
void _qib_schedule_send(struct rvt_qp *qp);
void qib_schedule_send(struct rvt_qp *qp);
static inline int qib_pkey_ok(u16 pkey1, u16 pkey2)
{
@ -878,7 +316,7 @@ static inline int qib_pkey_ok(u16 pkey1, u16 pkey2)
void qib_bad_pqkey(struct qib_ibport *ibp, __be16 trap_num, u32 key, u32 sl,
u32 qp1, u32 qp2, __be16 lid1, __be16 lid2);
void qib_cap_mask_chg(struct qib_ibport *ibp);
void qib_cap_mask_chg(struct rvt_dev_info *rdi, u8 port_num);
void qib_sys_guid_chg(struct qib_ibport *ibp);
void qib_node_desc_chg(struct qib_ibport *ibp);
int qib_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num,
@ -886,8 +324,8 @@ int qib_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num,
const struct ib_mad_hdr *in, size_t in_mad_size,
struct ib_mad_hdr *out, size_t *out_mad_size,
u16 *out_mad_pkey_index);
int qib_create_agents(struct qib_ibdev *dev);
void qib_free_agents(struct qib_ibdev *dev);
void qib_notify_create_mad_agent(struct rvt_dev_info *rdi, int port_idx);
void qib_notify_free_mad_agent(struct rvt_dev_info *rdi, int port_idx);
/*
* Compare the lower 24 bits of the two values.
@ -898,8 +336,6 @@ static inline int qib_cmp24(u32 a, u32 b)
return (((int) a) - ((int) b)) << 8;
}
struct qib_mcast *qib_mcast_find(struct qib_ibport *ibp, union ib_gid *mgid);
int qib_snapshot_counters(struct qib_pportdata *ppd, u64 *swords,
u64 *rwords, u64 *spkts, u64 *rpkts,
u64 *xmit_wait);
@ -907,35 +343,17 @@ int qib_snapshot_counters(struct qib_pportdata *ppd, u64 *swords,
int qib_get_counters(struct qib_pportdata *ppd,
struct qib_verbs_counters *cntrs);
int qib_multicast_attach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid);
__be32 qib_compute_aeth(struct rvt_qp *qp);
int qib_multicast_detach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid);
int qib_mcast_tree_empty(struct qib_ibport *ibp);
__be32 qib_compute_aeth(struct qib_qp *qp);
struct qib_qp *qib_lookup_qpn(struct qib_ibport *ibp, u32 qpn);
struct ib_qp *qib_create_qp(struct ib_pd *ibpd,
struct ib_qp_init_attr *init_attr,
struct ib_udata *udata);
int qib_destroy_qp(struct ib_qp *ibqp);
int qib_error_qp(struct qib_qp *qp, enum ib_wc_status err);
int qib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
int attr_mask, struct ib_udata *udata);
int qib_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
int attr_mask, struct ib_qp_init_attr *init_attr);
unsigned qib_free_all_qps(struct qib_devdata *dd);
void qib_init_qpn_table(struct qib_devdata *dd, struct qib_qpn_table *qpt);
void qib_free_qpn_table(struct qib_qpn_table *qpt);
/*
* Functions provided by qib driver for rdmavt to use
*/
unsigned qib_free_all_qps(struct rvt_dev_info *rdi);
void *qib_qp_priv_alloc(struct rvt_dev_info *rdi, struct rvt_qp *qp, gfp_t gfp);
void qib_qp_priv_free(struct rvt_dev_info *rdi, struct rvt_qp *qp);
void qib_notify_qp_reset(struct rvt_qp *qp);
int qib_alloc_qpn(struct rvt_dev_info *rdi, struct rvt_qpn_table *qpt,
enum ib_qp_type type, u8 port, gfp_t gfp);
#ifdef CONFIG_DEBUG_FS
@ -949,7 +367,7 @@ void qib_qp_iter_print(struct seq_file *s, struct qib_qp_iter *iter);
#endif
void qib_get_credit(struct qib_qp *qp, u32 aeth);
void qib_get_credit(struct rvt_qp *qp, u32 aeth);
unsigned qib_pkt_delay(u32 plen, u8 snd_mult, u8 rcv_mult);
@ -957,166 +375,66 @@ void qib_verbs_sdma_desc_avail(struct qib_pportdata *ppd, unsigned avail);
void qib_put_txreq(struct qib_verbs_txreq *tx);
int qib_verbs_send(struct qib_qp *qp, struct qib_ib_header *hdr,
u32 hdrwords, struct qib_sge_state *ss, u32 len);
int qib_verbs_send(struct rvt_qp *qp, struct qib_ib_header *hdr,
u32 hdrwords, struct rvt_sge_state *ss, u32 len);
void qib_copy_sge(struct qib_sge_state *ss, void *data, u32 length,
void qib_copy_sge(struct rvt_sge_state *ss, void *data, u32 length,
int release);
void qib_skip_sge(struct qib_sge_state *ss, u32 length, int release);
void qib_skip_sge(struct rvt_sge_state *ss, u32 length, int release);
void qib_uc_rcv(struct qib_ibport *ibp, struct qib_ib_header *hdr,
int has_grh, void *data, u32 tlen, struct qib_qp *qp);
int has_grh, void *data, u32 tlen, struct rvt_qp *qp);
void qib_rc_rcv(struct qib_ctxtdata *rcd, struct qib_ib_header *hdr,
int has_grh, void *data, u32 tlen, struct qib_qp *qp);
int has_grh, void *data, u32 tlen, struct rvt_qp *qp);
int qib_check_ah(struct ib_device *ibdev, struct ib_ah_attr *ah_attr);
int qib_check_send_wqe(struct rvt_qp *qp, struct rvt_swqe *wqe);
struct ib_ah *qib_create_qp0_ah(struct qib_ibport *ibp, u16 dlid);
void qib_rc_rnr_retry(unsigned long arg);
void qib_rc_send_complete(struct qib_qp *qp, struct qib_ib_header *hdr);
void qib_rc_send_complete(struct rvt_qp *qp, struct qib_ib_header *hdr);
void qib_rc_error(struct qib_qp *qp, enum ib_wc_status err);
void qib_rc_error(struct rvt_qp *qp, enum ib_wc_status err);
int qib_post_ud_send(struct qib_qp *qp, struct ib_send_wr *wr);
int qib_post_ud_send(struct rvt_qp *qp, struct ib_send_wr *wr);
void qib_ud_rcv(struct qib_ibport *ibp, struct qib_ib_header *hdr,
int has_grh, void *data, u32 tlen, struct qib_qp *qp);
int qib_alloc_lkey(struct qib_mregion *mr, int dma_region);
void qib_free_lkey(struct qib_mregion *mr);
int qib_lkey_ok(struct qib_lkey_table *rkt, struct qib_pd *pd,
struct qib_sge *isge, struct ib_sge *sge, int acc);
int qib_rkey_ok(struct qib_qp *qp, struct qib_sge *sge,
u32 len, u64 vaddr, u32 rkey, int acc);
int qib_post_srq_receive(struct ib_srq *ibsrq, struct ib_recv_wr *wr,
struct ib_recv_wr **bad_wr);
struct ib_srq *qib_create_srq(struct ib_pd *ibpd,
struct ib_srq_init_attr *srq_init_attr,
struct ib_udata *udata);
int qib_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
enum ib_srq_attr_mask attr_mask,
struct ib_udata *udata);
int qib_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr);
int qib_destroy_srq(struct ib_srq *ibsrq);
int qib_cq_init(struct qib_devdata *dd);
void qib_cq_exit(struct qib_devdata *dd);
void qib_cq_enter(struct qib_cq *cq, struct ib_wc *entry, int sig);
int qib_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *entry);
struct ib_cq *qib_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
struct ib_ucontext *context,
struct ib_udata *udata);
int qib_destroy_cq(struct ib_cq *ibcq);
int qib_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags notify_flags);
int qib_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata);
struct ib_mr *qib_get_dma_mr(struct ib_pd *pd, int acc);
struct ib_mr *qib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
u64 virt_addr, int mr_access_flags,
struct ib_udata *udata);
int qib_dereg_mr(struct ib_mr *ibmr);
struct ib_mr *qib_alloc_mr(struct ib_pd *pd,
enum ib_mr_type mr_type,
u32 max_entries);
int qib_map_mr_sg(struct ib_mr *ibmr,
struct scatterlist *sg,
int sg_nents);
int qib_reg_mr(struct qib_qp *qp, struct ib_reg_wr *wr);
struct ib_fmr *qib_alloc_fmr(struct ib_pd *pd, int mr_access_flags,
struct ib_fmr_attr *fmr_attr);
int qib_map_phys_fmr(struct ib_fmr *ibfmr, u64 *page_list,
int list_len, u64 iova);
int qib_unmap_fmr(struct list_head *fmr_list);
int qib_dealloc_fmr(struct ib_fmr *ibfmr);
static inline void qib_get_mr(struct qib_mregion *mr)
{
atomic_inc(&mr->refcount);
}
int has_grh, void *data, u32 tlen, struct rvt_qp *qp);
void mr_rcu_callback(struct rcu_head *list);
static inline void qib_put_mr(struct qib_mregion *mr)
{
if (unlikely(atomic_dec_and_test(&mr->refcount)))
call_rcu(&mr->list, mr_rcu_callback);
}
int qib_get_rwqe(struct rvt_qp *qp, int wr_id_only);
static inline void qib_put_ss(struct qib_sge_state *ss)
{
while (ss->num_sge) {
qib_put_mr(ss->sge.mr);
if (--ss->num_sge)
ss->sge = *ss->sg_list++;
}
}
void qib_release_mmap_info(struct kref *ref);
struct qib_mmap_info *qib_create_mmap_info(struct qib_ibdev *dev, u32 size,
struct ib_ucontext *context,
void *obj);
void qib_update_mmap_info(struct qib_ibdev *dev, struct qib_mmap_info *ip,
u32 size, void *obj);
int qib_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);
int qib_get_rwqe(struct qib_qp *qp, int wr_id_only);
void qib_migrate_qp(struct qib_qp *qp);
void qib_migrate_qp(struct rvt_qp *qp);
int qib_ruc_check_hdr(struct qib_ibport *ibp, struct qib_ib_header *hdr,
int has_grh, struct qib_qp *qp, u32 bth0);
int has_grh, struct rvt_qp *qp, u32 bth0);
u32 qib_make_grh(struct qib_ibport *ibp, struct ib_grh *hdr,
struct ib_global_route *grh, u32 hwords, u32 nwords);
void qib_make_ruc_header(struct qib_qp *qp, struct qib_other_headers *ohdr,
void qib_make_ruc_header(struct rvt_qp *qp, struct qib_other_headers *ohdr,
u32 bth0, u32 bth2);
void qib_do_send(struct work_struct *work);
void _qib_do_send(struct work_struct *work);
void qib_send_complete(struct qib_qp *qp, struct qib_swqe *wqe,
void qib_do_send(struct rvt_qp *qp);
void qib_send_complete(struct rvt_qp *qp, struct rvt_swqe *wqe,
enum ib_wc_status status);
void qib_send_rc_ack(struct qib_qp *qp);
void qib_send_rc_ack(struct rvt_qp *qp);
int qib_make_rc_req(struct qib_qp *qp);
int qib_make_rc_req(struct rvt_qp *qp);
int qib_make_uc_req(struct qib_qp *qp);
int qib_make_uc_req(struct rvt_qp *qp);
int qib_make_ud_req(struct qib_qp *qp);
int qib_make_ud_req(struct rvt_qp *qp);
int qib_register_ib_device(struct qib_devdata *);
@ -1150,11 +468,11 @@ extern const enum ib_wc_opcode ib_qib_wc_opcode[];
#define IB_PHYSPORTSTATE_CFG_ENH 0x10
#define IB_PHYSPORTSTATE_CFG_WAIT_ENH 0x13
extern const int ib_qib_state_ops[];
extern const int ib_rvt_state_ops[];
extern __be64 ib_qib_sys_image_guid; /* in network order */
extern unsigned int ib_qib_lkey_table_size;
extern unsigned int ib_rvt_lkey_table_size;
extern unsigned int ib_qib_max_cqes;
@ -1178,6 +496,4 @@ extern unsigned int ib_qib_max_srq_wrs;
extern const u32 ib_qib_rnr_table[];
extern struct ib_dma_mapping_ops qib_dma_mapping_ops;
#endif /* QIB_VERBS_H */

363
drivers/infiniband/hw/qib/qib_verbs_mcast.c
View File

@ -1,363 +0,0 @@
/*
* Copyright (c) 2006, 2007, 2008, 2009 QLogic Corporation. All rights reserved.
* Copyright (c) 2005, 2006 PathScale, Inc. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
* General Public License (GPL) Version 2, available from the file
* COPYING in the main directory of this source tree, or the
* OpenIB.org BSD license below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include <linux/rculist.h>
#include "qib.h"
/**
* qib_mcast_qp_alloc - alloc a struct to link a QP to mcast GID struct
* @qp: the QP to link
*/
static struct qib_mcast_qp *qib_mcast_qp_alloc(struct qib_qp *qp)
{
struct qib_mcast_qp *mqp;
mqp = kmalloc(sizeof(*mqp), GFP_KERNEL);
if (!mqp)
goto bail;
mqp->qp = qp;
atomic_inc(&qp->refcount);
bail:
return mqp;
}
static void qib_mcast_qp_free(struct qib_mcast_qp *mqp)
{
struct qib_qp *qp = mqp->qp;
/* Notify qib_destroy_qp() if it is waiting. */
if (atomic_dec_and_test(&qp->refcount))
wake_up(&qp->wait);
kfree(mqp);
}
/**
* qib_mcast_alloc - allocate the multicast GID structure
* @mgid: the multicast GID
*
* A list of QPs will be attached to this structure.
*/
static struct qib_mcast *qib_mcast_alloc(union ib_gid *mgid)
{
struct qib_mcast *mcast;
mcast = kmalloc(sizeof(*mcast), GFP_KERNEL);
if (!mcast)
goto bail;
mcast->mgid = *mgid;
INIT_LIST_HEAD(&mcast->qp_list);
init_waitqueue_head(&mcast->wait);
atomic_set(&mcast->refcount, 0);
mcast->n_attached = 0;
bail:
return mcast;
}
static void qib_mcast_free(struct qib_mcast *mcast)
{
struct qib_mcast_qp *p, *tmp;
list_for_each_entry_safe(p, tmp, &mcast->qp_list, list)
qib_mcast_qp_free(p);
kfree(mcast);
}
/**
* qib_mcast_find - search the global table for the given multicast GID
* @ibp: the IB port structure
* @mgid: the multicast GID to search for
*
* Returns NULL if not found.
*
* The caller is responsible for decrementing the reference count if found.
*/
struct qib_mcast *qib_mcast_find(struct qib_ibport *ibp, union ib_gid *mgid)
{
struct rb_node *n;
unsigned long flags;
struct qib_mcast *mcast;
spin_lock_irqsave(&ibp->lock, flags);
n = ibp->mcast_tree.rb_node;
while (n) {
int ret;
mcast = rb_entry(n, struct qib_mcast, rb_node);
ret = memcmp(mgid->raw, mcast->mgid.raw,
sizeof(union ib_gid));
if (ret < 0)
n = n->rb_left;
else if (ret > 0)
n = n->rb_right;
else {
atomic_inc(&mcast->refcount);
spin_unlock_irqrestore(&ibp->lock, flags);
goto bail;
}
}
spin_unlock_irqrestore(&ibp->lock, flags);
mcast = NULL;
bail:
return mcast;
}
/**
* qib_mcast_add - insert mcast GID into table and attach QP struct
* @mcast: the mcast GID table
* @mqp: the QP to attach
*
* Return zero if both were added. Return EEXIST if the GID was already in
* the table but the QP was added. Return ESRCH if the QP was already
* attached and neither structure was added.
*/
static int qib_mcast_add(struct qib_ibdev *dev, struct qib_ibport *ibp,
struct qib_mcast *mcast, struct qib_mcast_qp *mqp)
{
struct rb_node **n = &ibp->mcast_tree.rb_node;
struct rb_node *pn = NULL;
int ret;
spin_lock_irq(&ibp->lock);
while (*n) {
struct qib_mcast *tmcast;
struct qib_mcast_qp *p;
pn = *n;
tmcast = rb_entry(pn, struct qib_mcast, rb_node);
ret = memcmp(mcast->mgid.raw, tmcast->mgid.raw,
sizeof(union ib_gid));
if (ret < 0) {
n = &pn->rb_left;
continue;
}
if (ret > 0) {
n = &pn->rb_right;
continue;
}
/* Search the QP list to see if this is already there. */
list_for_each_entry_rcu(p, &tmcast->qp_list, list) {
if (p->qp == mqp->qp) {
ret = ESRCH;
goto bail;
}
}
if (tmcast->n_attached == ib_qib_max_mcast_qp_attached) {
ret = ENOMEM;
goto bail;
}
tmcast->n_attached++;
list_add_tail_rcu(&mqp->list, &tmcast->qp_list);
ret = EEXIST;
goto bail;
}
spin_lock(&dev->n_mcast_grps_lock);
if (dev->n_mcast_grps_allocated == ib_qib_max_mcast_grps) {
spin_unlock(&dev->n_mcast_grps_lock);
ret = ENOMEM;
goto bail;
}
dev->n_mcast_grps_allocated++;
spin_unlock(&dev->n_mcast_grps_lock);
mcast->n_attached++;
list_add_tail_rcu(&mqp->list, &mcast->qp_list);
atomic_inc(&mcast->refcount);
rb_link_node(&mcast->rb_node, pn, n);
rb_insert_color(&mcast->rb_node, &ibp->mcast_tree);
ret = 0;
bail:
spin_unlock_irq(&ibp->lock);
return ret;
}
int qib_multicast_attach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
{
struct qib_qp *qp = to_iqp(ibqp);
struct qib_ibdev *dev = to_idev(ibqp->device);
struct qib_ibport *ibp;
struct qib_mcast *mcast;
struct qib_mcast_qp *mqp;
int ret;
if (ibqp->qp_num <= 1 || qp->state == IB_QPS_RESET) {
ret = -EINVAL;
goto bail;
}
/*
* Allocate data structures since its better to do this outside of
* spin locks and it will most likely be needed.
*/
mcast = qib_mcast_alloc(gid);
if (mcast == NULL) {
ret = -ENOMEM;
goto bail;
}
mqp = qib_mcast_qp_alloc(qp);
if (mqp == NULL) {
qib_mcast_free(mcast);
ret = -ENOMEM;
goto bail;
}
ibp = to_iport(ibqp->device, qp->port_num);
switch (qib_mcast_add(dev, ibp, mcast, mqp)) {
case ESRCH:
/* Neither was used: OK to attach the same QP twice. */
qib_mcast_qp_free(mqp);
qib_mcast_free(mcast);
break;
case EEXIST: /* The mcast wasn't used */
qib_mcast_free(mcast);
break;
case ENOMEM:
/* Exceeded the maximum number of mcast groups. */
qib_mcast_qp_free(mqp);
qib_mcast_free(mcast);
ret = -ENOMEM;
goto bail;
default:
break;
}
ret = 0;
bail:
return ret;
}
int qib_multicast_detach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
{
struct qib_qp *qp = to_iqp(ibqp);
struct qib_ibdev *dev = to_idev(ibqp->device);
struct qib_ibport *ibp = to_iport(ibqp->device, qp->port_num);
struct qib_mcast *mcast = NULL;
struct qib_mcast_qp *p, *tmp, *delp = NULL;
struct rb_node *n;
int last = 0;
int ret;
if (ibqp->qp_num <= 1 || qp->state == IB_QPS_RESET)
return -EINVAL;
spin_lock_irq(&ibp->lock);
/* Find the GID in the mcast table. */
n = ibp->mcast_tree.rb_node;
while (1) {
if (n == NULL) {
spin_unlock_irq(&ibp->lock);
return -EINVAL;
}
mcast = rb_entry(n, struct qib_mcast, rb_node);
ret = memcmp(gid->raw, mcast->mgid.raw,
sizeof(union ib_gid));
if (ret < 0)
n = n->rb_left;
else if (ret > 0)
n = n->rb_right;
else
break;
}
/* Search the QP list. */
list_for_each_entry_safe(p, tmp, &mcast->qp_list, list) {
if (p->qp != qp)
continue;
/*
* We found it, so remove it, but don't poison the forward
* link until we are sure there are no list walkers.
*/
list_del_rcu(&p->list);
mcast->n_attached--;
delp = p;
/* If this was the last attached QP, remove the GID too. */
if (list_empty(&mcast->qp_list)) {
rb_erase(&mcast->rb_node, &ibp->mcast_tree);
last = 1;
}
break;
}
spin_unlock_irq(&ibp->lock);
/* QP not attached */
if (!delp)
return -EINVAL;
/*
* Wait for any list walkers to finish before freeing the
* list element.
*/
wait_event(mcast->wait, atomic_read(&mcast->refcount) <= 1);
qib_mcast_qp_free(delp);
if (last) {
atomic_dec(&mcast->refcount);
wait_event(mcast->wait, !atomic_read(&mcast->refcount));
qib_mcast_free(mcast);
spin_lock_irq(&dev->n_mcast_grps_lock);
dev->n_mcast_grps_allocated--;
spin_unlock_irq(&dev->n_mcast_grps_lock);
}
return 0;
}
int qib_mcast_tree_empty(struct qib_ibport *ibp)
{
return ibp->mcast_tree.rb_node == NULL;
}

1
drivers/infiniband/sw/Makefile 100644
View File

@ -0,0 +1 @@
obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt/

6
drivers/infiniband/sw/rdmavt/Kconfig 100644
View File

@ -0,0 +1,6 @@
config INFINIBAND_RDMAVT
tristate "RDMA verbs transport library"
depends on 64BIT
default m
---help---
This is a common software verbs provider for RDMA networks.

13
drivers/infiniband/sw/rdmavt/Makefile 100644
View File

@ -0,0 +1,13 @@
#
# rdmavt driver
#
#
#
# Called from the kernel module build system.
#
obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt.o
rdmavt-y := vt.o ah.o cq.o dma.o mad.o mcast.o mmap.o mr.o pd.o qp.o srq.o \
trace.o
CFLAGS_trace.o = -I$(src)

196
drivers/infiniband/sw/rdmavt/ah.c 100644
View File

@ -0,0 +1,196 @@
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <linux/slab.h>
#include "ah.h"
#include "vt.h" /* for prints */
/**
* rvt_check_ah - validate the attributes of AH
* @ibdev: the ib device
* @ah_attr: the attributes of the AH
*
* If driver supports a more detailed check_ah function call back to it
* otherwise just check the basics.
*
* Return: 0 on success
*/
int rvt_check_ah(struct ib_device *ibdev,
struct ib_ah_attr *ah_attr)
{
int err;
struct ib_port_attr port_attr;
struct rvt_dev_info *rdi = ib_to_rvt(ibdev);
enum rdma_link_layer link = rdma_port_get_link_layer(ibdev,
ah_attr->port_num);
err = ib_query_port(ibdev, ah_attr->port_num, &port_attr);
if (err)
return -EINVAL;
if (ah_attr->port_num < 1 ||
ah_attr->port_num > ibdev->phys_port_cnt)
return -EINVAL;
if (ah_attr->static_rate != IB_RATE_PORT_CURRENT &&
ib_rate_to_mbps(ah_attr->static_rate) < 0)
return -EINVAL;
if ((ah_attr->ah_flags & IB_AH_GRH) &&
ah_attr->grh.sgid_index >= port_attr.gid_tbl_len)
return -EINVAL;
if (link != IB_LINK_LAYER_ETHERNET) {
if (ah_attr->dlid == 0)
return -EINVAL;
if (ah_attr->dlid >= be16_to_cpu(IB_MULTICAST_LID_BASE) &&
ah_attr->dlid != be16_to_cpu(IB_LID_PERMISSIVE) &&
!(ah_attr->ah_flags & IB_AH_GRH))
return -EINVAL;
}
if (rdi->driver_f.check_ah)
return rdi->driver_f.check_ah(ibdev, ah_attr);
return 0;
}
EXPORT_SYMBOL(rvt_check_ah);
/**
* rvt_create_ah - create an address handle
* @pd: the protection domain
* @ah_attr: the attributes of the AH
*
* This may be called from interrupt context.
*
* Return: newly allocated ah
*/
struct ib_ah *rvt_create_ah(struct ib_pd *pd,
struct ib_ah_attr *ah_attr)
{
struct rvt_ah *ah;
struct rvt_dev_info *dev = ib_to_rvt(pd->device);
unsigned long flags;
if (rvt_check_ah(pd->device, ah_attr))
return ERR_PTR(-EINVAL);
ah = kmalloc(sizeof(*ah), GFP_ATOMIC);
if (!ah)
return ERR_PTR(-ENOMEM);
spin_lock_irqsave(&dev->n_ahs_lock, flags);
if (dev->n_ahs_allocated == dev->dparms.props.max_ah) {
spin_unlock(&dev->n_ahs_lock);
kfree(ah);
return ERR_PTR(-ENOMEM);
}
dev->n_ahs_allocated++;
spin_unlock_irqrestore(&dev->n_ahs_lock, flags);
ah->attr = *ah_attr;
atomic_set(&ah->refcount, 0);
if (dev->driver_f.notify_new_ah)
dev->driver_f.notify_new_ah(pd->device, ah_attr, ah);
return &ah->ibah;
}
/**
* rvt_destory_ah - Destory an address handle
* @ibah: address handle
*
* Return: 0 on success
*/
int rvt_destroy_ah(struct ib_ah *ibah)
{
struct rvt_dev_info *dev = ib_to_rvt(ibah->device);
struct rvt_ah *ah = ibah_to_rvtah(ibah);
unsigned long flags;
if (atomic_read(&ah->refcount) != 0)
return -EBUSY;
spin_lock_irqsave(&dev->n_ahs_lock, flags);
dev->n_ahs_allocated--;
spin_unlock_irqrestore(&dev->n_ahs_lock, flags);
kfree(ah);
return 0;
}
/**
* rvt_modify_ah - modify an ah with given attrs
* @ibah: address handle to modify
* @ah_attr: attrs to apply
*
* Return: 0 on success
*/
int rvt_modify_ah(struct ib_ah *ibah, struct ib_ah_attr *ah_attr)
{
struct rvt_ah *ah = ibah_to_rvtah(ibah);
if (rvt_check_ah(ibah->device, ah_attr))
return -EINVAL;
ah->attr = *ah_attr;
return 0;
}
/**
* rvt_query_ah - return attrs for ah
* @ibah: address handle to query
* @ah_attr: return info in this
*
* Return: always 0
*/
int rvt_query_ah(struct ib_ah *ibah, struct ib_ah_attr *ah_attr)
{
struct rvt_ah *ah = ibah_to_rvtah(ibah);
*ah_attr = ah->attr;
return 0;
}

59
drivers/infiniband/sw/rdmavt/ah.h 100644
View File

@ -0,0 +1,59 @@
#ifndef DEF_RVTAH_H
#define DEF_RVTAH_H
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <rdma/rdma_vt.h>
struct ib_ah *rvt_create_ah(struct ib_pd *pd,
struct ib_ah_attr *ah_attr);
int rvt_destroy_ah(struct ib_ah *ibah);
int rvt_modify_ah(struct ib_ah *ibah, struct ib_ah_attr *ah_attr);
int rvt_query_ah(struct ib_ah *ibah, struct ib_ah_attr *ah_attr);
#endif /* DEF_RVTAH_H */

325
drivers/staging/rdma/hfi1/cq.c → drivers/infiniband/sw/rdmavt/cq.c
View File

@ -1,12 +1,11 @@
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* Copyright(c) 2015 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
@ -18,8 +17,6 @@
*
* BSD LICENSE
*
* Copyright(c) 2015 Intel Corporation.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
@ -48,25 +45,23 @@
*
*/
#include <linux/err.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>
#include <linux/kthread.h>
#include "verbs.h"
#include "hfi.h"
#include "cq.h"
#include "vt.h"
/**
* hfi1_cq_enter - add a new entry to the completion queue
* rvt_cq_enter - add a new entry to the completion queue
* @cq: completion queue
* @entry: work completion entry to add
* @sig: true if @entry is a solicited entry
* @sig: true if @entry is solicited
*
* This may be called with qp->s_lock held.
*/
void hfi1_cq_enter(struct hfi1_cq *cq, struct ib_wc *entry, int solicited)
void rvt_cq_enter(struct rvt_cq *cq, struct ib_wc *entry, bool solicited)
{
struct hfi1_cq_wc *wc;
struct rvt_cq_wc *wc;
unsigned long flags;
u32 head;
u32 next;
@ -79,11 +74,13 @@ void hfi1_cq_enter(struct hfi1_cq *cq, struct ib_wc *entry, int solicited)
*/
wc = cq->queue;
head = wc->head;
if (head >= (unsigned) cq->ibcq.cqe) {
if (head >= (unsigned)cq->ibcq.cqe) {
head = cq->ibcq.cqe;
next = 0;
} else
} else {
next = head + 1;
}
if (unlikely(next == wc->tail)) {
spin_unlock_irqrestore(&cq->lock, flags);
if (cq->ibcq.event_handler) {
@ -114,8 +111,9 @@ void hfi1_cq_enter(struct hfi1_cq *cq, struct ib_wc *entry, int solicited)
wc->uqueue[head].port_num = entry->port_num;
/* Make sure entry is written before the head index. */
smp_wmb();
} else
} else {
wc->kqueue[head] = *entry;
}
wc->head = next;
if (cq->notify == IB_CQ_NEXT_COMP ||
@ -126,10 +124,10 @@ void hfi1_cq_enter(struct hfi1_cq *cq, struct ib_wc *entry, int solicited)
* This will cause send_complete() to be called in
* another thread.
*/
smp_read_barrier_depends(); /* see hfi1_cq_exit */
worker = cq->dd->worker;
smp_read_barrier_depends(); /* see rvt_cq_exit */
worker = cq->rdi->worker;
if (likely(worker)) {
cq->notify = IB_CQ_NONE;
cq->notify = RVT_CQ_NONE;
cq->triggered++;
queue_kthread_work(worker, &cq->comptask);
}
@ -137,59 +135,11 @@ void hfi1_cq_enter(struct hfi1_cq *cq, struct ib_wc *entry, int solicited)
spin_unlock_irqrestore(&cq->lock, flags);
}
/**
* hfi1_poll_cq - poll for work completion entries
* @ibcq: the completion queue to poll
* @num_entries: the maximum number of entries to return
* @entry: pointer to array where work completions are placed
*
* Returns the number of completion entries polled.
*
* This may be called from interrupt context. Also called by ib_poll_cq()
* in the generic verbs code.
*/
int hfi1_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *entry)
{
struct hfi1_cq *cq = to_icq(ibcq);
struct hfi1_cq_wc *wc;
unsigned long flags;
int npolled;
u32 tail;
/* The kernel can only poll a kernel completion queue */
if (cq->ip) {
npolled = -EINVAL;
goto bail;
}
spin_lock_irqsave(&cq->lock, flags);
wc = cq->queue;
tail = wc->tail;
if (tail > (u32) cq->ibcq.cqe)
tail = (u32) cq->ibcq.cqe;
for (npolled = 0; npolled < num_entries; ++npolled, ++entry) {
if (tail == wc->head)
break;
/* The kernel doesn't need a RMB since it has the lock. */
*entry = wc->kqueue[tail];
if (tail >= cq->ibcq.cqe)
tail = 0;
else
tail++;
}
wc->tail = tail;
spin_unlock_irqrestore(&cq->lock, flags);
bail:
return npolled;
}
EXPORT_SYMBOL(rvt_cq_enter);
static void send_complete(struct kthread_work *work)
{
struct hfi1_cq *cq = container_of(work, struct hfi1_cq, comptask);
struct rvt_cq *cq = container_of(work, struct rvt_cq, comptask);
/*
* The completion handler will most likely rearm the notification
@ -217,26 +167,25 @@ static void send_complete(struct kthread_work *work)
}
/**
* hfi1_create_cq - create a completion queue
* rvt_create_cq - create a completion queue
* @ibdev: the device this completion queue is attached to
* @attr: creation attributes
* @context: unused by the driver
* @context: unused by the QLogic_IB driver
* @udata: user data for libibverbs.so
*
* Returns a pointer to the completion queue or negative errno values
* for failure.
*
* Called by ib_create_cq() in the generic verbs code.
*
* Return: pointer to the completion queue or negative errno values
* for failure.
*/
struct ib_cq *hfi1_create_cq(
struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
struct ib_ucontext *context,
struct ib_udata *udata)
struct ib_cq *rvt_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
struct ib_ucontext *context,
struct ib_udata *udata)
{
struct hfi1_ibdev *dev = to_idev(ibdev);
struct hfi1_cq *cq;
struct hfi1_cq_wc *wc;
struct rvt_dev_info *rdi = ib_to_rvt(ibdev);
struct rvt_cq *cq;
struct rvt_cq_wc *wc;
struct ib_cq *ret;
u32 sz;
unsigned int entries = attr->cqe;
@ -244,11 +193,11 @@ struct ib_cq *hfi1_create_cq(
if (attr->flags)
return ERR_PTR(-EINVAL);
if (entries < 1 || entries > hfi1_max_cqes)
if (entries < 1 || entries > rdi->dparms.props.max_cqe)
return ERR_PTR(-EINVAL);
/* Allocate the completion queue structure. */
cq = kmalloc(sizeof(*cq), GFP_KERNEL);
cq = kzalloc(sizeof(*cq), GFP_KERNEL);
if (!cq)
return ERR_PTR(-ENOMEM);
@ -272,12 +221,12 @@ struct ib_cq *hfi1_create_cq(
/*
* Return the address of the WC as the offset to mmap.
* See hfi1_mmap() for details.
* See rvt_mmap() for details.
*/
if (udata && udata->outlen >= sizeof(__u64)) {
int err;
cq->ip = hfi1_create_mmap_info(dev, sz, context, wc);
cq->ip = rvt_create_mmap_info(rdi, sz, context, wc);
if (!cq->ip) {
ret = ERR_PTR(-ENOMEM);
goto bail_wc;
@ -289,23 +238,22 @@ struct ib_cq *hfi1_create_cq(
ret = ERR_PTR(err);
goto bail_ip;
}
} else
cq->ip = NULL;
}
spin_lock(&dev->n_cqs_lock);
if (dev->n_cqs_allocated == hfi1_max_cqs) {
spin_unlock(&dev->n_cqs_lock);
spin_lock(&rdi->n_cqs_lock);
if (rdi->n_cqs_allocated == rdi->dparms.props.max_cq) {
spin_unlock(&rdi->n_cqs_lock);
ret = ERR_PTR(-ENOMEM);
goto bail_ip;
}
dev->n_cqs_allocated++;
spin_unlock(&dev->n_cqs_lock);
rdi->n_cqs_allocated++;
spin_unlock(&rdi->n_cqs_lock);
if (cq->ip) {
spin_lock_irq(&dev->pending_lock);
list_add(&cq->ip->pending_mmaps, &dev->pending_mmaps);
spin_unlock_irq(&dev->pending_lock);
spin_lock_irq(&rdi->pending_lock);
list_add(&cq->ip->pending_mmaps, &rdi->pending_mmaps);
spin_unlock_irq(&rdi->pending_lock);
}
/*
@ -313,14 +261,11 @@ struct ib_cq *hfi1_create_cq(
* The number of entries should be >= the number requested or return
* an error.
*/
cq->dd = dd_from_dev(dev);
cq->rdi = rdi;
cq->ibcq.cqe = entries;
cq->notify = IB_CQ_NONE;
cq->triggered = 0;
cq->notify = RVT_CQ_NONE;
spin_lock_init(&cq->lock);
init_kthread_work(&cq->comptask, send_complete);
wc->head = 0;
wc->tail = 0;
cq->queue = wc;
ret = &cq->ibcq;
@ -338,24 +283,24 @@ done:
}
/**
* hfi1_destroy_cq - destroy a completion queue
* rvt_destroy_cq - destroy a completion queue
* @ibcq: the completion queue to destroy.
*
* Returns 0 for success.
*
* Called by ib_destroy_cq() in the generic verbs code.
*
* Return: always 0
*/
int hfi1_destroy_cq(struct ib_cq *ibcq)
int rvt_destroy_cq(struct ib_cq *ibcq)
{
struct hfi1_ibdev *dev = to_idev(ibcq->device);
struct hfi1_cq *cq = to_icq(ibcq);
struct rvt_cq *cq = ibcq_to_rvtcq(ibcq);
struct rvt_dev_info *rdi = cq->rdi;
flush_kthread_work(&cq->comptask);
spin_lock(&dev->n_cqs_lock);
dev->n_cqs_allocated--;
spin_unlock(&dev->n_cqs_lock);
spin_lock(&rdi->n_cqs_lock);
rdi->n_cqs_allocated--;
spin_unlock(&rdi->n_cqs_lock);
if (cq->ip)
kref_put(&cq->ip->ref, hfi1_release_mmap_info);
kref_put(&cq->ip->ref, rvt_release_mmap_info);
else
vfree(cq->queue);
kfree(cq);
@ -364,18 +309,18 @@ int hfi1_destroy_cq(struct ib_cq *ibcq)
}
/**
* hfi1_req_notify_cq - change the notification type for a completion queue
* rvt_req_notify_cq - change the notification type for a completion queue
* @ibcq: the completion queue
* @notify_flags: the type of notification to request
*
* Returns 0 for success.
*
* This may be called from interrupt context. Also called by
* ib_req_notify_cq() in the generic verbs code.
*
* Return: 0 for success.
*/
int hfi1_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags notify_flags)
int rvt_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags notify_flags)
{
struct hfi1_cq *cq = to_icq(ibcq);
struct rvt_cq *cq = ibcq_to_rvtcq(ibcq);
unsigned long flags;
int ret = 0;
@ -397,24 +342,23 @@ int hfi1_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags notify_flags)
}
/**
* hfi1_resize_cq - change the size of the CQ
* rvt_resize_cq - change the size of the CQ
* @ibcq: the completion queue
*
* Returns 0 for success.
* Return: 0 for success.
*/
int hfi1_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata)
int rvt_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata)
{
struct hfi1_cq *cq = to_icq(ibcq);
struct hfi1_cq_wc *old_wc;
struct hfi1_cq_wc *wc;
struct rvt_cq *cq = ibcq_to_rvtcq(ibcq);
struct rvt_cq_wc *old_wc;
struct rvt_cq_wc *wc;
u32 head, tail, n;
int ret;
u32 sz;
struct rvt_dev_info *rdi = cq->rdi;
if (cqe < 1 || cqe > hfi1_max_cqes) {
ret = -EINVAL;
goto bail;
}
if (cqe < 1 || cqe > rdi->dparms.props.max_cqe)
return -EINVAL;
/*
* Need to use vmalloc() if we want to support large #s of entries.
@ -425,10 +369,8 @@ int hfi1_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata)
else
sz += sizeof(struct ib_wc) * (cqe + 1);
wc = vmalloc_user(sz);
if (!wc) {
ret = -ENOMEM;
goto bail;
}
if (!wc)
return -ENOMEM;
/* Check that we can write the offset to mmap. */
if (udata && udata->outlen >= sizeof(__u64)) {
@ -446,11 +388,11 @@ int hfi1_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata)
*/
old_wc = cq->queue;
head = old_wc->head;
if (head > (u32) cq->ibcq.cqe)
head = (u32) cq->ibcq.cqe;
if (head > (u32)cq->ibcq.cqe)
head = (u32)cq->ibcq.cqe;
tail = old_wc->tail;
if (tail > (u32) cq->ibcq.cqe)
tail = (u32) cq->ibcq.cqe;
if (tail > (u32)cq->ibcq.cqe)
tail = (u32)cq->ibcq.cqe;
if (head < tail)
n = cq->ibcq.cqe + 1 + head - tail;
else
@ -464,7 +406,7 @@ int hfi1_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata)
wc->uqueue[n] = old_wc->uqueue[tail];
else
wc->kqueue[n] = old_wc->kqueue[tail];
if (tail == (u32) cq->ibcq.cqe)
if (tail == (u32)cq->ibcq.cqe)
tail = 0;
else
tail++;
@ -478,80 +420,131 @@ int hfi1_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata)
vfree(old_wc);
if (cq->ip) {
struct hfi1_ibdev *dev = to_idev(ibcq->device);
struct hfi1_mmap_info *ip = cq->ip;
struct rvt_mmap_info *ip = cq->ip;
hfi1_update_mmap_info(dev, ip, sz, wc);
rvt_update_mmap_info(rdi, ip, sz, wc);
/*
* Return the offset to mmap.
* See hfi1_mmap() for details.
* See rvt_mmap() for details.
*/
if (udata && udata->outlen >= sizeof(__u64)) {
ret = ib_copy_to_udata(udata, &ip->offset,
sizeof(ip->offset));
if (ret)
goto bail;
return ret;
}
spin_lock_irq(&dev->pending_lock);
spin_lock_irq(&rdi->pending_lock);
if (list_empty(&ip->pending_mmaps))
list_add(&ip->pending_mmaps, &dev->pending_mmaps);
spin_unlock_irq(&dev->pending_lock);
list_add(&ip->pending_mmaps, &rdi->pending_mmaps);
spin_unlock_irq(&rdi->pending_lock);
}
ret = 0;
goto bail;
return 0;
bail_unlock:
spin_unlock_irq(&cq->lock);
bail_free:
vfree(wc);
bail:
return ret;
}
int hfi1_cq_init(struct hfi1_devdata *dd)
/**
* rvt_poll_cq - poll for work completion entries
* @ibcq: the completion queue to poll
* @num_entries: the maximum number of entries to return
* @entry: pointer to array where work completions are placed
*
* This may be called from interrupt context. Also called by ib_poll_cq()
* in the generic verbs code.
*
* Return: the number of completion entries polled.
*/
int rvt_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *entry)
{
struct rvt_cq *cq = ibcq_to_rvtcq(ibcq);
struct rvt_cq_wc *wc;
unsigned long flags;
int npolled;
u32 tail;
/* The kernel can only poll a kernel completion queue */
if (cq->ip)
return -EINVAL;
spin_lock_irqsave(&cq->lock, flags);
wc = cq->queue;
tail = wc->tail;
if (tail > (u32)cq->ibcq.cqe)
tail = (u32)cq->ibcq.cqe;
for (npolled = 0; npolled < num_entries; ++npolled, ++entry) {
if (tail == wc->head)
break;
/* The kernel doesn't need a RMB since it has the lock. */
*entry = wc->kqueue[tail];
if (tail >= cq->ibcq.cqe)
tail = 0;
else
tail++;
}
wc->tail = tail;
spin_unlock_irqrestore(&cq->lock, flags);
return npolled;
}
/**
* rvt_driver_cq_init - Init cq resources on behalf of driver
* @rdi: rvt dev structure
*
* Return: 0 on success
*/
int rvt_driver_cq_init(struct rvt_dev_info *rdi)
{
int ret = 0;
int cpu;
struct task_struct *task;
if (dd->worker)
if (rdi->worker)
return 0;
dd->worker = kzalloc(sizeof(*dd->worker), GFP_KERNEL);
if (!dd->worker)
rdi->worker = kzalloc(sizeof(*rdi->worker), GFP_KERNEL);
if (!rdi->worker)
return -ENOMEM;
init_kthread_worker(dd->worker);
init_kthread_worker(rdi->worker);
task = kthread_create_on_node(
kthread_worker_fn,
dd->worker,
dd->assigned_node_id,
"hfi1_cq%d", dd->unit);
if (IS_ERR(task))
goto task_fail;
cpu = cpumask_first(cpumask_of_node(dd->assigned_node_id));
rdi->worker,
rdi->dparms.node,
"%s", rdi->dparms.cq_name);
if (IS_ERR(task)) {
kfree(rdi->worker);
rdi->worker = NULL;
return PTR_ERR(task);
}
cpu = cpumask_first(cpumask_of_node(rdi->dparms.node));
kthread_bind(task, cpu);
wake_up_process(task);
out:
return ret;
task_fail:
ret = PTR_ERR(task);
kfree(dd->worker);
dd->worker = NULL;
goto out;
}
void hfi1_cq_exit(struct hfi1_devdata *dd)
/**
* rvt_cq_exit - tear down cq reources
* @rdi: rvt dev structure
*/
void rvt_cq_exit(struct rvt_dev_info *rdi)
{
struct kthread_worker *worker;
worker = dd->worker;
worker = rdi->worker;
if (!worker)
return;
/* blocks future queuing from send_complete() */
dd->worker = NULL;
smp_wmb(); /* See hfi1_cq_enter */
rdi->worker = NULL;
smp_wmb(); /* See rdi_cq_enter */
flush_kthread_worker(worker);
kthread_stop(worker->task);
kfree(worker);

64
drivers/infiniband/sw/rdmavt/cq.h 100644
View File

@ -0,0 +1,64 @@
#ifndef DEF_RVTCQ_H
#define DEF_RVTCQ_H
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <rdma/rdma_vt.h>
#include <rdma/rdmavt_cq.h>
struct ib_cq *rvt_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
struct ib_ucontext *context,
struct ib_udata *udata);
int rvt_destroy_cq(struct ib_cq *ibcq);
int rvt_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags notify_flags);
int rvt_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata);
int rvt_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *entry);
int rvt_driver_cq_init(struct rvt_dev_info *rdi);
void rvt_cq_exit(struct rvt_dev_info *rdi);
#endif /* DEF_RVTCQ_H */

184
drivers/infiniband/sw/rdmavt/dma.c 100644
View File

@ -0,0 +1,184 @@
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <linux/types.h>
#include <linux/scatterlist.h>
#include <rdma/ib_verbs.h>
#include "dma.h"
#define BAD_DMA_ADDRESS ((u64)0)
/*
* The following functions implement driver specific replacements
* for the ib_dma_*() functions.
*
* These functions return kernel virtual addresses instead of
* device bus addresses since the driver uses the CPU to copy
* data instead of using hardware DMA.
*/
static int rvt_mapping_error(struct ib_device *dev, u64 dma_addr)
{
return dma_addr == BAD_DMA_ADDRESS;
}
static u64 rvt_dma_map_single(struct ib_device *dev, void *cpu_addr,
size_t size, enum dma_data_direction direction)
{
if (WARN_ON(!valid_dma_direction(direction)))
return BAD_DMA_ADDRESS;
return (u64)cpu_addr;
}
static void rvt_dma_unmap_single(struct ib_device *dev, u64 addr, size_t size,
enum dma_data_direction direction)
{
/* This is a stub, nothing to be done here */
}
static u64 rvt_dma_map_page(struct ib_device *dev, struct page *page,
unsigned long offset, size_t size,
enum dma_data_direction direction)
{
u64 addr;
if (WARN_ON(!valid_dma_direction(direction)))
return BAD_DMA_ADDRESS;
if (offset + size > PAGE_SIZE)
return BAD_DMA_ADDRESS;
addr = (u64)page_address(page);
if (addr)
addr += offset;
return addr;
}
static void rvt_dma_unmap_page(struct ib_device *dev, u64 addr, size_t size,
enum dma_data_direction direction)
{
/* This is a stub, nothing to be done here */
}
static int rvt_map_sg(struct ib_device *dev, struct scatterlist *sgl,
int nents, enum dma_data_direction direction)
{
struct scatterlist *sg;
u64 addr;
int i;
int ret = nents;
if (WARN_ON(!valid_dma_direction(direction)))
return 0;
for_each_sg(sgl, sg, nents, i) {
addr = (u64)page_address(sg_page(sg));
if (!addr) {
ret = 0;
break;
}
sg->dma_address = addr + sg->offset;
#ifdef CONFIG_NEED_SG_DMA_LENGTH
sg->dma_length = sg->length;
#endif
}
return ret;
}
static void rvt_unmap_sg(struct ib_device *dev,
struct scatterlist *sg, int nents,
enum dma_data_direction direction)
{
/* This is a stub, nothing to be done here */
}
static void rvt_sync_single_for_cpu(struct ib_device *dev, u64 addr,
size_t size, enum dma_data_direction dir)
{
}
static void rvt_sync_single_for_device(struct ib_device *dev, u64 addr,
size_t size,
enum dma_data_direction dir)
{
}
static void *rvt_dma_alloc_coherent(struct ib_device *dev, size_t size,
u64 *dma_handle, gfp_t flag)
{
struct page *p;
void *addr = NULL;
p = alloc_pages(flag, get_order(size));
if (p)
addr = page_address(p);
if (dma_handle)
*dma_handle = (u64)addr;
return addr;
}
static void rvt_dma_free_coherent(struct ib_device *dev, size_t size,
void *cpu_addr, u64 dma_handle)
{
free_pages((unsigned long)cpu_addr, get_order(size));
}
struct ib_dma_mapping_ops rvt_default_dma_mapping_ops = {
.mapping_error = rvt_mapping_error,
.map_single = rvt_dma_map_single,
.unmap_single = rvt_dma_unmap_single,
.map_page = rvt_dma_map_page,
.unmap_page = rvt_dma_unmap_page,
.map_sg = rvt_map_sg,
.unmap_sg = rvt_unmap_sg,
.sync_single_for_cpu = rvt_sync_single_for_cpu,
.sync_single_for_device = rvt_sync_single_for_device,
.alloc_coherent = rvt_dma_alloc_coherent,
.free_coherent = rvt_dma_free_coherent
};

53
drivers/infiniband/sw/rdmavt/dma.h 100644
View File

@ -0,0 +1,53 @@
#ifndef DEF_RDMAVTDMA_H
#define DEF_RDMAVTDMA_H
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
extern struct ib_dma_mapping_ops rvt_default_dma_mapping_ops;
#endif /* DEF_RDMAVTDMA_H */

171
drivers/infiniband/sw/rdmavt/mad.c 100644
View File

@ -0,0 +1,171 @@
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <rdma/ib_mad.h>
#include "mad.h"
#include "vt.h"
/**
* rvt_process_mad - process an incoming MAD packet
* @ibdev: the infiniband device this packet came in on
* @mad_flags: MAD flags
* @port_num: the port number this packet came in on, 1 based from ib core
* @in_wc: the work completion entry for this packet
* @in_grh: the global route header for this packet
* @in_mad: the incoming MAD
* @out_mad: any outgoing MAD reply
*
* Note that the verbs framework has already done the MAD sanity checks,
* and hop count/pointer updating for IB_MGMT_CLASS_SUBN_DIRECTED_ROUTE
* MADs.
*
* This is called by the ib_mad module.
*
* Return: IB_MAD_RESULT_SUCCESS or error
*/
int rvt_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num,
const struct ib_wc *in_wc, const struct ib_grh *in_grh,
const struct ib_mad_hdr *in, size_t in_mad_size,
struct ib_mad_hdr *out, size_t *out_mad_size,
u16 *out_mad_pkey_index)
{
/*
* MAD processing is quite different between hfi1 and qib. Therfore this
* is expected to be provided by the driver. Other drivers in the future
* may chose to implement this but it should not be made into a
* requirement.
*/
if (ibport_num_to_idx(ibdev, port_num) < 0)
return -EINVAL;
return IB_MAD_RESULT_FAILURE;
}
static void rvt_send_mad_handler(struct ib_mad_agent *agent,
struct ib_mad_send_wc *mad_send_wc)
{
ib_free_send_mad(mad_send_wc->send_buf);
}
/**
* rvt_create_mad_agents - create mad agents
* @rdi: rvt dev struct
*
* If driver needs to be notified of mad agent creation then call back
*
* Return 0 on success
*/
int rvt_create_mad_agents(struct rvt_dev_info *rdi)
{
struct ib_mad_agent *agent;
struct rvt_ibport *rvp;
int p;
int ret;
for (p = 0; p < rdi->dparms.nports; p++) {
rvp = rdi->ports[p];
agent = ib_register_mad_agent(&rdi->ibdev, p + 1,
IB_QPT_SMI,
NULL, 0, rvt_send_mad_handler,
NULL, NULL, 0);
if (IS_ERR(agent)) {
ret = PTR_ERR(agent);
goto err;
}
rvp->send_agent = agent;
if (rdi->driver_f.notify_create_mad_agent)
rdi->driver_f.notify_create_mad_agent(rdi, p);
}
return 0;
err:
for (p = 0; p < rdi->dparms.nports; p++) {
rvp = rdi->ports[p];
if (rvp->send_agent) {
agent = rvp->send_agent;
rvp->send_agent = NULL;
ib_unregister_mad_agent(agent);
if (rdi->driver_f.notify_free_mad_agent)
rdi->driver_f.notify_free_mad_agent(rdi, p);
}
}
return ret;
}
/**
* rvt_free_mad_agents - free up mad agents
* @rdi: rvt dev struct
*
* If driver needs notification of mad agent removal make the call back
*/
void rvt_free_mad_agents(struct rvt_dev_info *rdi)
{
struct ib_mad_agent *agent;
struct rvt_ibport *rvp;
int p;
for (p = 0; p < rdi->dparms.nports; p++) {
rvp = rdi->ports[p];
if (rvp->send_agent) {
agent = rvp->send_agent;
rvp->send_agent = NULL;
ib_unregister_mad_agent(agent);
}
if (rvp->sm_ah) {
ib_destroy_ah(&rvp->sm_ah->ibah);
rvp->sm_ah = NULL;
}
if (rdi->driver_f.notify_free_mad_agent)
rdi->driver_f.notify_free_mad_agent(rdi, p);
}
}

60
drivers/infiniband/sw/rdmavt/mad.h 100644
View File

@ -0,0 +1,60 @@
#ifndef DEF_RVTMAD_H
#define DEF_RVTMAD_H
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <rdma/rdma_vt.h>
int rvt_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num,
const struct ib_wc *in_wc, const struct ib_grh *in_grh,
const struct ib_mad_hdr *in, size_t in_mad_size,
struct ib_mad_hdr *out, size_t *out_mad_size,
u16 *out_mad_pkey_index);
int rvt_create_mad_agents(struct rvt_dev_info *rdi);
void rvt_free_mad_agents(struct rvt_dev_info *rdi);
#endif /* DEF_RVTMAD_H */

262
drivers/staging/rdma/hfi1/verbs_mcast.c → drivers/infiniband/sw/rdmavt/mcast.c
View File

@ -1,12 +1,11 @@
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* Copyright(c) 2015 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
@ -18,8 +17,6 @@
*
* BSD LICENSE
*
* Copyright(c) 2015 Intel Corporation.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
@ -48,17 +45,36 @@
*
*/
#include <linux/slab.h>
#include <linux/sched.h>
#include <linux/rculist.h>
#include <rdma/rdma_vt.h>
#include <rdma/rdmavt_qp.h>
#include "hfi.h"
#include "mcast.h"
/**
* rvt_driver_mcast - init resources for multicast
* @rdi: rvt dev struct
*
* This is per device that registers with rdmavt
*/
void rvt_driver_mcast_init(struct rvt_dev_info *rdi)
{
/*
* Anything that needs setup for multicast on a per driver or per rdi
* basis should be done in here.
*/
spin_lock_init(&rdi->n_mcast_grps_lock);
}
/**
* mcast_qp_alloc - alloc a struct to link a QP to mcast GID struct
* @qp: the QP to link
*/
static struct hfi1_mcast_qp *mcast_qp_alloc(struct hfi1_qp *qp)
static struct rvt_mcast_qp *rvt_mcast_qp_alloc(struct rvt_qp *qp)
{
struct hfi1_mcast_qp *mqp;
struct rvt_mcast_qp *mqp;
mqp = kmalloc(sizeof(*mqp), GFP_KERNEL);
if (!mqp)
@ -71,9 +87,9 @@ bail:
return mqp;
}
static void mcast_qp_free(struct hfi1_mcast_qp *mqp)
static void rvt_mcast_qp_free(struct rvt_mcast_qp *mqp)
{
struct hfi1_qp *qp = mqp->qp;
struct rvt_qp *qp = mqp->qp;
/* Notify hfi1_destroy_qp() if it is waiting. */
if (atomic_dec_and_test(&qp->refcount))
@ -88,11 +104,11 @@ static void mcast_qp_free(struct hfi1_mcast_qp *mqp)
*
* A list of QPs will be attached to this structure.
*/
static struct hfi1_mcast *mcast_alloc(union ib_gid *mgid)
static struct rvt_mcast *rvt_mcast_alloc(union ib_gid *mgid)
{
struct hfi1_mcast *mcast;
struct rvt_mcast *mcast;
mcast = kmalloc(sizeof(*mcast), GFP_KERNEL);
mcast = kzalloc(sizeof(*mcast), GFP_KERNEL);
if (!mcast)
goto bail;
@ -100,75 +116,72 @@ static struct hfi1_mcast *mcast_alloc(union ib_gid *mgid)
INIT_LIST_HEAD(&mcast->qp_list);
init_waitqueue_head(&mcast->wait);
atomic_set(&mcast->refcount, 0);
mcast->n_attached = 0;
bail:
return mcast;
}
static void mcast_free(struct hfi1_mcast *mcast)
static void rvt_mcast_free(struct rvt_mcast *mcast)
{
struct hfi1_mcast_qp *p, *tmp;
struct rvt_mcast_qp *p, *tmp;
list_for_each_entry_safe(p, tmp, &mcast->qp_list, list)
mcast_qp_free(p);
rvt_mcast_qp_free(p);
kfree(mcast);
}
/**
* hfi1_mcast_find - search the global table for the given multicast GID
* rvt_mcast_find - search the global table for the given multicast GID
* @ibp: the IB port structure
* @mgid: the multicast GID to search for
*
* Returns NULL if not found.
*
* The caller is responsible for decrementing the reference count if found.
*
* Return: NULL if not found.
*/
struct hfi1_mcast *hfi1_mcast_find(struct hfi1_ibport *ibp, union ib_gid *mgid)
struct rvt_mcast *rvt_mcast_find(struct rvt_ibport *ibp, union ib_gid *mgid)
{
struct rb_node *n;
unsigned long flags;
struct hfi1_mcast *mcast;
struct rvt_mcast *found = NULL;
spin_lock_irqsave(&ibp->lock, flags);
n = ibp->mcast_tree.rb_node;
while (n) {
int ret;
struct rvt_mcast *mcast;
mcast = rb_entry(n, struct hfi1_mcast, rb_node);
mcast = rb_entry(n, struct rvt_mcast, rb_node);
ret = memcmp(mgid->raw, mcast->mgid.raw,
sizeof(union ib_gid));
if (ret < 0)
if (ret < 0) {
n = n->rb_left;
else if (ret > 0)
} else if (ret > 0) {
n = n->rb_right;
else {
} else {
atomic_inc(&mcast->refcount);
spin_unlock_irqrestore(&ibp->lock, flags);
goto bail;
found = mcast;
break;
}
}
spin_unlock_irqrestore(&ibp->lock, flags);
mcast = NULL;
bail:
return mcast;
return found;
}
EXPORT_SYMBOL(rvt_mcast_find);
/**
* mcast_add - insert mcast GID into table and attach QP struct
* @mcast: the mcast GID table
* @mqp: the QP to attach
*
* Return zero if both were added. Return EEXIST if the GID was already in
* Return: zero if both were added. Return EEXIST if the GID was already in
* the table but the QP was added. Return ESRCH if the QP was already
* attached and neither structure was added.
*/
static int mcast_add(struct hfi1_ibdev *dev, struct hfi1_ibport *ibp,
struct hfi1_mcast *mcast, struct hfi1_mcast_qp *mqp)
static int rvt_mcast_add(struct rvt_dev_info *rdi, struct rvt_ibport *ibp,
struct rvt_mcast *mcast, struct rvt_mcast_qp *mqp)
{
struct rb_node **n = &ibp->mcast_tree.rb_node;
struct rb_node *pn = NULL;
@ -177,11 +190,11 @@ static int mcast_add(struct hfi1_ibdev *dev, struct hfi1_ibport *ibp,
spin_lock_irq(&ibp->lock);
while (*n) {
struct hfi1_mcast *tmcast;
struct hfi1_mcast_qp *p;
struct rvt_mcast *tmcast;
struct rvt_mcast_qp *p;
pn = *n;
tmcast = rb_entry(pn, struct hfi1_mcast, rb_node);
tmcast = rb_entry(pn, struct rvt_mcast, rb_node);
ret = memcmp(mcast->mgid.raw, tmcast->mgid.raw,
sizeof(union ib_gid));
@ -201,7 +214,8 @@ static int mcast_add(struct hfi1_ibdev *dev, struct hfi1_ibport *ibp,
goto bail;
}
}
if (tmcast->n_attached == hfi1_max_mcast_qp_attached) {
if (tmcast->n_attached ==
rdi->dparms.props.max_mcast_qp_attach) {
ret = ENOMEM;
goto bail;
}
@ -213,15 +227,15 @@ static int mcast_add(struct hfi1_ibdev *dev, struct hfi1_ibport *ibp,
goto bail;
}
spin_lock(&dev->n_mcast_grps_lock);
if (dev->n_mcast_grps_allocated == hfi1_max_mcast_grps) {
spin_unlock(&dev->n_mcast_grps_lock);
spin_lock(&rdi->n_mcast_grps_lock);
if (rdi->n_mcast_grps_allocated == rdi->dparms.props.max_mcast_grp) {
spin_unlock(&rdi->n_mcast_grps_lock);
ret = ENOMEM;
goto bail;
}
dev->n_mcast_grps_allocated++;
spin_unlock(&dev->n_mcast_grps_lock);
rdi->n_mcast_grps_allocated++;
spin_unlock(&rdi->n_mcast_grps_lock);
mcast->n_attached++;
@ -239,92 +253,98 @@ bail:
return ret;
}
int hfi1_multicast_attach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
/**
* rvt_attach_mcast - attach a qp to a multicast group
* @ibqp: Infiniband qp
* @igd: multicast guid
* @lid: multicast lid
*
* Return: 0 on success
*/
int rvt_attach_mcast(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
{
struct hfi1_qp *qp = to_iqp(ibqp);
struct hfi1_ibdev *dev = to_idev(ibqp->device);
struct hfi1_ibport *ibp;
struct hfi1_mcast *mcast;
struct hfi1_mcast_qp *mqp;
int ret;
struct rvt_qp *qp = ibqp_to_rvtqp(ibqp);
struct rvt_dev_info *rdi = ib_to_rvt(ibqp->device);
struct rvt_ibport *ibp = rdi->ports[qp->port_num - 1];
struct rvt_mcast *mcast;
struct rvt_mcast_qp *mqp;
int ret = -ENOMEM;
if (ibqp->qp_num <= 1 || qp->state == IB_QPS_RESET) {
ret = -EINVAL;
goto bail;
}
if (ibqp->qp_num <= 1 || qp->state == IB_QPS_RESET)
return -EINVAL;
/*
* Allocate data structures since its better to do this outside of
* spin locks and it will most likely be needed.
*/
mcast = mcast_alloc(gid);
if (mcast == NULL) {
ret = -ENOMEM;
goto bail;
}
mqp = mcast_qp_alloc(qp);
if (mqp == NULL) {
mcast_free(mcast);
ret = -ENOMEM;
goto bail;
}
ibp = to_iport(ibqp->device, qp->port_num);
switch (mcast_add(dev, ibp, mcast, mqp)) {
mcast = rvt_mcast_alloc(gid);
if (!mcast)
return -ENOMEM;
mqp = rvt_mcast_qp_alloc(qp);
if (!mqp)
goto bail_mcast;
switch (rvt_mcast_add(rdi, ibp, mcast, mqp)) {
case ESRCH:
/* Neither was used: OK to attach the same QP twice. */
mcast_qp_free(mqp);
mcast_free(mcast);
break;
case EEXIST: /* The mcast wasn't used */
mcast_free(mcast);
break;
ret = 0;
goto bail_mqp;
case EEXIST: /* The mcast wasn't used */
ret = 0;
goto bail_mcast;
case ENOMEM:
/* Exceeded the maximum number of mcast groups. */
mcast_qp_free(mqp);
mcast_free(mcast);
ret = -ENOMEM;
goto bail;
goto bail_mqp;
default:
break;
}
ret = 0;
return 0;
bail_mqp:
rvt_mcast_qp_free(mqp);
bail_mcast:
rvt_mcast_free(mcast);
bail:
return ret;
}
int hfi1_multicast_detach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
/**
* rvt_detach_mcast - remove a qp from a multicast group
* @ibqp: Infiniband qp
* @igd: multicast guid
* @lid: multicast lid
*
* Return: 0 on success
*/
int rvt_detach_mcast(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
{
struct hfi1_qp *qp = to_iqp(ibqp);
struct hfi1_ibdev *dev = to_idev(ibqp->device);
struct hfi1_ibport *ibp = to_iport(ibqp->device, qp->port_num);
struct hfi1_mcast *mcast = NULL;
struct hfi1_mcast_qp *p, *tmp;
struct rvt_qp *qp = ibqp_to_rvtqp(ibqp);
struct rvt_dev_info *rdi = ib_to_rvt(ibqp->device);
struct rvt_ibport *ibp = rdi->ports[qp->port_num - 1];
struct rvt_mcast *mcast = NULL;
struct rvt_mcast_qp *p, *tmp, *delp = NULL;
struct rb_node *n;
int last = 0;
int ret;
int ret = 0;
if (ibqp->qp_num <= 1 || qp->state == IB_QPS_RESET) {
ret = -EINVAL;
goto bail;
}
if (ibqp->qp_num <= 1 || qp->state == IB_QPS_RESET)
return -EINVAL;
spin_lock_irq(&ibp->lock);
/* Find the GID in the mcast table. */
n = ibp->mcast_tree.rb_node;
while (1) {
if (n == NULL) {
if (!n) {
spin_unlock_irq(&ibp->lock);
ret = -EINVAL;
goto bail;
return -EINVAL;
}
mcast = rb_entry(n, struct hfi1_mcast, rb_node);
mcast = rb_entry(n, struct rvt_mcast, rb_node);
ret = memcmp(gid->raw, mcast->mgid.raw,
sizeof(union ib_gid));
if (ret < 0)
@ -345,6 +365,7 @@ int hfi1_multicast_detach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
*/
list_del_rcu(&p->list);
mcast->n_attached--;
delp = p;
/* If this was the last attached QP, remove the GID too. */
if (list_empty(&mcast->qp_list)) {
@ -355,31 +376,42 @@ int hfi1_multicast_detach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
}
spin_unlock_irq(&ibp->lock);
/* QP not attached */
if (!delp)
return -EINVAL;
/*
* Wait for any list walkers to finish before freeing the
* list element.
*/
wait_event(mcast->wait, atomic_read(&mcast->refcount) <= 1);
rvt_mcast_qp_free(delp);
if (p) {
/*
* Wait for any list walkers to finish before freeing the
* list element.
*/
wait_event(mcast->wait, atomic_read(&mcast->refcount) <= 1);
mcast_qp_free(p);
}
if (last) {
atomic_dec(&mcast->refcount);
wait_event(mcast->wait, !atomic_read(&mcast->refcount));
mcast_free(mcast);
spin_lock_irq(&dev->n_mcast_grps_lock);
dev->n_mcast_grps_allocated--;
spin_unlock_irq(&dev->n_mcast_grps_lock);
rvt_mcast_free(mcast);
spin_lock_irq(&rdi->n_mcast_grps_lock);
rdi->n_mcast_grps_allocated--;
spin_unlock_irq(&rdi->n_mcast_grps_lock);
}
ret = 0;
bail:
return ret;
return 0;
}
int hfi1_mcast_tree_empty(struct hfi1_ibport *ibp)
/**
*rvt_mast_tree_empty - determine if any qps are attached to any mcast group
*@rdi: rvt dev struct
*
* Return: in use count
*/
int rvt_mcast_tree_empty(struct rvt_dev_info *rdi)
{
return ibp->mcast_tree.rb_node == NULL;
int i;
int in_use = 0;
for (i = 0; i < rdi->dparms.nports; i++)
if (rdi->ports[i]->mcast_tree.rb_node)
in_use++;
return in_use;
}

58
drivers/infiniband/sw/rdmavt/mcast.h 100644
View File

@ -0,0 +1,58 @@
#ifndef DEF_RVTMCAST_H
#define DEF_RVTMCAST_H
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <rdma/rdma_vt.h>
void rvt_driver_mcast_init(struct rvt_dev_info *rdi);
int rvt_attach_mcast(struct ib_qp *ibqp, union ib_gid *gid, u16 lid);
int rvt_detach_mcast(struct ib_qp *ibqp, union ib_gid *gid, u16 lid);
int rvt_mcast_tree_empty(struct rvt_dev_info *rdi);
#endif /* DEF_RVTMCAST_H */

144
drivers/staging/rdma/hfi1/mmap.c → drivers/infiniband/sw/rdmavt/mmap.c
View File

@ -1,12 +1,11 @@
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* Copyright(c) 2015 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
@ -18,8 +17,6 @@
*
* BSD LICENSE
*
* Copyright(c) 2015 Intel Corporation.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
@ -48,68 +45,74 @@
*
*/
#include <linux/module.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>
#include <linux/mm.h>
#include <linux/errno.h>
#include <asm/pgtable.h>
#include "verbs.h"
#include "mmap.h"
/**
* hfi1_release_mmap_info - free mmap info structure
* @ref: a pointer to the kref within struct hfi1_mmap_info
* rvt_mmap_init - init link list and lock for mem map
* @rdi: rvt dev struct
*/
void hfi1_release_mmap_info(struct kref *ref)
void rvt_mmap_init(struct rvt_dev_info *rdi)
{
struct hfi1_mmap_info *ip =
container_of(ref, struct hfi1_mmap_info, ref);
struct hfi1_ibdev *dev = to_idev(ip->context->device);
INIT_LIST_HEAD(&rdi->pending_mmaps);
spin_lock_init(&rdi->pending_lock);
rdi->mmap_offset = PAGE_SIZE;
spin_lock_init(&rdi->mmap_offset_lock);
}
spin_lock_irq(&dev->pending_lock);
/**
* rvt_release_mmap_info - free mmap info structure
* @ref: a pointer to the kref within struct rvt_mmap_info
*/
void rvt_release_mmap_info(struct kref *ref)
{
struct rvt_mmap_info *ip =
container_of(ref, struct rvt_mmap_info, ref);
struct rvt_dev_info *rdi = ib_to_rvt(ip->context->device);
spin_lock_irq(&rdi->pending_lock);
list_del(&ip->pending_mmaps);
spin_unlock_irq(&dev->pending_lock);
spin_unlock_irq(&rdi->pending_lock);
vfree(ip->obj);
kfree(ip);
}
/*
* open and close keep track of how many times the CQ is mapped,
* to avoid releasing it.
*/
static void hfi1_vma_open(struct vm_area_struct *vma)
static void rvt_vma_open(struct vm_area_struct *vma)
{
struct hfi1_mmap_info *ip = vma->vm_private_data;
struct rvt_mmap_info *ip = vma->vm_private_data;
kref_get(&ip->ref);
}
static void hfi1_vma_close(struct vm_area_struct *vma)
static void rvt_vma_close(struct vm_area_struct *vma)
{
struct hfi1_mmap_info *ip = vma->vm_private_data;
struct rvt_mmap_info *ip = vma->vm_private_data;
kref_put(&ip->ref, hfi1_release_mmap_info);
kref_put(&ip->ref, rvt_release_mmap_info);
}
static struct vm_operations_struct hfi1_vm_ops = {
.open = hfi1_vma_open,
.close = hfi1_vma_close,
static const struct vm_operations_struct rvt_vm_ops = {
.open = rvt_vma_open,
.close = rvt_vma_close,
};
/**
* hfi1_mmap - create a new mmap region
* rvt_mmap - create a new mmap region
* @context: the IB user context of the process making the mmap() call
* @vma: the VMA to be initialized
* Return zero if the mmap is OK. Otherwise, return an errno.
*
* Return: zero if the mmap is OK. Otherwise, return an errno.
*/
int hfi1_mmap(struct ib_ucontext *context, struct vm_area_struct *vma)
int rvt_mmap(struct ib_ucontext *context, struct vm_area_struct *vma)
{
struct hfi1_ibdev *dev = to_idev(context->device);
struct rvt_dev_info *rdi = ib_to_rvt(context->device);
unsigned long offset = vma->vm_pgoff << PAGE_SHIFT;
unsigned long size = vma->vm_end - vma->vm_start;
struct hfi1_mmap_info *ip, *pp;
struct rvt_mmap_info *ip, *pp;
int ret = -EINVAL;
/*
@ -117,53 +120,60 @@ int hfi1_mmap(struct ib_ucontext *context, struct vm_area_struct *vma)
* Normally, this list is very short since a call to create a
* CQ, QP, or SRQ is soon followed by a call to mmap().
*/
spin_lock_irq(&dev->pending_lock);
list_for_each_entry_safe(ip, pp, &dev->pending_mmaps,
spin_lock_irq(&rdi->pending_lock);
list_for_each_entry_safe(ip, pp, &rdi->pending_mmaps,
pending_mmaps) {
/* Only the creator is allowed to mmap the object */
if (context != ip->context || (__u64) offset != ip->offset)
if (context != ip->context || (__u64)offset != ip->offset)
continue;
/* Don't allow a mmap larger than the object. */
if (size > ip->size)
break;
list_del_init(&ip->pending_mmaps);
spin_unlock_irq(&dev->pending_lock);
spin_unlock_irq(&rdi->pending_lock);
ret = remap_vmalloc_range(vma, ip->obj, 0);
if (ret)
goto done;
vma->vm_ops = &hfi1_vm_ops;
vma->vm_ops = &rvt_vm_ops;
vma->vm_private_data = ip;
hfi1_vma_open(vma);
rvt_vma_open(vma);
goto done;
}
spin_unlock_irq(&dev->pending_lock);
spin_unlock_irq(&rdi->pending_lock);
done:
return ret;
}
/*
* Allocate information for hfi1_mmap
/**
* rvt_create_mmap_info - allocate information for hfi1_mmap
* @rdi: rvt dev struct
* @size: size in bytes to map
* @context: user context
* @obj: opaque pointer to a cq, wq etc
*
* Return: rvt_mmap struct on success
*/
struct hfi1_mmap_info *hfi1_create_mmap_info(struct hfi1_ibdev *dev,
u32 size,
struct ib_ucontext *context,
void *obj) {
struct hfi1_mmap_info *ip;
struct rvt_mmap_info *rvt_create_mmap_info(struct rvt_dev_info *rdi,
u32 size,
struct ib_ucontext *context,
void *obj)
{
struct rvt_mmap_info *ip;
ip = kmalloc(sizeof(*ip), GFP_KERNEL);
ip = kmalloc_node(sizeof(*ip), GFP_KERNEL, rdi->dparms.node);
if (!ip)
goto bail;
return ip;
size = PAGE_ALIGN(size);
spin_lock_irq(&dev->mmap_offset_lock);
if (dev->mmap_offset == 0)
dev->mmap_offset = PAGE_SIZE;
ip->offset = dev->mmap_offset;
dev->mmap_offset += size;
spin_unlock_irq(&dev->mmap_offset_lock);
spin_lock_irq(&rdi->mmap_offset_lock);
if (rdi->mmap_offset == 0)
rdi->mmap_offset = PAGE_SIZE;
ip->offset = rdi->mmap_offset;
rdi->mmap_offset += size;
spin_unlock_irq(&rdi->mmap_offset_lock);
INIT_LIST_HEAD(&ip->pending_mmaps);
ip->size = size;
@ -171,21 +181,27 @@ struct hfi1_mmap_info *hfi1_create_mmap_info(struct hfi1_ibdev *dev,
ip->obj = obj;
kref_init(&ip->ref);
bail:
return ip;
}
void hfi1_update_mmap_info(struct hfi1_ibdev *dev, struct hfi1_mmap_info *ip,
u32 size, void *obj)
/**
* rvt_update_mmap_info - update a mem map
* @rdi: rvt dev struct
* @ip: mmap info pointer
* @size: size to grow by
* @obj: opaque pointer to cq, wq, etc.
*/
void rvt_update_mmap_info(struct rvt_dev_info *rdi, struct rvt_mmap_info *ip,
u32 size, void *obj)
{
size = PAGE_ALIGN(size);
spin_lock_irq(&dev->mmap_offset_lock);
if (dev->mmap_offset == 0)
dev->mmap_offset = PAGE_SIZE;
ip->offset = dev->mmap_offset;
dev->mmap_offset += size;
spin_unlock_irq(&dev->mmap_offset_lock);
spin_lock_irq(&rdi->mmap_offset_lock);
if (rdi->mmap_offset == 0)
rdi->mmap_offset = PAGE_SIZE;
ip->offset = rdi->mmap_offset;
rdi->mmap_offset += size;
spin_unlock_irq(&rdi->mmap_offset_lock);
ip->size = size;
ip->obj = obj;

63
drivers/infiniband/sw/rdmavt/mmap.h 100644
View File

@ -0,0 +1,63 @@
#ifndef DEF_RDMAVTMMAP_H
#define DEF_RDMAVTMMAP_H
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <rdma/rdma_vt.h>
void rvt_mmap_init(struct rvt_dev_info *rdi);
void rvt_release_mmap_info(struct kref *ref);
int rvt_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);
struct rvt_mmap_info *rvt_create_mmap_info(struct rvt_dev_info *rdi,
u32 size,
struct ib_ucontext *context,
void *obj);
void rvt_update_mmap_info(struct rvt_dev_info *rdi, struct rvt_mmap_info *ip,
u32 size, void *obj);
#endif /* DEF_RDMAVTMMAP_H */

830
drivers/infiniband/sw/rdmavt/mr.c 100644
View File

@ -0,0 +1,830 @@
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <linux/slab.h>
#include <linux/vmalloc.h>
#include <rdma/ib_umem.h>
#include <rdma/rdma_vt.h>
#include "vt.h"
#include "mr.h"
/**
* rvt_driver_mr_init - Init MR resources per driver
* @rdi: rvt dev struct
*
* Do any intilization needed when a driver registers with rdmavt.
*
* Return: 0 on success or errno on failure
*/
int rvt_driver_mr_init(struct rvt_dev_info *rdi)
{
unsigned int lkey_table_size = rdi->dparms.lkey_table_size;
unsigned lk_tab_size;
int i;
/*
* The top hfi1_lkey_table_size bits are used to index the
* table. The lower 8 bits can be owned by the user (copied from
* the LKEY). The remaining bits act as a generation number or tag.
*/
if (!lkey_table_size)
return -EINVAL;
spin_lock_init(&rdi->lkey_table.lock);
/* ensure generation is at least 4 bits */
if (lkey_table_size > RVT_MAX_LKEY_TABLE_BITS) {
rvt_pr_warn(rdi, "lkey bits %u too large, reduced to %u\n",
lkey_table_size, RVT_MAX_LKEY_TABLE_BITS);
rdi->dparms.lkey_table_size = RVT_MAX_LKEY_TABLE_BITS;
lkey_table_size = rdi->dparms.lkey_table_size;
}
rdi->lkey_table.max = 1 << lkey_table_size;
lk_tab_size = rdi->lkey_table.max * sizeof(*rdi->lkey_table.table);
rdi->lkey_table.table = (struct rvt_mregion __rcu **)
vmalloc_node(lk_tab_size, rdi->dparms.node);
if (!rdi->lkey_table.table)
return -ENOMEM;
RCU_INIT_POINTER(rdi->dma_mr, NULL);
for (i = 0; i < rdi->lkey_table.max; i++)
RCU_INIT_POINTER(rdi->lkey_table.table[i], NULL);
return 0;
}
/**
*rvt_mr_exit: clean up MR
*@rdi: rvt dev structure
*
* called when drivers have unregistered or perhaps failed to register with us
*/
void rvt_mr_exit(struct rvt_dev_info *rdi)
{
if (rdi->dma_mr)
rvt_pr_err(rdi, "DMA MR not null!\n");
vfree(rdi->lkey_table.table);
}
static void rvt_deinit_mregion(struct rvt_mregion *mr)
{
int i = mr->mapsz;
mr->mapsz = 0;
while (i)
kfree(mr->map[--i]);
}
static int rvt_init_mregion(struct rvt_mregion *mr, struct ib_pd *pd,
int count)
{
int m, i = 0;
mr->mapsz = 0;
m = (count + RVT_SEGSZ - 1) / RVT_SEGSZ;
for (; i < m; i++) {
mr->map[i] = kzalloc(sizeof(*mr->map[0]), GFP_KERNEL);
if (!mr->map[i]) {
rvt_deinit_mregion(mr);
return -ENOMEM;
}
mr->mapsz++;
}
init_completion(&mr->comp);
/* count returning the ptr to user */
atomic_set(&mr->refcount, 1);
mr->pd = pd;
mr->max_segs = count;
return 0;
}
/**
* rvt_alloc_lkey - allocate an lkey
* @mr: memory region that this lkey protects
* @dma_region: 0->normal key, 1->restricted DMA key
*
* Returns 0 if successful, otherwise returns -errno.
*
* Increments mr reference count as required.
*
* Sets the lkey field mr for non-dma regions.
*
*/
static int rvt_alloc_lkey(struct rvt_mregion *mr, int dma_region)
{
unsigned long flags;
u32 r;
u32 n;
int ret = 0;
struct rvt_dev_info *dev = ib_to_rvt(mr->pd->device);
struct rvt_lkey_table *rkt = &dev->lkey_table;
rvt_get_mr(mr);
spin_lock_irqsave(&rkt->lock, flags);
/* special case for dma_mr lkey == 0 */
if (dma_region) {
struct rvt_mregion *tmr;
tmr = rcu_access_pointer(dev->dma_mr);
if (!tmr) {
rcu_assign_pointer(dev->dma_mr, mr);
mr->lkey_published = 1;
} else {
rvt_put_mr(mr);
}
goto success;
}
/* Find the next available LKEY */
r = rkt->next;
n = r;
for (;;) {
if (!rcu_access_pointer(rkt->table[r]))
break;
r = (r + 1) & (rkt->max - 1);
if (r == n)
goto bail;
}
rkt->next = (r + 1) & (rkt->max - 1);
/*
* Make sure lkey is never zero which is reserved to indicate an
* unrestricted LKEY.
*/
rkt->gen++;
/*
* bits are capped to ensure enough bits for generation number
*/
mr->lkey = (r << (32 - dev->dparms.lkey_table_size)) |
((((1 << (24 - dev->dparms.lkey_table_size)) - 1) & rkt->gen)
<< 8);
if (mr->lkey == 0) {
mr->lkey |= 1 << 8;
rkt->gen++;
}
rcu_assign_pointer(rkt->table[r], mr);
mr->lkey_published = 1;
success:
spin_unlock_irqrestore(&rkt->lock, flags);
out:
return ret;
bail:
rvt_put_mr(mr);
spin_unlock_irqrestore(&rkt->lock, flags);
ret = -ENOMEM;
goto out;
}
/**
* rvt_free_lkey - free an lkey
* @mr: mr to free from tables
*/
static void rvt_free_lkey(struct rvt_mregion *mr)
{
unsigned long flags;
u32 lkey = mr->lkey;
u32 r;
struct rvt_dev_info *dev = ib_to_rvt(mr->pd->device);
struct rvt_lkey_table *rkt = &dev->lkey_table;
int freed = 0;
spin_lock_irqsave(&rkt->lock, flags);
if (!mr->lkey_published)
goto out;
if (lkey == 0) {
RCU_INIT_POINTER(dev->dma_mr, NULL);
} else {
r = lkey >> (32 - dev->dparms.lkey_table_size);
RCU_INIT_POINTER(rkt->table[r], NULL);
}
mr->lkey_published = 0;
freed++;
out:
spin_unlock_irqrestore(&rkt->lock, flags);
if (freed) {
synchronize_rcu();
rvt_put_mr(mr);
}
}
static struct rvt_mr *__rvt_alloc_mr(int count, struct ib_pd *pd)
{
struct rvt_mr *mr;
int rval = -ENOMEM;
int m;
/* Allocate struct plus pointers to first level page tables. */
m = (count + RVT_SEGSZ - 1) / RVT_SEGSZ;
mr = kzalloc(sizeof(*mr) + m * sizeof(mr->mr.map[0]), GFP_KERNEL);
if (!mr)
goto bail;
rval = rvt_init_mregion(&mr->mr, pd, count);
if (rval)
goto bail;
/*
* ib_reg_phys_mr() will initialize mr->ibmr except for
* lkey and rkey.
*/
rval = rvt_alloc_lkey(&mr->mr, 0);
if (rval)
goto bail_mregion;
mr->ibmr.lkey = mr->mr.lkey;
mr->ibmr.rkey = mr->mr.lkey;
done:
return mr;
bail_mregion:
rvt_deinit_mregion(&mr->mr);
bail:
kfree(mr);
mr = ERR_PTR(rval);
goto done;
}
static void __rvt_free_mr(struct rvt_mr *mr)
{
rvt_deinit_mregion(&mr->mr);
rvt_free_lkey(&mr->mr);
vfree(mr);
}
/**
* rvt_get_dma_mr - get a DMA memory region
* @pd: protection domain for this memory region
* @acc: access flags
*
* Return: the memory region on success, otherwise returns an errno.
* Note that all DMA addresses should be created via the
* struct ib_dma_mapping_ops functions (see dma.c).
*/
struct ib_mr *rvt_get_dma_mr(struct ib_pd *pd, int acc)
{
struct rvt_mr *mr;
struct ib_mr *ret;
int rval;
if (ibpd_to_rvtpd(pd)->user)
return ERR_PTR(-EPERM);
mr = kzalloc(sizeof(*mr), GFP_KERNEL);
if (!mr) {
ret = ERR_PTR(-ENOMEM);
goto bail;
}
rval = rvt_init_mregion(&mr->mr, pd, 0);
if (rval) {
ret = ERR_PTR(rval);
goto bail;
}
rval = rvt_alloc_lkey(&mr->mr, 1);
if (rval) {
ret = ERR_PTR(rval);
goto bail_mregion;
}
mr->mr.access_flags = acc;
ret = &mr->ibmr;
done:
return ret;
bail_mregion:
rvt_deinit_mregion(&mr->mr);
bail:
kfree(mr);
goto done;
}
/**
* rvt_reg_user_mr - register a userspace memory region
* @pd: protection domain for this memory region
* @start: starting userspace address
* @length: length of region to register
* @mr_access_flags: access flags for this memory region
* @udata: unused by the driver
*
* Return: the memory region on success, otherwise returns an errno.
*/
struct ib_mr *rvt_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
u64 virt_addr, int mr_access_flags,
struct ib_udata *udata)
{
struct rvt_mr *mr;
struct ib_umem *umem;
struct scatterlist *sg;
int n, m, entry;
struct ib_mr *ret;
if (length == 0)
return ERR_PTR(-EINVAL);
umem = ib_umem_get(pd->uobject->context, start, length,
mr_access_flags, 0);
if (IS_ERR(umem))
return (void *)umem;
n = umem->nmap;
mr = __rvt_alloc_mr(n, pd);
if (IS_ERR(mr)) {
ret = (struct ib_mr *)mr;
goto bail_umem;
}
mr->mr.user_base = start;
mr->mr.iova = virt_addr;
mr->mr.length = length;
mr->mr.offset = ib_umem_offset(umem);
mr->mr.access_flags = mr_access_flags;
mr->umem = umem;
if (is_power_of_2(umem->page_size))
mr->mr.page_shift = ilog2(umem->page_size);
m = 0;
n = 0;
for_each_sg(umem->sg_head.sgl, sg, umem->nmap, entry) {
void *vaddr;
vaddr = page_address(sg_page(sg));
if (!vaddr) {
ret = ERR_PTR(-EINVAL);
goto bail_inval;
}
mr->mr.map[m]->segs[n].vaddr = vaddr;
mr->mr.map[m]->segs[n].length = umem->page_size;
n++;
if (n == RVT_SEGSZ) {
m++;
n = 0;
}
}
return &mr->ibmr;
bail_inval:
__rvt_free_mr(mr);
bail_umem:
ib_umem_release(umem);
return ret;
}
/**
* rvt_dereg_mr - unregister and free a memory region
* @ibmr: the memory region to free
*
*
* Note that this is called to free MRs created by rvt_get_dma_mr()
* or rvt_reg_user_mr().
*
* Returns 0 on success.
*/
int rvt_dereg_mr(struct ib_mr *ibmr)
{
struct rvt_mr *mr = to_imr(ibmr);
struct rvt_dev_info *rdi = ib_to_rvt(ibmr->pd->device);
int ret = 0;
unsigned long timeout;
rvt_free_lkey(&mr->mr);
rvt_put_mr(&mr->mr); /* will set completion if last */
timeout = wait_for_completion_timeout(&mr->mr.comp, 5 * HZ);
if (!timeout) {
rvt_pr_err(rdi,
"rvt_dereg_mr timeout mr %p pd %p refcount %u\n",
mr, mr->mr.pd, atomic_read(&mr->mr.refcount));
rvt_get_mr(&mr->mr);
ret = -EBUSY;
goto out;
}
rvt_deinit_mregion(&mr->mr);
if (mr->umem)
ib_umem_release(mr->umem);
kfree(mr);
out:
return ret;
}
/**
* rvt_alloc_mr - Allocate a memory region usable with the
* @pd: protection domain for this memory region
* @mr_type: mem region type
* @max_num_sg: Max number of segments allowed
*
* Return: the memory region on success, otherwise return an errno.
*/
struct ib_mr *rvt_alloc_mr(struct ib_pd *pd,
enum ib_mr_type mr_type,
u32 max_num_sg)
{
struct rvt_mr *mr;
if (mr_type != IB_MR_TYPE_MEM_REG)
return ERR_PTR(-EINVAL);
mr = __rvt_alloc_mr(max_num_sg, pd);
if (IS_ERR(mr))
return (struct ib_mr *)mr;
return &mr->ibmr;
}
/**
* rvt_alloc_fmr - allocate a fast memory region
* @pd: the protection domain for this memory region
* @mr_access_flags: access flags for this memory region
* @fmr_attr: fast memory region attributes
*
* Return: the memory region on success, otherwise returns an errno.
*/
struct ib_fmr *rvt_alloc_fmr(struct ib_pd *pd, int mr_access_flags,
struct ib_fmr_attr *fmr_attr)
{
struct rvt_fmr *fmr;
int m;
struct ib_fmr *ret;
int rval = -ENOMEM;
/* Allocate struct plus pointers to first level page tables. */
m = (fmr_attr->max_pages + RVT_SEGSZ - 1) / RVT_SEGSZ;
fmr = kzalloc(sizeof(*fmr) + m * sizeof(fmr->mr.map[0]), GFP_KERNEL);
if (!fmr)
goto bail;
rval = rvt_init_mregion(&fmr->mr, pd, fmr_attr->max_pages);
if (rval)
goto bail;
/*
* ib_alloc_fmr() will initialize fmr->ibfmr except for lkey &
* rkey.
*/
rval = rvt_alloc_lkey(&fmr->mr, 0);
if (rval)
goto bail_mregion;
fmr->ibfmr.rkey = fmr->mr.lkey;
fmr->ibfmr.lkey = fmr->mr.lkey;
/*
* Resources are allocated but no valid mapping (RKEY can't be
* used).
*/
fmr->mr.access_flags = mr_access_flags;
fmr->mr.max_segs = fmr_attr->max_pages;
fmr->mr.page_shift = fmr_attr->page_shift;
ret = &fmr->ibfmr;
done:
return ret;
bail_mregion:
rvt_deinit_mregion(&fmr->mr);
bail:
kfree(fmr);
ret = ERR_PTR(rval);
goto done;
}
/**
* rvt_map_phys_fmr - set up a fast memory region
* @ibmfr: the fast memory region to set up
* @page_list: the list of pages to associate with the fast memory region
* @list_len: the number of pages to associate with the fast memory region
* @iova: the virtual address of the start of the fast memory region
*
* This may be called from interrupt context.
*
* Return: 0 on success
*/
int rvt_map_phys_fmr(struct ib_fmr *ibfmr, u64 *page_list,
int list_len, u64 iova)
{
struct rvt_fmr *fmr = to_ifmr(ibfmr);
struct rvt_lkey_table *rkt;
unsigned long flags;
int m, n, i;
u32 ps;
struct rvt_dev_info *rdi = ib_to_rvt(ibfmr->device);
i = atomic_read(&fmr->mr.refcount);
if (i > 2)
return -EBUSY;
if (list_len > fmr->mr.max_segs)
return -EINVAL;
rkt = &rdi->lkey_table;
spin_lock_irqsave(&rkt->lock, flags);
fmr->mr.user_base = iova;
fmr->mr.iova = iova;
ps = 1 << fmr->mr.page_shift;
fmr->mr.length = list_len * ps;
m = 0;
n = 0;
for (i = 0; i < list_len; i++) {
fmr->mr.map[m]->segs[n].vaddr = (void *)page_list[i];
fmr->mr.map[m]->segs[n].length = ps;
if (++n == RVT_SEGSZ) {
m++;
n = 0;
}
}
spin_unlock_irqrestore(&rkt->lock, flags);
return 0;
}
/**
* rvt_unmap_fmr - unmap fast memory regions
* @fmr_list: the list of fast memory regions to unmap
*
* Return: 0 on success.
*/
int rvt_unmap_fmr(struct list_head *fmr_list)
{
struct rvt_fmr *fmr;
struct rvt_lkey_table *rkt;
unsigned long flags;
struct rvt_dev_info *rdi;
list_for_each_entry(fmr, fmr_list, ibfmr.list) {
rdi = ib_to_rvt(fmr->ibfmr.device);
rkt = &rdi->lkey_table;
spin_lock_irqsave(&rkt->lock, flags);
fmr->mr.user_base = 0;
fmr->mr.iova = 0;
fmr->mr.length = 0;
spin_unlock_irqrestore(&rkt->lock, flags);
}
return 0;
}
/**
* rvt_dealloc_fmr - deallocate a fast memory region
* @ibfmr: the fast memory region to deallocate
*
* Return: 0 on success.
*/
int rvt_dealloc_fmr(struct ib_fmr *ibfmr)
{
struct rvt_fmr *fmr = to_ifmr(ibfmr);
int ret = 0;
unsigned long timeout;
rvt_free_lkey(&fmr->mr);
rvt_put_mr(&fmr->mr); /* will set completion if last */
timeout = wait_for_completion_timeout(&fmr->mr.comp, 5 * HZ);
if (!timeout) {
rvt_get_mr(&fmr->mr);
ret = -EBUSY;
goto out;
}
rvt_deinit_mregion(&fmr->mr);
kfree(fmr);
out:
return ret;
}
/**
* rvt_lkey_ok - check IB SGE for validity and initialize
* @rkt: table containing lkey to check SGE against
* @pd: protection domain
* @isge: outgoing internal SGE
* @sge: SGE to check
* @acc: access flags
*
* Check the IB SGE for validity and initialize our internal version
* of it.
*
* Return: 1 if valid and successful, otherwise returns 0.
*
* increments the reference count upon success
*
*/
int rvt_lkey_ok(struct rvt_lkey_table *rkt, struct rvt_pd *pd,
struct rvt_sge *isge, struct ib_sge *sge, int acc)
{
struct rvt_mregion *mr;
unsigned n, m;
size_t off;
struct rvt_dev_info *dev = ib_to_rvt(pd->ibpd.device);
/*
* We use LKEY == zero for kernel virtual addresses
* (see rvt_get_dma_mr and dma.c).
*/
rcu_read_lock();
if (sge->lkey == 0) {
if (pd->user)
goto bail;
mr = rcu_dereference(dev->dma_mr);
if (!mr)
goto bail;
atomic_inc(&mr->refcount);
rcu_read_unlock();
isge->mr = mr;
isge->vaddr = (void *)sge->addr;
isge->length = sge->length;
isge->sge_length = sge->length;
isge->m = 0;
isge->n = 0;
goto ok;
}
mr = rcu_dereference(
rkt->table[(sge->lkey >> (32 - dev->dparms.lkey_table_size))]);
if (unlikely(!mr || mr->lkey != sge->lkey || mr->pd != &pd->ibpd))
goto bail;
off = sge->addr - mr->user_base;
if (unlikely(sge->addr < mr->user_base ||
off + sge->length > mr->length ||
(mr->access_flags & acc) != acc))
goto bail;
atomic_inc(&mr->refcount);
rcu_read_unlock();
off += mr->offset;
if (mr->page_shift) {
/*
* page sizes are uniform power of 2 so no loop is necessary
* entries_spanned_by_off is the number of times the loop below
* would have executed.
*/
size_t entries_spanned_by_off;
entries_spanned_by_off = off >> mr->page_shift;
off -= (entries_spanned_by_off << mr->page_shift);
m = entries_spanned_by_off / RVT_SEGSZ;
n = entries_spanned_by_off % RVT_SEGSZ;
} else {
m = 0;
n = 0;
while (off >= mr->map[m]->segs[n].length) {
off -= mr->map[m]->segs[n].length;
n++;
if (n >= RVT_SEGSZ) {
m++;
n = 0;
}
}
}
isge->mr = mr;
isge->vaddr = mr->map[m]->segs[n].vaddr + off;
isge->length = mr->map[m]->segs[n].length - off;
isge->sge_length = sge->length;
isge->m = m;
isge->n = n;
ok:
return 1;
bail:
rcu_read_unlock();
return 0;
}
EXPORT_SYMBOL(rvt_lkey_ok);
/**
* rvt_rkey_ok - check the IB virtual address, length, and RKEY
* @qp: qp for validation
* @sge: SGE state
* @len: length of data
* @vaddr: virtual address to place data
* @rkey: rkey to check
* @acc: access flags
*
* Return: 1 if successful, otherwise 0.
*
* increments the reference count upon success
*/
int rvt_rkey_ok(struct rvt_qp *qp, struct rvt_sge *sge,
u32 len, u64 vaddr, u32 rkey, int acc)
{
struct rvt_dev_info *dev = ib_to_rvt(qp->ibqp.device);
struct rvt_lkey_table *rkt = &dev->lkey_table;
struct rvt_mregion *mr;
unsigned n, m;
size_t off;
/*
* We use RKEY == zero for kernel virtual addresses
* (see rvt_get_dma_mr and dma.c).
*/
rcu_read_lock();
if (rkey == 0) {
struct rvt_pd *pd = ibpd_to_rvtpd(qp->ibqp.pd);
struct rvt_dev_info *rdi = ib_to_rvt(pd->ibpd.device);
if (pd->user)
goto bail;
mr = rcu_dereference(rdi->dma_mr);
if (!mr)
goto bail;
atomic_inc(&mr->refcount);
rcu_read_unlock();
sge->mr = mr;
sge->vaddr = (void *)vaddr;
sge->length = len;
sge->sge_length = len;
sge->m = 0;
sge->n = 0;
goto ok;
}
mr = rcu_dereference(
rkt->table[(rkey >> (32 - dev->dparms.lkey_table_size))]);
if (unlikely(!mr || mr->lkey != rkey || qp->ibqp.pd != mr->pd))
goto bail;
off = vaddr - mr->iova;
if (unlikely(vaddr < mr->iova || off + len > mr->length ||
(mr->access_flags & acc) == 0))
goto bail;
atomic_inc(&mr->refcount);
rcu_read_unlock();
off += mr->offset;
if (mr->page_shift) {
/*
* page sizes are uniform power of 2 so no loop is necessary
* entries_spanned_by_off is the number of times the loop below
* would have executed.
*/
size_t entries_spanned_by_off;
entries_spanned_by_off = off >> mr->page_shift;
off -= (entries_spanned_by_off << mr->page_shift);
m = entries_spanned_by_off / RVT_SEGSZ;
n = entries_spanned_by_off % RVT_SEGSZ;
} else {
m = 0;
n = 0;
while (off >= mr->map[m]->segs[n].length) {
off -= mr->map[m]->segs[n].length;
n++;
if (n >= RVT_SEGSZ) {
m++;
n = 0;
}
}
}
sge->mr = mr;
sge->vaddr = mr->map[m]->segs[n].vaddr + off;
sge->length = mr->map[m]->segs[n].length - off;
sge->sge_length = len;
sge->m = m;
sge->n = n;
ok:
return 1;
bail:
rcu_read_unlock();
return 0;
}
EXPORT_SYMBOL(rvt_rkey_ok);

92
drivers/infiniband/sw/rdmavt/mr.h 100644
View File

@ -0,0 +1,92 @@
#ifndef DEF_RVTMR_H
#define DEF_RVTMR_H
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <rdma/rdma_vt.h>
struct rvt_fmr {
struct ib_fmr ibfmr;
struct rvt_mregion mr; /* must be last */
};
struct rvt_mr {
struct ib_mr ibmr;
struct ib_umem *umem;
struct rvt_mregion mr; /* must be last */
};
static inline struct rvt_fmr *to_ifmr(struct ib_fmr *ibfmr)
{
return container_of(ibfmr, struct rvt_fmr, ibfmr);
}
static inline struct rvt_mr *to_imr(struct ib_mr *ibmr)
{
return container_of(ibmr, struct rvt_mr, ibmr);
}
int rvt_driver_mr_init(struct rvt_dev_info *rdi);
void rvt_mr_exit(struct rvt_dev_info *rdi);
/* Mem Regions */
struct ib_mr *rvt_get_dma_mr(struct ib_pd *pd, int acc);
struct ib_mr *rvt_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
u64 virt_addr, int mr_access_flags,
struct ib_udata *udata);
int rvt_dereg_mr(struct ib_mr *ibmr);
struct ib_mr *rvt_alloc_mr(struct ib_pd *pd,
enum ib_mr_type mr_type,
u32 max_num_sg);
struct ib_fmr *rvt_alloc_fmr(struct ib_pd *pd, int mr_access_flags,
struct ib_fmr_attr *fmr_attr);
int rvt_map_phys_fmr(struct ib_fmr *ibfmr, u64 *page_list,
int list_len, u64 iova);
int rvt_unmap_fmr(struct list_head *fmr_list);
int rvt_dealloc_fmr(struct ib_fmr *ibfmr);
#endif /* DEF_RVTMR_H */

119
drivers/infiniband/sw/rdmavt/pd.c 100644
View File

@ -0,0 +1,119 @@
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <linux/slab.h>
#include "pd.h"
/**
* rvt_alloc_pd - allocate a protection domain
* @ibdev: ib device
* @context: optional user context
* @udata: optional user data
*
* Allocate and keep track of a PD.
*
* Return: 0 on success
*/
struct ib_pd *rvt_alloc_pd(struct ib_device *ibdev,
struct ib_ucontext *context,
struct ib_udata *udata)
{
struct rvt_dev_info *dev = ib_to_rvt(ibdev);
struct rvt_pd *pd;
struct ib_pd *ret;
pd = kmalloc(sizeof(*pd), GFP_KERNEL);
if (!pd) {
ret = ERR_PTR(-ENOMEM);
goto bail;
}
/*
* While we could continue allocating protecetion domains, being
* constrained only by system resources. The IBTA spec defines that
* there is a max_pd limit that can be set and we need to check for
* that.
*/
spin_lock(&dev->n_pds_lock);
if (dev->n_pds_allocated == dev->dparms.props.max_pd) {
spin_unlock(&dev->n_pds_lock);
kfree(pd);
ret = ERR_PTR(-ENOMEM);
goto bail;
}
dev->n_pds_allocated++;
spin_unlock(&dev->n_pds_lock);
/* ib_alloc_pd() will initialize pd->ibpd. */
pd->user = udata ? 1 : 0;
ret = &pd->ibpd;
bail:
return ret;
}
/**
* rvt_dealloc_pd - Free PD
* @ibpd: Free up PD
*
* Return: always 0
*/
int rvt_dealloc_pd(struct ib_pd *ibpd)
{
struct rvt_pd *pd = ibpd_to_rvtpd(ibpd);
struct rvt_dev_info *dev = ib_to_rvt(ibpd->device);
spin_lock(&dev->n_pds_lock);
dev->n_pds_allocated--;
spin_unlock(&dev->n_pds_lock);
kfree(pd);
return 0;
}

58
drivers/infiniband/sw/rdmavt/pd.h 100644
View File

@ -0,0 +1,58 @@
#ifndef DEF_RDMAVTPD_H
#define DEF_RDMAVTPD_H
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <rdma/rdma_vt.h>
struct ib_pd *rvt_alloc_pd(struct ib_device *ibdev,
struct ib_ucontext *context,
struct ib_udata *udata);
int rvt_dealloc_pd(struct ib_pd *ibpd);
#endif /* DEF_RDMAVTPD_H */

1696
drivers/infiniband/sw/rdmavt/qp.c 100644
View File

File diff suppressed because it is too large Load Diff

69
drivers/infiniband/sw/rdmavt/qp.h 100644
View File

@ -0,0 +1,69 @@
#ifndef DEF_RVTQP_H
#define DEF_RVTQP_H
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <rdma/rdma_vt.h>
int rvt_driver_qp_init(struct rvt_dev_info *rdi);
void rvt_qp_exit(struct rvt_dev_info *rdi);
struct ib_qp *rvt_create_qp(struct ib_pd *ibpd,
struct ib_qp_init_attr *init_attr,
struct ib_udata *udata);
int rvt_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
int attr_mask, struct ib_udata *udata);
int rvt_destroy_qp(struct ib_qp *ibqp);
int rvt_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
int attr_mask, struct ib_qp_init_attr *init_attr);
int rvt_post_recv(struct ib_qp *ibqp, struct ib_recv_wr *wr,
struct ib_recv_wr **bad_wr);
int rvt_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
struct ib_send_wr **bad_wr);
int rvt_post_srq_recv(struct ib_srq *ibsrq, struct ib_recv_wr *wr,
struct ib_recv_wr **bad_wr);
#endif /* DEF_RVTQP_H */

204
drivers/staging/rdma/hfi1/srq.c → drivers/infiniband/sw/rdmavt/srq.c
View File

@ -1,12 +1,11 @@
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* Copyright(c) 2015 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
@ -18,8 +17,6 @@
*
* BSD LICENSE
*
* Copyright(c) 2015 Intel Corporation.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
@ -52,96 +49,50 @@
#include <linux/slab.h>
#include <linux/vmalloc.h>
#include "verbs.h"
#include "srq.h"
#include "vt.h"
/**
* hfi1_post_srq_receive - post a receive on a shared receive queue
* @ibsrq: the SRQ to post the receive on
* @wr: the list of work requests to post
* @bad_wr: A pointer to the first WR to cause a problem is put here
* rvt_driver_srq_init - init srq resources on a per driver basis
* @rdi: rvt dev structure
*
* This may be called from interrupt context.
* Do any initialization needed when a driver registers with rdmavt.
*/
int hfi1_post_srq_receive(struct ib_srq *ibsrq, struct ib_recv_wr *wr,
struct ib_recv_wr **bad_wr)
void rvt_driver_srq_init(struct rvt_dev_info *rdi)
{
struct hfi1_srq *srq = to_isrq(ibsrq);
struct hfi1_rwq *wq;
unsigned long flags;
int ret;
for (; wr; wr = wr->next) {
struct hfi1_rwqe *wqe;
u32 next;
int i;
if ((unsigned) wr->num_sge > srq->rq.max_sge) {
*bad_wr = wr;
ret = -EINVAL;
goto bail;
}
spin_lock_irqsave(&srq->rq.lock, flags);
wq = srq->rq.wq;
next = wq->head + 1;
if (next >= srq->rq.size)
next = 0;
if (next == wq->tail) {
spin_unlock_irqrestore(&srq->rq.lock, flags);
*bad_wr = wr;
ret = -ENOMEM;
goto bail;
}
wqe = get_rwqe_ptr(&srq->rq, wq->head);
wqe->wr_id = wr->wr_id;
wqe->num_sge = wr->num_sge;
for (i = 0; i < wr->num_sge; i++)
wqe->sg_list[i] = wr->sg_list[i];
/* Make sure queue entry is written before the head index. */
smp_wmb();
wq->head = next;
spin_unlock_irqrestore(&srq->rq.lock, flags);
}
ret = 0;
bail:
return ret;
spin_lock_init(&rdi->n_srqs_lock);
rdi->n_srqs_allocated = 0;
}
/**
* hfi1_create_srq - create a shared receive queue
* rvt_create_srq - create a shared receive queue
* @ibpd: the protection domain of the SRQ to create
* @srq_init_attr: the attributes of the SRQ
* @udata: data from libibverbs when creating a user SRQ
*
* Return: Allocated srq object
*/
struct ib_srq *hfi1_create_srq(struct ib_pd *ibpd,
struct ib_srq_init_attr *srq_init_attr,
struct ib_udata *udata)
struct ib_srq *rvt_create_srq(struct ib_pd *ibpd,
struct ib_srq_init_attr *srq_init_attr,
struct ib_udata *udata)
{
struct hfi1_ibdev *dev = to_idev(ibpd->device);
struct hfi1_srq *srq;
struct rvt_dev_info *dev = ib_to_rvt(ibpd->device);
struct rvt_srq *srq;
u32 sz;
struct ib_srq *ret;
if (srq_init_attr->srq_type != IB_SRQT_BASIC) {
ret = ERR_PTR(-ENOSYS);
goto done;
}
if (srq_init_attr->srq_type != IB_SRQT_BASIC)
return ERR_PTR(-ENOSYS);
if (srq_init_attr->attr.max_sge == 0 ||
srq_init_attr->attr.max_sge > hfi1_max_srq_sges ||
srq_init_attr->attr.max_sge > dev->dparms.props.max_srq_sge ||
srq_init_attr->attr.max_wr == 0 ||
srq_init_attr->attr.max_wr > hfi1_max_srq_wrs) {
ret = ERR_PTR(-EINVAL);
goto done;
}
srq_init_attr->attr.max_wr > dev->dparms.props.max_srq_wr)
return ERR_PTR(-EINVAL);
srq = kmalloc(sizeof(*srq), GFP_KERNEL);
if (!srq) {
ret = ERR_PTR(-ENOMEM);
goto done;
}
if (!srq)
return ERR_PTR(-ENOMEM);
/*
* Need to use vmalloc() if we want to support large #s of entries.
@ -149,8 +100,8 @@ struct ib_srq *hfi1_create_srq(struct ib_pd *ibpd,
srq->rq.size = srq_init_attr->attr.max_wr + 1;
srq->rq.max_sge = srq_init_attr->attr.max_sge;
sz = sizeof(struct ib_sge) * srq->rq.max_sge +
sizeof(struct hfi1_rwqe);
srq->rq.wq = vmalloc_user(sizeof(struct hfi1_rwq) + srq->rq.size * sz);
sizeof(struct rvt_rwqe);
srq->rq.wq = vmalloc_user(sizeof(struct rvt_rwq) + srq->rq.size * sz);
if (!srq->rq.wq) {
ret = ERR_PTR(-ENOMEM);
goto bail_srq;
@ -158,15 +109,15 @@ struct ib_srq *hfi1_create_srq(struct ib_pd *ibpd,
/*
* Return the address of the RWQ as the offset to mmap.
* See hfi1_mmap() for details.
* See rvt_mmap() for details.
*/
if (udata && udata->outlen >= sizeof(__u64)) {
int err;
u32 s = sizeof(struct hfi1_rwq) + srq->rq.size * sz;
u32 s = sizeof(struct rvt_rwq) + srq->rq.size * sz;
srq->ip =
hfi1_create_mmap_info(dev, s, ibpd->uobject->context,
srq->rq.wq);
rvt_create_mmap_info(dev, s, ibpd->uobject->context,
srq->rq.wq);
if (!srq->ip) {
ret = ERR_PTR(-ENOMEM);
goto bail_wq;
@ -178,8 +129,9 @@ struct ib_srq *hfi1_create_srq(struct ib_pd *ibpd,
ret = ERR_PTR(err);
goto bail_ip;
}
} else
} else {
srq->ip = NULL;
}
/*
* ib_create_srq() will initialize srq->ibsrq.
@ -190,7 +142,7 @@ struct ib_srq *hfi1_create_srq(struct ib_pd *ibpd,
srq->limit = srq_init_attr->attr.srq_limit;
spin_lock(&dev->n_srqs_lock);
if (dev->n_srqs_allocated == hfi1_max_srqs) {
if (dev->n_srqs_allocated == dev->dparms.props.max_srq) {
spin_unlock(&dev->n_srqs_lock);
ret = ERR_PTR(-ENOMEM);
goto bail_ip;
@ -205,8 +157,7 @@ struct ib_srq *hfi1_create_srq(struct ib_pd *ibpd,
spin_unlock_irq(&dev->pending_lock);
}
ret = &srq->ibsrq;
goto done;
return &srq->ibsrq;
bail_ip:
kfree(srq->ip);
@ -214,46 +165,44 @@ bail_wq:
vfree(srq->rq.wq);
bail_srq:
kfree(srq);
done:
return ret;
}
/**
* hfi1_modify_srq - modify a shared receive queue
* rvt_modify_srq - modify a shared receive queue
* @ibsrq: the SRQ to modify
* @attr: the new attributes of the SRQ
* @attr_mask: indicates which attributes to modify
* @udata: user data for libibverbs.so
*
* Return: 0 on success
*/
int hfi1_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
enum ib_srq_attr_mask attr_mask,
struct ib_udata *udata)
int rvt_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
enum ib_srq_attr_mask attr_mask,
struct ib_udata *udata)
{
struct hfi1_srq *srq = to_isrq(ibsrq);
struct hfi1_rwq *wq;
struct rvt_srq *srq = ibsrq_to_rvtsrq(ibsrq);
struct rvt_dev_info *dev = ib_to_rvt(ibsrq->device);
struct rvt_rwq *wq;
int ret = 0;
if (attr_mask & IB_SRQ_MAX_WR) {
struct hfi1_rwq *owq;
struct hfi1_rwqe *p;
struct rvt_rwq *owq;
struct rvt_rwqe *p;
u32 sz, size, n, head, tail;
/* Check that the requested sizes are below the limits. */
if ((attr->max_wr > hfi1_max_srq_wrs) ||
if ((attr->max_wr > dev->dparms.props.max_srq_wr) ||
((attr_mask & IB_SRQ_LIMIT) ?
attr->srq_limit : srq->limit) > attr->max_wr) {
ret = -EINVAL;
goto bail;
}
attr->srq_limit : srq->limit) > attr->max_wr)
return -EINVAL;
sz = sizeof(struct hfi1_rwqe) +
sz = sizeof(struct rvt_rwqe) +
srq->rq.max_sge * sizeof(struct ib_sge);
size = attr->max_wr + 1;
wq = vmalloc_user(sizeof(struct hfi1_rwq) + size * sz);
if (!wq) {
ret = -ENOMEM;
goto bail;
}
wq = vmalloc_user(sizeof(struct rvt_rwq) + size * sz);
if (!wq)
return -ENOMEM;
/* Check that we can write the offset to mmap. */
if (udata && udata->inlen >= sizeof(__u64)) {
@ -264,8 +213,8 @@ int hfi1_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
sizeof(offset_addr));
if (ret)
goto bail_free;
udata->outbuf =
(void __user *) (unsigned long) offset_addr;
udata->outbuf = (void __user *)
(unsigned long)offset_addr;
ret = ib_copy_to_udata(udata, &offset,
sizeof(offset));
if (ret)
@ -296,16 +245,16 @@ int hfi1_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
n = 0;
p = wq->wq;
while (tail != head) {
struct hfi1_rwqe *wqe;
struct rvt_rwqe *wqe;
int i;
wqe = get_rwqe_ptr(&srq->rq, tail);
wqe = rvt_get_rwqe_ptr(&srq->rq, tail);
p->wr_id = wqe->wr_id;
p->num_sge = wqe->num_sge;
for (i = 0; i < wqe->num_sge; i++)
p->sg_list[i] = wqe->sg_list[i];
n++;
p = (struct hfi1_rwqe *)((char *)p + sz);
p = (struct rvt_rwqe *)((char *)p + sz);
if (++tail >= srq->rq.size)
tail = 0;
}
@ -320,21 +269,21 @@ int hfi1_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
vfree(owq);
if (srq->ip) {
struct hfi1_mmap_info *ip = srq->ip;
struct hfi1_ibdev *dev = to_idev(srq->ibsrq.device);
u32 s = sizeof(struct hfi1_rwq) + size * sz;
struct rvt_mmap_info *ip = srq->ip;
struct rvt_dev_info *dev = ib_to_rvt(srq->ibsrq.device);
u32 s = sizeof(struct rvt_rwq) + size * sz;
hfi1_update_mmap_info(dev, ip, s, wq);
rvt_update_mmap_info(dev, ip, s, wq);
/*
* Return the offset to mmap.
* See hfi1_mmap() for details.
* See rvt_mmap() for details.
*/
if (udata && udata->inlen >= sizeof(__u64)) {
ret = ib_copy_to_udata(udata, &ip->offset,
sizeof(ip->offset));
if (ret)
goto bail;
return ret;
}
/*
@ -355,19 +304,24 @@ int hfi1_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
srq->limit = attr->srq_limit;
spin_unlock_irq(&srq->rq.lock);
}
goto bail;
return ret;
bail_unlock:
spin_unlock_irq(&srq->rq.lock);
bail_free:
vfree(wq);
bail:
return ret;
}
int hfi1_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr)
/** rvt_query_srq - query srq data
* @ibsrq: srq to query
* @attr: return info in attr
*
* Return: always 0
*/
int rvt_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr)
{
struct hfi1_srq *srq = to_isrq(ibsrq);
struct rvt_srq *srq = ibsrq_to_rvtsrq(ibsrq);
attr->max_wr = srq->rq.size - 1;
attr->max_sge = srq->rq.max_sge;
@ -376,19 +330,21 @@ int hfi1_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr)
}
/**
* hfi1_destroy_srq - destroy a shared receive queue
* @ibsrq: the SRQ to destroy
* rvt_destroy_srq - destory an srq
* @ibsrq: srq object to destroy
*
* Return always 0
*/
int hfi1_destroy_srq(struct ib_srq *ibsrq)
int rvt_destroy_srq(struct ib_srq *ibsrq)
{
struct hfi1_srq *srq = to_isrq(ibsrq);
struct hfi1_ibdev *dev = to_idev(ibsrq->device);
struct rvt_srq *srq = ibsrq_to_rvtsrq(ibsrq);
struct rvt_dev_info *dev = ib_to_rvt(ibsrq->device);
spin_lock(&dev->n_srqs_lock);
dev->n_srqs_allocated--;
spin_unlock(&dev->n_srqs_lock);
if (srq->ip)
kref_put(&srq->ip->ref, hfi1_release_mmap_info);
kref_put(&srq->ip->ref, rvt_release_mmap_info);
else
vfree(srq->rq.wq);
kfree(srq);

62
drivers/infiniband/sw/rdmavt/srq.h 100644
View File

@ -0,0 +1,62 @@
#ifndef DEF_RVTSRQ_H
#define DEF_RVTSRQ_H
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <rdma/rdma_vt.h>
void rvt_driver_srq_init(struct rvt_dev_info *rdi);
struct ib_srq *rvt_create_srq(struct ib_pd *ibpd,
struct ib_srq_init_attr *srq_init_attr,
struct ib_udata *udata);
int rvt_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
enum ib_srq_attr_mask attr_mask,
struct ib_udata *udata);
int rvt_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr);
int rvt_destroy_srq(struct ib_srq *ibsrq);
#endif /* DEF_RVTSRQ_H */

49
drivers/infiniband/sw/rdmavt/trace.c 100644
View File

@ -0,0 +1,49 @@
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#define CREATE_TRACE_POINTS
#include "trace.h"

187
drivers/infiniband/sw/rdmavt/trace.h 100644
View File

@ -0,0 +1,187 @@
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#undef TRACE_SYSTEM_VAR
#define TRACE_SYSTEM_VAR rdmavt
#if !defined(__RDMAVT_TRACE_H) || defined(TRACE_HEADER_MULTI_READ)
#define __RDMAVT_TRACE_H
#include <linux/tracepoint.h>
#include <linux/trace_seq.h>
#include <rdma/ib_verbs.h>
#include <rdma/rdma_vt.h>
#define RDI_DEV_ENTRY(rdi) __string(dev, rdi->driver_f.get_card_name(rdi))
#define RDI_DEV_ASSIGN(rdi) __assign_str(dev, rdi->driver_f.get_card_name(rdi))
#undef TRACE_SYSTEM
#define TRACE_SYSTEM rdmavt
TRACE_EVENT(rvt_dbg,
TP_PROTO(struct rvt_dev_info *rdi,
const char *msg),
TP_ARGS(rdi, msg),
TP_STRUCT__entry(
RDI_DEV_ENTRY(rdi)
__string(msg, msg)
),
TP_fast_assign(
RDI_DEV_ASSIGN(rdi);
__assign_str(msg, msg);
),
TP_printk("[%s]: %s", __get_str(dev), __get_str(msg))
);
#undef TRACE_SYSTEM
#define TRACE_SYSTEM rvt_qphash
DECLARE_EVENT_CLASS(rvt_qphash_template,
TP_PROTO(struct rvt_qp *qp, u32 bucket),
TP_ARGS(qp, bucket),
TP_STRUCT__entry(
RDI_DEV_ENTRY(ib_to_rvt(qp->ibqp.device))
__field(u32, qpn)
__field(u32, bucket)
),
TP_fast_assign(
RDI_DEV_ASSIGN(ib_to_rvt(qp->ibqp.device))
__entry->qpn = qp->ibqp.qp_num;
__entry->bucket = bucket;
),
TP_printk(
"[%s] qpn 0x%x bucket %u",
__get_str(dev),
__entry->qpn,
__entry->bucket
)
);
DEFINE_EVENT(rvt_qphash_template, rvt_qpinsert,
TP_PROTO(struct rvt_qp *qp, u32 bucket),
TP_ARGS(qp, bucket));
DEFINE_EVENT(rvt_qphash_template, rvt_qpremove,
TP_PROTO(struct rvt_qp *qp, u32 bucket),
TP_ARGS(qp, bucket));
#undef TRACE_SYSTEM
#define TRACE_SYSTEM rvt_tx
#define wr_opcode_name(opcode) { IB_WR_##opcode, #opcode }
#define show_wr_opcode(opcode) \
__print_symbolic(opcode, \
wr_opcode_name(RDMA_WRITE), \
wr_opcode_name(RDMA_WRITE_WITH_IMM), \
wr_opcode_name(SEND), \
wr_opcode_name(SEND_WITH_IMM), \
wr_opcode_name(RDMA_READ), \
wr_opcode_name(ATOMIC_CMP_AND_SWP), \
wr_opcode_name(ATOMIC_FETCH_AND_ADD), \
wr_opcode_name(LSO), \
wr_opcode_name(SEND_WITH_INV), \
wr_opcode_name(RDMA_READ_WITH_INV), \
wr_opcode_name(LOCAL_INV), \
wr_opcode_name(MASKED_ATOMIC_CMP_AND_SWP), \
wr_opcode_name(MASKED_ATOMIC_FETCH_AND_ADD))
#define POS_PRN \
"[%s] wr_id %llx qpn %x psn 0x%x lpsn 0x%x length %u opcode 0x%.2x,%s size %u avail %u head %u last %u"
TRACE_EVENT(
rvt_post_one_wr,
TP_PROTO(struct rvt_qp *qp, struct rvt_swqe *wqe),
TP_ARGS(qp, wqe),
TP_STRUCT__entry(
RDI_DEV_ENTRY(ib_to_rvt(qp->ibqp.device))
__field(u64, wr_id)
__field(u32, qpn)
__field(u32, psn)
__field(u32, lpsn)
__field(u32, length)
__field(u32, opcode)
__field(u32, size)
__field(u32, avail)
__field(u32, head)
__field(u32, last)
),
TP_fast_assign(
RDI_DEV_ASSIGN(ib_to_rvt(qp->ibqp.device))
__entry->wr_id = wqe->wr.wr_id;
__entry->qpn = qp->ibqp.qp_num;
__entry->psn = wqe->psn;
__entry->lpsn = wqe->lpsn;
__entry->length = wqe->length;
__entry->opcode = wqe->wr.opcode;
__entry->size = qp->s_size;
__entry->avail = qp->s_avail;
__entry->head = qp->s_head;
__entry->last = qp->s_last;
),
TP_printk(
POS_PRN,
__get_str(dev),
__entry->wr_id,
__entry->qpn,
__entry->psn,
__entry->lpsn,
__entry->length,
__entry->opcode, show_wr_opcode(__entry->opcode),
__entry->size,
__entry->avail,
__entry->head,
__entry->last
)
);
#endif /* __RDMAVT_TRACE_H */
#undef TRACE_INCLUDE_PATH
#undef TRACE_INCLUDE_FILE
#define TRACE_INCLUDE_PATH .
#define TRACE_INCLUDE_FILE trace
#include <trace/define_trace.h>

873
drivers/infiniband/sw/rdmavt/vt.c 100644
View File

@ -0,0 +1,873 @@
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <linux/module.h>
#include <linux/kernel.h>
#include "vt.h"
#include "trace.h"
#define RVT_UVERBS_ABI_VERSION 2
MODULE_LICENSE("Dual BSD/GPL");
MODULE_DESCRIPTION("RDMA Verbs Transport Library");
static int rvt_init(void)
{
/*
* rdmavt does not need to do anything special when it starts up. All it
* needs to do is sit and wait until a driver attempts registration.
*/
return 0;
}
module_init(rvt_init);
static void rvt_cleanup(void)
{
/*
* Nothing to do at exit time either. The module won't be able to be
* removed until all drivers are gone which means all the dev structs
* are gone so there is really nothing to do.
*/
}
module_exit(rvt_cleanup);
/**
* rvt_alloc_device - allocate rdi
* @size: how big of a structure to allocate
* @nports: number of ports to allocate array slots for
*
* Use IB core device alloc to allocate space for the rdi which is assumed to be
* inside of the ib_device. Any extra space that drivers require should be
* included in size.
*
* We also allocate a port array based on the number of ports.
*
* Return: pointer to allocated rdi
*/
struct rvt_dev_info *rvt_alloc_device(size_t size, int nports)
{
struct rvt_dev_info *rdi = ERR_PTR(-ENOMEM);
rdi = (struct rvt_dev_info *)ib_alloc_device(size);
if (!rdi)
return rdi;
rdi->ports = kcalloc(nports,
sizeof(struct rvt_ibport **),
GFP_KERNEL);
if (!rdi->ports)
ib_dealloc_device(&rdi->ibdev);
return rdi;
}
EXPORT_SYMBOL(rvt_alloc_device);
static int rvt_query_device(struct ib_device *ibdev,
struct ib_device_attr *props,
struct ib_udata *uhw)
{
struct rvt_dev_info *rdi = ib_to_rvt(ibdev);
if (uhw->inlen || uhw->outlen)
return -EINVAL;
/*
* Return rvt_dev_info.dparms.props contents
*/
*props = rdi->dparms.props;
return 0;
}
static int rvt_modify_device(struct ib_device *device,
int device_modify_mask,
struct ib_device_modify *device_modify)
{
/*
* There is currently no need to supply this based on qib and hfi1.
* Future drivers may need to implement this though.
*/
return -EOPNOTSUPP;
}
/**
* rvt_query_port: Passes the query port call to the driver
* @ibdev: Verbs IB dev
* @port_num: port number, 1 based from ib core
* @props: structure to hold returned properties
*
* Return: 0 on success
*/
static int rvt_query_port(struct ib_device *ibdev, u8 port_num,
struct ib_port_attr *props)
{
struct rvt_dev_info *rdi = ib_to_rvt(ibdev);
struct rvt_ibport *rvp;
int port_index = ibport_num_to_idx(ibdev, port_num);
if (port_index < 0)
return -EINVAL;
rvp = rdi->ports[port_index];
memset(props, 0, sizeof(*props));
props->sm_lid = rvp->sm_lid;
props->sm_sl = rvp->sm_sl;
props->port_cap_flags = rvp->port_cap_flags;
props->max_msg_sz = 0x80000000;
props->pkey_tbl_len = rvt_get_npkeys(rdi);
props->bad_pkey_cntr = rvp->pkey_violations;
props->qkey_viol_cntr = rvp->qkey_violations;
props->subnet_timeout = rvp->subnet_timeout;
props->init_type_reply = 0;
/* Populate the remaining ib_port_attr elements */
return rdi->driver_f.query_port_state(rdi, port_num, props);
}
/**
* rvt_modify_port
* @ibdev: Verbs IB dev
* @port_num: Port number, 1 based from ib core
* @port_modify_mask: How to change the port
* @props: Structure to fill in
*
* Return: 0 on success
*/
static int rvt_modify_port(struct ib_device *ibdev, u8 port_num,
int port_modify_mask, struct ib_port_modify *props)
{
struct rvt_dev_info *rdi = ib_to_rvt(ibdev);
struct rvt_ibport *rvp;
int ret = 0;
int port_index = ibport_num_to_idx(ibdev, port_num);
if (port_index < 0)
return -EINVAL;
rvp = rdi->ports[port_index];
rvp->port_cap_flags |= props->set_port_cap_mask;
rvp->port_cap_flags &= ~props->clr_port_cap_mask;
if (props->set_port_cap_mask || props->clr_port_cap_mask)
rdi->driver_f.cap_mask_chg(rdi, port_num);
if (port_modify_mask & IB_PORT_SHUTDOWN)
ret = rdi->driver_f.shut_down_port(rdi, port_num);
if (port_modify_mask & IB_PORT_RESET_QKEY_CNTR)
rvp->qkey_violations = 0;
return ret;
}
/**
* rvt_query_pkey - Return a pkey from the table at a given index
* @ibdev: Verbs IB dev
* @port_num: Port number, 1 based from ib core
* @intex: Index into pkey table
*
* Return: 0 on failure pkey otherwise
*/
static int rvt_query_pkey(struct ib_device *ibdev, u8 port_num, u16 index,
u16 *pkey)
{
/*
* Driver will be responsible for keeping rvt_dev_info.pkey_table up to
* date. This function will just return that value. There is no need to
* lock, if a stale value is read and sent to the user so be it there is
* no way to protect against that anyway.
*/
struct rvt_dev_info *rdi = ib_to_rvt(ibdev);
int port_index;
port_index = ibport_num_to_idx(ibdev, port_num);
if (port_index < 0)
return -EINVAL;
if (index >= rvt_get_npkeys(rdi))
return -EINVAL;
*pkey = rvt_get_pkey(rdi, port_index, index);
return 0;
}
/**
* rvt_query_gid - Return a gid from the table
* @ibdev: Verbs IB dev
* @port_num: Port number, 1 based from ib core
* @index: = Index in table
* @gid: Gid to return
*
* Return: 0 on success
*/
static int rvt_query_gid(struct ib_device *ibdev, u8 port_num,
int guid_index, union ib_gid *gid)
{
struct rvt_dev_info *rdi;
struct rvt_ibport *rvp;
int port_index;
/*
* Driver is responsible for updating the guid table. Which will be used
* to craft the return value. This will work similar to how query_pkey()
* is being done.
*/
port_index = ibport_num_to_idx(ibdev, port_num);
if (port_index < 0)
return -EINVAL;
rdi = ib_to_rvt(ibdev);
rvp = rdi->ports[port_index];
gid->global.subnet_prefix = rvp->gid_prefix;
return rdi->driver_f.get_guid_be(rdi, rvp, guid_index,
&gid->global.interface_id);
}
struct rvt_ucontext {
struct ib_ucontext ibucontext;
};
static inline struct rvt_ucontext *to_iucontext(struct ib_ucontext
*ibucontext)
{
return container_of(ibucontext, struct rvt_ucontext, ibucontext);
}
/**
* rvt_alloc_ucontext - Allocate a user context
* @ibdev: Vers IB dev
* @data: User data allocated
*/
static struct ib_ucontext *rvt_alloc_ucontext(struct ib_device *ibdev,
struct ib_udata *udata)
{
struct rvt_ucontext *context;
context = kmalloc(sizeof(*context), GFP_KERNEL);
if (!context)
return ERR_PTR(-ENOMEM);
return &context->ibucontext;
}
/**
*rvt_dealloc_ucontext - Free a user context
*@context - Free this
*/
static int rvt_dealloc_ucontext(struct ib_ucontext *context)
{
kfree(to_iucontext(context));
return 0;
}
static int rvt_get_port_immutable(struct ib_device *ibdev, u8 port_num,
struct ib_port_immutable *immutable)
{
struct rvt_dev_info *rdi = ib_to_rvt(ibdev);
struct ib_port_attr attr;
int err, port_index;
port_index = ibport_num_to_idx(ibdev, port_num);
if (port_index < 0)
return -EINVAL;
err = rvt_query_port(ibdev, port_num, &attr);
if (err)
return err;
immutable->pkey_tbl_len = attr.pkey_tbl_len;
immutable->gid_tbl_len = attr.gid_tbl_len;
immutable->core_cap_flags = rdi->dparms.core_cap_flags;
immutable->max_mad_size = rdi->dparms.max_mad_size;
return 0;
}
enum {
MISC,
QUERY_DEVICE,
MODIFY_DEVICE,
QUERY_PORT,
MODIFY_PORT,
QUERY_PKEY,
QUERY_GID,
ALLOC_UCONTEXT,
DEALLOC_UCONTEXT,
GET_PORT_IMMUTABLE,
CREATE_QP,
MODIFY_QP,
DESTROY_QP,
QUERY_QP,
POST_SEND,
POST_RECV,
POST_SRQ_RECV,
CREATE_AH,
DESTROY_AH,
MODIFY_AH,
QUERY_AH,
CREATE_SRQ,
MODIFY_SRQ,
DESTROY_SRQ,
QUERY_SRQ,
ATTACH_MCAST,
DETACH_MCAST,
GET_DMA_MR,
REG_USER_MR,
DEREG_MR,
ALLOC_MR,
ALLOC_FMR,
MAP_PHYS_FMR,
UNMAP_FMR,
DEALLOC_FMR,
MMAP,
CREATE_CQ,
DESTROY_CQ,
POLL_CQ,
REQ_NOTFIY_CQ,
RESIZE_CQ,
ALLOC_PD,
DEALLOC_PD,
_VERB_IDX_MAX /* Must always be last! */
};
static inline int check_driver_override(struct rvt_dev_info *rdi,
size_t offset, void *func)
{
if (!*(void **)((void *)&rdi->ibdev + offset)) {
*(void **)((void *)&rdi->ibdev + offset) = func;
return 0;
}
return 1;
}
static noinline int check_support(struct rvt_dev_info *rdi, int verb)
{
switch (verb) {
case MISC:
/*
* These functions are not part of verbs specifically but are
* required for rdmavt to function.
*/
if ((!rdi->driver_f.port_callback) ||
(!rdi->driver_f.get_card_name) ||
(!rdi->driver_f.get_pci_dev))
return -EINVAL;
break;
case QUERY_DEVICE:
check_driver_override(rdi, offsetof(struct ib_device,
query_device),
rvt_query_device);
break;
case MODIFY_DEVICE:
/*
* rdmavt does not support modify device currently drivers must
* provide.
*/
if (!check_driver_override(rdi, offsetof(struct ib_device,
modify_device),
rvt_modify_device))
return -EOPNOTSUPP;
break;
case QUERY_PORT:
if (!check_driver_override(rdi, offsetof(struct ib_device,
query_port),
rvt_query_port))
if (!rdi->driver_f.query_port_state)
return -EINVAL;
break;
case MODIFY_PORT:
if (!check_driver_override(rdi, offsetof(struct ib_device,
modify_port),
rvt_modify_port))
if (!rdi->driver_f.cap_mask_chg ||
!rdi->driver_f.shut_down_port)
return -EINVAL;
break;
case QUERY_PKEY:
check_driver_override(rdi, offsetof(struct ib_device,
query_pkey),
rvt_query_pkey);
break;
case QUERY_GID:
if (!check_driver_override(rdi, offsetof(struct ib_device,
query_gid),
rvt_query_gid))
if (!rdi->driver_f.get_guid_be)
return -EINVAL;
break;
case ALLOC_UCONTEXT:
check_driver_override(rdi, offsetof(struct ib_device,
alloc_ucontext),
rvt_alloc_ucontext);
break;
case DEALLOC_UCONTEXT:
check_driver_override(rdi, offsetof(struct ib_device,
dealloc_ucontext),
rvt_dealloc_ucontext);
break;
case GET_PORT_IMMUTABLE:
check_driver_override(rdi, offsetof(struct ib_device,
get_port_immutable),
rvt_get_port_immutable);
break;
case CREATE_QP:
if (!check_driver_override(rdi, offsetof(struct ib_device,
create_qp),
rvt_create_qp))
if (!rdi->driver_f.qp_priv_alloc ||
!rdi->driver_f.qp_priv_free ||
!rdi->driver_f.notify_qp_reset ||
!rdi->driver_f.flush_qp_waiters ||
!rdi->driver_f.stop_send_queue ||
!rdi->driver_f.quiesce_qp)
return -EINVAL;
break;
case MODIFY_QP:
if (!check_driver_override(rdi, offsetof(struct ib_device,
modify_qp),
rvt_modify_qp))
if (!rdi->driver_f.notify_qp_reset ||
!rdi->driver_f.schedule_send ||
!rdi->driver_f.get_pmtu_from_attr ||
!rdi->driver_f.flush_qp_waiters ||
!rdi->driver_f.stop_send_queue ||
!rdi->driver_f.quiesce_qp ||
!rdi->driver_f.notify_error_qp ||
!rdi->driver_f.mtu_from_qp ||
!rdi->driver_f.mtu_to_path_mtu ||
!rdi->driver_f.shut_down_port ||
!rdi->driver_f.cap_mask_chg)
return -EINVAL;
break;
case DESTROY_QP:
if (!check_driver_override(rdi, offsetof(struct ib_device,
destroy_qp),
rvt_destroy_qp))
if (!rdi->driver_f.qp_priv_free ||
!rdi->driver_f.notify_qp_reset ||
!rdi->driver_f.flush_qp_waiters ||
!rdi->driver_f.stop_send_queue ||
!rdi->driver_f.quiesce_qp)
return -EINVAL;
break;
case QUERY_QP:
check_driver_override(rdi, offsetof(struct ib_device,
query_qp),
rvt_query_qp);
break;
case POST_SEND:
if (!check_driver_override(rdi, offsetof(struct ib_device,
post_send),
rvt_post_send))
if (!rdi->driver_f.schedule_send ||
!rdi->driver_f.do_send)
return -EINVAL;
break;
case POST_RECV:
check_driver_override(rdi, offsetof(struct ib_device,
post_recv),
rvt_post_recv);
break;
case POST_SRQ_RECV:
check_driver_override(rdi, offsetof(struct ib_device,
post_srq_recv),
rvt_post_srq_recv);
break;
case CREATE_AH:
check_driver_override(rdi, offsetof(struct ib_device,
create_ah),
rvt_create_ah);
break;
case DESTROY_AH:
check_driver_override(rdi, offsetof(struct ib_device,
destroy_ah),
rvt_destroy_ah);
break;
case MODIFY_AH:
check_driver_override(rdi, offsetof(struct ib_device,
modify_ah),
rvt_modify_ah);
break;
case QUERY_AH:
check_driver_override(rdi, offsetof(struct ib_device,
query_ah),
rvt_query_ah);
break;
case CREATE_SRQ:
check_driver_override(rdi, offsetof(struct ib_device,
create_srq),
rvt_create_srq);
break;
case MODIFY_SRQ:
check_driver_override(rdi, offsetof(struct ib_device,
modify_srq),
rvt_modify_srq);
break;
case DESTROY_SRQ:
check_driver_override(rdi, offsetof(struct ib_device,
destroy_srq),
rvt_destroy_srq);
break;
case QUERY_SRQ:
check_driver_override(rdi, offsetof(struct ib_device,
query_srq),
rvt_query_srq);
break;
case ATTACH_MCAST:
check_driver_override(rdi, offsetof(struct ib_device,
attach_mcast),
rvt_attach_mcast);
break;
case DETACH_MCAST:
check_driver_override(rdi, offsetof(struct ib_device,
detach_mcast),
rvt_detach_mcast);
break;
case GET_DMA_MR:
check_driver_override(rdi, offsetof(struct ib_device,
get_dma_mr),
rvt_get_dma_mr);
break;
case REG_USER_MR:
check_driver_override(rdi, offsetof(struct ib_device,
reg_user_mr),
rvt_reg_user_mr);
break;
case DEREG_MR:
check_driver_override(rdi, offsetof(struct ib_device,
dereg_mr),
rvt_dereg_mr);
break;
case ALLOC_FMR:
check_driver_override(rdi, offsetof(struct ib_device,
alloc_fmr),
rvt_alloc_fmr);
break;
case ALLOC_MR:
check_driver_override(rdi, offsetof(struct ib_device,
alloc_mr),
rvt_alloc_mr);
break;
case MAP_PHYS_FMR:
check_driver_override(rdi, offsetof(struct ib_device,
map_phys_fmr),
rvt_map_phys_fmr);
break;
case UNMAP_FMR:
check_driver_override(rdi, offsetof(struct ib_device,
unmap_fmr),
rvt_unmap_fmr);
break;
case DEALLOC_FMR:
check_driver_override(rdi, offsetof(struct ib_device,
dealloc_fmr),
rvt_dealloc_fmr);
break;
case MMAP:
check_driver_override(rdi, offsetof(struct ib_device,
mmap),
rvt_mmap);
break;
case CREATE_CQ:
check_driver_override(rdi, offsetof(struct ib_device,
create_cq),
rvt_create_cq);
break;
case DESTROY_CQ:
check_driver_override(rdi, offsetof(struct ib_device,
destroy_cq),
rvt_destroy_cq);
break;
case POLL_CQ:
check_driver_override(rdi, offsetof(struct ib_device,
poll_cq),
rvt_poll_cq);
break;
case REQ_NOTFIY_CQ:
check_driver_override(rdi, offsetof(struct ib_device,
req_notify_cq),
rvt_req_notify_cq);
break;
case RESIZE_CQ:
check_driver_override(rdi, offsetof(struct ib_device,
resize_cq),
rvt_resize_cq);
break;
case ALLOC_PD:
check_driver_override(rdi, offsetof(struct ib_device,
alloc_pd),
rvt_alloc_pd);
break;
case DEALLOC_PD:
check_driver_override(rdi, offsetof(struct ib_device,
dealloc_pd),
rvt_dealloc_pd);
break;
default:
return -EINVAL;
}
return 0;
}
/**
* rvt_register_device - register a driver
* @rdi: main dev structure for all of rdmavt operations
*
* It is up to drivers to allocate the rdi and fill in the appropriate
* information.
*
* Return: 0 on success otherwise an errno.
*/
int rvt_register_device(struct rvt_dev_info *rdi)
{
int ret = 0, i;
if (!rdi)
return -EINVAL;
/*
* Check to ensure drivers have setup the required helpers for the verbs
* they want rdmavt to handle
*/
for (i = 0; i < _VERB_IDX_MAX; i++)
if (check_support(rdi, i)) {
pr_err("Driver support req not met at %d\n", i);
return -EINVAL;
}
/* Once we get past here we can use rvt_pr macros and tracepoints */
trace_rvt_dbg(rdi, "Driver attempting registration");
rvt_mmap_init(rdi);
/* Queue Pairs */
ret = rvt_driver_qp_init(rdi);
if (ret) {
pr_err("Error in driver QP init.\n");
return -EINVAL;
}
/* Address Handle */
spin_lock_init(&rdi->n_ahs_lock);
rdi->n_ahs_allocated = 0;
/* Shared Receive Queue */
rvt_driver_srq_init(rdi);
/* Multicast */
rvt_driver_mcast_init(rdi);
/* Mem Region */
ret = rvt_driver_mr_init(rdi);
if (ret) {
pr_err("Error in driver MR init.\n");
goto bail_no_mr;
}
/* Completion queues */
ret = rvt_driver_cq_init(rdi);
if (ret) {
pr_err("Error in driver CQ init.\n");
goto bail_mr;
}
/* DMA Operations */
rdi->ibdev.dma_ops =
rdi->ibdev.dma_ops ? : &rvt_default_dma_mapping_ops;
/* Protection Domain */
spin_lock_init(&rdi->n_pds_lock);
rdi->n_pds_allocated = 0;
/*
* There are some things which could be set by underlying drivers but
* really should be up to rdmavt to set. For instance drivers can't know
* exactly which functions rdmavt supports, nor do they know the ABI
* version, so we do all of this sort of stuff here.
*/
rdi->ibdev.uverbs_abi_ver = RVT_UVERBS_ABI_VERSION;
rdi->ibdev.uverbs_cmd_mask =
(1ull << IB_USER_VERBS_CMD_GET_CONTEXT) |
(1ull << IB_USER_VERBS_CMD_QUERY_DEVICE) |
(1ull << IB_USER_VERBS_CMD_QUERY_PORT) |
(1ull << IB_USER_VERBS_CMD_ALLOC_PD) |
(1ull << IB_USER_VERBS_CMD_DEALLOC_PD) |
(1ull << IB_USER_VERBS_CMD_CREATE_AH) |
(1ull << IB_USER_VERBS_CMD_MODIFY_AH) |
(1ull << IB_USER_VERBS_CMD_QUERY_AH) |
(1ull << IB_USER_VERBS_CMD_DESTROY_AH) |
(1ull << IB_USER_VERBS_CMD_REG_MR) |
(1ull << IB_USER_VERBS_CMD_DEREG_MR) |
(1ull << IB_USER_VERBS_CMD_CREATE_COMP_CHANNEL) |
(1ull << IB_USER_VERBS_CMD_CREATE_CQ) |
(1ull << IB_USER_VERBS_CMD_RESIZE_CQ) |
(1ull << IB_USER_VERBS_CMD_DESTROY_CQ) |
(1ull << IB_USER_VERBS_CMD_POLL_CQ) |
(1ull << IB_USER_VERBS_CMD_REQ_NOTIFY_CQ) |
(1ull << IB_USER_VERBS_CMD_CREATE_QP) |
(1ull << IB_USER_VERBS_CMD_QUERY_QP) |
(1ull << IB_USER_VERBS_CMD_MODIFY_QP) |
(1ull << IB_USER_VERBS_CMD_DESTROY_QP) |
(1ull << IB_USER_VERBS_CMD_POST_SEND) |
(1ull << IB_USER_VERBS_CMD_POST_RECV) |
(1ull << IB_USER_VERBS_CMD_ATTACH_MCAST) |
(1ull << IB_USER_VERBS_CMD_DETACH_MCAST) |
(1ull << IB_USER_VERBS_CMD_CREATE_SRQ) |
(1ull << IB_USER_VERBS_CMD_MODIFY_SRQ) |
(1ull << IB_USER_VERBS_CMD_QUERY_SRQ) |
(1ull << IB_USER_VERBS_CMD_DESTROY_SRQ) |
(1ull << IB_USER_VERBS_CMD_POST_SRQ_RECV);
rdi->ibdev.node_type = RDMA_NODE_IB_CA;
rdi->ibdev.num_comp_vectors = 1;
/* We are now good to announce we exist */
ret = ib_register_device(&rdi->ibdev, rdi->driver_f.port_callback);
if (ret) {
rvt_pr_err(rdi, "Failed to register driver with ib core.\n");
goto bail_cq;
}
rvt_create_mad_agents(rdi);
rvt_pr_info(rdi, "Registration with rdmavt done.\n");
return ret;
bail_cq:
rvt_cq_exit(rdi);
bail_mr:
rvt_mr_exit(rdi);
bail_no_mr:
rvt_qp_exit(rdi);
return ret;
}
EXPORT_SYMBOL(rvt_register_device);
/**
* rvt_unregister_device - remove a driver
* @rdi: rvt dev struct
*/
void rvt_unregister_device(struct rvt_dev_info *rdi)
{
trace_rvt_dbg(rdi, "Driver is unregistering.");
if (!rdi)
return;
rvt_free_mad_agents(rdi);
ib_unregister_device(&rdi->ibdev);
rvt_cq_exit(rdi);
rvt_mr_exit(rdi);
rvt_qp_exit(rdi);
}
EXPORT_SYMBOL(rvt_unregister_device);
/**
* rvt_init_port - init internal data for driver port
* @rdi: rvt dev strut
* @port: rvt port
* @port_index: 0 based index of ports, different from IB core port num
*
* Keep track of a list of ports. No need to have a detach port.
* They persist until the driver goes away.
*
* Return: always 0
*/
int rvt_init_port(struct rvt_dev_info *rdi, struct rvt_ibport *port,
int port_index, u16 *pkey_table)
{
rdi->ports[port_index] = port;
rdi->ports[port_index]->pkey_table = pkey_table;
return 0;
}
EXPORT_SYMBOL(rvt_init_port);

104
drivers/infiniband/sw/rdmavt/vt.h 100644
View File

@ -0,0 +1,104 @@
#ifndef DEF_RDMAVT_H
#define DEF_RDMAVT_H
/*
* Copyright(c) 2016 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* BSD LICENSE
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* - Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <rdma/rdma_vt.h>
#include <linux/pci.h>
#include "dma.h"
#include "pd.h"
#include "qp.h"
#include "ah.h"
#include "mr.h"
#include "srq.h"
#include "mcast.h"
#include "mmap.h"
#include "cq.h"
#include "mad.h"
#include "mmap.h"
#define rvt_pr_info(rdi, fmt, ...) \
__rvt_pr_info(rdi->driver_f.get_pci_dev(rdi), \
rdi->driver_f.get_card_name(rdi), \
fmt, \
##__VA_ARGS__)
#define rvt_pr_warn(rdi, fmt, ...) \
__rvt_pr_warn(rdi->driver_f.get_pci_dev(rdi), \
rdi->driver_f.get_card_name(rdi), \
fmt, \
##__VA_ARGS__)
#define rvt_pr_err(rdi, fmt, ...) \
__rvt_pr_err(rdi->driver_f.get_pci_dev(rdi), \
rdi->driver_f.get_card_name(rdi), \
fmt, \
##__VA_ARGS__)
#define __rvt_pr_info(pdev, name, fmt, ...) \
dev_info(&pdev->dev, "%s: " fmt, name, ##__VA_ARGS__)
#define __rvt_pr_warn(pdev, name, fmt, ...) \
dev_warn(&pdev->dev, "%s: " fmt, name, ##__VA_ARGS__)
#define __rvt_pr_err(pdev, name, fmt, ...) \
dev_err(&pdev->dev, "%s: " fmt, name, ##__VA_ARGS__)
static inline int ibport_num_to_idx(struct ib_device *ibdev, u8 port_num)
{
struct rvt_dev_info *rdi = ib_to_rvt(ibdev);
int port_index;
port_index = port_num - 1; /* IB ports start at 1 our arrays at 0 */
if ((port_index < 0) || (port_index >= rdi->dparms.nports))
return -EINVAL;
return port_index;
}
#endif /* DEF_RDMAVT_H */

2
drivers/infiniband/ulp/ipoib/ipoib.h
View File

@ -388,7 +388,7 @@ struct ipoib_dev_priv {
struct dentry *mcg_dentry;
struct dentry *path_dentry;
#endif
int hca_caps;
u64 hca_caps;
struct ipoib_ethtool_st ethtool;
struct timer_list poll_timer;
unsigned max_send_sge;

27
drivers/infiniband/ulp/ipoib/ipoib_ib.c
View File

@ -180,6 +180,7 @@ static void ipoib_ib_handle_rx_wc(struct net_device *dev, struct ib_wc *wc)
struct sk_buff *skb;
u64 mapping[IPOIB_UD_RX_SG];
union ib_gid *dgid;
union ib_gid *sgid;
ipoib_dbg_data(priv, "recv completion: id %d, status: %d\n",
wr_id, wc->status);
@ -203,13 +204,6 @@ static void ipoib_ib_handle_rx_wc(struct net_device *dev, struct ib_wc *wc)
return;
}
/*
* Drop packets that this interface sent, ie multicast packets
* that the HCA has replicated.
*/
if (wc->slid == priv->local_lid && wc->src_qp == priv->qp->qp_num)
goto repost;
memcpy(mapping, priv->rx_ring[wr_id].mapping,
IPOIB_UD_RX_SG * sizeof *mapping);
@ -239,6 +233,25 @@ static void ipoib_ib_handle_rx_wc(struct net_device *dev, struct ib_wc *wc)
else
skb->pkt_type = PACKET_MULTICAST;
sgid = &((struct ib_grh *)skb->data)->sgid;
/*
* Drop packets that this interface sent, ie multicast packets
* that the HCA has replicated.
*/
if (wc->slid == priv->local_lid && wc->src_qp == priv->qp->qp_num) {
int need_repost = 1;
if ((wc->wc_flags & IB_WC_GRH) &&
sgid->global.interface_id != priv->local_gid.global.interface_id)
need_repost = 0;
if (need_repost) {
dev_kfree_skb_any(skb);
goto repost;
}
}
skb_pull(skb, IB_GRH_BYTES);
skb->protocol = ((struct ipoib_header *) skb->data)->proto;

65
drivers/infiniband/ulp/ipoib/ipoib_main.c
View File

@ -51,6 +51,7 @@
#include <net/addrconf.h>
#include <linux/inetdevice.h>
#include <rdma/ib_cache.h>
#include <linux/pci.h>
#define DRV_VERSION "1.0.0"
@ -1590,11 +1591,67 @@ void ipoib_dev_cleanup(struct net_device *dev)
priv->tx_ring = NULL;
}
static int ipoib_set_vf_link_state(struct net_device *dev, int vf, int link_state)
{
struct ipoib_dev_priv *priv = netdev_priv(dev);
return ib_set_vf_link_state(priv->ca, vf, priv->port, link_state);
}
static int ipoib_get_vf_config(struct net_device *dev, int vf,
struct ifla_vf_info *ivf)
{
struct ipoib_dev_priv *priv = netdev_priv(dev);
int err;
err = ib_get_vf_config(priv->ca, vf, priv->port, ivf);
if (err)
return err;
ivf->vf = vf;
return 0;
}
static int ipoib_set_vf_guid(struct net_device *dev, int vf, u64 guid, int type)
{
struct ipoib_dev_priv *priv = netdev_priv(dev);
if (type != IFLA_VF_IB_NODE_GUID && type != IFLA_VF_IB_PORT_GUID)
return -EINVAL;
return ib_set_vf_guid(priv->ca, vf, priv->port, guid, type);
}
static int ipoib_get_vf_stats(struct net_device *dev, int vf,
struct ifla_vf_stats *vf_stats)
{
struct ipoib_dev_priv *priv = netdev_priv(dev);
return ib_get_vf_stats(priv->ca, vf, priv->port, vf_stats);
}
static const struct header_ops ipoib_header_ops = {
.create = ipoib_hard_header,
};
static const struct net_device_ops ipoib_netdev_ops = {
static const struct net_device_ops ipoib_netdev_ops_pf = {
.ndo_uninit = ipoib_uninit,
.ndo_open = ipoib_open,
.ndo_stop = ipoib_stop,
.ndo_change_mtu = ipoib_change_mtu,
.ndo_fix_features = ipoib_fix_features,
.ndo_start_xmit = ipoib_start_xmit,
.ndo_tx_timeout = ipoib_timeout,
.ndo_set_rx_mode = ipoib_set_mcast_list,
.ndo_get_iflink = ipoib_get_iflink,
.ndo_set_vf_link_state = ipoib_set_vf_link_state,
.ndo_get_vf_config = ipoib_get_vf_config,
.ndo_get_vf_stats = ipoib_get_vf_stats,
.ndo_set_vf_guid = ipoib_set_vf_guid,
};
static const struct net_device_ops ipoib_netdev_ops_vf = {
.ndo_uninit = ipoib_uninit,
.ndo_open = ipoib_open,
.ndo_stop = ipoib_stop,
@ -1610,7 +1667,11 @@ void ipoib_setup(struct net_device *dev)
{
struct ipoib_dev_priv *priv = netdev_priv(dev);
dev->netdev_ops = &ipoib_netdev_ops;
if (priv->hca_caps & IB_DEVICE_VIRTUAL_FUNCTION)
dev->netdev_ops = &ipoib_netdev_ops_vf;
else
dev->netdev_ops = &ipoib_netdev_ops_pf;
dev->header_ops = &ipoib_header_ops;
ipoib_set_ethtool_ops(dev);

2
drivers/infiniband/ulp/srpt/ib_srpt.c
View File

@ -839,7 +839,7 @@ static void srpt_zerolength_write_done(struct ib_cq *cq, struct ib_wc *wc)
if (srpt_set_ch_state(ch, CH_DISCONNECTED))
schedule_work(&ch->release_work);
else
WARN_ONCE("%s-%d\n", ch->sess_name, ch->qp->qp_num);
WARN_ONCE(1, "%s-%d\n", ch->sess_name, ch->qp->qp_num);
}
}

Some files were not shown because too many files have changed in this diff Show More