2008-10-22 16:47:49 -06:00
|
|
|
/*
|
|
|
|
* Copyright (c) 2007 Mellanox Technologies. All rights reserved.
|
|
|
|
*
|
|
|
|
* This software is available to you under a choice of one of two
|
|
|
|
* licenses. You may choose to be licensed under the terms of the GNU
|
|
|
|
* General Public License (GPL) Version 2, available from the file
|
|
|
|
* COPYING in the main directory of this source tree, or the
|
|
|
|
* OpenIB.org BSD license below:
|
|
|
|
*
|
|
|
|
* Redistribution and use in source and binary forms, with or
|
|
|
|
* without modification, are permitted provided that the following
|
|
|
|
* conditions are met:
|
|
|
|
*
|
|
|
|
* - Redistributions of source code must retain the above
|
|
|
|
* copyright notice, this list of conditions and the following
|
|
|
|
* disclaimer.
|
|
|
|
*
|
|
|
|
* - Redistributions in binary form must reproduce the above
|
|
|
|
* copyright notice, this list of conditions and the following
|
|
|
|
* disclaimer in the documentation and/or other materials
|
|
|
|
* provided with the distribution.
|
|
|
|
*
|
|
|
|
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
|
|
|
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
|
|
|
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
|
|
|
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
|
|
|
|
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
|
|
|
|
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
|
|
|
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
|
|
* SOFTWARE.
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <asm/page.h>
|
|
|
|
#include <linux/mlx4/cq.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 02:04:11 -06:00
|
|
|
#include <linux/slab.h>
|
2008-10-22 16:47:49 -06:00
|
|
|
#include <linux/mlx4/qp.h>
|
|
|
|
#include <linux/skbuff.h>
|
|
|
|
#include <linux/if_vlan.h>
|
2014-10-05 03:35:14 -06:00
|
|
|
#include <linux/prefetch.h>
|
2008-10-22 16:47:49 -06:00
|
|
|
#include <linux/vmalloc.h>
|
2010-08-23 21:46:46 -06:00
|
|
|
#include <linux/tcp.h>
|
2013-12-23 07:09:44 -07:00
|
|
|
#include <linux/ip.h>
|
2016-05-02 10:38:37 -06:00
|
|
|
#include <linux/ipv6.h>
|
2011-09-15 17:46:05 -06:00
|
|
|
#include <linux/moduleparam.h>
|
2020-04-15 10:46:52 -06:00
|
|
|
#include <linux/indirect_call_wrapper.h>
|
2008-10-22 16:47:49 -06:00
|
|
|
|
|
|
|
#include "mlx4_en.h"
|
|
|
|
|
|
|
|
int mlx4_en_create_tx_ring(struct mlx4_en_priv *priv,
|
net/mlx4: Change QP allocation scheme
When using BF (Blue-Flame), the QPN overrides the VLAN, CV, and SV fields
in the WQE. Thus, BF may only be used for QPNs with bits 6,7 unset.
The current Ethernet driver code reserves a Tx QP range with 256b alignment.
This is wrong because if there are more than 64 Tx QPs in use,
QPNs >= base + 65 will have bits 6/7 set.
This problem is not specific for the Ethernet driver, any entity that
tries to reserve more than 64 BF-enabled QPs should fail. Also, using
ranges is not necessary here and is wasteful.
The new mechanism introduced here will support reservation for
"Eth QPs eligible for BF" for all drivers: bare-metal, multi-PF, and VFs
(when hypervisors support WC in VMs). The flow we use is:
1. In mlx4_en, allocate Tx QPs one by one instead of a range allocation,
and request "BF enabled QPs" if BF is supported for the function
2. In the ALLOC_RES FW command, change param1 to:
a. param1[23:0] - number of QPs
b. param1[31-24] - flags controlling QPs reservation
Bit 31 refers to Eth blueflame supported QPs. Those QPs must have
bits 6 and 7 unset in order to be used in Ethernet.
Bits 24-30 of the flags are currently reserved.
When a function tries to allocate a QP, it states the required attributes
for this QP. Those attributes are considered "best-effort". If an attribute,
such as Ethernet BF enabled QP, is a must-have attribute, the function has
to check that attribute is supported before trying to do the allocation.
In a lower layer of the code, mlx4_qp_reserve_range masks out the bits
which are unsupported. If SRIOV is used, the PF validates those attributes
and masks out unsupported attributes as well. In order to notify VFs which
attributes are supported, the VF uses QUERY_FUNC_CAP command. This command's
mailbox is filled by the PF, which notifies which QP allocation attributes
it supports.
Signed-off-by: Eugenia Emantayev <eugenia@mellanox.co.il>
Signed-off-by: Matan Barak <matanb@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-12-11 01:57:54 -07:00
|
|
|
struct mlx4_en_tx_ring **pring, u32 size,
|
2013-12-19 12:20:14 -07:00
|
|
|
u16 stride, int node, int queue_index)
|
2008-10-22 16:47:49 -06:00
|
|
|
{
|
|
|
|
struct mlx4_en_dev *mdev = priv->mdev;
|
2013-11-07 03:19:52 -07:00
|
|
|
struct mlx4_en_tx_ring *ring;
|
2008-10-22 16:47:49 -06:00
|
|
|
int tmp;
|
|
|
|
int err;
|
|
|
|
|
2013-11-07 03:19:54 -07:00
|
|
|
ring = kzalloc_node(sizeof(*ring), GFP_KERNEL, node);
|
2013-11-07 03:19:52 -07:00
|
|
|
if (!ring) {
|
2018-12-13 04:03:37 -07:00
|
|
|
en_err(priv, "Failed allocating TX ring\n");
|
|
|
|
return -ENOMEM;
|
2013-11-07 03:19:52 -07:00
|
|
|
}
|
|
|
|
|
2008-10-22 16:47:49 -06:00
|
|
|
ring->size = size;
|
|
|
|
ring->size_mask = size - 1;
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
ring->sp_stride = stride;
|
2015-06-25 02:29:42 -06:00
|
|
|
ring->full_size = ring->size - HEADROOM - MAX_DESC_TXBBS;
|
2008-10-22 16:47:49 -06:00
|
|
|
|
|
|
|
tmp = size * sizeof(struct mlx4_en_tx_info);
|
2017-05-08 16:57:27 -06:00
|
|
|
ring->tx_info = kvmalloc_node(tmp, GFP_KERNEL, node);
|
2013-11-07 03:19:52 -07:00
|
|
|
if (!ring->tx_info) {
|
2017-05-08 16:57:27 -06:00
|
|
|
err = -ENOMEM;
|
|
|
|
goto err_ring;
|
2013-11-07 03:19:52 -07:00
|
|
|
}
|
2012-01-29 05:56:23 -07:00
|
|
|
|
2009-06-01 14:27:13 -06:00
|
|
|
en_dbg(DRV, priv, "Allocated tx_info ring at addr:%p size:%d\n",
|
2008-10-22 16:47:49 -06:00
|
|
|
ring->tx_info, tmp);
|
|
|
|
|
2013-11-07 03:19:54 -07:00
|
|
|
ring->bounce_buf = kmalloc_node(MAX_DESC_SIZE, GFP_KERNEL, node);
|
2008-10-22 16:47:49 -06:00
|
|
|
if (!ring->bounce_buf) {
|
2013-11-07 03:19:54 -07:00
|
|
|
ring->bounce_buf = kmalloc(MAX_DESC_SIZE, GFP_KERNEL);
|
|
|
|
if (!ring->bounce_buf) {
|
|
|
|
err = -ENOMEM;
|
|
|
|
goto err_info;
|
|
|
|
}
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
ring->buf_size = ALIGN(size * ring->sp_stride, MLX4_EN_PAGE_SIZE);
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2013-11-07 03:19:54 -07:00
|
|
|
/* Allocate HW buffers on provided NUMA node */
|
2015-01-25 07:59:35 -07:00
|
|
|
set_dev_node(&mdev->dev->persist->pdev->dev, node);
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
err = mlx4_alloc_hwq_res(mdev->dev, &ring->sp_wqres, ring->buf_size);
|
2015-01-25 07:59:35 -07:00
|
|
|
set_dev_node(&mdev->dev->persist->pdev->dev, mdev->dev->numa_node);
|
2008-10-22 16:47:49 -06:00
|
|
|
if (err) {
|
2009-06-01 14:27:13 -06:00
|
|
|
en_err(priv, "Failed allocating hwq resources\n");
|
2008-10-22 16:47:49 -06:00
|
|
|
goto err_bounce;
|
|
|
|
}
|
|
|
|
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
ring->buf = ring->sp_wqres.buf.direct.buf;
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2014-05-07 13:52:57 -06:00
|
|
|
en_dbg(DRV, priv, "Allocated TX ring (addr:%p) - buf:%p size:%d buf_size:%d dma:%llx\n",
|
|
|
|
ring, ring->buf, ring->size, ring->buf_size,
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
(unsigned long long) ring->sp_wqres.buf.direct.map);
|
2008-10-22 16:47:49 -06:00
|
|
|
|
net/mlx4: Change QP allocation scheme
When using BF (Blue-Flame), the QPN overrides the VLAN, CV, and SV fields
in the WQE. Thus, BF may only be used for QPNs with bits 6,7 unset.
The current Ethernet driver code reserves a Tx QP range with 256b alignment.
This is wrong because if there are more than 64 Tx QPs in use,
QPNs >= base + 65 will have bits 6/7 set.
This problem is not specific for the Ethernet driver, any entity that
tries to reserve more than 64 BF-enabled QPs should fail. Also, using
ranges is not necessary here and is wasteful.
The new mechanism introduced here will support reservation for
"Eth QPs eligible for BF" for all drivers: bare-metal, multi-PF, and VFs
(when hypervisors support WC in VMs). The flow we use is:
1. In mlx4_en, allocate Tx QPs one by one instead of a range allocation,
and request "BF enabled QPs" if BF is supported for the function
2. In the ALLOC_RES FW command, change param1 to:
a. param1[23:0] - number of QPs
b. param1[31-24] - flags controlling QPs reservation
Bit 31 refers to Eth blueflame supported QPs. Those QPs must have
bits 6 and 7 unset in order to be used in Ethernet.
Bits 24-30 of the flags are currently reserved.
When a function tries to allocate a QP, it states the required attributes
for this QP. Those attributes are considered "best-effort". If an attribute,
such as Ethernet BF enabled QP, is a must-have attribute, the function has
to check that attribute is supported before trying to do the allocation.
In a lower layer of the code, mlx4_qp_reserve_range masks out the bits
which are unsupported. If SRIOV is used, the PF validates those attributes
and masks out unsupported attributes as well. In order to notify VFs which
attributes are supported, the VF uses QUERY_FUNC_CAP command. This command's
mailbox is filled by the PF, which notifies which QP allocation attributes
it supports.
Signed-off-by: Eugenia Emantayev <eugenia@mellanox.co.il>
Signed-off-by: Matan Barak <matanb@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-12-11 01:57:54 -07:00
|
|
|
err = mlx4_qp_reserve_range(mdev->dev, 1, 1, &ring->qpn,
|
2017-06-21 00:29:36 -06:00
|
|
|
MLX4_RESERVE_ETH_BF_QP,
|
|
|
|
MLX4_RES_USAGE_DRIVER);
|
net/mlx4: Change QP allocation scheme
When using BF (Blue-Flame), the QPN overrides the VLAN, CV, and SV fields
in the WQE. Thus, BF may only be used for QPNs with bits 6,7 unset.
The current Ethernet driver code reserves a Tx QP range with 256b alignment.
This is wrong because if there are more than 64 Tx QPs in use,
QPNs >= base + 65 will have bits 6/7 set.
This problem is not specific for the Ethernet driver, any entity that
tries to reserve more than 64 BF-enabled QPs should fail. Also, using
ranges is not necessary here and is wasteful.
The new mechanism introduced here will support reservation for
"Eth QPs eligible for BF" for all drivers: bare-metal, multi-PF, and VFs
(when hypervisors support WC in VMs). The flow we use is:
1. In mlx4_en, allocate Tx QPs one by one instead of a range allocation,
and request "BF enabled QPs" if BF is supported for the function
2. In the ALLOC_RES FW command, change param1 to:
a. param1[23:0] - number of QPs
b. param1[31-24] - flags controlling QPs reservation
Bit 31 refers to Eth blueflame supported QPs. Those QPs must have
bits 6 and 7 unset in order to be used in Ethernet.
Bits 24-30 of the flags are currently reserved.
When a function tries to allocate a QP, it states the required attributes
for this QP. Those attributes are considered "best-effort". If an attribute,
such as Ethernet BF enabled QP, is a must-have attribute, the function has
to check that attribute is supported before trying to do the allocation.
In a lower layer of the code, mlx4_qp_reserve_range masks out the bits
which are unsupported. If SRIOV is used, the PF validates those attributes
and masks out unsupported attributes as well. In order to notify VFs which
attributes are supported, the VF uses QUERY_FUNC_CAP command. This command's
mailbox is filled by the PF, which notifies which QP allocation attributes
it supports.
Signed-off-by: Eugenia Emantayev <eugenia@mellanox.co.il>
Signed-off-by: Matan Barak <matanb@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-12-11 01:57:54 -07:00
|
|
|
if (err) {
|
|
|
|
en_err(priv, "failed reserving qp for TX ring\n");
|
2016-05-04 05:50:15 -06:00
|
|
|
goto err_hwq_res;
|
net/mlx4: Change QP allocation scheme
When using BF (Blue-Flame), the QPN overrides the VLAN, CV, and SV fields
in the WQE. Thus, BF may only be used for QPNs with bits 6,7 unset.
The current Ethernet driver code reserves a Tx QP range with 256b alignment.
This is wrong because if there are more than 64 Tx QPs in use,
QPNs >= base + 65 will have bits 6/7 set.
This problem is not specific for the Ethernet driver, any entity that
tries to reserve more than 64 BF-enabled QPs should fail. Also, using
ranges is not necessary here and is wasteful.
The new mechanism introduced here will support reservation for
"Eth QPs eligible for BF" for all drivers: bare-metal, multi-PF, and VFs
(when hypervisors support WC in VMs). The flow we use is:
1. In mlx4_en, allocate Tx QPs one by one instead of a range allocation,
and request "BF enabled QPs" if BF is supported for the function
2. In the ALLOC_RES FW command, change param1 to:
a. param1[23:0] - number of QPs
b. param1[31-24] - flags controlling QPs reservation
Bit 31 refers to Eth blueflame supported QPs. Those QPs must have
bits 6 and 7 unset in order to be used in Ethernet.
Bits 24-30 of the flags are currently reserved.
When a function tries to allocate a QP, it states the required attributes
for this QP. Those attributes are considered "best-effort". If an attribute,
such as Ethernet BF enabled QP, is a must-have attribute, the function has
to check that attribute is supported before trying to do the allocation.
In a lower layer of the code, mlx4_qp_reserve_range masks out the bits
which are unsupported. If SRIOV is used, the PF validates those attributes
and masks out unsupported attributes as well. In order to notify VFs which
attributes are supported, the VF uses QUERY_FUNC_CAP command. This command's
mailbox is filled by the PF, which notifies which QP allocation attributes
it supports.
Signed-off-by: Eugenia Emantayev <eugenia@mellanox.co.il>
Signed-off-by: Matan Barak <matanb@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-12-11 01:57:54 -07:00
|
|
|
}
|
|
|
|
|
2017-05-23 05:38:15 -06:00
|
|
|
err = mlx4_qp_alloc(mdev->dev, ring->qpn, &ring->sp_qp);
|
2008-10-22 16:47:49 -06:00
|
|
|
if (err) {
|
2009-06-01 14:27:13 -06:00
|
|
|
en_err(priv, "Failed allocating qp %d\n", ring->qpn);
|
net/mlx4: Change QP allocation scheme
When using BF (Blue-Flame), the QPN overrides the VLAN, CV, and SV fields
in the WQE. Thus, BF may only be used for QPNs with bits 6,7 unset.
The current Ethernet driver code reserves a Tx QP range with 256b alignment.
This is wrong because if there are more than 64 Tx QPs in use,
QPNs >= base + 65 will have bits 6/7 set.
This problem is not specific for the Ethernet driver, any entity that
tries to reserve more than 64 BF-enabled QPs should fail. Also, using
ranges is not necessary here and is wasteful.
The new mechanism introduced here will support reservation for
"Eth QPs eligible for BF" for all drivers: bare-metal, multi-PF, and VFs
(when hypervisors support WC in VMs). The flow we use is:
1. In mlx4_en, allocate Tx QPs one by one instead of a range allocation,
and request "BF enabled QPs" if BF is supported for the function
2. In the ALLOC_RES FW command, change param1 to:
a. param1[23:0] - number of QPs
b. param1[31-24] - flags controlling QPs reservation
Bit 31 refers to Eth blueflame supported QPs. Those QPs must have
bits 6 and 7 unset in order to be used in Ethernet.
Bits 24-30 of the flags are currently reserved.
When a function tries to allocate a QP, it states the required attributes
for this QP. Those attributes are considered "best-effort". If an attribute,
such as Ethernet BF enabled QP, is a must-have attribute, the function has
to check that attribute is supported before trying to do the allocation.
In a lower layer of the code, mlx4_qp_reserve_range masks out the bits
which are unsupported. If SRIOV is used, the PF validates those attributes
and masks out unsupported attributes as well. In order to notify VFs which
attributes are supported, the VF uses QUERY_FUNC_CAP command. This command's
mailbox is filled by the PF, which notifies which QP allocation attributes
it supports.
Signed-off-by: Eugenia Emantayev <eugenia@mellanox.co.il>
Signed-off-by: Matan Barak <matanb@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-12-11 01:57:54 -07:00
|
|
|
goto err_reserve;
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
ring->sp_qp.event = mlx4_en_sqp_event;
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2013-11-07 03:19:54 -07:00
|
|
|
err = mlx4_bf_alloc(mdev->dev, &ring->bf, node);
|
2011-03-22 16:38:52 -06:00
|
|
|
if (err) {
|
2014-05-07 13:52:57 -06:00
|
|
|
en_dbg(DRV, priv, "working without blueflame (%d)\n", err);
|
2011-03-22 16:38:52 -06:00
|
|
|
ring->bf.uar = &mdev->priv_uar;
|
|
|
|
ring->bf.uar->map = mdev->uar_map;
|
|
|
|
ring->bf_enabled = false;
|
2014-07-22 06:44:10 -06:00
|
|
|
ring->bf_alloced = false;
|
|
|
|
priv->pflags &= ~MLX4_EN_PRIV_FLAGS_BLUEFLAME;
|
|
|
|
} else {
|
|
|
|
ring->bf_alloced = true;
|
|
|
|
ring->bf_enabled = !!(priv->pflags &
|
|
|
|
MLX4_EN_PRIV_FLAGS_BLUEFLAME);
|
|
|
|
}
|
2011-03-22 16:38:52 -06:00
|
|
|
|
2013-04-23 00:06:49 -06:00
|
|
|
ring->hwtstamp_tx_type = priv->hwtstamp_config.tx_type;
|
2013-12-19 12:20:14 -07:00
|
|
|
ring->queue_index = queue_index;
|
|
|
|
|
2015-04-28 15:49:29 -06:00
|
|
|
if (queue_index < priv->num_tx_rings_p_up)
|
cpumask_set_cpu_local_first => cpumask_local_spread, lament
da91309e0a7e (cpumask: Utility function to set n'th cpu...) created a
genuinely weird function. I never saw it before, it went through DaveM.
(He only does this to make us other maintainers feel better about our own
mistakes.)
cpumask_set_cpu_local_first's purpose is say "I need to spread things
across N online cpus, choose the ones on this numa node first"; you call
it in a loop.
It can fail. One of the two callers ignores this, the other aborts and
fails the device open.
It can fail in two ways: allocating the off-stack cpumask, or through a
convoluted codepath which AFAICT can only occur if cpu_online_mask
changes. Which shouldn't happen, because if cpu_online_mask can change
while you call this, it could return a now-offline cpu anyway.
It contains a nonsensical test "!cpumask_of_node(numa_node)". This was
drawn to my attention by Geert, who said this causes a warning on Sparc.
It sets a single bit in a cpumask instead of returning a cpu number,
because that's what the callers want.
It could be made more efficient by passing the previous cpu rather than
an index, but that would be more invasive to the callers.
Fixes: da91309e0a7e8966d916a74cce42ed170fde06bf
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> (then rebased)
Tested-by: Amir Vadai <amirv@mellanox.com>
Acked-by: Amir Vadai <amirv@mellanox.com>
Acked-by: David S. Miller <davem@davemloft.net>
2015-05-08 11:44:13 -06:00
|
|
|
cpumask_set_cpu(cpumask_local_spread(queue_index,
|
|
|
|
priv->mdev->dev->numa_node),
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
&ring->sp_affinity_mask);
|
2013-04-23 00:06:49 -06:00
|
|
|
|
2013-11-07 03:19:52 -07:00
|
|
|
*pring = ring;
|
2008-10-22 16:47:49 -06:00
|
|
|
return 0;
|
|
|
|
|
net/mlx4: Change QP allocation scheme
When using BF (Blue-Flame), the QPN overrides the VLAN, CV, and SV fields
in the WQE. Thus, BF may only be used for QPNs with bits 6,7 unset.
The current Ethernet driver code reserves a Tx QP range with 256b alignment.
This is wrong because if there are more than 64 Tx QPs in use,
QPNs >= base + 65 will have bits 6/7 set.
This problem is not specific for the Ethernet driver, any entity that
tries to reserve more than 64 BF-enabled QPs should fail. Also, using
ranges is not necessary here and is wasteful.
The new mechanism introduced here will support reservation for
"Eth QPs eligible for BF" for all drivers: bare-metal, multi-PF, and VFs
(when hypervisors support WC in VMs). The flow we use is:
1. In mlx4_en, allocate Tx QPs one by one instead of a range allocation,
and request "BF enabled QPs" if BF is supported for the function
2. In the ALLOC_RES FW command, change param1 to:
a. param1[23:0] - number of QPs
b. param1[31-24] - flags controlling QPs reservation
Bit 31 refers to Eth blueflame supported QPs. Those QPs must have
bits 6 and 7 unset in order to be used in Ethernet.
Bits 24-30 of the flags are currently reserved.
When a function tries to allocate a QP, it states the required attributes
for this QP. Those attributes are considered "best-effort". If an attribute,
such as Ethernet BF enabled QP, is a must-have attribute, the function has
to check that attribute is supported before trying to do the allocation.
In a lower layer of the code, mlx4_qp_reserve_range masks out the bits
which are unsupported. If SRIOV is used, the PF validates those attributes
and masks out unsupported attributes as well. In order to notify VFs which
attributes are supported, the VF uses QUERY_FUNC_CAP command. This command's
mailbox is filled by the PF, which notifies which QP allocation attributes
it supports.
Signed-off-by: Eugenia Emantayev <eugenia@mellanox.co.il>
Signed-off-by: Matan Barak <matanb@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-12-11 01:57:54 -07:00
|
|
|
err_reserve:
|
|
|
|
mlx4_qp_release_range(mdev->dev, ring->qpn, 1);
|
2008-10-22 16:47:49 -06:00
|
|
|
err_hwq_res:
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
mlx4_free_hwq_res(mdev->dev, &ring->sp_wqres, ring->buf_size);
|
2008-10-22 16:47:49 -06:00
|
|
|
err_bounce:
|
|
|
|
kfree(ring->bounce_buf);
|
|
|
|
ring->bounce_buf = NULL;
|
2013-11-07 03:19:52 -07:00
|
|
|
err_info:
|
2014-10-05 03:35:12 -06:00
|
|
|
kvfree(ring->tx_info);
|
2008-10-22 16:47:49 -06:00
|
|
|
ring->tx_info = NULL;
|
2013-11-07 03:19:52 -07:00
|
|
|
err_ring:
|
|
|
|
kfree(ring);
|
|
|
|
*pring = NULL;
|
2008-10-22 16:47:49 -06:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
void mlx4_en_destroy_tx_ring(struct mlx4_en_priv *priv,
|
2013-11-07 03:19:52 -07:00
|
|
|
struct mlx4_en_tx_ring **pring)
|
2008-10-22 16:47:49 -06:00
|
|
|
{
|
|
|
|
struct mlx4_en_dev *mdev = priv->mdev;
|
2013-11-07 03:19:52 -07:00
|
|
|
struct mlx4_en_tx_ring *ring = *pring;
|
2009-06-01 14:27:13 -06:00
|
|
|
en_dbg(DRV, priv, "Destroying tx ring, qpn: %d\n", ring->qpn);
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2014-07-22 06:44:10 -06:00
|
|
|
if (ring->bf_alloced)
|
2011-03-22 16:38:52 -06:00
|
|
|
mlx4_bf_free(mdev->dev, &ring->bf);
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
mlx4_qp_remove(mdev->dev, &ring->sp_qp);
|
|
|
|
mlx4_qp_free(mdev->dev, &ring->sp_qp);
|
2015-06-25 02:29:41 -06:00
|
|
|
mlx4_qp_release_range(priv->mdev->dev, ring->qpn, 1);
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
mlx4_free_hwq_res(mdev->dev, &ring->sp_wqres, ring->buf_size);
|
2008-10-22 16:47:49 -06:00
|
|
|
kfree(ring->bounce_buf);
|
|
|
|
ring->bounce_buf = NULL;
|
2014-10-05 03:35:12 -06:00
|
|
|
kvfree(ring->tx_info);
|
2008-10-22 16:47:49 -06:00
|
|
|
ring->tx_info = NULL;
|
2013-11-07 03:19:52 -07:00
|
|
|
kfree(ring);
|
|
|
|
*pring = NULL;
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
int mlx4_en_activate_tx_ring(struct mlx4_en_priv *priv,
|
|
|
|
struct mlx4_en_tx_ring *ring,
|
2012-04-04 15:33:24 -06:00
|
|
|
int cq, int user_prio)
|
2008-10-22 16:47:49 -06:00
|
|
|
{
|
|
|
|
struct mlx4_en_dev *mdev = priv->mdev;
|
|
|
|
int err;
|
|
|
|
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
ring->sp_cqn = cq;
|
2008-10-22 16:47:49 -06:00
|
|
|
ring->prod = 0;
|
|
|
|
ring->cons = 0xffffffff;
|
|
|
|
ring->last_nr_txbb = 1;
|
|
|
|
memset(ring->tx_info, 0, ring->size * sizeof(struct mlx4_en_tx_info));
|
|
|
|
memset(ring->buf, 0, ring->buf_size);
|
2016-07-19 13:16:55 -06:00
|
|
|
ring->free_tx_desc = mlx4_en_free_tx_desc;
|
2008-10-22 16:47:49 -06:00
|
|
|
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
ring->sp_qp_state = MLX4_QP_STATE_RST;
|
|
|
|
ring->doorbell_qpn = cpu_to_be32(ring->sp_qp.qpn << 8);
|
2014-10-05 03:35:11 -06:00
|
|
|
ring->mr_key = cpu_to_be32(mdev->mr.key);
|
2008-10-22 16:47:49 -06:00
|
|
|
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
mlx4_en_fill_qp_context(priv, ring->size, ring->sp_stride, 1, 0, ring->qpn,
|
|
|
|
ring->sp_cqn, user_prio, &ring->sp_context);
|
2014-07-22 06:44:10 -06:00
|
|
|
if (ring->bf_alloced)
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
ring->sp_context.usr_page =
|
net/mlx4_core: Set UAR page size to 4KB regardless of system page size
problem description:
The current code sets UAR page size equal to system page size.
The ConnectX-3 and ConnectX-3 Pro HWs require minimum 128 UAR pages.
The mlx4 kernel drivers are not loaded if there is less than 128 UAR pages.
solution:
Always set UAR page to 4KB. This allows more UAR pages if the OS
has PAGE_SIZE larger than 4KB. For example, PowerPC kernel use 64KB
system page size, with 4MB uar region, there are 4MB/2/64KB = 32
uars (half for uar, half for blueflame). This does not meet minimum 128
UAR pages requirement. With 4KB UAR page, there are 4MB/2/4KB = 512 uars
which meet the minimum requirement.
Note that only codes in mlx4_core that deal with firmware know that uar
page size is 4KB. Codes that deal with usr page in cq and qp context
(mlx4_ib, mlx4_en and part of mlx4_core) still have the same assumption
that uar page size equals to system page size.
Note that with this implementation, on 64KB system page size kernel, there
are 16 uars per system page but only one uars is used. The other 15
uars are ignored because of the above assumption.
Regarding SR-IOV, mlx4_core in hypervisor will set the uar page size
to 4KB and mlx4_core code in virtual OS will obtain the uar page size from
firmware.
Regarding backward compatibility in SR-IOV, if hypervisor has this new code,
the virtual OS must be updated. If hypervisor has old code, and the virtual
OS has this new code, the new code will be backward compatible with the
old code. If the uar size is big enough, this new code in VF continues to
work with 64 KB uar page size (on PowerPc kernel). If the uar size does not
meet 128 uars requirement, this new code not loaded in VF and print the same
error message as the old code in Hypervisor.
Signed-off-by: Huy Nguyen <huyn@mellanox.com>
Reviewed-by: Yishai Hadas <yishaih@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-02-17 08:24:26 -07:00
|
|
|
cpu_to_be32(mlx4_to_hw_uar_index(mdev->dev,
|
|
|
|
ring->bf.uar->index));
|
2008-10-22 16:47:49 -06:00
|
|
|
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
err = mlx4_qp_to_ready(mdev->dev, &ring->sp_wqres.mtt, &ring->sp_context,
|
|
|
|
&ring->sp_qp, &ring->sp_qp_state);
|
|
|
|
if (!cpumask_empty(&ring->sp_affinity_mask))
|
|
|
|
netif_set_xps_queue(priv->dev, &ring->sp_affinity_mask,
|
2013-12-19 12:20:14 -07:00
|
|
|
ring->queue_index);
|
2008-10-22 16:47:49 -06:00
|
|
|
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
void mlx4_en_deactivate_tx_ring(struct mlx4_en_priv *priv,
|
|
|
|
struct mlx4_en_tx_ring *ring)
|
|
|
|
{
|
|
|
|
struct mlx4_en_dev *mdev = priv->mdev;
|
|
|
|
|
mlx4: reorganize struct mlx4_en_tx_ring
Goal is to reorganize this critical structure to increase performance.
ndo_start_xmit() should only dirty one cache line, and access as few
cache lines as possible.
Add sp_ (Slow Path) prefix to fields that are not used in fast path,
to make clear what is going on.
After this patch pahole reports something much better, as all
ndo_start_xmit() needed fields are packed into two cache lines instead
of seven or eight
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
struct netdev_queue * tx_queue; /* 0x10 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x18 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x20 0x8 */
/* XXX 24 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
unsigned int tx_dropped; /* 0x44 0x4 */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
struct mlx4_bf bf; /* 0x70 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
__be32 doorbell_qpn; /* 0x88 0x4 */
__be32 mr_key; /* 0x8c 0x4 */
u32 size; /* 0x90 0x4 */
u32 size_mask; /* 0x94 0x4 */
u32 full_size; /* 0x98 0x4 */
u32 buf_size; /* 0x9c 0x4 */
void * buf; /* 0xa0 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0xa8 0x8 */
int qpn; /* 0xb0 0x4 */
u8 queue_index; /* 0xb4 0x1 */
bool bf_enabled; /* 0xb5 0x1 */
bool bf_alloced; /* 0xb6 0x1 */
u8 hwtstamp_tx_type; /* 0xb7 0x1 */
u8 * bounce_buf; /* 0xb8 0x8 */
/* --- cacheline 3 boundary (192 bytes) --- */
long unsigned int queue_stopped; /* 0xc0 0x8 */
struct mlx4_hwq_resources sp_wqres; /* 0xc8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
struct mlx4_qp sp_qp; /* 0x120 0x30 */
/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
struct mlx4_qp_context sp_context; /* 0x150 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
cpumask_t sp_affinity_mask; /* 0x248 0x20 */
enum mlx4_qp_state sp_qp_state; /* 0x268 0x4 */
u16 sp_stride; /* 0x26c 0x2 */
u16 sp_cqn; /* 0x26e 0x2 */
/* size: 640, cachelines: 10, members: 36 */
/* sum members: 600, holes: 1, sum holes: 24 */
/* padding: 16 */
};
Instead of this silly placement :
struct mlx4_en_tx_ring {
u32 last_nr_txbb; /* 0 0x4 */
u32 cons; /* 0x4 0x4 */
long unsigned int wake_queue; /* 0x8 0x8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 prod; /* 0x40 0x4 */
/* XXX 4 bytes hole, try to pack */
long unsigned int bytes; /* 0x48 0x8 */
long unsigned int packets; /* 0x50 0x8 */
long unsigned int tx_csum; /* 0x58 0x8 */
long unsigned int tso_packets; /* 0x60 0x8 */
long unsigned int xmit_more; /* 0x68 0x8 */
unsigned int tx_dropped; /* 0x70 0x4 */
/* XXX 4 bytes hole, try to pack */
struct mlx4_bf bf; /* 0x78 0x18 */
/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
long unsigned int queue_stopped; /* 0x90 0x8 */
cpumask_t affinity_mask; /* 0x98 0x10 */
struct mlx4_qp qp; /* 0xa8 0x30 */
/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
struct mlx4_hwq_resources wqres; /* 0xd8 0x58 */
/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
u32 size; /* 0x130 0x4 */
u32 size_mask; /* 0x134 0x4 */
u16 stride; /* 0x138 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 full_size; /* 0x13c 0x4 */
/* --- cacheline 5 boundary (320 bytes) --- */
u16 cqn; /* 0x140 0x2 */
/* XXX 2 bytes hole, try to pack */
u32 buf_size; /* 0x144 0x4 */
__be32 doorbell_qpn; /* 0x148 0x4 */
__be32 mr_key; /* 0x14c 0x4 */
void * buf; /* 0x150 0x8 */
struct mlx4_en_tx_info * tx_info; /* 0x158 0x8 */
struct mlx4_en_rx_ring * recycle_ring; /* 0x160 0x8 */
u32 (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168 0x8 */
u8 * bounce_buf; /* 0x170 0x8 */
struct mlx4_qp_context context; /* 0x178 0xf8 */
/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
int qpn; /* 0x270 0x4 */
enum mlx4_qp_state qp_state; /* 0x274 0x4 */
u8 queue_index; /* 0x278 0x1 */
bool bf_enabled; /* 0x279 0x1 */
bool bf_alloced; /* 0x27a 0x1 */
/* XXX 5 bytes hole, try to pack */
/* --- cacheline 10 boundary (640 bytes) --- */
struct netdev_queue * tx_queue; /* 0x280 0x8 */
int hwtstamp_tx_type; /* 0x288 0x4 */
/* size: 704, cachelines: 11, members: 36 */
/* sum members: 587, holes: 6, sum holes: 65 */
/* padding: 52 */
};
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-22 16:56:10 -07:00
|
|
|
mlx4_qp_modify(mdev->dev, NULL, ring->sp_qp_state,
|
|
|
|
MLX4_QP_STATE_RST, NULL, 0, 0, &ring->sp_qp);
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
|
|
|
|
2015-06-25 02:29:42 -06:00
|
|
|
static inline bool mlx4_en_is_tx_ring_full(struct mlx4_en_tx_ring *ring)
|
|
|
|
{
|
|
|
|
return ring->prod - ring->cons > ring->full_size;
|
|
|
|
}
|
|
|
|
|
2013-07-25 10:21:23 -06:00
|
|
|
static void mlx4_en_stamp_wqe(struct mlx4_en_priv *priv,
|
|
|
|
struct mlx4_en_tx_ring *ring, int index,
|
|
|
|
u8 owner)
|
|
|
|
{
|
|
|
|
__be32 stamp = cpu_to_be32(STAMP_VAL | (!!owner << STAMP_SHIFT));
|
2017-06-15 05:35:39 -06:00
|
|
|
struct mlx4_en_tx_desc *tx_desc = ring->buf + (index << LOG_TXBB_SIZE);
|
2013-07-25 10:21:23 -06:00
|
|
|
struct mlx4_en_tx_info *tx_info = &ring->tx_info[index];
|
|
|
|
void *end = ring->buf + ring->buf_size;
|
|
|
|
__be32 *ptr = (__be32 *)tx_desc;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/* Optimize the common case when there are no wraparounds */
|
2017-06-15 05:35:39 -06:00
|
|
|
if (likely((void *)tx_desc +
|
|
|
|
(tx_info->nr_txbb << LOG_TXBB_SIZE) <= end)) {
|
2013-07-25 10:21:23 -06:00
|
|
|
/* Stamp the freed descriptor */
|
2017-06-15 05:35:39 -06:00
|
|
|
for (i = 0; i < tx_info->nr_txbb << LOG_TXBB_SIZE;
|
2013-07-25 10:21:23 -06:00
|
|
|
i += STAMP_STRIDE) {
|
|
|
|
*ptr = stamp;
|
|
|
|
ptr += STAMP_DWORDS;
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
/* Stamp the freed descriptor */
|
2017-06-15 05:35:39 -06:00
|
|
|
for (i = 0; i < tx_info->nr_txbb << LOG_TXBB_SIZE;
|
2013-07-25 10:21:23 -06:00
|
|
|
i += STAMP_STRIDE) {
|
|
|
|
*ptr = stamp;
|
|
|
|
ptr += STAMP_DWORDS;
|
|
|
|
if ((void *)ptr >= end) {
|
|
|
|
ptr = ring->buf;
|
|
|
|
stamp ^= cpu_to_be32(0x80000000);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-04-15 10:46:52 -06:00
|
|
|
INDIRECT_CALLABLE_DECLARE(u32 mlx4_en_free_tx_desc(struct mlx4_en_priv *priv,
|
|
|
|
struct mlx4_en_tx_ring *ring,
|
|
|
|
int index, u64 timestamp,
|
|
|
|
int napi_mode));
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2016-07-19 13:16:55 -06:00
|
|
|
u32 mlx4_en_free_tx_desc(struct mlx4_en_priv *priv,
|
|
|
|
struct mlx4_en_tx_ring *ring,
|
2017-06-15 05:35:31 -06:00
|
|
|
int index, u64 timestamp,
|
2016-07-19 13:16:55 -06:00
|
|
|
int napi_mode)
|
2008-10-22 16:47:49 -06:00
|
|
|
{
|
|
|
|
struct mlx4_en_tx_info *tx_info = &ring->tx_info[index];
|
2017-06-15 05:35:39 -06:00
|
|
|
struct mlx4_en_tx_desc *tx_desc = ring->buf + (index << LOG_TXBB_SIZE);
|
2008-10-22 16:47:49 -06:00
|
|
|
struct mlx4_wqe_data_seg *data = (void *) tx_desc + tx_info->data_offset;
|
|
|
|
void *end = ring->buf + ring->buf_size;
|
2014-10-05 03:35:13 -06:00
|
|
|
struct sk_buff *skb = tx_info->skb;
|
|
|
|
int nr_maps = tx_info->nr_maps;
|
2008-10-22 16:47:49 -06:00
|
|
|
int i;
|
2013-04-23 00:06:49 -06:00
|
|
|
|
2014-10-05 03:35:14 -06:00
|
|
|
/* We do not touch skb here, so prefetch skb->users location
|
|
|
|
* to speedup consume_skb()
|
|
|
|
*/
|
|
|
|
prefetchw(&skb->users);
|
|
|
|
|
2014-10-05 03:35:13 -06:00
|
|
|
if (unlikely(timestamp)) {
|
|
|
|
struct skb_shared_hwtstamps hwts;
|
|
|
|
|
|
|
|
mlx4_en_fill_hwtstamps(priv->mdev, &hwts, timestamp);
|
2013-04-23 00:06:49 -06:00
|
|
|
skb_tstamp_tx(skb, &hwts);
|
|
|
|
}
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2017-06-15 05:35:40 -06:00
|
|
|
if (!tx_info->inl) {
|
|
|
|
if (tx_info->linear)
|
|
|
|
dma_unmap_single(priv->ddev,
|
|
|
|
tx_info->map0_dma,
|
|
|
|
tx_info->map0_byte_count,
|
|
|
|
PCI_DMA_TODEVICE);
|
|
|
|
else
|
|
|
|
dma_unmap_page(priv->ddev,
|
|
|
|
tx_info->map0_dma,
|
|
|
|
tx_info->map0_byte_count,
|
|
|
|
PCI_DMA_TODEVICE);
|
|
|
|
/* Optimize the common case when there are no wraparounds */
|
|
|
|
if (likely((void *)tx_desc +
|
|
|
|
(tx_info->nr_txbb << LOG_TXBB_SIZE) <= end)) {
|
2014-10-05 03:35:13 -06:00
|
|
|
for (i = 1; i < nr_maps; i++) {
|
|
|
|
data++;
|
2012-03-05 21:03:34 -07:00
|
|
|
dma_unmap_page(priv->ddev,
|
2014-10-05 03:35:13 -06:00
|
|
|
(dma_addr_t)be64_to_cpu(data->addr),
|
|
|
|
be32_to_cpu(data->byte_count),
|
|
|
|
PCI_DMA_TODEVICE);
|
2009-01-08 11:57:15 -07:00
|
|
|
}
|
2017-06-15 05:35:40 -06:00
|
|
|
} else {
|
|
|
|
if ((void *)data >= end)
|
2011-06-16 13:08:06 -06:00
|
|
|
data = ring->buf + ((void *)data - end);
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2014-10-05 03:35:13 -06:00
|
|
|
for (i = 1; i < nr_maps; i++) {
|
|
|
|
data++;
|
2009-01-08 11:57:15 -07:00
|
|
|
/* Check for wraparound before unmapping */
|
|
|
|
if ((void *) data >= end)
|
2011-06-16 13:08:06 -06:00
|
|
|
data = ring->buf;
|
2012-03-05 21:03:34 -07:00
|
|
|
dma_unmap_page(priv->ddev,
|
2014-10-05 03:35:13 -06:00
|
|
|
(dma_addr_t)be64_to_cpu(data->addr),
|
|
|
|
be32_to_cpu(data->byte_count),
|
|
|
|
PCI_DMA_TODEVICE);
|
2009-01-08 11:57:15 -07:00
|
|
|
}
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
|
|
|
}
|
2016-03-11 01:44:08 -07:00
|
|
|
napi_consume_skb(skb, napi_mode);
|
|
|
|
|
2008-10-22 16:47:49 -06:00
|
|
|
return tx_info->nr_txbb;
|
|
|
|
}
|
|
|
|
|
2020-04-15 10:46:52 -06:00
|
|
|
INDIRECT_CALLABLE_DECLARE(u32 mlx4_en_recycle_tx_desc(struct mlx4_en_priv *priv,
|
|
|
|
struct mlx4_en_tx_ring *ring,
|
|
|
|
int index, u64 timestamp,
|
|
|
|
int napi_mode));
|
|
|
|
|
2016-07-19 13:16:55 -06:00
|
|
|
u32 mlx4_en_recycle_tx_desc(struct mlx4_en_priv *priv,
|
|
|
|
struct mlx4_en_tx_ring *ring,
|
2017-06-15 05:35:31 -06:00
|
|
|
int index, u64 timestamp,
|
2016-07-19 13:16:55 -06:00
|
|
|
int napi_mode)
|
|
|
|
{
|
|
|
|
struct mlx4_en_tx_info *tx_info = &ring->tx_info[index];
|
|
|
|
struct mlx4_en_rx_alloc frame = {
|
|
|
|
.page = tx_info->page,
|
|
|
|
.dma = tx_info->map0_dma,
|
|
|
|
};
|
|
|
|
|
2020-10-08 12:45:26 -06:00
|
|
|
if (!napi_mode || !mlx4_en_rx_recycle(ring->recycle_ring, &frame)) {
|
2016-07-19 13:16:55 -06:00
|
|
|
dma_unmap_page(priv->ddev, tx_info->map0_dma,
|
2017-03-08 09:17:06 -07:00
|
|
|
PAGE_SIZE, priv->dma_dir);
|
2016-07-19 13:16:55 -06:00
|
|
|
put_page(tx_info->page);
|
|
|
|
}
|
|
|
|
|
|
|
|
return tx_info->nr_txbb;
|
|
|
|
}
|
2008-10-22 16:47:49 -06:00
|
|
|
|
|
|
|
int mlx4_en_free_tx_buf(struct net_device *dev, struct mlx4_en_tx_ring *ring)
|
|
|
|
{
|
|
|
|
struct mlx4_en_priv *priv = netdev_priv(dev);
|
|
|
|
int cnt = 0;
|
|
|
|
|
|
|
|
/* Skip last polled descriptor */
|
|
|
|
ring->cons += ring->last_nr_txbb;
|
2009-06-01 14:27:13 -06:00
|
|
|
en_dbg(DRV, priv, "Freeing Tx buf - cons:0x%x prod:0x%x\n",
|
2008-10-22 16:47:49 -06:00
|
|
|
ring->cons, ring->prod);
|
|
|
|
|
|
|
|
if ((u32) (ring->prod - ring->cons) > ring->size) {
|
|
|
|
if (netif_msg_tx_err(priv))
|
2009-06-01 14:27:13 -06:00
|
|
|
en_warn(priv, "Tx consumer passed producer!\n");
|
2008-10-22 16:47:49 -06:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
while (ring->cons != ring->prod) {
|
2016-07-19 13:16:55 -06:00
|
|
|
ring->last_nr_txbb = ring->free_tx_desc(priv, ring,
|
2008-10-22 16:47:49 -06:00
|
|
|
ring->cons & ring->size_mask,
|
2017-06-15 05:35:31 -06:00
|
|
|
0, 0 /* Non-NAPI caller */);
|
2008-10-22 16:47:49 -06:00
|
|
|
ring->cons += ring->last_nr_txbb;
|
|
|
|
cnt++;
|
|
|
|
}
|
|
|
|
|
2016-11-02 09:12:24 -06:00
|
|
|
if (ring->tx_queue)
|
|
|
|
netdev_tx_reset_queue(ring->tx_queue);
|
2013-02-06 00:58:41 -07:00
|
|
|
|
2008-10-22 16:47:49 -06:00
|
|
|
if (cnt)
|
2009-06-01 14:27:13 -06:00
|
|
|
en_dbg(DRV, priv, "Freed %d uncompleted tx descriptors\n", cnt);
|
2008-10-22 16:47:49 -06:00
|
|
|
|
|
|
|
return cnt;
|
|
|
|
}
|
|
|
|
|
2020-04-22 10:13:29 -06:00
|
|
|
int mlx4_en_process_tx_cq(struct net_device *dev,
|
|
|
|
struct mlx4_en_cq *cq, int napi_budget)
|
2008-10-22 16:47:49 -06:00
|
|
|
{
|
|
|
|
struct mlx4_en_priv *priv = netdev_priv(dev);
|
|
|
|
struct mlx4_cq *mcq = &cq->mcq;
|
2016-11-02 09:12:24 -06:00
|
|
|
struct mlx4_en_tx_ring *ring = priv->tx_ring[cq->type][cq->ring];
|
2011-11-26 12:55:10 -07:00
|
|
|
struct mlx4_cqe *cqe;
|
2017-06-15 05:35:34 -06:00
|
|
|
u16 index, ring_index, stamp_index;
|
2008-10-22 16:47:49 -06:00
|
|
|
u32 txbbs_skipped = 0;
|
2013-07-25 10:21:23 -06:00
|
|
|
u32 txbbs_stamp = 0;
|
2011-11-26 12:55:10 -07:00
|
|
|
u32 cons_index = mcq->cons_index;
|
|
|
|
int size = cq->size;
|
|
|
|
u32 size_mask = ring->size_mask;
|
|
|
|
struct mlx4_cqe *buf = cq->buf;
|
2012-04-22 20:18:50 -06:00
|
|
|
u32 packets = 0;
|
|
|
|
u32 bytes = 0;
|
2012-10-21 08:59:24 -06:00
|
|
|
int factor = priv->cqe_factor;
|
2013-12-19 12:20:17 -07:00
|
|
|
int done = 0;
|
2014-07-08 02:28:12 -06:00
|
|
|
int budget = priv->tx_work_limit;
|
2014-10-05 03:35:15 -06:00
|
|
|
u32 last_nr_txbb;
|
|
|
|
u32 ring_cons;
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2017-06-15 05:35:34 -06:00
|
|
|
if (unlikely(!priv->port_up))
|
2020-04-22 10:13:29 -06:00
|
|
|
return 0;
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2014-10-08 09:19:27 -06:00
|
|
|
netdev_txq_bql_complete_prefetchw(ring->tx_queue);
|
|
|
|
|
2011-11-26 12:55:10 -07:00
|
|
|
index = cons_index & size_mask;
|
2014-09-18 02:51:01 -06:00
|
|
|
cqe = mlx4_en_get_cqe(buf, index, priv->cqe_size) + factor;
|
locking/atomics: COCCINELLE/treewide: Convert trivial ACCESS_ONCE() patterns to READ_ONCE()/WRITE_ONCE()
Please do not apply this to mainline directly, instead please re-run the
coccinelle script shown below and apply its output.
For several reasons, it is desirable to use {READ,WRITE}_ONCE() in
preference to ACCESS_ONCE(), and new code is expected to use one of the
former. So far, there's been no reason to change most existing uses of
ACCESS_ONCE(), as these aren't harmful, and changing them results in
churn.
However, for some features, the read/write distinction is critical to
correct operation. To distinguish these cases, separate read/write
accessors must be used. This patch migrates (most) remaining
ACCESS_ONCE() instances to {READ,WRITE}_ONCE(), using the following
coccinelle script:
----
// Convert trivial ACCESS_ONCE() uses to equivalent READ_ONCE() and
// WRITE_ONCE()
// $ make coccicheck COCCI=/home/mark/once.cocci SPFLAGS="--include-headers" MODE=patch
virtual patch
@ depends on patch @
expression E1, E2;
@@
- ACCESS_ONCE(E1) = E2
+ WRITE_ONCE(E1, E2)
@ depends on patch @
expression E;
@@
- ACCESS_ONCE(E)
+ READ_ONCE(E)
----
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: davem@davemloft.net
Cc: linux-arch@vger.kernel.org
Cc: mpe@ellerman.id.au
Cc: shuah@kernel.org
Cc: snitzer@redhat.com
Cc: thor.thayer@linux.intel.com
Cc: tj@kernel.org
Cc: viro@zeniv.linux.org.uk
Cc: will.deacon@arm.com
Link: http://lkml.kernel.org/r/1508792849-3115-19-git-send-email-paulmck@linux.vnet.ibm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-10-23 15:07:29 -06:00
|
|
|
last_nr_txbb = READ_ONCE(ring->last_nr_txbb);
|
|
|
|
ring_cons = READ_ONCE(ring->cons);
|
2014-10-05 03:35:15 -06:00
|
|
|
ring_index = ring_cons & size_mask;
|
2013-07-25 10:21:23 -06:00
|
|
|
stamp_index = ring_index;
|
2011-11-26 12:55:10 -07:00
|
|
|
|
|
|
|
/* Process all completed CQEs */
|
|
|
|
while (XNOR(cqe->owner_sr_opcode & MLX4_CQE_OWNER_MASK,
|
2013-12-19 12:20:17 -07:00
|
|
|
cons_index & size) && (done < budget)) {
|
2017-06-15 05:35:34 -06:00
|
|
|
u16 new_index;
|
|
|
|
|
2011-11-26 12:55:10 -07:00
|
|
|
/*
|
|
|
|
* make sure we read the CQE after we read the
|
|
|
|
* ownership bit
|
|
|
|
*/
|
2015-04-08 19:49:36 -06:00
|
|
|
dma_rmb();
|
2011-11-26 12:55:10 -07:00
|
|
|
|
2013-08-21 01:08:57 -06:00
|
|
|
if (unlikely((cqe->owner_sr_opcode & MLX4_CQE_OPCODE_MASK) ==
|
|
|
|
MLX4_CQE_OPCODE_ERROR)) {
|
|
|
|
struct mlx4_err_cqe *cqe_err = (struct mlx4_err_cqe *)cqe;
|
|
|
|
|
|
|
|
en_err(priv, "CQE error - vendor syndrome: 0x%x syndrome: 0x%x\n",
|
|
|
|
cqe_err->vendor_err_syndrome,
|
|
|
|
cqe_err->syndrome);
|
|
|
|
}
|
|
|
|
|
2011-11-26 12:55:10 -07:00
|
|
|
/* Skip over last polled CQE */
|
|
|
|
new_index = be16_to_cpu(cqe->wqe_index) & size_mask;
|
|
|
|
|
2008-10-22 16:47:49 -06:00
|
|
|
do {
|
2016-04-23 12:35:46 -06:00
|
|
|
u64 timestamp = 0;
|
|
|
|
|
2014-10-05 03:35:15 -06:00
|
|
|
txbbs_skipped += last_nr_txbb;
|
|
|
|
ring_index = (ring_index + last_nr_txbb) & size_mask;
|
2016-04-23 12:35:46 -06:00
|
|
|
|
|
|
|
if (unlikely(ring->tx_info[ring_index].ts_requested))
|
2013-04-23 00:06:49 -06:00
|
|
|
timestamp = mlx4_en_get_cqe_ts(cqe);
|
|
|
|
|
2011-11-26 12:55:10 -07:00
|
|
|
/* free next descriptor */
|
2020-04-15 10:46:52 -06:00
|
|
|
last_nr_txbb = INDIRECT_CALL_2(ring->free_tx_desc,
|
|
|
|
mlx4_en_free_tx_desc,
|
|
|
|
mlx4_en_recycle_tx_desc,
|
2011-11-26 12:55:10 -07:00
|
|
|
priv, ring, ring_index,
|
2017-06-15 05:35:31 -06:00
|
|
|
timestamp, napi_budget);
|
2013-07-25 10:21:23 -06:00
|
|
|
|
|
|
|
mlx4_en_stamp_wqe(priv, ring, stamp_index,
|
2014-10-05 03:35:15 -06:00
|
|
|
!!((ring_cons + txbbs_stamp) &
|
2013-07-25 10:21:23 -06:00
|
|
|
ring->size));
|
|
|
|
stamp_index = ring_index;
|
|
|
|
txbbs_stamp = txbbs_skipped;
|
2012-04-22 20:18:50 -06:00
|
|
|
packets++;
|
|
|
|
bytes += ring->tx_info[ring_index].nr_bytes;
|
2013-12-19 12:20:17 -07:00
|
|
|
} while ((++done < budget) && (ring_index != new_index));
|
2011-11-26 12:55:10 -07:00
|
|
|
|
|
|
|
++cons_index;
|
|
|
|
index = cons_index & size_mask;
|
2014-09-18 02:51:01 -06:00
|
|
|
cqe = mlx4_en_get_cqe(buf, index, priv->cqe_size) + factor;
|
2011-11-26 12:55:10 -07:00
|
|
|
}
|
2008-10-22 16:47:49 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* To prevent CQ overflow we first update CQ consumer and only then
|
|
|
|
* the ring consumer.
|
|
|
|
*/
|
2011-11-26 12:55:10 -07:00
|
|
|
mcq->cons_index = cons_index;
|
2008-10-22 16:47:49 -06:00
|
|
|
mlx4_cq_set_ci(mcq);
|
|
|
|
wmb();
|
2014-10-05 03:35:15 -06:00
|
|
|
|
|
|
|
/* we want to dirty this cache line once */
|
locking/atomics: COCCINELLE/treewide: Convert trivial ACCESS_ONCE() patterns to READ_ONCE()/WRITE_ONCE()
Please do not apply this to mainline directly, instead please re-run the
coccinelle script shown below and apply its output.
For several reasons, it is desirable to use {READ,WRITE}_ONCE() in
preference to ACCESS_ONCE(), and new code is expected to use one of the
former. So far, there's been no reason to change most existing uses of
ACCESS_ONCE(), as these aren't harmful, and changing them results in
churn.
However, for some features, the read/write distinction is critical to
correct operation. To distinguish these cases, separate read/write
accessors must be used. This patch migrates (most) remaining
ACCESS_ONCE() instances to {READ,WRITE}_ONCE(), using the following
coccinelle script:
----
// Convert trivial ACCESS_ONCE() uses to equivalent READ_ONCE() and
// WRITE_ONCE()
// $ make coccicheck COCCI=/home/mark/once.cocci SPFLAGS="--include-headers" MODE=patch
virtual patch
@ depends on patch @
expression E1, E2;
@@
- ACCESS_ONCE(E1) = E2
+ WRITE_ONCE(E1, E2)
@ depends on patch @
expression E;
@@
- ACCESS_ONCE(E)
+ READ_ONCE(E)
----
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: davem@davemloft.net
Cc: linux-arch@vger.kernel.org
Cc: mpe@ellerman.id.au
Cc: shuah@kernel.org
Cc: snitzer@redhat.com
Cc: thor.thayer@linux.intel.com
Cc: tj@kernel.org
Cc: viro@zeniv.linux.org.uk
Cc: will.deacon@arm.com
Link: http://lkml.kernel.org/r/1508792849-3115-19-git-send-email-paulmck@linux.vnet.ibm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-10-23 15:07:29 -06:00
|
|
|
WRITE_ONCE(ring->last_nr_txbb, last_nr_txbb);
|
|
|
|
WRITE_ONCE(ring->cons, ring_cons + txbbs_skipped);
|
2014-10-05 03:35:15 -06:00
|
|
|
|
2017-06-15 05:35:34 -06:00
|
|
|
if (cq->type == TX_XDP)
|
2020-04-22 10:13:29 -06:00
|
|
|
return done;
|
2016-07-19 13:16:55 -06:00
|
|
|
|
2012-04-22 20:18:50 -06:00
|
|
|
netdev_tx_completed_queue(ring->tx_queue, packets, bytes);
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2015-06-25 02:29:42 -06:00
|
|
|
/* Wakeup Tx queue if this stopped, and ring is not full.
|
2012-08-02 18:38:37 -06:00
|
|
|
*/
|
2015-06-25 02:29:42 -06:00
|
|
|
if (netif_tx_queue_stopped(ring->tx_queue) &&
|
|
|
|
!mlx4_en_is_tx_ring_full(ring)) {
|
2012-08-02 18:38:37 -06:00
|
|
|
netif_tx_wake_queue(ring->tx_queue);
|
2014-03-02 01:25:00 -07:00
|
|
|
ring->wake_queue++;
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
2017-06-15 05:35:34 -06:00
|
|
|
|
2020-04-22 10:13:29 -06:00
|
|
|
return done;
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
void mlx4_en_tx_irq(struct mlx4_cq *mcq)
|
|
|
|
{
|
|
|
|
struct mlx4_en_cq *cq = container_of(mcq, struct mlx4_en_cq, mcq);
|
|
|
|
struct mlx4_en_priv *priv = netdev_priv(cq->dev);
|
|
|
|
|
2014-10-29 17:54:45 -06:00
|
|
|
if (likely(priv->port_up))
|
|
|
|
napi_schedule_irqoff(&cq->napi);
|
2013-12-19 12:20:17 -07:00
|
|
|
else
|
|
|
|
mlx4_en_arm_cq(priv, cq);
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
|
|
|
|
2013-12-19 12:20:17 -07:00
|
|
|
/* TX CQ polling - called by NAPI */
|
|
|
|
int mlx4_en_poll_tx_cq(struct napi_struct *napi, int budget)
|
|
|
|
{
|
|
|
|
struct mlx4_en_cq *cq = container_of(napi, struct mlx4_en_cq, napi);
|
|
|
|
struct net_device *dev = cq->dev;
|
|
|
|
struct mlx4_en_priv *priv = netdev_priv(dev);
|
2020-04-22 10:13:29 -06:00
|
|
|
int work_done;
|
2013-12-19 12:20:17 -07:00
|
|
|
|
2020-04-22 10:13:29 -06:00
|
|
|
work_done = mlx4_en_process_tx_cq(dev, cq, budget);
|
|
|
|
if (work_done >= budget)
|
2014-07-08 02:28:12 -06:00
|
|
|
return budget;
|
2013-12-19 12:20:17 -07:00
|
|
|
|
2020-04-22 10:13:29 -06:00
|
|
|
if (napi_complete_done(napi, work_done))
|
|
|
|
mlx4_en_arm_cq(priv, cq);
|
2014-07-08 02:28:12 -06:00
|
|
|
|
|
|
|
return 0;
|
2013-12-19 12:20:17 -07:00
|
|
|
}
|
2008-10-22 16:47:49 -06:00
|
|
|
|
|
|
|
static struct mlx4_en_tx_desc *mlx4_en_bounce_to_desc(struct mlx4_en_priv *priv,
|
|
|
|
struct mlx4_en_tx_ring *ring,
|
|
|
|
u32 index,
|
|
|
|
unsigned int desc_size)
|
|
|
|
{
|
2017-06-15 05:35:39 -06:00
|
|
|
u32 copy = (ring->size - index) << LOG_TXBB_SIZE;
|
2008-10-22 16:47:49 -06:00
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = desc_size - copy - 4; i >= 0; i -= 4) {
|
|
|
|
if ((i & (TXBB_SIZE - 1)) == 0)
|
|
|
|
wmb();
|
|
|
|
|
|
|
|
*((u32 *) (ring->buf + i)) =
|
|
|
|
*((u32 *) (ring->bounce_buf + copy + i));
|
|
|
|
}
|
|
|
|
|
|
|
|
for (i = copy - 4; i >= 4 ; i -= 4) {
|
|
|
|
if ((i & (TXBB_SIZE - 1)) == 0)
|
|
|
|
wmb();
|
|
|
|
|
2017-06-15 05:35:39 -06:00
|
|
|
*((u32 *)(ring->buf + (index << LOG_TXBB_SIZE) + i)) =
|
2008-10-22 16:47:49 -06:00
|
|
|
*((u32 *) (ring->bounce_buf + i));
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Return real descriptor location */
|
2017-06-15 05:35:39 -06:00
|
|
|
return ring->buf + (index << LOG_TXBB_SIZE);
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
|
|
|
|
2014-10-05 03:35:20 -06:00
|
|
|
/* Decide if skb can be inlined in tx descriptor to avoid dma mapping
|
|
|
|
*
|
|
|
|
* It seems strange we do not simply use skb_copy_bits().
|
|
|
|
* This would allow to inline all skbs iff skb->len <= inline_thold
|
|
|
|
*
|
|
|
|
* Note that caller already checked skb was not a gso packet
|
|
|
|
*/
|
2014-10-05 03:35:09 -06:00
|
|
|
static bool is_inline(int inline_thold, const struct sk_buff *skb,
|
2014-10-05 03:35:17 -06:00
|
|
|
const struct skb_shared_info *shinfo,
|
2014-10-05 03:35:09 -06:00
|
|
|
void **pfrag)
|
2008-10-22 16:47:49 -06:00
|
|
|
{
|
|
|
|
void *ptr;
|
|
|
|
|
2014-10-05 03:35:20 -06:00
|
|
|
if (skb->len > inline_thold || !inline_thold)
|
|
|
|
return false;
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2014-10-05 03:35:20 -06:00
|
|
|
if (shinfo->nr_frags == 1) {
|
|
|
|
ptr = skb_frag_address_safe(&shinfo->frags[0]);
|
|
|
|
if (unlikely(!ptr))
|
|
|
|
return false;
|
|
|
|
*pfrag = ptr;
|
|
|
|
return true;
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
2014-10-05 03:35:20 -06:00
|
|
|
if (shinfo->nr_frags)
|
|
|
|
return false;
|
|
|
|
return true;
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
|
|
|
|
2014-10-05 03:35:09 -06:00
|
|
|
static int inline_size(const struct sk_buff *skb)
|
2008-10-22 16:47:49 -06:00
|
|
|
{
|
|
|
|
if (skb->len + CTRL_SIZE + sizeof(struct mlx4_wqe_inline_seg)
|
|
|
|
<= MLX4_INLINE_ALIGN)
|
|
|
|
return ALIGN(skb->len + CTRL_SIZE +
|
|
|
|
sizeof(struct mlx4_wqe_inline_seg), 16);
|
|
|
|
else
|
|
|
|
return ALIGN(skb->len + CTRL_SIZE + 2 *
|
|
|
|
sizeof(struct mlx4_wqe_inline_seg), 16);
|
|
|
|
}
|
|
|
|
|
2014-10-05 03:35:09 -06:00
|
|
|
static int get_real_size(const struct sk_buff *skb,
|
2014-10-05 03:35:17 -06:00
|
|
|
const struct skb_shared_info *shinfo,
|
2014-10-05 03:35:09 -06:00
|
|
|
struct net_device *dev,
|
2014-10-05 03:35:20 -06:00
|
|
|
int *lso_header_size,
|
|
|
|
bool *inline_ok,
|
|
|
|
void **pfrag)
|
2008-10-22 16:47:49 -06:00
|
|
|
{
|
|
|
|
struct mlx4_en_priv *priv = netdev_priv(dev);
|
|
|
|
int real_size;
|
|
|
|
|
2014-10-05 03:35:17 -06:00
|
|
|
if (shinfo->gso_size) {
|
2014-10-05 03:35:20 -06:00
|
|
|
*inline_ok = false;
|
2013-12-23 07:09:44 -07:00
|
|
|
if (skb->encapsulation)
|
|
|
|
*lso_header_size = (skb_inner_transport_header(skb) - skb->data) + inner_tcp_hdrlen(skb);
|
|
|
|
else
|
|
|
|
*lso_header_size = skb_transport_offset(skb) + tcp_hdrlen(skb);
|
2014-10-05 03:35:17 -06:00
|
|
|
real_size = CTRL_SIZE + shinfo->nr_frags * DS_SIZE +
|
2008-10-22 16:47:49 -06:00
|
|
|
ALIGN(*lso_header_size + 4, DS_SIZE);
|
|
|
|
if (unlikely(*lso_header_size != skb_headlen(skb))) {
|
|
|
|
/* We add a segment for the skb linear buffer only if
|
|
|
|
* it contains data */
|
|
|
|
if (*lso_header_size < skb_headlen(skb))
|
|
|
|
real_size += DS_SIZE;
|
|
|
|
else {
|
|
|
|
if (netif_msg_tx_err(priv))
|
2009-06-01 14:27:13 -06:00
|
|
|
en_warn(priv, "Non-linear headers\n");
|
2008-10-22 16:47:49 -06:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
*lso_header_size = 0;
|
2014-10-05 03:35:20 -06:00
|
|
|
*inline_ok = is_inline(priv->prof->inline_thold, skb,
|
|
|
|
shinfo, pfrag);
|
|
|
|
|
|
|
|
if (*inline_ok)
|
2008-10-22 16:47:49 -06:00
|
|
|
real_size = inline_size(skb);
|
2014-10-05 03:35:20 -06:00
|
|
|
else
|
|
|
|
real_size = CTRL_SIZE +
|
|
|
|
(shinfo->nr_frags + 1) * DS_SIZE;
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
return real_size;
|
|
|
|
}
|
|
|
|
|
2014-10-05 03:35:09 -06:00
|
|
|
static void build_inline_wqe(struct mlx4_en_tx_desc *tx_desc,
|
|
|
|
const struct sk_buff *skb,
|
2014-10-05 03:35:17 -06:00
|
|
|
const struct skb_shared_info *shinfo,
|
2016-07-19 13:16:54 -06:00
|
|
|
void *fragptr)
|
2008-10-22 16:47:49 -06:00
|
|
|
{
|
|
|
|
struct mlx4_wqe_inline_seg *inl = &tx_desc->inl;
|
2017-08-15 11:29:19 -06:00
|
|
|
int spc = MLX4_INLINE_ALIGN - CTRL_SIZE - sizeof(*inl);
|
2014-10-05 03:35:18 -06:00
|
|
|
unsigned int hlen = skb_headlen(skb);
|
2008-10-22 16:47:49 -06:00
|
|
|
|
|
|
|
if (skb->len <= spc) {
|
2014-03-02 01:24:59 -07:00
|
|
|
if (likely(skb->len >= MIN_PKT_LEN)) {
|
|
|
|
inl->byte_count = cpu_to_be32(1 << 31 | skb->len);
|
|
|
|
} else {
|
|
|
|
inl->byte_count = cpu_to_be32(1 << 31 | MIN_PKT_LEN);
|
|
|
|
memset(((void *)(inl + 1)) + skb->len, 0,
|
|
|
|
MIN_PKT_LEN - skb->len);
|
|
|
|
}
|
2014-10-05 03:35:18 -06:00
|
|
|
skb_copy_from_linear_data(skb, inl + 1, hlen);
|
2014-10-05 03:35:17 -06:00
|
|
|
if (shinfo->nr_frags)
|
2014-10-05 03:35:18 -06:00
|
|
|
memcpy(((void *)(inl + 1)) + hlen, fragptr,
|
2014-10-05 03:35:17 -06:00
|
|
|
skb_frag_size(&shinfo->frags[0]));
|
2008-10-22 16:47:49 -06:00
|
|
|
|
|
|
|
} else {
|
|
|
|
inl->byte_count = cpu_to_be32(1 << 31 | spc);
|
2014-10-05 03:35:18 -06:00
|
|
|
if (hlen <= spc) {
|
|
|
|
skb_copy_from_linear_data(skb, inl + 1, hlen);
|
|
|
|
if (hlen < spc) {
|
|
|
|
memcpy(((void *)(inl + 1)) + hlen,
|
|
|
|
fragptr, spc - hlen);
|
|
|
|
fragptr += spc - hlen;
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
|
|
|
inl = (void *) (inl + 1) + spc;
|
|
|
|
memcpy(((void *)(inl + 1)), fragptr, skb->len - spc);
|
|
|
|
} else {
|
|
|
|
skb_copy_from_linear_data(skb, inl + 1, spc);
|
|
|
|
inl = (void *) (inl + 1) + spc;
|
|
|
|
skb_copy_from_linear_data_offset(skb, spc, inl + 1,
|
2014-10-05 03:35:18 -06:00
|
|
|
hlen - spc);
|
2014-10-05 03:35:17 -06:00
|
|
|
if (shinfo->nr_frags)
|
2014-10-05 03:35:18 -06:00
|
|
|
memcpy(((void *)(inl + 1)) + hlen - spc,
|
2014-10-05 03:35:17 -06:00
|
|
|
fragptr,
|
|
|
|
skb_frag_size(&shinfo->frags[0]));
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
|
|
|
|
2015-04-08 19:49:36 -06:00
|
|
|
dma_wmb();
|
2008-10-22 16:47:49 -06:00
|
|
|
inl->byte_count = cpu_to_be32(1 << 31 | (skb->len - spc));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2014-01-10 01:18:26 -07:00
|
|
|
u16 mlx4_en_select_queue(struct net_device *dev, struct sk_buff *skb,
|
2019-03-20 04:02:06 -06:00
|
|
|
struct net_device *sb_dev)
|
2008-10-22 16:47:49 -06:00
|
|
|
{
|
2012-05-16 18:58:10 -06:00
|
|
|
struct mlx4_en_priv *priv = netdev_priv(dev);
|
2012-12-01 20:49:23 -07:00
|
|
|
u16 rings_p_up = priv->num_tx_rings_p_up;
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2017-01-29 09:56:16 -07:00
|
|
|
if (netdev_get_num_tc(dev))
|
2019-03-20 04:02:06 -06:00
|
|
|
return netdev_pick_tx(dev, skb, NULL);
|
2012-05-16 18:58:10 -06:00
|
|
|
|
2019-03-20 04:02:06 -06:00
|
|
|
return netdev_pick_tx(dev, skb, NULL) % rings_p_up;
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
|
|
|
|
2014-10-05 03:35:09 -06:00
|
|
|
static void mlx4_bf_copy(void __iomem *dst, const void *src,
|
|
|
|
unsigned int bytecnt)
|
2011-03-22 16:38:52 -06:00
|
|
|
{
|
|
|
|
__iowrite64_copy(dst, src, bytecnt / 8);
|
|
|
|
}
|
|
|
|
|
2016-07-19 13:16:54 -06:00
|
|
|
void mlx4_en_xmit_doorbell(struct mlx4_en_tx_ring *ring)
|
|
|
|
{
|
|
|
|
wmb();
|
|
|
|
/* Since there is no iowrite*_native() that writes the
|
|
|
|
* value as is, without byteswapping - using the one
|
|
|
|
* the doesn't do byteswapping in the relevant arch
|
|
|
|
* endianness.
|
|
|
|
*/
|
|
|
|
#if defined(__LITTLE_ENDIAN)
|
|
|
|
iowrite32(
|
|
|
|
#else
|
|
|
|
iowrite32be(
|
|
|
|
#endif
|
2017-10-09 07:59:50 -06:00
|
|
|
(__force u32)ring->doorbell_qpn,
|
2016-07-19 13:16:54 -06:00
|
|
|
ring->bf.uar->map + MLX4_SEND_DOORBELL);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void mlx4_en_tx_write_desc(struct mlx4_en_tx_ring *ring,
|
|
|
|
struct mlx4_en_tx_desc *tx_desc,
|
|
|
|
union mlx4_wqe_qpn_vlan qpn_vlan,
|
|
|
|
int desc_size, int bf_index,
|
|
|
|
__be32 op_own, bool bf_ok,
|
|
|
|
bool send_doorbell)
|
|
|
|
{
|
|
|
|
tx_desc->ctrl.qpn_vlan = qpn_vlan;
|
|
|
|
|
|
|
|
if (bf_ok) {
|
|
|
|
op_own |= htonl((bf_index & 0xffff) << 8);
|
|
|
|
/* Ensure new descriptor hits memory
|
|
|
|
* before setting ownership of this descriptor to HW
|
|
|
|
*/
|
|
|
|
dma_wmb();
|
|
|
|
tx_desc->ctrl.owner_opcode = op_own;
|
|
|
|
|
|
|
|
wmb();
|
|
|
|
|
|
|
|
mlx4_bf_copy(ring->bf.reg + ring->bf.offset, &tx_desc->ctrl,
|
|
|
|
desc_size);
|
|
|
|
|
|
|
|
wmb();
|
|
|
|
|
|
|
|
ring->bf.offset ^= ring->bf.buf_size;
|
|
|
|
} else {
|
|
|
|
/* Ensure new descriptor hits memory
|
|
|
|
* before setting ownership of this descriptor to HW
|
|
|
|
*/
|
|
|
|
dma_wmb();
|
|
|
|
tx_desc->ctrl.owner_opcode = op_own;
|
|
|
|
if (send_doorbell)
|
|
|
|
mlx4_en_xmit_doorbell(ring);
|
|
|
|
else
|
|
|
|
ring->xmit_more++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-06-15 05:35:35 -06:00
|
|
|
static bool mlx4_en_build_dma_wqe(struct mlx4_en_priv *priv,
|
|
|
|
struct skb_shared_info *shinfo,
|
|
|
|
struct mlx4_wqe_data_seg *data,
|
|
|
|
struct sk_buff *skb,
|
|
|
|
int lso_header_size,
|
|
|
|
__be32 mr_key,
|
|
|
|
struct mlx4_en_tx_info *tx_info)
|
|
|
|
{
|
|
|
|
struct device *ddev = priv->ddev;
|
|
|
|
dma_addr_t dma = 0;
|
|
|
|
u32 byte_count = 0;
|
|
|
|
int i_frag;
|
|
|
|
|
|
|
|
/* Map fragments if any */
|
|
|
|
for (i_frag = shinfo->nr_frags - 1; i_frag >= 0; i_frag--) {
|
2019-07-22 21:08:25 -06:00
|
|
|
const skb_frag_t *frag = &shinfo->frags[i_frag];
|
2017-06-15 05:35:35 -06:00
|
|
|
byte_count = skb_frag_size(frag);
|
|
|
|
dma = skb_frag_dma_map(ddev, frag,
|
|
|
|
0, byte_count,
|
|
|
|
DMA_TO_DEVICE);
|
|
|
|
if (dma_mapping_error(ddev, dma))
|
|
|
|
goto tx_drop_unmap;
|
|
|
|
|
|
|
|
data->addr = cpu_to_be64(dma);
|
|
|
|
data->lkey = mr_key;
|
|
|
|
dma_wmb();
|
|
|
|
data->byte_count = cpu_to_be32(byte_count);
|
|
|
|
--data;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Map linear part if needed */
|
|
|
|
if (tx_info->linear) {
|
|
|
|
byte_count = skb_headlen(skb) - lso_header_size;
|
|
|
|
|
|
|
|
dma = dma_map_single(ddev, skb->data +
|
|
|
|
lso_header_size, byte_count,
|
|
|
|
PCI_DMA_TODEVICE);
|
|
|
|
if (dma_mapping_error(ddev, dma))
|
|
|
|
goto tx_drop_unmap;
|
|
|
|
|
|
|
|
data->addr = cpu_to_be64(dma);
|
|
|
|
data->lkey = mr_key;
|
|
|
|
dma_wmb();
|
|
|
|
data->byte_count = cpu_to_be32(byte_count);
|
|
|
|
}
|
|
|
|
/* tx completion can avoid cache line miss for common cases */
|
|
|
|
tx_info->map0_dma = dma;
|
|
|
|
tx_info->map0_byte_count = byte_count;
|
|
|
|
|
|
|
|
return true;
|
|
|
|
|
|
|
|
tx_drop_unmap:
|
|
|
|
en_err(priv, "DMA mapping error\n");
|
|
|
|
|
|
|
|
while (++i_frag < shinfo->nr_frags) {
|
|
|
|
++data;
|
|
|
|
dma_unmap_page(ddev, (dma_addr_t)be64_to_cpu(data->addr),
|
|
|
|
be32_to_cpu(data->byte_count),
|
|
|
|
PCI_DMA_TODEVICE);
|
|
|
|
}
|
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2009-08-31 13:50:58 -06:00
|
|
|
netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev)
|
2008-10-22 16:47:49 -06:00
|
|
|
{
|
2014-10-05 03:35:17 -06:00
|
|
|
struct skb_shared_info *shinfo = skb_shinfo(skb);
|
2008-10-22 16:47:49 -06:00
|
|
|
struct mlx4_en_priv *priv = netdev_priv(dev);
|
2016-07-19 13:16:54 -06:00
|
|
|
union mlx4_wqe_qpn_vlan qpn_vlan = {};
|
2008-10-22 16:47:49 -06:00
|
|
|
struct mlx4_en_tx_ring *ring;
|
|
|
|
struct mlx4_en_tx_desc *tx_desc;
|
|
|
|
struct mlx4_wqe_data_seg *data;
|
|
|
|
struct mlx4_en_tx_info *tx_info;
|
2020-09-25 16:24:39 -06:00
|
|
|
u32 __maybe_unused ring_cons;
|
2017-06-15 05:35:35 -06:00
|
|
|
int tx_ind;
|
2008-10-22 16:47:49 -06:00
|
|
|
int nr_txbb;
|
|
|
|
int desc_size;
|
|
|
|
int real_size;
|
2011-03-22 16:38:52 -06:00
|
|
|
u32 index, bf_index;
|
2008-10-22 16:47:49 -06:00
|
|
|
__be32 op_own;
|
|
|
|
int lso_header_size;
|
2014-10-05 03:35:20 -06:00
|
|
|
void *fragptr = NULL;
|
2011-03-22 16:38:52 -06:00
|
|
|
bool bounce = false;
|
2014-09-25 08:17:49 -06:00
|
|
|
bool send_doorbell;
|
2014-10-06 10:30:35 -06:00
|
|
|
bool stop_queue;
|
2014-10-05 03:35:20 -06:00
|
|
|
bool inline_ok;
|
2017-06-15 05:35:35 -06:00
|
|
|
u8 data_offset;
|
2016-07-19 13:16:54 -06:00
|
|
|
bool bf_ok;
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2014-10-05 03:35:16 -06:00
|
|
|
tx_ind = skb_get_queue_mapping(skb);
|
2016-11-02 09:12:24 -06:00
|
|
|
ring = priv->tx_ring[TX][tx_ind];
|
2014-10-05 03:35:16 -06:00
|
|
|
|
2017-06-15 05:35:35 -06:00
|
|
|
if (unlikely(!priv->port_up))
|
2016-05-25 10:50:36 -06:00
|
|
|
goto tx_drop;
|
|
|
|
|
2014-10-05 03:35:16 -06:00
|
|
|
/* fetch ring->cons far ahead before needing it to avoid stall */
|
locking/atomics: COCCINELLE/treewide: Convert trivial ACCESS_ONCE() patterns to READ_ONCE()/WRITE_ONCE()
Please do not apply this to mainline directly, instead please re-run the
coccinelle script shown below and apply its output.
For several reasons, it is desirable to use {READ,WRITE}_ONCE() in
preference to ACCESS_ONCE(), and new code is expected to use one of the
former. So far, there's been no reason to change most existing uses of
ACCESS_ONCE(), as these aren't harmful, and changing them results in
churn.
However, for some features, the read/write distinction is critical to
correct operation. To distinguish these cases, separate read/write
accessors must be used. This patch migrates (most) remaining
ACCESS_ONCE() instances to {READ,WRITE}_ONCE(), using the following
coccinelle script:
----
// Convert trivial ACCESS_ONCE() uses to equivalent READ_ONCE() and
// WRITE_ONCE()
// $ make coccicheck COCCI=/home/mark/once.cocci SPFLAGS="--include-headers" MODE=patch
virtual patch
@ depends on patch @
expression E1, E2;
@@
- ACCESS_ONCE(E1) = E2
+ WRITE_ONCE(E1, E2)
@ depends on patch @
expression E;
@@
- ACCESS_ONCE(E)
+ READ_ONCE(E)
----
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: davem@davemloft.net
Cc: linux-arch@vger.kernel.org
Cc: mpe@ellerman.id.au
Cc: shuah@kernel.org
Cc: snitzer@redhat.com
Cc: thor.thayer@linux.intel.com
Cc: tj@kernel.org
Cc: viro@zeniv.linux.org.uk
Cc: will.deacon@arm.com
Link: http://lkml.kernel.org/r/1508792849-3115-19-git-send-email-paulmck@linux.vnet.ibm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-10-23 15:07:29 -06:00
|
|
|
ring_cons = READ_ONCE(ring->cons);
|
2014-10-05 03:35:16 -06:00
|
|
|
|
2014-10-05 03:35:20 -06:00
|
|
|
real_size = get_real_size(skb, shinfo, dev, &lso_header_size,
|
|
|
|
&inline_ok, &fragptr);
|
2008-10-22 16:47:49 -06:00
|
|
|
if (unlikely(!real_size))
|
2016-09-11 01:56:20 -06:00
|
|
|
goto tx_drop_count;
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2011-03-30 19:57:33 -06:00
|
|
|
/* Align descriptor to TXBB size */
|
2008-10-22 16:47:49 -06:00
|
|
|
desc_size = ALIGN(real_size, TXBB_SIZE);
|
2017-06-15 05:35:39 -06:00
|
|
|
nr_txbb = desc_size >> LOG_TXBB_SIZE;
|
2008-10-22 16:47:49 -06:00
|
|
|
if (unlikely(nr_txbb > MAX_DESC_TXBBS)) {
|
|
|
|
if (netif_msg_tx_err(priv))
|
2009-06-01 14:27:13 -06:00
|
|
|
en_warn(priv, "Oversized header or SG list\n");
|
2016-09-11 01:56:20 -06:00
|
|
|
goto tx_drop_count;
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
|
|
|
|
2016-07-19 13:16:54 -06:00
|
|
|
bf_ok = ring->bf_enabled;
|
2015-07-27 05:46:34 -06:00
|
|
|
if (skb_vlan_tag_present(skb)) {
|
2017-06-15 05:35:35 -06:00
|
|
|
u16 vlan_proto;
|
|
|
|
|
2016-07-19 13:16:54 -06:00
|
|
|
qpn_vlan.vlan_tag = cpu_to_be16(skb_vlan_tag_get(skb));
|
2015-07-27 05:46:34 -06:00
|
|
|
vlan_proto = be16_to_cpu(skb->vlan_proto);
|
2016-07-19 13:16:54 -06:00
|
|
|
if (vlan_proto == ETH_P_8021AD)
|
|
|
|
qpn_vlan.ins_vlan = MLX4_WQE_CTRL_INS_SVLAN;
|
|
|
|
else if (vlan_proto == ETH_P_8021Q)
|
|
|
|
qpn_vlan.ins_vlan = MLX4_WQE_CTRL_INS_CVLAN;
|
|
|
|
else
|
|
|
|
qpn_vlan.ins_vlan = 0;
|
|
|
|
bf_ok = false;
|
2015-07-27 05:46:34 -06:00
|
|
|
}
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2014-10-08 09:19:27 -06:00
|
|
|
netdev_txq_bql_enqueue_prefetchw(ring->tx_queue);
|
2014-10-05 03:35:14 -06:00
|
|
|
|
2008-10-22 16:47:49 -06:00
|
|
|
/* Track current inflight packets for performance analysis */
|
|
|
|
AVG_PERF_COUNTER(priv->pstats.inflight_avg,
|
2014-10-05 03:35:16 -06:00
|
|
|
(u32)(ring->prod - ring_cons - 1));
|
2008-10-22 16:47:49 -06:00
|
|
|
|
|
|
|
/* Packet is good - grab an index and transmit it */
|
|
|
|
index = ring->prod & ring->size_mask;
|
2011-03-22 16:38:52 -06:00
|
|
|
bf_index = ring->prod;
|
2008-10-22 16:47:49 -06:00
|
|
|
|
|
|
|
/* See if we have enough space for whole descriptor TXBB for setting
|
|
|
|
* SW ownership on next descriptor; if not, use a bounce buffer. */
|
|
|
|
if (likely(index + nr_txbb <= ring->size))
|
2017-06-15 05:35:39 -06:00
|
|
|
tx_desc = ring->buf + (index << LOG_TXBB_SIZE);
|
2011-03-22 16:38:52 -06:00
|
|
|
else {
|
2008-10-22 16:47:49 -06:00
|
|
|
tx_desc = (struct mlx4_en_tx_desc *) ring->bounce_buf;
|
2011-03-22 16:38:52 -06:00
|
|
|
bounce = true;
|
2016-07-19 13:16:54 -06:00
|
|
|
bf_ok = false;
|
2011-03-22 16:38:52 -06:00
|
|
|
}
|
2008-10-22 16:47:49 -06:00
|
|
|
|
|
|
|
/* Save skb in tx_info ring */
|
|
|
|
tx_info = &ring->tx_info[index];
|
|
|
|
tx_info->skb = skb;
|
|
|
|
tx_info->nr_txbb = nr_txbb;
|
|
|
|
|
2017-06-15 05:35:35 -06:00
|
|
|
if (!lso_header_size) {
|
|
|
|
data = &tx_desc->data;
|
|
|
|
data_offset = offsetof(struct mlx4_en_tx_desc, data);
|
|
|
|
} else {
|
|
|
|
int lso_align = ALIGN(lso_header_size + 4, DS_SIZE);
|
|
|
|
|
|
|
|
data = (void *)&tx_desc->lso + lso_align;
|
|
|
|
data_offset = offsetof(struct mlx4_en_tx_desc, lso) + lso_align;
|
|
|
|
}
|
2013-08-21 01:08:58 -06:00
|
|
|
|
|
|
|
/* valid only for none inline segments */
|
2017-06-15 05:35:35 -06:00
|
|
|
tx_info->data_offset = data_offset;
|
2013-08-21 01:08:58 -06:00
|
|
|
|
2014-10-05 03:35:20 -06:00
|
|
|
tx_info->inl = inline_ok;
|
|
|
|
|
2017-06-15 05:35:35 -06:00
|
|
|
tx_info->linear = lso_header_size < skb_headlen(skb) && !inline_ok;
|
2013-08-21 01:08:58 -06:00
|
|
|
|
2014-10-05 03:35:17 -06:00
|
|
|
tx_info->nr_maps = shinfo->nr_frags + tx_info->linear;
|
2014-10-05 03:35:13 -06:00
|
|
|
data += tx_info->nr_maps - 1;
|
2013-08-21 01:08:58 -06:00
|
|
|
|
2017-06-15 05:35:35 -06:00
|
|
|
if (!tx_info->inl)
|
|
|
|
if (!mlx4_en_build_dma_wqe(priv, shinfo, data, skb,
|
|
|
|
lso_header_size, ring->mr_key,
|
|
|
|
tx_info))
|
|
|
|
goto tx_drop_count;
|
2013-08-21 01:08:58 -06:00
|
|
|
|
2013-04-23 00:06:49 -06:00
|
|
|
/*
|
|
|
|
* For timestamping add flag to skb_shinfo and
|
|
|
|
* set flag for further reference
|
|
|
|
*/
|
2014-10-05 03:35:19 -06:00
|
|
|
tx_info->ts_requested = 0;
|
2014-10-05 03:35:09 -06:00
|
|
|
if (unlikely(ring->hwtstamp_tx_type == HWTSTAMP_TX_ON &&
|
|
|
|
shinfo->tx_flags & SKBTX_HW_TSTAMP)) {
|
|
|
|
shinfo->tx_flags |= SKBTX_IN_PROGRESS;
|
2013-04-23 00:06:49 -06:00
|
|
|
tx_info->ts_requested = 1;
|
|
|
|
}
|
|
|
|
|
2008-10-22 16:47:49 -06:00
|
|
|
/* Prepare ctrl segement apart opcode+ownership, which depends on
|
|
|
|
* whether LSO is used */
|
2011-11-26 12:55:19 -07:00
|
|
|
tx_desc->ctrl.srcrb_flags = priv->ctrl_flags;
|
2008-10-22 16:47:49 -06:00
|
|
|
if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
|
2014-10-30 07:59:27 -06:00
|
|
|
if (!skb->encapsulation)
|
|
|
|
tx_desc->ctrl.srcrb_flags |= cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM |
|
|
|
|
MLX4_WQE_CTRL_TCP_UDP_CSUM);
|
|
|
|
else
|
|
|
|
tx_desc->ctrl.srcrb_flags |= cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM);
|
2011-10-17 19:50:56 -06:00
|
|
|
ring->tx_csum++;
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
|
|
|
|
2013-02-06 19:25:19 -07:00
|
|
|
if (priv->flags & MLX4_EN_FLAG_ENABLE_HW_LOOPBACK) {
|
2013-08-21 01:08:59 -06:00
|
|
|
struct ethhdr *ethh;
|
|
|
|
|
2013-01-16 22:30:42 -07:00
|
|
|
/* Copy dst mac address to wqe. This allows loopback in eSwitch,
|
|
|
|
* so that VFs and PF can communicate with each other
|
|
|
|
*/
|
|
|
|
ethh = (struct ethhdr *)skb->data;
|
|
|
|
tx_desc->ctrl.srcrb_flags16[0] = get_unaligned((__be16 *)ethh->h_dest);
|
|
|
|
tx_desc->ctrl.imm = get_unaligned((__be32 *)(ethh->h_dest + 2));
|
|
|
|
}
|
|
|
|
|
2008-10-22 16:47:49 -06:00
|
|
|
/* Handle LSO (TSO) packets */
|
|
|
|
if (lso_header_size) {
|
2014-10-05 03:35:17 -06:00
|
|
|
int i;
|
|
|
|
|
2008-10-22 16:47:49 -06:00
|
|
|
/* Mark opcode as LSO */
|
|
|
|
op_own = cpu_to_be32(MLX4_OPCODE_LSO | (1 << 6)) |
|
|
|
|
((ring->prod & ring->size) ?
|
|
|
|
cpu_to_be32(MLX4_EN_BIT_DESC_OWN) : 0);
|
|
|
|
|
|
|
|
/* Fill in the LSO prefix */
|
|
|
|
tx_desc->lso.mss_hdr_size = cpu_to_be32(
|
2014-10-05 03:35:17 -06:00
|
|
|
shinfo->gso_size << 16 | lso_header_size);
|
2008-10-22 16:47:49 -06:00
|
|
|
|
|
|
|
/* Copy headers;
|
|
|
|
* note that we already verified that it is linear */
|
|
|
|
memcpy(tx_desc->lso.header, skb->data, lso_header_size);
|
|
|
|
|
2014-10-02 09:24:21 -06:00
|
|
|
ring->tso_packets++;
|
2014-10-05 03:35:17 -06:00
|
|
|
|
2017-04-05 09:49:02 -06:00
|
|
|
i = shinfo->gso_segs;
|
2012-04-22 20:18:50 -06:00
|
|
|
tx_info->nr_bytes = skb->len + (i - 1) * lso_header_size;
|
2008-10-22 16:47:49 -06:00
|
|
|
ring->packets += i;
|
|
|
|
} else {
|
|
|
|
/* Normal (Non LSO) packet */
|
|
|
|
op_own = cpu_to_be32(MLX4_OPCODE_SEND) |
|
|
|
|
((ring->prod & ring->size) ?
|
|
|
|
cpu_to_be32(MLX4_EN_BIT_DESC_OWN) : 0);
|
2012-04-22 20:18:50 -06:00
|
|
|
tx_info->nr_bytes = max_t(unsigned int, skb->len, ETH_ZLEN);
|
2008-10-22 16:47:49 -06:00
|
|
|
ring->packets++;
|
|
|
|
}
|
2012-04-22 20:18:50 -06:00
|
|
|
ring->bytes += tx_info->nr_bytes;
|
2008-10-22 16:47:49 -06:00
|
|
|
AVG_PERF_COUNTER(priv->pstats.tx_pktsz_avg, skb->len);
|
|
|
|
|
2014-10-05 03:35:20 -06:00
|
|
|
if (tx_info->inl)
|
2016-07-19 13:16:54 -06:00
|
|
|
build_inline_wqe(tx_desc, skb, shinfo, fragptr);
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2013-12-23 07:09:44 -07:00
|
|
|
if (skb->encapsulation) {
|
2016-05-02 10:38:37 -06:00
|
|
|
union {
|
|
|
|
struct iphdr *v4;
|
|
|
|
struct ipv6hdr *v6;
|
|
|
|
unsigned char *hdr;
|
|
|
|
} ip;
|
|
|
|
u8 proto;
|
|
|
|
|
|
|
|
ip.hdr = skb_inner_network_header(skb);
|
|
|
|
proto = (ip.v4->version == 4) ? ip.v4->protocol :
|
|
|
|
ip.v6->nexthdr;
|
|
|
|
|
|
|
|
if (proto == IPPROTO_TCP || proto == IPPROTO_UDP)
|
2013-12-23 07:09:44 -07:00
|
|
|
op_own |= cpu_to_be32(MLX4_WQE_CTRL_IIP | MLX4_WQE_CTRL_ILP);
|
|
|
|
else
|
|
|
|
op_own |= cpu_to_be32(MLX4_WQE_CTRL_IIP);
|
|
|
|
}
|
|
|
|
|
2008-10-22 16:47:49 -06:00
|
|
|
ring->prod += nr_txbb;
|
|
|
|
|
|
|
|
/* If we used a bounce buffer then copy descriptor back into place */
|
2014-10-05 03:35:09 -06:00
|
|
|
if (unlikely(bounce))
|
2008-10-22 16:47:49 -06:00
|
|
|
tx_desc = mlx4_en_bounce_to_desc(priv, ring, index, desc_size);
|
|
|
|
|
2013-04-23 00:06:50 -06:00
|
|
|
skb_tx_timestamp(skb);
|
|
|
|
|
2014-10-06 10:30:35 -06:00
|
|
|
/* Check available TXBBs And 2K spare for prefetch */
|
2015-06-25 02:29:42 -06:00
|
|
|
stop_queue = mlx4_en_is_tx_ring_full(ring);
|
2014-10-06 10:30:35 -06:00
|
|
|
if (unlikely(stop_queue)) {
|
|
|
|
netif_tx_stop_queue(ring->tx_queue);
|
|
|
|
ring->queue_stopped++;
|
|
|
|
}
|
2018-10-31 09:39:14 -06:00
|
|
|
|
|
|
|
send_doorbell = __netdev_tx_sent_queue(ring->tx_queue,
|
|
|
|
tx_info->nr_bytes,
|
2019-04-01 08:42:15 -06:00
|
|
|
netdev_xmit_more());
|
2014-09-25 08:17:49 -06:00
|
|
|
|
2014-10-05 03:35:11 -06:00
|
|
|
real_size = (real_size / 16) & 0x3f;
|
|
|
|
|
2016-07-19 13:16:54 -06:00
|
|
|
bf_ok &= desc_size <= MAX_BF && send_doorbell;
|
2015-07-27 05:46:34 -06:00
|
|
|
|
2016-07-19 13:16:54 -06:00
|
|
|
if (bf_ok)
|
|
|
|
qpn_vlan.bf_qpn = ring->doorbell_qpn | cpu_to_be32(real_size);
|
|
|
|
else
|
|
|
|
qpn_vlan.fence_size = real_size;
|
2014-10-05 03:35:09 -06:00
|
|
|
|
2016-07-19 13:16:54 -06:00
|
|
|
mlx4_en_tx_write_desc(ring, tx_desc, qpn_vlan, desc_size, bf_index,
|
|
|
|
op_own, bf_ok, send_doorbell);
|
2008-10-22 16:47:49 -06:00
|
|
|
|
2014-10-06 10:30:35 -06:00
|
|
|
if (unlikely(stop_queue)) {
|
|
|
|
/* If queue was emptied after the if (stop_queue) , and before
|
|
|
|
* the netif_tx_stop_queue() - need to wake the queue,
|
|
|
|
* or else it will remain stopped forever.
|
|
|
|
* Need a memory barrier to make sure ring->cons was not
|
|
|
|
* updated before queue was stopped.
|
|
|
|
*/
|
|
|
|
smp_rmb();
|
|
|
|
|
2015-06-25 02:29:42 -06:00
|
|
|
if (unlikely(!mlx4_en_is_tx_ring_full(ring))) {
|
2014-10-06 10:30:35 -06:00
|
|
|
netif_tx_wake_queue(ring->tx_queue);
|
|
|
|
ring->wake_queue++;
|
|
|
|
}
|
|
|
|
}
|
2009-07-05 20:23:38 -06:00
|
|
|
return NETDEV_TX_OK;
|
2009-06-20 16:15:31 -06:00
|
|
|
|
2016-09-11 01:56:20 -06:00
|
|
|
tx_drop_count:
|
|
|
|
ring->tx_dropped++;
|
2009-06-20 16:15:31 -06:00
|
|
|
tx_drop:
|
|
|
|
dev_kfree_skb_any(skb);
|
|
|
|
return NETDEV_TX_OK;
|
2008-10-22 16:47:49 -06:00
|
|
|
}
|
|
|
|
|
2017-06-15 05:35:36 -06:00
|
|
|
#define MLX4_EN_XDP_TX_NRTXBB 1
|
|
|
|
#define MLX4_EN_XDP_TX_REAL_SZ (((CTRL_SIZE + MLX4_EN_XDP_TX_NRTXBB * DS_SIZE) \
|
|
|
|
/ 16) & 0x3f)
|
|
|
|
|
2017-10-11 04:17:27 -06:00
|
|
|
void mlx4_en_init_tx_xdp_ring_descs(struct mlx4_en_priv *priv,
|
|
|
|
struct mlx4_en_tx_ring *ring)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < ring->size; i++) {
|
|
|
|
struct mlx4_en_tx_info *tx_info = &ring->tx_info[i];
|
|
|
|
struct mlx4_en_tx_desc *tx_desc = ring->buf +
|
|
|
|
(i << LOG_TXBB_SIZE);
|
|
|
|
|
|
|
|
tx_info->map0_byte_count = PAGE_SIZE;
|
|
|
|
tx_info->nr_txbb = MLX4_EN_XDP_TX_NRTXBB;
|
|
|
|
tx_info->data_offset = offsetof(struct mlx4_en_tx_desc, data);
|
|
|
|
tx_info->ts_requested = 0;
|
|
|
|
tx_info->nr_maps = 1;
|
|
|
|
tx_info->linear = 1;
|
|
|
|
tx_info->inl = 0;
|
|
|
|
|
|
|
|
tx_desc->data.lkey = ring->mr_key;
|
|
|
|
tx_desc->ctrl.qpn_vlan.fence_size = MLX4_EN_XDP_TX_REAL_SZ;
|
|
|
|
tx_desc->ctrl.srcrb_flags = priv->ctrl_flags;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-11-02 09:12:25 -06:00
|
|
|
netdev_tx_t mlx4_en_xmit_frame(struct mlx4_en_rx_ring *rx_ring,
|
|
|
|
struct mlx4_en_rx_alloc *frame,
|
2017-10-11 04:17:25 -06:00
|
|
|
struct mlx4_en_priv *priv, unsigned int length,
|
2017-06-15 05:35:36 -06:00
|
|
|
int tx_ind, bool *doorbell_pending)
|
2016-07-19 13:16:55 -06:00
|
|
|
{
|
|
|
|
struct mlx4_en_tx_desc *tx_desc;
|
|
|
|
struct mlx4_en_tx_info *tx_info;
|
2017-06-15 05:35:36 -06:00
|
|
|
struct mlx4_wqe_data_seg *data;
|
|
|
|
struct mlx4_en_tx_ring *ring;
|
2016-07-19 13:16:55 -06:00
|
|
|
dma_addr_t dma;
|
|
|
|
__be32 op_own;
|
2017-06-15 05:35:36 -06:00
|
|
|
int index;
|
2016-07-19 13:16:55 -06:00
|
|
|
|
2017-06-15 05:35:36 -06:00
|
|
|
if (unlikely(!priv->port_up))
|
|
|
|
goto tx_drop;
|
2016-07-19 13:16:55 -06:00
|
|
|
|
2016-11-02 09:12:24 -06:00
|
|
|
ring = priv->tx_ring[TX_XDP][tx_ind];
|
2016-07-19 13:16:55 -06:00
|
|
|
|
2017-06-15 05:35:36 -06:00
|
|
|
if (unlikely(mlx4_en_is_tx_ring_full(ring)))
|
2016-09-11 01:56:20 -06:00
|
|
|
goto tx_drop_count;
|
2016-07-19 13:16:55 -06:00
|
|
|
|
|
|
|
index = ring->prod & ring->size_mask;
|
|
|
|
tx_info = &ring->tx_info[index];
|
|
|
|
|
|
|
|
/* Track current inflight packets for performance analysis */
|
|
|
|
AVG_PERF_COUNTER(priv->pstats.inflight_avg,
|
2017-06-15 05:35:36 -06:00
|
|
|
(u32)(ring->prod - READ_ONCE(ring->cons) - 1));
|
2016-07-19 13:16:55 -06:00
|
|
|
|
2017-06-15 05:35:39 -06:00
|
|
|
tx_desc = ring->buf + (index << LOG_TXBB_SIZE);
|
2016-07-19 13:16:55 -06:00
|
|
|
data = &tx_desc->data;
|
|
|
|
|
|
|
|
dma = frame->dma;
|
|
|
|
|
|
|
|
tx_info->page = frame->page;
|
|
|
|
frame->page = NULL;
|
|
|
|
tx_info->map0_dma = dma;
|
|
|
|
tx_info->nr_bytes = max_t(unsigned int, length, ETH_ZLEN);
|
|
|
|
|
2016-12-07 16:53:13 -07:00
|
|
|
dma_sync_single_range_for_device(priv->ddev, dma, frame->page_offset,
|
|
|
|
length, PCI_DMA_TODEVICE);
|
2016-07-19 13:16:55 -06:00
|
|
|
|
2016-12-07 16:53:13 -07:00
|
|
|
data->addr = cpu_to_be64(dma + frame->page_offset);
|
2016-07-19 13:16:55 -06:00
|
|
|
dma_wmb();
|
|
|
|
data->byte_count = cpu_to_be32(length);
|
|
|
|
|
|
|
|
/* tx completion can avoid cache line miss for common cases */
|
|
|
|
|
|
|
|
op_own = cpu_to_be32(MLX4_OPCODE_SEND) |
|
|
|
|
((ring->prod & ring->size) ?
|
|
|
|
cpu_to_be32(MLX4_EN_BIT_DESC_OWN) : 0);
|
|
|
|
|
2016-11-02 09:12:25 -06:00
|
|
|
rx_ring->xdp_tx++;
|
2016-07-19 13:16:55 -06:00
|
|
|
AVG_PERF_COUNTER(priv->pstats.tx_pktsz_avg, length);
|
|
|
|
|
2017-06-15 05:35:36 -06:00
|
|
|
ring->prod += MLX4_EN_XDP_TX_NRTXBB;
|
2016-07-19 13:16:55 -06:00
|
|
|
|
2017-10-11 04:17:26 -06:00
|
|
|
/* Ensure new descriptor hits memory
|
|
|
|
* before setting ownership of this descriptor to HW
|
|
|
|
*/
|
|
|
|
dma_wmb();
|
|
|
|
tx_desc->ctrl.owner_opcode = op_own;
|
|
|
|
ring->xmit_more++;
|
2016-07-19 13:16:55 -06:00
|
|
|
|
2017-06-15 05:35:36 -06:00
|
|
|
*doorbell_pending = true;
|
2016-07-19 13:16:55 -06:00
|
|
|
|
|
|
|
return NETDEV_TX_OK;
|
|
|
|
|
2016-09-11 01:56:20 -06:00
|
|
|
tx_drop_count:
|
2016-11-02 09:12:25 -06:00
|
|
|
rx_ring->xdp_tx_full++;
|
2017-06-15 05:35:37 -06:00
|
|
|
*doorbell_pending = true;
|
2016-09-11 01:56:20 -06:00
|
|
|
tx_drop:
|
2016-07-19 13:16:55 -06:00
|
|
|
return NETDEV_TX_BUSY;
|
|
|
|
}
|