License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 08:07:57 -06:00
|
|
|
// SPDX-License-Identifier: GPL-2.0
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* INET An implementation of the TCP/IP protocol suite for the LINUX
|
|
|
|
* operating system. INET is implemented using the BSD Socket
|
|
|
|
* interface as the means of communication with the user level.
|
|
|
|
*
|
|
|
|
* Implementation of the Transmission Control Protocol(TCP).
|
|
|
|
*
|
2005-05-05 17:16:16 -06:00
|
|
|
* Authors: Ross Biro
|
2005-04-16 16:20:36 -06:00
|
|
|
* Fred N. van Kempen, <waltje@uWalt.NL.Mugnet.ORG>
|
|
|
|
* Mark Evans, <evansmp@uhura.aston.ac.uk>
|
|
|
|
* Corey Minyard <wf-rch!minyard@relay.EU.net>
|
|
|
|
* Florian La Roche, <flla@stud.uni-sb.de>
|
|
|
|
* Charles Hedrick, <hedrick@klinzhai.rutgers.edu>
|
|
|
|
* Linus Torvalds, <torvalds@cs.helsinki.fi>
|
|
|
|
* Alan Cox, <gw4pts@gw4pts.ampr.org>
|
|
|
|
* Matthew Dillon, <dillon@apollo.west.oic.com>
|
|
|
|
* Arnt Gulbrandsen, <agulbra@nvg.unit.no>
|
|
|
|
* Jorge Cwik, <jorge@laser.satlink.net>
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Changes:
|
|
|
|
* Pedro Roque : Fast Retransmit/Recovery.
|
|
|
|
* Two receive queues.
|
|
|
|
* Retransmit queue handled by TCP.
|
|
|
|
* Better retransmit timer handling.
|
|
|
|
* New congestion avoidance.
|
|
|
|
* Header prediction.
|
|
|
|
* Variable renaming.
|
|
|
|
*
|
|
|
|
* Eric : Fast Retransmit.
|
|
|
|
* Randy Scott : MSS option defines.
|
|
|
|
* Eric Schenk : Fixes to slow start algorithm.
|
|
|
|
* Eric Schenk : Yet another double ACK bug.
|
|
|
|
* Eric Schenk : Delayed ACK bug fixes.
|
|
|
|
* Eric Schenk : Floyd style fast retrans war avoidance.
|
|
|
|
* David S. Miller : Don't allow zero congestion window.
|
|
|
|
* Eric Schenk : Fix retransmitter so that it sends
|
|
|
|
* next packet on ack of previous packet.
|
|
|
|
* Andi Kleen : Moved open_request checking here
|
|
|
|
* and process RSTs for open_requests.
|
|
|
|
* Andi Kleen : Better prune_queue, and other fixes.
|
2005-11-10 18:13:47 -07:00
|
|
|
* Andrey Savochkin: Fix RTT measurements in the presence of
|
2005-04-16 16:20:36 -06:00
|
|
|
* timestamps.
|
|
|
|
* Andrey Savochkin: Check sequence numbers correctly when
|
|
|
|
* removing SACKs due to in sequence incoming
|
|
|
|
* data segments.
|
|
|
|
* Andi Kleen: Make sure we never ack data there is not
|
|
|
|
* enough room for. Also make this condition
|
|
|
|
* a fatal error if it might still happen.
|
2007-02-09 07:24:47 -07:00
|
|
|
* Andi Kleen: Add tcp_measure_rcv_mss to make
|
2005-04-16 16:20:36 -06:00
|
|
|
* connections with MSS<min(MTU,ann. MSS)
|
2007-02-09 07:24:47 -07:00
|
|
|
* work without delayed acks.
|
2005-04-16 16:20:36 -06:00
|
|
|
* Andi Kleen: Process packets with PSH set in the
|
|
|
|
* fast path.
|
|
|
|
* J Hadi Salim: ECN support
|
|
|
|
* Andrei Gurtov,
|
|
|
|
* Pasi Sarolahti,
|
|
|
|
* Panu Kuhlberg: Experimental audit of TCP (re)transmission
|
|
|
|
* engine. Lots of bugs are found.
|
|
|
|
* Pasi Sarolahti: F-RTO for dealing with spurious RTOs
|
|
|
|
*/
|
|
|
|
|
2012-03-12 01:03:32 -06:00
|
|
|
#define pr_fmt(fmt) "TCP: " fmt
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
#include <linux/mm.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 02:04:11 -06:00
|
|
|
#include <linux/slab.h>
|
2005-04-16 16:20:36 -06:00
|
|
|
#include <linux/module.h>
|
|
|
|
#include <linux/sysctl.h>
|
2009-03-21 14:36:17 -06:00
|
|
|
#include <linux/kernel.h>
|
2014-10-11 16:17:29 -06:00
|
|
|
#include <linux/prefetch.h>
|
2008-05-04 23:14:42 -06:00
|
|
|
#include <net/dst.h>
|
2005-04-16 16:20:36 -06:00
|
|
|
#include <net/tcp.h>
|
|
|
|
#include <net/inet_common.h>
|
|
|
|
#include <linux/ipsec.h>
|
|
|
|
#include <asm/unaligned.h>
|
2014-08-04 20:11:50 -06:00
|
|
|
#include <linux/errqueue.h>
|
2017-10-23 10:20:25 -06:00
|
|
|
#include <trace/events/tcp.h>
|
2019-05-08 17:46:14 -06:00
|
|
|
#include <linux/jump_label_ratelimit.h>
|
2018-06-29 22:26:57 -06:00
|
|
|
#include <net/busy_poll.h>
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-09-22 15:15:41 -06:00
|
|
|
int sysctl_tcp_max_orphans __read_mostly = NR_FILE;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
#define FLAG_DATA 0x01 /* Incoming frame contained data. */
|
|
|
|
#define FLAG_WIN_UPDATE 0x02 /* Incoming ACK was a window update. */
|
|
|
|
#define FLAG_DATA_ACKED 0x04 /* This ACK acknowledged new data. */
|
|
|
|
#define FLAG_RETRANS_DATA_ACKED 0x08 /* "" "" some of which was retransmitted. */
|
|
|
|
#define FLAG_SYN_ACKED 0x10 /* This ACK acknowledged SYN. */
|
|
|
|
#define FLAG_DATA_SACKED 0x20 /* New SACK. */
|
|
|
|
#define FLAG_ECE 0x40 /* ECE in this ACK */
|
2015-07-01 15:11:14 -06:00
|
|
|
#define FLAG_LOST_RETRANS 0x80 /* This ACK marks some retransmission lost */
|
2017-08-30 11:24:58 -06:00
|
|
|
#define FLAG_SLOWPATH 0x100 /* Do not skip RFC checks for window update.*/
|
2013-03-20 07:33:00 -06:00
|
|
|
#define FLAG_ORIG_SACK_ACKED 0x200 /* Never retransmitted data are (s)acked */
|
2007-08-02 20:46:58 -06:00
|
|
|
#define FLAG_SND_UNA_ADVANCED 0x400 /* Snd_una was changed (!= FLAG_DATA_ACKED) */
|
2007-10-26 00:03:52 -06:00
|
|
|
#define FLAG_DSACKING_ACK 0x800 /* SACK blocks contained D-SACK info */
|
tcp: fix xmit timer to only be reset if data ACKed/SACKed
Fix a TCP loss recovery performance bug raised recently on the netdev
list, in two threads:
(i) July 26, 2017: netdev thread "TCP fast retransmit issues"
(ii) July 26, 2017: netdev thread:
"[PATCH V2 net-next] TLP: Don't reschedule PTO when there's one
outstanding TLP retransmission"
The basic problem is that incoming TCP packets that did not indicate
forward progress could cause the xmit timer (TLP or RTO) to be rearmed
and pushed back in time. In certain corner cases this could result in
the following problems noted in these threads:
- Repeated ACKs coming in with bogus SACKs corrupted by middleboxes
could cause TCP to repeatedly schedule TLPs forever. We kept
sending TLPs after every ~200ms, which elicited bogus SACKs, which
caused more TLPs, ad infinitum; we never fired an RTO to fill in
the holes.
- Incoming data segments could, in some cases, cause us to reschedule
our RTO or TLP timer further out in time, for no good reason. This
could cause repeated inbound data to result in stalls in outbound
data, in the presence of packet loss.
This commit fixes these bugs by changing the TLP and RTO ACK
processing to:
(a) Only reschedule the xmit timer once per ACK.
(b) Only reschedule the xmit timer if tcp_clean_rtx_queue() deems the
ACK indicates sufficient forward progress (a packet was
cumulatively ACKed, or we got a SACK for a packet that was sent
before the most recent retransmit of the write queue head).
This brings us back into closer compliance with the RFCs, since, as
the comment for tcp_rearm_rto() notes, we should only restart the RTO
timer after forward progress on the connection. Previously we were
restarting the xmit timer even in these cases where there was no
forward progress.
As a side benefit, this commit simplifies and speeds up the TCP timer
arming logic. We had been calling inet_csk_reset_xmit_timer() three
times on normal ACKs that cumulatively acknowledged some data:
1) Once near the top of tcp_ack() to switch from TLP timer to RTO:
if (icsk->icsk_pending == ICSK_TIME_LOSS_PROBE)
tcp_rearm_rto(sk);
2) Once in tcp_clean_rtx_queue(), to update the RTO:
if (flag & FLAG_ACKED) {
tcp_rearm_rto(sk);
3) Once in tcp_ack() after tcp_fastretrans_alert() to switch from RTO
to TLP:
if (icsk->icsk_pending == ICSK_TIME_RETRANS)
tcp_schedule_loss_probe(sk);
This commit, by only rescheduling the xmit timer once per ACK,
simplifies the code and reduces CPU overhead.
This commit was tested in an A/B test with Google web server
traffic. SNMP stats and request latency metrics were within noise
levels, substantiating that for normal web traffic patterns this is a
rare issue. This commit was also tested with packetdrill tests to
verify that it fixes the timer behavior in the corner cases discussed
in the netdev threads mentioned above.
This patch is a bug fix patch intended to be queued for -stable
relases.
Fixes: 6ba8a3b19e76 ("tcp: Tail loss probe (TLP)")
Reported-by: Klavs Klavsen <kl@vsen.dk>
Reported-by: Mao Wenan <maowenan@huawei.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-03 07:19:54 -06:00
|
|
|
#define FLAG_SET_XMIT_TIMER 0x1000 /* Set TLP or RTO timer */
|
2007-12-31 05:49:21 -07:00
|
|
|
#define FLAG_SACK_RENEGING 0x2000 /* snd_una advanced to a sacked seq */
|
tcp: call tcp_replace_ts_recent() from tcp_ack()
commit bd090dfc634d (tcp: tcp_replace_ts_recent() should not be called
from tcp_validate_incoming()) introduced a TS ecr bug in slow path
processing.
1 A > B P. 1:10001(10000) ack 1 <nop,nop,TS val 1001 ecr 200>
2 B < A . 1:1(0) ack 1 win 257 <sack 9001:10001,TS val 300 ecr 1001>
3 A > B . 1:1001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
4 A > B . 1001:2001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
(ecr 200 should be ecr 300 in packets 3 & 4)
Problem is tcp_ack() can trigger send of new packets (retransmits),
reflecting the prior TSval, instead of the TSval contained in the
currently processed incoming packet.
Fix this by calling tcp_replace_ts_recent() from tcp_ack() after the
checks, but before the actions.
Reported-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-04-19 01:19:48 -06:00
|
|
|
#define FLAG_UPDATE_TS_RECENT 0x4000 /* tcp_replace_ts_recent() */
|
tcp: better validation of received ack sequences
Paul Fiterau Brostean reported :
<quote>
Linux TCP stack we analyze exhibits behavior that seems odd to me.
The scenario is as follows (all packets have empty payloads, no window
scaling, rcv/snd window size should not be a factor):
TEST HARNESS (CLIENT) LINUX SERVER
1. - LISTEN (server listen,
then accepts)
2. - --> <SEQ=100><CTL=SYN> --> SYN-RECEIVED
3. - <-- <SEQ=300><ACK=101><CTL=SYN,ACK> <-- SYN-RECEIVED
4. - --> <SEQ=101><ACK=301><CTL=ACK> --> ESTABLISHED
5. - <-- <SEQ=301><ACK=101><CTL=FIN,ACK> <-- FIN WAIT-1 (server
opts to close the data connection calling "close" on the connection
socket)
6. - --> <SEQ=101><ACK=99999><CTL=FIN,ACK> --> CLOSING (client sends
FIN,ACK with not yet sent acknowledgement number)
7. - <-- <SEQ=302><ACK=102><CTL=ACK> <-- CLOSING (ACK is 102
instead of 101, why?)
... (silence from CLIENT)
8. - <-- <SEQ=301><ACK=102><CTL=FIN,ACK> <-- CLOSING
(retransmission, again ACK is 102)
Now, note that packet 6 while having the expected sequence number,
acknowledges something that wasn't sent by the server. So I would
expect
the packet to maybe prompt an ACK response from the server, and then be
ignored. Yet it is not ignored and actually leads to an increase of the
acknowledgement number in the server's retransmission of the FIN,ACK
packet. The explanation I found is that the FIN in packet 6 was
processed, despite the acknowledgement number being unacceptable.
Further experiments indeed show that the server processes this FIN,
transitioning to CLOSING, then on receiving an ACK for the FIN it had
send in packet 5, the server (or better said connection) transitions
from CLOSING to TIME_WAIT (as signaled by netstat).
</quote>
Indeed, tcp_rcv_state_process() calls tcp_ack() but
does not exploit the @acceptable status but for TCP_SYN_RECV
state.
What we want here is to send a challenge ACK, if not in TCP_SYN_RECV
state. TCP_FIN_WAIT1 state is not the only state we should fix.
Add a FLAG_NO_CHALLENGE_ACK so that tcp_rcv_state_process()
can choose to send a challenge ACK and discard the packet instead
of wrongly change socket state.
With help from Neal Cardwell.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Paul Fiterau Brostean <p.fiterau-brostean@science.ru.nl>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-05-23 16:24:46 -06:00
|
|
|
#define FLAG_NO_CHALLENGE_ACK 0x8000 /* do not call tcp_send_challenge_ack() */
|
2018-01-17 13:11:00 -07:00
|
|
|
#define FLAG_ACK_MAYBE_DELAYED 0x10000 /* Likely a delayed ACK */
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
#define FLAG_ACKED (FLAG_DATA_ACKED|FLAG_SYN_ACKED)
|
|
|
|
#define FLAG_NOT_DUP (FLAG_DATA|FLAG_WIN_UPDATE|FLAG_ACKED)
|
2017-11-03 18:46:55 -06:00
|
|
|
#define FLAG_CA_ALERT (FLAG_DATA_SACKED|FLAG_ECE|FLAG_DSACKING_ACK)
|
2005-04-16 16:20:36 -06:00
|
|
|
#define FLAG_FORWARD_PROGRESS (FLAG_ACKED|FLAG_DATA_SACKED)
|
|
|
|
|
|
|
|
#define TCP_REMNANT (TCP_FLAG_FIN|TCP_FLAG_URG|TCP_FLAG_SYN|TCP_FLAG_PSH)
|
2007-05-27 03:04:16 -06:00
|
|
|
#define TCP_HP_BITS (~(TCP_RESERVED_BITS|TCP_FLAG_PSH))
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2016-02-02 11:33:04 -07:00
|
|
|
#define REXMIT_NONE 0 /* no loss recovery to do */
|
|
|
|
#define REXMIT_LOST 1 /* retransmit packets marked lost */
|
|
|
|
#define REXMIT_NEW 2 /* FRTO-style transmit of unsent/new packets */
|
|
|
|
|
2018-04-30 01:16:10 -06:00
|
|
|
#if IS_ENABLED(CONFIG_TLS_DEVICE)
|
2019-05-08 17:46:14 -06:00
|
|
|
static DEFINE_STATIC_KEY_DEFERRED_FALSE(clean_acked_data_enabled, HZ);
|
2018-04-30 01:16:10 -06:00
|
|
|
|
|
|
|
void clean_acked_data_enable(struct inet_connection_sock *icsk,
|
|
|
|
void (*cad)(struct sock *sk, u32 ack_seq))
|
|
|
|
{
|
|
|
|
icsk->icsk_clean_acked = cad;
|
2019-05-08 17:46:14 -06:00
|
|
|
static_branch_inc(&clean_acked_data_enabled.key);
|
2018-04-30 01:16:10 -06:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(clean_acked_data_enable);
|
|
|
|
|
|
|
|
void clean_acked_data_disable(struct inet_connection_sock *icsk)
|
|
|
|
{
|
2019-05-08 17:46:14 -06:00
|
|
|
static_branch_slow_dec_deferred(&clean_acked_data_enabled);
|
2018-04-30 01:16:10 -06:00
|
|
|
icsk->icsk_clean_acked = NULL;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(clean_acked_data_disable);
|
2019-05-08 17:46:14 -06:00
|
|
|
|
|
|
|
void clean_acked_data_flush(void)
|
|
|
|
{
|
|
|
|
static_key_deferred_flush(&clean_acked_data_enabled);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(clean_acked_data_flush);
|
2018-04-30 01:16:10 -06:00
|
|
|
#endif
|
|
|
|
|
2017-04-01 08:00:21 -06:00
|
|
|
static void tcp_gro_dev_warn(struct sock *sk, const struct sk_buff *skb,
|
|
|
|
unsigned int len)
|
tcp: warn on bogus MSS and try to amend it
There have been some reports lately about TCP connection stalls caused
by NIC drivers that aren't setting gso_size on aggregated packets on rx
path. This causes TCP to assume that the MSS is actually the size of the
aggregated packet, which is invalid.
Although the proper fix is to be done at each driver, it's often hard
and cumbersome for one to debug, come to such root cause and report/fix
it.
This patch amends this situation in two ways. First, it adds a warning
on when this situation occurs, so it gives a hint to those trying to
debug this. It also limit the maximum probed MSS to the adverised MSS,
as it should never be any higher than that.
The result is that the connection may not have the best performance ever
but it shouldn't stall, and the admin will have a hint on what to look
for.
Tested with virtio by forcing gso_size to 0.
v2: updated msg per David's suggestion
v3: use skb_iif to find the interface and also log its name, per Eric
Dumazet's suggestion. As the skb may be backlogged and the interface
gone by then, we need to check if the number still has a meaning.
v4: use helper tcp_gro_dev_warn() and avoid pr_warn_once inside __once, per
David's suggestion
Cc: Jonathan Maxwell <jmaxwell37@gmail.com>
Signed-off-by: Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-12-05 13:37:13 -07:00
|
|
|
{
|
|
|
|
static bool __once __read_mostly;
|
|
|
|
|
|
|
|
if (!__once) {
|
|
|
|
struct net_device *dev;
|
|
|
|
|
|
|
|
__once = true;
|
|
|
|
|
|
|
|
rcu_read_lock();
|
|
|
|
dev = dev_get_by_index_rcu(sock_net(sk), skb->skb_iif);
|
2017-04-01 08:00:21 -06:00
|
|
|
if (!dev || len >= dev->mtu)
|
|
|
|
pr_warn("%s: Driver has suspect GRO implementation, TCP performance may be compromised.\n",
|
|
|
|
dev ? dev->name : "Unknown driver");
|
tcp: warn on bogus MSS and try to amend it
There have been some reports lately about TCP connection stalls caused
by NIC drivers that aren't setting gso_size on aggregated packets on rx
path. This causes TCP to assume that the MSS is actually the size of the
aggregated packet, which is invalid.
Although the proper fix is to be done at each driver, it's often hard
and cumbersome for one to debug, come to such root cause and report/fix
it.
This patch amends this situation in two ways. First, it adds a warning
on when this situation occurs, so it gives a hint to those trying to
debug this. It also limit the maximum probed MSS to the adverised MSS,
as it should never be any higher than that.
The result is that the connection may not have the best performance ever
but it shouldn't stall, and the admin will have a hint on what to look
for.
Tested with virtio by forcing gso_size to 0.
v2: updated msg per David's suggestion
v3: use skb_iif to find the interface and also log its name, per Eric
Dumazet's suggestion. As the skb may be backlogged and the interface
gone by then, we need to check if the number still has a meaning.
v4: use helper tcp_gro_dev_warn() and avoid pr_warn_once inside __once, per
David's suggestion
Cc: Jonathan Maxwell <jmaxwell37@gmail.com>
Signed-off-by: Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-12-05 13:37:13 -07:00
|
|
|
rcu_read_unlock();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2007-02-09 07:24:47 -07:00
|
|
|
/* Adapt the MSS value used to make delayed ack decision to the
|
2005-04-16 16:20:36 -06:00
|
|
|
* real world.
|
2007-02-09 07:24:47 -07:00
|
|
|
*/
|
2007-12-31 15:57:14 -07:00
|
|
|
static void tcp_measure_rcv_mss(struct sock *sk, const struct sk_buff *skb)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-09 21:10:42 -06:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2007-02-09 07:24:47 -07:00
|
|
|
const unsigned int lss = icsk->icsk_ack.last_seg_size;
|
2005-08-09 21:10:42 -06:00
|
|
|
unsigned int len;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2007-02-09 07:24:47 -07:00
|
|
|
icsk->icsk_ack.last_seg_size = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* skb->len may jitter because of SACKs, even if peer
|
|
|
|
* sends good full-sized frames.
|
|
|
|
*/
|
2007-12-31 15:57:14 -07:00
|
|
|
len = skb_shinfo(skb)->gso_size ? : skb->len;
|
2005-08-09 21:10:42 -06:00
|
|
|
if (len >= icsk->icsk_ack.rcv_mss) {
|
tcp: warn on bogus MSS and try to amend it
There have been some reports lately about TCP connection stalls caused
by NIC drivers that aren't setting gso_size on aggregated packets on rx
path. This causes TCP to assume that the MSS is actually the size of the
aggregated packet, which is invalid.
Although the proper fix is to be done at each driver, it's often hard
and cumbersome for one to debug, come to such root cause and report/fix
it.
This patch amends this situation in two ways. First, it adds a warning
on when this situation occurs, so it gives a hint to those trying to
debug this. It also limit the maximum probed MSS to the adverised MSS,
as it should never be any higher than that.
The result is that the connection may not have the best performance ever
but it shouldn't stall, and the admin will have a hint on what to look
for.
Tested with virtio by forcing gso_size to 0.
v2: updated msg per David's suggestion
v3: use skb_iif to find the interface and also log its name, per Eric
Dumazet's suggestion. As the skb may be backlogged and the interface
gone by then, we need to check if the number still has a meaning.
v4: use helper tcp_gro_dev_warn() and avoid pr_warn_once inside __once, per
David's suggestion
Cc: Jonathan Maxwell <jmaxwell37@gmail.com>
Signed-off-by: Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-12-05 13:37:13 -07:00
|
|
|
icsk->icsk_ack.rcv_mss = min_t(unsigned int, len,
|
|
|
|
tcp_sk(sk)->advmss);
|
2017-04-01 08:00:21 -06:00
|
|
|
/* Account for possibly-removed options */
|
|
|
|
if (unlikely(len > icsk->icsk_ack.rcv_mss +
|
|
|
|
MAX_TCP_OPTION_SPACE))
|
|
|
|
tcp_gro_dev_warn(sk, skb, len);
|
2005-04-16 16:20:36 -06:00
|
|
|
} else {
|
|
|
|
/* Otherwise, we make more careful check taking into account,
|
|
|
|
* that SACKs block is variable.
|
|
|
|
*
|
|
|
|
* "len" is invariant segment length, including TCP header.
|
|
|
|
*/
|
2007-04-25 19:04:18 -06:00
|
|
|
len += skb->data - skb_transport_header(skb);
|
2009-11-10 02:51:18 -07:00
|
|
|
if (len >= TCP_MSS_DEFAULT + sizeof(struct tcphdr) ||
|
2005-04-16 16:20:36 -06:00
|
|
|
/* If PSH is not set, packet should be
|
|
|
|
* full sized, provided peer TCP is not badly broken.
|
|
|
|
* This observation (if it is correct 8)) allows
|
|
|
|
* to handle super-low mtu links fairly.
|
|
|
|
*/
|
|
|
|
(len >= TCP_MIN_MSS + sizeof(struct tcphdr) &&
|
2007-04-10 22:04:22 -06:00
|
|
|
!(tcp_flag_word(tcp_hdr(skb)) & TCP_REMNANT))) {
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Subtract also invariant (if peer is RFC compliant),
|
|
|
|
* tcp header plus fixed timestamp option length.
|
|
|
|
* Resulting "len" is MSS free of SACK jitter.
|
|
|
|
*/
|
2005-08-09 21:10:42 -06:00
|
|
|
len -= tcp_sk(sk)->tcp_header_len;
|
|
|
|
icsk->icsk_ack.last_seg_size = len;
|
2005-04-16 16:20:36 -06:00
|
|
|
if (len == lss) {
|
2005-08-09 21:10:42 -06:00
|
|
|
icsk->icsk_ack.rcv_mss = len;
|
2005-04-16 16:20:36 -06:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
2006-09-19 13:52:50 -06:00
|
|
|
if (icsk->icsk_ack.pending & ICSK_ACK_PUSHED)
|
|
|
|
icsk->icsk_ack.pending |= ICSK_ACK_PUSHED2;
|
2005-08-09 21:10:42 -06:00
|
|
|
icsk->icsk_ack.pending |= ICSK_ACK_PUSHED;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-05-21 16:08:56 -06:00
|
|
|
static void tcp_incr_quickack(struct sock *sk, unsigned int max_quickacks)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-09 21:10:42 -06:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2012-04-14 23:58:06 -06:00
|
|
|
unsigned int quickacks = tcp_sk(sk)->rcv_wnd / (2 * icsk->icsk_ack.rcv_mss);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2007-12-31 15:57:14 -07:00
|
|
|
if (quickacks == 0)
|
|
|
|
quickacks = 2;
|
2018-05-21 16:08:56 -06:00
|
|
|
quickacks = min(quickacks, max_quickacks);
|
2005-08-09 21:10:42 -06:00
|
|
|
if (quickacks > icsk->icsk_ack.quick)
|
2018-05-21 16:08:56 -06:00
|
|
|
icsk->icsk_ack.quick = quickacks;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
tcp: do not delay ACK in DCTCP upon CE status change
Per DCTCP RFC8257 (Section 3.2) the ACK reflecting the CE status change
has to be sent immediately so the sender can respond quickly:
""" When receiving packets, the CE codepoint MUST be processed as follows:
1. If the CE codepoint is set and DCTCP.CE is false, set DCTCP.CE to
true and send an immediate ACK.
2. If the CE codepoint is not set and DCTCP.CE is true, set DCTCP.CE
to false and send an immediate ACK.
"""
Previously DCTCP implementation may continue to delay the ACK. This
patch fixes that to implement the RFC by forcing an immediate ACK.
Tested with this packetdrill script provided by Larry Brakmo
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
0.000 setsockopt(3, SOL_TCP, TCP_CONGESTION, "dctcp", 5) = 0
0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < [ect0] SEW 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
0.100 > SE. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 8>
0.110 < [ect0] . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
+0 setsockopt(4, SOL_SOCKET, SO_DEBUG, [1], 4) = 0
0.200 < [ect0] . 1:1001(1000) ack 1 win 257
0.200 > [ect01] . 1:1(0) ack 1001
0.200 write(4, ..., 1) = 1
0.200 > [ect01] P. 1:2(1) ack 1001
0.200 < [ect0] . 1001:2001(1000) ack 2 win 257
+0.005 < [ce] . 2001:3001(1000) ack 2 win 257
+0.000 > [ect01] . 2:2(0) ack 2001
// Previously the ACK below would be delayed by 40ms
+0.000 > [ect01] E. 2:2(0) ack 3001
+0.500 < F. 9501:9501(0) ack 4 win 257
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-07-18 14:56:36 -06:00
|
|
|
void tcp_enter_quickack_mode(struct sock *sk, unsigned int max_quickacks)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-09 21:10:42 -06:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2018-05-21 16:08:56 -06:00
|
|
|
|
|
|
|
tcp_incr_quickack(sk, max_quickacks);
|
2019-01-25 11:53:19 -07:00
|
|
|
inet_csk_exit_pingpong_mode(sk);
|
2005-08-09 21:10:42 -06:00
|
|
|
icsk->icsk_ack.ato = TCP_ATO_MIN;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
tcp: do not delay ACK in DCTCP upon CE status change
Per DCTCP RFC8257 (Section 3.2) the ACK reflecting the CE status change
has to be sent immediately so the sender can respond quickly:
""" When receiving packets, the CE codepoint MUST be processed as follows:
1. If the CE codepoint is set and DCTCP.CE is false, set DCTCP.CE to
true and send an immediate ACK.
2. If the CE codepoint is not set and DCTCP.CE is true, set DCTCP.CE
to false and send an immediate ACK.
"""
Previously DCTCP implementation may continue to delay the ACK. This
patch fixes that to implement the RFC by forcing an immediate ACK.
Tested with this packetdrill script provided by Larry Brakmo
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
0.000 setsockopt(3, SOL_TCP, TCP_CONGESTION, "dctcp", 5) = 0
0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < [ect0] SEW 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
0.100 > SE. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 8>
0.110 < [ect0] . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
+0 setsockopt(4, SOL_SOCKET, SO_DEBUG, [1], 4) = 0
0.200 < [ect0] . 1:1001(1000) ack 1 win 257
0.200 > [ect01] . 1:1(0) ack 1001
0.200 write(4, ..., 1) = 1
0.200 > [ect01] P. 1:2(1) ack 1001
0.200 < [ect0] . 1001:2001(1000) ack 2 win 257
+0.005 < [ce] . 2001:3001(1000) ack 2 win 257
+0.000 > [ect01] . 2:2(0) ack 2001
// Previously the ACK below would be delayed by 40ms
+0.000 > [ect01] E. 2:2(0) ack 3001
+0.500 < F. 9501:9501(0) ack 4 win 257
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-07-18 14:56:36 -06:00
|
|
|
EXPORT_SYMBOL(tcp_enter_quickack_mode);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Send ACKs quickly, if "quick" count is not exhausted
|
|
|
|
* and the session is not interactive.
|
|
|
|
*/
|
|
|
|
|
2015-07-07 18:12:28 -06:00
|
|
|
static bool tcp_in_quickack_mode(struct sock *sk)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-09 21:10:42 -06:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2015-07-07 18:12:28 -06:00
|
|
|
const struct dst_entry *dst = __sk_dst_get(sk);
|
2012-05-16 17:15:34 -06:00
|
|
|
|
2015-07-07 18:12:28 -06:00
|
|
|
return (dst && dst_metric(dst, RTAX_QUICKACK)) ||
|
2019-01-25 11:53:19 -07:00
|
|
|
(icsk->icsk_ack.quick && !inet_csk_in_pingpong_mode(sk));
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2014-09-29 05:08:30 -06:00
|
|
|
static void tcp_ecn_queue_cwr(struct tcp_sock *tp)
|
2007-05-27 03:04:16 -06:00
|
|
|
{
|
2007-12-31 15:57:14 -07:00
|
|
|
if (tp->ecn_flags & TCP_ECN_OK)
|
2007-05-27 03:04:16 -06:00
|
|
|
tp->ecn_flags |= TCP_ECN_QUEUE_CWR;
|
|
|
|
}
|
|
|
|
|
tcp: avoid resetting ACK timer upon receiving packet with ECN CWR flag
Previously commit 9aee40006190 ("tcp: ack immediately when a cwr
packet arrives") calls tcp_enter_quickack_mode to force sending
two immediate ACKs upon receiving a packet w/ CWR flag. The side
effect is it'll also reset the delayed ACK timer and interactive
session tracking. This patch removes that side effect by using the
new ACK_NOW flag to force an immmediate ACK.
Packetdrill to demonstrate:
0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 setsockopt(3, SOL_TCP, TCP_CONGESTION, "dctcp", 5) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < [ect0] SEW 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > SE. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 8>
+.1 < [ect0] . 1:1(0) ack 1 win 257
+0 accept(3, ..., ...) = 4
+0 < [ect0] . 1:1001(1000) ack 1 win 257
+0 > [ect01] . 1:1(0) ack 1001
+0 write(4, ..., 1) = 1
+0 > [ect01] P. 1:2(1) ack 1001
+0 < [ect0] . 1001:2001(1000) ack 2 win 257
+0 write(4, ..., 1) = 1
+0 > [ect01] P. 2:3(1) ack 2001
+0 < [ect0] . 2001:3001(1000) ack 3 win 257
+0 < [ect0] . 3001:4001(1000) ack 3 win 257
// Ack delayed ...
+.01 < [ce] P. 4001:4501(500) ack 3 win 257
+0 > [ect01] . 3:3(0) ack 4001
+0 > [ect01] E. 3:3(0) ack 4501
+.001 read(4, ..., 4500) = 4500
+0 write(4, ..., 1) = 1
+0 > [ect01] PE. 3:4(1) ack 4501 win 100
+.01 < [ect0] W. 4501:5501(1000) ack 4 win 257
// No delayed ACK on CWR flag
+0 > [ect01] . 4:4(0) ack 5501
+.31 < [ect0] . 5501:6501(1000) ack 4 win 257
+0 > [ect01] . 4:4(0) ack 6501
Fixes: 9aee40006190 ("tcp: ack immediately when a cwr packet arrives")
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-08-09 10:38:12 -06:00
|
|
|
static void tcp_ecn_accept_cwr(struct sock *sk, const struct sk_buff *skb)
|
2007-05-27 03:04:16 -06:00
|
|
|
{
|
tcp: ack immediately when a cwr packet arrives
We observed high 99 and 99.9% latencies when doing RPCs with DCTCP. The
problem is triggered when the last packet of a request arrives CE
marked. The reply will carry the ECE mark causing TCP to shrink its cwnd
to 1 (because there are no packets in flight). When the 1st packet of
the next request arrives, the ACK was sometimes delayed even though it
is CWR marked, adding up to 40ms to the RPC latency.
This patch insures that CWR marked data packets arriving will be acked
immediately.
Packetdrill script to reproduce the problem:
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
0.000 setsockopt(3, SOL_TCP, TCP_CONGESTION, "dctcp", 5) = 0
0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < [ect0] SEW 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
0.100 > SE. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 8>
0.110 < [ect0] . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
0.200 < [ect0] . 1:1001(1000) ack 1 win 257
0.200 > [ect01] . 1:1(0) ack 1001
0.200 write(4, ..., 1) = 1
0.200 > [ect01] P. 1:2(1) ack 1001
0.200 < [ect0] . 1001:2001(1000) ack 2 win 257
0.200 write(4, ..., 1) = 1
0.200 > [ect01] P. 2:3(1) ack 2001
0.200 < [ect0] . 2001:3001(1000) ack 3 win 257
0.200 < [ect0] . 3001:4001(1000) ack 3 win 257
0.200 > [ect01] . 3:3(0) ack 4001
0.210 < [ce] P. 4001:4501(500) ack 3 win 257
+0.001 read(4, ..., 4500) = 4500
+0 write(4, ..., 1) = 1
+0 > [ect01] PE. 3:4(1) ack 4501
+0.010 < [ect0] W. 4501:5501(1000) ack 4 win 257
// Previously the ACK sequence below would be 4501, causing a long RTO
+0.040~+0.045 > [ect01] . 4:4(0) ack 5501 // delayed ack
+0.311 < [ect0] . 5501:6501(1000) ack 4 win 257 // More data
+0 > [ect01] . 4:4(0) ack 6501 // now acks everything
+0.500 < F. 9501:9501(0) ack 4 win 257
Modified based on comments by Neal Cardwell <ncardwell@google.com>
Signed-off-by: Lawrence Brakmo <brakmo@fb.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-07-23 18:49:39 -06:00
|
|
|
if (tcp_hdr(skb)->cwr) {
|
tcp: avoid resetting ACK timer upon receiving packet with ECN CWR flag
Previously commit 9aee40006190 ("tcp: ack immediately when a cwr
packet arrives") calls tcp_enter_quickack_mode to force sending
two immediate ACKs upon receiving a packet w/ CWR flag. The side
effect is it'll also reset the delayed ACK timer and interactive
session tracking. This patch removes that side effect by using the
new ACK_NOW flag to force an immmediate ACK.
Packetdrill to demonstrate:
0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 setsockopt(3, SOL_TCP, TCP_CONGESTION, "dctcp", 5) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < [ect0] SEW 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > SE. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 8>
+.1 < [ect0] . 1:1(0) ack 1 win 257
+0 accept(3, ..., ...) = 4
+0 < [ect0] . 1:1001(1000) ack 1 win 257
+0 > [ect01] . 1:1(0) ack 1001
+0 write(4, ..., 1) = 1
+0 > [ect01] P. 1:2(1) ack 1001
+0 < [ect0] . 1001:2001(1000) ack 2 win 257
+0 write(4, ..., 1) = 1
+0 > [ect01] P. 2:3(1) ack 2001
+0 < [ect0] . 2001:3001(1000) ack 3 win 257
+0 < [ect0] . 3001:4001(1000) ack 3 win 257
// Ack delayed ...
+.01 < [ce] P. 4001:4501(500) ack 3 win 257
+0 > [ect01] . 3:3(0) ack 4001
+0 > [ect01] E. 3:3(0) ack 4501
+.001 read(4, ..., 4500) = 4500
+0 write(4, ..., 1) = 1
+0 > [ect01] PE. 3:4(1) ack 4501 win 100
+.01 < [ect0] W. 4501:5501(1000) ack 4 win 257
// No delayed ACK on CWR flag
+0 > [ect01] . 4:4(0) ack 5501
+.31 < [ect0] . 5501:6501(1000) ack 4 win 257
+0 > [ect01] . 4:4(0) ack 6501
Fixes: 9aee40006190 ("tcp: ack immediately when a cwr packet arrives")
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-08-09 10:38:12 -06:00
|
|
|
tcp_sk(sk)->ecn_flags &= ~TCP_ECN_DEMAND_CWR;
|
tcp: ack immediately when a cwr packet arrives
We observed high 99 and 99.9% latencies when doing RPCs with DCTCP. The
problem is triggered when the last packet of a request arrives CE
marked. The reply will carry the ECE mark causing TCP to shrink its cwnd
to 1 (because there are no packets in flight). When the 1st packet of
the next request arrives, the ACK was sometimes delayed even though it
is CWR marked, adding up to 40ms to the RPC latency.
This patch insures that CWR marked data packets arriving will be acked
immediately.
Packetdrill script to reproduce the problem:
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
0.000 setsockopt(3, SOL_TCP, TCP_CONGESTION, "dctcp", 5) = 0
0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < [ect0] SEW 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
0.100 > SE. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 8>
0.110 < [ect0] . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
0.200 < [ect0] . 1:1001(1000) ack 1 win 257
0.200 > [ect01] . 1:1(0) ack 1001
0.200 write(4, ..., 1) = 1
0.200 > [ect01] P. 1:2(1) ack 1001
0.200 < [ect0] . 1001:2001(1000) ack 2 win 257
0.200 write(4, ..., 1) = 1
0.200 > [ect01] P. 2:3(1) ack 2001
0.200 < [ect0] . 2001:3001(1000) ack 3 win 257
0.200 < [ect0] . 3001:4001(1000) ack 3 win 257
0.200 > [ect01] . 3:3(0) ack 4001
0.210 < [ce] P. 4001:4501(500) ack 3 win 257
+0.001 read(4, ..., 4500) = 4500
+0 write(4, ..., 1) = 1
+0 > [ect01] PE. 3:4(1) ack 4501
+0.010 < [ect0] W. 4501:5501(1000) ack 4 win 257
// Previously the ACK sequence below would be 4501, causing a long RTO
+0.040~+0.045 > [ect01] . 4:4(0) ack 5501 // delayed ack
+0.311 < [ect0] . 5501:6501(1000) ack 4 win 257 // More data
+0 > [ect01] . 4:4(0) ack 6501 // now acks everything
+0.500 < F. 9501:9501(0) ack 4 win 257
Modified based on comments by Neal Cardwell <ncardwell@google.com>
Signed-off-by: Lawrence Brakmo <brakmo@fb.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-07-23 18:49:39 -06:00
|
|
|
|
|
|
|
/* If the sender is telling us it has entered CWR, then its
|
|
|
|
* cwnd may be very low (even just 1 packet), so we should ACK
|
|
|
|
* immediately.
|
|
|
|
*/
|
tcp: avoid resetting ACK timer upon receiving packet with ECN CWR flag
Previously commit 9aee40006190 ("tcp: ack immediately when a cwr
packet arrives") calls tcp_enter_quickack_mode to force sending
two immediate ACKs upon receiving a packet w/ CWR flag. The side
effect is it'll also reset the delayed ACK timer and interactive
session tracking. This patch removes that side effect by using the
new ACK_NOW flag to force an immmediate ACK.
Packetdrill to demonstrate:
0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 setsockopt(3, SOL_TCP, TCP_CONGESTION, "dctcp", 5) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < [ect0] SEW 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > SE. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 8>
+.1 < [ect0] . 1:1(0) ack 1 win 257
+0 accept(3, ..., ...) = 4
+0 < [ect0] . 1:1001(1000) ack 1 win 257
+0 > [ect01] . 1:1(0) ack 1001
+0 write(4, ..., 1) = 1
+0 > [ect01] P. 1:2(1) ack 1001
+0 < [ect0] . 1001:2001(1000) ack 2 win 257
+0 write(4, ..., 1) = 1
+0 > [ect01] P. 2:3(1) ack 2001
+0 < [ect0] . 2001:3001(1000) ack 3 win 257
+0 < [ect0] . 3001:4001(1000) ack 3 win 257
// Ack delayed ...
+.01 < [ce] P. 4001:4501(500) ack 3 win 257
+0 > [ect01] . 3:3(0) ack 4001
+0 > [ect01] E. 3:3(0) ack 4501
+.001 read(4, ..., 4500) = 4500
+0 write(4, ..., 1) = 1
+0 > [ect01] PE. 3:4(1) ack 4501 win 100
+.01 < [ect0] W. 4501:5501(1000) ack 4 win 257
// No delayed ACK on CWR flag
+0 > [ect01] . 4:4(0) ack 5501
+.31 < [ect0] . 5501:6501(1000) ack 4 win 257
+0 > [ect01] . 4:4(0) ack 6501
Fixes: 9aee40006190 ("tcp: ack immediately when a cwr packet arrives")
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-08-09 10:38:12 -06:00
|
|
|
inet_csk(sk)->icsk_ack.pending |= ICSK_ACK_NOW;
|
tcp: ack immediately when a cwr packet arrives
We observed high 99 and 99.9% latencies when doing RPCs with DCTCP. The
problem is triggered when the last packet of a request arrives CE
marked. The reply will carry the ECE mark causing TCP to shrink its cwnd
to 1 (because there are no packets in flight). When the 1st packet of
the next request arrives, the ACK was sometimes delayed even though it
is CWR marked, adding up to 40ms to the RPC latency.
This patch insures that CWR marked data packets arriving will be acked
immediately.
Packetdrill script to reproduce the problem:
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
0.000 setsockopt(3, SOL_TCP, TCP_CONGESTION, "dctcp", 5) = 0
0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < [ect0] SEW 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
0.100 > SE. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 8>
0.110 < [ect0] . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
0.200 < [ect0] . 1:1001(1000) ack 1 win 257
0.200 > [ect01] . 1:1(0) ack 1001
0.200 write(4, ..., 1) = 1
0.200 > [ect01] P. 1:2(1) ack 1001
0.200 < [ect0] . 1001:2001(1000) ack 2 win 257
0.200 write(4, ..., 1) = 1
0.200 > [ect01] P. 2:3(1) ack 2001
0.200 < [ect0] . 2001:3001(1000) ack 3 win 257
0.200 < [ect0] . 3001:4001(1000) ack 3 win 257
0.200 > [ect01] . 3:3(0) ack 4001
0.210 < [ce] P. 4001:4501(500) ack 3 win 257
+0.001 read(4, ..., 4500) = 4500
+0 write(4, ..., 1) = 1
+0 > [ect01] PE. 3:4(1) ack 4501
+0.010 < [ect0] W. 4501:5501(1000) ack 4 win 257
// Previously the ACK sequence below would be 4501, causing a long RTO
+0.040~+0.045 > [ect01] . 4:4(0) ack 5501 // delayed ack
+0.311 < [ect0] . 5501:6501(1000) ack 4 win 257 // More data
+0 > [ect01] . 4:4(0) ack 6501 // now acks everything
+0.500 < F. 9501:9501(0) ack 4 win 257
Modified based on comments by Neal Cardwell <ncardwell@google.com>
Signed-off-by: Lawrence Brakmo <brakmo@fb.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-07-23 18:49:39 -06:00
|
|
|
}
|
2007-05-27 03:04:16 -06:00
|
|
|
}
|
|
|
|
|
2014-09-29 05:08:30 -06:00
|
|
|
static void tcp_ecn_withdraw_cwr(struct tcp_sock *tp)
|
2007-05-27 03:04:16 -06:00
|
|
|
{
|
|
|
|
tp->ecn_flags &= ~TCP_ECN_DEMAND_CWR;
|
|
|
|
}
|
|
|
|
|
2018-06-04 16:29:51 -06:00
|
|
|
static void __tcp_ecn_check_ce(struct sock *sk, const struct sk_buff *skb)
|
2007-05-27 03:04:16 -06:00
|
|
|
{
|
2018-06-04 16:29:51 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2011-09-27 00:20:08 -06:00
|
|
|
switch (TCP_SKB_CB(skb)->ip_dsfield & INET_ECN_MASK) {
|
2011-09-22 14:02:19 -06:00
|
|
|
case INET_ECN_NOT_ECT:
|
2007-05-27 03:04:16 -06:00
|
|
|
/* Funny extension: if ECT is not set on a segment,
|
2011-09-22 14:02:19 -06:00
|
|
|
* and we already seen ECT on a previous segment,
|
|
|
|
* it is probably a retransmit.
|
|
|
|
*/
|
|
|
|
if (tp->ecn_flags & TCP_ECN_SEEN)
|
2018-06-27 09:47:21 -06:00
|
|
|
tcp_enter_quickack_mode(sk, 2);
|
2011-09-22 14:02:19 -06:00
|
|
|
break;
|
|
|
|
case INET_ECN_CE:
|
2018-06-04 16:29:51 -06:00
|
|
|
if (tcp_ca_needs_ecn(sk))
|
|
|
|
tcp_ca_event(sk, CA_EVENT_ECN_IS_CE);
|
2014-09-26 14:37:35 -06:00
|
|
|
|
2012-08-06 05:04:43 -06:00
|
|
|
if (!(tp->ecn_flags & TCP_ECN_DEMAND_CWR)) {
|
|
|
|
/* Better not delay acks, sender can have a very low cwnd */
|
2018-06-27 09:47:21 -06:00
|
|
|
tcp_enter_quickack_mode(sk, 2);
|
2012-08-06 05:04:43 -06:00
|
|
|
tp->ecn_flags |= TCP_ECN_DEMAND_CWR;
|
|
|
|
}
|
2014-09-26 14:37:35 -06:00
|
|
|
tp->ecn_flags |= TCP_ECN_SEEN;
|
|
|
|
break;
|
2011-09-22 14:02:19 -06:00
|
|
|
default:
|
2018-06-04 16:29:51 -06:00
|
|
|
if (tcp_ca_needs_ecn(sk))
|
|
|
|
tcp_ca_event(sk, CA_EVENT_ECN_NO_CE);
|
2011-09-22 14:02:19 -06:00
|
|
|
tp->ecn_flags |= TCP_ECN_SEEN;
|
2014-09-26 14:37:35 -06:00
|
|
|
break;
|
2007-05-27 03:04:16 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-06-04 16:29:51 -06:00
|
|
|
static void tcp_ecn_check_ce(struct sock *sk, const struct sk_buff *skb)
|
2014-09-29 05:08:30 -06:00
|
|
|
{
|
2018-06-04 16:29:51 -06:00
|
|
|
if (tcp_sk(sk)->ecn_flags & TCP_ECN_OK)
|
|
|
|
__tcp_ecn_check_ce(sk, skb);
|
2014-09-29 05:08:30 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
static void tcp_ecn_rcv_synack(struct tcp_sock *tp, const struct tcphdr *th)
|
2007-05-27 03:04:16 -06:00
|
|
|
{
|
2007-12-31 15:57:14 -07:00
|
|
|
if ((tp->ecn_flags & TCP_ECN_OK) && (!th->ece || th->cwr))
|
2007-05-27 03:04:16 -06:00
|
|
|
tp->ecn_flags &= ~TCP_ECN_OK;
|
|
|
|
}
|
|
|
|
|
2014-09-29 05:08:30 -06:00
|
|
|
static void tcp_ecn_rcv_syn(struct tcp_sock *tp, const struct tcphdr *th)
|
2007-05-27 03:04:16 -06:00
|
|
|
{
|
2007-12-31 15:57:14 -07:00
|
|
|
if ((tp->ecn_flags & TCP_ECN_OK) && (!th->ece || !th->cwr))
|
2007-05-27 03:04:16 -06:00
|
|
|
tp->ecn_flags &= ~TCP_ECN_OK;
|
|
|
|
}
|
|
|
|
|
2014-09-29 05:08:30 -06:00
|
|
|
static bool tcp_ecn_rcv_ecn_echo(const struct tcp_sock *tp, const struct tcphdr *th)
|
2007-05-27 03:04:16 -06:00
|
|
|
{
|
2007-12-31 15:57:14 -07:00
|
|
|
if (th->ece && !th->syn && (tp->ecn_flags & TCP_ECN_OK))
|
2012-05-16 17:15:34 -06:00
|
|
|
return true;
|
|
|
|
return false;
|
2007-05-27 03:04:16 -06:00
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Buffer size and advertised window tuning.
|
|
|
|
*
|
|
|
|
* 1. Tuning sk->sk_sndbuf, when connection enters established state.
|
|
|
|
*/
|
|
|
|
|
2013-10-01 11:23:44 -06:00
|
|
|
static void tcp_sndbuf_expand(struct sock *sk)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2013-10-01 11:23:44 -06:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
2016-09-19 21:39:20 -06:00
|
|
|
const struct tcp_congestion_ops *ca_ops = inet_csk(sk)->icsk_ca_ops;
|
2013-10-01 11:23:44 -06:00
|
|
|
int sndmem, per_mss;
|
|
|
|
u32 nr_segs;
|
|
|
|
|
|
|
|
/* Worst case is non GSO/TSO : each frame consumes one skb
|
|
|
|
* and skb->head is kmalloced using power of two area of memory
|
|
|
|
*/
|
|
|
|
per_mss = max_t(u32, tp->rx_opt.mss_clamp, tp->mss_cache) +
|
|
|
|
MAX_TCP_HEADER +
|
|
|
|
SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
|
|
|
|
|
|
|
|
per_mss = roundup_pow_of_two(per_mss) +
|
|
|
|
SKB_DATA_ALIGN(sizeof(struct sk_buff));
|
|
|
|
|
|
|
|
nr_segs = max_t(u32, TCP_INIT_CWND, tp->snd_cwnd);
|
|
|
|
nr_segs = max_t(u32, nr_segs, tp->reordering + 1);
|
|
|
|
|
|
|
|
/* Fast Recovery (RFC 5681 3.2) :
|
|
|
|
* Cubic needs 1.7 factor, rounded to 2 to include
|
2018-02-11 15:34:03 -07:00
|
|
|
* extra cushion (application might react slowly to EPOLLOUT)
|
2013-10-01 11:23:44 -06:00
|
|
|
*/
|
2016-09-19 21:39:20 -06:00
|
|
|
sndmem = ca_ops->sndbuf_expand ? ca_ops->sndbuf_expand(sk) : 2;
|
|
|
|
sndmem *= nr_segs * per_mss;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2011-10-13 12:24:42 -06:00
|
|
|
if (sk->sk_sndbuf < sndmem)
|
2017-11-07 01:29:28 -07:00
|
|
|
sk->sk_sndbuf = min(sndmem, sock_net(sk)->ipv4.sysctl_tcp_wmem[2]);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* 2. Tuning advertised window (window_clamp, rcv_ssthresh)
|
|
|
|
*
|
|
|
|
* All tcp_full_space() is split to two parts: "network" buffer, allocated
|
|
|
|
* forward and advertised in receiver window (tp->rcv_wnd) and
|
|
|
|
* "application buffer", required to isolate scheduling/application
|
|
|
|
* latencies from network.
|
|
|
|
* window_clamp is maximal advertised window. It can be less than
|
|
|
|
* tcp_full_space(), in this case tcp_full_space() - window_clamp
|
|
|
|
* is reserved for "application" buffer. The less window_clamp is
|
|
|
|
* the smoother our behaviour from viewpoint of network, but the lower
|
|
|
|
* throughput and the higher sensitivity of the connection to losses. 8)
|
|
|
|
*
|
|
|
|
* rcv_ssthresh is more strict window_clamp used at "slow start"
|
|
|
|
* phase to predict further behaviour of this connection.
|
|
|
|
* It is used for two goals:
|
|
|
|
* - to enforce header prediction at sender, even when application
|
|
|
|
* requires some significant "application buffer". It is check #1.
|
|
|
|
* - to prevent pruning of receive queue because of misprediction
|
|
|
|
* of receiver window. Check #2.
|
|
|
|
*
|
|
|
|
* The scheme does not work when sender sends good segments opening
|
2005-11-10 18:13:47 -07:00
|
|
|
* window and then starts to feed us spaghetti. But it should work
|
2005-04-16 16:20:36 -06:00
|
|
|
* in common situations. Otherwise, we have to rely on queue collapsing.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* Slow part of check#2. */
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
static int __tcp_grow_window(const struct sock *sk, const struct sk_buff *skb)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Optimize this! */
|
2017-10-26 22:55:09 -06:00
|
|
|
int truesize = tcp_win_from_space(sk, skb->truesize) >> 1;
|
2017-11-07 01:29:28 -07:00
|
|
|
int window = tcp_win_from_space(sk, sock_net(sk)->ipv4.sysctl_tcp_rmem[2]) >> 1;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
while (tp->rcv_ssthresh <= window) {
|
|
|
|
if (truesize <= skb->len)
|
2005-08-09 21:10:42 -06:00
|
|
|
return 2 * inet_csk(sk)->icsk_ack.rcv_mss;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
truesize >>= 1;
|
|
|
|
window >>= 1;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2011-10-21 03:22:42 -06:00
|
|
|
static void tcp_grow_window(struct sock *sk, const struct sk_buff *skb)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2019-04-16 11:55:20 -06:00
|
|
|
int room;
|
|
|
|
|
|
|
|
room = min_t(int, tp->window_clamp, tcp_space(sk)) - tp->rcv_ssthresh;
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Check #1 */
|
2019-04-16 11:55:20 -06:00
|
|
|
if (room > 0 && !tcp_under_memory_pressure(sk)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
int incr;
|
|
|
|
|
|
|
|
/* Check #2. Increase window, if skb with such overhead
|
|
|
|
* will fit to rcvbuf in future.
|
|
|
|
*/
|
2017-10-26 22:55:09 -06:00
|
|
|
if (tcp_win_from_space(sk, skb->truesize) <= skb->len)
|
2007-12-31 15:57:14 -07:00
|
|
|
incr = 2 * tp->advmss;
|
2005-04-16 16:20:36 -06:00
|
|
|
else
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
incr = __tcp_grow_window(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
if (incr) {
|
2012-04-16 17:28:07 -06:00
|
|
|
incr = max_t(int, incr, 2 * skb->len);
|
2019-04-16 11:55:20 -06:00
|
|
|
tp->rcv_ssthresh += min(room, incr);
|
2005-08-09 21:10:42 -06:00
|
|
|
inet_csk(sk)->icsk_ack.quick |= 1;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-09-27 12:21:19 -06:00
|
|
|
/* 3. Try to fixup all. It is made immediately after connection enters
|
2005-04-16 16:20:36 -06:00
|
|
|
* established state.
|
|
|
|
*/
|
2012-08-31 06:29:11 -06:00
|
|
|
void tcp_init_buffer_space(struct sock *sk)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2017-10-26 22:55:08 -06:00
|
|
|
int tcp_app_win = sock_net(sk)->ipv4.sysctl_tcp_app_win;
|
2005-04-16 16:20:36 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int maxwin;
|
|
|
|
|
|
|
|
if (!(sk->sk_userlocks & SOCK_SNDBUF_LOCK))
|
2013-10-01 11:23:44 -06:00
|
|
|
tcp_sndbuf_expand(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2018-10-01 16:42:32 -06:00
|
|
|
tp->rcvq_space.space = min_t(u32, tp->rcv_wnd, TCP_INIT_CWND * tp->advmss);
|
2017-05-16 15:00:14 -06:00
|
|
|
tcp_mstamp_refresh(tp);
|
2017-04-25 11:15:41 -06:00
|
|
|
tp->rcvq_space.time = tp->tcp_mstamp;
|
2013-09-20 14:56:58 -06:00
|
|
|
tp->rcvq_space.seq = tp->copied_seq;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
maxwin = tcp_full_space(sk);
|
|
|
|
|
|
|
|
if (tp->window_clamp >= maxwin) {
|
|
|
|
tp->window_clamp = maxwin;
|
|
|
|
|
2017-10-26 22:55:08 -06:00
|
|
|
if (tcp_app_win && maxwin > 4 * tp->advmss)
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->window_clamp = max(maxwin -
|
2017-10-26 22:55:08 -06:00
|
|
|
(maxwin >> tcp_app_win),
|
2005-04-16 16:20:36 -06:00
|
|
|
4 * tp->advmss);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Force reservation of one segment. */
|
2017-10-26 22:55:08 -06:00
|
|
|
if (tcp_app_win &&
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->window_clamp > 2 * tp->advmss &&
|
|
|
|
tp->window_clamp + tp->advmss > maxwin)
|
|
|
|
tp->window_clamp = max(2 * tp->advmss, maxwin - tp->advmss);
|
|
|
|
|
|
|
|
tp->rcv_ssthresh = min(tp->rcv_ssthresh, tp->window_clamp);
|
2017-05-16 15:00:04 -06:00
|
|
|
tp->snd_cwnd_stamp = tcp_jiffies32;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2018-09-27 12:21:19 -06:00
|
|
|
/* 4. Recalculate window clamp after socket hit its memory bounds. */
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
static void tcp_clamp_window(struct sock *sk)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-08-10 01:03:31 -06:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2017-11-07 01:29:28 -07:00
|
|
|
struct net *net = sock_net(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2005-08-10 01:03:31 -06:00
|
|
|
icsk->icsk_ack.quick = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-11-07 01:29:28 -07:00
|
|
|
if (sk->sk_rcvbuf < net->ipv4.sysctl_tcp_rmem[2] &&
|
2005-11-10 18:11:48 -07:00
|
|
|
!(sk->sk_userlocks & SOCK_RCVBUF_LOCK) &&
|
2015-05-15 13:39:27 -06:00
|
|
|
!tcp_under_memory_pressure(sk) &&
|
2011-12-11 14:47:02 -07:00
|
|
|
sk_memory_allocated(sk) < sk_prot_mem_limits(sk, 0)) {
|
2005-11-10 18:11:48 -07:00
|
|
|
sk->sk_rcvbuf = min(atomic_read(&sk->sk_rmem_alloc),
|
2017-11-07 01:29:28 -07:00
|
|
|
net->ipv4.sysctl_tcp_rmem[2]);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2005-11-10 18:11:48 -07:00
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf)
|
2007-12-31 15:57:14 -07:00
|
|
|
tp->rcv_ssthresh = min(tp->window_clamp, 2U * tp->advmss);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2006-01-03 17:03:49 -07:00
|
|
|
/* Initialize RCV_MSS value.
|
|
|
|
* RCV_MSS is an our guess about MSS used by the peer.
|
|
|
|
* We haven't any direct information about the MSS.
|
|
|
|
* It's better to underestimate the RCV_MSS rather than overestimate.
|
|
|
|
* Overestimations make us ACKing less frequently than needed.
|
|
|
|
* Underestimations are more easy to detect and fix by tcp_measure_rcv_mss().
|
|
|
|
*/
|
|
|
|
void tcp_initialize_rcv_mss(struct sock *sk)
|
|
|
|
{
|
2011-10-21 03:22:42 -06:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
2006-01-03 17:03:49 -07:00
|
|
|
unsigned int hint = min_t(unsigned int, tp->advmss, tp->mss_cache);
|
|
|
|
|
2007-12-31 15:57:14 -07:00
|
|
|
hint = min(hint, tp->rcv_wnd / 2);
|
2009-11-10 02:51:18 -07:00
|
|
|
hint = min(hint, TCP_MSS_DEFAULT);
|
2006-01-03 17:03:49 -07:00
|
|
|
hint = max(hint, TCP_MIN_MSS);
|
|
|
|
|
|
|
|
inet_csk(sk)->icsk_ack.rcv_mss = hint;
|
|
|
|
}
|
2010-07-09 15:22:10 -06:00
|
|
|
EXPORT_SYMBOL(tcp_initialize_rcv_mss);
|
2006-01-03 17:03:49 -07:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Receiver "autotuning" code.
|
|
|
|
*
|
|
|
|
* The algorithm for RTT estimation w/o timestamps is based on
|
|
|
|
* Dynamic Right-Sizing (DRS) by Wu Feng and Mike Fisk of LANL.
|
2010-10-18 03:03:14 -06:00
|
|
|
* <http://public.lanl.gov/radiant/pubs.html#DRS>
|
2005-04-16 16:20:36 -06:00
|
|
|
*
|
|
|
|
* More detail on this code can be found at
|
2010-10-18 03:03:14 -06:00
|
|
|
* <http://staff.psc.edu/jheffner/>,
|
2005-04-16 16:20:36 -06:00
|
|
|
* though this reference is out of date. A new paper
|
|
|
|
* is pending.
|
|
|
|
*/
|
|
|
|
static void tcp_rcv_rtt_update(struct tcp_sock *tp, u32 sample, int win_dep)
|
|
|
|
{
|
2017-04-25 11:15:41 -06:00
|
|
|
u32 new_sample = tp->rcv_rtt_est.rtt_us;
|
2005-04-16 16:20:36 -06:00
|
|
|
long m = sample;
|
|
|
|
|
|
|
|
if (new_sample != 0) {
|
|
|
|
/* If we sample in larger samples in the non-timestamp
|
|
|
|
* case, we could grossly overestimate the RTT especially
|
|
|
|
* with chatty applications or bulk transfer apps which
|
|
|
|
* are stalled on filesystem I/O.
|
|
|
|
*
|
|
|
|
* Also, since we are only going for a minimum in the
|
2005-11-15 16:17:10 -07:00
|
|
|
* non-timestamp case, we do not smooth things out
|
2005-11-10 18:13:47 -07:00
|
|
|
* else with timestamps disabled convergence takes too
|
2005-04-16 16:20:36 -06:00
|
|
|
* long.
|
|
|
|
*/
|
|
|
|
if (!win_dep) {
|
|
|
|
m -= (new_sample >> 3);
|
|
|
|
new_sample += m;
|
2012-04-10 01:59:20 -06:00
|
|
|
} else {
|
|
|
|
m <<= 3;
|
|
|
|
if (m < new_sample)
|
|
|
|
new_sample = m;
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
} else {
|
2005-11-10 18:13:47 -07:00
|
|
|
/* No previous measure. */
|
2005-04-16 16:20:36 -06:00
|
|
|
new_sample = m << 3;
|
|
|
|
}
|
|
|
|
|
2017-04-25 11:15:41 -06:00
|
|
|
tp->rcv_rtt_est.rtt_us = new_sample;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline void tcp_rcv_rtt_measure(struct tcp_sock *tp)
|
|
|
|
{
|
2017-04-25 11:15:41 -06:00
|
|
|
u32 delta_us;
|
|
|
|
|
2017-05-16 15:00:14 -06:00
|
|
|
if (tp->rcv_rtt_est.time == 0)
|
2005-04-16 16:20:36 -06:00
|
|
|
goto new_measure;
|
|
|
|
if (before(tp->rcv_nxt, tp->rcv_rtt_est.seq))
|
|
|
|
return;
|
2017-05-16 15:00:14 -06:00
|
|
|
delta_us = tcp_stamp_us_delta(tp->tcp_mstamp, tp->rcv_rtt_est.time);
|
2017-12-12 17:28:58 -07:00
|
|
|
if (!delta_us)
|
|
|
|
delta_us = 1;
|
2017-04-25 11:15:41 -06:00
|
|
|
tcp_rcv_rtt_update(tp, delta_us, 1);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
new_measure:
|
|
|
|
tp->rcv_rtt_est.seq = tp->rcv_nxt + tp->rcv_wnd;
|
2017-04-25 11:15:41 -06:00
|
|
|
tp->rcv_rtt_est.time = tp->tcp_mstamp;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2007-12-31 15:57:14 -07:00
|
|
|
static inline void tcp_rcv_rtt_measure_ts(struct sock *sk,
|
|
|
|
const struct sk_buff *skb)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-09 21:10:42 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2017-05-16 15:00:14 -06:00
|
|
|
|
2018-06-19 22:42:50 -06:00
|
|
|
if (tp->rx_opt.rcv_tsecr == tp->rcv_rtt_last_tsecr)
|
|
|
|
return;
|
|
|
|
tp->rcv_rtt_last_tsecr = tp->rx_opt.rcv_tsecr;
|
|
|
|
|
|
|
|
if (TCP_SKB_CB(skb)->end_seq -
|
|
|
|
TCP_SKB_CB(skb)->seq >= inet_csk(sk)->icsk_ack.rcv_mss) {
|
2017-05-16 15:00:14 -06:00
|
|
|
u32 delta = tcp_time_stamp(tp) - tp->rx_opt.rcv_tsecr;
|
2017-12-12 17:28:58 -07:00
|
|
|
u32 delta_us;
|
2017-05-16 15:00:14 -06:00
|
|
|
|
2018-11-24 10:12:24 -07:00
|
|
|
if (likely(delta < INT_MAX / (USEC_PER_SEC / TCP_TS_HZ))) {
|
|
|
|
if (!delta)
|
|
|
|
delta = 1;
|
|
|
|
delta_us = delta * (USEC_PER_SEC / TCP_TS_HZ);
|
|
|
|
tcp_rcv_rtt_update(tp, delta_us, 0);
|
|
|
|
}
|
2017-05-16 15:00:14 -06:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This function should be called every time data is copied to user space.
|
|
|
|
* It calculates the appropriate TCP receive buffer space.
|
|
|
|
*/
|
|
|
|
void tcp_rcv_space_adjust(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2017-12-10 18:55:03 -07:00
|
|
|
u32 copied;
|
2005-04-16 16:20:36 -06:00
|
|
|
int time;
|
2007-02-09 07:24:47 -07:00
|
|
|
|
2018-04-20 09:18:26 -06:00
|
|
|
trace_tcp_rcv_space_adjust(sk);
|
|
|
|
|
2017-12-06 12:08:19 -07:00
|
|
|
tcp_mstamp_refresh(tp);
|
2017-05-16 15:00:14 -06:00
|
|
|
time = tcp_stamp_us_delta(tp->tcp_mstamp, tp->rcvq_space.time);
|
2017-04-25 11:15:41 -06:00
|
|
|
if (time < (tp->rcv_rtt_est.rtt_us >> 3) || tp->rcv_rtt_est.rtt_us == 0)
|
2005-04-16 16:20:36 -06:00
|
|
|
return;
|
2007-02-09 07:24:47 -07:00
|
|
|
|
2013-09-20 14:56:58 -06:00
|
|
|
/* Number of bytes copied to user in last RTT */
|
|
|
|
copied = tp->copied_seq - tp->rcvq_space.seq;
|
|
|
|
if (copied <= tp->rcvq_space.space)
|
|
|
|
goto new_measure;
|
|
|
|
|
|
|
|
/* A bit of theory :
|
|
|
|
* copied = bytes received in previous RTT, our base window
|
|
|
|
* To cope with packet losses, we need a 2x factor
|
|
|
|
* To cope with slow start, and sender growing its cwin by 100 %
|
|
|
|
* every RTT, we need a 4x factor, because the ACK we are sending
|
|
|
|
* now is for the next RTT, not the current one :
|
|
|
|
* <prev RTT . ><current RTT .. ><next RTT .... >
|
|
|
|
*/
|
|
|
|
|
2017-10-27 08:47:22 -06:00
|
|
|
if (sock_net(sk)->ipv4.sysctl_tcp_moderate_rcvbuf &&
|
2013-09-20 14:56:58 -06:00
|
|
|
!(sk->sk_userlocks & SOCK_RCVBUF_LOCK)) {
|
2017-12-10 18:55:03 -07:00
|
|
|
int rcvmem, rcvbuf;
|
2017-12-10 18:55:04 -07:00
|
|
|
u64 rcvwin, grow;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2013-09-20 14:56:58 -06:00
|
|
|
/* minimal window to cope with packet losses, assuming
|
|
|
|
* steady state. Add some cushion because of small variations.
|
|
|
|
*/
|
2017-12-10 18:55:03 -07:00
|
|
|
rcvwin = ((u64)copied << 1) + 16 * tp->advmss;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-12-10 18:55:04 -07:00
|
|
|
/* Accommodate for sender rate increase (eg. slow start) */
|
|
|
|
grow = rcvwin * (copied - tp->rcvq_space.space);
|
|
|
|
do_div(grow, tp->rcvq_space.space);
|
|
|
|
rcvwin += (grow << 1);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2013-09-20 14:56:58 -06:00
|
|
|
rcvmem = SKB_TRUESIZE(tp->advmss + MAX_TCP_HEADER);
|
2017-10-26 22:55:09 -06:00
|
|
|
while (tcp_win_from_space(sk, rcvmem) < tp->advmss)
|
2013-09-20 14:56:58 -06:00
|
|
|
rcvmem += 128;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-12-10 18:55:03 -07:00
|
|
|
do_div(rcvwin, tp->advmss);
|
|
|
|
rcvbuf = min_t(u64, rcvwin * rcvmem,
|
|
|
|
sock_net(sk)->ipv4.sysctl_tcp_rmem[2]);
|
2013-09-20 14:56:58 -06:00
|
|
|
if (rcvbuf > sk->sk_rcvbuf) {
|
|
|
|
sk->sk_rcvbuf = rcvbuf;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2013-09-20 14:56:58 -06:00
|
|
|
/* Make the window clamp follow along. */
|
2017-12-10 18:55:02 -07:00
|
|
|
tp->window_clamp = tcp_win_from_space(sk, rcvbuf);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
2013-09-20 14:56:58 -06:00
|
|
|
tp->rcvq_space.space = copied;
|
2007-02-09 07:24:47 -07:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
new_measure:
|
|
|
|
tp->rcvq_space.seq = tp->copied_seq;
|
2017-04-25 11:15:41 -06:00
|
|
|
tp->rcvq_space.time = tp->tcp_mstamp;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* There is something which you must keep in mind when you analyze the
|
|
|
|
* behavior of the tp->ato delayed ack timeout interval. When a
|
|
|
|
* connection starts up, we want to ack as quickly as possible. The
|
|
|
|
* problem is that "good" TCP's do slow start at the beginning of data
|
|
|
|
* transmission. The means that until we send the first few ACK's the
|
|
|
|
* sender will sit on his end and only queue most of his data, because
|
|
|
|
* he can only send snd_cwnd unacked packets at any given time. For
|
|
|
|
* each ACK we send, he increments snd_cwnd and transmits more of his
|
|
|
|
* queue. -DaveM
|
|
|
|
*/
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
static void tcp_event_data_recv(struct sock *sk, struct sk_buff *skb)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-08-09 21:10:42 -06:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
u32 now;
|
|
|
|
|
2005-08-09 21:10:42 -06:00
|
|
|
inet_csk_schedule_ack(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2005-08-09 21:10:42 -06:00
|
|
|
tcp_measure_rcv_mss(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
tcp_rcv_rtt_measure(tp);
|
2007-02-09 07:24:47 -07:00
|
|
|
|
2017-05-16 15:00:07 -06:00
|
|
|
now = tcp_jiffies32;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2005-08-09 21:10:42 -06:00
|
|
|
if (!icsk->icsk_ack.ato) {
|
2005-04-16 16:20:36 -06:00
|
|
|
/* The _first_ data packet received, initialize
|
|
|
|
* delayed ACK engine.
|
|
|
|
*/
|
2018-05-21 16:08:56 -06:00
|
|
|
tcp_incr_quickack(sk, TCP_MAX_QUICKACKS);
|
2005-08-09 21:10:42 -06:00
|
|
|
icsk->icsk_ack.ato = TCP_ATO_MIN;
|
2005-04-16 16:20:36 -06:00
|
|
|
} else {
|
2005-08-09 21:10:42 -06:00
|
|
|
int m = now - icsk->icsk_ack.lrcvtime;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2007-12-31 15:57:14 -07:00
|
|
|
if (m <= TCP_ATO_MIN / 2) {
|
2005-04-16 16:20:36 -06:00
|
|
|
/* The fastest case is the first. */
|
2005-08-09 21:10:42 -06:00
|
|
|
icsk->icsk_ack.ato = (icsk->icsk_ack.ato >> 1) + TCP_ATO_MIN / 2;
|
|
|
|
} else if (m < icsk->icsk_ack.ato) {
|
|
|
|
icsk->icsk_ack.ato = (icsk->icsk_ack.ato >> 1) + m;
|
|
|
|
if (icsk->icsk_ack.ato > icsk->icsk_rto)
|
|
|
|
icsk->icsk_ack.ato = icsk->icsk_rto;
|
|
|
|
} else if (m > icsk->icsk_rto) {
|
2005-11-10 18:13:47 -07:00
|
|
|
/* Too long gap. Apparently sender failed to
|
2005-04-16 16:20:36 -06:00
|
|
|
* restart window, so that we send ACKs quickly.
|
|
|
|
*/
|
2018-05-21 16:08:56 -06:00
|
|
|
tcp_incr_quickack(sk, TCP_MAX_QUICKACKS);
|
2007-12-31 01:11:19 -07:00
|
|
|
sk_mem_reclaim(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
2005-08-09 21:10:42 -06:00
|
|
|
icsk->icsk_ack.lrcvtime = now;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2018-06-04 16:29:51 -06:00
|
|
|
tcp_ecn_check_ce(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
if (skb->len >= 128)
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
tcp_grow_window(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Called to compute a smoothed rtt estimate. The data fed to this
|
|
|
|
* routine either comes from timestamps, or from segments that were
|
|
|
|
* known _not_ to have been retransmitted [see Karn/Partridge
|
|
|
|
* Proceedings SIGCOMM 87]. The algorithm is from the SIGCOMM 88
|
|
|
|
* piece by Van Jacobson.
|
|
|
|
* NOTE: the next three routines used to be one big routine.
|
|
|
|
* To save cycles in the RFC 1323 implementation it was better to break
|
|
|
|
* it up into three procedures. -- erics
|
|
|
|
*/
|
2014-02-26 15:02:48 -07:00
|
|
|
static void tcp_rtt_estimator(struct sock *sk, long mrtt_us)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-10 01:03:31 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2014-02-26 15:02:48 -07:00
|
|
|
long m = mrtt_us; /* RTT */
|
|
|
|
u32 srtt = tp->srtt_us;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* The following amusing code comes from Jacobson's
|
|
|
|
* article in SIGCOMM '88. Note that rtt and mdev
|
|
|
|
* are scaled versions of rtt and mean deviation.
|
2007-02-09 07:24:47 -07:00
|
|
|
* This is designed to be as fast as possible
|
2005-04-16 16:20:36 -06:00
|
|
|
* m stands for "measurement".
|
|
|
|
*
|
|
|
|
* On a 1990 paper the rto value is changed to:
|
|
|
|
* RTO = rtt + 4 * mdev
|
|
|
|
*
|
|
|
|
* Funny. This algorithm seems to be very broken.
|
|
|
|
* These formulae increase RTO, when it should be decreased, increase
|
2005-11-15 16:17:10 -07:00
|
|
|
* too slowly, when it should be increased quickly, decrease too quickly
|
2005-04-16 16:20:36 -06:00
|
|
|
* etc. I guess in BSD RTO takes ONE value, so that it is absolutely
|
|
|
|
* does not matter how to _calculate_ it. Seems, it was trap
|
|
|
|
* that VJ failed to avoid. 8)
|
|
|
|
*/
|
tcp: remove 1ms offset in srtt computation
TCP pacing depends on an accurate srtt estimation.
Current srtt estimation is using jiffie resolution,
and has an artificial offset of at least 1 ms, which can produce
slowdowns when FQ/pacing is used, especially in DC world,
where typical rtt is below 1 ms.
We are planning a switch to usec resolution for linux-3.15,
but in the meantime, this patch removes the 1 ms offset.
All we need is to have tp->srtt minimal value of 1 to differentiate
the case of srtt being initialized or not, not 8.
The problematic behavior was observed on a 40Gbit testbed,
where 32 concurrent netperf were reaching 12Gbps of aggregate
speed, instead of line speed.
This patch also has the effect of reporting more accurate srtt and send
rates to iproute2 ss command as in :
$ ss -i dst cca2
Netid State Recv-Q Send-Q Local Address:Port
Peer Address:Port
tcp ESTAB 0 0 10.244.129.1:56984
10.244.129.2:12865
cubic wscale:6,6 rto:200 rtt:0.25/0.25 ato:40 mss:1448 cwnd:10 send
463.4Mbps rcv_rtt:1 rcv_space:29200
tcp ESTAB 0 390960 10.244.129.1:60247
10.244.129.2:50204
cubic wscale:6,6 rto:200 rtt:0.875/0.75 mss:1448 cwnd:73 ssthresh:51
send 966.4Mbps unacked:73 retrans:0/121 rcv_space:29200
Reported-by: Vytautas Valancius <valas@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-02-06 16:57:10 -07:00
|
|
|
if (srtt != 0) {
|
|
|
|
m -= (srtt >> 3); /* m is now error in rtt est */
|
|
|
|
srtt += m; /* rtt = 7/8 rtt + 1/8 new */
|
2005-04-16 16:20:36 -06:00
|
|
|
if (m < 0) {
|
|
|
|
m = -m; /* m is now abs(error) */
|
2014-02-26 15:02:48 -07:00
|
|
|
m -= (tp->mdev_us >> 2); /* similar update on mdev */
|
2005-04-16 16:20:36 -06:00
|
|
|
/* This is similar to one of Eifel findings.
|
|
|
|
* Eifel blocks mdev updates when rtt decreases.
|
|
|
|
* This solution is a bit different: we use finer gain
|
|
|
|
* for mdev in this case (alpha*beta).
|
|
|
|
* Like Eifel it also prevents growth of rto,
|
|
|
|
* but also it limits too fast rto decreases,
|
|
|
|
* happening in pure Eifel.
|
|
|
|
*/
|
|
|
|
if (m > 0)
|
|
|
|
m >>= 3;
|
|
|
|
} else {
|
2014-02-26 15:02:48 -07:00
|
|
|
m -= (tp->mdev_us >> 2); /* similar update on mdev */
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2014-02-26 15:02:48 -07:00
|
|
|
tp->mdev_us += m; /* mdev = 3/4 mdev + 1/4 new */
|
|
|
|
if (tp->mdev_us > tp->mdev_max_us) {
|
|
|
|
tp->mdev_max_us = tp->mdev_us;
|
|
|
|
if (tp->mdev_max_us > tp->rttvar_us)
|
|
|
|
tp->rttvar_us = tp->mdev_max_us;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
if (after(tp->snd_una, tp->rtt_seq)) {
|
2014-02-26 15:02:48 -07:00
|
|
|
if (tp->mdev_max_us < tp->rttvar_us)
|
|
|
|
tp->rttvar_us -= (tp->rttvar_us - tp->mdev_max_us) >> 2;
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->rtt_seq = tp->snd_nxt;
|
2014-02-26 15:02:48 -07:00
|
|
|
tp->mdev_max_us = tcp_rto_min_us(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
} else {
|
|
|
|
/* no previous measure. */
|
tcp: remove 1ms offset in srtt computation
TCP pacing depends on an accurate srtt estimation.
Current srtt estimation is using jiffie resolution,
and has an artificial offset of at least 1 ms, which can produce
slowdowns when FQ/pacing is used, especially in DC world,
where typical rtt is below 1 ms.
We are planning a switch to usec resolution for linux-3.15,
but in the meantime, this patch removes the 1 ms offset.
All we need is to have tp->srtt minimal value of 1 to differentiate
the case of srtt being initialized or not, not 8.
The problematic behavior was observed on a 40Gbit testbed,
where 32 concurrent netperf were reaching 12Gbps of aggregate
speed, instead of line speed.
This patch also has the effect of reporting more accurate srtt and send
rates to iproute2 ss command as in :
$ ss -i dst cca2
Netid State Recv-Q Send-Q Local Address:Port
Peer Address:Port
tcp ESTAB 0 0 10.244.129.1:56984
10.244.129.2:12865
cubic wscale:6,6 rto:200 rtt:0.25/0.25 ato:40 mss:1448 cwnd:10 send
463.4Mbps rcv_rtt:1 rcv_space:29200
tcp ESTAB 0 390960 10.244.129.1:60247
10.244.129.2:50204
cubic wscale:6,6 rto:200 rtt:0.875/0.75 mss:1448 cwnd:73 ssthresh:51
send 966.4Mbps unacked:73 retrans:0/121 rcv_space:29200
Reported-by: Vytautas Valancius <valas@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-02-06 16:57:10 -07:00
|
|
|
srtt = m << 3; /* take the measured time to be rtt */
|
2014-02-26 15:02:48 -07:00
|
|
|
tp->mdev_us = m << 1; /* make sure rto = 3*rtt */
|
|
|
|
tp->rttvar_us = max(tp->mdev_us, tcp_rto_min_us(sk));
|
|
|
|
tp->mdev_max_us = tp->rttvar_us;
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->rtt_seq = tp->snd_nxt;
|
|
|
|
}
|
2014-02-26 15:02:48 -07:00
|
|
|
tp->srtt_us = max(1U, srtt);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
tcp: TSO packets automatic sizing
After hearing many people over past years complaining against TSO being
bursty or even buggy, we are proud to present automatic sizing of TSO
packets.
One part of the problem is that tcp_tso_should_defer() uses an heuristic
relying on upcoming ACKS instead of a timer, but more generally, having
big TSO packets makes little sense for low rates, as it tends to create
micro bursts on the network, and general consensus is to reduce the
buffering amount.
This patch introduces a per socket sk_pacing_rate, that approximates
the current sending rate, and allows us to size the TSO packets so
that we try to send one packet every ms.
This field could be set by other transports.
Patch has no impact for high speed flows, where having large TSO packets
makes sense to reach line rate.
For other flows, this helps better packet scheduling and ACK clocking.
This patch increases performance of TCP flows in lossy environments.
A new sysctl (tcp_min_tso_segs) is added, to specify the
minimal size of a TSO packet (default being 2).
A follow-up patch will provide a new packet scheduler (FQ), using
sk_pacing_rate as an input to perform optional per flow pacing.
This explains why we chose to set sk_pacing_rate to twice the current
rate, allowing 'slow start' ramp up.
sk_pacing_rate = 2 * cwnd * mss / srtt
v2: Neal Cardwell reported a suspect deferring of last two segments on
initial write of 10 MSS, I had to change tcp_tso_should_defer() to take
into account tp->xmit_size_goal_segs
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Van Jacobson <vanj@google.com>
Cc: Tom Herbert <therbert@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-27 06:46:32 -06:00
|
|
|
static void tcp_update_pacing_rate(struct sock *sk)
|
|
|
|
{
|
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
u64 rate;
|
|
|
|
|
|
|
|
/* set sk_pacing_rate to 200 % of current rate (mss * cwnd / srtt) */
|
2015-08-21 18:38:02 -06:00
|
|
|
rate = (u64)tp->mss_cache * ((USEC_PER_SEC / 100) << 3);
|
|
|
|
|
|
|
|
/* current rate is (cwnd * mss) / srtt
|
|
|
|
* In Slow Start [1], set sk_pacing_rate to 200 % the current rate.
|
|
|
|
* In Congestion Avoidance phase, set it to 120 % the current rate.
|
|
|
|
*
|
|
|
|
* [1] : Normal Slow Start condition is (tp->snd_cwnd < tp->snd_ssthresh)
|
|
|
|
* If snd_cwnd >= (tp->snd_ssthresh / 2), we are approaching
|
|
|
|
* end of slow start and should slow down.
|
|
|
|
*/
|
|
|
|
if (tp->snd_cwnd < tp->snd_ssthresh / 2)
|
2017-10-27 08:47:31 -06:00
|
|
|
rate *= sock_net(sk)->ipv4.sysctl_tcp_pacing_ss_ratio;
|
2015-08-21 18:38:02 -06:00
|
|
|
else
|
2017-10-27 08:47:32 -06:00
|
|
|
rate *= sock_net(sk)->ipv4.sysctl_tcp_pacing_ca_ratio;
|
tcp: TSO packets automatic sizing
After hearing many people over past years complaining against TSO being
bursty or even buggy, we are proud to present automatic sizing of TSO
packets.
One part of the problem is that tcp_tso_should_defer() uses an heuristic
relying on upcoming ACKS instead of a timer, but more generally, having
big TSO packets makes little sense for low rates, as it tends to create
micro bursts on the network, and general consensus is to reduce the
buffering amount.
This patch introduces a per socket sk_pacing_rate, that approximates
the current sending rate, and allows us to size the TSO packets so
that we try to send one packet every ms.
This field could be set by other transports.
Patch has no impact for high speed flows, where having large TSO packets
makes sense to reach line rate.
For other flows, this helps better packet scheduling and ACK clocking.
This patch increases performance of TCP flows in lossy environments.
A new sysctl (tcp_min_tso_segs) is added, to specify the
minimal size of a TSO packet (default being 2).
A follow-up patch will provide a new packet scheduler (FQ), using
sk_pacing_rate as an input to perform optional per flow pacing.
This explains why we chose to set sk_pacing_rate to twice the current
rate, allowing 'slow start' ramp up.
sk_pacing_rate = 2 * cwnd * mss / srtt
v2: Neal Cardwell reported a suspect deferring of last two segments on
initial write of 10 MSS, I had to change tcp_tso_should_defer() to take
into account tp->xmit_size_goal_segs
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Van Jacobson <vanj@google.com>
Cc: Tom Herbert <therbert@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-27 06:46:32 -06:00
|
|
|
|
|
|
|
rate *= max(tp->snd_cwnd, tp->packets_out);
|
|
|
|
|
2014-02-26 15:02:48 -07:00
|
|
|
if (likely(tp->srtt_us))
|
|
|
|
do_div(rate, tp->srtt_us);
|
tcp: TSO packets automatic sizing
After hearing many people over past years complaining against TSO being
bursty or even buggy, we are proud to present automatic sizing of TSO
packets.
One part of the problem is that tcp_tso_should_defer() uses an heuristic
relying on upcoming ACKS instead of a timer, but more generally, having
big TSO packets makes little sense for low rates, as it tends to create
micro bursts on the network, and general consensus is to reduce the
buffering amount.
This patch introduces a per socket sk_pacing_rate, that approximates
the current sending rate, and allows us to size the TSO packets so
that we try to send one packet every ms.
This field could be set by other transports.
Patch has no impact for high speed flows, where having large TSO packets
makes sense to reach line rate.
For other flows, this helps better packet scheduling and ACK clocking.
This patch increases performance of TCP flows in lossy environments.
A new sysctl (tcp_min_tso_segs) is added, to specify the
minimal size of a TSO packet (default being 2).
A follow-up patch will provide a new packet scheduler (FQ), using
sk_pacing_rate as an input to perform optional per flow pacing.
This explains why we chose to set sk_pacing_rate to twice the current
rate, allowing 'slow start' ramp up.
sk_pacing_rate = 2 * cwnd * mss / srtt
v2: Neal Cardwell reported a suspect deferring of last two segments on
initial write of 10 MSS, I had to change tcp_tso_should_defer() to take
into account tp->xmit_size_goal_segs
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Van Jacobson <vanj@google.com>
Cc: Tom Herbert <therbert@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-27 06:46:32 -06:00
|
|
|
|
locking/atomics, net/ipv4/tcp_input.c: Convert ACCESS_ONCE() to READ_ONCE()/WRITE_ONCE()
For several reasons, it is desirable to use {READ,WRITE}_ONCE() in
preference to ACCESS_ONCE(), and new code is expected to use one of the
former. So far, there's been no reason to change most existing uses of
ACCESS_ONCE(), as these aren't currently harmful.
However, for some features it is necessary to instrument reads and
writes separately, which is not possible with ACCESS_ONCE(). This
distinction is critical to correct operation.
It's possible to transform the bulk of kernel code using the Coccinelle
script below. However, this doesn't handle comments, leaving references
to ACCESS_ONCE() instances which have been removed. As a preparatory
step, this patch converts the IPv4 TCP input code and comments to use
{READ,WRITE}_ONCE() consistently.
----
virtual patch
@ depends on patch @
expression E1, E2;
@@
- ACCESS_ONCE(E1) = E2
+ WRITE_ONCE(E1, E2)
@ depends on patch @
expression E;
@@
- ACCESS_ONCE(E)
+ READ_ONCE(E)
----
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-arch@vger.kernel.org
Cc: mpe@ellerman.id.au
Cc: shuah@kernel.org
Cc: snitzer@redhat.com
Cc: thor.thayer@linux.intel.com
Cc: tj@kernel.org
Cc: viro@zeniv.linux.org.uk
Cc: will.deacon@arm.com
Link: http://lkml.kernel.org/r/1508792849-3115-8-git-send-email-paulmck@linux.vnet.ibm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-10-23 15:07:18 -06:00
|
|
|
/* WRITE_ONCE() is needed because sch_fq fetches sk_pacing_rate
|
2013-10-09 18:14:52 -06:00
|
|
|
* without any lock. We want to make sure compiler wont store
|
|
|
|
* intermediate values in this location.
|
|
|
|
*/
|
locking/atomics, net/ipv4/tcp_input.c: Convert ACCESS_ONCE() to READ_ONCE()/WRITE_ONCE()
For several reasons, it is desirable to use {READ,WRITE}_ONCE() in
preference to ACCESS_ONCE(), and new code is expected to use one of the
former. So far, there's been no reason to change most existing uses of
ACCESS_ONCE(), as these aren't currently harmful.
However, for some features it is necessary to instrument reads and
writes separately, which is not possible with ACCESS_ONCE(). This
distinction is critical to correct operation.
It's possible to transform the bulk of kernel code using the Coccinelle
script below. However, this doesn't handle comments, leaving references
to ACCESS_ONCE() instances which have been removed. As a preparatory
step, this patch converts the IPv4 TCP input code and comments to use
{READ,WRITE}_ONCE() consistently.
----
virtual patch
@ depends on patch @
expression E1, E2;
@@
- ACCESS_ONCE(E1) = E2
+ WRITE_ONCE(E1, E2)
@ depends on patch @
expression E;
@@
- ACCESS_ONCE(E)
+ READ_ONCE(E)
----
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-arch@vger.kernel.org
Cc: mpe@ellerman.id.au
Cc: shuah@kernel.org
Cc: snitzer@redhat.com
Cc: thor.thayer@linux.intel.com
Cc: tj@kernel.org
Cc: viro@zeniv.linux.org.uk
Cc: will.deacon@arm.com
Link: http://lkml.kernel.org/r/1508792849-3115-8-git-send-email-paulmck@linux.vnet.ibm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-10-23 15:07:18 -06:00
|
|
|
WRITE_ONCE(sk->sk_pacing_rate, min_t(u64, rate,
|
|
|
|
sk->sk_max_pacing_rate));
|
tcp: TSO packets automatic sizing
After hearing many people over past years complaining against TSO being
bursty or even buggy, we are proud to present automatic sizing of TSO
packets.
One part of the problem is that tcp_tso_should_defer() uses an heuristic
relying on upcoming ACKS instead of a timer, but more generally, having
big TSO packets makes little sense for low rates, as it tends to create
micro bursts on the network, and general consensus is to reduce the
buffering amount.
This patch introduces a per socket sk_pacing_rate, that approximates
the current sending rate, and allows us to size the TSO packets so
that we try to send one packet every ms.
This field could be set by other transports.
Patch has no impact for high speed flows, where having large TSO packets
makes sense to reach line rate.
For other flows, this helps better packet scheduling and ACK clocking.
This patch increases performance of TCP flows in lossy environments.
A new sysctl (tcp_min_tso_segs) is added, to specify the
minimal size of a TSO packet (default being 2).
A follow-up patch will provide a new packet scheduler (FQ), using
sk_pacing_rate as an input to perform optional per flow pacing.
This explains why we chose to set sk_pacing_rate to twice the current
rate, allowing 'slow start' ramp up.
sk_pacing_rate = 2 * cwnd * mss / srtt
v2: Neal Cardwell reported a suspect deferring of last two segments on
initial write of 10 MSS, I had to change tcp_tso_should_defer() to take
into account tp->xmit_size_goal_segs
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Van Jacobson <vanj@google.com>
Cc: Tom Herbert <therbert@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-27 06:46:32 -06:00
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Calculate rto without backoff. This is the second half of Van Jacobson's
|
|
|
|
* routine referred to above.
|
|
|
|
*/
|
2013-12-29 12:39:51 -07:00
|
|
|
static void tcp_set_rto(struct sock *sk)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-09 21:10:42 -06:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Old crap is replaced with new one. 8)
|
|
|
|
*
|
|
|
|
* More seriously:
|
|
|
|
* 1. If rtt variance happened to be less 50msec, it is hallucination.
|
|
|
|
* It cannot be less due to utterly erratic ACK generation made
|
|
|
|
* at least by solaris and freebsd. "Erratic ACKs" has _nothing_
|
|
|
|
* to do with delayed acks, because at cwnd>2 true delack timeout
|
|
|
|
* is invisible. Actually, Linux-2.4 also generates erratic
|
2005-11-10 18:13:47 -07:00
|
|
|
* ACKs in some circumstances.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
Revert Backoff [v3]: Revert RTO on ICMP destination unreachable
Here, an ICMP host/network unreachable message, whose payload fits to
TCP's SND.UNA, is taken as an indication that the RTO retransmission has
not been lost due to congestion, but because of a route failure
somewhere along the path.
With true congestion, a router won't trigger such a message and the
patched TCP will operate as standard TCP.
This patch reverts one RTO backoff, if an ICMP host/network unreachable
message, whose payload fits to TCP's SND.UNA, arrives.
Based on the new RTO, the retransmission timer is reset to reflect the
remaining time, or - if the revert clocked out the timer - a retransmission
is sent out immediately.
Backoffs are only reverted, if TCP is in RTO loss recovery, i.e. if
there have been retransmissions and reversible backoffs, already.
Changes from v2:
1) Renaming of skb in tcp_v4_err() moved to another patch.
2) Reintroduced tcp_bound_rto() and __tcp_set_rto().
3) Fixed code comments.
Signed-off-by: Damian Lukowski <damian@tvk.rwth-aachen.de>
Acked-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-08-25 18:16:31 -06:00
|
|
|
inet_csk(sk)->icsk_rto = __tcp_set_rto(tp);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* 2. Fixups made earlier cannot be right.
|
|
|
|
* If we do not estimate RTO correctly without them,
|
|
|
|
* all the algo is pure shit and should be replaced
|
2005-11-10 18:13:47 -07:00
|
|
|
* with correct one. It is exactly, which we pretend to do.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
|
|
|
|
2008-12-05 23:43:08 -07:00
|
|
|
/* NOTE: clamping at TCP_RTO_MIN is not required, current algo
|
|
|
|
* guarantees that rto is higher.
|
|
|
|
*/
|
Revert Backoff [v3]: Revert RTO on ICMP destination unreachable
Here, an ICMP host/network unreachable message, whose payload fits to
TCP's SND.UNA, is taken as an indication that the RTO retransmission has
not been lost due to congestion, but because of a route failure
somewhere along the path.
With true congestion, a router won't trigger such a message and the
patched TCP will operate as standard TCP.
This patch reverts one RTO backoff, if an ICMP host/network unreachable
message, whose payload fits to TCP's SND.UNA, arrives.
Based on the new RTO, the retransmission timer is reset to reflect the
remaining time, or - if the revert clocked out the timer - a retransmission
is sent out immediately.
Backoffs are only reverted, if TCP is in RTO loss recovery, i.e. if
there have been retransmissions and reversible backoffs, already.
Changes from v2:
1) Renaming of skb in tcp_v4_err() moved to another patch.
2) Reintroduced tcp_bound_rto() and __tcp_set_rto().
3) Fixed code comments.
Signed-off-by: Damian Lukowski <damian@tvk.rwth-aachen.de>
Acked-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-08-25 18:16:31 -06:00
|
|
|
tcp_bound_rto(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2011-10-21 03:22:42 -06:00
|
|
|
__u32 tcp_init_cwnd(const struct tcp_sock *tp, const struct dst_entry *dst)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
__u32 cwnd = (dst ? dst_metric(dst, RTAX_INITCWND) : 0);
|
|
|
|
|
2010-08-29 13:23:12 -06:00
|
|
|
if (!cwnd)
|
2011-02-02 18:05:11 -07:00
|
|
|
cwnd = TCP_INIT_CWND;
|
2005-04-16 16:20:36 -06:00
|
|
|
return min_t(__u32, cwnd, tp->snd_cwnd_clamp);
|
|
|
|
}
|
|
|
|
|
2007-10-26 00:03:52 -06:00
|
|
|
/* Take a notice that peer is sending D-SACKs */
|
2007-08-09 06:14:46 -06:00
|
|
|
static void tcp_dsack_seen(struct tcp_sock *tp)
|
|
|
|
{
|
2011-12-20 06:23:24 -07:00
|
|
|
tp->rx_opt.sack_ok |= TCP_DSACK_SEEN;
|
2017-11-03 17:38:48 -06:00
|
|
|
tp->rack.dsack_seen = 1;
|
2018-07-31 18:46:23 -06:00
|
|
|
tp->dsack_dups++;
|
2007-08-09 06:14:46 -06:00
|
|
|
}
|
|
|
|
|
2017-11-08 14:01:27 -07:00
|
|
|
/* It's reordering when higher sequence was delivered (i.e. sacked) before
|
|
|
|
* some lower never-retransmitted sequence ("low_seq"). The maximum reordering
|
|
|
|
* distance is approximated in full-mss packet distance ("reordering").
|
|
|
|
*/
|
|
|
|
static void tcp_check_sack_reordering(struct sock *sk, const u32 low_seq,
|
|
|
|
const int ts)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-10 01:03:31 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2017-11-08 14:01:27 -07:00
|
|
|
const u32 mss = tp->mss_cache;
|
|
|
|
u32 fack, metric;
|
2008-07-03 02:05:41 -06:00
|
|
|
|
2017-11-08 14:01:27 -07:00
|
|
|
fack = tcp_highest_sack_seq(tp);
|
|
|
|
if (!before(low_seq, fack))
|
2017-05-16 15:39:02 -06:00
|
|
|
return;
|
|
|
|
|
2017-11-08 14:01:27 -07:00
|
|
|
metric = fack - low_seq;
|
|
|
|
if ((metric > tp->reordering * mss) && mss) {
|
2005-04-16 16:20:36 -06:00
|
|
|
#if FASTRETRANS_DEBUG > 1
|
2012-05-15 08:11:54 -06:00
|
|
|
pr_debug("Disorder%d %d %u f%u s%u rr%d\n",
|
|
|
|
tp->rx_opt.sack_ok, inet_csk(sk)->icsk_ca_state,
|
|
|
|
tp->reordering,
|
2017-11-08 14:01:27 -07:00
|
|
|
0,
|
2012-05-15 08:11:54 -06:00
|
|
|
tp->sacked_out,
|
|
|
|
tp->undo_marker ? tp->undo_retrans : 0);
|
2005-04-16 16:20:36 -06:00
|
|
|
#endif
|
2017-11-08 14:01:27 -07:00
|
|
|
tp->reordering = min_t(u32, (metric + mss - 1) / mss,
|
|
|
|
sock_net(sk)->ipv4.sysctl_tcp_max_reordering);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2012-05-02 07:30:03 -06:00
|
|
|
|
2017-04-04 15:15:40 -06:00
|
|
|
/* This exciting event is worth to be remembered. 8) */
|
2018-07-31 18:46:24 -06:00
|
|
|
tp->reord_seen++;
|
2017-11-08 14:01:27 -07:00
|
|
|
NET_INC_STATS(sock_net(sk),
|
|
|
|
ts ? LINUX_MIB_TCPTSREORDER : LINUX_MIB_TCPSACKREORDER);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2008-09-20 22:20:20 -06:00
|
|
|
/* This must be called before lost_out is incremented */
|
2008-09-20 22:18:55 -06:00
|
|
|
static void tcp_verify_retransmit_hint(struct tcp_sock *tp, struct sk_buff *skb)
|
|
|
|
{
|
2015-04-03 02:17:26 -06:00
|
|
|
if (!tp->retransmit_skb_hint ||
|
2008-09-20 22:18:55 -06:00
|
|
|
before(TCP_SKB_CB(skb)->seq,
|
|
|
|
TCP_SKB_CB(tp->retransmit_skb_hint)->seq))
|
2008-09-20 22:20:20 -06:00
|
|
|
tp->retransmit_skb_hint = skb;
|
2008-09-20 22:18:55 -06:00
|
|
|
}
|
|
|
|
|
2016-09-19 21:39:13 -06:00
|
|
|
/* Sum the number of packets on the wire we have marked as lost.
|
|
|
|
* There are two cases we care about here:
|
|
|
|
* a) Packet hasn't been marked lost (nor retransmitted),
|
|
|
|
* and this is the first loss.
|
|
|
|
* b) Packet has been marked both lost and retransmitted,
|
|
|
|
* and this means we think it was lost again.
|
|
|
|
*/
|
|
|
|
static void tcp_sum_lost(struct tcp_sock *tp, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
__u8 sacked = TCP_SKB_CB(skb)->sacked;
|
|
|
|
|
|
|
|
if (!(sacked & TCPCB_LOST) ||
|
|
|
|
((sacked & TCPCB_LOST) && (sacked & TCPCB_SACKED_RETRANS)))
|
|
|
|
tp->lost += tcp_skb_pcount(skb);
|
|
|
|
}
|
|
|
|
|
2008-09-20 22:19:22 -06:00
|
|
|
static void tcp_skb_mark_lost(struct tcp_sock *tp, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
if (!(TCP_SKB_CB(skb)->sacked & (TCPCB_LOST|TCPCB_SACKED_ACKED))) {
|
|
|
|
tcp_verify_retransmit_hint(tp, skb);
|
|
|
|
|
|
|
|
tp->lost_out += tcp_skb_pcount(skb);
|
2016-09-19 21:39:13 -06:00
|
|
|
tcp_sum_lost(tp, skb);
|
2008-09-20 22:19:22 -06:00
|
|
|
TCP_SKB_CB(skb)->sacked |= TCPCB_LOST;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-10-16 22:57:47 -06:00
|
|
|
void tcp_skb_mark_lost_uncond_verify(struct tcp_sock *tp, struct sk_buff *skb)
|
2008-09-20 22:20:20 -06:00
|
|
|
{
|
|
|
|
tcp_verify_retransmit_hint(tp, skb);
|
|
|
|
|
2016-09-19 21:39:13 -06:00
|
|
|
tcp_sum_lost(tp, skb);
|
2008-09-20 22:20:20 -06:00
|
|
|
if (!(TCP_SKB_CB(skb)->sacked & (TCPCB_LOST|TCPCB_SACKED_ACKED))) {
|
|
|
|
tp->lost_out += tcp_skb_pcount(skb);
|
|
|
|
TCP_SKB_CB(skb)->sacked |= TCPCB_LOST;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* This procedure tags the retransmission queue when SACKs arrive.
|
|
|
|
*
|
|
|
|
* We have three tag bits: SACKED(S), RETRANS(R) and LOST(L).
|
|
|
|
* Packets in queue with these bits set are counted in variables
|
|
|
|
* sacked_out, retrans_out and lost_out, correspondingly.
|
|
|
|
*
|
|
|
|
* Valid combinations are:
|
|
|
|
* Tag InFlight Description
|
|
|
|
* 0 1 - orig segment is in flight.
|
|
|
|
* S 0 - nothing flies, orig reached receiver.
|
|
|
|
* L 0 - nothing flies, orig lost by net.
|
|
|
|
* R 2 - both orig and retransmit are in flight.
|
|
|
|
* L|R 1 - orig is lost, retransmit is in flight.
|
|
|
|
* S|R 1 - orig reached receiver, retrans is still in flight.
|
|
|
|
* (L|S|R is logically valid, it could occur when L|R is sacked,
|
|
|
|
* but it is equivalent to plain S and code short-curcuits it to S.
|
|
|
|
* L|S is logically invalid, it would mean -1 packet in flight 8))
|
|
|
|
*
|
|
|
|
* These 6 states form finite state machine, controlled by the following events:
|
|
|
|
* 1. New ACK (+SACK) arrives. (tcp_sacktag_write_queue())
|
|
|
|
* 2. Retransmission. (tcp_retransmit_skb(), tcp_xmit_retransmit_queue())
|
2012-01-19 07:42:21 -07:00
|
|
|
* 3. Loss detection event of two flavors:
|
2005-04-16 16:20:36 -06:00
|
|
|
* A. Scoreboard estimator decided the packet is lost.
|
|
|
|
* A'. Reno "three dupacks" marks head of queue lost.
|
2012-01-19 07:42:21 -07:00
|
|
|
* B. SACK arrives sacking SND.NXT at the moment, when the
|
2005-04-16 16:20:36 -06:00
|
|
|
* segment was retransmitted.
|
|
|
|
* 4. D-SACK added new rule: D-SACK changes any tag to S.
|
|
|
|
*
|
|
|
|
* It is pleasant to note, that state diagram turns out to be commutative,
|
|
|
|
* so that we are allowed not to be bothered by order of our actions,
|
|
|
|
* when multiple events arrive simultaneously. (see the function below).
|
|
|
|
*
|
|
|
|
* Reordering detection.
|
|
|
|
* --------------------
|
|
|
|
* Reordering metric is maximal distance, which a packet can be displaced
|
|
|
|
* in packet stream. With SACKs we can estimate it:
|
|
|
|
*
|
|
|
|
* 1. SACK fills old hole and the corresponding segment was not
|
|
|
|
* ever retransmitted -> reordering. Alas, we cannot use it
|
|
|
|
* when segment was retransmitted.
|
|
|
|
* 2. The last flaw is solved with D-SACK. D-SACK arrives
|
|
|
|
* for retransmitted and already SACKed segment -> reordering..
|
|
|
|
* Both of these heuristics are not used in Loss state, when we cannot
|
|
|
|
* account for retransmits accurately.
|
2007-08-24 23:54:44 -06:00
|
|
|
*
|
|
|
|
* SACK block validation.
|
|
|
|
* ----------------------
|
|
|
|
*
|
|
|
|
* SACK block range validation checks that the received SACK block fits to
|
|
|
|
* the expected sequence limits, i.e., it is between SND.UNA and SND.NXT.
|
|
|
|
* Note that SND.UNA is not included to the range though being valid because
|
2007-10-01 16:28:17 -06:00
|
|
|
* it means that the receiver is rather inconsistent with itself reporting
|
|
|
|
* SACK reneging when it should advance SND.UNA. Such SACK block this is
|
|
|
|
* perfectly valid, however, in light of RFC2018 which explicitly states
|
|
|
|
* that "SACK block MUST reflect the newest segment. Even if the newest
|
|
|
|
* segment is going to be discarded ...", not that it looks very clever
|
|
|
|
* in case of head skb. Due to potentional receiver driven attacks, we
|
|
|
|
* choose to avoid immediate execution of a walk in write queue due to
|
|
|
|
* reneging and defer head skb's loss recovery to standard loss recovery
|
|
|
|
* procedure that will eventually trigger (nothing forbids us doing this).
|
2007-08-24 23:54:44 -06:00
|
|
|
*
|
|
|
|
* Implements also blockage to start_seq wrap-around. Problem lies in the
|
|
|
|
* fact that though start_seq (s) is before end_seq (i.e., not reversed),
|
|
|
|
* there's no guarantee that it will be before snd_nxt (n). The problem
|
|
|
|
* happens when start_seq resides between end_seq wrap (e_w) and snd_nxt
|
|
|
|
* wrap (s_w):
|
|
|
|
*
|
|
|
|
* <- outs wnd -> <- wrapzone ->
|
|
|
|
* u e n u_w e_w s n_w
|
|
|
|
* | | | | | | |
|
|
|
|
* |<------------+------+----- TCP seqno space --------------+---------->|
|
|
|
|
* ...-- <2^31 ->| |<--------...
|
|
|
|
* ...---- >2^31 ------>| |<--------...
|
|
|
|
*
|
|
|
|
* Current code wouldn't be vulnerable but it's better still to discard such
|
|
|
|
* crazy SACK blocks. Doing this check for start_seq alone closes somewhat
|
|
|
|
* similar case (end_seq after snd_nxt wrap) as earlier reversed check in
|
|
|
|
* snd_nxt wrap -> snd_una region will then become "well defined", i.e.,
|
|
|
|
* equal to the ideal case (infinite seqno space without wrap caused issues).
|
|
|
|
*
|
|
|
|
* With D-SACK the lower bound is extended to cover sequence space below
|
|
|
|
* SND.UNA down to undo_marker, which is the last point of interest. Yet
|
2007-10-26 00:03:52 -06:00
|
|
|
* again, D-SACK block must not to go across snd_una (for the same reason as
|
2007-08-24 23:54:44 -06:00
|
|
|
* for the normal SACK blocks, explained above). But there all simplicity
|
|
|
|
* ends, TCP might receive valid D-SACKs below that. As long as they reside
|
|
|
|
* fully below undo_marker they do not affect behavior in anyway and can
|
|
|
|
* therefore be safely ignored. In rare cases (which are more or less
|
|
|
|
* theoretical ones), the D-SACK will nicely cross that boundary due to skb
|
|
|
|
* fragmentation and packet reordering past skb's retransmission. To consider
|
|
|
|
* them correctly, the acceptable range must be extended even more though
|
|
|
|
* the exact amount is rather hard to quantify. However, tp->max_window can
|
|
|
|
* be used as an exaggerated estimate.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
2012-05-16 17:15:34 -06:00
|
|
|
static bool tcp_is_sackblock_valid(struct tcp_sock *tp, bool is_dsack,
|
|
|
|
u32 start_seq, u32 end_seq)
|
2007-08-24 23:54:44 -06:00
|
|
|
{
|
|
|
|
/* Too far in future, or reversed (interpretation is ambiguous) */
|
|
|
|
if (after(end_seq, tp->snd_nxt) || !before(start_seq, end_seq))
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2007-08-24 23:54:44 -06:00
|
|
|
|
|
|
|
/* Nasty start_seq wrap-around check (see comments above) */
|
|
|
|
if (!before(start_seq, tp->snd_nxt))
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2007-08-24 23:54:44 -06:00
|
|
|
|
2007-10-26 00:03:52 -06:00
|
|
|
/* In outstanding window? ...This is valid exit for D-SACKs too.
|
2007-08-24 23:54:44 -06:00
|
|
|
* start_seq == snd_una is non-sensical (see comments above)
|
|
|
|
*/
|
|
|
|
if (after(start_seq, tp->snd_una))
|
2012-05-16 17:15:34 -06:00
|
|
|
return true;
|
2007-08-24 23:54:44 -06:00
|
|
|
|
|
|
|
if (!is_dsack || !tp->undo_marker)
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2007-08-24 23:54:44 -06:00
|
|
|
|
|
|
|
/* ...Then it's D-SACK, and must reside below snd_una completely */
|
2011-09-18 20:37:34 -06:00
|
|
|
if (after(end_seq, tp->snd_una))
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2007-08-24 23:54:44 -06:00
|
|
|
|
|
|
|
if (!before(start_seq, tp->undo_marker))
|
2012-05-16 17:15:34 -06:00
|
|
|
return true;
|
2007-08-24 23:54:44 -06:00
|
|
|
|
|
|
|
/* Too old */
|
|
|
|
if (!after(end_seq, tp->undo_marker))
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2007-08-24 23:54:44 -06:00
|
|
|
|
|
|
|
/* Undo_marker boundary crossing (overestimates a lot). Known already:
|
|
|
|
* start_seq < undo_marker and end_seq >= undo_marker.
|
|
|
|
*/
|
|
|
|
return !before(start_seq, end_seq - tp->max_window);
|
|
|
|
}
|
|
|
|
|
2012-05-16 17:15:34 -06:00
|
|
|
static bool tcp_check_dsack(struct sock *sk, const struct sk_buff *ack_skb,
|
|
|
|
struct tcp_sack_block_wire *sp, int num_sacks,
|
|
|
|
u32 prior_snd_una)
|
2007-06-18 23:43:06 -06:00
|
|
|
{
|
2008-07-16 21:29:51 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2008-05-02 17:26:16 -06:00
|
|
|
u32 start_seq_0 = get_unaligned_be32(&sp[0].start_seq);
|
|
|
|
u32 end_seq_0 = get_unaligned_be32(&sp[0].end_seq);
|
2012-05-16 17:15:34 -06:00
|
|
|
bool dup_sack = false;
|
2007-06-18 23:43:06 -06:00
|
|
|
|
|
|
|
if (before(start_seq_0, TCP_SKB_CB(ack_skb)->ack_seq)) {
|
2012-05-16 17:15:34 -06:00
|
|
|
dup_sack = true;
|
2007-08-09 06:14:46 -06:00
|
|
|
tcp_dsack_seen(tp);
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPDSACKRECV);
|
2007-06-18 23:43:06 -06:00
|
|
|
} else if (num_sacks > 1) {
|
2008-05-02 17:26:16 -06:00
|
|
|
u32 end_seq_1 = get_unaligned_be32(&sp[1].end_seq);
|
|
|
|
u32 start_seq_1 = get_unaligned_be32(&sp[1].start_seq);
|
2007-06-18 23:43:06 -06:00
|
|
|
|
|
|
|
if (!after(end_seq_0, end_seq_1) &&
|
|
|
|
!before(start_seq_0, start_seq_1)) {
|
2012-05-16 17:15:34 -06:00
|
|
|
dup_sack = true;
|
2007-08-09 06:14:46 -06:00
|
|
|
tcp_dsack_seen(tp);
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk),
|
2008-07-16 21:31:16 -06:00
|
|
|
LINUX_MIB_TCPDSACKOFORECV);
|
2007-06-18 23:43:06 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* D-SACK for already forgotten data... Do dumb counting. */
|
2014-07-02 13:07:16 -06:00
|
|
|
if (dup_sack && tp->undo_marker && tp->undo_retrans > 0 &&
|
2007-06-18 23:43:06 -06:00
|
|
|
!after(end_seq_0, prior_snd_una) &&
|
|
|
|
after(end_seq_0, tp->undo_marker))
|
|
|
|
tp->undo_retrans--;
|
|
|
|
|
|
|
|
return dup_sack;
|
|
|
|
}
|
|
|
|
|
2008-12-05 23:42:22 -07:00
|
|
|
struct tcp_sacktag_state {
|
2017-11-08 14:01:27 -07:00
|
|
|
u32 reord;
|
2015-04-30 17:10:58 -06:00
|
|
|
/* Timestamps for earliest and latest never-retransmitted segment
|
|
|
|
* that was SACKed. RTO needs the earliest RTT to stay conservative,
|
|
|
|
* but congestion control should still get an accurate delay signal.
|
|
|
|
*/
|
2017-05-16 15:00:14 -06:00
|
|
|
u64 first_sackt;
|
|
|
|
u64 last_sackt;
|
tcp: track data delivery rate for a TCP connection
This patch generates data delivery rate (throughput) samples on a
per-ACK basis. These rate samples can be used by congestion control
modules, and specifically will be used by TCP BBR in later patches in
this series.
Key state:
tp->delivered: Tracks the total number of data packets (original or not)
delivered so far. This is an already-existing field.
tp->delivered_mstamp: the last time tp->delivered was updated.
Algorithm:
A rate sample is calculated as (d1 - d0)/(t1 - t0) on a per-ACK basis:
d1: the current tp->delivered after processing the ACK
t1: the current time after processing the ACK
d0: the prior tp->delivered when the acked skb was transmitted
t0: the prior tp->delivered_mstamp when the acked skb was transmitted
When an skb is transmitted, we snapshot d0 and t0 in its control
block in tcp_rate_skb_sent().
When an ACK arrives, it may SACK and ACK some skbs. For each SACKed
or ACKed skb, tcp_rate_skb_delivered() updates the rate_sample struct
to reflect the latest (d0, t0).
Finally, tcp_rate_gen() generates a rate sample by storing
(d1 - d0) in rs->delivered and (t1 - t0) in rs->interval_us.
One caveat: if an skb was sent with no packets in flight, then
tp->delivered_mstamp may be either invalid (if the connection is
starting) or outdated (if the connection was idle). In that case,
we'll re-stamp tp->delivered_mstamp.
At first glance it seems t0 should always be the time when an skb was
transmitted, but actually this could over-estimate the rate due to
phase mismatch between transmit and ACK events. To track the delivery
rate, we ensure that if packets are in flight then t0 and and t1 are
times at which packets were marked delivered.
If the initial and final RTTs are different then one may be corrupted
by some sort of noise. The noise we see most often is sending gaps
caused by delayed, compressed, or stretched acks. This either affects
both RTTs equally or artificially reduces the final RTT. We approach
this by recording the info we need to compute the initial RTT
(duration of the "send phase" of the window) when we recorded the
associated inflight. Then, for a filter to avoid bandwidth
overestimates, we generalize the per-sample bandwidth computation
from:
bw = delivered / ack_phase_rtt
to the following:
bw = delivered / max(send_phase_rtt, ack_phase_rtt)
In large-scale experiments, this filtering approach incorporating
send_phase_rtt is effective at avoiding bandwidth overestimates due to
ACK compression or stretched ACKs.
Signed-off-by: Van Jacobson <vanj@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-19 21:39:14 -06:00
|
|
|
struct rate_sample *rate;
|
2014-02-26 15:02:48 -07:00
|
|
|
int flag;
|
2017-10-05 23:21:27 -06:00
|
|
|
unsigned int mss_now;
|
2008-12-05 23:42:22 -07:00
|
|
|
};
|
|
|
|
|
2007-10-11 18:34:25 -06:00
|
|
|
/* Check if skb is fully within the SACK block. In presence of GSO skbs,
|
|
|
|
* the incoming SACK may not exactly match but we can find smaller MSS
|
|
|
|
* aligned portion of it that matches. Therefore we might need to fragment
|
|
|
|
* which may fail and creates some hassle (caller must handle error case
|
|
|
|
* returns).
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
*
|
|
|
|
* FIXME: this could be merged to shift decision code
|
2007-10-11 18:34:25 -06:00
|
|
|
*/
|
2007-10-26 04:57:36 -06:00
|
|
|
static int tcp_match_skb_to_sack(struct sock *sk, struct sk_buff *skb,
|
2012-05-16 17:15:34 -06:00
|
|
|
u32 start_seq, u32 end_seq)
|
2007-10-11 18:34:25 -06:00
|
|
|
{
|
2012-05-16 17:15:34 -06:00
|
|
|
int err;
|
|
|
|
bool in_sack;
|
2007-10-11 18:34:25 -06:00
|
|
|
unsigned int pkt_len;
|
2008-11-24 22:13:50 -07:00
|
|
|
unsigned int mss;
|
2007-10-11 18:34:25 -06:00
|
|
|
|
|
|
|
in_sack = !after(start_seq, TCP_SKB_CB(skb)->seq) &&
|
|
|
|
!before(end_seq, TCP_SKB_CB(skb)->end_seq);
|
|
|
|
|
|
|
|
if (tcp_skb_pcount(skb) > 1 && !in_sack &&
|
|
|
|
after(TCP_SKB_CB(skb)->end_seq, start_seq)) {
|
2008-11-24 22:13:50 -07:00
|
|
|
mss = tcp_skb_mss(skb);
|
2007-10-11 18:34:25 -06:00
|
|
|
in_sack = !after(start_seq, TCP_SKB_CB(skb)->seq);
|
|
|
|
|
2008-11-24 22:13:50 -07:00
|
|
|
if (!in_sack) {
|
2007-10-11 18:34:25 -06:00
|
|
|
pkt_len = start_seq - TCP_SKB_CB(skb)->seq;
|
2008-11-24 22:13:50 -07:00
|
|
|
if (pkt_len < mss)
|
|
|
|
pkt_len = mss;
|
|
|
|
} else {
|
2007-10-11 18:34:25 -06:00
|
|
|
pkt_len = end_seq - TCP_SKB_CB(skb)->seq;
|
2008-11-24 22:13:50 -07:00
|
|
|
if (pkt_len < mss)
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Round if necessary so that SACKs cover only full MSSes
|
|
|
|
* and/or the remaining small portion (if present)
|
|
|
|
*/
|
|
|
|
if (pkt_len > mss) {
|
|
|
|
unsigned int new_len = (pkt_len / mss) * mss;
|
2017-05-10 18:01:27 -06:00
|
|
|
if (!in_sack && new_len < pkt_len)
|
2008-11-24 22:13:50 -07:00
|
|
|
new_len += mss;
|
|
|
|
pkt_len = new_len;
|
|
|
|
}
|
2017-05-10 18:01:27 -06:00
|
|
|
|
|
|
|
if (pkt_len >= skb->len && !in_sack)
|
|
|
|
return 0;
|
|
|
|
|
2017-10-05 23:21:27 -06:00
|
|
|
err = tcp_fragment(sk, TCP_FRAG_IN_RTX_QUEUE, skb,
|
|
|
|
pkt_len, mss, GFP_ATOMIC);
|
2007-10-11 18:34:25 -06:00
|
|
|
if (err < 0)
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
return in_sack;
|
|
|
|
}
|
|
|
|
|
2012-02-12 11:37:09 -07:00
|
|
|
/* Mark the given newly-SACKed range as such, adjusting counters and hints. */
|
|
|
|
static u8 tcp_sacktag_one(struct sock *sk,
|
|
|
|
struct tcp_sacktag_state *state, u8 sacked,
|
|
|
|
u32 start_seq, u32 end_seq,
|
2014-02-26 15:02:48 -07:00
|
|
|
int dup_sack, int pcount,
|
2017-05-16 15:00:14 -06:00
|
|
|
u64 xmit_time)
|
2007-11-15 20:44:56 -07:00
|
|
|
{
|
2007-12-01 15:48:06 -07:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2007-11-15 20:44:56 -07:00
|
|
|
|
|
|
|
/* Account D-SACK for retransmitted packet. */
|
|
|
|
if (dup_sack && (sacked & TCPCB_RETRANS)) {
|
2014-07-02 13:07:16 -06:00
|
|
|
if (tp->undo_marker && tp->undo_retrans > 0 &&
|
2012-02-12 11:37:09 -07:00
|
|
|
after(end_seq, tp->undo_marker))
|
2007-11-15 20:44:56 -07:00
|
|
|
tp->undo_retrans--;
|
2017-11-08 14:01:27 -07:00
|
|
|
if ((sacked & TCPCB_SACKED_ACKED) &&
|
|
|
|
before(start_seq, state->reord))
|
|
|
|
state->reord = start_seq;
|
2007-11-15 20:44:56 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Nothing to do; acked frame is about to be dropped (was ACKed). */
|
2012-02-12 11:37:09 -07:00
|
|
|
if (!after(end_seq, tp->snd_una))
|
2008-12-05 23:42:22 -07:00
|
|
|
return sacked;
|
2007-11-15 20:44:56 -07:00
|
|
|
|
|
|
|
if (!(sacked & TCPCB_SACKED_ACKED)) {
|
2017-04-25 11:15:38 -06:00
|
|
|
tcp_rack_advance(tp, sacked, end_seq, xmit_time);
|
2015-10-16 22:57:46 -06:00
|
|
|
|
2007-11-15 20:44:56 -07:00
|
|
|
if (sacked & TCPCB_SACKED_RETRANS) {
|
|
|
|
/* If the segment is not tagged as lost,
|
|
|
|
* we do not clear RETRANS, believing
|
|
|
|
* that retransmission is still in flight.
|
|
|
|
*/
|
|
|
|
if (sacked & TCPCB_LOST) {
|
2008-12-05 23:42:22 -07:00
|
|
|
sacked &= ~(TCPCB_LOST|TCPCB_SACKED_RETRANS);
|
2008-11-24 22:14:43 -07:00
|
|
|
tp->lost_out -= pcount;
|
|
|
|
tp->retrans_out -= pcount;
|
2007-11-15 20:44:56 -07:00
|
|
|
}
|
|
|
|
} else {
|
|
|
|
if (!(sacked & TCPCB_RETRANS)) {
|
|
|
|
/* New sack for not retransmitted frame,
|
|
|
|
* which was in hole. It is reordering.
|
|
|
|
*/
|
2012-02-12 11:37:09 -07:00
|
|
|
if (before(start_seq,
|
2017-11-08 14:01:27 -07:00
|
|
|
tcp_highest_sack_seq(tp)) &&
|
|
|
|
before(start_seq, state->reord))
|
|
|
|
state->reord = start_seq;
|
|
|
|
|
2013-03-20 07:33:00 -06:00
|
|
|
if (!after(end_seq, tp->high_seq))
|
|
|
|
state->flag |= FLAG_ORIG_SACK_ACKED;
|
2017-05-16 15:00:14 -06:00
|
|
|
if (state->first_sackt == 0)
|
|
|
|
state->first_sackt = xmit_time;
|
|
|
|
state->last_sackt = xmit_time;
|
2007-11-15 20:44:56 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
if (sacked & TCPCB_LOST) {
|
2008-12-05 23:42:22 -07:00
|
|
|
sacked &= ~TCPCB_LOST;
|
2008-11-24 22:14:43 -07:00
|
|
|
tp->lost_out -= pcount;
|
2007-11-15 20:44:56 -07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-12-05 23:42:22 -07:00
|
|
|
sacked |= TCPCB_SACKED_ACKED;
|
|
|
|
state->flag |= FLAG_DATA_SACKED;
|
2008-11-24 22:14:43 -07:00
|
|
|
tp->sacked_out += pcount;
|
2016-02-02 11:33:06 -07:00
|
|
|
tp->delivered += pcount; /* Out-of-order packets delivered */
|
2007-11-15 20:44:56 -07:00
|
|
|
|
|
|
|
/* Lost marker hint past SACKed? Tweak RFC3517 cnt */
|
2017-11-08 14:01:26 -07:00
|
|
|
if (tp->lost_skb_hint &&
|
2012-02-12 11:37:09 -07:00
|
|
|
before(start_seq, TCP_SKB_CB(tp->lost_skb_hint)->seq))
|
2008-11-24 22:14:43 -07:00
|
|
|
tp->lost_cnt_hint += pcount;
|
2007-11-15 20:44:56 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
/* D-SACK. We can detect redundant retransmission in S|R and plain R
|
|
|
|
* frames and clear it. undo_retrans is decreased above, L|R frames
|
|
|
|
* are accounted above as well.
|
|
|
|
*/
|
2008-12-05 23:42:22 -07:00
|
|
|
if (dup_sack && (sacked & TCPCB_SACKED_RETRANS)) {
|
|
|
|
sacked &= ~TCPCB_SACKED_RETRANS;
|
2008-11-24 22:14:43 -07:00
|
|
|
tp->retrans_out -= pcount;
|
2007-11-15 20:44:56 -07:00
|
|
|
}
|
|
|
|
|
2008-12-05 23:42:22 -07:00
|
|
|
return sacked;
|
2007-11-15 20:44:56 -07:00
|
|
|
}
|
|
|
|
|
2012-02-12 11:37:10 -07:00
|
|
|
/* Shift newly-SACKed bytes from this skb to the immediately previous
|
|
|
|
* already-SACKed sk_buff. Mark the newly-SACKed bytes as such.
|
|
|
|
*/
|
2017-10-05 23:21:26 -06:00
|
|
|
static bool tcp_shifted_skb(struct sock *sk, struct sk_buff *prev,
|
|
|
|
struct sk_buff *skb,
|
2012-05-16 17:15:34 -06:00
|
|
|
struct tcp_sacktag_state *state,
|
|
|
|
unsigned int pcount, int shifted, int mss,
|
|
|
|
bool dup_sack)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2012-02-12 11:37:10 -07:00
|
|
|
u32 start_seq = TCP_SKB_CB(skb)->seq; /* start of newly-SACKed */
|
|
|
|
u32 end_seq = start_seq + shifted; /* end of newly-SACKed */
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
|
|
|
|
BUG_ON(!pcount);
|
|
|
|
|
2012-02-26 03:06:19 -07:00
|
|
|
/* Adjust counters and hints for the newly sacked sequence
|
|
|
|
* range but discard the return value since prev is already
|
|
|
|
* marked. We must tag the range first because the seq
|
|
|
|
* advancement below implicitly advances
|
|
|
|
* tcp_highest_sack_seq() when skb is highest_sack.
|
|
|
|
*/
|
|
|
|
tcp_sacktag_one(sk, state, TCP_SKB_CB(skb)->sacked,
|
2013-07-22 17:20:47 -06:00
|
|
|
start_seq, end_seq, dup_sack, pcount,
|
2018-09-21 09:51:47 -06:00
|
|
|
tcp_skb_timestamp_us(skb));
|
tcp: track data delivery rate for a TCP connection
This patch generates data delivery rate (throughput) samples on a
per-ACK basis. These rate samples can be used by congestion control
modules, and specifically will be used by TCP BBR in later patches in
this series.
Key state:
tp->delivered: Tracks the total number of data packets (original or not)
delivered so far. This is an already-existing field.
tp->delivered_mstamp: the last time tp->delivered was updated.
Algorithm:
A rate sample is calculated as (d1 - d0)/(t1 - t0) on a per-ACK basis:
d1: the current tp->delivered after processing the ACK
t1: the current time after processing the ACK
d0: the prior tp->delivered when the acked skb was transmitted
t0: the prior tp->delivered_mstamp when the acked skb was transmitted
When an skb is transmitted, we snapshot d0 and t0 in its control
block in tcp_rate_skb_sent().
When an ACK arrives, it may SACK and ACK some skbs. For each SACKed
or ACKed skb, tcp_rate_skb_delivered() updates the rate_sample struct
to reflect the latest (d0, t0).
Finally, tcp_rate_gen() generates a rate sample by storing
(d1 - d0) in rs->delivered and (t1 - t0) in rs->interval_us.
One caveat: if an skb was sent with no packets in flight, then
tp->delivered_mstamp may be either invalid (if the connection is
starting) or outdated (if the connection was idle). In that case,
we'll re-stamp tp->delivered_mstamp.
At first glance it seems t0 should always be the time when an skb was
transmitted, but actually this could over-estimate the rate due to
phase mismatch between transmit and ACK events. To track the delivery
rate, we ensure that if packets are in flight then t0 and and t1 are
times at which packets were marked delivered.
If the initial and final RTTs are different then one may be corrupted
by some sort of noise. The noise we see most often is sending gaps
caused by delayed, compressed, or stretched acks. This either affects
both RTTs equally or artificially reduces the final RTT. We approach
this by recording the info we need to compute the initial RTT
(duration of the "send phase" of the window) when we recorded the
associated inflight. Then, for a filter to avoid bandwidth
overestimates, we generalize the per-sample bandwidth computation
from:
bw = delivered / ack_phase_rtt
to the following:
bw = delivered / max(send_phase_rtt, ack_phase_rtt)
In large-scale experiments, this filtering approach incorporating
send_phase_rtt is effective at avoiding bandwidth overestimates due to
ACK compression or stretched ACKs.
Signed-off-by: Van Jacobson <vanj@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-19 21:39:14 -06:00
|
|
|
tcp_rate_skb_delivered(sk, skb, state->rate);
|
2012-02-26 03:06:19 -07:00
|
|
|
|
|
|
|
if (skb == tp->lost_skb_hint)
|
2012-02-13 13:22:08 -07:00
|
|
|
tp->lost_cnt_hint += pcount;
|
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
TCP_SKB_CB(prev)->end_seq += shifted;
|
|
|
|
TCP_SKB_CB(skb)->seq += shifted;
|
|
|
|
|
2014-09-24 05:11:22 -06:00
|
|
|
tcp_skb_pcount_add(prev, pcount);
|
|
|
|
BUG_ON(tcp_skb_pcount(skb) < pcount);
|
|
|
|
tcp_skb_pcount_add(skb, -pcount);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
|
|
|
|
/* When we're adding to gso_segs == 1, gso_size will be zero,
|
|
|
|
* in theory this shouldn't be necessary but as long as DSACK
|
|
|
|
* code can come after this skb later on it's better to keep
|
|
|
|
* setting gso_size to something.
|
|
|
|
*/
|
2015-06-11 10:15:18 -06:00
|
|
|
if (!TCP_SKB_CB(prev)->tcp_gso_size)
|
|
|
|
TCP_SKB_CB(prev)->tcp_gso_size = mss;
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
|
|
|
|
/* CHECKME: To clear or not to clear? Mimics normal skb currently */
|
2015-06-11 10:15:16 -06:00
|
|
|
if (tcp_skb_pcount(skb) <= 1)
|
2015-06-11 10:15:18 -06:00
|
|
|
TCP_SKB_CB(skb)->tcp_gso_size = 0;
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
|
|
|
|
/* Difference in this won't matter, both ACKed by the same cumul. ACK */
|
|
|
|
TCP_SKB_CB(prev)->sacked |= (TCP_SKB_CB(skb)->sacked & TCPCB_EVER_RETRANS);
|
|
|
|
|
|
|
|
if (skb->len > 0) {
|
|
|
|
BUG_ON(!tcp_skb_pcount(skb));
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_SACKSHIFTED);
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Whole SKB was eaten :-) */
|
|
|
|
|
2008-11-24 22:26:56 -07:00
|
|
|
if (skb == tp->retransmit_skb_hint)
|
|
|
|
tp->retransmit_skb_hint = prev;
|
|
|
|
if (skb == tp->lost_skb_hint) {
|
|
|
|
tp->lost_skb_hint = prev;
|
|
|
|
tp->lost_cnt_hint -= tcp_skb_pcount(prev);
|
|
|
|
}
|
|
|
|
|
tcp: do not forget FIN in tcp_shifted_skb()
Yuchung found following problem :
There are bugs in the SACK processing code, merging part in
tcp_shift_skb_data(), that incorrectly resets or ignores the sacked
skbs FIN flag. When a receiver first SACK the FIN sequence, and later
throw away ofo queue (e.g., sack-reneging), the sender will stop
retransmitting the FIN flag, and hangs forever.
Following packetdrill test can be used to reproduce the bug.
$ cat sack-merge-bug.pkt
`sysctl -q net.ipv4.tcp_fack=0`
// Establish a connection and send 10 MSS.
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+.000 bind(3, ..., ...) = 0
+.000 listen(3, 1) = 0
+.050 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+.000 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>
+.001 < . 1:1(0) ack 1 win 1024
+.000 accept(3, ..., ...) = 4
+.100 write(4, ..., 12000) = 12000
+.000 shutdown(4, SHUT_WR) = 0
+.000 > . 1:10001(10000) ack 1
+.050 < . 1:1(0) ack 2001 win 257
+.000 > FP. 10001:12001(2000) ack 1
+.050 < . 1:1(0) ack 2001 win 257 <sack 10001:11001,nop,nop>
+.050 < . 1:1(0) ack 2001 win 257 <sack 10001:12002,nop,nop>
// SACK reneg
+.050 < . 1:1(0) ack 12001 win 257
+0 %{ print "unacked: ",tcpi_unacked }%
+5 %{ print "" }%
First, a typo inverted left/right of one OR operation, then
code forgot to advance end_seq if the merged skb carried FIN.
Bug was added in 2.6.29 by commit 832d11c5cd076ab
("tcp: Try to restore large SKBs while SACK processing")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-04 11:31:41 -06:00
|
|
|
TCP_SKB_CB(prev)->tcp_flags |= TCP_SKB_CB(skb)->tcp_flags;
|
tcp: Handle eor bit when coalescing skb
This patch:
1. Prevent next_skb from coalescing to the prev_skb if
TCP_SKB_CB(prev_skb)->eor is set
2. Update the TCP_SKB_CB(prev_skb)->eor if coalescing is
allowed
Packetdrill script for testing:
~~~~~~
+0 `sysctl -q -w net.ipv4.tcp_min_tso_segs=10`
+0 `sysctl -q -w net.ipv4.tcp_no_metrics_save=1`
+0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
0.100 < S 0:0(0) win 32792 <mss 1460,sackOK,nop,nop,nop,wscale 7>
0.100 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 7>
0.200 < . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
+0 setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4) = 0
0.200 sendto(4, ..., 730, MSG_EOR, ..., ...) = 730
0.200 sendto(4, ..., 730, MSG_EOR, ..., ...) = 730
0.200 write(4, ..., 11680) = 11680
0.200 > P. 1:731(730) ack 1
0.200 > P. 731:1461(730) ack 1
0.200 > . 1461:8761(7300) ack 1
0.200 > P. 8761:13141(4380) ack 1
0.300 < . 1:1(0) ack 1 win 257 <sack 1461:13141,nop,nop>
0.300 > P. 1:731(730) ack 1
0.300 > P. 731:1461(730) ack 1
0.400 < . 1:1(0) ack 13141 win 257
0.400 close(4) = 0
0.400 > F. 13141:13141(0) ack 1
0.500 < F. 1:1(0) ack 13142 win 257
0.500 > . 13142:13142(0) ack 2
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Cc: Willem de Bruijn <willemb@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-04-25 15:44:49 -06:00
|
|
|
TCP_SKB_CB(prev)->eor = TCP_SKB_CB(skb)->eor;
|
tcp: do not forget FIN in tcp_shifted_skb()
Yuchung found following problem :
There are bugs in the SACK processing code, merging part in
tcp_shift_skb_data(), that incorrectly resets or ignores the sacked
skbs FIN flag. When a receiver first SACK the FIN sequence, and later
throw away ofo queue (e.g., sack-reneging), the sender will stop
retransmitting the FIN flag, and hangs forever.
Following packetdrill test can be used to reproduce the bug.
$ cat sack-merge-bug.pkt
`sysctl -q net.ipv4.tcp_fack=0`
// Establish a connection and send 10 MSS.
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+.000 bind(3, ..., ...) = 0
+.000 listen(3, 1) = 0
+.050 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+.000 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>
+.001 < . 1:1(0) ack 1 win 1024
+.000 accept(3, ..., ...) = 4
+.100 write(4, ..., 12000) = 12000
+.000 shutdown(4, SHUT_WR) = 0
+.000 > . 1:10001(10000) ack 1
+.050 < . 1:1(0) ack 2001 win 257
+.000 > FP. 10001:12001(2000) ack 1
+.050 < . 1:1(0) ack 2001 win 257 <sack 10001:11001,nop,nop>
+.050 < . 1:1(0) ack 2001 win 257 <sack 10001:12002,nop,nop>
// SACK reneg
+.050 < . 1:1(0) ack 12001 win 257
+0 %{ print "unacked: ",tcpi_unacked }%
+5 %{ print "" }%
First, a typo inverted left/right of one OR operation, then
code forgot to advance end_seq if the merged skb carried FIN.
Bug was added in 2.6.29 by commit 832d11c5cd076ab
("tcp: Try to restore large SKBs while SACK processing")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-04 11:31:41 -06:00
|
|
|
if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
|
|
|
|
TCP_SKB_CB(prev)->end_seq++;
|
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
if (skb == tcp_highest_sack(sk))
|
|
|
|
tcp_advance_highest_sack(sk, skb);
|
|
|
|
|
tcp: Merge tx_flags and tskey in tcp_shifted_skb
After receiving sacks, tcp_shifted_skb() will collapse
skbs if possible. tx_flags and tskey also have to be
merged.
This patch reuses the tcp_skb_collapse_tstamp() to handle
them.
BPF Output Before:
~~~~~
<no-output-due-to-missing-tstamp-event>
BPF Output After:
~~~~~
<...>-2024 [007] d.s. 88.644374: : ee_data:14599
Packetdrill Script:
~~~~~
+0 `sysctl -q -w net.ipv4.tcp_min_tso_segs=10`
+0 `sysctl -q -w net.ipv4.tcp_no_metrics_save=1`
+0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
0.100 < S 0:0(0) win 32792 <mss 1460,sackOK,nop,nop,nop,wscale 7>
0.100 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 7>
0.200 < . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
+0 setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4) = 0
0.200 write(4, ..., 1460) = 1460
+0 setsockopt(4, SOL_SOCKET, 37, [2688], 4) = 0
0.200 write(4, ..., 13140) = 13140
0.200 > P. 1:1461(1460) ack 1
0.200 > . 1461:8761(7300) ack 1
0.200 > P. 8761:14601(5840) ack 1
0.300 < . 1:1(0) ack 1 win 257 <sack 1461:14601,nop,nop>
0.300 > P. 1:1461(1460) ack 1
0.400 < . 1:1(0) ack 14601 win 257
0.400 close(4) = 0
0.400 > F. 14601:14601(0) ack 1
0.500 < F. 1:1(0) ack 14602 win 257
0.500 > . 14602:14602(0) ack 2
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Cc: Willem de Bruijn <willemb@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
Tested-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-04-19 23:39:29 -06:00
|
|
|
tcp_skb_collapse_tstamp(prev, skb);
|
2017-05-16 15:00:14 -06:00
|
|
|
if (unlikely(TCP_SKB_CB(prev)->tx.delivered_mstamp))
|
|
|
|
TCP_SKB_CB(prev)->tx.delivered_mstamp = 0;
|
tcp: track data delivery rate for a TCP connection
This patch generates data delivery rate (throughput) samples on a
per-ACK basis. These rate samples can be used by congestion control
modules, and specifically will be used by TCP BBR in later patches in
this series.
Key state:
tp->delivered: Tracks the total number of data packets (original or not)
delivered so far. This is an already-existing field.
tp->delivered_mstamp: the last time tp->delivered was updated.
Algorithm:
A rate sample is calculated as (d1 - d0)/(t1 - t0) on a per-ACK basis:
d1: the current tp->delivered after processing the ACK
t1: the current time after processing the ACK
d0: the prior tp->delivered when the acked skb was transmitted
t0: the prior tp->delivered_mstamp when the acked skb was transmitted
When an skb is transmitted, we snapshot d0 and t0 in its control
block in tcp_rate_skb_sent().
When an ACK arrives, it may SACK and ACK some skbs. For each SACKed
or ACKed skb, tcp_rate_skb_delivered() updates the rate_sample struct
to reflect the latest (d0, t0).
Finally, tcp_rate_gen() generates a rate sample by storing
(d1 - d0) in rs->delivered and (t1 - t0) in rs->interval_us.
One caveat: if an skb was sent with no packets in flight, then
tp->delivered_mstamp may be either invalid (if the connection is
starting) or outdated (if the connection was idle). In that case,
we'll re-stamp tp->delivered_mstamp.
At first glance it seems t0 should always be the time when an skb was
transmitted, but actually this could over-estimate the rate due to
phase mismatch between transmit and ACK events. To track the delivery
rate, we ensure that if packets are in flight then t0 and and t1 are
times at which packets were marked delivered.
If the initial and final RTTs are different then one may be corrupted
by some sort of noise. The noise we see most often is sending gaps
caused by delayed, compressed, or stretched acks. This either affects
both RTTs equally or artificially reduces the final RTT. We approach
this by recording the info we need to compute the initial RTT
(duration of the "send phase" of the window) when we recorded the
associated inflight. Then, for a filter to avoid bandwidth
overestimates, we generalize the per-sample bandwidth computation
from:
bw = delivered / ack_phase_rtt
to the following:
bw = delivered / max(send_phase_rtt, ack_phase_rtt)
In large-scale experiments, this filtering approach incorporating
send_phase_rtt is effective at avoiding bandwidth overestimates due to
ACK compression or stretched ACKs.
Signed-off-by: Van Jacobson <vanj@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-19 21:39:14 -06:00
|
|
|
|
2017-10-05 23:21:27 -06:00
|
|
|
tcp_rtx_queue_unlink_and_free(skb, sk);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_SACKMERGED);
|
2008-11-24 22:27:22 -07:00
|
|
|
|
2012-05-16 17:15:34 -06:00
|
|
|
return true;
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
/* I wish gso_size would have a bit more sane initialization than
|
|
|
|
* something-or-zero which complicates things
|
|
|
|
*/
|
2011-10-21 03:22:42 -06:00
|
|
|
static int tcp_skb_seglen(const struct sk_buff *skb)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
{
|
2008-12-05 23:41:26 -07:00
|
|
|
return tcp_skb_pcount(skb) == 1 ? skb->len : tcp_skb_mss(skb);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Shifting pages past head area doesn't work */
|
2011-10-21 03:22:42 -06:00
|
|
|
static int skb_can_shift(const struct sk_buff *skb)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
{
|
|
|
|
return !skb_headlen(skb) && skb_is_nonlinear(skb);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Try collapsing SACK blocks spanning across multiple skbs to a single
|
|
|
|
* skb.
|
|
|
|
*/
|
|
|
|
static struct sk_buff *tcp_shift_skb_data(struct sock *sk, struct sk_buff *skb,
|
2008-12-05 23:42:22 -07:00
|
|
|
struct tcp_sacktag_state *state,
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
u32 start_seq, u32 end_seq,
|
2012-05-16 17:15:34 -06:00
|
|
|
bool dup_sack)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *prev;
|
|
|
|
int mss;
|
|
|
|
int pcount = 0;
|
|
|
|
int len;
|
|
|
|
int in_sack;
|
|
|
|
|
|
|
|
/* Normally R but no L won't result in plain S */
|
|
|
|
if (!dup_sack &&
|
2008-12-05 23:41:06 -07:00
|
|
|
(TCP_SKB_CB(skb)->sacked & (TCPCB_LOST|TCPCB_SACKED_RETRANS)) == TCPCB_SACKED_RETRANS)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
goto fallback;
|
|
|
|
if (!skb_can_shift(skb))
|
|
|
|
goto fallback;
|
|
|
|
/* This frame is about to be dropped (was ACKed). */
|
|
|
|
if (!after(TCP_SKB_CB(skb)->end_seq, tp->snd_una))
|
|
|
|
goto fallback;
|
|
|
|
|
|
|
|
/* Can only happen with delayed DSACK + discard craziness */
|
2017-10-05 23:21:27 -06:00
|
|
|
prev = skb_rb_prev(skb);
|
|
|
|
if (!prev)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
goto fallback;
|
|
|
|
|
|
|
|
if ((TCP_SKB_CB(prev)->sacked & TCPCB_TAGBITS) != TCPCB_SACKED_ACKED)
|
|
|
|
goto fallback;
|
|
|
|
|
tcp: Handle eor bit when coalescing skb
This patch:
1. Prevent next_skb from coalescing to the prev_skb if
TCP_SKB_CB(prev_skb)->eor is set
2. Update the TCP_SKB_CB(prev_skb)->eor if coalescing is
allowed
Packetdrill script for testing:
~~~~~~
+0 `sysctl -q -w net.ipv4.tcp_min_tso_segs=10`
+0 `sysctl -q -w net.ipv4.tcp_no_metrics_save=1`
+0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
0.100 < S 0:0(0) win 32792 <mss 1460,sackOK,nop,nop,nop,wscale 7>
0.100 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 7>
0.200 < . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
+0 setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4) = 0
0.200 sendto(4, ..., 730, MSG_EOR, ..., ...) = 730
0.200 sendto(4, ..., 730, MSG_EOR, ..., ...) = 730
0.200 write(4, ..., 11680) = 11680
0.200 > P. 1:731(730) ack 1
0.200 > P. 731:1461(730) ack 1
0.200 > . 1461:8761(7300) ack 1
0.200 > P. 8761:13141(4380) ack 1
0.300 < . 1:1(0) ack 1 win 257 <sack 1461:13141,nop,nop>
0.300 > P. 1:731(730) ack 1
0.300 > P. 731:1461(730) ack 1
0.400 < . 1:1(0) ack 13141 win 257
0.400 close(4) = 0
0.400 > F. 13141:13141(0) ack 1
0.500 < F. 1:1(0) ack 13142 win 257
0.500 > . 13142:13142(0) ack 2
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Cc: Willem de Bruijn <willemb@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-04-25 15:44:49 -06:00
|
|
|
if (!tcp_skb_can_collapse_to(prev))
|
|
|
|
goto fallback;
|
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
in_sack = !after(start_seq, TCP_SKB_CB(skb)->seq) &&
|
|
|
|
!before(end_seq, TCP_SKB_CB(skb)->end_seq);
|
|
|
|
|
|
|
|
if (in_sack) {
|
|
|
|
len = skb->len;
|
|
|
|
pcount = tcp_skb_pcount(skb);
|
2008-12-05 23:41:26 -07:00
|
|
|
mss = tcp_skb_seglen(skb);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
|
|
|
|
/* TODO: Fix DSACKs to not fragment already SACKed and we can
|
|
|
|
* drop this restriction as unnecessary
|
|
|
|
*/
|
2008-12-05 23:41:26 -07:00
|
|
|
if (mss != tcp_skb_seglen(prev))
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
goto fallback;
|
|
|
|
} else {
|
|
|
|
if (!after(TCP_SKB_CB(skb)->end_seq, start_seq))
|
|
|
|
goto noop;
|
|
|
|
/* CHECKME: This is non-MSS split case only?, this will
|
|
|
|
* cause skipped skbs due to advancing loop btw, original
|
|
|
|
* has that feature too
|
|
|
|
*/
|
|
|
|
if (tcp_skb_pcount(skb) <= 1)
|
|
|
|
goto noop;
|
|
|
|
|
|
|
|
in_sack = !after(start_seq, TCP_SKB_CB(skb)->seq);
|
|
|
|
if (!in_sack) {
|
|
|
|
/* TODO: head merge to next could be attempted here
|
|
|
|
* if (!after(TCP_SKB_CB(skb)->end_seq, end_seq)),
|
|
|
|
* though it might not be worth of the additional hassle
|
|
|
|
*
|
|
|
|
* ...we can probably just fallback to what was done
|
|
|
|
* previously. We could try merging non-SACKed ones
|
|
|
|
* as well but it probably isn't going to buy off
|
|
|
|
* because later SACKs might again split them, and
|
|
|
|
* it would make skb timestamp tracking considerably
|
|
|
|
* harder problem.
|
|
|
|
*/
|
|
|
|
goto fallback;
|
|
|
|
}
|
|
|
|
|
|
|
|
len = end_seq - TCP_SKB_CB(skb)->seq;
|
|
|
|
BUG_ON(len < 0);
|
|
|
|
BUG_ON(len > skb->len);
|
|
|
|
|
|
|
|
/* MSS boundaries should be honoured or else pcount will
|
|
|
|
* severely break even though it makes things bit trickier.
|
|
|
|
* Optimize common case to avoid most of the divides
|
|
|
|
*/
|
|
|
|
mss = tcp_skb_mss(skb);
|
|
|
|
|
|
|
|
/* TODO: Fix DSACKs to not fragment already SACKed and we can
|
|
|
|
* drop this restriction as unnecessary
|
|
|
|
*/
|
2008-12-05 23:41:26 -07:00
|
|
|
if (mss != tcp_skb_seglen(prev))
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
goto fallback;
|
|
|
|
|
|
|
|
if (len == mss) {
|
|
|
|
pcount = 1;
|
|
|
|
} else if (len < mss) {
|
|
|
|
goto noop;
|
|
|
|
} else {
|
|
|
|
pcount = len / mss;
|
|
|
|
len = pcount * mss;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-03-05 12:35:04 -07:00
|
|
|
/* tcp_sacktag_one() won't SACK-tag ranges below snd_una */
|
|
|
|
if (!after(TCP_SKB_CB(skb)->seq + len, tp->snd_una))
|
|
|
|
goto fallback;
|
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
if (!skb_shift(prev, skb, len))
|
|
|
|
goto fallback;
|
2017-10-05 23:21:26 -06:00
|
|
|
if (!tcp_shifted_skb(sk, prev, skb, state, pcount, len, mss, dup_sack))
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
goto out;
|
|
|
|
|
|
|
|
/* Hole filled allows collapsing with the next as well, this is very
|
|
|
|
* useful when hole on every nth skb pattern happens
|
|
|
|
*/
|
2017-10-05 23:21:27 -06:00
|
|
|
skb = skb_rb_next(prev);
|
|
|
|
if (!skb)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
goto out;
|
|
|
|
|
2008-12-05 23:40:47 -07:00
|
|
|
if (!skb_can_shift(skb) ||
|
|
|
|
((TCP_SKB_CB(skb)->sacked & TCPCB_TAGBITS) != TCPCB_SACKED_ACKED) ||
|
2008-12-05 23:41:26 -07:00
|
|
|
(mss != tcp_skb_seglen(skb)))
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
goto out;
|
|
|
|
|
|
|
|
len = skb->len;
|
|
|
|
if (skb_shift(prev, skb, len)) {
|
|
|
|
pcount += tcp_skb_pcount(skb);
|
2017-10-05 23:21:26 -06:00
|
|
|
tcp_shifted_skb(sk, prev, skb, state, tcp_skb_pcount(skb),
|
|
|
|
len, mss, 0);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
out:
|
|
|
|
return prev;
|
|
|
|
|
|
|
|
noop:
|
|
|
|
return skb;
|
|
|
|
|
|
|
|
fallback:
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_SACKSHIFTFALLBACK);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
static struct sk_buff *tcp_sacktag_walk(struct sk_buff *skb, struct sock *sk,
|
|
|
|
struct tcp_sack_block *next_dup,
|
2008-12-05 23:42:22 -07:00
|
|
|
struct tcp_sacktag_state *state,
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
u32 start_seq, u32 end_seq,
|
2012-05-16 17:15:34 -06:00
|
|
|
bool dup_sack_in)
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
{
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *tmp;
|
|
|
|
|
2017-10-05 23:21:27 -06:00
|
|
|
skb_rbtree_walk_from(skb) {
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
int in_sack = 0;
|
2012-05-16 17:15:34 -06:00
|
|
|
bool dup_sack = dup_sack_in;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
|
|
|
|
/* queue is in-order => we can short-circuit the walk early */
|
|
|
|
if (!before(TCP_SKB_CB(skb)->seq, end_seq))
|
|
|
|
break;
|
|
|
|
|
2015-04-03 02:17:27 -06:00
|
|
|
if (next_dup &&
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
before(TCP_SKB_CB(skb)->seq, next_dup->end_seq)) {
|
|
|
|
in_sack = tcp_match_skb_to_sack(sk, skb,
|
|
|
|
next_dup->start_seq,
|
|
|
|
next_dup->end_seq);
|
|
|
|
if (in_sack > 0)
|
2012-05-16 17:15:34 -06:00
|
|
|
dup_sack = true;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
}
|
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
/* skb reference here is a bit tricky to get right, since
|
|
|
|
* shifting can eat and free both this skb and the next,
|
|
|
|
* so not even _safe variant of the loop is enough.
|
|
|
|
*/
|
|
|
|
if (in_sack <= 0) {
|
2008-12-05 23:42:22 -07:00
|
|
|
tmp = tcp_shift_skb_data(sk, skb, state,
|
|
|
|
start_seq, end_seq, dup_sack);
|
2015-04-03 02:17:27 -06:00
|
|
|
if (tmp) {
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
if (tmp != skb) {
|
|
|
|
skb = tmp;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
in_sack = 0;
|
|
|
|
} else {
|
|
|
|
in_sack = tcp_match_skb_to_sack(sk, skb,
|
|
|
|
start_seq,
|
|
|
|
end_seq);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
if (unlikely(in_sack < 0))
|
|
|
|
break;
|
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
if (in_sack) {
|
2012-02-12 11:37:09 -07:00
|
|
|
TCP_SKB_CB(skb)->sacked =
|
|
|
|
tcp_sacktag_one(sk,
|
|
|
|
state,
|
|
|
|
TCP_SKB_CB(skb)->sacked,
|
|
|
|
TCP_SKB_CB(skb)->seq,
|
|
|
|
TCP_SKB_CB(skb)->end_seq,
|
|
|
|
dup_sack,
|
2013-07-22 17:20:47 -06:00
|
|
|
tcp_skb_pcount(skb),
|
2018-09-21 09:51:47 -06:00
|
|
|
tcp_skb_timestamp_us(skb));
|
tcp: track data delivery rate for a TCP connection
This patch generates data delivery rate (throughput) samples on a
per-ACK basis. These rate samples can be used by congestion control
modules, and specifically will be used by TCP BBR in later patches in
this series.
Key state:
tp->delivered: Tracks the total number of data packets (original or not)
delivered so far. This is an already-existing field.
tp->delivered_mstamp: the last time tp->delivered was updated.
Algorithm:
A rate sample is calculated as (d1 - d0)/(t1 - t0) on a per-ACK basis:
d1: the current tp->delivered after processing the ACK
t1: the current time after processing the ACK
d0: the prior tp->delivered when the acked skb was transmitted
t0: the prior tp->delivered_mstamp when the acked skb was transmitted
When an skb is transmitted, we snapshot d0 and t0 in its control
block in tcp_rate_skb_sent().
When an ACK arrives, it may SACK and ACK some skbs. For each SACKed
or ACKed skb, tcp_rate_skb_delivered() updates the rate_sample struct
to reflect the latest (d0, t0).
Finally, tcp_rate_gen() generates a rate sample by storing
(d1 - d0) in rs->delivered and (t1 - t0) in rs->interval_us.
One caveat: if an skb was sent with no packets in flight, then
tp->delivered_mstamp may be either invalid (if the connection is
starting) or outdated (if the connection was idle). In that case,
we'll re-stamp tp->delivered_mstamp.
At first glance it seems t0 should always be the time when an skb was
transmitted, but actually this could over-estimate the rate due to
phase mismatch between transmit and ACK events. To track the delivery
rate, we ensure that if packets are in flight then t0 and and t1 are
times at which packets were marked delivered.
If the initial and final RTTs are different then one may be corrupted
by some sort of noise. The noise we see most often is sending gaps
caused by delayed, compressed, or stretched acks. This either affects
both RTTs equally or artificially reduces the final RTT. We approach
this by recording the info we need to compute the initial RTT
(duration of the "send phase" of the window) when we recorded the
associated inflight. Then, for a filter to avoid bandwidth
overestimates, we generalize the per-sample bandwidth computation
from:
bw = delivered / ack_phase_rtt
to the following:
bw = delivered / max(send_phase_rtt, ack_phase_rtt)
In large-scale experiments, this filtering approach incorporating
send_phase_rtt is effective at avoiding bandwidth overestimates due to
ACK compression or stretched ACKs.
Signed-off-by: Van Jacobson <vanj@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-19 21:39:14 -06:00
|
|
|
tcp_rate_skb_delivered(sk, skb, state->rate);
|
2017-10-04 13:59:58 -06:00
|
|
|
if (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED)
|
|
|
|
list_del_init(&skb->tcp_tsorted_anchor);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-24 22:20:15 -07:00
|
|
|
if (!before(TCP_SKB_CB(skb)->seq,
|
|
|
|
tcp_highest_sack_seq(tp)))
|
|
|
|
tcp_advance_highest_sack(sk, skb);
|
|
|
|
}
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
}
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
|
2019-02-25 02:42:33 -07:00
|
|
|
static struct sk_buff *tcp_sacktag_bsearch(struct sock *sk, u32 seq)
|
2017-10-05 23:21:27 -06:00
|
|
|
{
|
|
|
|
struct rb_node *parent, **p = &sk->tcp_rtx_queue.rb_node;
|
|
|
|
struct sk_buff *skb;
|
|
|
|
|
|
|
|
while (*p) {
|
|
|
|
parent = *p;
|
|
|
|
skb = rb_to_skb(parent);
|
|
|
|
if (before(seq, TCP_SKB_CB(skb)->seq)) {
|
|
|
|
p = &parent->rb_left;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
if (!before(seq, TCP_SKB_CB(skb)->end_seq)) {
|
|
|
|
p = &parent->rb_right;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
static struct sk_buff *tcp_sacktag_skip(struct sk_buff *skb, struct sock *sk,
|
2008-12-05 23:42:22 -07:00
|
|
|
u32 skip_to_seq)
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
{
|
2017-10-05 23:21:27 -06:00
|
|
|
if (skb && after(TCP_SKB_CB(skb)->seq, skip_to_seq))
|
|
|
|
return skb;
|
2008-03-03 13:10:16 -07:00
|
|
|
|
2019-02-25 02:42:33 -07:00
|
|
|
return tcp_sacktag_bsearch(sk, skip_to_seq);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
static struct sk_buff *tcp_maybe_skipping_dsack(struct sk_buff *skb,
|
|
|
|
struct sock *sk,
|
|
|
|
struct tcp_sack_block *next_dup,
|
2008-12-05 23:42:22 -07:00
|
|
|
struct tcp_sacktag_state *state,
|
|
|
|
u32 skip_to_seq)
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
{
|
2015-04-03 02:17:26 -06:00
|
|
|
if (!next_dup)
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
return skb;
|
|
|
|
|
|
|
|
if (before(next_dup->start_seq, skip_to_seq)) {
|
2019-02-25 02:42:33 -07:00
|
|
|
skb = tcp_sacktag_skip(skb, sk, next_dup->start_seq);
|
2008-12-05 23:42:22 -07:00
|
|
|
skb = tcp_sacktag_walk(skb, sk, NULL, state,
|
|
|
|
next_dup->start_seq, next_dup->end_seq,
|
|
|
|
1);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
|
2011-10-21 03:22:42 -06:00
|
|
|
static int tcp_sack_cache_ok(const struct tcp_sock *tp, const struct tcp_sack_block *cache)
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
{
|
|
|
|
return cache < tp->recv_sack_cache + ARRAY_SIZE(tp->recv_sack_cache);
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
static int
|
2011-10-21 03:22:42 -06:00
|
|
|
tcp_sacktag_write_queue(struct sock *sk, const struct sk_buff *ack_skb,
|
2015-04-30 17:10:57 -06:00
|
|
|
u32 prior_snd_una, struct tcp_sacktag_state *state)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2011-10-21 03:22:42 -06:00
|
|
|
const unsigned char *ptr = (skb_transport_header(ack_skb) +
|
|
|
|
TCP_SKB_CB(ack_skb)->sacked);
|
2007-11-15 20:49:47 -07:00
|
|
|
struct tcp_sack_block_wire *sp_wire = (struct tcp_sack_block_wire *)(ptr+2);
|
2008-07-19 01:07:02 -06:00
|
|
|
struct tcp_sack_block sp[TCP_NUM_SACKS];
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
struct tcp_sack_block *cache;
|
|
|
|
struct sk_buff *skb;
|
2008-07-19 01:07:02 -06:00
|
|
|
int num_sacks = min(TCP_NUM_SACKS, (ptr[1] - TCPOLEN_SACK_BASE) >> 3);
|
2007-11-15 20:49:47 -07:00
|
|
|
int used_sacks;
|
2012-05-16 17:15:34 -06:00
|
|
|
bool found_dup_sack = false;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
int i, j;
|
2007-02-05 00:35:57 -07:00
|
|
|
int first_sack_index;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2015-04-30 17:10:57 -06:00
|
|
|
state->flag = 0;
|
2017-11-08 14:01:27 -07:00
|
|
|
state->reord = tp->snd_nxt;
|
2008-12-05 23:42:22 -07:00
|
|
|
|
2017-11-08 14:01:27 -07:00
|
|
|
if (!tp->sacked_out)
|
2007-12-01 15:48:06 -07:00
|
|
|
tcp_highest_sack_reset(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2008-07-16 21:29:51 -06:00
|
|
|
found_dup_sack = tcp_check_dsack(sk, ack_skb, sp_wire,
|
2007-06-18 23:43:06 -06:00
|
|
|
num_sacks, prior_snd_una);
|
tcp: track data delivery rate for a TCP connection
This patch generates data delivery rate (throughput) samples on a
per-ACK basis. These rate samples can be used by congestion control
modules, and specifically will be used by TCP BBR in later patches in
this series.
Key state:
tp->delivered: Tracks the total number of data packets (original or not)
delivered so far. This is an already-existing field.
tp->delivered_mstamp: the last time tp->delivered was updated.
Algorithm:
A rate sample is calculated as (d1 - d0)/(t1 - t0) on a per-ACK basis:
d1: the current tp->delivered after processing the ACK
t1: the current time after processing the ACK
d0: the prior tp->delivered when the acked skb was transmitted
t0: the prior tp->delivered_mstamp when the acked skb was transmitted
When an skb is transmitted, we snapshot d0 and t0 in its control
block in tcp_rate_skb_sent().
When an ACK arrives, it may SACK and ACK some skbs. For each SACKed
or ACKed skb, tcp_rate_skb_delivered() updates the rate_sample struct
to reflect the latest (d0, t0).
Finally, tcp_rate_gen() generates a rate sample by storing
(d1 - d0) in rs->delivered and (t1 - t0) in rs->interval_us.
One caveat: if an skb was sent with no packets in flight, then
tp->delivered_mstamp may be either invalid (if the connection is
starting) or outdated (if the connection was idle). In that case,
we'll re-stamp tp->delivered_mstamp.
At first glance it seems t0 should always be the time when an skb was
transmitted, but actually this could over-estimate the rate due to
phase mismatch between transmit and ACK events. To track the delivery
rate, we ensure that if packets are in flight then t0 and and t1 are
times at which packets were marked delivered.
If the initial and final RTTs are different then one may be corrupted
by some sort of noise. The noise we see most often is sending gaps
caused by delayed, compressed, or stretched acks. This either affects
both RTTs equally or artificially reduces the final RTT. We approach
this by recording the info we need to compute the initial RTT
(duration of the "send phase" of the window) when we recorded the
associated inflight. Then, for a filter to avoid bandwidth
overestimates, we generalize the per-sample bandwidth computation
from:
bw = delivered / ack_phase_rtt
to the following:
bw = delivered / max(send_phase_rtt, ack_phase_rtt)
In large-scale experiments, this filtering approach incorporating
send_phase_rtt is effective at avoiding bandwidth overestimates due to
ACK compression or stretched ACKs.
Signed-off-by: Van Jacobson <vanj@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-19 21:39:14 -06:00
|
|
|
if (found_dup_sack) {
|
2015-04-30 17:10:57 -06:00
|
|
|
state->flag |= FLAG_DSACKING_ACK;
|
tcp: track data delivery rate for a TCP connection
This patch generates data delivery rate (throughput) samples on a
per-ACK basis. These rate samples can be used by congestion control
modules, and specifically will be used by TCP BBR in later patches in
this series.
Key state:
tp->delivered: Tracks the total number of data packets (original or not)
delivered so far. This is an already-existing field.
tp->delivered_mstamp: the last time tp->delivered was updated.
Algorithm:
A rate sample is calculated as (d1 - d0)/(t1 - t0) on a per-ACK basis:
d1: the current tp->delivered after processing the ACK
t1: the current time after processing the ACK
d0: the prior tp->delivered when the acked skb was transmitted
t0: the prior tp->delivered_mstamp when the acked skb was transmitted
When an skb is transmitted, we snapshot d0 and t0 in its control
block in tcp_rate_skb_sent().
When an ACK arrives, it may SACK and ACK some skbs. For each SACKed
or ACKed skb, tcp_rate_skb_delivered() updates the rate_sample struct
to reflect the latest (d0, t0).
Finally, tcp_rate_gen() generates a rate sample by storing
(d1 - d0) in rs->delivered and (t1 - t0) in rs->interval_us.
One caveat: if an skb was sent with no packets in flight, then
tp->delivered_mstamp may be either invalid (if the connection is
starting) or outdated (if the connection was idle). In that case,
we'll re-stamp tp->delivered_mstamp.
At first glance it seems t0 should always be the time when an skb was
transmitted, but actually this could over-estimate the rate due to
phase mismatch between transmit and ACK events. To track the delivery
rate, we ensure that if packets are in flight then t0 and and t1 are
times at which packets were marked delivered.
If the initial and final RTTs are different then one may be corrupted
by some sort of noise. The noise we see most often is sending gaps
caused by delayed, compressed, or stretched acks. This either affects
both RTTs equally or artificially reduces the final RTT. We approach
this by recording the info we need to compute the initial RTT
(duration of the "send phase" of the window) when we recorded the
associated inflight. Then, for a filter to avoid bandwidth
overestimates, we generalize the per-sample bandwidth computation
from:
bw = delivered / ack_phase_rtt
to the following:
bw = delivered / max(send_phase_rtt, ack_phase_rtt)
In large-scale experiments, this filtering approach incorporating
send_phase_rtt is effective at avoiding bandwidth overestimates due to
ACK compression or stretched ACKs.
Signed-off-by: Van Jacobson <vanj@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-19 21:39:14 -06:00
|
|
|
tp->delivered++; /* A spurious retransmission is delivered */
|
|
|
|
}
|
2007-02-05 00:36:42 -07:00
|
|
|
|
|
|
|
/* Eliminate too old ACKs, but take into
|
|
|
|
* account more or less fresh ones, they can
|
|
|
|
* contain valid SACK info.
|
|
|
|
*/
|
|
|
|
if (before(TCP_SKB_CB(ack_skb)->ack_seq, prior_snd_una - tp->max_window))
|
|
|
|
return 0;
|
|
|
|
|
2007-11-14 16:47:18 -07:00
|
|
|
if (!tp->packets_out)
|
|
|
|
goto out;
|
|
|
|
|
2007-11-15 20:49:47 -07:00
|
|
|
used_sacks = 0;
|
|
|
|
first_sack_index = 0;
|
|
|
|
for (i = 0; i < num_sacks; i++) {
|
2012-05-16 17:15:34 -06:00
|
|
|
bool dup_sack = !i && found_dup_sack;
|
2007-11-15 20:49:47 -07:00
|
|
|
|
2008-05-02 17:26:16 -06:00
|
|
|
sp[used_sacks].start_seq = get_unaligned_be32(&sp_wire[i].start_seq);
|
|
|
|
sp[used_sacks].end_seq = get_unaligned_be32(&sp_wire[i].end_seq);
|
2007-11-15 20:49:47 -07:00
|
|
|
|
|
|
|
if (!tcp_is_sackblock_valid(tp, dup_sack,
|
|
|
|
sp[used_sacks].start_seq,
|
|
|
|
sp[used_sacks].end_seq)) {
|
2008-07-03 02:05:41 -06:00
|
|
|
int mib_idx;
|
|
|
|
|
2007-11-15 20:49:47 -07:00
|
|
|
if (dup_sack) {
|
|
|
|
if (!tp->undo_marker)
|
2008-07-03 02:05:41 -06:00
|
|
|
mib_idx = LINUX_MIB_TCPDSACKIGNOREDNOUNDO;
|
2007-11-15 20:49:47 -07:00
|
|
|
else
|
2008-07-03 02:05:41 -06:00
|
|
|
mib_idx = LINUX_MIB_TCPDSACKIGNOREDOLD;
|
2007-11-15 20:49:47 -07:00
|
|
|
} else {
|
|
|
|
/* Don't count olds caused by ACK reordering */
|
|
|
|
if ((TCP_SKB_CB(ack_skb)->ack_seq != tp->snd_una) &&
|
|
|
|
!after(sp[used_sacks].end_seq, tp->snd_una))
|
|
|
|
continue;
|
2008-07-03 02:05:41 -06:00
|
|
|
mib_idx = LINUX_MIB_TCPSACKDISCARD;
|
2007-11-15 20:49:47 -07:00
|
|
|
}
|
2008-07-03 02:05:41 -06:00
|
|
|
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), mib_idx);
|
2007-11-15 20:49:47 -07:00
|
|
|
if (i == 0)
|
|
|
|
first_sack_index = -1;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Ignore very old stuff early */
|
|
|
|
if (!after(sp[used_sacks].end_seq, prior_snd_una))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
used_sacks++;
|
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
/* order SACK blocks to allow in order walk of the retrans queue */
|
|
|
|
for (i = used_sacks - 1; i > 0; i--) {
|
2007-12-31 15:57:14 -07:00
|
|
|
for (j = 0; j < i; j++) {
|
|
|
|
if (after(sp[j].start_seq, sp[j + 1].start_seq)) {
|
2009-03-21 14:36:17 -06:00
|
|
|
swap(sp[j], sp[j + 1]);
|
2005-11-10 18:14:59 -07:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
/* Track where the first SACK block goes to */
|
|
|
|
if (j == first_sack_index)
|
2007-12-31 15:57:14 -07:00
|
|
|
first_sack_index = j + 1;
|
2005-11-10 18:14:59 -07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-10-05 23:21:27 -06:00
|
|
|
state->mss_now = tcp_current_mss(sk);
|
|
|
|
skb = NULL;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
i = 0;
|
|
|
|
|
|
|
|
if (!tp->sacked_out) {
|
|
|
|
/* It's already past, so skip checking against it */
|
|
|
|
cache = tp->recv_sack_cache + ARRAY_SIZE(tp->recv_sack_cache);
|
|
|
|
} else {
|
|
|
|
cache = tp->recv_sack_cache;
|
|
|
|
/* Skip empty blocks in at head of the cache */
|
|
|
|
while (tcp_sack_cache_ok(tp, cache) && !cache->start_seq &&
|
|
|
|
!cache->end_seq)
|
|
|
|
cache++;
|
2007-02-05 00:35:57 -07:00
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
while (i < used_sacks) {
|
2007-11-15 20:49:47 -07:00
|
|
|
u32 start_seq = sp[i].start_seq;
|
|
|
|
u32 end_seq = sp[i].end_seq;
|
2012-05-16 17:15:34 -06:00
|
|
|
bool dup_sack = (found_dup_sack && (i == first_sack_index));
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
struct tcp_sack_block *next_dup = NULL;
|
[TCP]: Process DSACKs that reside within a SACK block
DSACK inside another SACK block were missed if start_seq of DSACK
was larger than SACK block's because sorting prioritizes full
processing of the SACK block before DSACK. After SACK block
sorting situation is like this:
SSSSSSSSS
D
SSSSSS
SSSSSSS
Because write_queue is walked in-order, when the first SACK block
has been processed, TCP is already past the skb for which the
DSACK arrived and we haven't taught it to backtrack (nor should
we), so TCP just continues processing by going to the next SACK
block after the DSACK (if any).
Whenever such DSACK is present, do an embedded checking during
the previous SACK block.
If the DSACK is below snd_una, there won't be overlapping SACK
block, and thus no problem in that case. Also if start_seq of
the DSACK is equal to the actual block, it will be processed
first.
Tested this by using netem to duplicate 15% of packets, and
by printing SACK block when found_dup_sack is true and the
selected skb in the dup_sack = 1 branch (if taken):
SACK block 0: 4344-5792 (relative to snd_una 2019137317)
SACK block 1: 4344-5792 (relative to snd_una 2019137317)
equal start seqnos => next_dup = 0, dup_sack = 1 won't occur...
SACK block 0: 5792-7240 (relative to snd_una 2019214061)
SACK block 1: 2896-7240 (relative to snd_una 2019214061)
DSACK skb match 5792-7240 (relative to snd_una)
...and next_dup = 1 case (after the not shown start_seq sort),
went to dup_sack = 1 branch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-01 01:09:37 -06:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
if (found_dup_sack && ((i + 1) == first_sack_index))
|
|
|
|
next_dup = &sp[i + 1];
|
2005-04-16 16:20:36 -06:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
/* Skip too early cached blocks */
|
|
|
|
while (tcp_sack_cache_ok(tp, cache) &&
|
|
|
|
!before(start_seq, cache->end_seq))
|
|
|
|
cache++;
|
|
|
|
|
|
|
|
/* Can skip some work by looking recv_sack_cache? */
|
|
|
|
if (tcp_sack_cache_ok(tp, cache) && !dup_sack &&
|
|
|
|
after(end_seq, cache->start_seq)) {
|
|
|
|
|
|
|
|
/* Head todo? */
|
|
|
|
if (before(start_seq, cache->start_seq)) {
|
2019-02-25 02:42:33 -07:00
|
|
|
skb = tcp_sacktag_skip(skb, sk, start_seq);
|
2007-12-31 15:57:14 -07:00
|
|
|
skb = tcp_sacktag_walk(skb, sk, next_dup,
|
2015-04-30 17:10:57 -06:00
|
|
|
state,
|
2007-12-31 15:57:14 -07:00
|
|
|
start_seq,
|
|
|
|
cache->start_seq,
|
2008-12-05 23:42:22 -07:00
|
|
|
dup_sack);
|
2007-02-05 00:35:57 -07:00
|
|
|
}
|
2005-11-10 18:14:59 -07:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
/* Rest of the block already fully processed? */
|
2007-11-16 17:17:05 -07:00
|
|
|
if (!after(end_seq, cache->end_seq))
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
goto advance_sp;
|
2007-11-16 17:17:05 -07:00
|
|
|
|
2007-12-31 15:57:14 -07:00
|
|
|
skb = tcp_maybe_skipping_dsack(skb, sk, next_dup,
|
2015-04-30 17:10:57 -06:00
|
|
|
state,
|
2008-12-05 23:42:22 -07:00
|
|
|
cache->end_seq);
|
[TCP]: Process DSACKs that reside within a SACK block
DSACK inside another SACK block were missed if start_seq of DSACK
was larger than SACK block's because sorting prioritizes full
processing of the SACK block before DSACK. After SACK block
sorting situation is like this:
SSSSSSSSS
D
SSSSSS
SSSSSSS
Because write_queue is walked in-order, when the first SACK block
has been processed, TCP is already past the skb for which the
DSACK arrived and we haven't taught it to backtrack (nor should
we), so TCP just continues processing by going to the next SACK
block after the DSACK (if any).
Whenever such DSACK is present, do an embedded checking during
the previous SACK block.
If the DSACK is below snd_una, there won't be overlapping SACK
block, and thus no problem in that case. Also if start_seq of
the DSACK is equal to the actual block, it will be processed
first.
Tested this by using netem to duplicate 15% of packets, and
by printing SACK block when found_dup_sack is true and the
selected skb in the dup_sack = 1 branch (if taken):
SACK block 0: 4344-5792 (relative to snd_una 2019137317)
SACK block 1: 4344-5792 (relative to snd_una 2019137317)
equal start seqnos => next_dup = 0, dup_sack = 1 won't occur...
SACK block 0: 5792-7240 (relative to snd_una 2019214061)
SACK block 1: 2896-7240 (relative to snd_una 2019214061)
DSACK skb match 5792-7240 (relative to snd_una)
...and next_dup = 1 case (after the not shown start_seq sort),
went to dup_sack = 1 branch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-01 01:09:37 -06:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
/* ...tail remains todo... */
|
2007-12-01 15:48:06 -07:00
|
|
|
if (tcp_highest_sack_seq(tp) == cache->end_seq) {
|
2007-11-16 17:17:05 -07:00
|
|
|
/* ...but better entrypoint exists! */
|
2007-12-01 15:48:06 -07:00
|
|
|
skb = tcp_highest_sack(sk);
|
2015-04-03 02:17:26 -06:00
|
|
|
if (!skb)
|
2007-12-01 15:48:06 -07:00
|
|
|
break;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
cache++;
|
|
|
|
goto walk;
|
[TCP]: Process DSACKs that reside within a SACK block
DSACK inside another SACK block were missed if start_seq of DSACK
was larger than SACK block's because sorting prioritizes full
processing of the SACK block before DSACK. After SACK block
sorting situation is like this:
SSSSSSSSS
D
SSSSSS
SSSSSSS
Because write_queue is walked in-order, when the first SACK block
has been processed, TCP is already past the skb for which the
DSACK arrived and we haven't taught it to backtrack (nor should
we), so TCP just continues processing by going to the next SACK
block after the DSACK (if any).
Whenever such DSACK is present, do an embedded checking during
the previous SACK block.
If the DSACK is below snd_una, there won't be overlapping SACK
block, and thus no problem in that case. Also if start_seq of
the DSACK is equal to the actual block, it will be processed
first.
Tested this by using netem to duplicate 15% of packets, and
by printing SACK block when found_dup_sack is true and the
selected skb in the dup_sack = 1 branch (if taken):
SACK block 0: 4344-5792 (relative to snd_una 2019137317)
SACK block 1: 4344-5792 (relative to snd_una 2019137317)
equal start seqnos => next_dup = 0, dup_sack = 1 won't occur...
SACK block 0: 5792-7240 (relative to snd_una 2019214061)
SACK block 1: 2896-7240 (relative to snd_una 2019214061)
DSACK skb match 5792-7240 (relative to snd_una)
...and next_dup = 1 case (after the not shown start_seq sort),
went to dup_sack = 1 branch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-01 01:09:37 -06:00
|
|
|
}
|
|
|
|
|
2019-02-25 02:42:33 -07:00
|
|
|
skb = tcp_sacktag_skip(skb, sk, cache->end_seq);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
/* Check overlap against next cached too (past this one already) */
|
|
|
|
cache++;
|
|
|
|
continue;
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2007-12-01 15:48:06 -07:00
|
|
|
if (!before(start_seq, tcp_highest_sack_seq(tp))) {
|
|
|
|
skb = tcp_highest_sack(sk);
|
2015-04-03 02:17:26 -06:00
|
|
|
if (!skb)
|
2007-12-01 15:48:06 -07:00
|
|
|
break;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2019-02-25 02:42:33 -07:00
|
|
|
skb = tcp_sacktag_skip(skb, sk, start_seq);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
|
|
|
|
walk:
|
2015-04-30 17:10:57 -06:00
|
|
|
skb = tcp_sacktag_walk(skb, sk, next_dup, state,
|
2008-12-05 23:42:22 -07:00
|
|
|
start_seq, end_seq, dup_sack);
|
2007-11-10 22:24:19 -07:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
advance_sp:
|
|
|
|
i++;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-15 20:50:37 -07:00
|
|
|
/* Clear the head of the cache sack blocks so we can skip it next time */
|
|
|
|
for (i = 0; i < ARRAY_SIZE(tp->recv_sack_cache) - used_sacks; i++) {
|
|
|
|
tp->recv_sack_cache[i].start_seq = 0;
|
|
|
|
tp->recv_sack_cache[i].end_seq = 0;
|
|
|
|
}
|
|
|
|
for (j = 0; j < used_sacks; j++)
|
|
|
|
tp->recv_sack_cache[i++] = sp[j];
|
|
|
|
|
2017-11-08 14:01:27 -07:00
|
|
|
if (inet_csk(sk)->icsk_ca_state != TCP_CA_Loss || tp->undo_marker)
|
|
|
|
tcp_check_sack_reordering(sk, state->reord, 0);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2015-04-29 12:28:30 -06:00
|
|
|
tcp_verify_left_out(tp);
|
2007-11-14 16:47:18 -07:00
|
|
|
out:
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
#if FASTRETRANS_DEBUG > 0
|
2008-07-25 22:43:18 -06:00
|
|
|
WARN_ON((int)tp->sacked_out < 0);
|
|
|
|
WARN_ON((int)tp->lost_out < 0);
|
|
|
|
WARN_ON((int)tp->retrans_out < 0);
|
|
|
|
WARN_ON((int)tcp_packets_in_flight(tp) < 0);
|
2005-04-16 16:20:36 -06:00
|
|
|
#endif
|
2015-04-30 17:10:57 -06:00
|
|
|
return state->flag;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2008-04-07 23:33:07 -06:00
|
|
|
/* Limits sacked_out so that sum with lost_out isn't ever larger than
|
2012-05-16 17:15:34 -06:00
|
|
|
* packets_out. Returns false if sacked_out adjustement wasn't necessary.
|
2007-02-22 00:01:36 -07:00
|
|
|
*/
|
2012-05-16 17:15:34 -06:00
|
|
|
static bool tcp_limit_reno_sacked(struct tcp_sock *tp)
|
2007-05-27 02:52:00 -06:00
|
|
|
{
|
|
|
|
u32 holes;
|
|
|
|
|
|
|
|
holes = max(tp->lost_out, 1U);
|
|
|
|
holes = min(holes, tp->packets_out);
|
|
|
|
|
|
|
|
if ((tp->sacked_out + holes) > tp->packets_out) {
|
|
|
|
tp->sacked_out = tp->packets_out - holes;
|
2012-05-16 17:15:34 -06:00
|
|
|
return true;
|
2007-05-27 02:52:00 -06:00
|
|
|
}
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2008-04-07 23:33:07 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* If we receive more dupacks than we expected counting segments
|
|
|
|
* in assumption of absent reordering, interpret this as reordering.
|
|
|
|
* The only another reason could be bug in receiver TCP.
|
|
|
|
*/
|
|
|
|
static void tcp_check_reno_reordering(struct sock *sk, const int addend)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2017-11-08 14:01:27 -07:00
|
|
|
|
|
|
|
if (!tcp_limit_reno_sacked(tp))
|
|
|
|
return;
|
|
|
|
|
|
|
|
tp->reordering = min_t(u32, tp->packets_out + addend,
|
|
|
|
sock_net(sk)->ipv4.sysctl_tcp_max_reordering);
|
2018-07-31 18:46:24 -06:00
|
|
|
tp->reord_seen++;
|
2017-11-08 14:01:27 -07:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPRENOREORDER);
|
2007-05-27 02:52:00 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Emulate SACKs for SACKless connection: account for a new dupack. */
|
|
|
|
|
2018-11-27 15:42:01 -07:00
|
|
|
static void tcp_add_reno_sack(struct sock *sk, int num_dupack)
|
2007-05-27 02:52:00 -06:00
|
|
|
{
|
2018-11-27 15:42:01 -07:00
|
|
|
if (num_dupack) {
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
u32 prior_sacked = tp->sacked_out;
|
|
|
|
s32 delivered;
|
2016-02-02 11:33:06 -07:00
|
|
|
|
2018-11-27 15:42:01 -07:00
|
|
|
tp->sacked_out += num_dupack;
|
|
|
|
tcp_check_reno_reordering(sk, 0);
|
|
|
|
delivered = tp->sacked_out - prior_sacked;
|
|
|
|
if (delivered > 0)
|
|
|
|
tp->delivered += delivered;
|
|
|
|
tcp_verify_left_out(tp);
|
|
|
|
}
|
2007-05-27 02:52:00 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Account for ACK, ACKing some data in Reno Recovery phase. */
|
|
|
|
|
|
|
|
static void tcp_remove_reno_sacks(struct sock *sk, int acked)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
if (acked > 0) {
|
|
|
|
/* One ACK acked hole. The rest eat duplicate ACKs. */
|
2016-02-02 11:33:06 -07:00
|
|
|
tp->delivered += max_t(int, acked - tp->sacked_out, 1);
|
2007-12-31 15:57:14 -07:00
|
|
|
if (acked - 1 >= tp->sacked_out)
|
2007-05-27 02:52:00 -06:00
|
|
|
tp->sacked_out = 0;
|
|
|
|
else
|
2007-12-31 15:57:14 -07:00
|
|
|
tp->sacked_out -= acked - 1;
|
2007-05-27 02:52:00 -06:00
|
|
|
}
|
|
|
|
tcp_check_reno_reordering(sk, acked);
|
2007-08-09 05:44:16 -06:00
|
|
|
tcp_verify_left_out(tp);
|
2007-05-27 02:52:00 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline void tcp_reset_reno_sack(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
tp->sacked_out = 0;
|
|
|
|
}
|
|
|
|
|
2014-08-22 15:15:22 -06:00
|
|
|
void tcp_clear_retrans(struct tcp_sock *tp)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
tp->retrans_out = 0;
|
|
|
|
tp->lost_out = 0;
|
|
|
|
tp->undo_marker = 0;
|
2014-07-02 13:07:16 -06:00
|
|
|
tp->undo_retrans = -1;
|
2014-08-22 15:15:22 -06:00
|
|
|
tp->sacked_out = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2014-08-22 15:15:22 -06:00
|
|
|
static inline void tcp_init_undo(struct tcp_sock *tp)
|
2007-10-11 18:34:57 -06:00
|
|
|
{
|
2014-08-22 15:15:22 -06:00
|
|
|
tp->undo_marker = tp->snd_una;
|
|
|
|
/* Retransmission still in flight may cause DSACKs later. */
|
|
|
|
tp->undo_retrans = tp->retrans_out ? : -1;
|
2007-10-11 18:34:57 -06:00
|
|
|
}
|
|
|
|
|
2018-05-16 17:40:16 -06:00
|
|
|
static bool tcp_is_rack(const struct sock *sk)
|
|
|
|
{
|
|
|
|
return sock_net(sk)->ipv4.sysctl_tcp_recovery & TCP_RACK_LOSS_DETECTION;
|
|
|
|
}
|
|
|
|
|
2018-05-16 17:40:14 -06:00
|
|
|
/* If we detect SACK reneging, forget all SACK information
|
2005-04-16 16:20:36 -06:00
|
|
|
* and reset tags completely, otherwise preserve SACKs. If receiver
|
|
|
|
* dropped its ofo queue, we will know this due to reneging detection.
|
|
|
|
*/
|
2018-05-16 17:40:14 -06:00
|
|
|
static void tcp_timeout_mark_lost(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2018-05-16 17:40:17 -06:00
|
|
|
struct sk_buff *skb, *head;
|
2018-05-16 17:40:14 -06:00
|
|
|
bool is_reneg; /* is receiver reneging on SACKs? */
|
|
|
|
|
2018-05-16 17:40:17 -06:00
|
|
|
head = tcp_rtx_queue_head(sk);
|
|
|
|
is_reneg = head && (TCP_SKB_CB(head)->sacked & TCPCB_SACKED_ACKED);
|
2018-05-16 17:40:14 -06:00
|
|
|
if (is_reneg) {
|
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPSACKRENEGING);
|
|
|
|
tp->sacked_out = 0;
|
|
|
|
/* Mark SACK reneging until we recover from this loss event. */
|
|
|
|
tp->is_sack_reneg = 1;
|
|
|
|
} else if (tcp_is_reno(tp)) {
|
|
|
|
tcp_reset_reno_sack(tp);
|
|
|
|
}
|
|
|
|
|
2018-05-16 17:40:17 -06:00
|
|
|
skb = head;
|
2018-05-16 17:40:14 -06:00
|
|
|
skb_rbtree_walk_from(skb) {
|
|
|
|
if (is_reneg)
|
|
|
|
TCP_SKB_CB(skb)->sacked &= ~TCPCB_SACKED_ACKED;
|
2018-05-16 17:40:17 -06:00
|
|
|
else if (tcp_is_rack(sk) && skb != head &&
|
|
|
|
tcp_rack_skb_timeout(tp, skb, 0) > 0)
|
|
|
|
continue; /* Don't mark recently sent ones lost yet */
|
2018-05-16 17:40:14 -06:00
|
|
|
tcp_mark_skb_lost(sk, skb);
|
|
|
|
}
|
|
|
|
tcp_verify_left_out(tp);
|
|
|
|
tcp_clear_all_retrans_hints(tp);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Enter Loss state. */
|
tcp: reduce spurious retransmits due to transient SACK reneging
This commit reduces spurious retransmits due to apparent SACK reneging
by only reacting to SACK reneging that persists for a short delay.
When a sequence space hole at snd_una is filled, some TCP receivers
send a series of ACKs as they apparently scan their out-of-order queue
and cumulatively ACK all the packets that have now been consecutiveyly
received. This is essentially misbehavior B in "Misbehaviors in TCP
SACK generation" ACM SIGCOMM Computer Communication Review, April
2011, so we suspect that this is from several common OSes (Windows
2000, Windows Server 2003, Windows XP). However, this issue has also
been seen in other cases, e.g. the netdev thread "TCP being hoodwinked
into spurious retransmissions by lack of timestamps?" from March 2014,
where the receiver was thought to be a BSD box.
Since snd_una would temporarily be adjacent to a previously SACKed
range in these scenarios, this receiver behavior triggered the Linux
SACK reneging code path in the sender. This led the sender to clear
the SACK scoreboard, enter CA_Loss, and spuriously retransmit
(potentially) every packet from the entire write queue at line rate
just a few milliseconds before the ACK for each packet arrives at the
sender.
To avoid such situations, now when a sender sees apparent reneging it
does not yet retransmit, but rather adjusts the RTO timer to give the
receiver a little time (max(RTT/2, 10ms)) to send us some more ACKs
that will restore sanity to the SACK scoreboard. If the reneging
persists until this RTO then, as before, we clear the SACK scoreboard
and enter CA_Loss.
A 10ms delay tolerates a receiver sending such a stream of ACKs at
56Kbit/sec. And to allow for receivers with slower or more congested
paths, we wait for at least RTT/2.
We validated the resulting max(RTT/2, 10ms) delay formula with a mix
of North American and South American Google web server traffic, and
found that for ACKs displaying transient reneging:
(1) 90% of inter-ACK delays were less than 10ms
(2) 99% of inter-ACK delays were less than RTT/2
In tests on Google web servers this commit reduced reneging events by
75%-90% (as measured by the TcpExtTCPSACKReneging counter), without
any measurable impact on latency for user HTTP and SPDY requests.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-04 17:12:29 -06:00
|
|
|
void tcp_enter_loss(struct sock *sk)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-10 01:03:31 -06:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2016-02-03 00:46:52 -07:00
|
|
|
struct net *net = sock_net(sk);
|
2017-04-07 12:42:05 -06:00
|
|
|
bool new_recovery = icsk->icsk_ca_state < TCP_CA_Recovery;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2018-05-16 17:40:15 -06:00
|
|
|
tcp_timeout_mark_lost(sk);
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Reduce ssthresh if it has not yet been made inside this window. */
|
2013-03-20 07:33:00 -06:00
|
|
|
if (icsk->icsk_ca_state <= TCP_CA_Disorder ||
|
|
|
|
!after(tp->high_seq, tp->snd_una) ||
|
2005-08-10 01:03:31 -06:00
|
|
|
(icsk->icsk_ca_state == TCP_CA_Loss && !icsk->icsk_retransmits)) {
|
|
|
|
tp->prior_ssthresh = tcp_current_ssthresh(sk);
|
2017-08-03 21:38:51 -06:00
|
|
|
tp->prior_cwnd = tp->snd_cwnd;
|
2005-08-10 01:03:31 -06:00
|
|
|
tp->snd_ssthresh = icsk->icsk_ca_ops->ssthresh(sk);
|
|
|
|
tcp_ca_event(sk, CA_EVENT_LOSS);
|
2014-08-22 15:15:22 -06:00
|
|
|
tcp_init_undo(tp);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2018-05-16 17:40:17 -06:00
|
|
|
tp->snd_cwnd = tcp_packets_in_flight(tp) + 1;
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->snd_cwnd_cnt = 0;
|
2017-05-16 15:00:04 -06:00
|
|
|
tp->snd_cwnd_stamp = tcp_jiffies32;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2013-08-12 17:41:25 -06:00
|
|
|
/* Timeout in disordered state after receiving substantial DUPACKs
|
|
|
|
* suggests that the degree of reordering is over-estimated.
|
|
|
|
*/
|
|
|
|
if (icsk->icsk_ca_state <= TCP_CA_Disorder &&
|
2016-02-03 00:46:52 -07:00
|
|
|
tp->sacked_out >= net->ipv4.sysctl_tcp_reordering)
|
2013-08-12 17:41:25 -06:00
|
|
|
tp->reordering = min_t(unsigned int, tp->reordering,
|
2016-02-03 00:46:52 -07:00
|
|
|
net->ipv4.sysctl_tcp_reordering);
|
2005-08-10 01:03:31 -06:00
|
|
|
tcp_set_ca_state(sk, TCP_CA_Loss);
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->high_seq = tp->snd_nxt;
|
2014-09-29 05:08:30 -06:00
|
|
|
tcp_ecn_queue_cwr(tp);
|
2013-03-20 07:33:00 -06:00
|
|
|
|
2017-04-07 12:42:05 -06:00
|
|
|
/* F-RTO RFC5682 sec 3.1 step 1: retransmit SND.UNA if no previous
|
|
|
|
* loss recovery is underway except recurring timeout(s) on
|
|
|
|
* the same SND.UNA (sec 3.2). Disable F-RTO on path MTU probing
|
2013-03-20 07:33:00 -06:00
|
|
|
*/
|
2017-10-26 22:55:10 -06:00
|
|
|
tp->frto = net->ipv4.sysctl_tcp_frto &&
|
2017-04-07 12:42:05 -06:00
|
|
|
(new_recovery || icsk->icsk_retransmits) &&
|
|
|
|
!inet_csk(sk)->icsk_mtup.probe_size;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2007-12-31 05:49:21 -07:00
|
|
|
/* If ACK arrived pointing to a remembered SACK, it means that our
|
|
|
|
* remembered SACKs do not reflect real state of receiver i.e.
|
|
|
|
* receiver _host_ is heavily congested (or buggy).
|
|
|
|
*
|
tcp: reduce spurious retransmits due to transient SACK reneging
This commit reduces spurious retransmits due to apparent SACK reneging
by only reacting to SACK reneging that persists for a short delay.
When a sequence space hole at snd_una is filled, some TCP receivers
send a series of ACKs as they apparently scan their out-of-order queue
and cumulatively ACK all the packets that have now been consecutiveyly
received. This is essentially misbehavior B in "Misbehaviors in TCP
SACK generation" ACM SIGCOMM Computer Communication Review, April
2011, so we suspect that this is from several common OSes (Windows
2000, Windows Server 2003, Windows XP). However, this issue has also
been seen in other cases, e.g. the netdev thread "TCP being hoodwinked
into spurious retransmissions by lack of timestamps?" from March 2014,
where the receiver was thought to be a BSD box.
Since snd_una would temporarily be adjacent to a previously SACKed
range in these scenarios, this receiver behavior triggered the Linux
SACK reneging code path in the sender. This led the sender to clear
the SACK scoreboard, enter CA_Loss, and spuriously retransmit
(potentially) every packet from the entire write queue at line rate
just a few milliseconds before the ACK for each packet arrives at the
sender.
To avoid such situations, now when a sender sees apparent reneging it
does not yet retransmit, but rather adjusts the RTO timer to give the
receiver a little time (max(RTT/2, 10ms)) to send us some more ACKs
that will restore sanity to the SACK scoreboard. If the reneging
persists until this RTO then, as before, we clear the SACK scoreboard
and enter CA_Loss.
A 10ms delay tolerates a receiver sending such a stream of ACKs at
56Kbit/sec. And to allow for receivers with slower or more congested
paths, we wait for at least RTT/2.
We validated the resulting max(RTT/2, 10ms) delay formula with a mix
of North American and South American Google web server traffic, and
found that for ACKs displaying transient reneging:
(1) 90% of inter-ACK delays were less than 10ms
(2) 99% of inter-ACK delays were less than RTT/2
In tests on Google web servers this commit reduced reneging events by
75%-90% (as measured by the TcpExtTCPSACKReneging counter), without
any measurable impact on latency for user HTTP and SPDY requests.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-04 17:12:29 -06:00
|
|
|
* To avoid big spurious retransmission bursts due to transient SACK
|
|
|
|
* scoreboard oddities that look like reneging, we give the receiver a
|
|
|
|
* little time (max(RTT/2, 10ms)) to send us some more ACKs that will
|
|
|
|
* restore sanity to the SACK scoreboard. If the apparent reneging
|
|
|
|
* persists until this RTO then we'll clear the SACK scoreboard.
|
2007-12-31 05:49:21 -07:00
|
|
|
*/
|
2012-05-16 17:15:34 -06:00
|
|
|
static bool tcp_check_sack_reneging(struct sock *sk, int flag)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2007-12-31 05:49:21 -07:00
|
|
|
if (flag & FLAG_SACK_RENEGING) {
|
tcp: reduce spurious retransmits due to transient SACK reneging
This commit reduces spurious retransmits due to apparent SACK reneging
by only reacting to SACK reneging that persists for a short delay.
When a sequence space hole at snd_una is filled, some TCP receivers
send a series of ACKs as they apparently scan their out-of-order queue
and cumulatively ACK all the packets that have now been consecutiveyly
received. This is essentially misbehavior B in "Misbehaviors in TCP
SACK generation" ACM SIGCOMM Computer Communication Review, April
2011, so we suspect that this is from several common OSes (Windows
2000, Windows Server 2003, Windows XP). However, this issue has also
been seen in other cases, e.g. the netdev thread "TCP being hoodwinked
into spurious retransmissions by lack of timestamps?" from March 2014,
where the receiver was thought to be a BSD box.
Since snd_una would temporarily be adjacent to a previously SACKed
range in these scenarios, this receiver behavior triggered the Linux
SACK reneging code path in the sender. This led the sender to clear
the SACK scoreboard, enter CA_Loss, and spuriously retransmit
(potentially) every packet from the entire write queue at line rate
just a few milliseconds before the ACK for each packet arrives at the
sender.
To avoid such situations, now when a sender sees apparent reneging it
does not yet retransmit, but rather adjusts the RTO timer to give the
receiver a little time (max(RTT/2, 10ms)) to send us some more ACKs
that will restore sanity to the SACK scoreboard. If the reneging
persists until this RTO then, as before, we clear the SACK scoreboard
and enter CA_Loss.
A 10ms delay tolerates a receiver sending such a stream of ACKs at
56Kbit/sec. And to allow for receivers with slower or more congested
paths, we wait for at least RTT/2.
We validated the resulting max(RTT/2, 10ms) delay formula with a mix
of North American and South American Google web server traffic, and
found that for ACKs displaying transient reneging:
(1) 90% of inter-ACK delays were less than 10ms
(2) 99% of inter-ACK delays were less than RTT/2
In tests on Google web servers this commit reduced reneging events by
75%-90% (as measured by the TcpExtTCPSACKReneging counter), without
any measurable impact on latency for user HTTP and SPDY requests.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-04 17:12:29 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
unsigned long delay = max(usecs_to_jiffies(tp->srtt_us >> 4),
|
|
|
|
msecs_to_jiffies(10));
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2005-08-09 21:10:42 -06:00
|
|
|
inet_csk_reset_xmit_timer(sk, ICSK_TIME_RETRANS,
|
tcp: reduce spurious retransmits due to transient SACK reneging
This commit reduces spurious retransmits due to apparent SACK reneging
by only reacting to SACK reneging that persists for a short delay.
When a sequence space hole at snd_una is filled, some TCP receivers
send a series of ACKs as they apparently scan their out-of-order queue
and cumulatively ACK all the packets that have now been consecutiveyly
received. This is essentially misbehavior B in "Misbehaviors in TCP
SACK generation" ACM SIGCOMM Computer Communication Review, April
2011, so we suspect that this is from several common OSes (Windows
2000, Windows Server 2003, Windows XP). However, this issue has also
been seen in other cases, e.g. the netdev thread "TCP being hoodwinked
into spurious retransmissions by lack of timestamps?" from March 2014,
where the receiver was thought to be a BSD box.
Since snd_una would temporarily be adjacent to a previously SACKed
range in these scenarios, this receiver behavior triggered the Linux
SACK reneging code path in the sender. This led the sender to clear
the SACK scoreboard, enter CA_Loss, and spuriously retransmit
(potentially) every packet from the entire write queue at line rate
just a few milliseconds before the ACK for each packet arrives at the
sender.
To avoid such situations, now when a sender sees apparent reneging it
does not yet retransmit, but rather adjusts the RTO timer to give the
receiver a little time (max(RTT/2, 10ms)) to send us some more ACKs
that will restore sanity to the SACK scoreboard. If the reneging
persists until this RTO then, as before, we clear the SACK scoreboard
and enter CA_Loss.
A 10ms delay tolerates a receiver sending such a stream of ACKs at
56Kbit/sec. And to allow for receivers with slower or more congested
paths, we wait for at least RTT/2.
We validated the resulting max(RTT/2, 10ms) delay formula with a mix
of North American and South American Google web server traffic, and
found that for ACKs displaying transient reneging:
(1) 90% of inter-ACK delays were less than 10ms
(2) 99% of inter-ACK delays were less than RTT/2
In tests on Google web servers this commit reduced reneging events by
75%-90% (as measured by the TcpExtTCPSACKReneging counter), without
any measurable impact on latency for user HTTP and SPDY requests.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-04 17:12:29 -06:00
|
|
|
delay, TCP_RTO_MAX);
|
2012-05-16 17:15:34 -06:00
|
|
|
return true;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2007-11-15 20:39:31 -07:00
|
|
|
/* Heurestics to calculate number of duplicate ACKs. There's no dupACKs
|
|
|
|
* counter when SACK is enabled (without SACK, sacked_out is used for
|
|
|
|
* that purpose).
|
|
|
|
*
|
|
|
|
* With reordering, holes may still be in flight, so RFC3517 recovery
|
|
|
|
* uses pure sacked_out (total number of SACKed segments) even though
|
|
|
|
* it violates the RFC that uses duplicate ACKs, often these are equal
|
|
|
|
* but when e.g. out-of-window ACKs or packet duplication occurs,
|
|
|
|
* they differ. Since neither occurs due to loss, TCP should really
|
|
|
|
* ignore them.
|
|
|
|
*/
|
2011-10-21 03:22:42 -06:00
|
|
|
static inline int tcp_dupack_heuristics(const struct tcp_sock *tp)
|
2007-11-15 20:39:31 -07:00
|
|
|
{
|
2017-11-08 14:01:26 -07:00
|
|
|
return tp->sacked_out + 1;
|
2007-11-15 20:39:31 -07:00
|
|
|
}
|
|
|
|
|
2017-11-08 14:01:26 -07:00
|
|
|
/* Linux NewReno/SACK/ECN state machine.
|
2005-04-16 16:20:36 -06:00
|
|
|
* --------------------------------------
|
|
|
|
*
|
|
|
|
* "Open" Normal state, no dubious events, fast path.
|
|
|
|
* "Disorder" In all the respects it is "Open",
|
|
|
|
* but requires a bit more attention. It is entered when
|
|
|
|
* we see some SACKs or dupacks. It is split of "Open"
|
|
|
|
* mainly to move some processing from fast path to slow one.
|
|
|
|
* "CWR" CWND was reduced due to some Congestion Notification event.
|
|
|
|
* It can be ECN, ICMP source quench, local device congestion.
|
|
|
|
* "Recovery" CWND was reduced, we are fast-retransmitting.
|
|
|
|
* "Loss" CWND was reduced due to RTO timeout or SACK reneging.
|
|
|
|
*
|
|
|
|
* tcp_fastretrans_alert() is entered:
|
|
|
|
* - each incoming ACK, if state is not "Open"
|
|
|
|
* - when arrived ACK is unusual, namely:
|
|
|
|
* * SACK
|
|
|
|
* * Duplicate ACK.
|
|
|
|
* * ECN ECE.
|
|
|
|
*
|
|
|
|
* Counting packets in flight is pretty simple.
|
|
|
|
*
|
|
|
|
* in_flight = packets_out - left_out + retrans_out
|
|
|
|
*
|
|
|
|
* packets_out is SND.NXT-SND.UNA counted in packets.
|
|
|
|
*
|
|
|
|
* retrans_out is number of retransmitted segments.
|
|
|
|
*
|
|
|
|
* left_out is number of segments left network, but not ACKed yet.
|
|
|
|
*
|
|
|
|
* left_out = sacked_out + lost_out
|
|
|
|
*
|
|
|
|
* sacked_out: Packets, which arrived to receiver out of order
|
|
|
|
* and hence not ACKed. With SACKs this number is simply
|
|
|
|
* amount of SACKed data. Even without SACKs
|
|
|
|
* it is easy to give pretty reliable estimate of this number,
|
|
|
|
* counting duplicate ACKs.
|
|
|
|
*
|
|
|
|
* lost_out: Packets lost by network. TCP has no explicit
|
|
|
|
* "loss notification" feedback from network (for now).
|
|
|
|
* It means that this number can be only _guessed_.
|
|
|
|
* Actually, it is the heuristics to predict lossage that
|
|
|
|
* distinguishes different algorithms.
|
|
|
|
*
|
|
|
|
* F.e. after RTO, when all the queue is considered as lost,
|
|
|
|
* lost_out = packets_out and in_flight = retrans_out.
|
|
|
|
*
|
2017-01-12 23:11:36 -07:00
|
|
|
* Essentially, we have now a few algorithms detecting
|
2005-04-16 16:20:36 -06:00
|
|
|
* lost packets.
|
|
|
|
*
|
2017-01-12 23:11:36 -07:00
|
|
|
* If the receiver supports SACK:
|
|
|
|
*
|
|
|
|
* RFC6675/3517: It is the conventional algorithm. A packet is
|
|
|
|
* considered lost if the number of higher sequence packets
|
|
|
|
* SACKed is greater than or equal the DUPACK thoreshold
|
|
|
|
* (reordering). This is implemented in tcp_mark_head_lost and
|
|
|
|
* tcp_update_scoreboard.
|
|
|
|
*
|
|
|
|
* RACK (draft-ietf-tcpm-rack-01): it is a newer algorithm
|
|
|
|
* (2017-) that checks timing instead of counting DUPACKs.
|
|
|
|
* Essentially a packet is considered lost if it's not S/ACKed
|
|
|
|
* after RTT + reordering_window, where both metrics are
|
|
|
|
* dynamically measured and adjusted. This is implemented in
|
|
|
|
* tcp_rack_mark_lost.
|
|
|
|
*
|
|
|
|
* If the receiver does not support SACK:
|
|
|
|
*
|
|
|
|
* NewReno (RFC6582): in Recovery we assume that one segment
|
2005-04-16 16:20:36 -06:00
|
|
|
* is lost (classic Reno). While we are in Recovery and
|
|
|
|
* a partial ACK arrives, we assume that one more packet
|
|
|
|
* is lost (NewReno). This heuristics are the same in NewReno
|
|
|
|
* and SACK.
|
|
|
|
*
|
|
|
|
* Really tricky (and requiring careful tuning) part of algorithm
|
|
|
|
* is hidden in functions tcp_time_to_recover() and tcp_xmit_retransmit_queue().
|
|
|
|
* The first determines the moment _when_ we should reduce CWND and,
|
|
|
|
* hence, slow down forward transmission. In fact, it determines the moment
|
|
|
|
* when we decide that hole is caused by loss, rather than by a reorder.
|
|
|
|
*
|
|
|
|
* tcp_xmit_retransmit_queue() decides, _what_ we should retransmit to fill
|
|
|
|
* holes, caused by lost packets.
|
|
|
|
*
|
|
|
|
* And the most logically complicated part of algorithm is undo
|
|
|
|
* heuristics. We detect false retransmits due to both too early
|
|
|
|
* fast retransmit (reordering) and underestimated RTO, analyzing
|
|
|
|
* timestamps and D-SACKs. When we detect that some segments were
|
|
|
|
* retransmitted by mistake and CWND reduction was wrong, we undo
|
|
|
|
* window reduction and abort recovery phase. This logic is hidden
|
|
|
|
* inside several functions named tcp_try_undo_<something>.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* This function decides, when we should leave Disordered state
|
|
|
|
* and enter Recovery phase, reducing congestion window.
|
|
|
|
*
|
|
|
|
* Main question: may we further continue forward transmission
|
|
|
|
* with the same cwnd?
|
|
|
|
*/
|
2012-05-16 17:15:34 -06:00
|
|
|
static bool tcp_time_to_recover(struct sock *sk, int flag)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Trick#1: The loss is proven. */
|
|
|
|
if (tp->lost_out)
|
2012-05-16 17:15:34 -06:00
|
|
|
return true;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Not-A-Trick#2 : Classic rule... */
|
2018-05-16 17:40:11 -06:00
|
|
|
if (!tcp_is_rack(sk) && tcp_dupack_heuristics(tp) > tp->reordering)
|
2012-05-16 17:15:34 -06:00
|
|
|
return true;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2012-01-19 07:42:21 -07:00
|
|
|
/* Detect loss in event "A" above by marking head of queue up as lost.
|
2017-11-08 14:01:26 -07:00
|
|
|
* For non-SACK(Reno) senders, the first "packets" number of segments
|
2012-01-19 07:42:21 -07:00
|
|
|
* are considered lost. For RFC3517 SACK, a segment is considered lost if it
|
|
|
|
* has at least tp->reordering SACKed seqments above it; "packets" refers to
|
|
|
|
* the maximum SACKed segments to pass before reaching this limit.
|
2007-11-15 20:39:31 -07:00
|
|
|
*/
|
2010-10-13 19:42:30 -06:00
|
|
|
static void tcp_mark_head_lost(struct sock *sk, int packets, int mark_head)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
struct sk_buff *skb;
|
tcp: fix tcp_mark_head_lost to check skb len before fragmenting
This commit fixes a corner case in tcp_mark_head_lost() which was
causing the WARN_ON(len > skb->len) in tcp_fragment() to fire.
tcp_mark_head_lost() was assuming that if a packet has
tcp_skb_pcount(skb) of N, then it's safe to fragment off a prefix of
M*mss bytes, for any M < N. But with the tricky way TCP pcounts are
maintained, this is not always true.
For example, suppose the sender sends 4 1-byte packets and have the
last 3 packet sacked. It will merge the last 3 packets in the write
queue into an skb with pcount = 3 and len = 3 bytes. If another
recovery happens after a sack reneging event, tcp_mark_head_lost()
may attempt to split the skb assuming it has more than 2*MSS bytes.
This sounds very counterintuitive, but as the commit description for
the related commit c0638c247f55 ("tcp: don't fragment SACKed skbs in
tcp_mark_head_lost()") notes, this is because tcp_shifted_skb()
coalesces adjacent regions of SACKed skbs, and when doing this it
preserves the sum of their packet counts in order to reflect the
real-world dynamics on the wire. The c0638c247f55 commit tried to
avoid problems by not fragmenting SACKed skbs, since SACKed skbs are
where the non-proportionality between pcount and skb->len/mss is known
to be possible. However, that commit did not handle the case where
during a reneging event one of these weird SACKed skbs becomes an
un-SACKed skb, which tcp_mark_head_lost() can then try to fragment.
The fix is to simply mark the entire skb lost when this happens.
This makes the recovery slightly more aggressive in such corner
cases before we detect reordering. But once we detect reordering
this code path is by-passed because FACK is disabled.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-01-25 15:01:53 -07:00
|
|
|
int cnt, oldcnt, lost;
|
2008-04-07 23:32:38 -06:00
|
|
|
unsigned int mss;
|
2012-01-19 07:42:21 -07:00
|
|
|
/* Use SACK to deduce losses of new sequences sent during recovery */
|
|
|
|
const u32 loss_high = tcp_is_sack(tp) ? tp->snd_nxt : tp->high_seq;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2008-07-25 22:43:18 -06:00
|
|
|
WARN_ON(packets > tp->packets_out);
|
2017-10-05 23:21:24 -06:00
|
|
|
skb = tp->lost_skb_hint;
|
|
|
|
if (skb) {
|
2010-10-13 19:42:30 -06:00
|
|
|
/* Head already handled? */
|
2017-10-05 23:21:24 -06:00
|
|
|
if (mark_head && after(TCP_SKB_CB(skb)->seq, tp->snd_una))
|
2010-10-13 19:42:30 -06:00
|
|
|
return;
|
2017-10-05 23:21:24 -06:00
|
|
|
cnt = tp->lost_cnt_hint;
|
2005-11-10 18:14:59 -07:00
|
|
|
} else {
|
2017-10-05 23:21:27 -06:00
|
|
|
skb = tcp_rtx_queue_head(sk);
|
2005-11-10 18:14:59 -07:00
|
|
|
cnt = 0;
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-10-05 23:21:27 -06:00
|
|
|
skb_rbtree_walk_from(skb) {
|
2005-11-10 18:14:59 -07:00
|
|
|
/* TODO: do this better */
|
|
|
|
/* this is not the most efficient way to do this... */
|
|
|
|
tp->lost_skb_hint = skb;
|
|
|
|
tp->lost_cnt_hint = cnt;
|
2007-11-15 20:39:31 -07:00
|
|
|
|
2012-01-19 07:42:21 -07:00
|
|
|
if (after(TCP_SKB_CB(skb)->end_seq, loss_high))
|
2008-04-07 23:32:38 -06:00
|
|
|
break;
|
|
|
|
|
|
|
|
oldcnt = cnt;
|
2017-11-08 14:01:26 -07:00
|
|
|
if (tcp_is_reno(tp) ||
|
2007-11-15 20:39:31 -07:00
|
|
|
(TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED))
|
|
|
|
cnt += tcp_skb_pcount(skb);
|
|
|
|
|
2008-04-07 23:32:38 -06:00
|
|
|
if (cnt > packets) {
|
2017-11-08 14:01:26 -07:00
|
|
|
if (tcp_is_sack(tp) ||
|
2012-03-02 14:36:51 -07:00
|
|
|
(TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED) ||
|
2010-09-24 07:22:06 -06:00
|
|
|
(oldcnt >= packets))
|
2008-04-07 23:32:38 -06:00
|
|
|
break;
|
|
|
|
|
2015-06-11 10:15:18 -06:00
|
|
|
mss = tcp_skb_mss(skb);
|
tcp: fix tcp_mark_head_lost to check skb len before fragmenting
This commit fixes a corner case in tcp_mark_head_lost() which was
causing the WARN_ON(len > skb->len) in tcp_fragment() to fire.
tcp_mark_head_lost() was assuming that if a packet has
tcp_skb_pcount(skb) of N, then it's safe to fragment off a prefix of
M*mss bytes, for any M < N. But with the tricky way TCP pcounts are
maintained, this is not always true.
For example, suppose the sender sends 4 1-byte packets and have the
last 3 packet sacked. It will merge the last 3 packets in the write
queue into an skb with pcount = 3 and len = 3 bytes. If another
recovery happens after a sack reneging event, tcp_mark_head_lost()
may attempt to split the skb assuming it has more than 2*MSS bytes.
This sounds very counterintuitive, but as the commit description for
the related commit c0638c247f55 ("tcp: don't fragment SACKed skbs in
tcp_mark_head_lost()") notes, this is because tcp_shifted_skb()
coalesces adjacent regions of SACKed skbs, and when doing this it
preserves the sum of their packet counts in order to reflect the
real-world dynamics on the wire. The c0638c247f55 commit tried to
avoid problems by not fragmenting SACKed skbs, since SACKed skbs are
where the non-proportionality between pcount and skb->len/mss is known
to be possible. However, that commit did not handle the case where
during a reneging event one of these weird SACKed skbs becomes an
un-SACKed skb, which tcp_mark_head_lost() can then try to fragment.
The fix is to simply mark the entire skb lost when this happens.
This makes the recovery slightly more aggressive in such corner
cases before we detect reordering. But once we detect reordering
this code path is by-passed because FACK is disabled.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-01-25 15:01:53 -07:00
|
|
|
/* If needed, chop off the prefix to mark as lost. */
|
|
|
|
lost = (packets - oldcnt) * mss;
|
|
|
|
if (lost < skb->len &&
|
2017-10-05 23:21:27 -06:00
|
|
|
tcp_fragment(sk, TCP_FRAG_IN_RTX_QUEUE, skb,
|
|
|
|
lost, mss, GFP_ATOMIC) < 0)
|
2008-04-07 23:32:38 -06:00
|
|
|
break;
|
|
|
|
cnt = packets;
|
|
|
|
}
|
|
|
|
|
2008-09-20 22:19:22 -06:00
|
|
|
tcp_skb_mark_lost(tp, skb);
|
2010-10-13 19:42:30 -06:00
|
|
|
|
|
|
|
if (mark_head)
|
|
|
|
break;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2007-08-09 05:44:16 -06:00
|
|
|
tcp_verify_left_out(tp);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Account newly detected lost packet(s) */
|
|
|
|
|
2007-11-15 20:39:31 -07:00
|
|
|
static void tcp_update_scoreboard(struct sock *sk, int fast_rexmit)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2018-05-16 17:40:12 -06:00
|
|
|
if (tcp_is_sack(tp)) {
|
2007-11-15 20:39:31 -07:00
|
|
|
int sacked_upto = tp->sacked_out - tp->reordering;
|
2010-10-13 19:42:30 -06:00
|
|
|
if (sacked_upto >= 0)
|
|
|
|
tcp_mark_head_lost(sk, sacked_upto, 0);
|
|
|
|
else if (fast_rexmit)
|
|
|
|
tcp_mark_head_lost(sk, 1, 1);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-10-16 22:57:44 -06:00
|
|
|
static bool tcp_tsopt_ecr_before(const struct tcp_sock *tp, u32 when)
|
|
|
|
{
|
|
|
|
return tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr &&
|
|
|
|
before(tp->rx_opt.rcv_tsecr, when);
|
|
|
|
}
|
|
|
|
|
2015-10-16 22:57:46 -06:00
|
|
|
/* skb is spurious retransmitted if the returned timestamp echo
|
|
|
|
* reply is prior to the skb transmission time
|
|
|
|
*/
|
|
|
|
static bool tcp_skb_spurious_retrans(const struct tcp_sock *tp,
|
|
|
|
const struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
return (TCP_SKB_CB(skb)->sacked & TCPCB_RETRANS) &&
|
|
|
|
tcp_tsopt_ecr_before(tp, tcp_skb_timestamp(skb));
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Nothing was retransmitted or returned timestamp is less
|
|
|
|
* than timestamp of the first retransmission.
|
|
|
|
*/
|
2012-07-19 15:32:18 -06:00
|
|
|
static inline bool tcp_packet_delayed(const struct tcp_sock *tp)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2019-04-29 16:46:13 -06:00
|
|
|
return tp->retrans_stamp &&
|
2015-10-16 22:57:44 -06:00
|
|
|
tcp_tsopt_ecr_before(tp, tp->retrans_stamp);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Undo procedures. */
|
|
|
|
|
2014-11-04 12:15:08 -07:00
|
|
|
/* We can clear retrans_stamp when there are no retransmissions in the
|
|
|
|
* window. It would seem that it is trivially available for us in
|
|
|
|
* tp->retrans_out, however, that kind of assumptions doesn't consider
|
|
|
|
* what will happen if errors occur when sending retransmission for the
|
|
|
|
* second time. ...It could the that such segment has only
|
|
|
|
* TCPCB_EVER_RETRANS set at the present time. It seems that checking
|
|
|
|
* the head skb is enough except for some reneging corner cases that
|
|
|
|
* are not worth the effort.
|
|
|
|
*
|
|
|
|
* Main reason for all this complexity is the fact that connection dying
|
|
|
|
* time now depends on the validity of the retrans_stamp, in particular,
|
|
|
|
* that successive retransmissions of a segment must not advance
|
|
|
|
* retrans_stamp under any conditions.
|
|
|
|
*/
|
|
|
|
static bool tcp_any_retrans_done(const struct sock *sk)
|
|
|
|
{
|
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *skb;
|
|
|
|
|
|
|
|
if (tp->retrans_out)
|
|
|
|
return true;
|
|
|
|
|
2017-10-05 23:21:27 -06:00
|
|
|
skb = tcp_rtx_queue_head(sk);
|
2014-11-04 12:15:08 -07:00
|
|
|
if (unlikely(skb && TCP_SKB_CB(skb)->sacked & TCPCB_EVER_RETRANS))
|
|
|
|
return true;
|
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
static void DBGUNDO(struct sock *sk, const char *msg)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2017-09-10 20:02:25 -06:00
|
|
|
#if FASTRETRANS_DEBUG > 1
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
struct inet_sock *inet = inet_sk(sk);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
|
2008-04-14 05:09:36 -06:00
|
|
|
if (sk->sk_family == AF_INET) {
|
2012-05-15 08:11:54 -06:00
|
|
|
pr_debug("Undo %s %pI4/%u c%u l%u ss%u/%u p%u\n",
|
|
|
|
msg,
|
|
|
|
&inet->inet_daddr, ntohs(inet->inet_dport),
|
|
|
|
tp->snd_cwnd, tcp_left_out(tp),
|
|
|
|
tp->snd_ssthresh, tp->prior_ssthresh,
|
|
|
|
tp->packets_out);
|
2008-04-14 05:09:36 -06:00
|
|
|
}
|
2011-12-10 02:48:31 -07:00
|
|
|
#if IS_ENABLED(CONFIG_IPV6)
|
2008-04-14 05:09:36 -06:00
|
|
|
else if (sk->sk_family == AF_INET6) {
|
2012-05-15 08:11:54 -06:00
|
|
|
pr_debug("Undo %s %pI6/%u c%u l%u ss%u/%u p%u\n",
|
|
|
|
msg,
|
2016-09-22 18:54:00 -06:00
|
|
|
&sk->sk_v6_daddr, ntohs(inet->inet_dport),
|
2012-05-15 08:11:54 -06:00
|
|
|
tp->snd_cwnd, tcp_left_out(tp),
|
|
|
|
tp->snd_ssthresh, tp->prior_ssthresh,
|
|
|
|
tp->packets_out);
|
2008-04-14 05:09:36 -06:00
|
|
|
}
|
|
|
|
#endif
|
2005-04-16 16:20:36 -06:00
|
|
|
#endif
|
2017-09-10 20:02:25 -06:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2013-05-29 08:20:13 -06:00
|
|
|
static void tcp_undo_cwnd_reduction(struct sock *sk, bool unmark_loss)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-10 01:03:31 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2013-05-29 08:20:12 -06:00
|
|
|
if (unmark_loss) {
|
|
|
|
struct sk_buff *skb;
|
|
|
|
|
2017-10-05 23:21:27 -06:00
|
|
|
skb_rbtree_walk(skb, &sk->tcp_rtx_queue) {
|
2013-05-29 08:20:12 -06:00
|
|
|
TCP_SKB_CB(skb)->sacked &= ~TCPCB_LOST;
|
|
|
|
}
|
|
|
|
tp->lost_out = 0;
|
|
|
|
tcp_clear_all_retrans_hints(tp);
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
if (tp->prior_ssthresh) {
|
2005-08-10 01:03:31 -06:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
|
2016-11-21 06:18:38 -07:00
|
|
|
tp->snd_cwnd = icsk->icsk_ca_ops->undo_cwnd(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2013-05-29 08:20:13 -06:00
|
|
|
if (tp->prior_ssthresh > tp->snd_ssthresh) {
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->snd_ssthresh = tp->prior_ssthresh;
|
2014-09-29 05:08:30 -06:00
|
|
|
tcp_ecn_withdraw_cwr(tp);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
2017-05-16 15:00:04 -06:00
|
|
|
tp->snd_cwnd_stamp = tcp_jiffies32;
|
2013-05-29 08:20:13 -06:00
|
|
|
tp->undo_marker = 0;
|
2017-12-07 12:33:31 -07:00
|
|
|
tp->rack.advanced = 1; /* Force RACK to re-exam losses */
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2012-07-19 15:32:18 -06:00
|
|
|
static inline bool tcp_may_undo(const struct tcp_sock *tp)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2007-12-31 15:57:14 -07:00
|
|
|
return tp->undo_marker && (!tp->undo_retrans || tcp_packet_delayed(tp));
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* People celebrate: "We love our President!" */
|
2012-05-16 17:15:34 -06:00
|
|
|
static bool tcp_try_undo_recovery(struct sock *sk)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
if (tcp_may_undo(tp)) {
|
2008-07-03 02:05:41 -06:00
|
|
|
int mib_idx;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Happy end! We did not retransmit anything
|
|
|
|
* or our original transmission succeeded.
|
|
|
|
*/
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
DBGUNDO(sk, inet_csk(sk)->icsk_ca_state == TCP_CA_Loss ? "loss" : "retrans");
|
2013-05-29 08:20:13 -06:00
|
|
|
tcp_undo_cwnd_reduction(sk, false);
|
2005-08-10 01:03:31 -06:00
|
|
|
if (inet_csk(sk)->icsk_ca_state == TCP_CA_Loss)
|
2008-07-03 02:05:41 -06:00
|
|
|
mib_idx = LINUX_MIB_TCPLOSSUNDO;
|
2005-04-16 16:20:36 -06:00
|
|
|
else
|
2008-07-03 02:05:41 -06:00
|
|
|
mib_idx = LINUX_MIB_TCPFULLUNDO;
|
|
|
|
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), mib_idx);
|
2017-11-03 17:38:48 -06:00
|
|
|
} else if (tp->rack.reo_wnd_persist) {
|
|
|
|
tp->rack.reo_wnd_persist--;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2007-08-09 06:14:46 -06:00
|
|
|
if (tp->snd_una == tp->high_seq && tcp_is_reno(tp)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Hold old state until something *above* high_seq
|
|
|
|
* is ACKed. For Reno it is MUST to prevent false
|
|
|
|
* fast retransmits (RFC2582). SACK TCP is safe. */
|
2014-11-04 12:15:08 -07:00
|
|
|
if (!tcp_any_retrans_done(sk))
|
|
|
|
tp->retrans_stamp = 0;
|
2012-05-16 17:15:34 -06:00
|
|
|
return true;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2005-08-10 01:03:31 -06:00
|
|
|
tcp_set_ca_state(sk, TCP_CA_Open);
|
2017-12-07 14:41:34 -07:00
|
|
|
tp->is_sack_reneg = 0;
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Try to undo cwnd reduction, because D-SACKs acked all retransmitted data */
|
2013-05-29 08:20:14 -06:00
|
|
|
static bool tcp_try_undo_dsack(struct sock *sk)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
if (tp->undo_marker && !tp->undo_retrans) {
|
2017-11-03 17:38:48 -06:00
|
|
|
tp->rack.reo_wnd_persist = min(TCP_RACK_RECOVERY_THRESH,
|
|
|
|
tp->rack.reo_wnd_persist + 1);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
DBGUNDO(sk, "D-SACK");
|
2013-05-29 08:20:13 -06:00
|
|
|
tcp_undo_cwnd_reduction(sk, false);
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPDSACKUNDO);
|
2013-05-29 08:20:14 -06:00
|
|
|
return true;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2013-05-29 08:20:14 -06:00
|
|
|
return false;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2013-03-20 07:33:00 -06:00
|
|
|
/* Undo during loss recovery after partial ACK or using F-RTO. */
|
|
|
|
static bool tcp_try_undo_loss(struct sock *sk, bool frto_undo)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2013-03-20 07:33:00 -06:00
|
|
|
if (frto_undo || tcp_may_undo(tp)) {
|
2013-05-29 08:20:13 -06:00
|
|
|
tcp_undo_cwnd_reduction(sk, true);
|
2005-11-10 18:14:59 -07:00
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
DBGUNDO(sk, "partial loss");
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPLOSSUNDO);
|
2013-03-20 07:33:00 -06:00
|
|
|
if (frto_undo)
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk),
|
2016-04-27 17:44:39 -06:00
|
|
|
LINUX_MIB_TCPSPURIOUSRTOS);
|
2005-08-09 21:10:42 -06:00
|
|
|
inet_csk(sk)->icsk_retransmits = 0;
|
2017-12-07 14:41:34 -07:00
|
|
|
if (frto_undo || tcp_is_sack(tp)) {
|
2005-08-10 01:03:31 -06:00
|
|
|
tcp_set_ca_state(sk, TCP_CA_Open);
|
2017-12-07 14:41:34 -07:00
|
|
|
tp->is_sack_reneg = 0;
|
|
|
|
}
|
2012-05-16 17:15:34 -06:00
|
|
|
return true;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2015-07-01 15:11:15 -06:00
|
|
|
/* The cwnd reduction in CWR and Recovery uses the PRR algorithm in RFC 6937.
|
2012-09-02 11:38:03 -06:00
|
|
|
* It computes the number of packets to send (sndcnt) based on packets newly
|
|
|
|
* delivered:
|
|
|
|
* 1) If the packets in flight is larger than ssthresh, PRR spreads the
|
|
|
|
* cwnd reductions across a full RTT.
|
2015-07-01 15:11:15 -06:00
|
|
|
* 2) Otherwise PRR uses packet conservation to send as much as delivered.
|
|
|
|
* But when the retransmits are acked without further losses, PRR
|
|
|
|
* slow starts cwnd up to ssthresh to speed up the recovery.
|
2012-09-02 11:38:03 -06:00
|
|
|
*/
|
2014-07-14 08:58:32 -06:00
|
|
|
static void tcp_init_cwnd_reduction(struct sock *sk)
|
2012-09-02 11:38:04 -06:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
tp->high_seq = tp->snd_nxt;
|
2013-03-11 04:00:44 -06:00
|
|
|
tp->tlp_high_seq = 0;
|
2012-09-02 11:38:04 -06:00
|
|
|
tp->snd_cwnd_cnt = 0;
|
|
|
|
tp->prior_cwnd = tp->snd_cwnd;
|
|
|
|
tp->prr_delivered = 0;
|
|
|
|
tp->prr_out = 0;
|
2014-07-14 08:58:32 -06:00
|
|
|
tp->snd_ssthresh = inet_csk(sk)->icsk_ca_ops->ssthresh(sk);
|
2014-09-29 05:08:30 -06:00
|
|
|
tcp_ecn_queue_cwr(tp);
|
2012-09-02 11:38:04 -06:00
|
|
|
}
|
|
|
|
|
2017-01-12 23:11:33 -07:00
|
|
|
void tcp_cwnd_reduction(struct sock *sk, int newly_acked_sacked, int flag)
|
2012-09-02 11:38:03 -06:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int sndcnt = 0;
|
|
|
|
int delta = tp->snd_ssthresh - tcp_packets_in_flight(tp);
|
|
|
|
|
2016-01-06 13:42:38 -07:00
|
|
|
if (newly_acked_sacked <= 0 || WARN_ON_ONCE(!tp->prior_cwnd))
|
|
|
|
return;
|
|
|
|
|
2012-09-02 11:38:04 -06:00
|
|
|
tp->prr_delivered += newly_acked_sacked;
|
2015-07-01 15:11:15 -06:00
|
|
|
if (delta < 0) {
|
2012-09-02 11:38:03 -06:00
|
|
|
u64 dividend = (u64)tp->snd_ssthresh * tp->prr_delivered +
|
|
|
|
tp->prior_cwnd - 1;
|
|
|
|
sndcnt = div_u64(dividend, tp->prior_cwnd) - tp->prr_out;
|
2018-11-11 05:10:10 -07:00
|
|
|
} else if ((flag & (FLAG_RETRANS_DATA_ACKED | FLAG_LOST_RETRANS)) ==
|
|
|
|
FLAG_RETRANS_DATA_ACKED) {
|
2012-09-02 11:38:03 -06:00
|
|
|
sndcnt = min_t(int, delta,
|
|
|
|
max_t(int, tp->prr_delivered - tp->prr_out,
|
|
|
|
newly_acked_sacked) + 1);
|
2015-07-01 15:11:15 -06:00
|
|
|
} else {
|
|
|
|
sndcnt = min(delta, newly_acked_sacked);
|
2012-09-02 11:38:03 -06:00
|
|
|
}
|
2016-02-02 11:33:05 -07:00
|
|
|
/* Force a fast retransmit upon entering fast recovery */
|
|
|
|
sndcnt = max(sndcnt, (tp->prr_out ? 0 : 1));
|
2012-09-02 11:38:03 -06:00
|
|
|
tp->snd_cwnd = tcp_packets_in_flight(tp) + sndcnt;
|
|
|
|
}
|
|
|
|
|
2012-09-02 11:38:04 -06:00
|
|
|
static inline void tcp_end_cwnd_reduction(struct sock *sk)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-10 01:03:31 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2011-08-21 14:21:57 -06:00
|
|
|
|
2016-09-19 21:39:21 -06:00
|
|
|
if (inet_csk(sk)->icsk_ca_ops->cong_control)
|
|
|
|
return;
|
|
|
|
|
2012-09-02 11:38:04 -06:00
|
|
|
/* Reset cwnd to ssthresh in CWR or Recovery (unless it's undone) */
|
2017-08-01 14:22:32 -06:00
|
|
|
if (tp->snd_ssthresh < TCP_INFINITE_SSTHRESH &&
|
|
|
|
(inet_csk(sk)->icsk_ca_state == TCP_CA_CWR || tp->undo_marker)) {
|
2012-09-02 11:38:04 -06:00
|
|
|
tp->snd_cwnd = tp->snd_ssthresh;
|
2017-05-16 15:00:04 -06:00
|
|
|
tp->snd_cwnd_stamp = tcp_jiffies32;
|
2011-03-14 04:57:03 -06:00
|
|
|
}
|
2005-08-10 01:03:31 -06:00
|
|
|
tcp_ca_event(sk, CA_EVENT_COMPLETE_CWR);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2012-09-02 11:38:04 -06:00
|
|
|
/* Enter CWR state. Disable cwnd undo since congestion is proven with ECN */
|
2014-07-14 08:58:32 -06:00
|
|
|
void tcp_enter_cwr(struct sock *sk)
|
2012-09-02 11:38:02 -06:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
tp->prior_ssthresh = 0;
|
2012-09-02 11:38:04 -06:00
|
|
|
if (inet_csk(sk)->icsk_ca_state < TCP_CA_CWR) {
|
2012-09-02 11:38:02 -06:00
|
|
|
tp->undo_marker = 0;
|
2014-07-14 08:58:32 -06:00
|
|
|
tcp_init_cwnd_reduction(sk);
|
2012-09-02 11:38:02 -06:00
|
|
|
tcp_set_ca_state(sk, TCP_CA_CWR);
|
|
|
|
}
|
|
|
|
}
|
2015-06-10 11:08:16 -06:00
|
|
|
EXPORT_SYMBOL(tcp_enter_cwr);
|
2012-09-02 11:38:02 -06:00
|
|
|
|
2008-06-04 12:34:22 -06:00
|
|
|
static void tcp_try_keep_open(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int state = TCP_CA_Open;
|
|
|
|
|
2011-11-16 01:58:04 -07:00
|
|
|
if (tcp_left_out(tp) || tcp_any_retrans_done(sk))
|
2008-06-04 12:34:22 -06:00
|
|
|
state = TCP_CA_Disorder;
|
|
|
|
|
|
|
|
if (inet_csk(sk)->icsk_ca_state != state) {
|
|
|
|
tcp_set_ca_state(sk, state);
|
|
|
|
tp->high_seq = tp->snd_nxt;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-02-02 11:33:05 -07:00
|
|
|
static void tcp_try_to_open(struct sock *sk, int flag)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2007-08-09 05:45:17 -06:00
|
|
|
tcp_verify_left_out(tp);
|
|
|
|
|
2013-03-20 07:32:58 -06:00
|
|
|
if (!tcp_any_retrans_done(sk))
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->retrans_stamp = 0;
|
|
|
|
|
2007-12-31 15:57:14 -07:00
|
|
|
if (flag & FLAG_ECE)
|
2014-07-14 08:58:32 -06:00
|
|
|
tcp_enter_cwr(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2005-08-10 01:03:31 -06:00
|
|
|
if (inet_csk(sk)->icsk_ca_state != TCP_CA_CWR) {
|
2008-06-04 12:34:22 -06:00
|
|
|
tcp_try_keep_open(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2006-03-20 18:53:41 -07:00
|
|
|
static void tcp_mtup_probe_failed(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
|
|
|
|
icsk->icsk_mtup.search_high = icsk->icsk_mtup.probe_size - 1;
|
|
|
|
icsk->icsk_mtup.probe_size = 0;
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMTUPFAIL);
|
2006-03-20 18:53:41 -07:00
|
|
|
}
|
|
|
|
|
2009-03-14 08:23:04 -06:00
|
|
|
static void tcp_mtup_probe_success(struct sock *sk)
|
2006-03-20 18:53:41 -07:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
|
|
|
|
/* FIXME: breaks with very large cwnd */
|
|
|
|
tp->prior_ssthresh = tcp_current_ssthresh(sk);
|
|
|
|
tp->snd_cwnd = tp->snd_cwnd *
|
|
|
|
tcp_mss_to_mtu(sk, tp->mss_cache) /
|
|
|
|
icsk->icsk_mtup.probe_size;
|
|
|
|
tp->snd_cwnd_cnt = 0;
|
2017-05-16 15:00:04 -06:00
|
|
|
tp->snd_cwnd_stamp = tcp_jiffies32;
|
2010-10-06 22:18:02 -06:00
|
|
|
tp->snd_ssthresh = tcp_current_ssthresh(sk);
|
2006-03-20 18:53:41 -07:00
|
|
|
|
|
|
|
icsk->icsk_mtup.search_low = icsk->icsk_mtup.probe_size;
|
|
|
|
icsk->icsk_mtup.probe_size = 0;
|
|
|
|
tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMTUPSUCCESS);
|
2006-03-20 18:53:41 -07:00
|
|
|
}
|
|
|
|
|
2008-11-24 22:11:55 -07:00
|
|
|
/* Do a simple retransmit without using the backoff mechanisms in
|
|
|
|
* tcp_timer. This is used for path mtu discovery.
|
|
|
|
* The socket is already locked here.
|
|
|
|
*/
|
|
|
|
void tcp_simple_retransmit(struct sock *sk)
|
|
|
|
{
|
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *skb;
|
2009-03-14 08:23:05 -06:00
|
|
|
unsigned int mss = tcp_current_mss(sk);
|
2008-11-24 22:11:55 -07:00
|
|
|
|
2017-10-05 23:21:27 -06:00
|
|
|
skb_rbtree_walk(skb, &sk->tcp_rtx_queue) {
|
2008-12-05 23:41:26 -07:00
|
|
|
if (tcp_skb_seglen(skb) > mss &&
|
2008-11-24 22:11:55 -07:00
|
|
|
!(TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED)) {
|
|
|
|
if (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_RETRANS) {
|
|
|
|
TCP_SKB_CB(skb)->sacked &= ~TCPCB_SACKED_RETRANS;
|
|
|
|
tp->retrans_out -= tcp_skb_pcount(skb);
|
|
|
|
}
|
|
|
|
tcp_skb_mark_lost_uncond_verify(tp, skb);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
tcp_clear_retrans_hints_partial(tp);
|
|
|
|
|
2017-11-07 16:33:43 -07:00
|
|
|
if (!tp->lost_out)
|
2008-11-24 22:11:55 -07:00
|
|
|
return;
|
|
|
|
|
|
|
|
if (tcp_is_reno(tp))
|
|
|
|
tcp_limit_reno_sacked(tp);
|
|
|
|
|
|
|
|
tcp_verify_left_out(tp);
|
|
|
|
|
|
|
|
/* Don't muck with the congestion window here.
|
|
|
|
* Reason is that we do not increase amount of _data_
|
|
|
|
* in network, but units changed and effective
|
|
|
|
* cwnd/ssthresh really reduced now.
|
|
|
|
*/
|
|
|
|
if (icsk->icsk_ca_state != TCP_CA_Loss) {
|
|
|
|
tp->high_seq = tp->snd_nxt;
|
|
|
|
tp->snd_ssthresh = tcp_current_ssthresh(sk);
|
|
|
|
tp->prior_ssthresh = 0;
|
|
|
|
tp->undo_marker = 0;
|
|
|
|
tcp_set_ca_state(sk, TCP_CA_Loss);
|
|
|
|
}
|
|
|
|
tcp_xmit_retransmit_queue(sk);
|
|
|
|
}
|
2010-07-09 15:22:10 -06:00
|
|
|
EXPORT_SYMBOL(tcp_simple_retransmit);
|
2008-11-24 22:11:55 -07:00
|
|
|
|
2017-01-12 23:11:33 -07:00
|
|
|
void tcp_enter_recovery(struct sock *sk, bool ece_ack)
|
2012-05-02 07:30:02 -06:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int mib_idx;
|
|
|
|
|
|
|
|
if (tcp_is_reno(tp))
|
|
|
|
mib_idx = LINUX_MIB_TCPRENORECOVERY;
|
|
|
|
else
|
|
|
|
mib_idx = LINUX_MIB_TCPSACKRECOVERY;
|
|
|
|
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), mib_idx);
|
2012-05-02 07:30:02 -06:00
|
|
|
|
|
|
|
tp->prior_ssthresh = 0;
|
2014-08-22 15:15:22 -06:00
|
|
|
tcp_init_undo(tp);
|
2012-05-02 07:30:02 -06:00
|
|
|
|
2015-07-01 15:11:14 -06:00
|
|
|
if (!tcp_in_cwnd_reduction(sk)) {
|
2012-05-02 07:30:02 -06:00
|
|
|
if (!ece_ack)
|
|
|
|
tp->prior_ssthresh = tcp_current_ssthresh(sk);
|
2014-07-14 08:58:32 -06:00
|
|
|
tcp_init_cwnd_reduction(sk);
|
2012-05-02 07:30:02 -06:00
|
|
|
}
|
|
|
|
tcp_set_ca_state(sk, TCP_CA_Recovery);
|
|
|
|
}
|
|
|
|
|
2013-03-20 07:32:59 -06:00
|
|
|
/* Process an ACK in CA_Loss state. Move to CA_Open if lost data are
|
|
|
|
* recovered or spurious. Otherwise retransmits more on partial ACKs.
|
|
|
|
*/
|
2018-11-27 15:42:01 -07:00
|
|
|
static void tcp_process_loss(struct sock *sk, int flag, int num_dupack,
|
2016-02-02 11:33:04 -07:00
|
|
|
int *rexmit)
|
2013-03-20 07:32:59 -06:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2013-03-20 07:33:00 -06:00
|
|
|
bool recovered = !before(tp->snd_una, tp->high_seq);
|
2013-03-20 07:32:59 -06:00
|
|
|
|
2015-05-18 13:31:44 -06:00
|
|
|
if ((flag & FLAG_SND_UNA_ADVANCED) &&
|
|
|
|
tcp_try_undo_loss(sk, false))
|
|
|
|
return;
|
|
|
|
|
2017-01-12 23:11:37 -07:00
|
|
|
if (tp->frto) { /* F-RTO RFC5682 sec 3.1 (sack enhanced version). */
|
2018-02-27 15:15:02 -07:00
|
|
|
/* Step 3.b. A timeout is spurious if not all data are
|
|
|
|
* lost, i.e., never-retransmitted data are (s)acked.
|
|
|
|
*/
|
|
|
|
if ((flag & FLAG_ORIG_SACK_ACKED) &&
|
|
|
|
tcp_try_undo_loss(sk, true))
|
|
|
|
return;
|
|
|
|
|
2015-05-18 13:31:45 -06:00
|
|
|
if (after(tp->snd_nxt, tp->high_seq)) {
|
2018-11-27 15:42:01 -07:00
|
|
|
if (flag & FLAG_DATA_SACKED || num_dupack)
|
2015-05-18 13:31:45 -06:00
|
|
|
tp->frto = 0; /* Step 3.a. loss was real */
|
2013-03-20 07:33:00 -06:00
|
|
|
} else if (flag & FLAG_SND_UNA_ADVANCED && !recovered) {
|
|
|
|
tp->high_seq = tp->snd_nxt;
|
2016-02-02 11:33:04 -07:00
|
|
|
/* Step 2.b. Try send new data (but deferred until cwnd
|
|
|
|
* is updated in tcp_ack()). Otherwise fall back to
|
|
|
|
* the conventional recovery.
|
|
|
|
*/
|
2017-10-05 23:21:27 -06:00
|
|
|
if (!tcp_write_queue_empty(sk) &&
|
2016-02-02 11:33:04 -07:00
|
|
|
after(tcp_wnd_end(tp), tp->snd_nxt)) {
|
|
|
|
*rexmit = REXMIT_NEW;
|
|
|
|
return;
|
|
|
|
}
|
2013-03-20 07:33:00 -06:00
|
|
|
tp->frto = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (recovered) {
|
|
|
|
/* F-RTO RFC5682 sec 3.1 step 2.a and 1st part of step 3.a */
|
2013-03-20 07:32:59 -06:00
|
|
|
tcp_try_undo_recovery(sk);
|
|
|
|
return;
|
|
|
|
}
|
2013-03-20 07:33:00 -06:00
|
|
|
if (tcp_is_reno(tp)) {
|
|
|
|
/* A Reno DUPACK means new data in F-RTO step 2.b above are
|
|
|
|
* delivered. Lower inflight to clock out (re)tranmissions.
|
|
|
|
*/
|
2018-11-27 15:42:01 -07:00
|
|
|
if (after(tp->snd_nxt, tp->high_seq) && num_dupack)
|
|
|
|
tcp_add_reno_sack(sk, num_dupack);
|
2013-03-20 07:33:00 -06:00
|
|
|
else if (flag & FLAG_SND_UNA_ADVANCED)
|
|
|
|
tcp_reset_reno_sack(tp);
|
|
|
|
}
|
2016-02-02 11:33:04 -07:00
|
|
|
*rexmit = REXMIT_LOST;
|
2013-03-20 07:32:59 -06:00
|
|
|
}
|
|
|
|
|
2013-05-29 08:20:12 -06:00
|
|
|
/* Undo during fast recovery after partial ACK. */
|
2017-11-08 14:01:27 -07:00
|
|
|
static bool tcp_try_undo_partial(struct sock *sk, u32 prior_snd_una)
|
2013-05-29 08:20:12 -06:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2013-05-29 08:20:13 -06:00
|
|
|
if (tp->undo_marker && tcp_packet_delayed(tp)) {
|
2013-05-29 08:20:12 -06:00
|
|
|
/* Plain luck! Hole if filled with delayed
|
2017-11-08 14:01:27 -07:00
|
|
|
* packet, rather than with a retransmit. Check reordering.
|
2013-05-29 08:20:12 -06:00
|
|
|
*/
|
2017-11-08 14:01:27 -07:00
|
|
|
tcp_check_sack_reordering(sk, prior_snd_una, 1);
|
2013-05-29 08:20:13 -06:00
|
|
|
|
|
|
|
/* We are getting evidence that the reordering degree is higher
|
|
|
|
* than we realized. If there are no retransmits out then we
|
|
|
|
* can undo. Otherwise we clock out new packets but do not
|
|
|
|
* mark more packets lost or retransmit more.
|
|
|
|
*/
|
2016-02-02 11:33:05 -07:00
|
|
|
if (tp->retrans_out)
|
2013-05-29 08:20:13 -06:00
|
|
|
return true;
|
|
|
|
|
2013-05-29 08:20:12 -06:00
|
|
|
if (!tcp_any_retrans_done(sk))
|
|
|
|
tp->retrans_stamp = 0;
|
|
|
|
|
2013-05-29 08:20:13 -06:00
|
|
|
DBGUNDO(sk, "partial recovery");
|
|
|
|
tcp_undo_cwnd_reduction(sk, true);
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPPARTIALUNDO);
|
2013-05-29 08:20:13 -06:00
|
|
|
tcp_try_keep_open(sk);
|
|
|
|
return true;
|
2013-05-29 08:20:12 -06:00
|
|
|
}
|
2013-05-29 08:20:13 -06:00
|
|
|
return false;
|
2013-05-29 08:20:12 -06:00
|
|
|
}
|
|
|
|
|
2018-05-16 17:40:12 -06:00
|
|
|
static void tcp_identify_packet_loss(struct sock *sk, int *ack_flag)
|
2017-01-12 23:11:35 -07:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2018-05-16 17:40:12 -06:00
|
|
|
if (tcp_rtx_queue_empty(sk))
|
|
|
|
return;
|
|
|
|
|
|
|
|
if (unlikely(tcp_is_reno(tp))) {
|
|
|
|
tcp_newreno_mark_lost(sk, *ack_flag & FLAG_SND_UNA_ADVANCED);
|
|
|
|
} else if (tcp_is_rack(sk)) {
|
2017-01-12 23:11:35 -07:00
|
|
|
u32 prior_retrans = tp->retrans_out;
|
|
|
|
|
2017-04-25 11:15:34 -06:00
|
|
|
tcp_rack_mark_lost(sk);
|
2017-01-12 23:11:35 -07:00
|
|
|
if (prior_retrans > tp->retrans_out)
|
|
|
|
*ack_flag |= FLAG_LOST_RETRANS;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-11-08 14:01:27 -07:00
|
|
|
static bool tcp_force_fast_retransmit(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
return after(tcp_highest_sack_seq(tp),
|
|
|
|
tp->snd_una + tp->reordering * tp->mss_cache);
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Process an event, which can update packets-in-flight not trivially.
|
|
|
|
* Main goal of this function is to calculate new estimate for left_out,
|
|
|
|
* taking into account both packets sitting in receiver's buffer and
|
|
|
|
* packets lost by network.
|
|
|
|
*
|
2016-02-02 11:33:05 -07:00
|
|
|
* Besides that it updates the congestion state when packet loss or ECN
|
|
|
|
* is detected. But it does not reduce the cwnd, it is done by the
|
|
|
|
* congestion control later.
|
2005-04-16 16:20:36 -06:00
|
|
|
*
|
|
|
|
* It does _not_ decide what to send, it is made in function
|
|
|
|
* tcp_xmit_retransmit_queue().
|
|
|
|
*/
|
2017-11-08 14:01:27 -07:00
|
|
|
static void tcp_fastretrans_alert(struct sock *sk, const u32 prior_snd_una,
|
2018-11-27 15:42:01 -07:00
|
|
|
int num_dupack, int *ack_flag, int *rexmit)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-10 01:03:31 -06:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2016-02-02 11:33:05 -07:00
|
|
|
int fast_rexmit = 0, flag = *ack_flag;
|
2018-11-27 15:42:01 -07:00
|
|
|
bool do_lost = num_dupack || ((flag & FLAG_DATA_SACKED) &&
|
|
|
|
tcp_force_fast_retransmit(sk));
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-10-05 23:21:25 -06:00
|
|
|
if (!tp->packets_out && tp->sacked_out)
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->sacked_out = 0;
|
|
|
|
|
2007-02-09 07:24:47 -07:00
|
|
|
/* Now state machine starts.
|
2005-04-16 16:20:36 -06:00
|
|
|
* A. ECE, hence prohibit cwnd undoing, the reduction is required. */
|
2007-12-31 15:57:14 -07:00
|
|
|
if (flag & FLAG_ECE)
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->prior_ssthresh = 0;
|
|
|
|
|
|
|
|
/* B. In all the states check for reneging SACKs. */
|
2007-12-31 05:49:21 -07:00
|
|
|
if (tcp_check_sack_reneging(sk, flag))
|
2005-04-16 16:20:36 -06:00
|
|
|
return;
|
|
|
|
|
2012-01-19 07:42:21 -07:00
|
|
|
/* C. Check consistency of the current state. */
|
2007-08-09 05:44:16 -06:00
|
|
|
tcp_verify_left_out(tp);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2012-01-19 07:42:21 -07:00
|
|
|
/* D. Check state exit conditions. State can be terminated
|
2005-04-16 16:20:36 -06:00
|
|
|
* when high_seq is ACKed. */
|
2005-08-10 01:03:31 -06:00
|
|
|
if (icsk->icsk_ca_state == TCP_CA_Open) {
|
2008-07-25 22:43:18 -06:00
|
|
|
WARN_ON(tp->retrans_out != 0);
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->retrans_stamp = 0;
|
|
|
|
} else if (!before(tp->snd_una, tp->high_seq)) {
|
2005-08-10 01:03:31 -06:00
|
|
|
switch (icsk->icsk_ca_state) {
|
2005-04-16 16:20:36 -06:00
|
|
|
case TCP_CA_CWR:
|
|
|
|
/* CWR is to be held something *above* high_seq
|
|
|
|
* is ACKed for CWR bit to reach receiver. */
|
|
|
|
if (tp->snd_una != tp->high_seq) {
|
2012-09-02 11:38:04 -06:00
|
|
|
tcp_end_cwnd_reduction(sk);
|
2005-08-10 01:03:31 -06:00
|
|
|
tcp_set_ca_state(sk, TCP_CA_Open);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
|
|
|
|
case TCP_CA_Recovery:
|
2007-08-09 06:14:46 -06:00
|
|
|
if (tcp_is_reno(tp))
|
2005-04-16 16:20:36 -06:00
|
|
|
tcp_reset_reno_sack(tp);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
if (tcp_try_undo_recovery(sk))
|
2005-04-16 16:20:36 -06:00
|
|
|
return;
|
2012-09-02 11:38:04 -06:00
|
|
|
tcp_end_cwnd_reduction(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-01-19 07:42:21 -07:00
|
|
|
/* E. Process state. */
|
2005-08-10 01:03:31 -06:00
|
|
|
switch (icsk->icsk_ca_state) {
|
2005-04-16 16:20:36 -06:00
|
|
|
case TCP_CA_Recovery:
|
2007-08-02 20:46:58 -06:00
|
|
|
if (!(flag & FLAG_SND_UNA_ADVANCED)) {
|
2018-11-27 15:42:01 -07:00
|
|
|
if (tcp_is_reno(tp))
|
|
|
|
tcp_add_reno_sack(sk, num_dupack);
|
2013-05-29 08:20:13 -06:00
|
|
|
} else {
|
2017-11-08 14:01:27 -07:00
|
|
|
if (tcp_try_undo_partial(sk, prior_snd_una))
|
2013-05-29 08:20:13 -06:00
|
|
|
return;
|
|
|
|
/* Partial ACK arrived. Force fast retransmit. */
|
|
|
|
do_lost = tcp_is_reno(tp) ||
|
2017-11-08 14:01:27 -07:00
|
|
|
tcp_force_fast_retransmit(sk);
|
2013-05-29 08:20:13 -06:00
|
|
|
}
|
2013-05-29 08:20:14 -06:00
|
|
|
if (tcp_try_undo_dsack(sk)) {
|
|
|
|
tcp_try_keep_open(sk);
|
|
|
|
return;
|
|
|
|
}
|
2018-05-16 17:40:12 -06:00
|
|
|
tcp_identify_packet_loss(sk, ack_flag);
|
2005-04-16 16:20:36 -06:00
|
|
|
break;
|
|
|
|
case TCP_CA_Loss:
|
2018-11-27 15:42:01 -07:00
|
|
|
tcp_process_loss(sk, flag, num_dupack, rexmit);
|
2018-05-16 17:40:12 -06:00
|
|
|
tcp_identify_packet_loss(sk, ack_flag);
|
2017-01-12 23:11:35 -07:00
|
|
|
if (!(icsk->icsk_ca_state == TCP_CA_Open ||
|
|
|
|
(*ack_flag & FLAG_LOST_RETRANS)))
|
2005-04-16 16:20:36 -06:00
|
|
|
return;
|
2015-07-01 15:11:14 -06:00
|
|
|
/* Change state if cwnd is undone or retransmits are lost */
|
2017-10-16 14:48:55 -06:00
|
|
|
/* fall through */
|
2005-04-16 16:20:36 -06:00
|
|
|
default:
|
2007-08-09 06:14:46 -06:00
|
|
|
if (tcp_is_reno(tp)) {
|
2007-08-02 20:46:58 -06:00
|
|
|
if (flag & FLAG_SND_UNA_ADVANCED)
|
2005-04-16 16:20:36 -06:00
|
|
|
tcp_reset_reno_sack(tp);
|
2018-11-27 15:42:01 -07:00
|
|
|
tcp_add_reno_sack(sk, num_dupack);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2011-11-16 01:58:04 -07:00
|
|
|
if (icsk->icsk_ca_state <= TCP_CA_Disorder)
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
tcp_try_undo_dsack(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2018-05-16 17:40:12 -06:00
|
|
|
tcp_identify_packet_loss(sk, ack_flag);
|
2012-05-02 07:30:04 -06:00
|
|
|
if (!tcp_time_to_recover(sk, flag)) {
|
2016-02-02 11:33:05 -07:00
|
|
|
tcp_try_to_open(sk, flag);
|
2005-04-16 16:20:36 -06:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2006-03-20 18:53:41 -07:00
|
|
|
/* MTU probe failure: don't reduce cwnd */
|
|
|
|
if (icsk->icsk_ca_state < TCP_CA_CWR &&
|
|
|
|
icsk->icsk_mtup.probe_size &&
|
2006-03-20 22:32:58 -07:00
|
|
|
tp->snd_una == tp->mtu_probe.probe_seq_start) {
|
2006-03-20 18:53:41 -07:00
|
|
|
tcp_mtup_probe_failed(sk);
|
|
|
|
/* Restores the reduction we did in tcp_mtup_probe() */
|
|
|
|
tp->snd_cwnd++;
|
|
|
|
tcp_simple_retransmit(sk);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Otherwise enter Recovery state */
|
2012-05-02 07:30:02 -06:00
|
|
|
tcp_enter_recovery(sk, (flag & FLAG_ECE));
|
2007-11-15 20:39:31 -07:00
|
|
|
fast_rexmit = 1;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2018-05-16 17:40:11 -06:00
|
|
|
if (!tcp_is_rack(sk) && do_lost)
|
2007-11-15 20:39:31 -07:00
|
|
|
tcp_update_scoreboard(sk, fast_rexmit);
|
2016-02-02 11:33:04 -07:00
|
|
|
*rexmit = REXMIT_LOST;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2018-01-17 13:11:00 -07:00
|
|
|
static void tcp_update_rtt_min(struct sock *sk, u32 rtt_us, const int flag)
|
2015-10-16 22:57:42 -06:00
|
|
|
{
|
2017-10-27 08:47:28 -06:00
|
|
|
u32 wlen = sock_net(sk)->ipv4.sysctl_tcp_min_rtt_wlen * HZ;
|
2016-09-19 21:39:10 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2018-01-17 13:11:00 -07:00
|
|
|
if ((flag & FLAG_ACK_MAYBE_DELAYED) && rtt_us > tcp_min_rtt(tp)) {
|
|
|
|
/* If the remote keeps returning delayed ACKs, eventually
|
|
|
|
* the min filter would pick it up and overestimate the
|
|
|
|
* prop. delay when it expires. Skip suspected delayed ACKs.
|
|
|
|
*/
|
|
|
|
return;
|
|
|
|
}
|
2017-05-16 15:00:13 -06:00
|
|
|
minmax_running_min(&tp->rtt_min, wlen, tcp_jiffies32,
|
2016-09-19 21:39:10 -06:00
|
|
|
rtt_us ? : jiffies_to_usecs(1));
|
2015-10-16 22:57:42 -06:00
|
|
|
}
|
|
|
|
|
2017-05-31 12:30:53 -06:00
|
|
|
static bool tcp_ack_update_rtt(struct sock *sk, const int flag,
|
|
|
|
long seq_rtt_us, long sack_rtt_us,
|
|
|
|
long ca_rtt_us, struct rate_sample *rs)
|
2008-12-05 23:43:26 -07:00
|
|
|
{
|
2013-07-22 17:20:46 -06:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
/* Prefer RTT measured from ACK's timing to TS-ECR. This is because
|
|
|
|
* broken middle-boxes or peers may corrupt TS-ECR fields. But
|
|
|
|
* Karn's algorithm forbids taking RTT if some retransmitted data
|
|
|
|
* is acked (RFC6298).
|
|
|
|
*/
|
2014-02-26 15:02:48 -07:00
|
|
|
if (seq_rtt_us < 0)
|
|
|
|
seq_rtt_us = sack_rtt_us;
|
2013-07-22 17:20:48 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* RTTM Rule: A TSecr value received in a segment is used to
|
|
|
|
* update the averaged RTT measurement only if the segment
|
|
|
|
* acknowledges some new data, i.e., only if it advances the
|
|
|
|
* left edge of the send window.
|
|
|
|
* See draft-ietf-tcplw-high-performance-00, section 3.3.
|
|
|
|
*/
|
2014-02-26 15:02:48 -07:00
|
|
|
if (seq_rtt_us < 0 && tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr &&
|
2017-05-16 15:00:14 -06:00
|
|
|
flag & FLAG_ACKED) {
|
|
|
|
u32 delta = tcp_time_stamp(tp) - tp->rx_opt.rcv_tsecr;
|
|
|
|
|
2018-11-24 10:12:24 -07:00
|
|
|
if (likely(delta < INT_MAX / (USEC_PER_SEC / TCP_TS_HZ))) {
|
|
|
|
seq_rtt_us = delta * (USEC_PER_SEC / TCP_TS_HZ);
|
|
|
|
ca_rtt_us = seq_rtt_us;
|
|
|
|
}
|
2017-05-16 15:00:14 -06:00
|
|
|
}
|
2017-05-31 12:30:53 -06:00
|
|
|
rs->rtt_us = ca_rtt_us; /* RTT of last (S)ACKed packet (or -1) */
|
2014-02-26 15:02:48 -07:00
|
|
|
if (seq_rtt_us < 0)
|
2013-07-22 17:20:48 -06:00
|
|
|
return false;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2015-10-16 22:57:42 -06:00
|
|
|
/* ca_rtt_us >= 0 is counting on the invariant that ca_rtt_us is
|
|
|
|
* always taken together with ACK, SACK, or TS-opts. Any negative
|
|
|
|
* values will be skipped with the seq_rtt_us < 0 check above.
|
|
|
|
*/
|
2018-01-17 13:11:00 -07:00
|
|
|
tcp_update_rtt_min(sk, ca_rtt_us, flag);
|
2014-02-26 15:02:48 -07:00
|
|
|
tcp_rtt_estimator(sk, seq_rtt_us);
|
2013-07-22 17:20:46 -06:00
|
|
|
tcp_set_rto(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2013-07-22 17:20:46 -06:00
|
|
|
/* RFC6298: only reset backoff on valid RTT measurement. */
|
|
|
|
inet_csk(sk)->icsk_backoff = 0;
|
2013-07-22 17:20:48 -06:00
|
|
|
return true;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2013-07-22 17:20:45 -06:00
|
|
|
/* Compute time elapsed between (last) SYNACK and the ACK completing 3WHS. */
|
2015-09-18 12:36:14 -06:00
|
|
|
void tcp_synack_rtt_meas(struct sock *sk, struct request_sock *req)
|
2013-07-22 17:20:45 -06:00
|
|
|
{
|
2017-05-31 12:30:53 -06:00
|
|
|
struct rate_sample rs;
|
2015-09-18 12:36:14 -06:00
|
|
|
long rtt_us = -1L;
|
2013-07-22 17:20:45 -06:00
|
|
|
|
2017-05-16 15:00:14 -06:00
|
|
|
if (req && !req->num_retrans && tcp_rsk(req)->snt_synack)
|
|
|
|
rtt_us = tcp_stamp_us_delta(tcp_clock_us(), tcp_rsk(req)->snt_synack);
|
2015-09-18 12:36:14 -06:00
|
|
|
|
2017-05-31 12:30:53 -06:00
|
|
|
tcp_ack_update_rtt(sk, FLAG_SYN_ACKED, rtt_us, -1L, rtt_us, &rs);
|
2013-07-22 17:20:45 -06:00
|
|
|
}
|
|
|
|
|
2015-09-18 12:36:14 -06:00
|
|
|
|
2014-05-02 22:18:05 -06:00
|
|
|
static void tcp_cong_avoid(struct sock *sk, u32 ack, u32 acked)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-10 01:03:31 -06:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2014-05-02 22:18:05 -06:00
|
|
|
|
|
|
|
icsk->icsk_ca_ops->cong_avoid(sk, ack, acked);
|
2017-05-16 15:00:04 -06:00
|
|
|
tcp_sk(sk)->snd_cwnd_stamp = tcp_jiffies32;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Restart timer after forward progress on connection.
|
|
|
|
* RFC2988 recommends to restart timer to now+rto.
|
|
|
|
*/
|
2012-05-02 07:30:04 -06:00
|
|
|
void tcp_rearm_rto(struct sock *sk)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
tcp: Tail loss probe (TLP)
This patch series implement the Tail loss probe (TLP) algorithm described
in http://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01. The
first patch implements the basic algorithm.
TLP's goal is to reduce tail latency of short transactions. It achieves
this by converting retransmission timeouts (RTOs) occuring due
to tail losses (losses at end of transactions) into fast recovery.
TLP transmits one packet in two round-trips when a connection is in
Open state and isn't receiving any ACKs. The transmitted packet, aka
loss probe, can be either new or a retransmission. When there is tail
loss, the ACK from a loss probe triggers FACK/early-retransmit based
fast recovery, thus avoiding a costly RTO. In the absence of loss,
there is no change in the connection state.
PTO stands for probe timeout. It is a timer event indicating
that an ACK is overdue and triggers a loss probe packet. The PTO value
is set to max(2*SRTT, 10ms) and is adjusted to account for delayed
ACK timer when there is only one oustanding packet.
TLP Algorithm
On transmission of new data in Open state:
-> packets_out > 1: schedule PTO in max(2*SRTT, 10ms).
-> packets_out == 1: schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
-> PTO = min(PTO, RTO)
Conditions for scheduling PTO:
-> Connection is in Open state.
-> Connection is either cwnd limited or no new data to send.
-> Number of probes per tail loss episode is limited to one.
-> Connection is SACK enabled.
When PTO fires:
new_segment_exists:
-> transmit new segment.
-> packets_out++. cwnd remains same.
no_new_packet:
-> retransmit the last segment.
Its ACK triggers FACK or early retransmit based recovery.
ACK path:
-> rearm RTO at start of ACK processing.
-> reschedule PTO if need be.
In addition, the patch includes a small variation to the Early Retransmit
(ER) algorithm, such that ER and TLP together can in principle recover any
N-degree of tail loss through fast recovery. TLP is controlled by the same
sysctl as ER, tcp_early_retrans sysctl.
tcp_early_retrans==0; disables TLP and ER.
==1; enables RFC5827 ER.
==2; delayed ER.
==3; TLP and delayed ER. [DEFAULT]
==4; TLP only.
The TLP patch series have been extensively tested on Google Web servers.
It is most effective for short Web trasactions, where it reduced RTOs by 15%
and improved HTTP response time (average by 6%, 99th percentile by 10%).
The transmitted probes account for <0.5% of the overall transmissions.
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-03-11 04:00:43 -06:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2012-05-02 07:30:04 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
|
2012-08-31 06:29:13 -06:00
|
|
|
/* If the retrans timer is currently being used by Fast Open
|
|
|
|
* for SYN-ACK retrans purpose, stay put.
|
|
|
|
*/
|
|
|
|
if (tp->fastopen_rsk)
|
|
|
|
return;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
if (!tp->packets_out) {
|
2005-08-09 21:10:42 -06:00
|
|
|
inet_csk_clear_xmit_timer(sk, ICSK_TIME_RETRANS);
|
2005-04-16 16:20:36 -06:00
|
|
|
} else {
|
2012-05-02 07:30:04 -06:00
|
|
|
u32 rto = inet_csk(sk)->icsk_rto;
|
|
|
|
/* Offset the time elapsed after installing regular RTO */
|
2017-01-12 23:11:39 -07:00
|
|
|
if (icsk->icsk_pending == ICSK_TIME_REO_TIMEOUT ||
|
tcp: Tail loss probe (TLP)
This patch series implement the Tail loss probe (TLP) algorithm described
in http://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01. The
first patch implements the basic algorithm.
TLP's goal is to reduce tail latency of short transactions. It achieves
this by converting retransmission timeouts (RTOs) occuring due
to tail losses (losses at end of transactions) into fast recovery.
TLP transmits one packet in two round-trips when a connection is in
Open state and isn't receiving any ACKs. The transmitted packet, aka
loss probe, can be either new or a retransmission. When there is tail
loss, the ACK from a loss probe triggers FACK/early-retransmit based
fast recovery, thus avoiding a costly RTO. In the absence of loss,
there is no change in the connection state.
PTO stands for probe timeout. It is a timer event indicating
that an ACK is overdue and triggers a loss probe packet. The PTO value
is set to max(2*SRTT, 10ms) and is adjusted to account for delayed
ACK timer when there is only one oustanding packet.
TLP Algorithm
On transmission of new data in Open state:
-> packets_out > 1: schedule PTO in max(2*SRTT, 10ms).
-> packets_out == 1: schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
-> PTO = min(PTO, RTO)
Conditions for scheduling PTO:
-> Connection is in Open state.
-> Connection is either cwnd limited or no new data to send.
-> Number of probes per tail loss episode is limited to one.
-> Connection is SACK enabled.
When PTO fires:
new_segment_exists:
-> transmit new segment.
-> packets_out++. cwnd remains same.
no_new_packet:
-> retransmit the last segment.
Its ACK triggers FACK or early retransmit based recovery.
ACK path:
-> rearm RTO at start of ACK processing.
-> reschedule PTO if need be.
In addition, the patch includes a small variation to the Early Retransmit
(ER) algorithm, such that ER and TLP together can in principle recover any
N-degree of tail loss through fast recovery. TLP is controlled by the same
sysctl as ER, tcp_early_retrans sysctl.
tcp_early_retrans==0; disables TLP and ER.
==1; enables RFC5827 ER.
==2; delayed ER.
==3; TLP and delayed ER. [DEFAULT]
==4; TLP only.
The TLP patch series have been extensively tested on Google Web servers.
It is most effective for short Web trasactions, where it reduced RTOs by 15%
and improved HTTP response time (average by 6%, 99th percentile by 10%).
The transmitted probes account for <0.5% of the overall transmissions.
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-03-11 04:00:43 -06:00
|
|
|
icsk->icsk_pending == ICSK_TIME_LOSS_PROBE) {
|
2017-08-03 07:19:52 -06:00
|
|
|
s64 delta_us = tcp_rto_delta_us(sk);
|
2017-05-18 10:15:58 -06:00
|
|
|
/* delta_us may not be positive if the socket is locked
|
tcp: Tail loss probe (TLP)
This patch series implement the Tail loss probe (TLP) algorithm described
in http://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01. The
first patch implements the basic algorithm.
TLP's goal is to reduce tail latency of short transactions. It achieves
this by converting retransmission timeouts (RTOs) occuring due
to tail losses (losses at end of transactions) into fast recovery.
TLP transmits one packet in two round-trips when a connection is in
Open state and isn't receiving any ACKs. The transmitted packet, aka
loss probe, can be either new or a retransmission. When there is tail
loss, the ACK from a loss probe triggers FACK/early-retransmit based
fast recovery, thus avoiding a costly RTO. In the absence of loss,
there is no change in the connection state.
PTO stands for probe timeout. It is a timer event indicating
that an ACK is overdue and triggers a loss probe packet. The PTO value
is set to max(2*SRTT, 10ms) and is adjusted to account for delayed
ACK timer when there is only one oustanding packet.
TLP Algorithm
On transmission of new data in Open state:
-> packets_out > 1: schedule PTO in max(2*SRTT, 10ms).
-> packets_out == 1: schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
-> PTO = min(PTO, RTO)
Conditions for scheduling PTO:
-> Connection is in Open state.
-> Connection is either cwnd limited or no new data to send.
-> Number of probes per tail loss episode is limited to one.
-> Connection is SACK enabled.
When PTO fires:
new_segment_exists:
-> transmit new segment.
-> packets_out++. cwnd remains same.
no_new_packet:
-> retransmit the last segment.
Its ACK triggers FACK or early retransmit based recovery.
ACK path:
-> rearm RTO at start of ACK processing.
-> reschedule PTO if need be.
In addition, the patch includes a small variation to the Early Retransmit
(ER) algorithm, such that ER and TLP together can in principle recover any
N-degree of tail loss through fast recovery. TLP is controlled by the same
sysctl as ER, tcp_early_retrans sysctl.
tcp_early_retrans==0; disables TLP and ER.
==1; enables RFC5827 ER.
==2; delayed ER.
==3; TLP and delayed ER. [DEFAULT]
==4; TLP only.
The TLP patch series have been extensively tested on Google Web servers.
It is most effective for short Web trasactions, where it reduced RTOs by 15%
and improved HTTP response time (average by 6%, 99th percentile by 10%).
The transmitted probes account for <0.5% of the overall transmissions.
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-03-11 04:00:43 -06:00
|
|
|
* when the retrans timer fires and is rescheduled.
|
2012-05-02 07:30:04 -06:00
|
|
|
*/
|
2017-08-16 15:53:36 -06:00
|
|
|
rto = usecs_to_jiffies(max_t(int, delta_us, 1));
|
2012-05-02 07:30:04 -06:00
|
|
|
}
|
2018-10-23 12:54:16 -06:00
|
|
|
tcp_reset_xmit_timer(sk, ICSK_TIME_RETRANS, rto,
|
|
|
|
TCP_RTO_MAX, tcp_rtx_queue_head(sk));
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2012-05-02 07:30:04 -06:00
|
|
|
}
|
|
|
|
|
tcp: fix xmit timer to only be reset if data ACKed/SACKed
Fix a TCP loss recovery performance bug raised recently on the netdev
list, in two threads:
(i) July 26, 2017: netdev thread "TCP fast retransmit issues"
(ii) July 26, 2017: netdev thread:
"[PATCH V2 net-next] TLP: Don't reschedule PTO when there's one
outstanding TLP retransmission"
The basic problem is that incoming TCP packets that did not indicate
forward progress could cause the xmit timer (TLP or RTO) to be rearmed
and pushed back in time. In certain corner cases this could result in
the following problems noted in these threads:
- Repeated ACKs coming in with bogus SACKs corrupted by middleboxes
could cause TCP to repeatedly schedule TLPs forever. We kept
sending TLPs after every ~200ms, which elicited bogus SACKs, which
caused more TLPs, ad infinitum; we never fired an RTO to fill in
the holes.
- Incoming data segments could, in some cases, cause us to reschedule
our RTO or TLP timer further out in time, for no good reason. This
could cause repeated inbound data to result in stalls in outbound
data, in the presence of packet loss.
This commit fixes these bugs by changing the TLP and RTO ACK
processing to:
(a) Only reschedule the xmit timer once per ACK.
(b) Only reschedule the xmit timer if tcp_clean_rtx_queue() deems the
ACK indicates sufficient forward progress (a packet was
cumulatively ACKed, or we got a SACK for a packet that was sent
before the most recent retransmit of the write queue head).
This brings us back into closer compliance with the RFCs, since, as
the comment for tcp_rearm_rto() notes, we should only restart the RTO
timer after forward progress on the connection. Previously we were
restarting the xmit timer even in these cases where there was no
forward progress.
As a side benefit, this commit simplifies and speeds up the TCP timer
arming logic. We had been calling inet_csk_reset_xmit_timer() three
times on normal ACKs that cumulatively acknowledged some data:
1) Once near the top of tcp_ack() to switch from TLP timer to RTO:
if (icsk->icsk_pending == ICSK_TIME_LOSS_PROBE)
tcp_rearm_rto(sk);
2) Once in tcp_clean_rtx_queue(), to update the RTO:
if (flag & FLAG_ACKED) {
tcp_rearm_rto(sk);
3) Once in tcp_ack() after tcp_fastretrans_alert() to switch from RTO
to TLP:
if (icsk->icsk_pending == ICSK_TIME_RETRANS)
tcp_schedule_loss_probe(sk);
This commit, by only rescheduling the xmit timer once per ACK,
simplifies the code and reduces CPU overhead.
This commit was tested in an A/B test with Google web server
traffic. SNMP stats and request latency metrics were within noise
levels, substantiating that for normal web traffic patterns this is a
rare issue. This commit was also tested with packetdrill tests to
verify that it fixes the timer behavior in the corner cases discussed
in the netdev threads mentioned above.
This patch is a bug fix patch intended to be queued for -stable
relases.
Fixes: 6ba8a3b19e76 ("tcp: Tail loss probe (TLP)")
Reported-by: Klavs Klavsen <kl@vsen.dk>
Reported-by: Mao Wenan <maowenan@huawei.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-03 07:19:54 -06:00
|
|
|
/* Try to schedule a loss probe; if that doesn't work, then schedule an RTO. */
|
|
|
|
static void tcp_set_xmit_timer(struct sock *sk)
|
|
|
|
{
|
tcp: when scheduling TLP, time of RTO should account for current ACK
Fix the TLP scheduling logic so that when scheduling a TLP probe, we
ensure that the estimated time at which an RTO would fire accounts for
the fact that ACKs indicating forward progress should push back RTO
times.
After the following fix:
df92c8394e6e ("tcp: fix xmit timer to only be reset if data ACKed/SACKed")
we had an unintentional behavior change in the following kind of
scenario: suppose the RTT variance has been very low recently. Then
suppose we send out a flight of N packets and our RTT is 100ms:
t=0: send a flight of N packets
t=100ms: receive an ACK for N-1 packets
The response before df92c8394e6e that was:
-> schedule a TLP for now + RTO_interval
The response after df92c8394e6e is:
-> schedule a TLP for t=0 + RTO_interval
Since RTO_interval = srtt + RTT_variance, this means that we have
scheduled a TLP timer at a point in the future that only accounts for
RTT_variance. If the RTT_variance term is small, this means that the
timer fires soon.
Before df92c8394e6e this would not happen, because in that code, when
we receive an ACK for a prefix of flight, we did:
1) Near the top of tcp_ack(), switch from TLP timer to RTO
at write_queue_head->paket_tx_time + RTO_interval:
if (icsk->icsk_pending == ICSK_TIME_LOSS_PROBE)
tcp_rearm_rto(sk);
2) In tcp_clean_rtx_queue(), update the RTO to now + RTO_interval:
if (flag & FLAG_ACKED) {
tcp_rearm_rto(sk);
3) In tcp_ack() after tcp_fastretrans_alert() switch from RTO
to TLP at now + RTO_interval:
if (icsk->icsk_pending == ICSK_TIME_RETRANS)
tcp_schedule_loss_probe(sk);
In df92c8394e6e we removed that 3-phase dance, and instead directly
set the TLP timer once: we set the TLP timer in cases like this to
write_queue_head->packet_tx_time + RTO_interval. So if the RTT
variance is small, then this means that this is setting the TLP timer
to fire quite soon. This means if the ACK for the tail of the flight
takes longer than an RTT to arrive (often due to delayed ACKs), then
the TLP timer fires too quickly.
Fixes: df92c8394e6e ("tcp: fix xmit timer to only be reset if data ACKed/SACKed")
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-11-17 19:06:14 -07:00
|
|
|
if (!tcp_schedule_loss_probe(sk, true))
|
tcp: fix xmit timer to only be reset if data ACKed/SACKed
Fix a TCP loss recovery performance bug raised recently on the netdev
list, in two threads:
(i) July 26, 2017: netdev thread "TCP fast retransmit issues"
(ii) July 26, 2017: netdev thread:
"[PATCH V2 net-next] TLP: Don't reschedule PTO when there's one
outstanding TLP retransmission"
The basic problem is that incoming TCP packets that did not indicate
forward progress could cause the xmit timer (TLP or RTO) to be rearmed
and pushed back in time. In certain corner cases this could result in
the following problems noted in these threads:
- Repeated ACKs coming in with bogus SACKs corrupted by middleboxes
could cause TCP to repeatedly schedule TLPs forever. We kept
sending TLPs after every ~200ms, which elicited bogus SACKs, which
caused more TLPs, ad infinitum; we never fired an RTO to fill in
the holes.
- Incoming data segments could, in some cases, cause us to reschedule
our RTO or TLP timer further out in time, for no good reason. This
could cause repeated inbound data to result in stalls in outbound
data, in the presence of packet loss.
This commit fixes these bugs by changing the TLP and RTO ACK
processing to:
(a) Only reschedule the xmit timer once per ACK.
(b) Only reschedule the xmit timer if tcp_clean_rtx_queue() deems the
ACK indicates sufficient forward progress (a packet was
cumulatively ACKed, or we got a SACK for a packet that was sent
before the most recent retransmit of the write queue head).
This brings us back into closer compliance with the RFCs, since, as
the comment for tcp_rearm_rto() notes, we should only restart the RTO
timer after forward progress on the connection. Previously we were
restarting the xmit timer even in these cases where there was no
forward progress.
As a side benefit, this commit simplifies and speeds up the TCP timer
arming logic. We had been calling inet_csk_reset_xmit_timer() three
times on normal ACKs that cumulatively acknowledged some data:
1) Once near the top of tcp_ack() to switch from TLP timer to RTO:
if (icsk->icsk_pending == ICSK_TIME_LOSS_PROBE)
tcp_rearm_rto(sk);
2) Once in tcp_clean_rtx_queue(), to update the RTO:
if (flag & FLAG_ACKED) {
tcp_rearm_rto(sk);
3) Once in tcp_ack() after tcp_fastretrans_alert() to switch from RTO
to TLP:
if (icsk->icsk_pending == ICSK_TIME_RETRANS)
tcp_schedule_loss_probe(sk);
This commit, by only rescheduling the xmit timer once per ACK,
simplifies the code and reduces CPU overhead.
This commit was tested in an A/B test with Google web server
traffic. SNMP stats and request latency metrics were within noise
levels, substantiating that for normal web traffic patterns this is a
rare issue. This commit was also tested with packetdrill tests to
verify that it fixes the timer behavior in the corner cases discussed
in the netdev threads mentioned above.
This patch is a bug fix patch intended to be queued for -stable
relases.
Fixes: 6ba8a3b19e76 ("tcp: Tail loss probe (TLP)")
Reported-by: Klavs Klavsen <kl@vsen.dk>
Reported-by: Mao Wenan <maowenan@huawei.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-03 07:19:54 -06:00
|
|
|
tcp_rearm_rto(sk);
|
|
|
|
}
|
|
|
|
|
2007-09-20 12:33:43 -06:00
|
|
|
/* If we get here, the whole TSO packet has not been acked. */
|
2007-10-09 02:28:45 -06:00
|
|
|
static u32 tcp_tso_acked(struct sock *sk, struct sk_buff *skb)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2007-09-20 12:33:43 -06:00
|
|
|
u32 packets_acked;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2007-09-20 12:33:43 -06:00
|
|
|
BUG_ON(!after(TCP_SKB_CB(skb)->end_seq, tp->snd_una));
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
packets_acked = tcp_skb_pcount(skb);
|
2007-09-20 12:33:43 -06:00
|
|
|
if (tcp_trim_head(sk, skb, tp->snd_una - TCP_SKB_CB(skb)->seq))
|
2005-04-16 16:20:36 -06:00
|
|
|
return 0;
|
|
|
|
packets_acked -= tcp_skb_pcount(skb);
|
|
|
|
|
|
|
|
if (packets_acked) {
|
|
|
|
BUG_ON(tcp_skb_pcount(skb) == 0);
|
2007-09-20 12:33:43 -06:00
|
|
|
BUG_ON(!before(TCP_SKB_CB(skb)->seq, TCP_SKB_CB(skb)->end_seq));
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2007-10-09 02:28:45 -06:00
|
|
|
return packets_acked;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2014-10-11 16:17:29 -06:00
|
|
|
static void tcp_ack_tstamp(struct sock *sk, struct sk_buff *skb,
|
|
|
|
u32 prior_snd_una)
|
|
|
|
{
|
|
|
|
const struct skb_shared_info *shinfo;
|
|
|
|
|
|
|
|
/* Avoid cache line misses to get skb_shinfo() and shinfo->tx_flags */
|
2016-04-02 21:08:08 -06:00
|
|
|
if (likely(!TCP_SKB_CB(skb)->txstamp_ack))
|
2014-10-11 16:17:29 -06:00
|
|
|
return;
|
|
|
|
|
|
|
|
shinfo = skb_shinfo(skb);
|
2016-04-27 21:39:01 -06:00
|
|
|
if (!before(shinfo->tskey, prior_snd_una) &&
|
2017-10-04 13:59:58 -06:00
|
|
|
before(shinfo->tskey, tcp_sk(sk)->snd_una)) {
|
|
|
|
tcp_skb_tsorted_save(skb) {
|
|
|
|
__skb_tstamp_tx(skb, NULL, sk, SCM_TSTAMP_ACK);
|
|
|
|
} tcp_skb_tsorted_restore(skb);
|
|
|
|
}
|
2014-10-11 16:17:29 -06:00
|
|
|
}
|
|
|
|
|
2007-09-20 12:33:43 -06:00
|
|
|
/* Remove acknowledged frames from the retransmission queue. If our packet
|
|
|
|
* is before the ack sequence we can discard it as it's confirmed to have
|
|
|
|
* arrived at the other end.
|
|
|
|
*/
|
2017-11-08 14:01:27 -07:00
|
|
|
static int tcp_clean_rtx_queue(struct sock *sk, u32 prior_fack,
|
|
|
|
u32 prior_snd_una,
|
2017-01-12 23:11:32 -07:00
|
|
|
struct tcp_sacktag_state *sack)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-11-10 17:56:12 -07:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2017-05-16 15:00:14 -06:00
|
|
|
u64 first_ackt, last_ackt;
|
2014-02-26 15:02:48 -07:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
u32 prior_sacked = tp->sacked_out;
|
2017-11-08 14:01:27 -07:00
|
|
|
u32 reord = tp->snd_nxt; /* lowest acked un-retx un-sacked seq */
|
2017-10-05 23:21:27 -06:00
|
|
|
struct sk_buff *skb, *next;
|
2013-10-02 06:19:51 -06:00
|
|
|
bool fully_acked = true;
|
2015-04-30 17:10:58 -06:00
|
|
|
long sack_rtt_us = -1L;
|
2014-02-26 15:02:48 -07:00
|
|
|
long seq_rtt_us = -1L;
|
2015-04-30 17:10:58 -06:00
|
|
|
long ca_rtt_us = -1L;
|
2007-12-30 05:37:55 -07:00
|
|
|
u32 pkts_acked = 0;
|
2016-06-08 22:16:44 -06:00
|
|
|
u32 last_in_flight = 0;
|
2013-10-24 09:59:27 -06:00
|
|
|
bool rtt_update;
|
2014-02-26 15:02:48 -07:00
|
|
|
int flag = 0;
|
|
|
|
|
2017-05-16 15:00:14 -06:00
|
|
|
first_ackt = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-10-05 23:21:27 -06:00
|
|
|
for (skb = skb_rb_first(&sk->tcp_rtx_queue); skb; skb = next) {
|
2007-02-09 07:24:47 -07:00
|
|
|
struct tcp_skb_cb *scb = TCP_SKB_CB(skb);
|
2017-11-08 14:01:27 -07:00
|
|
|
const u32 start_seq = scb->seq;
|
2007-09-20 12:33:43 -06:00
|
|
|
u8 sacked = scb->sacked;
|
2014-02-26 15:02:48 -07:00
|
|
|
u32 acked_pcount;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2014-10-11 16:17:29 -06:00
|
|
|
tcp_ack_tstamp(sk, skb, prior_snd_una);
|
2014-08-12 12:53:16 -06:00
|
|
|
|
[TCP]: use non-delayed ACK for congestion control RTT
When a delayed ACK representing two packets arrives, there are two RTT
samples available, one for each packet. The first (in order of seq
number) will be artificially long due to the delay waiting for the
second packet, the second will trigger the ACK and so will not itself
be delayed.
According to rfc1323, the SRTT used for RTO calculation should use the
first rtt, so receivers echo the timestamp from the first packet in
the delayed ack. For congestion control however, it seems measuring
delayed ack delay is not desirable as it varies independently of
congestion.
The patch below causes seq_rtt and last_ackt to be updated with any
available later packet rtts which should have less (and hopefully
zero) delack delay. The rtt value then gets passed to
ca_ops->pkts_acked().
Where TCP_CONG_RTT_STAMP was set, effort was made to supress RTTs from
within a TSO chunk (!fully_acked), using only the final ACK (which
includes any TSO delay) to generate RTTs. This patch removes these
checks so RTTs are passed for each ACK to ca_ops->pkts_acked().
For non-delay based congestion control (cubic, h-tcp), rtt is
sometimes used for rtt-scaling. In shortening the RTT, this may make
them a little less aggressive. Delay-based schemes (eg vegas, veno,
illinois) should get a cleaner, more accurate congestion signal,
particularly for small cwnds. The congestion control module can
potentially also filter out bad RTTs due to the delayed ack alarm by
looking at the associated cnt which (where delayed acking is in use)
should probably be 1 if the alarm went off or greater if the ACK was
triggered by a packet.
Signed-off-by: Gavin McCullagh <gavin.mccullagh@nuim.ie>
Acked-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-12-29 20:11:21 -07:00
|
|
|
/* Determine how many packets and what bytes were acked, tso and else */
|
2005-04-16 16:20:36 -06:00
|
|
|
if (after(scb->end_seq, tp->snd_una)) {
|
2007-10-09 02:28:45 -06:00
|
|
|
if (tcp_skb_pcount(skb) == 1 ||
|
|
|
|
!after(tp->snd_una, scb->seq))
|
|
|
|
break;
|
|
|
|
|
2007-12-30 05:37:55 -07:00
|
|
|
acked_pcount = tcp_tso_acked(sk, skb);
|
|
|
|
if (!acked_pcount)
|
2007-10-09 02:28:45 -06:00
|
|
|
break;
|
2012-05-16 17:15:34 -06:00
|
|
|
fully_acked = false;
|
2007-10-09 02:28:45 -06:00
|
|
|
} else {
|
2007-12-30 05:37:55 -07:00
|
|
|
acked_pcount = tcp_skb_pcount(skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2014-10-11 16:17:29 -06:00
|
|
|
if (unlikely(sacked & TCPCB_RETRANS)) {
|
2007-12-30 05:35:27 -07:00
|
|
|
if (sacked & TCPCB_SACKED_RETRANS)
|
2007-12-30 05:37:55 -07:00
|
|
|
tp->retrans_out -= acked_pcount;
|
2007-12-30 05:35:27 -07:00
|
|
|
flag |= FLAG_RETRANS_DATA_ACKED;
|
2015-04-10 18:17:49 -06:00
|
|
|
} else if (!(sacked & TCPCB_SACKED_ACKED)) {
|
2018-09-21 09:51:47 -06:00
|
|
|
last_ackt = tcp_skb_timestamp_us(skb);
|
2017-05-16 15:00:14 -06:00
|
|
|
WARN_ON_ONCE(last_ackt == 0);
|
|
|
|
if (!first_ackt)
|
2014-02-26 15:02:48 -07:00
|
|
|
first_ackt = last_ackt;
|
|
|
|
|
2016-06-08 22:16:44 -06:00
|
|
|
last_in_flight = TCP_SKB_CB(skb)->tx.in_flight;
|
2017-11-08 14:01:27 -07:00
|
|
|
if (before(start_seq, reord))
|
|
|
|
reord = start_seq;
|
2015-04-10 18:17:49 -06:00
|
|
|
if (!after(scb->end_seq, tp->high_seq))
|
|
|
|
flag |= FLAG_ORIG_SACK_ACKED;
|
2005-11-10 17:56:12 -07:00
|
|
|
}
|
2007-12-30 05:35:27 -07:00
|
|
|
|
2016-02-02 11:33:06 -07:00
|
|
|
if (sacked & TCPCB_SACKED_ACKED) {
|
2007-12-30 05:37:55 -07:00
|
|
|
tp->sacked_out -= acked_pcount;
|
2016-02-02 11:33:06 -07:00
|
|
|
} else if (tcp_is_sack(tp)) {
|
|
|
|
tp->delivered += acked_pcount;
|
|
|
|
if (!tcp_skb_spurious_retrans(tp, skb))
|
2017-01-12 23:11:34 -07:00
|
|
|
tcp_rack_advance(tp, sacked, scb->end_seq,
|
2018-09-21 09:51:47 -06:00
|
|
|
tcp_skb_timestamp_us(skb));
|
2016-02-02 11:33:06 -07:00
|
|
|
}
|
2007-12-30 05:35:27 -07:00
|
|
|
if (sacked & TCPCB_LOST)
|
2007-12-30 05:37:55 -07:00
|
|
|
tp->lost_out -= acked_pcount;
|
2007-12-30 05:35:27 -07:00
|
|
|
|
2007-12-30 05:37:55 -07:00
|
|
|
tp->packets_out -= acked_pcount;
|
|
|
|
pkts_acked += acked_pcount;
|
tcp: track data delivery rate for a TCP connection
This patch generates data delivery rate (throughput) samples on a
per-ACK basis. These rate samples can be used by congestion control
modules, and specifically will be used by TCP BBR in later patches in
this series.
Key state:
tp->delivered: Tracks the total number of data packets (original or not)
delivered so far. This is an already-existing field.
tp->delivered_mstamp: the last time tp->delivered was updated.
Algorithm:
A rate sample is calculated as (d1 - d0)/(t1 - t0) on a per-ACK basis:
d1: the current tp->delivered after processing the ACK
t1: the current time after processing the ACK
d0: the prior tp->delivered when the acked skb was transmitted
t0: the prior tp->delivered_mstamp when the acked skb was transmitted
When an skb is transmitted, we snapshot d0 and t0 in its control
block in tcp_rate_skb_sent().
When an ACK arrives, it may SACK and ACK some skbs. For each SACKed
or ACKed skb, tcp_rate_skb_delivered() updates the rate_sample struct
to reflect the latest (d0, t0).
Finally, tcp_rate_gen() generates a rate sample by storing
(d1 - d0) in rs->delivered and (t1 - t0) in rs->interval_us.
One caveat: if an skb was sent with no packets in flight, then
tp->delivered_mstamp may be either invalid (if the connection is
starting) or outdated (if the connection was idle). In that case,
we'll re-stamp tp->delivered_mstamp.
At first glance it seems t0 should always be the time when an skb was
transmitted, but actually this could over-estimate the rate due to
phase mismatch between transmit and ACK events. To track the delivery
rate, we ensure that if packets are in flight then t0 and and t1 are
times at which packets were marked delivered.
If the initial and final RTTs are different then one may be corrupted
by some sort of noise. The noise we see most often is sending gaps
caused by delayed, compressed, or stretched acks. This either affects
both RTTs equally or artificially reduces the final RTT. We approach
this by recording the info we need to compute the initial RTT
(duration of the "send phase" of the window) when we recorded the
associated inflight. Then, for a filter to avoid bandwidth
overestimates, we generalize the per-sample bandwidth computation
from:
bw = delivered / ack_phase_rtt
to the following:
bw = delivered / max(send_phase_rtt, ack_phase_rtt)
In large-scale experiments, this filtering approach incorporating
send_phase_rtt is effective at avoiding bandwidth overestimates due to
ACK compression or stretched ACKs.
Signed-off-by: Van Jacobson <vanj@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-19 21:39:14 -06:00
|
|
|
tcp_rate_skb_delivered(sk, skb, sack->rate);
|
2007-10-09 02:28:45 -06:00
|
|
|
|
2007-09-20 12:34:38 -06:00
|
|
|
/* Initial outgoing SYN's get put onto the write_queue
|
|
|
|
* just like anything else we transmit. It is not
|
|
|
|
* true data, and if we misinform our callers that
|
|
|
|
* this ACK acks real data, we will erroneously exit
|
|
|
|
* connection startup slow start one packet too
|
|
|
|
* quickly. This is severely frowned upon behavior.
|
|
|
|
*/
|
2014-10-11 16:17:29 -06:00
|
|
|
if (likely(!(scb->tcp_flags & TCPHDR_SYN))) {
|
2007-09-20 12:34:38 -06:00
|
|
|
flag |= FLAG_DATA_ACKED;
|
|
|
|
} else {
|
|
|
|
flag |= FLAG_SYN_ACKED;
|
|
|
|
tp->retrans_stamp = 0;
|
|
|
|
}
|
|
|
|
|
2007-10-09 02:28:45 -06:00
|
|
|
if (!fully_acked)
|
|
|
|
break;
|
|
|
|
|
2017-10-05 23:21:27 -06:00
|
|
|
next = skb_rb_next(skb);
|
2014-10-11 16:17:29 -06:00
|
|
|
if (unlikely(skb == tp->retransmit_skb_hint))
|
2008-09-20 22:25:15 -06:00
|
|
|
tp->retransmit_skb_hint = NULL;
|
2014-10-11 16:17:29 -06:00
|
|
|
if (unlikely(skb == tp->lost_skb_hint))
|
2008-09-20 22:25:52 -06:00
|
|
|
tp->lost_skb_hint = NULL;
|
2017-10-05 23:21:27 -06:00
|
|
|
tcp_rtx_queue_unlink_and_free(skb, sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2016-11-28 00:07:14 -07:00
|
|
|
if (!skb)
|
|
|
|
tcp_chrono_stop(sk, TCP_CHRONO_BUSY);
|
|
|
|
|
2008-10-07 15:43:06 -06:00
|
|
|
if (likely(between(tp->snd_up, prior_snd_una, tp->snd_una)))
|
|
|
|
tp->snd_up = tp->snd_una;
|
|
|
|
|
2007-12-31 05:49:21 -07:00
|
|
|
if (skb && (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED))
|
|
|
|
flag |= FLAG_SACK_RENEGING;
|
|
|
|
|
2017-05-16 15:00:14 -06:00
|
|
|
if (likely(first_ackt) && !(flag & FLAG_RETRANS_DATA_ACKED)) {
|
|
|
|
seq_rtt_us = tcp_stamp_us_delta(tp->tcp_mstamp, first_ackt);
|
|
|
|
ca_rtt_us = tcp_stamp_us_delta(tp->tcp_mstamp, last_ackt);
|
2018-01-17 13:11:00 -07:00
|
|
|
|
|
|
|
if (pkts_acked == 1 && last_in_flight < tp->mss_cache &&
|
|
|
|
last_in_flight && !prior_sacked && fully_acked &&
|
|
|
|
sack->rate->prior_delivered + 1 == tp->delivered &&
|
|
|
|
!(flag & (FLAG_CA_ALERT | FLAG_SYN_ACKED))) {
|
|
|
|
/* Conservatively mark a delayed ACK. It's typically
|
|
|
|
* from a lone runt packet over the round trip to
|
|
|
|
* a receiver w/o out-of-order or CE events.
|
|
|
|
*/
|
|
|
|
flag |= FLAG_ACK_MAYBE_DELAYED;
|
|
|
|
}
|
2015-04-30 17:10:58 -06:00
|
|
|
}
|
2017-05-16 15:00:14 -06:00
|
|
|
if (sack->first_sackt) {
|
|
|
|
sack_rtt_us = tcp_stamp_us_delta(tp->tcp_mstamp, sack->first_sackt);
|
|
|
|
ca_rtt_us = tcp_stamp_us_delta(tp->tcp_mstamp, sack->last_sackt);
|
2014-02-26 15:02:48 -07:00
|
|
|
}
|
2015-10-16 22:57:42 -06:00
|
|
|
rtt_update = tcp_ack_update_rtt(sk, flag, seq_rtt_us, sack_rtt_us,
|
2017-05-31 12:30:53 -06:00
|
|
|
ca_rtt_us, sack->rate);
|
2013-07-22 17:20:48 -06:00
|
|
|
|
2007-09-20 12:33:43 -06:00
|
|
|
if (flag & FLAG_ACKED) {
|
tcp: fix xmit timer to only be reset if data ACKed/SACKed
Fix a TCP loss recovery performance bug raised recently on the netdev
list, in two threads:
(i) July 26, 2017: netdev thread "TCP fast retransmit issues"
(ii) July 26, 2017: netdev thread:
"[PATCH V2 net-next] TLP: Don't reschedule PTO when there's one
outstanding TLP retransmission"
The basic problem is that incoming TCP packets that did not indicate
forward progress could cause the xmit timer (TLP or RTO) to be rearmed
and pushed back in time. In certain corner cases this could result in
the following problems noted in these threads:
- Repeated ACKs coming in with bogus SACKs corrupted by middleboxes
could cause TCP to repeatedly schedule TLPs forever. We kept
sending TLPs after every ~200ms, which elicited bogus SACKs, which
caused more TLPs, ad infinitum; we never fired an RTO to fill in
the holes.
- Incoming data segments could, in some cases, cause us to reschedule
our RTO or TLP timer further out in time, for no good reason. This
could cause repeated inbound data to result in stalls in outbound
data, in the presence of packet loss.
This commit fixes these bugs by changing the TLP and RTO ACK
processing to:
(a) Only reschedule the xmit timer once per ACK.
(b) Only reschedule the xmit timer if tcp_clean_rtx_queue() deems the
ACK indicates sufficient forward progress (a packet was
cumulatively ACKed, or we got a SACK for a packet that was sent
before the most recent retransmit of the write queue head).
This brings us back into closer compliance with the RFCs, since, as
the comment for tcp_rearm_rto() notes, we should only restart the RTO
timer after forward progress on the connection. Previously we were
restarting the xmit timer even in these cases where there was no
forward progress.
As a side benefit, this commit simplifies and speeds up the TCP timer
arming logic. We had been calling inet_csk_reset_xmit_timer() three
times on normal ACKs that cumulatively acknowledged some data:
1) Once near the top of tcp_ack() to switch from TLP timer to RTO:
if (icsk->icsk_pending == ICSK_TIME_LOSS_PROBE)
tcp_rearm_rto(sk);
2) Once in tcp_clean_rtx_queue(), to update the RTO:
if (flag & FLAG_ACKED) {
tcp_rearm_rto(sk);
3) Once in tcp_ack() after tcp_fastretrans_alert() to switch from RTO
to TLP:
if (icsk->icsk_pending == ICSK_TIME_RETRANS)
tcp_schedule_loss_probe(sk);
This commit, by only rescheduling the xmit timer once per ACK,
simplifies the code and reduces CPU overhead.
This commit was tested in an A/B test with Google web server
traffic. SNMP stats and request latency metrics were within noise
levels, substantiating that for normal web traffic patterns this is a
rare issue. This commit was also tested with packetdrill tests to
verify that it fixes the timer behavior in the corner cases discussed
in the netdev threads mentioned above.
This patch is a bug fix patch intended to be queued for -stable
relases.
Fixes: 6ba8a3b19e76 ("tcp: Tail loss probe (TLP)")
Reported-by: Klavs Klavsen <kl@vsen.dk>
Reported-by: Mao Wenan <maowenan@huawei.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-03 07:19:54 -06:00
|
|
|
flag |= FLAG_SET_XMIT_TIMER; /* set TLP or RTO timer */
|
2009-03-14 08:23:04 -06:00
|
|
|
if (unlikely(icsk->icsk_mtup.probe_size &&
|
|
|
|
!after(tp->mtu_probe.probe_seq_end, tp->snd_una))) {
|
|
|
|
tcp_mtup_probe_success(sk);
|
|
|
|
}
|
|
|
|
|
2007-11-10 22:22:18 -07:00
|
|
|
if (tcp_is_reno(tp)) {
|
|
|
|
tcp_remove_reno_sacks(sk, pkts_acked);
|
2018-06-29 04:07:53 -06:00
|
|
|
|
|
|
|
/* If any of the cumulatively ACKed segments was
|
|
|
|
* retransmitted, non-SACK case cannot confirm that
|
|
|
|
* progress was due to original transmission due to
|
|
|
|
* lack of TCPCB_SACKED_ACKED bits even if some of
|
|
|
|
* the packets may have been never retransmitted.
|
|
|
|
*/
|
|
|
|
if (flag & FLAG_RETRANS_DATA_ACKED)
|
|
|
|
flag &= ~FLAG_ORIG_SACK_ACKED;
|
2007-11-10 22:22:18 -07:00
|
|
|
} else {
|
2009-02-27 21:44:28 -07:00
|
|
|
int delta;
|
|
|
|
|
2007-11-10 22:22:18 -07:00
|
|
|
/* Non-retransmitted hole got filled? That's reordering */
|
2017-11-08 14:01:27 -07:00
|
|
|
if (before(reord, prior_fack))
|
|
|
|
tcp_check_sack_reordering(sk, reord, 0);
|
2008-09-20 22:25:52 -06:00
|
|
|
|
2017-11-08 14:01:26 -07:00
|
|
|
delta = prior_sacked - tp->sacked_out;
|
2009-02-27 21:44:28 -07:00
|
|
|
tp->lost_cnt_hint -= min(tp->lost_cnt_hint, delta);
|
2007-11-10 22:22:18 -07:00
|
|
|
}
|
2014-02-26 15:02:48 -07:00
|
|
|
} else if (skb && rtt_update && sack_rtt_us >= 0 &&
|
2018-09-21 09:51:47 -06:00
|
|
|
sack_rtt_us > tcp_stamp_us_delta(tp->tcp_mstamp,
|
|
|
|
tcp_skb_timestamp_us(skb))) {
|
2013-10-24 09:59:27 -06:00
|
|
|
/* Do not re-arm RTO if the sack RTT is measured from data sent
|
|
|
|
* after when the head was last (re)transmitted. Otherwise the
|
|
|
|
* timeout may continue to extend in loss recovery.
|
|
|
|
*/
|
tcp: fix xmit timer to only be reset if data ACKed/SACKed
Fix a TCP loss recovery performance bug raised recently on the netdev
list, in two threads:
(i) July 26, 2017: netdev thread "TCP fast retransmit issues"
(ii) July 26, 2017: netdev thread:
"[PATCH V2 net-next] TLP: Don't reschedule PTO when there's one
outstanding TLP retransmission"
The basic problem is that incoming TCP packets that did not indicate
forward progress could cause the xmit timer (TLP or RTO) to be rearmed
and pushed back in time. In certain corner cases this could result in
the following problems noted in these threads:
- Repeated ACKs coming in with bogus SACKs corrupted by middleboxes
could cause TCP to repeatedly schedule TLPs forever. We kept
sending TLPs after every ~200ms, which elicited bogus SACKs, which
caused more TLPs, ad infinitum; we never fired an RTO to fill in
the holes.
- Incoming data segments could, in some cases, cause us to reschedule
our RTO or TLP timer further out in time, for no good reason. This
could cause repeated inbound data to result in stalls in outbound
data, in the presence of packet loss.
This commit fixes these bugs by changing the TLP and RTO ACK
processing to:
(a) Only reschedule the xmit timer once per ACK.
(b) Only reschedule the xmit timer if tcp_clean_rtx_queue() deems the
ACK indicates sufficient forward progress (a packet was
cumulatively ACKed, or we got a SACK for a packet that was sent
before the most recent retransmit of the write queue head).
This brings us back into closer compliance with the RFCs, since, as
the comment for tcp_rearm_rto() notes, we should only restart the RTO
timer after forward progress on the connection. Previously we were
restarting the xmit timer even in these cases where there was no
forward progress.
As a side benefit, this commit simplifies and speeds up the TCP timer
arming logic. We had been calling inet_csk_reset_xmit_timer() three
times on normal ACKs that cumulatively acknowledged some data:
1) Once near the top of tcp_ack() to switch from TLP timer to RTO:
if (icsk->icsk_pending == ICSK_TIME_LOSS_PROBE)
tcp_rearm_rto(sk);
2) Once in tcp_clean_rtx_queue(), to update the RTO:
if (flag & FLAG_ACKED) {
tcp_rearm_rto(sk);
3) Once in tcp_ack() after tcp_fastretrans_alert() to switch from RTO
to TLP:
if (icsk->icsk_pending == ICSK_TIME_RETRANS)
tcp_schedule_loss_probe(sk);
This commit, by only rescheduling the xmit timer once per ACK,
simplifies the code and reduces CPU overhead.
This commit was tested in an A/B test with Google web server
traffic. SNMP stats and request latency metrics were within noise
levels, substantiating that for normal web traffic patterns this is a
rare issue. This commit was also tested with packetdrill tests to
verify that it fixes the timer behavior in the corner cases discussed
in the netdev threads mentioned above.
This patch is a bug fix patch intended to be queued for -stable
relases.
Fixes: 6ba8a3b19e76 ("tcp: Tail loss probe (TLP)")
Reported-by: Klavs Klavsen <kl@vsen.dk>
Reported-by: Mao Wenan <maowenan@huawei.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-03 07:19:54 -06:00
|
|
|
flag |= FLAG_SET_XMIT_TIMER; /* set TLP or RTO timer */
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2016-05-11 11:02:13 -06:00
|
|
|
if (icsk->icsk_ca_ops->pkts_acked) {
|
|
|
|
struct ack_sample sample = { .pkts_acked = pkts_acked,
|
2017-05-31 12:30:53 -06:00
|
|
|
.rtt_us = sack->rate->rtt_us,
|
2016-06-08 22:16:44 -06:00
|
|
|
.in_flight = last_in_flight };
|
2016-05-11 11:02:13 -06:00
|
|
|
|
|
|
|
icsk->icsk_ca_ops->pkts_acked(sk, &sample);
|
|
|
|
}
|
2015-04-30 17:10:59 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
#if FASTRETRANS_DEBUG > 0
|
2008-07-25 22:43:18 -06:00
|
|
|
WARN_ON((int)tp->sacked_out < 0);
|
|
|
|
WARN_ON((int)tp->lost_out < 0);
|
|
|
|
WARN_ON((int)tp->retrans_out < 0);
|
2007-08-09 06:14:46 -06:00
|
|
|
if (!tp->packets_out && tcp_is_sack(tp)) {
|
2007-10-09 02:59:42 -06:00
|
|
|
icsk = inet_csk(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
if (tp->lost_out) {
|
2012-05-15 08:11:54 -06:00
|
|
|
pr_debug("Leak l=%u %d\n",
|
|
|
|
tp->lost_out, icsk->icsk_ca_state);
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->lost_out = 0;
|
|
|
|
}
|
|
|
|
if (tp->sacked_out) {
|
2012-05-15 08:11:54 -06:00
|
|
|
pr_debug("Leak s=%u %d\n",
|
|
|
|
tp->sacked_out, icsk->icsk_ca_state);
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->sacked_out = 0;
|
|
|
|
}
|
|
|
|
if (tp->retrans_out) {
|
2012-05-15 08:11:54 -06:00
|
|
|
pr_debug("Leak r=%u %d\n",
|
|
|
|
tp->retrans_out, icsk->icsk_ca_state);
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->retrans_out = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
#endif
|
2007-09-20 12:33:43 -06:00
|
|
|
return flag;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
static void tcp_ack_probe(struct sock *sk)
|
|
|
|
{
|
2005-08-09 21:10:42 -06:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2017-10-05 23:21:27 -06:00
|
|
|
struct sk_buff *head = tcp_send_head(sk);
|
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Was it a usable window open? */
|
2017-10-05 23:21:27 -06:00
|
|
|
if (!head)
|
|
|
|
return;
|
|
|
|
if (!after(TCP_SKB_CB(head)->end_seq, tcp_wnd_end(tp))) {
|
2005-08-09 21:10:42 -06:00
|
|
|
icsk->icsk_backoff = 0;
|
|
|
|
inet_csk_clear_xmit_timer(sk, ICSK_TIME_PROBE0);
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Socket must be waked up by subsequent tcp_data_snd_check().
|
|
|
|
* This function is not for random using!
|
|
|
|
*/
|
|
|
|
} else {
|
tcp: adjust window probe timers to safer values
With the advent of small rto timers in datacenter TCP,
(ip route ... rto_min x), the following can happen :
1) Qdisc is full, transmit fails.
TCP sets a timer based on icsk_rto to retry the transmit, without
exponential backoff.
With low icsk_rto, and lot of sockets, all cpus are servicing timer
interrupts like crazy.
Intent of the code was to retry with a timer between 200 (TCP_RTO_MIN)
and 500ms (TCP_RESOURCE_PROBE_INTERVAL)
2) Receivers can send zero windows if they don't drain their receive queue.
TCP sends zero window probes, based on icsk_rto current value, with
exponential backoff.
With /proc/sys/net/ipv4/tcp_retries2 being 15 (or even smaller in
some cases), sender can abort in less than one or two minutes !
If receiver stops the sender, it obviously doesn't care of very tight
rto. Probability of dropping the ACK reopening the window is not
worth the risk.
Lets change the base timer to be at least 200ms (TCP_RTO_MIN) for these
events (but not normal RTO based retransmits)
A followup patch adds a new SNMP counter, as it would have helped a lot
diagnosing this issue.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-05-06 15:26:24 -06:00
|
|
|
unsigned long when = tcp_probe0_when(sk, TCP_RTO_MAX);
|
2014-09-22 14:19:44 -06:00
|
|
|
|
2018-10-23 12:54:16 -06:00
|
|
|
tcp_reset_xmit_timer(sk, ICSK_TIME_PROBE0,
|
|
|
|
when, TCP_RTO_MAX, NULL);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-07-19 15:32:18 -06:00
|
|
|
static inline bool tcp_ack_is_dubious(const struct sock *sk, const int flag)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2010-09-22 14:43:57 -06:00
|
|
|
return !(flag & FLAG_NOT_DUP) || (flag & FLAG_CA_ALERT) ||
|
|
|
|
inet_csk(sk)->icsk_ca_state != TCP_CA_Open;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2013-08-21 18:29:23 -06:00
|
|
|
/* Decide wheather to run the increase function of congestion control. */
|
2012-07-19 15:32:18 -06:00
|
|
|
static inline bool tcp_may_raise_cwnd(const struct sock *sk, const int flag)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2013-08-21 18:29:23 -06:00
|
|
|
/* If reordering is high then always grow cwnd whenever data is
|
|
|
|
* delivered regardless of its ordering. Otherwise stay conservative
|
2013-09-05 16:36:09 -06:00
|
|
|
* and only grow cwnd on in-order delivery (RFC5681). A stretched ACK w/
|
2013-08-21 18:29:23 -06:00
|
|
|
* new SACK or ECE mark may first advance cwnd here and later reduce
|
|
|
|
* cwnd in tcp_fastretrans_alert() based on more states.
|
|
|
|
*/
|
2016-02-03 00:46:52 -07:00
|
|
|
if (tcp_sk(sk)->reordering > sock_net(sk)->ipv4.sysctl_tcp_reordering)
|
2013-08-21 18:29:23 -06:00
|
|
|
return flag & FLAG_FORWARD_PROGRESS;
|
|
|
|
|
2013-09-05 16:36:09 -06:00
|
|
|
return flag & FLAG_DATA_ACKED;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2016-02-02 11:33:09 -07:00
|
|
|
/* The "ultimate" congestion control function that aims to replace the rigid
|
|
|
|
* cwnd increase and decrease control (tcp_cong_avoid,tcp_*cwnd_reduction).
|
|
|
|
* It's called toward the end of processing an ACK with precise rate
|
|
|
|
* information. All transmission or retransmission are delayed afterwards.
|
|
|
|
*/
|
|
|
|
static void tcp_cong_control(struct sock *sk, u32 ack, u32 acked_sacked,
|
2016-09-19 21:39:21 -06:00
|
|
|
int flag, const struct rate_sample *rs)
|
2016-02-02 11:33:09 -07:00
|
|
|
{
|
2016-09-19 21:39:21 -06:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
|
|
|
|
if (icsk->icsk_ca_ops->cong_control) {
|
|
|
|
icsk->icsk_ca_ops->cong_control(sk, rs);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2016-02-02 11:33:09 -07:00
|
|
|
if (tcp_in_cwnd_reduction(sk)) {
|
|
|
|
/* Reduce cwnd if state mandates */
|
|
|
|
tcp_cwnd_reduction(sk, acked_sacked, flag);
|
|
|
|
} else if (tcp_may_raise_cwnd(sk, flag)) {
|
|
|
|
/* Advance cwnd if state allows */
|
|
|
|
tcp_cong_avoid(sk, ack, acked_sacked);
|
|
|
|
}
|
|
|
|
tcp_update_pacing_rate(sk);
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Check that window update is acceptable.
|
|
|
|
* The function assumes that snd_una<=ack<=snd_next.
|
|
|
|
*/
|
2012-07-19 15:32:18 -06:00
|
|
|
static inline bool tcp_may_update_window(const struct tcp_sock *tp,
|
2007-12-31 15:57:14 -07:00
|
|
|
const u32 ack, const u32 ack_seq,
|
|
|
|
const u32 nwin)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2010-09-22 14:43:57 -06:00
|
|
|
return after(ack, tp->snd_una) ||
|
2005-04-16 16:20:36 -06:00
|
|
|
after(ack_seq, tp->snd_wl1) ||
|
2010-09-22 14:43:57 -06:00
|
|
|
(ack_seq == tp->snd_wl1 && nwin > tp->snd_wnd);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2015-04-28 16:28:17 -06:00
|
|
|
/* If we update tp->snd_una, also update tp->bytes_acked */
|
|
|
|
static void tcp_snd_una_update(struct tcp_sock *tp, u32 ack)
|
|
|
|
{
|
|
|
|
u32 delta = ack - tp->snd_una;
|
|
|
|
|
2016-05-03 17:56:03 -06:00
|
|
|
sock_owned_by_me((struct sock *)tp);
|
2015-04-28 16:28:17 -06:00
|
|
|
tp->bytes_acked += delta;
|
|
|
|
tp->snd_una = ack;
|
|
|
|
}
|
|
|
|
|
2015-04-28 16:28:18 -06:00
|
|
|
/* If we update tp->rcv_nxt, also update tp->bytes_received */
|
|
|
|
static void tcp_rcv_nxt_update(struct tcp_sock *tp, u32 seq)
|
|
|
|
{
|
|
|
|
u32 delta = seq - tp->rcv_nxt;
|
|
|
|
|
2016-05-03 17:56:03 -06:00
|
|
|
sock_owned_by_me((struct sock *)tp);
|
2015-04-28 16:28:18 -06:00
|
|
|
tp->bytes_received += delta;
|
|
|
|
tp->rcv_nxt = seq;
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Update our send window.
|
|
|
|
*
|
|
|
|
* Window update algorithm, described in RFC793/RFC1122 (used in linux-2.2
|
|
|
|
* and in FreeBSD. NetBSD's one is even worse.) is wrong.
|
|
|
|
*/
|
2011-10-21 03:22:42 -06:00
|
|
|
static int tcp_ack_update_window(struct sock *sk, const struct sk_buff *skb, u32 ack,
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
u32 ack_seq)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
int flag = 0;
|
2007-04-10 22:04:22 -06:00
|
|
|
u32 nwin = ntohs(tcp_hdr(skb)->window);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2007-04-10 22:04:22 -06:00
|
|
|
if (likely(!tcp_hdr(skb)->syn))
|
2005-04-16 16:20:36 -06:00
|
|
|
nwin <<= tp->rx_opt.snd_wscale;
|
|
|
|
|
|
|
|
if (tcp_may_update_window(tp, ack, ack_seq, nwin)) {
|
|
|
|
flag |= FLAG_WIN_UPDATE;
|
2009-03-02 23:42:02 -07:00
|
|
|
tcp_update_wl(tp, ack_seq);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
if (tp->snd_wnd != nwin) {
|
|
|
|
tp->snd_wnd = nwin;
|
|
|
|
|
2017-08-30 11:24:58 -06:00
|
|
|
/* Note, it is the only place, where
|
|
|
|
* fast path is recovered for sending TCP.
|
|
|
|
*/
|
|
|
|
tp->pred_flags = 0;
|
|
|
|
tcp_fast_path_check(sk);
|
|
|
|
|
2017-10-05 23:21:27 -06:00
|
|
|
if (!tcp_write_queue_empty(sk))
|
tcp: fix slow start after idle vs TSO/GSO
slow start after idle might reduce cwnd, but we perform this
after first packet was cooked and sent.
With TSO/GSO, it means that we might send a full TSO packet
even if cwnd should have been reduced to IW10.
Moving the SSAI check in skb_entail() makes sense, because
we slightly reduce number of times this check is done,
especially for large send() and TCP Small queue callbacks from
softirq context.
As Neal pointed out, we also need to perform the check
if/when receive window opens.
Tested:
Following packetdrill test demonstrates the problem
// Test of slow start after idle
`sysctl -q net.ipv4.tcp_slow_start_after_idle=1`
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < S 0:0(0) win 65535 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>
+.100 < . 1:1(0) ack 1 win 511
+0 accept(3, ..., ...) = 4
+0 setsockopt(4, SOL_SOCKET, SO_SNDBUF, [200000], 4) = 0
+0 write(4, ..., 26000) = 26000
+0 > . 1:5001(5000) ack 1
+0 > . 5001:10001(5000) ack 1
+0 %{ assert tcpi_snd_cwnd == 10 }%
+.100 < . 1:1(0) ack 10001 win 511
+0 %{ assert tcpi_snd_cwnd == 20, tcpi_snd_cwnd }%
+0 > . 10001:20001(10000) ack 1
+0 > P. 20001:26001(6000) ack 1
+.100 < . 1:1(0) ack 26001 win 511
+0 %{ assert tcpi_snd_cwnd == 36, tcpi_snd_cwnd }%
+4 write(4, ..., 20000) = 20000
// If slow start after idle works properly, we should send 5 MSS here (cwnd/2)
+0 > . 26001:31001(5000) ack 1
+0 %{ assert tcpi_snd_cwnd == 10, tcpi_snd_cwnd }%
+0 > . 31001:36001(5000) ack 1
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-08-21 13:30:00 -06:00
|
|
|
tcp_slow_start_after_idle_check(sk);
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
if (nwin > tp->max_window) {
|
|
|
|
tp->max_window = nwin;
|
2005-12-14 00:26:10 -07:00
|
|
|
tcp_sync_mss(sk, inet_csk(sk)->icsk_pmtu_cookie);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-04-28 16:28:17 -06:00
|
|
|
tcp_snd_una_update(tp, ack);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
return flag;
|
|
|
|
}
|
|
|
|
|
2016-07-14 09:38:40 -06:00
|
|
|
static bool __tcp_oow_rate_limited(struct net *net, int mib_idx,
|
|
|
|
u32 *last_oow_ack_time)
|
|
|
|
{
|
|
|
|
if (*last_oow_ack_time) {
|
2017-05-16 15:00:10 -06:00
|
|
|
s32 elapsed = (s32)(tcp_jiffies32 - *last_oow_ack_time);
|
2016-07-14 09:38:40 -06:00
|
|
|
|
2017-10-27 08:47:30 -06:00
|
|
|
if (0 <= elapsed && elapsed < net->ipv4.sysctl_tcp_invalid_ratelimit) {
|
2016-07-14 09:38:40 -06:00
|
|
|
NET_INC_STATS(net, mib_idx);
|
|
|
|
return true; /* rate-limited: don't send yet! */
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-05-16 15:00:10 -06:00
|
|
|
*last_oow_ack_time = tcp_jiffies32;
|
2016-07-14 09:38:40 -06:00
|
|
|
|
|
|
|
return false; /* not rate-limited: go ahead, send dupack now! */
|
|
|
|
}
|
|
|
|
|
2015-03-16 22:06:20 -06:00
|
|
|
/* Return true if we're currently rate-limiting out-of-window ACKs and
|
|
|
|
* thus shouldn't send a dupack right now. We rate-limit dupacks in
|
|
|
|
* response to out-of-window SYNs or ACKs to mitigate ACK loops or DoS
|
|
|
|
* attacks that send repeated SYNs or ACKs for the same connection. To
|
|
|
|
* do this, we do not send a duplicate SYNACK or ACK if the remote
|
|
|
|
* endpoint is sending out-of-window SYNs or pure ACKs at a high rate.
|
|
|
|
*/
|
|
|
|
bool tcp_oow_rate_limited(struct net *net, const struct sk_buff *skb,
|
|
|
|
int mib_idx, u32 *last_oow_ack_time)
|
|
|
|
{
|
|
|
|
/* Data packets without SYNs are not likely part of an ACK loop. */
|
|
|
|
if ((TCP_SKB_CB(skb)->seq != TCP_SKB_CB(skb)->end_seq) &&
|
|
|
|
!tcp_hdr(skb)->syn)
|
2016-07-14 09:38:40 -06:00
|
|
|
return false;
|
2015-03-16 22:06:20 -06:00
|
|
|
|
2016-07-14 09:38:40 -06:00
|
|
|
return __tcp_oow_rate_limited(net, mib_idx, last_oow_ack_time);
|
2015-03-16 22:06:20 -06:00
|
|
|
}
|
|
|
|
|
2012-10-21 13:57:11 -06:00
|
|
|
/* RFC 5961 7 [ACK Throttling] */
|
2015-02-06 14:04:40 -07:00
|
|
|
static void tcp_send_challenge_ack(struct sock *sk, const struct sk_buff *skb)
|
2012-10-21 13:57:11 -06:00
|
|
|
{
|
|
|
|
/* unprotected vars, we dont care of overwrites */
|
|
|
|
static u32 challenge_timestamp;
|
|
|
|
static unsigned int challenge_count;
|
2015-02-06 14:04:40 -07:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2017-10-27 08:47:26 -06:00
|
|
|
struct net *net = sock_net(sk);
|
2016-07-10 02:04:02 -06:00
|
|
|
u32 count, now;
|
2015-02-06 14:04:40 -07:00
|
|
|
|
|
|
|
/* First check our per-socket dupack rate limit. */
|
2017-10-27 08:47:26 -06:00
|
|
|
if (__tcp_oow_rate_limited(net,
|
2016-07-14 09:38:40 -06:00
|
|
|
LINUX_MIB_TCPACKSKIPPEDCHALLENGE,
|
|
|
|
&tp->last_oow_ack_time))
|
2015-02-06 14:04:40 -07:00
|
|
|
return;
|
2012-10-21 13:57:11 -06:00
|
|
|
|
2016-07-10 02:04:02 -06:00
|
|
|
/* Then check host-wide RFC 5961 rate limit. */
|
2015-02-06 14:04:40 -07:00
|
|
|
now = jiffies / HZ;
|
2012-10-21 13:57:11 -06:00
|
|
|
if (now != challenge_timestamp) {
|
2017-10-27 08:47:26 -06:00
|
|
|
u32 ack_limit = net->ipv4.sysctl_tcp_challenge_ack_limit;
|
|
|
|
u32 half = (ack_limit + 1) >> 1;
|
2016-07-10 02:04:02 -06:00
|
|
|
|
2012-10-21 13:57:11 -06:00
|
|
|
challenge_timestamp = now;
|
2017-10-27 08:47:26 -06:00
|
|
|
WRITE_ONCE(challenge_count, half + prandom_u32_max(ack_limit));
|
2012-10-21 13:57:11 -06:00
|
|
|
}
|
2016-07-10 02:04:02 -06:00
|
|
|
count = READ_ONCE(challenge_count);
|
|
|
|
if (count > 0) {
|
|
|
|
WRITE_ONCE(challenge_count, count - 1);
|
2017-10-27 08:47:26 -06:00
|
|
|
NET_INC_STATS(net, LINUX_MIB_TCPCHALLENGEACK);
|
2012-10-21 13:57:11 -06:00
|
|
|
tcp_send_ack(sk);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
tcp: call tcp_replace_ts_recent() from tcp_ack()
commit bd090dfc634d (tcp: tcp_replace_ts_recent() should not be called
from tcp_validate_incoming()) introduced a TS ecr bug in slow path
processing.
1 A > B P. 1:10001(10000) ack 1 <nop,nop,TS val 1001 ecr 200>
2 B < A . 1:1(0) ack 1 win 257 <sack 9001:10001,TS val 300 ecr 1001>
3 A > B . 1:1001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
4 A > B . 1001:2001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
(ecr 200 should be ecr 300 in packets 3 & 4)
Problem is tcp_ack() can trigger send of new packets (retransmits),
reflecting the prior TSval, instead of the TSval contained in the
currently processed incoming packet.
Fix this by calling tcp_replace_ts_recent() from tcp_ack() after the
checks, but before the actions.
Reported-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-04-19 01:19:48 -06:00
|
|
|
static void tcp_store_ts_recent(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
tp->rx_opt.ts_recent = tp->rx_opt.rcv_tsval;
|
2018-07-11 04:16:12 -06:00
|
|
|
tp->rx_opt.ts_recent_stamp = ktime_get_seconds();
|
tcp: call tcp_replace_ts_recent() from tcp_ack()
commit bd090dfc634d (tcp: tcp_replace_ts_recent() should not be called
from tcp_validate_incoming()) introduced a TS ecr bug in slow path
processing.
1 A > B P. 1:10001(10000) ack 1 <nop,nop,TS val 1001 ecr 200>
2 B < A . 1:1(0) ack 1 win 257 <sack 9001:10001,TS val 300 ecr 1001>
3 A > B . 1:1001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
4 A > B . 1001:2001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
(ecr 200 should be ecr 300 in packets 3 & 4)
Problem is tcp_ack() can trigger send of new packets (retransmits),
reflecting the prior TSval, instead of the TSval contained in the
currently processed incoming packet.
Fix this by calling tcp_replace_ts_recent() from tcp_ack() after the
checks, but before the actions.
Reported-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-04-19 01:19:48 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
static void tcp_replace_ts_recent(struct tcp_sock *tp, u32 seq)
|
|
|
|
{
|
|
|
|
if (tp->rx_opt.saw_tstamp && !after(seq, tp->rcv_wup)) {
|
|
|
|
/* PAWS bug workaround wrt. ACK frames, the PAWS discard
|
|
|
|
* extra check below makes sure this can only happen
|
|
|
|
* for pure ACK frames. -DaveM
|
|
|
|
*
|
|
|
|
* Not only, also it occurs for expired timestamps.
|
|
|
|
*/
|
|
|
|
|
|
|
|
if (tcp_paws_check(&tp->rx_opt, 0))
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2013-03-11 04:00:44 -06:00
|
|
|
/* This routine deals with acks during a TLP episode.
|
tcp: avoid reducing cwnd when ACK+DSACK is received
With TLP, the peer may reply to a probe with an
ACK+D-SACK, with ack value set to tlp_high_seq. In the current code,
such ACK+DSACK will be missed and only at next, higher ack will the TLP
episode be considered done. Since the DSACK is not present anymore,
this will cost a cwnd reduction.
This patch ensures that this scenario does not cause a cwnd reduction, since
receiving an ACK+DSACK indicates that both the initial segment and the probe
have been received by the peer.
The following packetdrill test, from Neal Cardwell, validates this patch:
// Establish a connection.
0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>
+.020 < . 1:1(0) ack 1 win 257
+0 accept(3, ..., ...) = 4
// Send 1 packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1:1001(1000) ack 1
// Loss probe retransmission.
// packets_out == 1 => schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
// In this case, this means: 1.5*RTT + 200ms = 230ms
+.230 > P. 1:1001(1000) ack 1
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Receiver ACKs at tlp_high_seq with a DSACK,
// indicating they received the original packet and probe.
+.020 < . 1:1(0) ack 1001 win 257 <sack 1:1001,nop,nop>
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Send another packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1001:2001(1000) ack 1
// Receiver ACKs above tlp_high_seq, which should end the TLP episode
// if we haven't already. We should not reduce cwnd.
+.020 < . 1:1(0) ack 2001 win 257
+0 %{ assert tcpi_snd_cwnd == 10, tcpi_snd_cwnd }%
Credits:
-Gregory helped in finding that tcp_process_tlp_ack was where the cwnd
got reduced in our MPTCP tests.
-Neal wrote the packetdrill test above
-Yuchung reworked the patch to make it more readable.
Cc: Gregory Detal <gregory.detal@uclouvain.be>
Cc: Nandita Dukkipati <nanditad@google.com>
Tested-by: Neal Cardwell <ncardwell@google.com>
Reviewed-by: Yuchung Cheng <ycheng@google.com>
Reviewed-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: Sébastien Barré <sebastien.barre@uclouvain.be>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-01-12 02:30:40 -07:00
|
|
|
* We mark the end of a TLP episode on receiving TLP dupack or when
|
|
|
|
* ack is after tlp_high_seq.
|
2013-03-11 04:00:44 -06:00
|
|
|
* Ref: loss detection algorithm in draft-dukkipati-tcpm-tcp-loss-probe.
|
|
|
|
*/
|
|
|
|
static void tcp_process_tlp_ack(struct sock *sk, u32 ack, int flag)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
tcp: avoid reducing cwnd when ACK+DSACK is received
With TLP, the peer may reply to a probe with an
ACK+D-SACK, with ack value set to tlp_high_seq. In the current code,
such ACK+DSACK will be missed and only at next, higher ack will the TLP
episode be considered done. Since the DSACK is not present anymore,
this will cost a cwnd reduction.
This patch ensures that this scenario does not cause a cwnd reduction, since
receiving an ACK+DSACK indicates that both the initial segment and the probe
have been received by the peer.
The following packetdrill test, from Neal Cardwell, validates this patch:
// Establish a connection.
0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>
+.020 < . 1:1(0) ack 1 win 257
+0 accept(3, ..., ...) = 4
// Send 1 packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1:1001(1000) ack 1
// Loss probe retransmission.
// packets_out == 1 => schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
// In this case, this means: 1.5*RTT + 200ms = 230ms
+.230 > P. 1:1001(1000) ack 1
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Receiver ACKs at tlp_high_seq with a DSACK,
// indicating they received the original packet and probe.
+.020 < . 1:1(0) ack 1001 win 257 <sack 1:1001,nop,nop>
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Send another packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1001:2001(1000) ack 1
// Receiver ACKs above tlp_high_seq, which should end the TLP episode
// if we haven't already. We should not reduce cwnd.
+.020 < . 1:1(0) ack 2001 win 257
+0 %{ assert tcpi_snd_cwnd == 10, tcpi_snd_cwnd }%
Credits:
-Gregory helped in finding that tcp_process_tlp_ack was where the cwnd
got reduced in our MPTCP tests.
-Neal wrote the packetdrill test above
-Yuchung reworked the patch to make it more readable.
Cc: Gregory Detal <gregory.detal@uclouvain.be>
Cc: Nandita Dukkipati <nanditad@google.com>
Tested-by: Neal Cardwell <ncardwell@google.com>
Reviewed-by: Yuchung Cheng <ycheng@google.com>
Reviewed-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: Sébastien Barré <sebastien.barre@uclouvain.be>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-01-12 02:30:40 -07:00
|
|
|
if (before(ack, tp->tlp_high_seq))
|
2013-03-11 04:00:44 -06:00
|
|
|
return;
|
|
|
|
|
tcp: avoid reducing cwnd when ACK+DSACK is received
With TLP, the peer may reply to a probe with an
ACK+D-SACK, with ack value set to tlp_high_seq. In the current code,
such ACK+DSACK will be missed and only at next, higher ack will the TLP
episode be considered done. Since the DSACK is not present anymore,
this will cost a cwnd reduction.
This patch ensures that this scenario does not cause a cwnd reduction, since
receiving an ACK+DSACK indicates that both the initial segment and the probe
have been received by the peer.
The following packetdrill test, from Neal Cardwell, validates this patch:
// Establish a connection.
0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>
+.020 < . 1:1(0) ack 1 win 257
+0 accept(3, ..., ...) = 4
// Send 1 packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1:1001(1000) ack 1
// Loss probe retransmission.
// packets_out == 1 => schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
// In this case, this means: 1.5*RTT + 200ms = 230ms
+.230 > P. 1:1001(1000) ack 1
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Receiver ACKs at tlp_high_seq with a DSACK,
// indicating they received the original packet and probe.
+.020 < . 1:1(0) ack 1001 win 257 <sack 1:1001,nop,nop>
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Send another packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1001:2001(1000) ack 1
// Receiver ACKs above tlp_high_seq, which should end the TLP episode
// if we haven't already. We should not reduce cwnd.
+.020 < . 1:1(0) ack 2001 win 257
+0 %{ assert tcpi_snd_cwnd == 10, tcpi_snd_cwnd }%
Credits:
-Gregory helped in finding that tcp_process_tlp_ack was where the cwnd
got reduced in our MPTCP tests.
-Neal wrote the packetdrill test above
-Yuchung reworked the patch to make it more readable.
Cc: Gregory Detal <gregory.detal@uclouvain.be>
Cc: Nandita Dukkipati <nanditad@google.com>
Tested-by: Neal Cardwell <ncardwell@google.com>
Reviewed-by: Yuchung Cheng <ycheng@google.com>
Reviewed-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: Sébastien Barré <sebastien.barre@uclouvain.be>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-01-12 02:30:40 -07:00
|
|
|
if (flag & FLAG_DSACKING_ACK) {
|
|
|
|
/* This DSACK means original and TLP probe arrived; no loss */
|
|
|
|
tp->tlp_high_seq = 0;
|
|
|
|
} else if (after(ack, tp->tlp_high_seq)) {
|
|
|
|
/* ACK advances: there was a loss, so reduce cwnd. Reset
|
|
|
|
* tlp_high_seq in tcp_init_cwnd_reduction()
|
|
|
|
*/
|
|
|
|
tcp_init_cwnd_reduction(sk);
|
|
|
|
tcp_set_ca_state(sk, TCP_CA_CWR);
|
|
|
|
tcp_end_cwnd_reduction(sk);
|
|
|
|
tcp_try_keep_open(sk);
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk),
|
2016-04-27 17:44:39 -06:00
|
|
|
LINUX_MIB_TCPLOSSPROBERECOVERY);
|
tcp: avoid reducing cwnd when ACK+DSACK is received
With TLP, the peer may reply to a probe with an
ACK+D-SACK, with ack value set to tlp_high_seq. In the current code,
such ACK+DSACK will be missed and only at next, higher ack will the TLP
episode be considered done. Since the DSACK is not present anymore,
this will cost a cwnd reduction.
This patch ensures that this scenario does not cause a cwnd reduction, since
receiving an ACK+DSACK indicates that both the initial segment and the probe
have been received by the peer.
The following packetdrill test, from Neal Cardwell, validates this patch:
// Establish a connection.
0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>
+.020 < . 1:1(0) ack 1 win 257
+0 accept(3, ..., ...) = 4
// Send 1 packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1:1001(1000) ack 1
// Loss probe retransmission.
// packets_out == 1 => schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
// In this case, this means: 1.5*RTT + 200ms = 230ms
+.230 > P. 1:1001(1000) ack 1
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Receiver ACKs at tlp_high_seq with a DSACK,
// indicating they received the original packet and probe.
+.020 < . 1:1(0) ack 1001 win 257 <sack 1:1001,nop,nop>
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Send another packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1001:2001(1000) ack 1
// Receiver ACKs above tlp_high_seq, which should end the TLP episode
// if we haven't already. We should not reduce cwnd.
+.020 < . 1:1(0) ack 2001 win 257
+0 %{ assert tcpi_snd_cwnd == 10, tcpi_snd_cwnd }%
Credits:
-Gregory helped in finding that tcp_process_tlp_ack was where the cwnd
got reduced in our MPTCP tests.
-Neal wrote the packetdrill test above
-Yuchung reworked the patch to make it more readable.
Cc: Gregory Detal <gregory.detal@uclouvain.be>
Cc: Nandita Dukkipati <nanditad@google.com>
Tested-by: Neal Cardwell <ncardwell@google.com>
Reviewed-by: Yuchung Cheng <ycheng@google.com>
Reviewed-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: Sébastien Barré <sebastien.barre@uclouvain.be>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-01-12 02:30:40 -07:00
|
|
|
} else if (!(flag & (FLAG_SND_UNA_ADVANCED |
|
|
|
|
FLAG_NOT_DUP | FLAG_DATA_SACKED))) {
|
|
|
|
/* Pure dupack: original and TLP probe arrived; no loss */
|
2013-03-11 04:00:44 -06:00
|
|
|
tp->tlp_high_seq = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2014-09-26 14:37:34 -06:00
|
|
|
static inline void tcp_in_ack_event(struct sock *sk, u32 flags)
|
|
|
|
{
|
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
|
|
|
|
if (icsk->icsk_ca_ops->in_ack_event)
|
|
|
|
icsk->icsk_ca_ops->in_ack_event(sk, flags);
|
|
|
|
}
|
|
|
|
|
2016-02-02 11:33:04 -07:00
|
|
|
/* Congestion control has updated the cwnd already. So if we're in
|
|
|
|
* loss recovery then now we do any new sends (for FRTO) or
|
|
|
|
* retransmits (for CA_Loss or CA_recovery) that make sense.
|
|
|
|
*/
|
|
|
|
static void tcp_xmit_recovery(struct sock *sk, int rexmit)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2019-04-29 16:46:13 -06:00
|
|
|
if (rexmit == REXMIT_NONE || sk->sk_state == TCP_SYN_SENT)
|
2016-02-02 11:33:04 -07:00
|
|
|
return;
|
|
|
|
|
|
|
|
if (unlikely(rexmit == 2)) {
|
|
|
|
__tcp_push_pending_frames(sk, tcp_current_mss(sk),
|
|
|
|
TCP_NAGLE_OFF);
|
|
|
|
if (after(tp->snd_nxt, tp->high_seq))
|
|
|
|
return;
|
|
|
|
tp->frto = 0;
|
|
|
|
}
|
|
|
|
tcp_xmit_retransmit_queue(sk);
|
|
|
|
}
|
|
|
|
|
2018-04-18 00:18:47 -06:00
|
|
|
/* Returns the number of packets newly acked or sacked by the current ACK */
|
|
|
|
static u32 tcp_newly_delivered(struct sock *sk, u32 prior_delivered, int flag)
|
|
|
|
{
|
2018-04-18 00:18:49 -06:00
|
|
|
const struct net *net = sock_net(sk);
|
2018-04-18 00:18:47 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
u32 delivered;
|
|
|
|
|
|
|
|
delivered = tp->delivered - prior_delivered;
|
2018-04-18 00:18:49 -06:00
|
|
|
NET_ADD_STATS(net, LINUX_MIB_TCPDELIVERED, delivered);
|
|
|
|
if (flag & FLAG_ECE) {
|
2018-04-18 00:18:48 -06:00
|
|
|
tp->delivered_ce += delivered;
|
2018-04-18 00:18:49 -06:00
|
|
|
NET_ADD_STATS(net, LINUX_MIB_TCPDELIVEREDCE, delivered);
|
|
|
|
}
|
2018-04-18 00:18:47 -06:00
|
|
|
return delivered;
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* This routine deals with incoming acks, but not outgoing ones. */
|
2011-10-21 03:22:42 -06:00
|
|
|
static int tcp_ack(struct sock *sk, const struct sk_buff *skb, int flag)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-10 01:03:31 -06:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2015-04-30 17:10:57 -06:00
|
|
|
struct tcp_sacktag_state sack_state;
|
tcp: track data delivery rate for a TCP connection
This patch generates data delivery rate (throughput) samples on a
per-ACK basis. These rate samples can be used by congestion control
modules, and specifically will be used by TCP BBR in later patches in
this series.
Key state:
tp->delivered: Tracks the total number of data packets (original or not)
delivered so far. This is an already-existing field.
tp->delivered_mstamp: the last time tp->delivered was updated.
Algorithm:
A rate sample is calculated as (d1 - d0)/(t1 - t0) on a per-ACK basis:
d1: the current tp->delivered after processing the ACK
t1: the current time after processing the ACK
d0: the prior tp->delivered when the acked skb was transmitted
t0: the prior tp->delivered_mstamp when the acked skb was transmitted
When an skb is transmitted, we snapshot d0 and t0 in its control
block in tcp_rate_skb_sent().
When an ACK arrives, it may SACK and ACK some skbs. For each SACKed
or ACKed skb, tcp_rate_skb_delivered() updates the rate_sample struct
to reflect the latest (d0, t0).
Finally, tcp_rate_gen() generates a rate sample by storing
(d1 - d0) in rs->delivered and (t1 - t0) in rs->interval_us.
One caveat: if an skb was sent with no packets in flight, then
tp->delivered_mstamp may be either invalid (if the connection is
starting) or outdated (if the connection was idle). In that case,
we'll re-stamp tp->delivered_mstamp.
At first glance it seems t0 should always be the time when an skb was
transmitted, but actually this could over-estimate the rate due to
phase mismatch between transmit and ACK events. To track the delivery
rate, we ensure that if packets are in flight then t0 and and t1 are
times at which packets were marked delivered.
If the initial and final RTTs are different then one may be corrupted
by some sort of noise. The noise we see most often is sending gaps
caused by delayed, compressed, or stretched acks. This either affects
both RTTs equally or artificially reduces the final RTT. We approach
this by recording the info we need to compute the initial RTT
(duration of the "send phase" of the window) when we recorded the
associated inflight. Then, for a filter to avoid bandwidth
overestimates, we generalize the per-sample bandwidth computation
from:
bw = delivered / ack_phase_rtt
to the following:
bw = delivered / max(send_phase_rtt, ack_phase_rtt)
In large-scale experiments, this filtering approach incorporating
send_phase_rtt is effective at avoiding bandwidth overestimates due to
ACK compression or stretched ACKs.
Signed-off-by: Van Jacobson <vanj@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-19 21:39:14 -06:00
|
|
|
struct rate_sample rs = { .prior_delivered = 0 };
|
2005-04-16 16:20:36 -06:00
|
|
|
u32 prior_snd_una = tp->snd_una;
|
2017-12-07 14:41:34 -07:00
|
|
|
bool is_sack_reneg = tp->is_sack_reneg;
|
2005-04-16 16:20:36 -06:00
|
|
|
u32 ack_seq = TCP_SKB_CB(skb)->seq;
|
|
|
|
u32 ack = TCP_SKB_CB(skb)->ack_seq;
|
2018-11-27 15:42:01 -07:00
|
|
|
int num_dupack = 0;
|
2013-05-21 09:12:07 -06:00
|
|
|
int prior_packets = tp->packets_out;
|
tcp: track data delivery rate for a TCP connection
This patch generates data delivery rate (throughput) samples on a
per-ACK basis. These rate samples can be used by congestion control
modules, and specifically will be used by TCP BBR in later patches in
this series.
Key state:
tp->delivered: Tracks the total number of data packets (original or not)
delivered so far. This is an already-existing field.
tp->delivered_mstamp: the last time tp->delivered was updated.
Algorithm:
A rate sample is calculated as (d1 - d0)/(t1 - t0) on a per-ACK basis:
d1: the current tp->delivered after processing the ACK
t1: the current time after processing the ACK
d0: the prior tp->delivered when the acked skb was transmitted
t0: the prior tp->delivered_mstamp when the acked skb was transmitted
When an skb is transmitted, we snapshot d0 and t0 in its control
block in tcp_rate_skb_sent().
When an ACK arrives, it may SACK and ACK some skbs. For each SACKed
or ACKed skb, tcp_rate_skb_delivered() updates the rate_sample struct
to reflect the latest (d0, t0).
Finally, tcp_rate_gen() generates a rate sample by storing
(d1 - d0) in rs->delivered and (t1 - t0) in rs->interval_us.
One caveat: if an skb was sent with no packets in flight, then
tp->delivered_mstamp may be either invalid (if the connection is
starting) or outdated (if the connection was idle). In that case,
we'll re-stamp tp->delivered_mstamp.
At first glance it seems t0 should always be the time when an skb was
transmitted, but actually this could over-estimate the rate due to
phase mismatch between transmit and ACK events. To track the delivery
rate, we ensure that if packets are in flight then t0 and and t1 are
times at which packets were marked delivered.
If the initial and final RTTs are different then one may be corrupted
by some sort of noise. The noise we see most often is sending gaps
caused by delayed, compressed, or stretched acks. This either affects
both RTTs equally or artificially reduces the final RTT. We approach
this by recording the info we need to compute the initial RTT
(duration of the "send phase" of the window) when we recorded the
associated inflight. Then, for a filter to avoid bandwidth
overestimates, we generalize the per-sample bandwidth computation
from:
bw = delivered / ack_phase_rtt
to the following:
bw = delivered / max(send_phase_rtt, ack_phase_rtt)
In large-scale experiments, this filtering approach incorporating
send_phase_rtt is effective at avoiding bandwidth overestimates due to
ACK compression or stretched ACKs.
Signed-off-by: Van Jacobson <vanj@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-19 21:39:14 -06:00
|
|
|
u32 delivered = tp->delivered;
|
|
|
|
u32 lost = tp->lost;
|
2016-02-02 11:33:04 -07:00
|
|
|
int rexmit = REXMIT_NONE; /* Flag to (re)transmit to recover losses */
|
2017-11-08 14:01:27 -07:00
|
|
|
u32 prior_fack;
|
2015-04-30 17:10:57 -06:00
|
|
|
|
2017-05-16 15:00:14 -06:00
|
|
|
sack_state.first_sackt = 0;
|
tcp: track data delivery rate for a TCP connection
This patch generates data delivery rate (throughput) samples on a
per-ACK basis. These rate samples can be used by congestion control
modules, and specifically will be used by TCP BBR in later patches in
this series.
Key state:
tp->delivered: Tracks the total number of data packets (original or not)
delivered so far. This is an already-existing field.
tp->delivered_mstamp: the last time tp->delivered was updated.
Algorithm:
A rate sample is calculated as (d1 - d0)/(t1 - t0) on a per-ACK basis:
d1: the current tp->delivered after processing the ACK
t1: the current time after processing the ACK
d0: the prior tp->delivered when the acked skb was transmitted
t0: the prior tp->delivered_mstamp when the acked skb was transmitted
When an skb is transmitted, we snapshot d0 and t0 in its control
block in tcp_rate_skb_sent().
When an ACK arrives, it may SACK and ACK some skbs. For each SACKed
or ACKed skb, tcp_rate_skb_delivered() updates the rate_sample struct
to reflect the latest (d0, t0).
Finally, tcp_rate_gen() generates a rate sample by storing
(d1 - d0) in rs->delivered and (t1 - t0) in rs->interval_us.
One caveat: if an skb was sent with no packets in flight, then
tp->delivered_mstamp may be either invalid (if the connection is
starting) or outdated (if the connection was idle). In that case,
we'll re-stamp tp->delivered_mstamp.
At first glance it seems t0 should always be the time when an skb was
transmitted, but actually this could over-estimate the rate due to
phase mismatch between transmit and ACK events. To track the delivery
rate, we ensure that if packets are in flight then t0 and and t1 are
times at which packets were marked delivered.
If the initial and final RTTs are different then one may be corrupted
by some sort of noise. The noise we see most often is sending gaps
caused by delayed, compressed, or stretched acks. This either affects
both RTTs equally or artificially reduces the final RTT. We approach
this by recording the info we need to compute the initial RTT
(duration of the "send phase" of the window) when we recorded the
associated inflight. Then, for a filter to avoid bandwidth
overestimates, we generalize the per-sample bandwidth computation
from:
bw = delivered / ack_phase_rtt
to the following:
bw = delivered / max(send_phase_rtt, ack_phase_rtt)
In large-scale experiments, this filtering approach incorporating
send_phase_rtt is effective at avoiding bandwidth overestimates due to
ACK compression or stretched ACKs.
Signed-off-by: Van Jacobson <vanj@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-19 21:39:14 -06:00
|
|
|
sack_state.rate = &rs;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-10-05 23:21:27 -06:00
|
|
|
/* We very likely will need to access rtx queue. */
|
|
|
|
prefetch(sk->tcp_rtx_queue.rb_node);
|
2014-10-11 16:17:29 -06:00
|
|
|
|
2009-03-22 22:49:57 -06:00
|
|
|
/* If the ack is older than previous acks
|
2005-04-16 16:20:36 -06:00
|
|
|
* then we can probably ignore it.
|
|
|
|
*/
|
2012-10-21 13:57:11 -06:00
|
|
|
if (before(ack, prior_snd_una)) {
|
|
|
|
/* RFC 5961 5.2 [Blind Data Injection Attack].[Mitigation] */
|
|
|
|
if (before(ack, prior_snd_una - tp->max_window)) {
|
tcp: better validation of received ack sequences
Paul Fiterau Brostean reported :
<quote>
Linux TCP stack we analyze exhibits behavior that seems odd to me.
The scenario is as follows (all packets have empty payloads, no window
scaling, rcv/snd window size should not be a factor):
TEST HARNESS (CLIENT) LINUX SERVER
1. - LISTEN (server listen,
then accepts)
2. - --> <SEQ=100><CTL=SYN> --> SYN-RECEIVED
3. - <-- <SEQ=300><ACK=101><CTL=SYN,ACK> <-- SYN-RECEIVED
4. - --> <SEQ=101><ACK=301><CTL=ACK> --> ESTABLISHED
5. - <-- <SEQ=301><ACK=101><CTL=FIN,ACK> <-- FIN WAIT-1 (server
opts to close the data connection calling "close" on the connection
socket)
6. - --> <SEQ=101><ACK=99999><CTL=FIN,ACK> --> CLOSING (client sends
FIN,ACK with not yet sent acknowledgement number)
7. - <-- <SEQ=302><ACK=102><CTL=ACK> <-- CLOSING (ACK is 102
instead of 101, why?)
... (silence from CLIENT)
8. - <-- <SEQ=301><ACK=102><CTL=FIN,ACK> <-- CLOSING
(retransmission, again ACK is 102)
Now, note that packet 6 while having the expected sequence number,
acknowledges something that wasn't sent by the server. So I would
expect
the packet to maybe prompt an ACK response from the server, and then be
ignored. Yet it is not ignored and actually leads to an increase of the
acknowledgement number in the server's retransmission of the FIN,ACK
packet. The explanation I found is that the FIN in packet 6 was
processed, despite the acknowledgement number being unacceptable.
Further experiments indeed show that the server processes this FIN,
transitioning to CLOSING, then on receiving an ACK for the FIN it had
send in packet 5, the server (or better said connection) transitions
from CLOSING to TIME_WAIT (as signaled by netstat).
</quote>
Indeed, tcp_rcv_state_process() calls tcp_ack() but
does not exploit the @acceptable status but for TCP_SYN_RECV
state.
What we want here is to send a challenge ACK, if not in TCP_SYN_RECV
state. TCP_FIN_WAIT1 state is not the only state we should fix.
Add a FLAG_NO_CHALLENGE_ACK so that tcp_rcv_state_process()
can choose to send a challenge ACK and discard the packet instead
of wrongly change socket state.
With help from Neal Cardwell.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Paul Fiterau Brostean <p.fiterau-brostean@science.ru.nl>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-05-23 16:24:46 -06:00
|
|
|
if (!(flag & FLAG_NO_CHALLENGE_ACK))
|
|
|
|
tcp_send_challenge_ack(sk, skb);
|
2012-10-21 13:57:11 -06:00
|
|
|
return -1;
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
goto old_ack;
|
2012-10-21 13:57:11 -06:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2009-03-22 22:49:57 -06:00
|
|
|
/* If the ack includes data we haven't sent yet, discard
|
|
|
|
* this segment (RFC793 Section 3.9).
|
|
|
|
*/
|
|
|
|
if (after(ack, tp->snd_nxt))
|
2019-02-25 03:33:48 -07:00
|
|
|
return -1;
|
2009-03-22 22:49:57 -06:00
|
|
|
|
2014-08-14 14:13:07 -06:00
|
|
|
if (after(ack, prior_snd_una)) {
|
2007-08-02 20:46:58 -06:00
|
|
|
flag |= FLAG_SND_UNA_ADVANCED;
|
2014-08-14 14:13:07 -06:00
|
|
|
icsk->icsk_retransmits = 0;
|
2018-04-30 01:16:10 -06:00
|
|
|
|
|
|
|
#if IS_ENABLED(CONFIG_TLS_DEVICE)
|
2019-05-08 17:46:14 -06:00
|
|
|
if (static_branch_unlikely(&clean_acked_data_enabled.key))
|
2018-04-30 01:16:10 -06:00
|
|
|
if (icsk->icsk_clean_acked)
|
|
|
|
icsk->icsk_clean_acked(sk, ack);
|
|
|
|
#endif
|
2014-08-14 14:13:07 -06:00
|
|
|
}
|
2007-08-02 20:46:58 -06:00
|
|
|
|
2017-11-14 22:02:19 -07:00
|
|
|
prior_fack = tcp_is_sack(tp) ? tcp_highest_sack_seq(tp) : tp->snd_una;
|
tcp: track data delivery rate for a TCP connection
This patch generates data delivery rate (throughput) samples on a
per-ACK basis. These rate samples can be used by congestion control
modules, and specifically will be used by TCP BBR in later patches in
this series.
Key state:
tp->delivered: Tracks the total number of data packets (original or not)
delivered so far. This is an already-existing field.
tp->delivered_mstamp: the last time tp->delivered was updated.
Algorithm:
A rate sample is calculated as (d1 - d0)/(t1 - t0) on a per-ACK basis:
d1: the current tp->delivered after processing the ACK
t1: the current time after processing the ACK
d0: the prior tp->delivered when the acked skb was transmitted
t0: the prior tp->delivered_mstamp when the acked skb was transmitted
When an skb is transmitted, we snapshot d0 and t0 in its control
block in tcp_rate_skb_sent().
When an ACK arrives, it may SACK and ACK some skbs. For each SACKed
or ACKed skb, tcp_rate_skb_delivered() updates the rate_sample struct
to reflect the latest (d0, t0).
Finally, tcp_rate_gen() generates a rate sample by storing
(d1 - d0) in rs->delivered and (t1 - t0) in rs->interval_us.
One caveat: if an skb was sent with no packets in flight, then
tp->delivered_mstamp may be either invalid (if the connection is
starting) or outdated (if the connection was idle). In that case,
we'll re-stamp tp->delivered_mstamp.
At first glance it seems t0 should always be the time when an skb was
transmitted, but actually this could over-estimate the rate due to
phase mismatch between transmit and ACK events. To track the delivery
rate, we ensure that if packets are in flight then t0 and and t1 are
times at which packets were marked delivered.
If the initial and final RTTs are different then one may be corrupted
by some sort of noise. The noise we see most often is sending gaps
caused by delayed, compressed, or stretched acks. This either affects
both RTTs equally or artificially reduces the final RTT. We approach
this by recording the info we need to compute the initial RTT
(duration of the "send phase" of the window) when we recorded the
associated inflight. Then, for a filter to avoid bandwidth
overestimates, we generalize the per-sample bandwidth computation
from:
bw = delivered / ack_phase_rtt
to the following:
bw = delivered / max(send_phase_rtt, ack_phase_rtt)
In large-scale experiments, this filtering approach incorporating
send_phase_rtt is effective at avoiding bandwidth overestimates due to
ACK compression or stretched ACKs.
Signed-off-by: Van Jacobson <vanj@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-19 21:39:14 -06:00
|
|
|
rs.prior_in_flight = tcp_packets_in_flight(tp);
|
2007-11-10 22:22:18 -07:00
|
|
|
|
tcp: call tcp_replace_ts_recent() from tcp_ack()
commit bd090dfc634d (tcp: tcp_replace_ts_recent() should not be called
from tcp_validate_incoming()) introduced a TS ecr bug in slow path
processing.
1 A > B P. 1:10001(10000) ack 1 <nop,nop,TS val 1001 ecr 200>
2 B < A . 1:1(0) ack 1 win 257 <sack 9001:10001,TS val 300 ecr 1001>
3 A > B . 1:1001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
4 A > B . 1001:2001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
(ecr 200 should be ecr 300 in packets 3 & 4)
Problem is tcp_ack() can trigger send of new packets (retransmits),
reflecting the prior TSval, instead of the TSval contained in the
currently processed incoming packet.
Fix this by calling tcp_replace_ts_recent() from tcp_ack() after the
checks, but before the actions.
Reported-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-04-19 01:19:48 -06:00
|
|
|
/* ts_recent update must be made after we are sure that the packet
|
|
|
|
* is in window.
|
|
|
|
*/
|
|
|
|
if (flag & FLAG_UPDATE_TS_RECENT)
|
|
|
|
tcp_replace_ts_recent(tp, TCP_SKB_CB(skb)->seq);
|
|
|
|
|
2018-11-11 05:10:10 -07:00
|
|
|
if ((flag & (FLAG_SLOWPATH | FLAG_SND_UNA_ADVANCED)) ==
|
|
|
|
FLAG_SND_UNA_ADVANCED) {
|
2017-08-30 11:24:58 -06:00
|
|
|
/* Window is constant, pure forward advance.
|
|
|
|
* No more checks are required.
|
|
|
|
* Note, we use the fact that SND.UNA>=SND.WL2.
|
|
|
|
*/
|
|
|
|
tcp_update_wl(tp, ack_seq);
|
|
|
|
tcp_snd_una_update(tp, ack);
|
|
|
|
flag |= FLAG_WIN_UPDATE;
|
|
|
|
|
|
|
|
tcp_in_ack_event(sk, CA_ACK_WIN_UPDATE);
|
|
|
|
|
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPHPACKS);
|
|
|
|
} else {
|
2017-08-30 11:24:57 -06:00
|
|
|
u32 ack_ev_flags = CA_ACK_SLOWPATH;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-08-30 11:24:57 -06:00
|
|
|
if (ack_seq != TCP_SKB_CB(skb)->end_seq)
|
|
|
|
flag |= FLAG_DATA;
|
|
|
|
else
|
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPPUREACKS);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-08-30 11:24:57 -06:00
|
|
|
flag |= tcp_ack_update_window(sk, skb, ack, ack_seq);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-08-30 11:24:57 -06:00
|
|
|
if (TCP_SKB_CB(skb)->sacked)
|
|
|
|
flag |= tcp_sacktag_write_queue(sk, skb, prior_snd_una,
|
|
|
|
&sack_state);
|
2014-09-26 14:37:35 -06:00
|
|
|
|
2017-08-30 11:24:57 -06:00
|
|
|
if (tcp_ecn_rcv_ecn_echo(tp, tcp_hdr(skb))) {
|
|
|
|
flag |= FLAG_ECE;
|
|
|
|
ack_ev_flags |= CA_ACK_ECE;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (flag & FLAG_WIN_UPDATE)
|
|
|
|
ack_ev_flags |= CA_ACK_WIN_UPDATE;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-08-30 11:24:57 -06:00
|
|
|
tcp_in_ack_event(sk, ack_ev_flags);
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* We passed data and got it acked, remove any soft error
|
|
|
|
* log. Something worked...
|
|
|
|
*/
|
|
|
|
sk->sk_err_soft = 0;
|
tcp: Clear probes_out more aggressively in tcp_ack().
This is based upon an excellent bug report from Eric Dumazet.
tcp_ack() should clear ->icsk_probes_out even if there are packets
outstanding. Otherwise if we get a sequence of ACKs while we do have
packets outstanding over and over again, we'll never clear the
probes_out value and eventually think the connection is too sick and
we'll reset it.
This appears to be some "optimization" added to tcp_ack() in the 2.4.x
timeframe. In 2.2.x, probes_out is pretty much always cleared by
tcp_ack().
Here is Eric's original report:
----------------------------------------
Apparently, we can in some situations reset TCP connections in a couple of seconds when some frames are lost.
In order to reproduce the problem, please try the following program on linux-2.6.25.*
Setup some iptables rules to allow two frames per second sent on loopback interface to tcp destination port 12000
iptables -N SLOWLO
iptables -A SLOWLO -m hashlimit --hashlimit 2 --hashlimit-burst 1 --hashlimit-mode dstip --hashlimit-name slow2 -j ACCEPT
iptables -A SLOWLO -j DROP
iptables -A OUTPUT -o lo -p tcp --dport 12000 -j SLOWLO
Then run the attached program and see the output :
# ./loop
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,1)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,3)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,5)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,7)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,9)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,11)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,201ms,13)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,188ms,15)
write(): Connection timed out
wrote 890 bytes but was interrupted after 9 seconds
ESTAB 0 0 127.0.0.1:12000 127.0.0.1:54455
Exiting read() because no data available (4000 ms timeout).
read 860 bytes
While this tcp session makes progress (sending frames with 50 bytes of payload, every 500ms), linux tcp stack decides to reset it, when tcp_retries 2 is reached (default value : 15)
tcpdump :
15:30:28.856695 IP 127.0.0.1.56554 > 127.0.0.1.12000: S 33788768:33788768(0) win 32792 <mss 16396,nop,nop,sackOK,nop,wscale 7>
15:30:28.856711 IP 127.0.0.1.12000 > 127.0.0.1.56554: S 33899253:33899253(0) ack 33788769 win 32792 <mss 16396,nop,nop,sackOK,nop,wscale 7>
15:30:29.356947 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 1:61(60) ack 1 win 257
15:30:29.356966 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 61 win 257
15:30:29.866415 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 61:111(50) ack 1 win 257
15:30:29.866427 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 111 win 257
15:30:30.366516 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 111:161(50) ack 1 win 257
15:30:30.366527 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 161 win 257
15:30:30.876196 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 161:211(50) ack 1 win 257
15:30:30.876207 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 211 win 257
15:30:31.376282 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 211:261(50) ack 1 win 257
15:30:31.376290 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 261 win 257
15:30:31.885619 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 261:311(50) ack 1 win 257
15:30:31.885631 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 311 win 257
15:30:32.385705 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 311:361(50) ack 1 win 257
15:30:32.385715 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 361 win 257
15:30:32.895249 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 361:411(50) ack 1 win 257
15:30:32.895266 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 411 win 257
15:30:33.395341 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 411:461(50) ack 1 win 257
15:30:33.395351 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 461 win 257
15:30:33.918085 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 461:511(50) ack 1 win 257
15:30:33.918096 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 511 win 257
15:30:34.418163 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 511:561(50) ack 1 win 257
15:30:34.418172 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 561 win 257
15:30:34.927685 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 561:611(50) ack 1 win 257
15:30:34.927698 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 611 win 257
15:30:35.427757 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 611:661(50) ack 1 win 257
15:30:35.427766 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 661 win 257
15:30:35.937359 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 661:711(50) ack 1 win 257
15:30:35.937376 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 711 win 257
15:30:36.437451 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 711:761(50) ack 1 win 257
15:30:36.437464 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 761 win 257
15:30:36.947022 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 761:811(50) ack 1 win 257
15:30:36.947039 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 811 win 257
15:30:37.447135 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 811:861(50) ack 1 win 257
15:30:37.447203 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 861 win 257
15:30:41.448171 IP 127.0.0.1.12000 > 127.0.0.1.56554: F 1:1(0) ack 861 win 257
15:30:41.448189 IP 127.0.0.1.56554 > 127.0.0.1.12000: R 33789629:33789629(0) win 0
Source of program :
/*
* small producer/consumer program.
* setup a listener on 127.0.0.1:12000
* Forks a child
* child connect to 127.0.0.1, and sends 10 bytes on this tcp socket every 100 ms
* Father accepts connection, and read all data
*/
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
#include <stdio.h>
#include <time.h>
#include <sys/poll.h>
int port = 12000;
char buffer[4096];
int main(int argc, char *argv[])
{
int lfd = socket(AF_INET, SOCK_STREAM, 0);
struct sockaddr_in socket_address;
time_t t0, t1;
int on = 1, sfd, res;
unsigned long total = 0;
socklen_t alen = sizeof(socket_address);
pid_t pid;
time(&t0);
socket_address.sin_family = AF_INET;
socket_address.sin_port = htons(port);
socket_address.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
if (lfd == -1) {
perror("socket()");
return 1;
}
setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(int));
if (bind(lfd, (struct sockaddr *)&socket_address, sizeof(socket_address)) == -1) {
perror("bind");
close(lfd);
return 1;
}
if (listen(lfd, 1) == -1) {
perror("listen()");
close(lfd);
return 1;
}
pid = fork();
if (pid == 0) {
int i, cfd = socket(AF_INET, SOCK_STREAM, 0);
close(lfd);
if (connect(cfd, (struct sockaddr *)&socket_address, sizeof(socket_address)) == -1) {
perror("connect()");
return 1;
}
for (i = 0 ; ;) {
res = write(cfd, "blablabla\n", 10);
if (res > 0) total += res;
else if (res == -1) {
perror("write()");
break;
} else break;
usleep(100000);
if (++i == 10) {
system("ss -on dst 127.0.0.1:12000");
i = 0;
}
}
time(&t1);
fprintf(stderr, "wrote %lu bytes but was interrupted after %g seconds\n", total, difftime(t1, t0));
system("ss -on | grep 127.0.0.1:12000");
close(cfd);
return 0;
}
sfd = accept(lfd, (struct sockaddr *)&socket_address, &alen);
if (sfd == -1) {
perror("accept");
return 1;
}
close(lfd);
while (1) {
struct pollfd pfd[1];
pfd[0].fd = sfd;
pfd[0].events = POLLIN;
if (poll(pfd, 1, 4000) == 0) {
fprintf(stderr, "Exiting read() because no data available (4000 ms timeout).\n");
break;
}
res = read(sfd, buffer, sizeof(buffer));
if (res > 0) total += res;
else if (res == 0) break;
else perror("read()");
}
fprintf(stderr, "read %lu bytes\n", total);
close(sfd);
return 0;
}
----------------------------------------
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-07-23 17:38:45 -06:00
|
|
|
icsk->icsk_probes_out = 0;
|
2017-05-16 15:00:07 -06:00
|
|
|
tp->rcv_tstamp = tcp_jiffies32;
|
2005-04-16 16:20:36 -06:00
|
|
|
if (!prior_packets)
|
|
|
|
goto no_queue;
|
|
|
|
|
|
|
|
/* See if we can take anything off of the retransmit queue. */
|
2017-11-08 14:01:27 -07:00
|
|
|
flag |= tcp_clean_rtx_queue(sk, prior_fack, prior_snd_una, &sack_state);
|
2011-08-21 14:21:57 -06:00
|
|
|
|
2017-11-03 17:38:48 -06:00
|
|
|
tcp_rack_update_reo_wnd(sk, &rs);
|
2011-08-21 14:21:57 -06:00
|
|
|
|
tcp: fix xmit timer to only be reset if data ACKed/SACKed
Fix a TCP loss recovery performance bug raised recently on the netdev
list, in two threads:
(i) July 26, 2017: netdev thread "TCP fast retransmit issues"
(ii) July 26, 2017: netdev thread:
"[PATCH V2 net-next] TLP: Don't reschedule PTO when there's one
outstanding TLP retransmission"
The basic problem is that incoming TCP packets that did not indicate
forward progress could cause the xmit timer (TLP or RTO) to be rearmed
and pushed back in time. In certain corner cases this could result in
the following problems noted in these threads:
- Repeated ACKs coming in with bogus SACKs corrupted by middleboxes
could cause TCP to repeatedly schedule TLPs forever. We kept
sending TLPs after every ~200ms, which elicited bogus SACKs, which
caused more TLPs, ad infinitum; we never fired an RTO to fill in
the holes.
- Incoming data segments could, in some cases, cause us to reschedule
our RTO or TLP timer further out in time, for no good reason. This
could cause repeated inbound data to result in stalls in outbound
data, in the presence of packet loss.
This commit fixes these bugs by changing the TLP and RTO ACK
processing to:
(a) Only reschedule the xmit timer once per ACK.
(b) Only reschedule the xmit timer if tcp_clean_rtx_queue() deems the
ACK indicates sufficient forward progress (a packet was
cumulatively ACKed, or we got a SACK for a packet that was sent
before the most recent retransmit of the write queue head).
This brings us back into closer compliance with the RFCs, since, as
the comment for tcp_rearm_rto() notes, we should only restart the RTO
timer after forward progress on the connection. Previously we were
restarting the xmit timer even in these cases where there was no
forward progress.
As a side benefit, this commit simplifies and speeds up the TCP timer
arming logic. We had been calling inet_csk_reset_xmit_timer() three
times on normal ACKs that cumulatively acknowledged some data:
1) Once near the top of tcp_ack() to switch from TLP timer to RTO:
if (icsk->icsk_pending == ICSK_TIME_LOSS_PROBE)
tcp_rearm_rto(sk);
2) Once in tcp_clean_rtx_queue(), to update the RTO:
if (flag & FLAG_ACKED) {
tcp_rearm_rto(sk);
3) Once in tcp_ack() after tcp_fastretrans_alert() to switch from RTO
to TLP:
if (icsk->icsk_pending == ICSK_TIME_RETRANS)
tcp_schedule_loss_probe(sk);
This commit, by only rescheduling the xmit timer once per ACK,
simplifies the code and reduces CPU overhead.
This commit was tested in an A/B test with Google web server
traffic. SNMP stats and request latency metrics were within noise
levels, substantiating that for normal web traffic patterns this is a
rare issue. This commit was also tested with packetdrill tests to
verify that it fixes the timer behavior in the corner cases discussed
in the netdev threads mentioned above.
This patch is a bug fix patch intended to be queued for -stable
relases.
Fixes: 6ba8a3b19e76 ("tcp: Tail loss probe (TLP)")
Reported-by: Klavs Klavsen <kl@vsen.dk>
Reported-by: Mao Wenan <maowenan@huawei.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-03 07:19:54 -06:00
|
|
|
if (tp->tlp_high_seq)
|
|
|
|
tcp_process_tlp_ack(sk, ack, flag);
|
|
|
|
/* If needed, reset TLP/RTO timer; RACK may later override this. */
|
|
|
|
if (flag & FLAG_SET_XMIT_TIMER)
|
|
|
|
tcp_set_xmit_timer(sk);
|
|
|
|
|
2005-08-10 01:03:31 -06:00
|
|
|
if (tcp_ack_is_dubious(sk, flag)) {
|
2018-11-27 15:42:01 -07:00
|
|
|
if (!(flag & (FLAG_SND_UNA_ADVANCED | FLAG_NOT_DUP))) {
|
|
|
|
num_dupack = 1;
|
|
|
|
/* Consider if pure acks were aggregated in tcp_add_backlog() */
|
|
|
|
if (!(flag & FLAG_DATA))
|
|
|
|
num_dupack = max_t(u16, 1, skb_shinfo(skb)->gso_segs);
|
|
|
|
}
|
|
|
|
tcp_fastretrans_alert(sk, prior_snd_una, num_dupack, &flag,
|
2017-11-08 14:01:27 -07:00
|
|
|
&rexmit);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2013-03-11 04:00:44 -06:00
|
|
|
|
2017-02-06 14:14:14 -07:00
|
|
|
if ((flag & FLAG_FORWARD_PROGRESS) || !(flag & FLAG_NOT_DUP))
|
|
|
|
sk_dst_confirm(sk);
|
tcp: Tail loss probe (TLP)
This patch series implement the Tail loss probe (TLP) algorithm described
in http://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01. The
first patch implements the basic algorithm.
TLP's goal is to reduce tail latency of short transactions. It achieves
this by converting retransmission timeouts (RTOs) occuring due
to tail losses (losses at end of transactions) into fast recovery.
TLP transmits one packet in two round-trips when a connection is in
Open state and isn't receiving any ACKs. The transmitted packet, aka
loss probe, can be either new or a retransmission. When there is tail
loss, the ACK from a loss probe triggers FACK/early-retransmit based
fast recovery, thus avoiding a costly RTO. In the absence of loss,
there is no change in the connection state.
PTO stands for probe timeout. It is a timer event indicating
that an ACK is overdue and triggers a loss probe packet. The PTO value
is set to max(2*SRTT, 10ms) and is adjusted to account for delayed
ACK timer when there is only one oustanding packet.
TLP Algorithm
On transmission of new data in Open state:
-> packets_out > 1: schedule PTO in max(2*SRTT, 10ms).
-> packets_out == 1: schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
-> PTO = min(PTO, RTO)
Conditions for scheduling PTO:
-> Connection is in Open state.
-> Connection is either cwnd limited or no new data to send.
-> Number of probes per tail loss episode is limited to one.
-> Connection is SACK enabled.
When PTO fires:
new_segment_exists:
-> transmit new segment.
-> packets_out++. cwnd remains same.
no_new_packet:
-> retransmit the last segment.
Its ACK triggers FACK or early retransmit based recovery.
ACK path:
-> rearm RTO at start of ACK processing.
-> reschedule PTO if need be.
In addition, the patch includes a small variation to the Early Retransmit
(ER) algorithm, such that ER and TLP together can in principle recover any
N-degree of tail loss through fast recovery. TLP is controlled by the same
sysctl as ER, tcp_early_retrans sysctl.
tcp_early_retrans==0; disables TLP and ER.
==1; enables RFC5827 ER.
==2; delayed ER.
==3; TLP and delayed ER. [DEFAULT]
==4; TLP only.
The TLP patch series have been extensively tested on Google Web servers.
It is most effective for short Web trasactions, where it reduced RTOs by 15%
and improved HTTP response time (average by 6%, 99th percentile by 10%).
The transmitted probes account for <0.5% of the overall transmissions.
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-03-11 04:00:43 -06:00
|
|
|
|
2018-04-18 00:18:47 -06:00
|
|
|
delivered = tcp_newly_delivered(sk, delivered, flag);
|
tcp: track data delivery rate for a TCP connection
This patch generates data delivery rate (throughput) samples on a
per-ACK basis. These rate samples can be used by congestion control
modules, and specifically will be used by TCP BBR in later patches in
this series.
Key state:
tp->delivered: Tracks the total number of data packets (original or not)
delivered so far. This is an already-existing field.
tp->delivered_mstamp: the last time tp->delivered was updated.
Algorithm:
A rate sample is calculated as (d1 - d0)/(t1 - t0) on a per-ACK basis:
d1: the current tp->delivered after processing the ACK
t1: the current time after processing the ACK
d0: the prior tp->delivered when the acked skb was transmitted
t0: the prior tp->delivered_mstamp when the acked skb was transmitted
When an skb is transmitted, we snapshot d0 and t0 in its control
block in tcp_rate_skb_sent().
When an ACK arrives, it may SACK and ACK some skbs. For each SACKed
or ACKed skb, tcp_rate_skb_delivered() updates the rate_sample struct
to reflect the latest (d0, t0).
Finally, tcp_rate_gen() generates a rate sample by storing
(d1 - d0) in rs->delivered and (t1 - t0) in rs->interval_us.
One caveat: if an skb was sent with no packets in flight, then
tp->delivered_mstamp may be either invalid (if the connection is
starting) or outdated (if the connection was idle). In that case,
we'll re-stamp tp->delivered_mstamp.
At first glance it seems t0 should always be the time when an skb was
transmitted, but actually this could over-estimate the rate due to
phase mismatch between transmit and ACK events. To track the delivery
rate, we ensure that if packets are in flight then t0 and and t1 are
times at which packets were marked delivered.
If the initial and final RTTs are different then one may be corrupted
by some sort of noise. The noise we see most often is sending gaps
caused by delayed, compressed, or stretched acks. This either affects
both RTTs equally or artificially reduces the final RTT. We approach
this by recording the info we need to compute the initial RTT
(duration of the "send phase" of the window) when we recorded the
associated inflight. Then, for a filter to avoid bandwidth
overestimates, we generalize the per-sample bandwidth computation
from:
bw = delivered / ack_phase_rtt
to the following:
bw = delivered / max(send_phase_rtt, ack_phase_rtt)
In large-scale experiments, this filtering approach incorporating
send_phase_rtt is effective at avoiding bandwidth overestimates due to
ACK compression or stretched ACKs.
Signed-off-by: Van Jacobson <vanj@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-19 21:39:14 -06:00
|
|
|
lost = tp->lost - lost; /* freshly marked lost */
|
2018-01-17 13:11:01 -07:00
|
|
|
rs.is_ack_delayed = !!(flag & FLAG_ACK_MAYBE_DELAYED);
|
2017-12-07 14:41:34 -07:00
|
|
|
tcp_rate_gen(sk, delivered, lost, is_sack_reneg, sack_state.rate);
|
2017-01-12 23:11:32 -07:00
|
|
|
tcp_cong_control(sk, ack, delivered, flag, sack_state.rate);
|
2016-02-02 11:33:04 -07:00
|
|
|
tcp_xmit_recovery(sk, rexmit);
|
2005-04-16 16:20:36 -06:00
|
|
|
return 1;
|
|
|
|
|
|
|
|
no_queue:
|
2011-11-16 01:58:02 -07:00
|
|
|
/* If data was DSACKed, see if we can undo a cwnd reduction. */
|
2018-04-18 00:18:47 -06:00
|
|
|
if (flag & FLAG_DSACKING_ACK) {
|
2018-11-27 15:42:01 -07:00
|
|
|
tcp_fastretrans_alert(sk, prior_snd_una, num_dupack, &flag,
|
2017-11-08 14:01:27 -07:00
|
|
|
&rexmit);
|
2018-04-18 00:18:47 -06:00
|
|
|
tcp_newly_delivered(sk, delivered, flag);
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
/* If this ack opens up a zero window, clear backoff. It was
|
|
|
|
* being used to time the probes, and is probably far higher than
|
|
|
|
* it needs to be for normal retransmission.
|
|
|
|
*/
|
2017-10-05 23:21:27 -06:00
|
|
|
tcp_ack_probe(sk);
|
2013-03-11 04:00:44 -06:00
|
|
|
|
|
|
|
if (tp->tlp_high_seq)
|
|
|
|
tcp_process_tlp_ack(sk, ack, flag);
|
2005-04-16 16:20:36 -06:00
|
|
|
return 1;
|
|
|
|
|
|
|
|
old_ack:
|
2011-11-16 01:58:03 -07:00
|
|
|
/* If data was SACKed, tag it and see if we should send more data.
|
|
|
|
* If data was DSACKed, see if we can undo a cwnd reduction.
|
|
|
|
*/
|
2008-06-04 12:34:22 -06:00
|
|
|
if (TCP_SKB_CB(skb)->sacked) {
|
2013-07-22 17:20:47 -06:00
|
|
|
flag |= tcp_sacktag_write_queue(sk, skb, prior_snd_una,
|
2015-04-30 17:10:57 -06:00
|
|
|
&sack_state);
|
2018-11-27 15:42:01 -07:00
|
|
|
tcp_fastretrans_alert(sk, prior_snd_una, num_dupack, &flag,
|
2017-11-08 14:01:27 -07:00
|
|
|
&rexmit);
|
2018-04-18 00:18:47 -06:00
|
|
|
tcp_newly_delivered(sk, delivered, flag);
|
2016-02-02 11:33:04 -07:00
|
|
|
tcp_xmit_recovery(sk, rexmit);
|
2008-06-04 12:34:22 -06:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2015-04-06 15:37:26 -06:00
|
|
|
static void tcp_parse_fastopen_option(int len, const unsigned char *cookie,
|
|
|
|
bool syn, struct tcp_fastopen_cookie *foc,
|
|
|
|
bool exp_opt)
|
|
|
|
{
|
|
|
|
/* Valid only in SYN or SYN-ACK with an even length. */
|
|
|
|
if (!foc || !syn || len < 0 || (len & 1))
|
|
|
|
return;
|
|
|
|
|
|
|
|
if (len >= TCP_FASTOPEN_COOKIE_MIN &&
|
|
|
|
len <= TCP_FASTOPEN_COOKIE_MAX)
|
|
|
|
memcpy(foc->val, cookie, len);
|
|
|
|
else if (len != 0)
|
|
|
|
len = -1;
|
|
|
|
foc->len = len;
|
|
|
|
foc->exp = exp_opt;
|
|
|
|
}
|
|
|
|
|
2017-10-25 03:01:45 -06:00
|
|
|
static void smc_parse_options(const struct tcphdr *th,
|
|
|
|
struct tcp_options_received *opt_rx,
|
|
|
|
const unsigned char *ptr,
|
|
|
|
int opsize)
|
|
|
|
{
|
|
|
|
#if IS_ENABLED(CONFIG_SMC)
|
|
|
|
if (static_branch_unlikely(&tcp_have_smc)) {
|
|
|
|
if (th->syn && !(opsize & 1) &&
|
|
|
|
opsize >= TCPOLEN_EXP_SMC_BASE &&
|
|
|
|
get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC)
|
|
|
|
opt_rx->smc_ok = 1;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Look for tcp options. Normally only called on SYN and SYNACK packets.
|
|
|
|
* But, this can also be called on packets in the established flow when
|
|
|
|
* the fast version below fails.
|
|
|
|
*/
|
2017-06-07 11:34:36 -06:00
|
|
|
void tcp_parse_options(const struct net *net,
|
|
|
|
const struct sk_buff *skb,
|
2013-03-17 02:23:34 -06:00
|
|
|
struct tcp_options_received *opt_rx, int estab,
|
2012-07-19 00:43:05 -06:00
|
|
|
struct tcp_fastopen_cookie *foc)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2011-10-21 03:22:42 -06:00
|
|
|
const unsigned char *ptr;
|
|
|
|
const struct tcphdr *th = tcp_hdr(skb);
|
2007-12-31 15:57:14 -07:00
|
|
|
int length = (th->doff * 4) - sizeof(struct tcphdr);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2011-10-21 03:22:42 -06:00
|
|
|
ptr = (const unsigned char *)(th + 1);
|
2005-04-16 16:20:36 -06:00
|
|
|
opt_rx->saw_tstamp = 0;
|
|
|
|
|
2007-03-08 21:45:19 -07:00
|
|
|
while (length > 0) {
|
2007-12-31 15:57:14 -07:00
|
|
|
int opcode = *ptr++;
|
2005-04-16 16:20:36 -06:00
|
|
|
int opsize;
|
|
|
|
|
|
|
|
switch (opcode) {
|
2008-01-03 21:36:55 -07:00
|
|
|
case TCPOPT_EOL:
|
|
|
|
return;
|
|
|
|
case TCPOPT_NOP: /* Ref: RFC 793 section 3.1 */
|
|
|
|
length--;
|
|
|
|
continue;
|
|
|
|
default:
|
2019-05-29 02:10:59 -06:00
|
|
|
if (length < 2)
|
|
|
|
return;
|
2008-01-03 21:36:55 -07:00
|
|
|
opsize = *ptr++;
|
|
|
|
if (opsize < 2) /* "silly options" */
|
2005-04-16 16:20:36 -06:00
|
|
|
return;
|
2008-01-03 21:36:55 -07:00
|
|
|
if (opsize > length)
|
|
|
|
return; /* don't parse partial options */
|
|
|
|
switch (opcode) {
|
|
|
|
case TCPOPT_MSS:
|
|
|
|
if (opsize == TCPOLEN_MSS && th->syn && !estab) {
|
2008-05-02 17:26:16 -06:00
|
|
|
u16 in_mss = get_unaligned_be16(ptr);
|
2008-01-03 21:36:55 -07:00
|
|
|
if (in_mss) {
|
|
|
|
if (opt_rx->user_mss &&
|
|
|
|
opt_rx->user_mss < in_mss)
|
|
|
|
in_mss = opt_rx->user_mss;
|
|
|
|
opt_rx->mss_clamp = in_mss;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2008-01-03 21:36:55 -07:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
case TCPOPT_WINDOW:
|
|
|
|
if (opsize == TCPOLEN_WINDOW && th->syn &&
|
2017-06-07 11:34:38 -06:00
|
|
|
!estab && net->ipv4.sysctl_tcp_window_scaling) {
|
2008-01-03 21:36:55 -07:00
|
|
|
__u8 snd_wscale = *(__u8 *)ptr;
|
|
|
|
opt_rx->wscale_ok = 1;
|
2017-04-04 07:09:48 -06:00
|
|
|
if (snd_wscale > TCP_MAX_WSCALE) {
|
|
|
|
net_info_ratelimited("%s: Illegal window scaling value %d > %u received\n",
|
2012-05-13 15:56:26 -06:00
|
|
|
__func__,
|
2017-04-04 07:09:48 -06:00
|
|
|
snd_wscale,
|
|
|
|
TCP_MAX_WSCALE);
|
|
|
|
snd_wscale = TCP_MAX_WSCALE;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2008-01-03 21:36:55 -07:00
|
|
|
opt_rx->snd_wscale = snd_wscale;
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case TCPOPT_TIMESTAMP:
|
|
|
|
if ((opsize == TCPOLEN_TIMESTAMP) &&
|
|
|
|
((estab && opt_rx->tstamp_ok) ||
|
2017-06-07 11:34:39 -06:00
|
|
|
(!estab && net->ipv4.sysctl_tcp_timestamps))) {
|
2008-01-03 21:36:55 -07:00
|
|
|
opt_rx->saw_tstamp = 1;
|
2008-05-02 17:26:16 -06:00
|
|
|
opt_rx->rcv_tsval = get_unaligned_be32(ptr);
|
|
|
|
opt_rx->rcv_tsecr = get_unaligned_be32(ptr + 4);
|
2008-01-03 21:36:55 -07:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
case TCPOPT_SACK_PERM:
|
|
|
|
if (opsize == TCPOLEN_SACK_PERM && th->syn &&
|
2017-06-07 11:34:37 -06:00
|
|
|
!estab && net->ipv4.sysctl_tcp_sack) {
|
2011-12-20 06:23:24 -07:00
|
|
|
opt_rx->sack_ok = TCP_SACK_SEEN;
|
2008-01-03 21:36:55 -07:00
|
|
|
tcp_sack_reset(opt_rx);
|
|
|
|
}
|
|
|
|
break;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2008-01-03 21:36:55 -07:00
|
|
|
case TCPOPT_SACK:
|
|
|
|
if ((opsize >= (TCPOLEN_SACK_BASE + TCPOLEN_SACK_PERBLOCK)) &&
|
|
|
|
!((opsize - TCPOLEN_SACK_BASE) % TCPOLEN_SACK_PERBLOCK) &&
|
|
|
|
opt_rx->sack_ok) {
|
|
|
|
TCP_SKB_CB(skb)->sacked = (ptr - 2) - (unsigned char *)th;
|
|
|
|
}
|
|
|
|
break;
|
2006-11-14 20:07:45 -07:00
|
|
|
#ifdef CONFIG_TCP_MD5SIG
|
2008-01-03 21:36:55 -07:00
|
|
|
case TCPOPT_MD5SIG:
|
|
|
|
/*
|
|
|
|
* The MD5 Hash has already been
|
|
|
|
* checked (see tcp_v{4,6}_do_rcv()).
|
|
|
|
*/
|
|
|
|
break;
|
2006-11-14 20:07:45 -07:00
|
|
|
#endif
|
2015-04-06 15:37:26 -06:00
|
|
|
case TCPOPT_FASTOPEN:
|
|
|
|
tcp_parse_fastopen_option(
|
|
|
|
opsize - TCPOLEN_FASTOPEN_BASE,
|
|
|
|
ptr, th->syn, foc, false);
|
|
|
|
break;
|
|
|
|
|
2012-07-19 00:43:05 -06:00
|
|
|
case TCPOPT_EXP:
|
|
|
|
/* Fast Open option shares code 254 using a
|
2015-04-06 15:37:26 -06:00
|
|
|
* 16 bits magic number.
|
2012-07-19 00:43:05 -06:00
|
|
|
*/
|
2015-04-06 15:37:26 -06:00
|
|
|
if (opsize >= TCPOLEN_EXP_FASTOPEN_BASE &&
|
|
|
|
get_unaligned_be16(ptr) ==
|
|
|
|
TCPOPT_FASTOPEN_MAGIC)
|
|
|
|
tcp_parse_fastopen_option(opsize -
|
|
|
|
TCPOLEN_EXP_FASTOPEN_BASE,
|
|
|
|
ptr + 2, th->syn, foc, true);
|
2017-10-25 03:01:45 -06:00
|
|
|
else
|
|
|
|
smc_parse_options(th, opt_rx, ptr,
|
|
|
|
opsize);
|
2012-07-19 00:43:05 -06:00
|
|
|
break;
|
|
|
|
|
|
|
|
}
|
2008-01-03 21:36:55 -07:00
|
|
|
ptr += opsize-2;
|
|
|
|
length -= opsize;
|
2007-04-20 18:09:22 -06:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
2010-07-09 15:22:10 -06:00
|
|
|
EXPORT_SYMBOL(tcp_parse_options);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2012-05-16 17:15:34 -06:00
|
|
|
static bool tcp_parse_aligned_timestamp(struct tcp_sock *tp, const struct tcphdr *th)
|
2008-08-23 06:12:29 -06:00
|
|
|
{
|
2011-10-21 03:22:42 -06:00
|
|
|
const __be32 *ptr = (const __be32 *)(th + 1);
|
2008-08-23 06:12:29 -06:00
|
|
|
|
|
|
|
if (*ptr == htonl((TCPOPT_NOP << 24) | (TCPOPT_NOP << 16)
|
|
|
|
| (TCPOPT_TIMESTAMP << 8) | TCPOLEN_TIMESTAMP)) {
|
|
|
|
tp->rx_opt.saw_tstamp = 1;
|
|
|
|
++ptr;
|
|
|
|
tp->rx_opt.rcv_tsval = ntohl(*ptr);
|
|
|
|
++ptr;
|
2013-08-27 02:21:55 -06:00
|
|
|
if (*ptr)
|
|
|
|
tp->rx_opt.rcv_tsecr = ntohl(*ptr) - tp->tsoffset;
|
|
|
|
else
|
|
|
|
tp->rx_opt.rcv_tsecr = 0;
|
2012-05-16 17:15:34 -06:00
|
|
|
return true;
|
2008-08-23 06:12:29 -06:00
|
|
|
}
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2008-08-23 06:12:29 -06:00
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Fast parse options. This hopes to only see timestamps.
|
|
|
|
* If it is wrong it falls back on tcp_parse_options().
|
|
|
|
*/
|
2017-06-07 11:34:36 -06:00
|
|
|
static bool tcp_fast_parse_options(const struct net *net,
|
|
|
|
const struct sk_buff *skb,
|
2013-03-17 02:23:34 -06:00
|
|
|
const struct tcphdr *th, struct tcp_sock *tp)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2009-12-02 11:25:27 -07:00
|
|
|
/* In the spirit of fast parsing, compare doff directly to constant
|
|
|
|
* values. Because equality is used, short doff can be ignored here.
|
|
|
|
*/
|
|
|
|
if (th->doff == (sizeof(*th) / 4)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->rx_opt.saw_tstamp = 0;
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2005-04-16 16:20:36 -06:00
|
|
|
} else if (tp->rx_opt.tstamp_ok &&
|
2009-12-02 11:25:27 -07:00
|
|
|
th->doff == ((sizeof(*th) + TCPOLEN_TSTAMP_ALIGNED) / 4)) {
|
2008-08-23 06:12:29 -06:00
|
|
|
if (tcp_parse_aligned_timestamp(tp, th))
|
2012-05-16 17:15:34 -06:00
|
|
|
return true;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2013-02-10 22:50:19 -07:00
|
|
|
|
2017-06-07 11:34:36 -06:00
|
|
|
tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL);
|
2013-08-27 02:21:55 -06:00
|
|
|
if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
|
2013-02-10 22:50:19 -07:00
|
|
|
tp->rx_opt.rcv_tsecr -= tp->tsoffset;
|
|
|
|
|
2012-05-16 17:15:34 -06:00
|
|
|
return true;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2008-04-16 21:29:53 -06:00
|
|
|
#ifdef CONFIG_TCP_MD5SIG
|
|
|
|
/*
|
|
|
|
* Parse MD5 Signature option
|
|
|
|
*/
|
2011-10-21 03:22:42 -06:00
|
|
|
const u8 *tcp_parse_md5sig_option(const struct tcphdr *th)
|
2008-04-16 21:29:53 -06:00
|
|
|
{
|
2011-10-21 03:22:42 -06:00
|
|
|
int length = (th->doff << 2) - sizeof(*th);
|
|
|
|
const u8 *ptr = (const u8 *)(th + 1);
|
2008-04-16 21:29:53 -06:00
|
|
|
|
tcp: don't read out-of-bounds opsize
The old code reads the "opsize" variable from out-of-bounds memory (first
byte behind the segment) if a broken TCP segment ends directly after an
opcode that is neither EOL nor NOP.
The result of the read isn't used for anything, so the worst thing that
could theoretically happen is a pagefault; and since the physmap is usually
mostly contiguous, even that seems pretty unlikely.
The following C reproducer triggers the uninitialized read - however, you
can't actually see anything happen unless you put something like a
pr_warn() in tcp_parse_md5sig_option() to print the opsize.
====================================
#define _GNU_SOURCE
#include <arpa/inet.h>
#include <stdlib.h>
#include <errno.h>
#include <stdarg.h>
#include <net/if.h>
#include <linux/if.h>
#include <linux/ip.h>
#include <linux/tcp.h>
#include <linux/in.h>
#include <linux/if_tun.h>
#include <err.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <assert.h>
void systemf(const char *command, ...) {
char *full_command;
va_list ap;
va_start(ap, command);
if (vasprintf(&full_command, command, ap) == -1)
err(1, "vasprintf");
va_end(ap);
printf("systemf: <<<%s>>>\n", full_command);
system(full_command);
}
char *devname;
int tun_alloc(char *name) {
int fd = open("/dev/net/tun", O_RDWR);
if (fd == -1)
err(1, "open tun dev");
static struct ifreq req = { .ifr_flags = IFF_TUN|IFF_NO_PI };
strcpy(req.ifr_name, name);
if (ioctl(fd, TUNSETIFF, &req))
err(1, "TUNSETIFF");
devname = req.ifr_name;
printf("device name: %s\n", devname);
return fd;
}
#define IPADDR(a,b,c,d) (((a)<<0)+((b)<<8)+((c)<<16)+((d)<<24))
void sum_accumulate(unsigned int *sum, void *data, int len) {
assert((len&2)==0);
for (int i=0; i<len/2; i++) {
*sum += ntohs(((unsigned short *)data)[i]);
}
}
unsigned short sum_final(unsigned int sum) {
sum = (sum >> 16) + (sum & 0xffff);
sum = (sum >> 16) + (sum & 0xffff);
return htons(~sum);
}
void fix_ip_sum(struct iphdr *ip) {
unsigned int sum = 0;
sum_accumulate(&sum, ip, sizeof(*ip));
ip->check = sum_final(sum);
}
void fix_tcp_sum(struct iphdr *ip, struct tcphdr *tcp) {
unsigned int sum = 0;
struct {
unsigned int saddr;
unsigned int daddr;
unsigned char pad;
unsigned char proto_num;
unsigned short tcp_len;
} fakehdr = {
.saddr = ip->saddr,
.daddr = ip->daddr,
.proto_num = ip->protocol,
.tcp_len = htons(ntohs(ip->tot_len) - ip->ihl*4)
};
sum_accumulate(&sum, &fakehdr, sizeof(fakehdr));
sum_accumulate(&sum, tcp, tcp->doff*4);
tcp->check = sum_final(sum);
}
int main(void) {
int tun_fd = tun_alloc("inject_dev%d");
systemf("ip link set %s up", devname);
systemf("ip addr add 192.168.42.1/24 dev %s", devname);
struct {
struct iphdr ip;
struct tcphdr tcp;
unsigned char tcp_opts[20];
} __attribute__((packed)) syn_packet = {
.ip = {
.ihl = sizeof(struct iphdr)/4,
.version = 4,
.tot_len = htons(sizeof(syn_packet)),
.ttl = 30,
.protocol = IPPROTO_TCP,
/* FIXUP check */
.saddr = IPADDR(192,168,42,2),
.daddr = IPADDR(192,168,42,1)
},
.tcp = {
.source = htons(1),
.dest = htons(1337),
.seq = 0x12345678,
.doff = (sizeof(syn_packet.tcp)+sizeof(syn_packet.tcp_opts))/4,
.syn = 1,
.window = htons(64),
.check = 0 /*FIXUP*/
},
.tcp_opts = {
/* INVALID: trailing MD5SIG opcode after NOPs */
1, 1, 1, 1, 1,
1, 1, 1, 1, 1,
1, 1, 1, 1, 1,
1, 1, 1, 1, 19
}
};
fix_ip_sum(&syn_packet.ip);
fix_tcp_sum(&syn_packet.ip, &syn_packet.tcp);
while (1) {
int write_res = write(tun_fd, &syn_packet, sizeof(syn_packet));
if (write_res != sizeof(syn_packet))
err(1, "packet write failed");
}
}
====================================
Fixes: cfb6eeb4c860 ("[TCP]: MD5 Signature Option (RFC2385) support.")
Signed-off-by: Jann Horn <jannh@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-20 07:57:30 -06:00
|
|
|
/* If not enough data remaining, we can short cut */
|
|
|
|
while (length >= TCPOLEN_MD5SIG) {
|
2008-04-16 21:29:53 -06:00
|
|
|
int opcode = *ptr++;
|
|
|
|
int opsize;
|
|
|
|
|
2013-12-22 23:37:26 -07:00
|
|
|
switch (opcode) {
|
2008-04-16 21:29:53 -06:00
|
|
|
case TCPOPT_EOL:
|
|
|
|
return NULL;
|
|
|
|
case TCPOPT_NOP:
|
|
|
|
length--;
|
|
|
|
continue;
|
|
|
|
default:
|
|
|
|
opsize = *ptr++;
|
|
|
|
if (opsize < 2 || opsize > length)
|
|
|
|
return NULL;
|
|
|
|
if (opcode == TCPOPT_MD5SIG)
|
2010-08-07 21:24:28 -06:00
|
|
|
return opsize == TCPOLEN_MD5SIG ? ptr : NULL;
|
2008-04-16 21:29:53 -06:00
|
|
|
}
|
|
|
|
ptr += opsize - 2;
|
|
|
|
length -= opsize;
|
|
|
|
}
|
|
|
|
return NULL;
|
|
|
|
}
|
2010-07-09 15:22:10 -06:00
|
|
|
EXPORT_SYMBOL(tcp_parse_md5sig_option);
|
2008-04-16 21:29:53 -06:00
|
|
|
#endif
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Sorry, PAWS as specified is broken wrt. pure-ACKs -DaveM
|
|
|
|
*
|
|
|
|
* It is not fatal. If this ACK does _not_ change critical state (seqs, window)
|
|
|
|
* it can pass through stack. So, the following predicate verifies that
|
|
|
|
* this segment is not used for anything but congestion avoidance or
|
|
|
|
* fast retransmit. Moreover, we even are able to eliminate most of such
|
|
|
|
* second order effects, if we apply some small "replay" window (~RTO)
|
|
|
|
* to timestamp space.
|
|
|
|
*
|
|
|
|
* All these measures still do not guarantee that we reject wrapped ACKs
|
|
|
|
* on networks with high bandwidth, when sequence space is recycled fastly,
|
|
|
|
* but it guarantees that such events will be very rare and do not affect
|
|
|
|
* connection seriously. This doesn't look nice, but alas, PAWS is really
|
|
|
|
* buggy extension.
|
|
|
|
*
|
|
|
|
* [ Later note. Even worse! It is buggy for segments _with_ data. RFC
|
|
|
|
* states that events when retransmit arrives after original data are rare.
|
|
|
|
* It is a blatant lie. VJ forgot about fast retransmit! 8)8) It is
|
|
|
|
* the biggest problem on large power networks even with minor reordering.
|
|
|
|
* OK, let's give it small replay window. If peer clock is even 1hz, it is safe
|
|
|
|
* up to bandwidth of 18Gigabit/sec. 8) ]
|
|
|
|
*/
|
|
|
|
|
2005-08-09 21:10:42 -06:00
|
|
|
static int tcp_disordered_ack(const struct sock *sk, const struct sk_buff *skb)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2011-10-21 03:22:42 -06:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
const struct tcphdr *th = tcp_hdr(skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
u32 seq = TCP_SKB_CB(skb)->seq;
|
|
|
|
u32 ack = TCP_SKB_CB(skb)->ack_seq;
|
|
|
|
|
|
|
|
return (/* 1. Pure ACK with correct sequence number. */
|
|
|
|
(th->ack && seq == TCP_SKB_CB(skb)->end_seq && seq == tp->rcv_nxt) &&
|
|
|
|
|
|
|
|
/* 2. ... and duplicate ACK. */
|
|
|
|
ack == tp->snd_una &&
|
|
|
|
|
|
|
|
/* 3. ... and does not update window. */
|
|
|
|
!tcp_may_update_window(tp, ack, seq, ntohs(th->window) << tp->rx_opt.snd_wscale) &&
|
|
|
|
|
|
|
|
/* 4. ... and sits in replay window. */
|
2005-08-09 21:10:42 -06:00
|
|
|
(s32)(tp->rx_opt.ts_recent - tp->rx_opt.rcv_tsval) <= (inet_csk(sk)->icsk_rto * 1024) / HZ);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2012-07-19 15:32:18 -06:00
|
|
|
static inline bool tcp_paws_discard(const struct sock *sk,
|
2007-12-31 15:57:14 -07:00
|
|
|
const struct sk_buff *skb)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-09 21:10:42 -06:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
2009-03-14 08:23:03 -06:00
|
|
|
|
|
|
|
return !tcp_paws_check(&tp->rx_opt, TCP_PAWS_WINDOW) &&
|
|
|
|
!tcp_disordered_ack(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Check segment sequence number for validity.
|
|
|
|
*
|
|
|
|
* Segment controls are considered valid, if the segment
|
|
|
|
* fits to the window after truncation to the window. Acceptability
|
|
|
|
* of data (and SYN, FIN, of course) is checked separately.
|
|
|
|
* See tcp_data_queue(), for example.
|
|
|
|
*
|
|
|
|
* Also, controls (RST is main one) are accepted using RCV.WUP instead
|
|
|
|
* of RCV.NXT. Peer still did not advance his SND.UNA when we
|
|
|
|
* delayed ACK, so that hisSND.UNA<=ourRCV.WUP.
|
|
|
|
* (borrowed from freebsd)
|
|
|
|
*/
|
|
|
|
|
2012-07-19 15:32:18 -06:00
|
|
|
static inline bool tcp_sequence(const struct tcp_sock *tp, u32 seq, u32 end_seq)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
return !before(end_seq, tp->rcv_wup) &&
|
|
|
|
!after(seq, tp->rcv_nxt + tcp_receive_window(tp));
|
|
|
|
}
|
|
|
|
|
|
|
|
/* When we get a reset we do this. */
|
2012-08-31 06:29:11 -06:00
|
|
|
void tcp_reset(struct sock *sk)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2017-10-23 10:20:25 -06:00
|
|
|
trace_tcp_receive_reset(sk);
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* We want the right error as BSD sees it (and indeed as we do). */
|
|
|
|
switch (sk->sk_state) {
|
2007-12-31 15:57:14 -07:00
|
|
|
case TCP_SYN_SENT:
|
|
|
|
sk->sk_err = ECONNREFUSED;
|
|
|
|
break;
|
|
|
|
case TCP_CLOSE_WAIT:
|
|
|
|
sk->sk_err = EPIPE;
|
|
|
|
break;
|
|
|
|
case TCP_CLOSE:
|
|
|
|
return;
|
|
|
|
default:
|
|
|
|
sk->sk_err = ECONNRESET;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2010-09-20 16:42:05 -06:00
|
|
|
/* This barrier is coupled with smp_rmb() in tcp_poll() */
|
|
|
|
smp_wmb();
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2018-02-27 16:32:18 -07:00
|
|
|
tcp_write_queue_purge(sk);
|
2017-04-18 10:45:51 -06:00
|
|
|
tcp_done(sk);
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
if (!sock_flag(sk, SOCK_DEAD))
|
|
|
|
sk->sk_error_report(sk);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Process the FIN bit. This now behaves as it is supposed to work
|
|
|
|
* and the FIN takes effect when it is validly part of sequence
|
|
|
|
* space. Not before when we get holes.
|
|
|
|
*
|
|
|
|
* If we are ESTABLISHED, a received fin moves us to CLOSE-WAIT
|
|
|
|
* (and thence onto LAST-ACK and finally, CLOSE, we never enter
|
|
|
|
* TIME-WAIT)
|
|
|
|
*
|
|
|
|
* If we are in FINWAIT-1, a received FIN indicates simultaneous
|
|
|
|
* close and we go into CLOSING (and later onto TIME-WAIT)
|
|
|
|
*
|
|
|
|
* If we are in FINWAIT-2, a received FIN moves us to TIME-WAIT.
|
|
|
|
*/
|
2016-02-06 12:16:28 -07:00
|
|
|
void tcp_fin(struct sock *sk)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-08-09 21:10:42 -06:00
|
|
|
inet_csk_schedule_ack(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
sk->sk_shutdown |= RCV_SHUTDOWN;
|
|
|
|
sock_set_flag(sk, SOCK_DONE);
|
|
|
|
|
|
|
|
switch (sk->sk_state) {
|
2007-12-31 15:57:14 -07:00
|
|
|
case TCP_SYN_RECV:
|
|
|
|
case TCP_ESTABLISHED:
|
|
|
|
/* Move to CLOSE_WAIT */
|
|
|
|
tcp_set_state(sk, TCP_CLOSE_WAIT);
|
2019-01-25 11:53:19 -07:00
|
|
|
inet_csk_enter_pingpong_mode(sk);
|
2007-12-31 15:57:14 -07:00
|
|
|
break;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2007-12-31 15:57:14 -07:00
|
|
|
case TCP_CLOSE_WAIT:
|
|
|
|
case TCP_CLOSING:
|
|
|
|
/* Received a retransmission of the FIN, do
|
|
|
|
* nothing.
|
|
|
|
*/
|
|
|
|
break;
|
|
|
|
case TCP_LAST_ACK:
|
|
|
|
/* RFC793: Remain in the LAST-ACK state. */
|
|
|
|
break;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2007-12-31 15:57:14 -07:00
|
|
|
case TCP_FIN_WAIT1:
|
|
|
|
/* This case occurs when a simultaneous close
|
|
|
|
* happens, we must ack the received FIN and
|
|
|
|
* enter the CLOSING state.
|
|
|
|
*/
|
|
|
|
tcp_send_ack(sk);
|
|
|
|
tcp_set_state(sk, TCP_CLOSING);
|
|
|
|
break;
|
|
|
|
case TCP_FIN_WAIT2:
|
|
|
|
/* Received a FIN -- send ACK and enter TIME_WAIT. */
|
|
|
|
tcp_send_ack(sk);
|
|
|
|
tcp_time_wait(sk, TCP_TIME_WAIT, 0);
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
/* Only TCP_LISTEN and TCP_CLOSE are left, in these
|
|
|
|
* cases we should never reach this piece of code.
|
|
|
|
*/
|
2012-03-11 12:36:11 -06:00
|
|
|
pr_err("%s: Impossible, sk->sk_state=%d\n",
|
2008-03-05 21:47:47 -07:00
|
|
|
__func__, sk->sk_state);
|
2007-12-31 15:57:14 -07:00
|
|
|
break;
|
2007-04-20 18:09:22 -06:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* It _is_ possible, that we have something out-of-order _after_ FIN.
|
|
|
|
* Probably, we should reset in this case. For now drop them.
|
|
|
|
*/
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
skb_rbtree_purge(&tp->out_of_order_queue);
|
2007-08-09 06:14:46 -06:00
|
|
|
if (tcp_is_sack(tp))
|
2005-04-16 16:20:36 -06:00
|
|
|
tcp_sack_reset(&tp->rx_opt);
|
2007-12-31 01:11:19 -07:00
|
|
|
sk_mem_reclaim(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
if (!sock_flag(sk, SOCK_DEAD)) {
|
|
|
|
sk->sk_state_change(sk);
|
|
|
|
|
|
|
|
/* Do not send POLL_HUP for half duplex close. */
|
|
|
|
if (sk->sk_shutdown == SHUTDOWN_MASK ||
|
|
|
|
sk->sk_state == TCP_CLOSE)
|
2007-11-26 05:10:50 -07:00
|
|
|
sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_HUP);
|
2005-04-16 16:20:36 -06:00
|
|
|
else
|
2007-11-26 05:10:50 -07:00
|
|
|
sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_IN);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-05-16 17:15:34 -06:00
|
|
|
static inline bool tcp_sack_extend(struct tcp_sack_block *sp, u32 seq,
|
2007-12-31 15:57:14 -07:00
|
|
|
u32 end_seq)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
if (!after(seq, sp->end_seq) && !after(sp->start_seq, end_seq)) {
|
|
|
|
if (before(seq, sp->start_seq))
|
|
|
|
sp->start_seq = seq;
|
|
|
|
if (after(end_seq, sp->end_seq))
|
|
|
|
sp->end_seq = end_seq;
|
2012-05-16 17:15:34 -06:00
|
|
|
return true;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2008-07-16 21:29:51 -06:00
|
|
|
static void tcp_dsack_set(struct sock *sk, u32 seq, u32 end_seq)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2008-07-16 21:29:51 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2017-10-26 22:55:07 -06:00
|
|
|
if (tcp_is_sack(tp) && sock_net(sk)->ipv4.sysctl_tcp_dsack) {
|
2008-07-03 02:05:41 -06:00
|
|
|
int mib_idx;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
if (before(seq, tp->rcv_nxt))
|
2008-07-03 02:05:41 -06:00
|
|
|
mib_idx = LINUX_MIB_TCPDSACKOLDSENT;
|
2005-04-16 16:20:36 -06:00
|
|
|
else
|
2008-07-03 02:05:41 -06:00
|
|
|
mib_idx = LINUX_MIB_TCPDSACKOFOSENT;
|
|
|
|
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), mib_idx);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
tp->rx_opt.dsack = 1;
|
|
|
|
tp->duplicate_sack[0].start_seq = seq;
|
|
|
|
tp->duplicate_sack[0].end_seq = end_seq;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-07-16 21:29:51 -06:00
|
|
|
static void tcp_dsack_extend(struct sock *sk, u32 seq, u32 end_seq)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2008-07-16 21:29:51 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
if (!tp->rx_opt.dsack)
|
2008-07-16 21:29:51 -06:00
|
|
|
tcp_dsack_set(sk, seq, end_seq);
|
2005-04-16 16:20:36 -06:00
|
|
|
else
|
|
|
|
tcp_sack_extend(tp->duplicate_sack, seq, end_seq);
|
|
|
|
}
|
|
|
|
|
2018-08-29 15:53:56 -06:00
|
|
|
static void tcp_rcv_spurious_retrans(struct sock *sk, const struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
/* When the ACK path fails or drops most ACKs, the sender would
|
|
|
|
* timeout and spuriously retransmit the same segment repeatedly.
|
|
|
|
* The receiver remembers and reflects via DSACKs. Leverage the
|
|
|
|
* DSACK state and change the txhash to re-route speculatively.
|
|
|
|
*/
|
|
|
|
if (TCP_SKB_CB(skb)->seq == tcp_sk(sk)->duplicate_sack[0].start_seq)
|
|
|
|
sk_rethink_txhash(sk);
|
|
|
|
}
|
|
|
|
|
2011-10-21 03:22:42 -06:00
|
|
|
static void tcp_send_dupack(struct sock *sk, const struct sk_buff *skb)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
if (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
|
|
|
|
before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt)) {
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_DELAYEDACKLOST);
|
2018-05-21 16:08:56 -06:00
|
|
|
tcp_enter_quickack_mode(sk, TCP_MAX_QUICKACKS);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-10-26 22:55:07 -06:00
|
|
|
if (tcp_is_sack(tp) && sock_net(sk)->ipv4.sysctl_tcp_dsack) {
|
2005-04-16 16:20:36 -06:00
|
|
|
u32 end_seq = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
|
2018-08-29 15:53:56 -06:00
|
|
|
tcp_rcv_spurious_retrans(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
if (after(TCP_SKB_CB(skb)->end_seq, tp->rcv_nxt))
|
|
|
|
end_seq = tp->rcv_nxt;
|
2008-07-16 21:29:51 -06:00
|
|
|
tcp_dsack_set(sk, TCP_SKB_CB(skb)->seq, end_seq);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
tcp_send_ack(sk);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* These routines update the SACK block as out-of-order packets arrive or
|
|
|
|
* in-order packets close up the sequence space.
|
|
|
|
*/
|
|
|
|
static void tcp_sack_maybe_coalesce(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
int this_sack;
|
|
|
|
struct tcp_sack_block *sp = &tp->selective_acks[0];
|
2007-12-31 15:57:14 -07:00
|
|
|
struct tcp_sack_block *swalk = sp + 1;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* See if the recent change to the first SACK eats into
|
|
|
|
* or hits the sequence space of other SACK blocks, if so coalesce.
|
|
|
|
*/
|
2007-12-31 15:57:14 -07:00
|
|
|
for (this_sack = 1; this_sack < tp->rx_opt.num_sacks;) {
|
2005-04-16 16:20:36 -06:00
|
|
|
if (tcp_sack_extend(sp, swalk->start_seq, swalk->end_seq)) {
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/* Zap SWALK, by moving every further SACK up by one slot.
|
|
|
|
* Decrease num_sacks.
|
|
|
|
*/
|
|
|
|
tp->rx_opt.num_sacks--;
|
2007-12-31 15:57:14 -07:00
|
|
|
for (i = this_sack; i < tp->rx_opt.num_sacks; i++)
|
|
|
|
sp[i] = sp[i + 1];
|
2005-04-16 16:20:36 -06:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
this_sack++, swalk++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void tcp_sack_new_ofo_skb(struct sock *sk, u32 seq, u32 end_seq)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct tcp_sack_block *sp = &tp->selective_acks[0];
|
|
|
|
int cur_sacks = tp->rx_opt.num_sacks;
|
|
|
|
int this_sack;
|
|
|
|
|
|
|
|
if (!cur_sacks)
|
|
|
|
goto new_sack;
|
|
|
|
|
2007-12-31 15:57:14 -07:00
|
|
|
for (this_sack = 0; this_sack < cur_sacks; this_sack++, sp++) {
|
2005-04-16 16:20:36 -06:00
|
|
|
if (tcp_sack_extend(sp, seq, end_seq)) {
|
|
|
|
/* Rotate this_sack to the first one. */
|
2007-12-31 15:57:14 -07:00
|
|
|
for (; this_sack > 0; this_sack--, sp--)
|
2009-03-21 14:36:17 -06:00
|
|
|
swap(*sp, *(sp - 1));
|
2005-04-16 16:20:36 -06:00
|
|
|
if (cur_sacks > 1)
|
|
|
|
tcp_sack_maybe_coalesce(tp);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Could not find an adjacent existing SACK, build a new one,
|
|
|
|
* put it at the front, and shift everyone else down. We
|
|
|
|
* always know there is at least one SACK present already here.
|
|
|
|
*
|
|
|
|
* If the sack array is full, forget about the last one.
|
|
|
|
*/
|
2008-07-19 01:07:02 -06:00
|
|
|
if (this_sack >= TCP_NUM_SACKS) {
|
2018-11-20 06:53:59 -07:00
|
|
|
if (tp->compressed_ack > TCP_FASTRETRANS_THRESH)
|
tcp: add SACK compression
When TCP receives an out-of-order packet, it immediately sends
a SACK packet, generating network load but also forcing the
receiver to send 1-MSS pathological packets, increasing its
RTX queue length/depth, and thus processing time.
Wifi networks suffer from this aggressive behavior, but generally
speaking, all these SACK packets add fuel to the fire when networks
are under congestion.
This patch adds a high resolution timer and tp->compressed_ack counter.
Instead of sending a SACK, we program this timer with a small delay,
based on RTT and capped to 1 ms :
delay = min ( 5 % of RTT, 1 ms)
If subsequent SACKs need to be sent while the timer has not yet
expired, we simply increment tp->compressed_ack.
When timer expires, a SACK is sent with the latest information.
Whenever an ACK is sent (if data is sent, or if in-order
data is received) timer is canceled.
Note that tcp_sack_new_ofo_skb() is able to force a SACK to be sent
if the sack blocks need to be shuffled, even if the timer has not
expired.
A new SNMP counter is added in the following patch.
Two other patches add sysctls to allow changing the 1,000,000 and 44
values that this commit hard-coded.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Toke Høiland-Jørgensen <toke@toke.dk>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-05-17 15:47:26 -06:00
|
|
|
tcp_send_ack(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
this_sack--;
|
|
|
|
tp->rx_opt.num_sacks--;
|
|
|
|
sp--;
|
|
|
|
}
|
2007-03-08 21:45:19 -07:00
|
|
|
for (; this_sack > 0; this_sack--, sp--)
|
2007-12-31 15:57:14 -07:00
|
|
|
*sp = *(sp - 1);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
new_sack:
|
|
|
|
/* Build the new head SACK, and we're done. */
|
|
|
|
sp->start_seq = seq;
|
|
|
|
sp->end_seq = end_seq;
|
|
|
|
tp->rx_opt.num_sacks++;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* RCV.NXT advances, some SACKs should be eaten. */
|
|
|
|
|
|
|
|
static void tcp_sack_remove(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
struct tcp_sack_block *sp = &tp->selective_acks[0];
|
|
|
|
int num_sacks = tp->rx_opt.num_sacks;
|
|
|
|
int this_sack;
|
|
|
|
|
|
|
|
/* Empty ofo queue, hence, all the SACKs are eaten. Clear. */
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
if (RB_EMPTY_ROOT(&tp->out_of_order_queue)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->rx_opt.num_sacks = 0;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2007-12-31 15:57:14 -07:00
|
|
|
for (this_sack = 0; this_sack < num_sacks;) {
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Check if the start of the sack is covered by RCV.NXT. */
|
|
|
|
if (!before(tp->rcv_nxt, sp->start_seq)) {
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/* RCV.NXT must cover all the block! */
|
2008-07-25 22:43:18 -06:00
|
|
|
WARN_ON(before(tp->rcv_nxt, sp->end_seq));
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Zap this SACK, by moving forward any other SACKS. */
|
2013-12-22 23:37:26 -07:00
|
|
|
for (i = this_sack+1; i < num_sacks; i++)
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->selective_acks[i-1] = tp->selective_acks[i];
|
|
|
|
num_sacks--;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
this_sack++;
|
|
|
|
sp++;
|
|
|
|
}
|
2009-03-14 08:23:01 -06:00
|
|
|
tp->rx_opt.num_sacks = num_sacks;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2014-09-19 09:26:20 -06:00
|
|
|
/**
|
|
|
|
* tcp_try_coalesce - try to merge skb to prior one
|
|
|
|
* @sk: socket
|
2017-08-22 15:08:48 -06:00
|
|
|
* @dest: destination queue
|
2014-09-19 09:26:20 -06:00
|
|
|
* @to: prior buffer
|
|
|
|
* @from: buffer to add in queue
|
|
|
|
* @fragstolen: pointer to boolean
|
|
|
|
*
|
|
|
|
* Before queueing skb @from after @to, try to merge them
|
|
|
|
* to reduce overall memory use and queue lengths, if cost is small.
|
|
|
|
* Packets in ofo or receive queues can stay a long time.
|
|
|
|
* Better try to coalesce them right now to avoid future collapses.
|
|
|
|
* Returns true if caller should free @from instead of queueing it
|
|
|
|
*/
|
|
|
|
static bool tcp_try_coalesce(struct sock *sk,
|
|
|
|
struct sk_buff *to,
|
|
|
|
struct sk_buff *from,
|
|
|
|
bool *fragstolen)
|
|
|
|
{
|
|
|
|
int delta;
|
|
|
|
|
|
|
|
*fragstolen = false;
|
|
|
|
|
|
|
|
/* Its possible this segment overlaps with prior segment in queue */
|
|
|
|
if (TCP_SKB_CB(from)->seq != TCP_SKB_CB(to)->end_seq)
|
|
|
|
return false;
|
|
|
|
|
2018-07-13 05:33:38 -06:00
|
|
|
#ifdef CONFIG_TLS_DEVICE
|
|
|
|
if (from->decrypted != to->decrypted)
|
|
|
|
return false;
|
|
|
|
#endif
|
|
|
|
|
2014-09-19 09:26:20 -06:00
|
|
|
if (!skb_try_coalesce(to, from, fragstolen, &delta))
|
|
|
|
return false;
|
|
|
|
|
|
|
|
atomic_add(delta, &sk->sk_rmem_alloc);
|
|
|
|
sk_mem_charge(sk, delta);
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPRCVCOALESCE);
|
2014-09-19 09:26:20 -06:00
|
|
|
TCP_SKB_CB(to)->end_seq = TCP_SKB_CB(from)->end_seq;
|
|
|
|
TCP_SKB_CB(to)->ack_seq = TCP_SKB_CB(from)->ack_seq;
|
|
|
|
TCP_SKB_CB(to)->tcp_flags |= TCP_SKB_CB(from)->tcp_flags;
|
2017-08-22 15:08:48 -06:00
|
|
|
|
|
|
|
if (TCP_SKB_CB(from)->has_rxtstamp) {
|
|
|
|
TCP_SKB_CB(to)->has_rxtstamp = true;
|
2017-09-19 06:14:24 -06:00
|
|
|
to->tstamp = from->tstamp;
|
2018-11-20 01:15:02 -07:00
|
|
|
skb_hwtstamps(to)->hwtstamp = skb_hwtstamps(from)->hwtstamp;
|
2017-08-22 15:08:48 -06:00
|
|
|
}
|
|
|
|
|
2014-09-19 09:26:20 -06:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2018-07-23 10:28:21 -06:00
|
|
|
static bool tcp_ooo_try_coalesce(struct sock *sk,
|
|
|
|
struct sk_buff *to,
|
|
|
|
struct sk_buff *from,
|
|
|
|
bool *fragstolen)
|
|
|
|
{
|
|
|
|
bool res = tcp_try_coalesce(sk, to, from, fragstolen);
|
|
|
|
|
|
|
|
/* In case tcp_drop() is called later, update to->gso_segs */
|
|
|
|
if (res) {
|
|
|
|
u32 gso_segs = max_t(u16, 1, skb_shinfo(to)->gso_segs) +
|
|
|
|
max_t(u16, 1, skb_shinfo(from)->gso_segs);
|
|
|
|
|
|
|
|
skb_shinfo(to)->gso_segs = min_t(u32, gso_segs, 0xFFFF);
|
|
|
|
}
|
|
|
|
return res;
|
|
|
|
}
|
|
|
|
|
2016-04-01 09:52:19 -06:00
|
|
|
static void tcp_drop(struct sock *sk, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
sk_drops_add(sk, skb);
|
|
|
|
__kfree_skb(skb);
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* This one checks to see if we can put data from the
|
|
|
|
* out_of_order queue into the receive_queue.
|
|
|
|
*/
|
|
|
|
static void tcp_ofo_queue(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
__u32 dsack_high = tp->rcv_nxt;
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
bool fin, fragstolen, eaten;
|
2014-09-19 09:26:20 -06:00
|
|
|
struct sk_buff *skb, *tail;
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
struct rb_node *p;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
p = rb_first(&tp->out_of_order_queue);
|
|
|
|
while (p) {
|
2017-10-05 23:21:21 -06:00
|
|
|
skb = rb_to_skb(p);
|
2005-04-16 16:20:36 -06:00
|
|
|
if (after(TCP_SKB_CB(skb)->seq, tp->rcv_nxt))
|
|
|
|
break;
|
|
|
|
|
|
|
|
if (before(TCP_SKB_CB(skb)->seq, dsack_high)) {
|
|
|
|
__u32 dsack = dsack_high;
|
|
|
|
if (before(TCP_SKB_CB(skb)->end_seq, dsack_high))
|
|
|
|
dsack_high = TCP_SKB_CB(skb)->end_seq;
|
2008-07-16 21:29:51 -06:00
|
|
|
tcp_dsack_extend(sk, TCP_SKB_CB(skb)->seq, dsack);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
p = rb_next(p);
|
|
|
|
rb_erase(&skb->rbnode, &tp->out_of_order_queue);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
if (unlikely(!after(TCP_SKB_CB(skb)->end_seq, tp->rcv_nxt))) {
|
2016-04-01 09:52:19 -06:00
|
|
|
tcp_drop(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2014-09-19 09:26:20 -06:00
|
|
|
tail = skb_peek_tail(&sk->sk_receive_queue);
|
2017-09-19 06:14:24 -06:00
|
|
|
eaten = tail && tcp_try_coalesce(sk, tail, skb, &fragstolen);
|
2015-04-28 16:28:18 -06:00
|
|
|
tcp_rcv_nxt_update(tp, TCP_SKB_CB(skb)->end_seq);
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
fin = TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN;
|
2014-09-19 09:26:20 -06:00
|
|
|
if (!eaten)
|
|
|
|
__skb_queue_tail(&sk->sk_receive_queue, skb);
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
else
|
2014-09-19 09:26:20 -06:00
|
|
|
kfree_skb_partial(skb, fragstolen);
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
|
|
|
|
if (unlikely(fin)) {
|
|
|
|
tcp_fin(sk);
|
|
|
|
/* tcp_fin() purges tp->out_of_order_queue,
|
|
|
|
* so we must end this loop right now.
|
|
|
|
*/
|
|
|
|
break;
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-05-16 17:15:34 -06:00
|
|
|
static bool tcp_prune_ofo_queue(struct sock *sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
static int tcp_prune_queue(struct sock *sk);
|
|
|
|
|
netvm: prevent a stream-specific deadlock
This patch series is based on top of "Swap-over-NBD without deadlocking
v15" as it depends on the same reservation of PF_MEMALLOC reserves logic.
When a user or administrator requires swap for their application, they
create a swap partition and file, format it with mkswap and activate it
with swapon. In diskless systems this is not an option so if swap if
required then swapping over the network is considered. The two likely
scenarios are when blade servers are used as part of a cluster where the
form factor or maintenance costs do not allow the use of disks and thin
clients.
The Linux Terminal Server Project recommends the use of the Network Block
Device (NBD) for swap but this is not always an option. There is no
guarantee that the network attached storage (NAS) device is running Linux
or supports NBD. However, it is likely that it supports NFS so there are
users that want support for swapping over NFS despite any performance
concern. Some distributions currently carry patches that support swapping
over NFS but it would be preferable to support it in the mainline kernel.
Patch 1 avoids a stream-specific deadlock that potentially affects TCP.
Patch 2 is a small modification to SELinux to avoid using PFMEMALLOC
reserves.
Patch 3 adds three helpers for filesystems to handle swap cache pages.
For example, page_file_mapping() returns page->mapping for
file-backed pages and the address_space of the underlying
swap file for swap cache pages.
Patch 4 adds two address_space_operations to allow a filesystem
to pin all metadata relevant to a swapfile in memory. Upon
successful activation, the swapfile is marked SWP_FILE and
the address space operation ->direct_IO is used for writing
and ->readpage for reading in swap pages.
Patch 5 notes that patch 3 is bolting
filesystem-specific-swapfile-support onto the side and that
the default handlers have different information to what
is available to the filesystem. This patch refactors the
code so that there are generic handlers for each of the new
address_space operations.
Patch 6 adds an API to allow a vector of kernel addresses to be
translated to struct pages and pinned for IO.
Patch 7 adds support for using highmem pages for swap by kmapping
the pages before calling the direct_IO handler.
Patch 8 updates NFS to use the helpers from patch 3 where necessary.
Patch 9 avoids setting PF_private on PG_swapcache pages within NFS.
Patch 10 implements the new swapfile-related address_space operations
for NFS and teaches the direct IO handler how to manage
kernel addresses.
Patch 11 prevents page allocator recursions in NFS by using GFP_NOIO
where appropriate.
Patch 12 fixes a NULL pointer dereference that occurs when using
swap-over-NFS.
With the patches applied, it is possible to mount a swapfile that is on an
NFS filesystem. Swap performance is not great with a swap stress test
taking roughly twice as long to complete than if the swap device was
backed by NBD.
This patch: netvm: prevent a stream-specific deadlock
It could happen that all !SOCK_MEMALLOC sockets have buffered so much data
that we're over the global rmem limit. This will prevent SOCK_MEMALLOC
buffers from receiving data, which will prevent userspace from running,
which is needed to reduce the buffered data.
Fix this by exempting the SOCK_MEMALLOC sockets from the rmem limit. Once
this change it applied, it is important that sockets that set
SOCK_MEMALLOC do not clear the flag until the socket is being torn down.
If this happens, a warning is generated and the tokens reclaimed to avoid
accounting errors until the bug is fixed.
[davem@davemloft.net: Warning about clearing SOCK_MEMALLOC]
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: David S. Miller <davem@davemloft.net>
Acked-by: Rik van Riel <riel@redhat.com>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Cc: Neil Brown <neilb@suse.de>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Mike Christie <michaelc@cs.wisc.edu>
Cc: Eric B Munson <emunson@mgebm.net>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Cc: Mel Gorman <mgorman@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-07-31 17:44:41 -06:00
|
|
|
static int tcp_try_rmem_schedule(struct sock *sk, struct sk_buff *skb,
|
|
|
|
unsigned int size)
|
2008-04-15 01:33:38 -06:00
|
|
|
{
|
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf ||
|
netvm: prevent a stream-specific deadlock
This patch series is based on top of "Swap-over-NBD without deadlocking
v15" as it depends on the same reservation of PF_MEMALLOC reserves logic.
When a user or administrator requires swap for their application, they
create a swap partition and file, format it with mkswap and activate it
with swapon. In diskless systems this is not an option so if swap if
required then swapping over the network is considered. The two likely
scenarios are when blade servers are used as part of a cluster where the
form factor or maintenance costs do not allow the use of disks and thin
clients.
The Linux Terminal Server Project recommends the use of the Network Block
Device (NBD) for swap but this is not always an option. There is no
guarantee that the network attached storage (NAS) device is running Linux
or supports NBD. However, it is likely that it supports NFS so there are
users that want support for swapping over NFS despite any performance
concern. Some distributions currently carry patches that support swapping
over NFS but it would be preferable to support it in the mainline kernel.
Patch 1 avoids a stream-specific deadlock that potentially affects TCP.
Patch 2 is a small modification to SELinux to avoid using PFMEMALLOC
reserves.
Patch 3 adds three helpers for filesystems to handle swap cache pages.
For example, page_file_mapping() returns page->mapping for
file-backed pages and the address_space of the underlying
swap file for swap cache pages.
Patch 4 adds two address_space_operations to allow a filesystem
to pin all metadata relevant to a swapfile in memory. Upon
successful activation, the swapfile is marked SWP_FILE and
the address space operation ->direct_IO is used for writing
and ->readpage for reading in swap pages.
Patch 5 notes that patch 3 is bolting
filesystem-specific-swapfile-support onto the side and that
the default handlers have different information to what
is available to the filesystem. This patch refactors the
code so that there are generic handlers for each of the new
address_space operations.
Patch 6 adds an API to allow a vector of kernel addresses to be
translated to struct pages and pinned for IO.
Patch 7 adds support for using highmem pages for swap by kmapping
the pages before calling the direct_IO handler.
Patch 8 updates NFS to use the helpers from patch 3 where necessary.
Patch 9 avoids setting PF_private on PG_swapcache pages within NFS.
Patch 10 implements the new swapfile-related address_space operations
for NFS and teaches the direct IO handler how to manage
kernel addresses.
Patch 11 prevents page allocator recursions in NFS by using GFP_NOIO
where appropriate.
Patch 12 fixes a NULL pointer dereference that occurs when using
swap-over-NFS.
With the patches applied, it is possible to mount a swapfile that is on an
NFS filesystem. Swap performance is not great with a swap stress test
taking roughly twice as long to complete than if the swap device was
backed by NBD.
This patch: netvm: prevent a stream-specific deadlock
It could happen that all !SOCK_MEMALLOC sockets have buffered so much data
that we're over the global rmem limit. This will prevent SOCK_MEMALLOC
buffers from receiving data, which will prevent userspace from running,
which is needed to reduce the buffered data.
Fix this by exempting the SOCK_MEMALLOC sockets from the rmem limit. Once
this change it applied, it is important that sockets that set
SOCK_MEMALLOC do not clear the flag until the socket is being torn down.
If this happens, a warning is generated and the tokens reclaimed to avoid
accounting errors until the bug is fixed.
[davem@davemloft.net: Warning about clearing SOCK_MEMALLOC]
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: David S. Miller <davem@davemloft.net>
Acked-by: Rik van Riel <riel@redhat.com>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Cc: Neil Brown <neilb@suse.de>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Mike Christie <michaelc@cs.wisc.edu>
Cc: Eric B Munson <emunson@mgebm.net>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Cc: Mel Gorman <mgorman@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-07-31 17:44:41 -06:00
|
|
|
!sk_rmem_schedule(sk, skb, size)) {
|
2008-04-15 01:33:38 -06:00
|
|
|
|
|
|
|
if (tcp_prune_queue(sk) < 0)
|
|
|
|
return -1;
|
|
|
|
|
tcp: refine tcp_prune_ofo_queue() to not drop all packets
Over the years, TCP BDP has increased a lot, and is typically
in the order of ~10 Mbytes with help of clever Congestion Control
modules.
In presence of packet losses, TCP stores incoming packets into an out of
order queue, and number of skbs sitting there waiting for the missing
packets to be received can match the BDP (~10 Mbytes)
In some cases, TCP needs to make room for incoming skbs, and current
strategy can simply remove all skbs in the out of order queue as a last
resort, incurring a huge penalty, both for receiver and sender.
Unfortunately these 'last resort events' are quite frequent, forcing
sender to send all packets again, stalling the flow and wasting a lot of
resources.
This patch cleans only a part of the out of order queue in order
to meet the memory constraints.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Cc: C. Stephen Gun <csg@google.com>
Cc: Van Jacobson <vanj@google.com>
Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-08-17 15:17:09 -06:00
|
|
|
while (!sk_rmem_schedule(sk, skb, size)) {
|
2008-04-15 21:26:34 -06:00
|
|
|
if (!tcp_prune_ofo_queue(sk))
|
|
|
|
return -1;
|
2008-04-15 01:33:38 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2012-03-18 05:06:44 -06:00
|
|
|
static void tcp_data_queue_ofo(struct sock *sk, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2017-10-05 23:21:21 -06:00
|
|
|
struct rb_node **p, *parent;
|
2012-03-18 05:06:44 -06:00
|
|
|
struct sk_buff *skb1;
|
|
|
|
u32 seq, end_seq;
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
bool fragstolen;
|
2012-03-18 05:06:44 -06:00
|
|
|
|
2018-06-04 16:29:51 -06:00
|
|
|
tcp_ecn_check_ce(sk, skb);
|
2012-03-18 05:06:44 -06:00
|
|
|
|
netvm: prevent a stream-specific deadlock
This patch series is based on top of "Swap-over-NBD without deadlocking
v15" as it depends on the same reservation of PF_MEMALLOC reserves logic.
When a user or administrator requires swap for their application, they
create a swap partition and file, format it with mkswap and activate it
with swapon. In diskless systems this is not an option so if swap if
required then swapping over the network is considered. The two likely
scenarios are when blade servers are used as part of a cluster where the
form factor or maintenance costs do not allow the use of disks and thin
clients.
The Linux Terminal Server Project recommends the use of the Network Block
Device (NBD) for swap but this is not always an option. There is no
guarantee that the network attached storage (NAS) device is running Linux
or supports NBD. However, it is likely that it supports NFS so there are
users that want support for swapping over NFS despite any performance
concern. Some distributions currently carry patches that support swapping
over NFS but it would be preferable to support it in the mainline kernel.
Patch 1 avoids a stream-specific deadlock that potentially affects TCP.
Patch 2 is a small modification to SELinux to avoid using PFMEMALLOC
reserves.
Patch 3 adds three helpers for filesystems to handle swap cache pages.
For example, page_file_mapping() returns page->mapping for
file-backed pages and the address_space of the underlying
swap file for swap cache pages.
Patch 4 adds two address_space_operations to allow a filesystem
to pin all metadata relevant to a swapfile in memory. Upon
successful activation, the swapfile is marked SWP_FILE and
the address space operation ->direct_IO is used for writing
and ->readpage for reading in swap pages.
Patch 5 notes that patch 3 is bolting
filesystem-specific-swapfile-support onto the side and that
the default handlers have different information to what
is available to the filesystem. This patch refactors the
code so that there are generic handlers for each of the new
address_space operations.
Patch 6 adds an API to allow a vector of kernel addresses to be
translated to struct pages and pinned for IO.
Patch 7 adds support for using highmem pages for swap by kmapping
the pages before calling the direct_IO handler.
Patch 8 updates NFS to use the helpers from patch 3 where necessary.
Patch 9 avoids setting PF_private on PG_swapcache pages within NFS.
Patch 10 implements the new swapfile-related address_space operations
for NFS and teaches the direct IO handler how to manage
kernel addresses.
Patch 11 prevents page allocator recursions in NFS by using GFP_NOIO
where appropriate.
Patch 12 fixes a NULL pointer dereference that occurs when using
swap-over-NFS.
With the patches applied, it is possible to mount a swapfile that is on an
NFS filesystem. Swap performance is not great with a swap stress test
taking roughly twice as long to complete than if the swap device was
backed by NBD.
This patch: netvm: prevent a stream-specific deadlock
It could happen that all !SOCK_MEMALLOC sockets have buffered so much data
that we're over the global rmem limit. This will prevent SOCK_MEMALLOC
buffers from receiving data, which will prevent userspace from running,
which is needed to reduce the buffered data.
Fix this by exempting the SOCK_MEMALLOC sockets from the rmem limit. Once
this change it applied, it is important that sockets that set
SOCK_MEMALLOC do not clear the flag until the socket is being torn down.
If this happens, a warning is generated and the tokens reclaimed to avoid
accounting errors until the bug is fixed.
[davem@davemloft.net: Warning about clearing SOCK_MEMALLOC]
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: David S. Miller <davem@davemloft.net>
Acked-by: Rik van Riel <riel@redhat.com>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Cc: Neil Brown <neilb@suse.de>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Mike Christie <michaelc@cs.wisc.edu>
Cc: Eric B Munson <emunson@mgebm.net>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Cc: Mel Gorman <mgorman@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-07-31 17:44:41 -06:00
|
|
|
if (unlikely(tcp_try_rmem_schedule(sk, skb, skb->truesize))) {
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPOFODROP);
|
2016-04-01 09:52:19 -06:00
|
|
|
tcp_drop(sk, skb);
|
2012-03-18 05:06:44 -06:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2017-08-30 11:24:58 -06:00
|
|
|
/* Disable header prediction. */
|
|
|
|
tp->pred_flags = 0;
|
2012-03-18 05:06:44 -06:00
|
|
|
inet_csk_schedule_ack(sk);
|
|
|
|
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPOFOQUEUE);
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
seq = TCP_SKB_CB(skb)->seq;
|
|
|
|
end_seq = TCP_SKB_CB(skb)->end_seq;
|
2012-03-18 05:06:44 -06:00
|
|
|
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
p = &tp->out_of_order_queue.rb_node;
|
|
|
|
if (RB_EMPTY_ROOT(&tp->out_of_order_queue)) {
|
2012-03-18 05:06:44 -06:00
|
|
|
/* Initial out of order segment, build 1 SACK. */
|
|
|
|
if (tcp_is_sack(tp)) {
|
|
|
|
tp->rx_opt.num_sacks = 1;
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
tp->selective_acks[0].start_seq = seq;
|
|
|
|
tp->selective_acks[0].end_seq = end_seq;
|
2012-03-18 05:06:44 -06:00
|
|
|
}
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
rb_link_node(&skb->rbnode, NULL, p);
|
|
|
|
rb_insert_color(&skb->rbnode, &tp->out_of_order_queue);
|
|
|
|
tp->ooo_last_skb = skb;
|
2012-03-18 05:06:44 -06:00
|
|
|
goto end;
|
|
|
|
}
|
|
|
|
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
/* In the typical case, we are adding an skb to the end of the list.
|
|
|
|
* Use of ooo_last_skb avoids the O(Log(N)) rbtree lookup.
|
|
|
|
*/
|
2018-07-23 10:28:21 -06:00
|
|
|
if (tcp_ooo_try_coalesce(sk, tp->ooo_last_skb,
|
|
|
|
skb, &fragstolen)) {
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
coalesce_done:
|
|
|
|
tcp_grow_window(sk, skb);
|
|
|
|
kfree_skb_partial(skb, fragstolen);
|
|
|
|
skb = NULL;
|
|
|
|
goto add_sack;
|
|
|
|
}
|
2016-09-09 15:22:45 -06:00
|
|
|
/* Can avoid an rbtree lookup if we are adding skb after ooo_last_skb */
|
|
|
|
if (!before(seq, TCP_SKB_CB(tp->ooo_last_skb)->end_seq)) {
|
|
|
|
parent = &tp->ooo_last_skb->rbnode;
|
|
|
|
p = &parent->rb_right;
|
|
|
|
goto insert;
|
|
|
|
}
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
|
|
|
|
/* Find place to insert this segment. Handle overlaps on the way. */
|
|
|
|
parent = NULL;
|
|
|
|
while (*p) {
|
|
|
|
parent = *p;
|
2017-10-05 23:21:21 -06:00
|
|
|
skb1 = rb_to_skb(parent);
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
if (before(seq, TCP_SKB_CB(skb1)->seq)) {
|
|
|
|
p = &parent->rb_left;
|
|
|
|
continue;
|
2012-03-18 05:06:44 -06:00
|
|
|
}
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
if (before(seq, TCP_SKB_CB(skb1)->end_seq)) {
|
|
|
|
if (!after(end_seq, TCP_SKB_CB(skb1)->end_seq)) {
|
|
|
|
/* All the bits are present. Drop. */
|
|
|
|
NET_INC_STATS(sock_net(sk),
|
|
|
|
LINUX_MIB_TCPOFOMERGE);
|
2018-07-23 10:28:20 -06:00
|
|
|
tcp_drop(sk, skb);
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
skb = NULL;
|
|
|
|
tcp_dsack_set(sk, seq, end_seq);
|
|
|
|
goto add_sack;
|
|
|
|
}
|
|
|
|
if (after(seq, TCP_SKB_CB(skb1)->seq)) {
|
|
|
|
/* Partial overlap. */
|
|
|
|
tcp_dsack_set(sk, seq, TCP_SKB_CB(skb1)->end_seq);
|
|
|
|
} else {
|
|
|
|
/* skb's seq == skb1's seq and skb covers skb1.
|
|
|
|
* Replace skb1 with skb.
|
|
|
|
*/
|
|
|
|
rb_replace_node(&skb1->rbnode, &skb->rbnode,
|
|
|
|
&tp->out_of_order_queue);
|
|
|
|
tcp_dsack_extend(sk,
|
|
|
|
TCP_SKB_CB(skb1)->seq,
|
|
|
|
TCP_SKB_CB(skb1)->end_seq);
|
|
|
|
NET_INC_STATS(sock_net(sk),
|
|
|
|
LINUX_MIB_TCPOFOMERGE);
|
2018-07-23 10:28:20 -06:00
|
|
|
tcp_drop(sk, skb1);
|
tcp: fix a stale ooo_last_skb after a replace
When skb replaces another one in ooo queue, I forgot to also
update tp->ooo_last_skb as well, if the replaced skb was the last one
in the queue.
To fix this, we simply can re-use the code that runs after an insertion,
trying to merge skbs at the right of current skb.
This not only fixes the bug, but also remove all small skbs that might
be a subset of the new one.
Example:
We receive segments 2001:3001, 4001:5001
Then we receive 2001:8001 : We should replace 2001:3001 with the big
skb, but also remove 4001:50001 from the queue to save space.
packetdrill test demonstrating the bug
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 7>
+0.100 < . 1:1(0) ack 1 win 1024
+0 accept(3, ..., ...) = 4
+0.01 < . 1001:2001(1000) ack 1 win 1024
+0 > . 1:1(0) ack 1 <nop,nop, sack 1001:2001>
+0.01 < . 1001:3001(2000) ack 1 win 1024
+0 > . 1:1(0) ack 1 <nop,nop, sack 1001:2001 1001:3001>
Fixes: 9f5afeae5152 ("tcp: use an RB tree for ooo receive queue")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Yuchung Cheng <ycheng@google.com>
Cc: Yaogong Wang <wygivan@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-13 23:55:05 -06:00
|
|
|
goto merge_right;
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
}
|
2018-07-23 10:28:21 -06:00
|
|
|
} else if (tcp_ooo_try_coalesce(sk, skb1,
|
|
|
|
skb, &fragstolen)) {
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
goto coalesce_done;
|
2012-03-18 05:06:44 -06:00
|
|
|
}
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
p = &parent->rb_right;
|
2012-03-18 05:06:44 -06:00
|
|
|
}
|
2016-09-09 15:22:45 -06:00
|
|
|
insert:
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
/* Insert segment into RB tree. */
|
|
|
|
rb_link_node(&skb->rbnode, parent, p);
|
|
|
|
rb_insert_color(&skb->rbnode, &tp->out_of_order_queue);
|
2012-03-18 05:06:44 -06:00
|
|
|
|
tcp: fix a stale ooo_last_skb after a replace
When skb replaces another one in ooo queue, I forgot to also
update tp->ooo_last_skb as well, if the replaced skb was the last one
in the queue.
To fix this, we simply can re-use the code that runs after an insertion,
trying to merge skbs at the right of current skb.
This not only fixes the bug, but also remove all small skbs that might
be a subset of the new one.
Example:
We receive segments 2001:3001, 4001:5001
Then we receive 2001:8001 : We should replace 2001:3001 with the big
skb, but also remove 4001:50001 from the queue to save space.
packetdrill test demonstrating the bug
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 7>
+0.100 < . 1:1(0) ack 1 win 1024
+0 accept(3, ..., ...) = 4
+0.01 < . 1001:2001(1000) ack 1 win 1024
+0 > . 1:1(0) ack 1 <nop,nop, sack 1001:2001>
+0.01 < . 1001:3001(2000) ack 1 win 1024
+0 > . 1:1(0) ack 1 <nop,nop, sack 1001:2001 1001:3001>
Fixes: 9f5afeae5152 ("tcp: use an RB tree for ooo receive queue")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Yuchung Cheng <ycheng@google.com>
Cc: Yaogong Wang <wygivan@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-13 23:55:05 -06:00
|
|
|
merge_right:
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
/* Remove other segments covered by skb. */
|
2017-10-05 23:21:21 -06:00
|
|
|
while ((skb1 = skb_rb_next(skb)) != NULL) {
|
2012-03-18 05:06:44 -06:00
|
|
|
if (!after(end_seq, TCP_SKB_CB(skb1)->seq))
|
|
|
|
break;
|
|
|
|
if (before(end_seq, TCP_SKB_CB(skb1)->end_seq)) {
|
|
|
|
tcp_dsack_extend(sk, TCP_SKB_CB(skb1)->seq,
|
|
|
|
end_seq);
|
|
|
|
break;
|
|
|
|
}
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
rb_erase(&skb1->rbnode, &tp->out_of_order_queue);
|
2012-03-18 05:06:44 -06:00
|
|
|
tcp_dsack_extend(sk, TCP_SKB_CB(skb1)->seq,
|
|
|
|
TCP_SKB_CB(skb1)->end_seq);
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPOFOMERGE);
|
2016-04-01 09:52:19 -06:00
|
|
|
tcp_drop(sk, skb1);
|
2012-03-18 05:06:44 -06:00
|
|
|
}
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
/* If there is no skb after us, we are the last_skb ! */
|
2017-10-05 23:21:21 -06:00
|
|
|
if (!skb1)
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
tp->ooo_last_skb = skb;
|
2012-03-18 05:06:44 -06:00
|
|
|
|
|
|
|
add_sack:
|
|
|
|
if (tcp_is_sack(tp))
|
|
|
|
tcp_sack_new_ofo_skb(sk, seq, end_seq);
|
|
|
|
end:
|
2013-09-06 11:35:58 -06:00
|
|
|
if (skb) {
|
|
|
|
tcp_grow_window(sk, skb);
|
2017-01-24 15:57:36 -07:00
|
|
|
skb_condense(skb);
|
2012-03-18 05:06:44 -06:00
|
|
|
skb_set_owner_r(skb, sk);
|
2013-09-06 11:35:58 -06:00
|
|
|
}
|
2012-03-18 05:06:44 -06:00
|
|
|
}
|
|
|
|
|
2018-11-26 15:49:12 -07:00
|
|
|
static int __must_check tcp_queue_rcv(struct sock *sk, struct sk_buff *skb,
|
|
|
|
bool *fragstolen)
|
2012-05-02 03:58:29 -06:00
|
|
|
{
|
|
|
|
int eaten;
|
|
|
|
struct sk_buff *tail = skb_peek_tail(&sk->sk_receive_queue);
|
|
|
|
|
|
|
|
eaten = (tail &&
|
2017-09-19 06:14:24 -06:00
|
|
|
tcp_try_coalesce(sk, tail,
|
2017-08-22 15:08:48 -06:00
|
|
|
skb, fragstolen)) ? 1 : 0;
|
2015-04-28 16:28:18 -06:00
|
|
|
tcp_rcv_nxt_update(tcp_sk(sk), TCP_SKB_CB(skb)->end_seq);
|
2012-05-02 03:58:29 -06:00
|
|
|
if (!eaten) {
|
|
|
|
__skb_queue_tail(&sk->sk_receive_queue, skb);
|
|
|
|
skb_set_owner_r(skb, sk);
|
|
|
|
}
|
|
|
|
return eaten;
|
|
|
|
}
|
2012-03-18 05:06:44 -06:00
|
|
|
|
2012-05-09 19:49:41 -06:00
|
|
|
int tcp_send_rcvq(struct sock *sk, struct msghdr *msg, size_t size)
|
|
|
|
{
|
2014-09-17 04:14:42 -06:00
|
|
|
struct sk_buff *skb;
|
2015-11-18 22:03:33 -07:00
|
|
|
int err = -ENOMEM;
|
|
|
|
int data_len = 0;
|
2012-05-09 19:49:41 -06:00
|
|
|
bool fragstolen;
|
|
|
|
|
2012-10-28 23:05:33 -06:00
|
|
|
if (size == 0)
|
|
|
|
return 0;
|
|
|
|
|
2015-11-18 22:03:33 -07:00
|
|
|
if (size > PAGE_SIZE) {
|
|
|
|
int npages = min_t(size_t, size >> PAGE_SHIFT, MAX_SKB_FRAGS);
|
|
|
|
|
|
|
|
data_len = npages << PAGE_SHIFT;
|
|
|
|
size = data_len + (size & ~PAGE_MASK);
|
|
|
|
}
|
|
|
|
skb = alloc_skb_with_frags(size - data_len, data_len,
|
|
|
|
PAGE_ALLOC_COSTLY_ORDER,
|
|
|
|
&err, sk->sk_allocation);
|
2012-05-09 19:49:41 -06:00
|
|
|
if (!skb)
|
|
|
|
goto err;
|
|
|
|
|
2015-11-18 22:03:33 -07:00
|
|
|
skb_put(skb, size - data_len);
|
|
|
|
skb->data_len = data_len;
|
|
|
|
skb->len = size;
|
|
|
|
|
2018-06-27 22:22:56 -06:00
|
|
|
if (tcp_try_rmem_schedule(sk, skb, skb->truesize)) {
|
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPRCVQDROP);
|
netvm: prevent a stream-specific deadlock
This patch series is based on top of "Swap-over-NBD without deadlocking
v15" as it depends on the same reservation of PF_MEMALLOC reserves logic.
When a user or administrator requires swap for their application, they
create a swap partition and file, format it with mkswap and activate it
with swapon. In diskless systems this is not an option so if swap if
required then swapping over the network is considered. The two likely
scenarios are when blade servers are used as part of a cluster where the
form factor or maintenance costs do not allow the use of disks and thin
clients.
The Linux Terminal Server Project recommends the use of the Network Block
Device (NBD) for swap but this is not always an option. There is no
guarantee that the network attached storage (NAS) device is running Linux
or supports NBD. However, it is likely that it supports NFS so there are
users that want support for swapping over NFS despite any performance
concern. Some distributions currently carry patches that support swapping
over NFS but it would be preferable to support it in the mainline kernel.
Patch 1 avoids a stream-specific deadlock that potentially affects TCP.
Patch 2 is a small modification to SELinux to avoid using PFMEMALLOC
reserves.
Patch 3 adds three helpers for filesystems to handle swap cache pages.
For example, page_file_mapping() returns page->mapping for
file-backed pages and the address_space of the underlying
swap file for swap cache pages.
Patch 4 adds two address_space_operations to allow a filesystem
to pin all metadata relevant to a swapfile in memory. Upon
successful activation, the swapfile is marked SWP_FILE and
the address space operation ->direct_IO is used for writing
and ->readpage for reading in swap pages.
Patch 5 notes that patch 3 is bolting
filesystem-specific-swapfile-support onto the side and that
the default handlers have different information to what
is available to the filesystem. This patch refactors the
code so that there are generic handlers for each of the new
address_space operations.
Patch 6 adds an API to allow a vector of kernel addresses to be
translated to struct pages and pinned for IO.
Patch 7 adds support for using highmem pages for swap by kmapping
the pages before calling the direct_IO handler.
Patch 8 updates NFS to use the helpers from patch 3 where necessary.
Patch 9 avoids setting PF_private on PG_swapcache pages within NFS.
Patch 10 implements the new swapfile-related address_space operations
for NFS and teaches the direct IO handler how to manage
kernel addresses.
Patch 11 prevents page allocator recursions in NFS by using GFP_NOIO
where appropriate.
Patch 12 fixes a NULL pointer dereference that occurs when using
swap-over-NFS.
With the patches applied, it is possible to mount a swapfile that is on an
NFS filesystem. Swap performance is not great with a swap stress test
taking roughly twice as long to complete than if the swap device was
backed by NBD.
This patch: netvm: prevent a stream-specific deadlock
It could happen that all !SOCK_MEMALLOC sockets have buffered so much data
that we're over the global rmem limit. This will prevent SOCK_MEMALLOC
buffers from receiving data, which will prevent userspace from running,
which is needed to reduce the buffered data.
Fix this by exempting the SOCK_MEMALLOC sockets from the rmem limit. Once
this change it applied, it is important that sockets that set
SOCK_MEMALLOC do not clear the flag until the socket is being torn down.
If this happens, a warning is generated and the tokens reclaimed to avoid
accounting errors until the bug is fixed.
[davem@davemloft.net: Warning about clearing SOCK_MEMALLOC]
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: David S. Miller <davem@davemloft.net>
Acked-by: Rik van Riel <riel@redhat.com>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Cc: Neil Brown <neilb@suse.de>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Mike Christie <michaelc@cs.wisc.edu>
Cc: Eric B Munson <emunson@mgebm.net>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Cc: Mel Gorman <mgorman@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-07-31 17:44:41 -06:00
|
|
|
goto err_free;
|
2018-06-27 22:22:56 -06:00
|
|
|
}
|
netvm: prevent a stream-specific deadlock
This patch series is based on top of "Swap-over-NBD without deadlocking
v15" as it depends on the same reservation of PF_MEMALLOC reserves logic.
When a user or administrator requires swap for their application, they
create a swap partition and file, format it with mkswap and activate it
with swapon. In diskless systems this is not an option so if swap if
required then swapping over the network is considered. The two likely
scenarios are when blade servers are used as part of a cluster where the
form factor or maintenance costs do not allow the use of disks and thin
clients.
The Linux Terminal Server Project recommends the use of the Network Block
Device (NBD) for swap but this is not always an option. There is no
guarantee that the network attached storage (NAS) device is running Linux
or supports NBD. However, it is likely that it supports NFS so there are
users that want support for swapping over NFS despite any performance
concern. Some distributions currently carry patches that support swapping
over NFS but it would be preferable to support it in the mainline kernel.
Patch 1 avoids a stream-specific deadlock that potentially affects TCP.
Patch 2 is a small modification to SELinux to avoid using PFMEMALLOC
reserves.
Patch 3 adds three helpers for filesystems to handle swap cache pages.
For example, page_file_mapping() returns page->mapping for
file-backed pages and the address_space of the underlying
swap file for swap cache pages.
Patch 4 adds two address_space_operations to allow a filesystem
to pin all metadata relevant to a swapfile in memory. Upon
successful activation, the swapfile is marked SWP_FILE and
the address space operation ->direct_IO is used for writing
and ->readpage for reading in swap pages.
Patch 5 notes that patch 3 is bolting
filesystem-specific-swapfile-support onto the side and that
the default handlers have different information to what
is available to the filesystem. This patch refactors the
code so that there are generic handlers for each of the new
address_space operations.
Patch 6 adds an API to allow a vector of kernel addresses to be
translated to struct pages and pinned for IO.
Patch 7 adds support for using highmem pages for swap by kmapping
the pages before calling the direct_IO handler.
Patch 8 updates NFS to use the helpers from patch 3 where necessary.
Patch 9 avoids setting PF_private on PG_swapcache pages within NFS.
Patch 10 implements the new swapfile-related address_space operations
for NFS and teaches the direct IO handler how to manage
kernel addresses.
Patch 11 prevents page allocator recursions in NFS by using GFP_NOIO
where appropriate.
Patch 12 fixes a NULL pointer dereference that occurs when using
swap-over-NFS.
With the patches applied, it is possible to mount a swapfile that is on an
NFS filesystem. Swap performance is not great with a swap stress test
taking roughly twice as long to complete than if the swap device was
backed by NBD.
This patch: netvm: prevent a stream-specific deadlock
It could happen that all !SOCK_MEMALLOC sockets have buffered so much data
that we're over the global rmem limit. This will prevent SOCK_MEMALLOC
buffers from receiving data, which will prevent userspace from running,
which is needed to reduce the buffered data.
Fix this by exempting the SOCK_MEMALLOC sockets from the rmem limit. Once
this change it applied, it is important that sockets that set
SOCK_MEMALLOC do not clear the flag until the socket is being torn down.
If this happens, a warning is generated and the tokens reclaimed to avoid
accounting errors until the bug is fixed.
[davem@davemloft.net: Warning about clearing SOCK_MEMALLOC]
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: David S. Miller <davem@davemloft.net>
Acked-by: Rik van Riel <riel@redhat.com>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Cc: Neil Brown <neilb@suse.de>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Mike Christie <michaelc@cs.wisc.edu>
Cc: Eric B Munson <emunson@mgebm.net>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Cc: Mel Gorman <mgorman@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-07-31 17:44:41 -06:00
|
|
|
|
2015-11-18 22:03:33 -07:00
|
|
|
err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size);
|
|
|
|
if (err)
|
2012-05-09 19:49:41 -06:00
|
|
|
goto err_free;
|
|
|
|
|
|
|
|
TCP_SKB_CB(skb)->seq = tcp_sk(sk)->rcv_nxt;
|
|
|
|
TCP_SKB_CB(skb)->end_seq = TCP_SKB_CB(skb)->seq + size;
|
|
|
|
TCP_SKB_CB(skb)->ack_seq = tcp_sk(sk)->snd_una - 1;
|
|
|
|
|
2018-11-26 15:49:12 -07:00
|
|
|
if (tcp_queue_rcv(sk, skb, &fragstolen)) {
|
2012-05-09 19:49:41 -06:00
|
|
|
WARN_ON_ONCE(fragstolen); /* should not happen */
|
|
|
|
__kfree_skb(skb);
|
|
|
|
}
|
|
|
|
return size;
|
|
|
|
|
|
|
|
err_free:
|
|
|
|
kfree_skb(skb);
|
|
|
|
err:
|
2015-11-18 22:03:33 -07:00
|
|
|
return err;
|
|
|
|
|
2012-05-09 19:49:41 -06:00
|
|
|
}
|
|
|
|
|
tcp: avoid extra wakeups for SO_RCVLOWAT users
SO_RCVLOWAT is properly handled in tcp_poll(), so that POLLIN is only
generated when enough bytes are available in receive queue, after
David change (commit c7004482e8dc "tcp: Respect SO_RCVLOWAT in tcp_poll().")
But TCP still calls sk->sk_data_ready() for each chunk added in receive
queue, meaning thread is awaken, and goes back to sleep shortly after.
Tested:
tcp_mmap test program, receiving 32768 MB of data with SO_RCVLOWAT set to 512KB
-> Should get ~2 wakeups (c-switches) per MB, regardless of how many
(tiny or big) packets were received.
High speed (mostly full size GRO packets)
received 32768 MB (100 % mmap'ed) in 8.03112 s, 34.2266 Gbit,
cpu usage user:0.037 sys:1.404, 43.9758 usec per MB, 65497 c-switches
received 32768 MB (99.9954 % mmap'ed) in 7.98453 s, 34.4263 Gbit,
cpu usage user:0.03 sys:1.422, 44.3115 usec per MB, 65485 c-switches
Low speed (sender is ratelimited and sends 1-MSS at a time, so GRO is not helping)
received 22474.5 MB (100 % mmap'ed) in 6015.35 s, 0.0313414 Gbit,
cpu usage user:0.05 sys:1.586, 72.7952 usec per MB, 44950 c-switches
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-16 11:33:37 -06:00
|
|
|
void tcp_data_ready(struct sock *sk)
|
|
|
|
{
|
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int avail = tp->rcv_nxt - tp->copied_seq;
|
|
|
|
|
|
|
|
if (avail < sk->sk_rcvlowat && !sock_flag(sk, SOCK_DONE))
|
|
|
|
return;
|
|
|
|
|
|
|
|
sk->sk_data_ready(sk);
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
static void tcp_data_queue(struct sock *sk, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2017-08-01 08:02:44 -06:00
|
|
|
bool fragstolen;
|
|
|
|
int eaten;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2016-04-01 09:52:19 -06:00
|
|
|
if (TCP_SKB_CB(skb)->seq == TCP_SKB_CB(skb)->end_seq) {
|
|
|
|
__kfree_skb(skb);
|
|
|
|
return;
|
|
|
|
}
|
2010-04-28 16:31:51 -06:00
|
|
|
skb_dst_drop(skb);
|
2014-09-24 08:17:02 -06:00
|
|
|
__skb_pull(skb, tcp_hdr(skb)->doff * 4);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
tcp: avoid resetting ACK timer upon receiving packet with ECN CWR flag
Previously commit 9aee40006190 ("tcp: ack immediately when a cwr
packet arrives") calls tcp_enter_quickack_mode to force sending
two immediate ACKs upon receiving a packet w/ CWR flag. The side
effect is it'll also reset the delayed ACK timer and interactive
session tracking. This patch removes that side effect by using the
new ACK_NOW flag to force an immmediate ACK.
Packetdrill to demonstrate:
0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 setsockopt(3, SOL_TCP, TCP_CONGESTION, "dctcp", 5) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < [ect0] SEW 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > SE. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 8>
+.1 < [ect0] . 1:1(0) ack 1 win 257
+0 accept(3, ..., ...) = 4
+0 < [ect0] . 1:1001(1000) ack 1 win 257
+0 > [ect01] . 1:1(0) ack 1001
+0 write(4, ..., 1) = 1
+0 > [ect01] P. 1:2(1) ack 1001
+0 < [ect0] . 1001:2001(1000) ack 2 win 257
+0 write(4, ..., 1) = 1
+0 > [ect01] P. 2:3(1) ack 2001
+0 < [ect0] . 2001:3001(1000) ack 3 win 257
+0 < [ect0] . 3001:4001(1000) ack 3 win 257
// Ack delayed ...
+.01 < [ce] P. 4001:4501(500) ack 3 win 257
+0 > [ect01] . 3:3(0) ack 4001
+0 > [ect01] E. 3:3(0) ack 4501
+.001 read(4, ..., 4500) = 4500
+0 write(4, ..., 1) = 1
+0 > [ect01] PE. 3:4(1) ack 4501 win 100
+.01 < [ect0] W. 4501:5501(1000) ack 4 win 257
// No delayed ACK on CWR flag
+0 > [ect01] . 4:4(0) ack 5501
+.31 < [ect0] . 5501:6501(1000) ack 4 win 257
+0 > [ect01] . 4:4(0) ack 6501
Fixes: 9aee40006190 ("tcp: ack immediately when a cwr packet arrives")
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-08-09 10:38:12 -06:00
|
|
|
tcp_ecn_accept_cwr(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2009-03-14 08:23:01 -06:00
|
|
|
tp->rx_opt.dsack = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Queue data for delivery to the user.
|
|
|
|
* Packets in sequence go to the receive queue.
|
|
|
|
* Out of sequence packets to the out_of_order_queue.
|
|
|
|
*/
|
|
|
|
if (TCP_SKB_CB(skb)->seq == tp->rcv_nxt) {
|
2018-06-24 08:02:54 -06:00
|
|
|
if (tcp_receive_window(tp) == 0) {
|
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPZEROWINDOWDROP);
|
2005-04-16 16:20:36 -06:00
|
|
|
goto out_of_window;
|
2018-06-24 08:02:54 -06:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Ok. In sequence. In window. */
|
|
|
|
queue_and_out:
|
2017-08-01 08:02:44 -06:00
|
|
|
if (skb_queue_len(&sk->sk_receive_queue) == 0)
|
|
|
|
sk_forced_mem_schedule(sk, skb->truesize);
|
2018-06-27 22:22:56 -06:00
|
|
|
else if (tcp_try_rmem_schedule(sk, skb, skb->truesize)) {
|
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPRCVQDROP);
|
2017-08-01 08:02:44 -06:00
|
|
|
goto drop;
|
2018-06-27 22:22:56 -06:00
|
|
|
}
|
2017-08-01 08:02:44 -06:00
|
|
|
|
2018-11-26 15:49:12 -07:00
|
|
|
eaten = tcp_queue_rcv(sk, skb, &fragstolen);
|
2007-03-08 21:45:19 -07:00
|
|
|
if (skb->len)
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
tcp_event_data_recv(sk, skb);
|
2014-09-24 08:17:02 -06:00
|
|
|
if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
|
2011-10-20 15:44:03 -06:00
|
|
|
tcp_fin(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
if (!RB_EMPTY_ROOT(&tp->out_of_order_queue)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
tcp_ofo_queue(sk);
|
|
|
|
|
2018-08-09 10:38:11 -06:00
|
|
|
/* RFC5681. 4.2. SHOULD send immediate ACK, when
|
2005-04-16 16:20:36 -06:00
|
|
|
* gap in queue is filled.
|
|
|
|
*/
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
if (RB_EMPTY_ROOT(&tp->out_of_order_queue))
|
2018-08-09 10:38:11 -06:00
|
|
|
inet_csk(sk)->icsk_ack.pending |= ICSK_ACK_NOW;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
if (tp->rx_opt.num_sacks)
|
|
|
|
tcp_sack_remove(tp);
|
|
|
|
|
2017-08-30 11:24:58 -06:00
|
|
|
tcp_fast_path_check(sk);
|
|
|
|
|
2012-05-02 01:55:58 -06:00
|
|
|
if (eaten > 0)
|
|
|
|
kfree_skb_partial(skb, fragstolen);
|
2012-09-17 06:51:39 -06:00
|
|
|
if (!sock_flag(sk, SOCK_DEAD))
|
tcp: avoid extra wakeups for SO_RCVLOWAT users
SO_RCVLOWAT is properly handled in tcp_poll(), so that POLLIN is only
generated when enough bytes are available in receive queue, after
David change (commit c7004482e8dc "tcp: Respect SO_RCVLOWAT in tcp_poll().")
But TCP still calls sk->sk_data_ready() for each chunk added in receive
queue, meaning thread is awaken, and goes back to sleep shortly after.
Tested:
tcp_mmap test program, receiving 32768 MB of data with SO_RCVLOWAT set to 512KB
-> Should get ~2 wakeups (c-switches) per MB, regardless of how many
(tiny or big) packets were received.
High speed (mostly full size GRO packets)
received 32768 MB (100 % mmap'ed) in 8.03112 s, 34.2266 Gbit,
cpu usage user:0.037 sys:1.404, 43.9758 usec per MB, 65497 c-switches
received 32768 MB (99.9954 % mmap'ed) in 7.98453 s, 34.4263 Gbit,
cpu usage user:0.03 sys:1.422, 44.3115 usec per MB, 65485 c-switches
Low speed (sender is ratelimited and sends 1-MSS at a time, so GRO is not helping)
received 22474.5 MB (100 % mmap'ed) in 6015.35 s, 0.0313414 Gbit,
cpu usage user:0.05 sys:1.586, 72.7952 usec per MB, 44950 c-switches
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-16 11:33:37 -06:00
|
|
|
tcp_data_ready(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!after(TCP_SKB_CB(skb)->end_seq, tp->rcv_nxt)) {
|
2018-08-29 15:53:56 -06:00
|
|
|
tcp_rcv_spurious_retrans(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
/* A retransmit, 2nd most common case. Force an immediate ack. */
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_DELAYEDACKLOST);
|
2008-07-16 21:29:51 -06:00
|
|
|
tcp_dsack_set(sk, TCP_SKB_CB(skb)->seq, TCP_SKB_CB(skb)->end_seq);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
out_of_window:
|
2018-05-21 16:08:56 -06:00
|
|
|
tcp_enter_quickack_mode(sk, TCP_MAX_QUICKACKS);
|
2005-08-09 21:10:42 -06:00
|
|
|
inet_csk_schedule_ack(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
drop:
|
2016-04-01 09:52:19 -06:00
|
|
|
tcp_drop(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Out of window. F.e. zero window probe. */
|
|
|
|
if (!before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt + tcp_receive_window(tp)))
|
|
|
|
goto out_of_window;
|
|
|
|
|
|
|
|
if (before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt)) {
|
|
|
|
/* Partial packet, seq < rcv_next < end_seq */
|
2008-07-16 21:29:51 -06:00
|
|
|
tcp_dsack_set(sk, TCP_SKB_CB(skb)->seq, tp->rcv_nxt);
|
2007-02-09 07:24:47 -07:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* If window is closed, drop tail of packet. But after
|
|
|
|
* remembering D-SACK for its head made in previous line.
|
|
|
|
*/
|
2018-06-24 08:02:54 -06:00
|
|
|
if (!tcp_receive_window(tp)) {
|
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPZEROWINDOWDROP);
|
2005-04-16 16:20:36 -06:00
|
|
|
goto out_of_window;
|
2018-06-24 08:02:54 -06:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
goto queue_and_out;
|
|
|
|
}
|
|
|
|
|
2012-03-18 05:06:44 -06:00
|
|
|
tcp_data_queue_ofo(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
static struct sk_buff *tcp_skb_next(struct sk_buff *skb, struct sk_buff_head *list)
|
|
|
|
{
|
|
|
|
if (list)
|
|
|
|
return !skb_queue_is_last(list, skb) ? skb->next : NULL;
|
|
|
|
|
2017-10-05 23:21:21 -06:00
|
|
|
return skb_rb_next(skb);
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
}
|
|
|
|
|
2008-08-23 06:11:41 -06:00
|
|
|
static struct sk_buff *tcp_collapse_one(struct sock *sk, struct sk_buff *skb,
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
struct sk_buff_head *list,
|
|
|
|
struct rb_root *root)
|
2008-08-23 06:11:41 -06:00
|
|
|
{
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
struct sk_buff *next = tcp_skb_next(skb, list);
|
2009-05-28 22:35:47 -06:00
|
|
|
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
if (list)
|
|
|
|
__skb_unlink(skb, list);
|
|
|
|
else
|
|
|
|
rb_erase(&skb->rbnode, root);
|
2008-08-23 06:11:41 -06:00
|
|
|
|
|
|
|
__kfree_skb(skb);
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPRCVCOLLAPSED);
|
2008-08-23 06:11:41 -06:00
|
|
|
|
|
|
|
return next;
|
|
|
|
}
|
|
|
|
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
/* Insert skb into rb tree, ordered by TCP_SKB_CB(skb)->seq */
|
2017-10-05 23:21:27 -06:00
|
|
|
void tcp_rbtree_insert(struct rb_root *root, struct sk_buff *skb)
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
{
|
|
|
|
struct rb_node **p = &root->rb_node;
|
|
|
|
struct rb_node *parent = NULL;
|
|
|
|
struct sk_buff *skb1;
|
|
|
|
|
|
|
|
while (*p) {
|
|
|
|
parent = *p;
|
2017-10-05 23:21:21 -06:00
|
|
|
skb1 = rb_to_skb(parent);
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
if (before(TCP_SKB_CB(skb)->seq, TCP_SKB_CB(skb1)->seq))
|
|
|
|
p = &parent->rb_left;
|
|
|
|
else
|
|
|
|
p = &parent->rb_right;
|
|
|
|
}
|
|
|
|
rb_link_node(&skb->rbnode, parent, p);
|
|
|
|
rb_insert_color(&skb->rbnode, root);
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Collapse contiguous sequence of skbs head..tail with
|
|
|
|
* sequence numbers start..end.
|
2009-05-28 22:35:47 -06:00
|
|
|
*
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
* If tail is NULL, this means until the end of the queue.
|
2009-05-28 22:35:47 -06:00
|
|
|
*
|
2005-04-16 16:20:36 -06:00
|
|
|
* Segments with FIN/SYN are not collapsed (only because this
|
|
|
|
* simplifies code)
|
|
|
|
*/
|
|
|
|
static void
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
|
|
|
|
struct sk_buff *head, struct sk_buff *tail, u32 start, u32 end)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
struct sk_buff *skb = head, *n;
|
|
|
|
struct sk_buff_head tmp;
|
2009-05-28 22:35:47 -06:00
|
|
|
bool end_of_skbs;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2005-11-10 18:13:47 -07:00
|
|
|
/* First, check that queue is collapsible and find
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
* the point where collapsing can be useful.
|
|
|
|
*/
|
2009-05-28 22:35:47 -06:00
|
|
|
restart:
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
for (end_of_skbs = true; skb != NULL && skb != tail; skb = n) {
|
|
|
|
n = tcp_skb_next(skb, list);
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* No new bits? It is possible on ofo queue. */
|
|
|
|
if (!before(start, TCP_SKB_CB(skb)->end_seq)) {
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
skb = tcp_collapse_one(sk, skb, list, root);
|
2009-05-28 22:35:47 -06:00
|
|
|
if (!skb)
|
|
|
|
break;
|
|
|
|
goto restart;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* The first skb to collapse is:
|
|
|
|
* - not SYN/FIN and
|
|
|
|
* - bloated or contains data before "start" or
|
|
|
|
* overlaps to the next one.
|
|
|
|
*/
|
2014-09-15 05:19:51 -06:00
|
|
|
if (!(TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN)) &&
|
2017-10-26 22:55:09 -06:00
|
|
|
(tcp_win_from_space(sk, skb->truesize) > skb->len ||
|
2009-05-28 22:35:47 -06:00
|
|
|
before(TCP_SKB_CB(skb)->seq, start))) {
|
|
|
|
end_of_skbs = false;
|
2005-04-16 16:20:36 -06:00
|
|
|
break;
|
2009-05-28 22:35:47 -06:00
|
|
|
}
|
|
|
|
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
if (n && n != tail &&
|
|
|
|
TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(n)->seq) {
|
|
|
|
end_of_skbs = false;
|
|
|
|
break;
|
2009-05-28 22:35:47 -06:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Decided to skip this, advance start seq. */
|
|
|
|
start = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
}
|
2014-09-15 05:19:51 -06:00
|
|
|
if (end_of_skbs ||
|
|
|
|
(TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN)))
|
2005-04-16 16:20:36 -06:00
|
|
|
return;
|
|
|
|
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
__skb_queue_head_init(&tmp);
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
while (before(start, end)) {
|
2014-09-15 05:19:53 -06:00
|
|
|
int copy = min_t(int, SKB_MAX_ORDER(0, 0), end - start);
|
2005-04-16 16:20:36 -06:00
|
|
|
struct sk_buff *nskb;
|
|
|
|
|
2014-09-15 05:19:53 -06:00
|
|
|
nskb = alloc_skb(copy, GFP_ATOMIC);
|
2005-04-16 16:20:36 -06:00
|
|
|
if (!nskb)
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
break;
|
2007-03-10 08:47:22 -07:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
memcpy(nskb->cb, skb->cb, sizeof(skb->cb));
|
2018-07-13 05:33:38 -06:00
|
|
|
#ifdef CONFIG_TLS_DEVICE
|
|
|
|
nskb->decrypted = skb->decrypted;
|
|
|
|
#endif
|
2005-04-16 16:20:36 -06:00
|
|
|
TCP_SKB_CB(nskb)->seq = TCP_SKB_CB(nskb)->end_seq = start;
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
if (list)
|
|
|
|
__skb_queue_before(list, skb, nskb);
|
|
|
|
else
|
|
|
|
__skb_queue_tail(&tmp, nskb); /* defer rbtree insertion */
|
2007-12-31 01:11:19 -07:00
|
|
|
skb_set_owner_r(nskb, sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Copy data, releasing collapsed skbs. */
|
|
|
|
while (copy > 0) {
|
|
|
|
int offset = start - TCP_SKB_CB(skb)->seq;
|
|
|
|
int size = TCP_SKB_CB(skb)->end_seq - start;
|
|
|
|
|
2006-01-08 23:24:28 -07:00
|
|
|
BUG_ON(offset < 0);
|
2005-04-16 16:20:36 -06:00
|
|
|
if (size > 0) {
|
|
|
|
size = min(copy, size);
|
|
|
|
if (skb_copy_bits(skb, offset, skb_put(nskb, size), size))
|
|
|
|
BUG();
|
|
|
|
TCP_SKB_CB(nskb)->end_seq += size;
|
|
|
|
copy -= size;
|
|
|
|
start += size;
|
|
|
|
}
|
|
|
|
if (!before(start, TCP_SKB_CB(skb)->end_seq)) {
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
skb = tcp_collapse_one(sk, skb, list, root);
|
2009-05-28 22:35:47 -06:00
|
|
|
if (!skb ||
|
|
|
|
skb == tail ||
|
2014-09-15 05:19:51 -06:00
|
|
|
(TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN)))
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
goto end;
|
2018-07-13 05:33:38 -06:00
|
|
|
#ifdef CONFIG_TLS_DEVICE
|
|
|
|
if (skb->decrypted != nskb->decrypted)
|
|
|
|
goto end;
|
|
|
|
#endif
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
end:
|
|
|
|
skb_queue_walk_safe(&tmp, skb, n)
|
|
|
|
tcp_rbtree_insert(root, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Collapse ofo queue. Algorithm: select contiguous sequence of skbs
|
|
|
|
* and tcp_collapse() them until all the queue is collapsed.
|
|
|
|
*/
|
|
|
|
static void tcp_collapse_ofo_queue(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2018-07-23 10:28:19 -06:00
|
|
|
u32 range_truesize, sum_tiny = 0;
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
struct sk_buff *skb, *head;
|
2005-04-16 16:20:36 -06:00
|
|
|
u32 start, end;
|
|
|
|
|
2017-10-05 23:21:21 -06:00
|
|
|
skb = skb_rb_first(&tp->out_of_order_queue);
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
new_range:
|
|
|
|
if (!skb) {
|
2017-10-05 23:21:21 -06:00
|
|
|
tp->ooo_last_skb = skb_rb_last(&tp->out_of_order_queue);
|
2005-04-16 16:20:36 -06:00
|
|
|
return;
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
start = TCP_SKB_CB(skb)->seq;
|
|
|
|
end = TCP_SKB_CB(skb)->end_seq;
|
2018-07-23 10:28:19 -06:00
|
|
|
range_truesize = skb->truesize;
|
2009-05-28 22:35:47 -06:00
|
|
|
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
for (head = skb;;) {
|
2017-10-05 23:21:21 -06:00
|
|
|
skb = skb_rb_next(skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
/* Range is terminated when we see a gap or when
|
|
|
|
* we are at the queue end.
|
|
|
|
*/
|
2009-05-28 22:35:47 -06:00
|
|
|
if (!skb ||
|
2005-04-16 16:20:36 -06:00
|
|
|
after(TCP_SKB_CB(skb)->seq, end) ||
|
|
|
|
before(TCP_SKB_CB(skb)->end_seq, start)) {
|
2018-07-23 10:28:19 -06:00
|
|
|
/* Do not attempt collapsing tiny skbs */
|
|
|
|
if (range_truesize != head->truesize ||
|
|
|
|
end - start >= SKB_WITH_OVERHEAD(SK_MEM_QUANTUM)) {
|
|
|
|
tcp_collapse(sk, NULL, &tp->out_of_order_queue,
|
|
|
|
head, skb, start, end);
|
|
|
|
} else {
|
|
|
|
sum_tiny += range_truesize;
|
|
|
|
if (sum_tiny > sk->sk_rcvbuf >> 3)
|
|
|
|
return;
|
|
|
|
}
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
goto new_range;
|
|
|
|
}
|
|
|
|
|
2018-07-23 10:28:19 -06:00
|
|
|
range_truesize += skb->truesize;
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
if (unlikely(before(TCP_SKB_CB(skb)->seq, start)))
|
2005-04-16 16:20:36 -06:00
|
|
|
start = TCP_SKB_CB(skb)->seq;
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
if (after(TCP_SKB_CB(skb)->end_seq, end))
|
2005-04-16 16:20:36 -06:00
|
|
|
end = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-04-15 01:33:38 -06:00
|
|
|
/*
|
tcp: refine tcp_prune_ofo_queue() to not drop all packets
Over the years, TCP BDP has increased a lot, and is typically
in the order of ~10 Mbytes with help of clever Congestion Control
modules.
In presence of packet losses, TCP stores incoming packets into an out of
order queue, and number of skbs sitting there waiting for the missing
packets to be received can match the BDP (~10 Mbytes)
In some cases, TCP needs to make room for incoming skbs, and current
strategy can simply remove all skbs in the out of order queue as a last
resort, incurring a huge penalty, both for receiver and sender.
Unfortunately these 'last resort events' are quite frequent, forcing
sender to send all packets again, stalling the flow and wasting a lot of
resources.
This patch cleans only a part of the out of order queue in order
to meet the memory constraints.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Cc: C. Stephen Gun <csg@google.com>
Cc: Van Jacobson <vanj@google.com>
Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-08-17 15:17:09 -06:00
|
|
|
* Clean the out-of-order queue to make room.
|
|
|
|
* We drop high sequences packets to :
|
|
|
|
* 1) Let a chance for holes to be filled.
|
|
|
|
* 2) not add too big latencies if thousands of packets sit there.
|
|
|
|
* (But if application shrinks SO_RCVBUF, we could still end up
|
|
|
|
* freeing whole queue here)
|
2018-07-23 10:28:17 -06:00
|
|
|
* 3) Drop at least 12.5 % of sk_rcvbuf to avoid malicious attacks.
|
tcp: refine tcp_prune_ofo_queue() to not drop all packets
Over the years, TCP BDP has increased a lot, and is typically
in the order of ~10 Mbytes with help of clever Congestion Control
modules.
In presence of packet losses, TCP stores incoming packets into an out of
order queue, and number of skbs sitting there waiting for the missing
packets to be received can match the BDP (~10 Mbytes)
In some cases, TCP needs to make room for incoming skbs, and current
strategy can simply remove all skbs in the out of order queue as a last
resort, incurring a huge penalty, both for receiver and sender.
Unfortunately these 'last resort events' are quite frequent, forcing
sender to send all packets again, stalling the flow and wasting a lot of
resources.
This patch cleans only a part of the out of order queue in order
to meet the memory constraints.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Cc: C. Stephen Gun <csg@google.com>
Cc: Van Jacobson <vanj@google.com>
Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-08-17 15:17:09 -06:00
|
|
|
*
|
|
|
|
* Return true if queue has shrunk.
|
2008-04-15 01:33:38 -06:00
|
|
|
*/
|
2012-05-16 17:15:34 -06:00
|
|
|
static bool tcp_prune_ofo_queue(struct sock *sk)
|
2008-04-15 01:33:38 -06:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
struct rb_node *node, *prev;
|
2018-07-23 10:28:17 -06:00
|
|
|
int goal;
|
2008-04-15 01:33:38 -06:00
|
|
|
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
if (RB_EMPTY_ROOT(&tp->out_of_order_queue))
|
tcp: refine tcp_prune_ofo_queue() to not drop all packets
Over the years, TCP BDP has increased a lot, and is typically
in the order of ~10 Mbytes with help of clever Congestion Control
modules.
In presence of packet losses, TCP stores incoming packets into an out of
order queue, and number of skbs sitting there waiting for the missing
packets to be received can match the BDP (~10 Mbytes)
In some cases, TCP needs to make room for incoming skbs, and current
strategy can simply remove all skbs in the out of order queue as a last
resort, incurring a huge penalty, both for receiver and sender.
Unfortunately these 'last resort events' are quite frequent, forcing
sender to send all packets again, stalling the flow and wasting a lot of
resources.
This patch cleans only a part of the out of order queue in order
to meet the memory constraints.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Cc: C. Stephen Gun <csg@google.com>
Cc: Van Jacobson <vanj@google.com>
Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-08-17 15:17:09 -06:00
|
|
|
return false;
|
2008-04-15 01:33:38 -06:00
|
|
|
|
tcp: refine tcp_prune_ofo_queue() to not drop all packets
Over the years, TCP BDP has increased a lot, and is typically
in the order of ~10 Mbytes with help of clever Congestion Control
modules.
In presence of packet losses, TCP stores incoming packets into an out of
order queue, and number of skbs sitting there waiting for the missing
packets to be received can match the BDP (~10 Mbytes)
In some cases, TCP needs to make room for incoming skbs, and current
strategy can simply remove all skbs in the out of order queue as a last
resort, incurring a huge penalty, both for receiver and sender.
Unfortunately these 'last resort events' are quite frequent, forcing
sender to send all packets again, stalling the flow and wasting a lot of
resources.
This patch cleans only a part of the out of order queue in order
to meet the memory constraints.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Cc: C. Stephen Gun <csg@google.com>
Cc: Van Jacobson <vanj@google.com>
Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-08-17 15:17:09 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_OFOPRUNED);
|
2018-07-23 10:28:17 -06:00
|
|
|
goal = sk->sk_rcvbuf >> 3;
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
node = &tp->ooo_last_skb->rbnode;
|
|
|
|
do {
|
|
|
|
prev = rb_prev(node);
|
|
|
|
rb_erase(node, &tp->out_of_order_queue);
|
2018-07-23 10:28:17 -06:00
|
|
|
goal -= rb_to_skb(node)->truesize;
|
2017-10-05 23:21:21 -06:00
|
|
|
tcp_drop(sk, rb_to_skb(node));
|
2018-07-23 10:28:17 -06:00
|
|
|
if (!prev || goal <= 0) {
|
|
|
|
sk_mem_reclaim(sk);
|
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) <= sk->sk_rcvbuf &&
|
|
|
|
!tcp_under_memory_pressure(sk))
|
|
|
|
break;
|
|
|
|
goal = sk->sk_rcvbuf >> 3;
|
|
|
|
}
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
node = prev;
|
|
|
|
} while (node);
|
2017-10-05 23:21:21 -06:00
|
|
|
tp->ooo_last_skb = rb_to_skb(prev);
|
tcp: refine tcp_prune_ofo_queue() to not drop all packets
Over the years, TCP BDP has increased a lot, and is typically
in the order of ~10 Mbytes with help of clever Congestion Control
modules.
In presence of packet losses, TCP stores incoming packets into an out of
order queue, and number of skbs sitting there waiting for the missing
packets to be received can match the BDP (~10 Mbytes)
In some cases, TCP needs to make room for incoming skbs, and current
strategy can simply remove all skbs in the out of order queue as a last
resort, incurring a huge penalty, both for receiver and sender.
Unfortunately these 'last resort events' are quite frequent, forcing
sender to send all packets again, stalling the flow and wasting a lot of
resources.
This patch cleans only a part of the out of order queue in order
to meet the memory constraints.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Cc: C. Stephen Gun <csg@google.com>
Cc: Van Jacobson <vanj@google.com>
Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-08-17 15:17:09 -06:00
|
|
|
|
|
|
|
/* Reset SACK state. A conforming SACK implementation will
|
|
|
|
* do the same at a timeout based retransmit. When a connection
|
|
|
|
* is in a sad state like this, we care only about integrity
|
|
|
|
* of the connection not performance.
|
|
|
|
*/
|
|
|
|
if (tp->rx_opt.sack_ok)
|
|
|
|
tcp_sack_reset(&tp->rx_opt);
|
|
|
|
return true;
|
2008-04-15 01:33:38 -06:00
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Reduce allocated memory if we can, trying to get
|
|
|
|
* the socket within its memory limits again.
|
|
|
|
*
|
|
|
|
* Return less than zero if we should start dropping frames
|
|
|
|
* until the socket owning process reads some of the data
|
|
|
|
* to stabilize the situation.
|
|
|
|
*/
|
|
|
|
static int tcp_prune_queue(struct sock *sk)
|
|
|
|
{
|
2007-02-09 07:24:47 -07:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_PRUNECALLED);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) >= sk->sk_rcvbuf)
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
tcp_clamp_window(sk);
|
2015-05-15 13:39:27 -06:00
|
|
|
else if (tcp_under_memory_pressure(sk))
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->rcv_ssthresh = min(tp->rcv_ssthresh, 4U * tp->advmss);
|
|
|
|
|
2018-07-23 10:28:18 -06:00
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) <= sk->sk_rcvbuf)
|
|
|
|
return 0;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
tcp_collapse_ofo_queue(sk);
|
2009-05-28 22:35:47 -06:00
|
|
|
if (!skb_queue_empty(&sk->sk_receive_queue))
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-07 15:49:28 -06:00
|
|
|
tcp_collapse(sk, &sk->sk_receive_queue, NULL,
|
2009-05-28 22:35:47 -06:00
|
|
|
skb_peek(&sk->sk_receive_queue),
|
|
|
|
NULL,
|
|
|
|
tp->copied_seq, tp->rcv_nxt);
|
2007-12-31 01:11:19 -07:00
|
|
|
sk_mem_reclaim(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) <= sk->sk_rcvbuf)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/* Collapsing did not help, destructive actions follow.
|
|
|
|
* This must not ever occur. */
|
|
|
|
|
2008-04-15 01:33:38 -06:00
|
|
|
tcp_prune_ofo_queue(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) <= sk->sk_rcvbuf)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/* If we are really being abused, tell the caller to silently
|
|
|
|
* drop receive data on the floor. It will get retransmitted
|
|
|
|
* and hopefully then we'll have sufficient space.
|
|
|
|
*/
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_RCVPRUNED);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Massive buffer overcommit. */
|
2017-08-30 11:24:58 -06:00
|
|
|
tp->pred_flags = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
2012-05-16 17:15:34 -06:00
|
|
|
static bool tcp_should_expand_sndbuf(const struct sock *sk)
|
2005-07-05 16:21:10 -06:00
|
|
|
{
|
2011-10-21 03:22:42 -06:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
|
2005-07-05 16:21:10 -06:00
|
|
|
/* If the user specified a specific send buffer setting, do
|
|
|
|
* not modify it.
|
|
|
|
*/
|
|
|
|
if (sk->sk_userlocks & SOCK_SNDBUF_LOCK)
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2005-07-05 16:21:10 -06:00
|
|
|
|
|
|
|
/* If we are under global TCP memory pressure, do not expand. */
|
2015-05-15 13:39:27 -06:00
|
|
|
if (tcp_under_memory_pressure(sk))
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2005-07-05 16:21:10 -06:00
|
|
|
|
|
|
|
/* If we are under soft global TCP memory pressure, do not expand. */
|
2011-12-11 14:47:02 -07:00
|
|
|
if (sk_memory_allocated(sk) >= sk_prot_mem_limits(sk, 0))
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2005-07-05 16:21:10 -06:00
|
|
|
|
|
|
|
/* If we filled the congestion window, do not expand. */
|
2015-02-20 11:33:16 -07:00
|
|
|
if (tcp_packets_in_flight(tp) >= tp->snd_cwnd)
|
2012-05-16 17:15:34 -06:00
|
|
|
return false;
|
2005-07-05 16:21:10 -06:00
|
|
|
|
2012-05-16 17:15:34 -06:00
|
|
|
return true;
|
2005-07-05 16:21:10 -06:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* When incoming ACK allowed to free some skb from write_queue,
|
|
|
|
* we remember this event in flag SOCK_QUEUE_SHRUNK and wake up socket
|
|
|
|
* on the exit from tcp input handler.
|
|
|
|
*
|
|
|
|
* PROBLEM: sndbuf expansion does not work well with largesend.
|
|
|
|
*/
|
|
|
|
static void tcp_new_space(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
if (tcp_should_expand_sndbuf(sk)) {
|
2013-10-01 11:23:44 -06:00
|
|
|
tcp_sndbuf_expand(sk);
|
2017-05-16 15:00:04 -06:00
|
|
|
tp->snd_cwnd_stamp = tcp_jiffies32;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
sk->sk_write_space(sk);
|
|
|
|
}
|
|
|
|
|
2006-01-03 17:03:49 -07:00
|
|
|
static void tcp_check_space(struct sock *sk)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
if (sock_flag(sk, SOCK_QUEUE_SHRUNK)) {
|
|
|
|
sock_reset_flag(sk, SOCK_QUEUE_SHRUNK);
|
2015-04-20 14:05:07 -06:00
|
|
|
/* pairs with tcp_poll() */
|
2017-01-24 19:49:41 -07:00
|
|
|
smp_mb();
|
2005-04-16 16:20:36 -06:00
|
|
|
if (sk->sk_socket &&
|
2016-11-28 00:07:16 -07:00
|
|
|
test_bit(SOCK_NOSPACE, &sk->sk_socket->flags)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
tcp_new_space(sk);
|
2016-11-28 00:07:16 -07:00
|
|
|
if (!test_bit(SOCK_NOSPACE, &sk->sk_socket->flags))
|
|
|
|
tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED);
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
static inline void tcp_data_snd_check(struct sock *sk)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
tcp_push_pending_frames(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
tcp_check_space(sk);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Check if sending an ack is needed.
|
|
|
|
*/
|
|
|
|
static void __tcp_ack_snd_check(struct sock *sk, int ofo_possible)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
tcp: add SACK compression
When TCP receives an out-of-order packet, it immediately sends
a SACK packet, generating network load but also forcing the
receiver to send 1-MSS pathological packets, increasing its
RTX queue length/depth, and thus processing time.
Wifi networks suffer from this aggressive behavior, but generally
speaking, all these SACK packets add fuel to the fire when networks
are under congestion.
This patch adds a high resolution timer and tp->compressed_ack counter.
Instead of sending a SACK, we program this timer with a small delay,
based on RTT and capped to 1 ms :
delay = min ( 5 % of RTT, 1 ms)
If subsequent SACKs need to be sent while the timer has not yet
expired, we simply increment tp->compressed_ack.
When timer expires, a SACK is sent with the latest information.
Whenever an ACK is sent (if data is sent, or if in-order
data is received) timer is canceled.
Note that tcp_sack_new_ofo_skb() is able to force a SACK to be sent
if the sack blocks need to be shuffled, even if the timer has not
expired.
A new SNMP counter is added in the following patch.
Two other patches add sysctls to allow changing the 1,000,000 and 44
values that this commit hard-coded.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Toke Høiland-Jørgensen <toke@toke.dk>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-05-17 15:47:26 -06:00
|
|
|
unsigned long rtt, delay;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* More than one full frame received... */
|
2009-11-23 11:41:23 -07:00
|
|
|
if (((tp->rcv_nxt - tp->rcv_wup) > inet_csk(sk)->icsk_ack.rcv_mss &&
|
2005-04-16 16:20:36 -06:00
|
|
|
/* ... and right edge of window advances far enough.
|
2018-04-16 11:33:36 -06:00
|
|
|
* (tcp_recvmsg() will send ACK otherwise).
|
|
|
|
* If application uses SO_RCVLOWAT, we want send ack now if
|
|
|
|
* we have not received enough bytes to satisfy the condition.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
2018-04-16 11:33:36 -06:00
|
|
|
(tp->rcv_nxt - tp->copied_seq < sk->sk_rcvlowat ||
|
|
|
|
__tcp_select_window(sk) >= tp->rcv_wnd)) ||
|
2005-04-16 16:20:36 -06:00
|
|
|
/* We ACK each frame or... */
|
2018-08-09 10:38:09 -06:00
|
|
|
tcp_in_quickack_mode(sk) ||
|
|
|
|
/* Protocol state mandates a one-time immediate ACK */
|
|
|
|
inet_csk(sk)->icsk_ack.pending & ICSK_ACK_NOW) {
|
tcp: add SACK compression
When TCP receives an out-of-order packet, it immediately sends
a SACK packet, generating network load but also forcing the
receiver to send 1-MSS pathological packets, increasing its
RTX queue length/depth, and thus processing time.
Wifi networks suffer from this aggressive behavior, but generally
speaking, all these SACK packets add fuel to the fire when networks
are under congestion.
This patch adds a high resolution timer and tp->compressed_ack counter.
Instead of sending a SACK, we program this timer with a small delay,
based on RTT and capped to 1 ms :
delay = min ( 5 % of RTT, 1 ms)
If subsequent SACKs need to be sent while the timer has not yet
expired, we simply increment tp->compressed_ack.
When timer expires, a SACK is sent with the latest information.
Whenever an ACK is sent (if data is sent, or if in-order
data is received) timer is canceled.
Note that tcp_sack_new_ofo_skb() is able to force a SACK to be sent
if the sack blocks need to be shuffled, even if the timer has not
expired.
A new SNMP counter is added in the following patch.
Two other patches add sysctls to allow changing the 1,000,000 and 44
values that this commit hard-coded.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Toke Høiland-Jørgensen <toke@toke.dk>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-05-17 15:47:26 -06:00
|
|
|
send_now:
|
2005-04-16 16:20:36 -06:00
|
|
|
tcp_send_ack(sk);
|
tcp: add SACK compression
When TCP receives an out-of-order packet, it immediately sends
a SACK packet, generating network load but also forcing the
receiver to send 1-MSS pathological packets, increasing its
RTX queue length/depth, and thus processing time.
Wifi networks suffer from this aggressive behavior, but generally
speaking, all these SACK packets add fuel to the fire when networks
are under congestion.
This patch adds a high resolution timer and tp->compressed_ack counter.
Instead of sending a SACK, we program this timer with a small delay,
based on RTT and capped to 1 ms :
delay = min ( 5 % of RTT, 1 ms)
If subsequent SACKs need to be sent while the timer has not yet
expired, we simply increment tp->compressed_ack.
When timer expires, a SACK is sent with the latest information.
Whenever an ACK is sent (if data is sent, or if in-order
data is received) timer is canceled.
Note that tcp_sack_new_ofo_skb() is able to force a SACK to be sent
if the sack blocks need to be shuffled, even if the timer has not
expired.
A new SNMP counter is added in the following patch.
Two other patches add sysctls to allow changing the 1,000,000 and 44
values that this commit hard-coded.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Toke Høiland-Jørgensen <toke@toke.dk>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-05-17 15:47:26 -06:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!ofo_possible || RB_EMPTY_ROOT(&tp->out_of_order_queue)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
tcp_send_delayed_ack(sk);
|
tcp: add SACK compression
When TCP receives an out-of-order packet, it immediately sends
a SACK packet, generating network load but also forcing the
receiver to send 1-MSS pathological packets, increasing its
RTX queue length/depth, and thus processing time.
Wifi networks suffer from this aggressive behavior, but generally
speaking, all these SACK packets add fuel to the fire when networks
are under congestion.
This patch adds a high resolution timer and tp->compressed_ack counter.
Instead of sending a SACK, we program this timer with a small delay,
based on RTT and capped to 1 ms :
delay = min ( 5 % of RTT, 1 ms)
If subsequent SACKs need to be sent while the timer has not yet
expired, we simply increment tp->compressed_ack.
When timer expires, a SACK is sent with the latest information.
Whenever an ACK is sent (if data is sent, or if in-order
data is received) timer is canceled.
Note that tcp_sack_new_ofo_skb() is able to force a SACK to be sent
if the sack blocks need to be shuffled, even if the timer has not
expired.
A new SNMP counter is added in the following patch.
Two other patches add sysctls to allow changing the 1,000,000 and 44
values that this commit hard-coded.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Toke Høiland-Jørgensen <toke@toke.dk>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-05-17 15:47:26 -06:00
|
|
|
return;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
tcp: add SACK compression
When TCP receives an out-of-order packet, it immediately sends
a SACK packet, generating network load but also forcing the
receiver to send 1-MSS pathological packets, increasing its
RTX queue length/depth, and thus processing time.
Wifi networks suffer from this aggressive behavior, but generally
speaking, all these SACK packets add fuel to the fire when networks
are under congestion.
This patch adds a high resolution timer and tp->compressed_ack counter.
Instead of sending a SACK, we program this timer with a small delay,
based on RTT and capped to 1 ms :
delay = min ( 5 % of RTT, 1 ms)
If subsequent SACKs need to be sent while the timer has not yet
expired, we simply increment tp->compressed_ack.
When timer expires, a SACK is sent with the latest information.
Whenever an ACK is sent (if data is sent, or if in-order
data is received) timer is canceled.
Note that tcp_sack_new_ofo_skb() is able to force a SACK to be sent
if the sack blocks need to be shuffled, even if the timer has not
expired.
A new SNMP counter is added in the following patch.
Two other patches add sysctls to allow changing the 1,000,000 and 44
values that this commit hard-coded.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Toke Høiland-Jørgensen <toke@toke.dk>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-05-17 15:47:26 -06:00
|
|
|
|
2018-05-17 15:47:29 -06:00
|
|
|
if (!tcp_is_sack(tp) ||
|
|
|
|
tp->compressed_ack >= sock_net(sk)->ipv4.sysctl_tcp_comp_sack_nr)
|
tcp: add SACK compression
When TCP receives an out-of-order packet, it immediately sends
a SACK packet, generating network load but also forcing the
receiver to send 1-MSS pathological packets, increasing its
RTX queue length/depth, and thus processing time.
Wifi networks suffer from this aggressive behavior, but generally
speaking, all these SACK packets add fuel to the fire when networks
are under congestion.
This patch adds a high resolution timer and tp->compressed_ack counter.
Instead of sending a SACK, we program this timer with a small delay,
based on RTT and capped to 1 ms :
delay = min ( 5 % of RTT, 1 ms)
If subsequent SACKs need to be sent while the timer has not yet
expired, we simply increment tp->compressed_ack.
When timer expires, a SACK is sent with the latest information.
Whenever an ACK is sent (if data is sent, or if in-order
data is received) timer is canceled.
Note that tcp_sack_new_ofo_skb() is able to force a SACK to be sent
if the sack blocks need to be shuffled, even if the timer has not
expired.
A new SNMP counter is added in the following patch.
Two other patches add sysctls to allow changing the 1,000,000 and 44
values that this commit hard-coded.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Toke Høiland-Jørgensen <toke@toke.dk>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-05-17 15:47:26 -06:00
|
|
|
goto send_now;
|
2018-11-20 06:53:59 -07:00
|
|
|
|
|
|
|
if (tp->compressed_ack_rcv_nxt != tp->rcv_nxt) {
|
|
|
|
tp->compressed_ack_rcv_nxt = tp->rcv_nxt;
|
|
|
|
if (tp->compressed_ack > TCP_FASTRETRANS_THRESH)
|
|
|
|
NET_ADD_STATS(sock_net(sk), LINUX_MIB_TCPACKCOMPRESSED,
|
|
|
|
tp->compressed_ack - TCP_FASTRETRANS_THRESH);
|
|
|
|
tp->compressed_ack = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (++tp->compressed_ack <= TCP_FASTRETRANS_THRESH)
|
|
|
|
goto send_now;
|
tcp: add SACK compression
When TCP receives an out-of-order packet, it immediately sends
a SACK packet, generating network load but also forcing the
receiver to send 1-MSS pathological packets, increasing its
RTX queue length/depth, and thus processing time.
Wifi networks suffer from this aggressive behavior, but generally
speaking, all these SACK packets add fuel to the fire when networks
are under congestion.
This patch adds a high resolution timer and tp->compressed_ack counter.
Instead of sending a SACK, we program this timer with a small delay,
based on RTT and capped to 1 ms :
delay = min ( 5 % of RTT, 1 ms)
If subsequent SACKs need to be sent while the timer has not yet
expired, we simply increment tp->compressed_ack.
When timer expires, a SACK is sent with the latest information.
Whenever an ACK is sent (if data is sent, or if in-order
data is received) timer is canceled.
Note that tcp_sack_new_ofo_skb() is able to force a SACK to be sent
if the sack blocks need to be shuffled, even if the timer has not
expired.
A new SNMP counter is added in the following patch.
Two other patches add sysctls to allow changing the 1,000,000 and 44
values that this commit hard-coded.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Toke Høiland-Jørgensen <toke@toke.dk>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-05-17 15:47:26 -06:00
|
|
|
|
|
|
|
if (hrtimer_is_queued(&tp->compressed_ack_timer))
|
|
|
|
return;
|
|
|
|
|
2018-05-17 15:47:28 -06:00
|
|
|
/* compress ack timer : 5 % of rtt, but no more than tcp_comp_sack_delay_ns */
|
tcp: add SACK compression
When TCP receives an out-of-order packet, it immediately sends
a SACK packet, generating network load but also forcing the
receiver to send 1-MSS pathological packets, increasing its
RTX queue length/depth, and thus processing time.
Wifi networks suffer from this aggressive behavior, but generally
speaking, all these SACK packets add fuel to the fire when networks
are under congestion.
This patch adds a high resolution timer and tp->compressed_ack counter.
Instead of sending a SACK, we program this timer with a small delay,
based on RTT and capped to 1 ms :
delay = min ( 5 % of RTT, 1 ms)
If subsequent SACKs need to be sent while the timer has not yet
expired, we simply increment tp->compressed_ack.
When timer expires, a SACK is sent with the latest information.
Whenever an ACK is sent (if data is sent, or if in-order
data is received) timer is canceled.
Note that tcp_sack_new_ofo_skb() is able to force a SACK to be sent
if the sack blocks need to be shuffled, even if the timer has not
expired.
A new SNMP counter is added in the following patch.
Two other patches add sysctls to allow changing the 1,000,000 and 44
values that this commit hard-coded.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Toke Høiland-Jørgensen <toke@toke.dk>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-05-17 15:47:26 -06:00
|
|
|
|
|
|
|
rtt = tp->rcv_rtt_est.rtt_us;
|
|
|
|
if (tp->srtt_us && tp->srtt_us < rtt)
|
|
|
|
rtt = tp->srtt_us;
|
|
|
|
|
2018-05-17 15:47:28 -06:00
|
|
|
delay = min_t(unsigned long, sock_net(sk)->ipv4.sysctl_tcp_comp_sack_delay_ns,
|
tcp: add SACK compression
When TCP receives an out-of-order packet, it immediately sends
a SACK packet, generating network load but also forcing the
receiver to send 1-MSS pathological packets, increasing its
RTX queue length/depth, and thus processing time.
Wifi networks suffer from this aggressive behavior, but generally
speaking, all these SACK packets add fuel to the fire when networks
are under congestion.
This patch adds a high resolution timer and tp->compressed_ack counter.
Instead of sending a SACK, we program this timer with a small delay,
based on RTT and capped to 1 ms :
delay = min ( 5 % of RTT, 1 ms)
If subsequent SACKs need to be sent while the timer has not yet
expired, we simply increment tp->compressed_ack.
When timer expires, a SACK is sent with the latest information.
Whenever an ACK is sent (if data is sent, or if in-order
data is received) timer is canceled.
Note that tcp_sack_new_ofo_skb() is able to force a SACK to be sent
if the sack blocks need to be shuffled, even if the timer has not
expired.
A new SNMP counter is added in the following patch.
Two other patches add sysctls to allow changing the 1,000,000 and 44
values that this commit hard-coded.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Toke Høiland-Jørgensen <toke@toke.dk>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-05-17 15:47:26 -06:00
|
|
|
rtt * (NSEC_PER_USEC >> 3)/20);
|
|
|
|
sock_hold(sk);
|
|
|
|
hrtimer_start(&tp->compressed_ack_timer, ns_to_ktime(delay),
|
|
|
|
HRTIMER_MODE_REL_PINNED_SOFT);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2006-01-03 17:03:49 -07:00
|
|
|
static inline void tcp_ack_snd_check(struct sock *sk)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-08-09 21:10:42 -06:00
|
|
|
if (!inet_csk_ack_scheduled(sk)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
/* We sent a data segment already. */
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
__tcp_ack_snd_check(sk, 1);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This routine is only called when we have urgent data
|
2005-11-10 18:13:47 -07:00
|
|
|
* signaled. Its the 'slow' part of tcp_urg. It could be
|
2005-04-16 16:20:36 -06:00
|
|
|
* moved inline now as tcp_urg is only called from one
|
|
|
|
* place. We handle URGent data wrong. We have to - as
|
|
|
|
* BSD still doesn't use the correction from RFC961.
|
|
|
|
* For 1003.1g we should support a new option TCP_STDURG to permit
|
|
|
|
* either form (or just set the sysctl tcp_stdurg).
|
|
|
|
*/
|
2007-02-09 07:24:47 -07:00
|
|
|
|
2011-10-21 03:22:42 -06:00
|
|
|
static void tcp_check_urg(struct sock *sk, const struct tcphdr *th)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
u32 ptr = ntohs(th->urg_ptr);
|
|
|
|
|
2017-10-26 22:55:01 -06:00
|
|
|
if (ptr && !sock_net(sk)->ipv4.sysctl_tcp_stdurg)
|
2005-04-16 16:20:36 -06:00
|
|
|
ptr--;
|
|
|
|
ptr += ntohl(th->seq);
|
|
|
|
|
|
|
|
/* Ignore urgent data that we've already seen and read. */
|
|
|
|
if (after(tp->copied_seq, ptr))
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Do not replay urg ptr.
|
|
|
|
*
|
|
|
|
* NOTE: interesting situation not covered by specs.
|
|
|
|
* Misbehaving sender may send urg ptr, pointing to segment,
|
|
|
|
* which we already have in ofo queue. We are not able to fetch
|
|
|
|
* such data and will stay in TCP_URG_NOTYET until will be eaten
|
|
|
|
* by recvmsg(). Seems, we are not obliged to handle such wicked
|
|
|
|
* situations. But it is worth to think about possibility of some
|
|
|
|
* DoSes using some hypothetical application level deadlock.
|
|
|
|
*/
|
|
|
|
if (before(ptr, tp->rcv_nxt))
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Do we already have a newer (or duplicate) urgent pointer? */
|
|
|
|
if (tp->urg_data && !after(ptr, tp->urg_seq))
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Tell the world about our new urgent pointer. */
|
|
|
|
sk_send_sigurg(sk);
|
|
|
|
|
|
|
|
/* We may be adding urgent data when the last byte read was
|
|
|
|
* urgent. To do this requires some care. We cannot just ignore
|
|
|
|
* tp->copied_seq since we would read the last urgent byte again
|
|
|
|
* as data, nor can we alter copied_seq until this data arrives
|
2005-11-10 18:13:47 -07:00
|
|
|
* or we break the semantics of SIOCATMARK (and thus sockatmark())
|
2005-04-16 16:20:36 -06:00
|
|
|
*
|
|
|
|
* NOTE. Double Dutch. Rendering to plain English: author of comment
|
|
|
|
* above did something sort of send("A", MSG_OOB); send("B", MSG_OOB);
|
|
|
|
* and expect that both A and B disappear from stream. This is _wrong_.
|
|
|
|
* Though this happens in BSD with high probability, this is occasional.
|
|
|
|
* Any application relying on this is buggy. Note also, that fix "works"
|
|
|
|
* only in this artificial test. Insert some normal data between A and B and we will
|
|
|
|
* decline of BSD again. Verdict: it is better to remove to trap
|
|
|
|
* buggy users.
|
|
|
|
*/
|
|
|
|
if (tp->urg_seq == tp->copied_seq && tp->urg_data &&
|
2007-12-31 15:57:14 -07:00
|
|
|
!sock_flag(sk, SOCK_URGINLINE) && tp->copied_seq != tp->rcv_nxt) {
|
2005-04-16 16:20:36 -06:00
|
|
|
struct sk_buff *skb = skb_peek(&sk->sk_receive_queue);
|
|
|
|
tp->copied_seq++;
|
|
|
|
if (skb && !before(tp->copied_seq, TCP_SKB_CB(skb)->end_seq)) {
|
2005-08-09 20:25:21 -06:00
|
|
|
__skb_unlink(skb, &sk->sk_receive_queue);
|
2005-04-16 16:20:36 -06:00
|
|
|
__kfree_skb(skb);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2007-12-31 15:57:14 -07:00
|
|
|
tp->urg_data = TCP_URG_NOTYET;
|
|
|
|
tp->urg_seq = ptr;
|
2017-08-30 11:24:58 -06:00
|
|
|
|
|
|
|
/* Disable header prediction. */
|
|
|
|
tp->pred_flags = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* This is the 'fast' part of urgent handling. */
|
2011-10-21 03:22:42 -06:00
|
|
|
static void tcp_urg(struct sock *sk, struct sk_buff *skb, const struct tcphdr *th)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
/* Check if we get a new urgent pointer - normally not. */
|
|
|
|
if (th->urg)
|
2007-12-31 15:57:14 -07:00
|
|
|
tcp_check_urg(sk, th);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Do we wait for any urgent data? - normally not... */
|
|
|
|
if (tp->urg_data == TCP_URG_NOTYET) {
|
|
|
|
u32 ptr = tp->urg_seq - ntohl(th->seq) + (th->doff * 4) -
|
|
|
|
th->syn;
|
|
|
|
|
2007-02-09 07:24:47 -07:00
|
|
|
/* Is the urgent pointer pointing into this packet? */
|
2005-04-16 16:20:36 -06:00
|
|
|
if (ptr < skb->len) {
|
|
|
|
u8 tmp;
|
|
|
|
if (skb_copy_bits(skb, ptr, &tmp, 1))
|
|
|
|
BUG();
|
|
|
|
tp->urg_data = TCP_URG_VALID | tmp;
|
|
|
|
if (!sock_flag(sk, SOCK_DEAD))
|
2014-04-11 14:15:36 -06:00
|
|
|
sk->sk_data_ready(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
tcp: accept RST for rcv_nxt - 1 after receiving a FIN
Using a Mac OSX box as a client connecting to a Linux server, we have found
that when certain applications (such as 'ab'), are abruptly terminated
(via ^C), a FIN is sent followed by a RST packet on tcp connections. The
FIN is accepted by the Linux stack but the RST is sent with the same
sequence number as the FIN, and Linux responds with a challenge ACK per
RFC 5961. The OSX client then sometimes (they are rate-limited) does not
reply with any RST as would be expected on a closed socket.
This results in sockets accumulating on the Linux server left mostly in
the CLOSE_WAIT state, although LAST_ACK and CLOSING are also possible.
This sequence of events can tie up a lot of resources on the Linux server
since there may be a lot of data in write buffers at the time of the RST.
Accepting a RST equal to rcv_nxt - 1, after we have already successfully
processed a FIN, has made a significant difference for us in practice, by
freeing up unneeded resources in a more expedient fashion.
A packetdrill test demonstrating the behavior:
// testing mac osx rst behavior
// Establish a connection
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < S 0:0(0) win 32768 <mss 1460,nop,wscale 10>
0.100 > S. 0:0(0) ack 1 <mss 1460,nop,wscale 5>
0.200 < . 1:1(0) ack 1 win 32768
0.200 accept(3, ..., ...) = 4
// Client closes the connection
0.300 < F. 1:1(0) ack 1 win 32768
// now send rst with same sequence
0.300 < R. 1:1(0) ack 1 win 32768
// make sure we are in TCP_CLOSE
0.400 %{
assert tcpi_state == 7
}%
Signed-off-by: Jason Baron <jbaron@akamai.com>
Cc: Eric Dumazet <edumazet@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-01-17 11:37:19 -07:00
|
|
|
/* Accept RST for rcv_nxt - 1 after a FIN.
|
|
|
|
* When tcp connections are abruptly terminated from Mac OSX (via ^C), a
|
|
|
|
* FIN is sent followed by a RST packet. The RST is sent with the same
|
|
|
|
* sequence number as the FIN, and thus according to RFC 5961 a challenge
|
|
|
|
* ACK should be sent. However, Mac OSX rate limits replies to challenge
|
|
|
|
* ACKs on the closed socket. In addition middleboxes can drop either the
|
|
|
|
* challenge ACK or a subsequent RST.
|
|
|
|
*/
|
|
|
|
static bool tcp_reset_check(const struct sock *sk, const struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
return unlikely(TCP_SKB_CB(skb)->seq == (tp->rcv_nxt - 1) &&
|
|
|
|
(1 << sk->sk_state) & (TCPF_CLOSE_WAIT | TCPF_LAST_ACK |
|
|
|
|
TCPF_CLOSING));
|
|
|
|
}
|
|
|
|
|
2008-08-23 06:10:12 -06:00
|
|
|
/* Does PAWS and seqno based validation of an incoming segment, flags will
|
|
|
|
* play significant role here.
|
|
|
|
*/
|
2012-07-16 19:41:30 -06:00
|
|
|
static bool tcp_validate_incoming(struct sock *sk, struct sk_buff *skb,
|
|
|
|
const struct tcphdr *th, int syn_inerr)
|
2008-08-23 06:10:12 -06:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
tcp: accept RST if SEQ matches right edge of right-most SACK block
RFC 5961 advises to only accept RST packets containing a seq number
matching the next expected seq number instead of the whole receive
window in order to avoid spoofing attacks.
However, this situation is not optimal in the case SACK is in use at the
time the RST is sent. I recently run into a scenario in which packet
losses were high while uploading data to a server, and userspace was
willing to frequently terminate connections by sending a RST. In
this case, the ACK sent on the receiver side (rcv_nxt) is frozen waiting
for a lost packet retransmission and SACK blocks are used to let the
client continue uploading data. At some point later on, the client sends
the RST (snd_nxt), which matches the next expected seq number of the
right-most SACK block on the receiver side which is going forward
receiving data.
In this scenario, as RFC 5961 defines, the RST SEQ doesn't match the
frozen main ACK at receiver side and thus gets dropped and a challenge
ACK is sent, which gets usually lost due to network conditions. The main
consequence is that the connection stays alive for a while even if it
made sense to accept the RST. This can get really bad if lots of
connections like this one are created in few seconds, allocating all the
resources of the server easily.
For security reasons, not all SACK blocks are checked (there could be a
big amount of SACK blocks => acceptable SEQ numbers). Furthermore, it
wouldn't make sense to check for RST in blocks other than the right-most
received one because the sender is not expected to be sending new data
after the RST. For simplicity, only up to the 4 most recently updated
SACK blocks (selective_acks[4] field) are compared to find the
right-most block, as usually those are the ones with bigger probability
to contain it.
This patch was tested in a 3.18 kernel and probed to improve the
situation in the scenario described above.
Signed-off-by: Pau Espin Pedrol <pau.espin@tessares.net>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Tested-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-06-07 08:30:34 -06:00
|
|
|
bool rst_seq_match = false;
|
2008-08-23 06:10:12 -06:00
|
|
|
|
|
|
|
/* RFC1323: H1. Apply PAWS check first. */
|
2017-06-07 11:34:36 -06:00
|
|
|
if (tcp_fast_parse_options(sock_net(sk), skb, th, tp) &&
|
|
|
|
tp->rx_opt.saw_tstamp &&
|
2008-08-23 06:10:12 -06:00
|
|
|
tcp_paws_discard(sk, skb)) {
|
|
|
|
if (!th->rst) {
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_PAWSESTABREJECTED);
|
2015-02-06 14:04:40 -07:00
|
|
|
if (!tcp_oow_rate_limited(sock_net(sk), skb,
|
|
|
|
LINUX_MIB_TCPACKSKIPPEDPAWS,
|
|
|
|
&tp->last_oow_ack_time))
|
|
|
|
tcp_send_dupack(sk, skb);
|
2008-08-23 06:10:12 -06:00
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
/* Reset is accepted even if it did not pass PAWS. */
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Step 1: check sequence number */
|
|
|
|
if (!tcp_sequence(tp, TCP_SKB_CB(skb)->seq, TCP_SKB_CB(skb)->end_seq)) {
|
|
|
|
/* RFC793, page 37: "In all states except SYN-SENT, all reset
|
|
|
|
* (RST) segments are validated by checking their SEQ-fields."
|
|
|
|
* And page 69: "If an incoming segment is not acceptable,
|
|
|
|
* an acknowledgment should be sent in reply (unless the RST
|
|
|
|
* bit is set, if so drop the segment and return)".
|
|
|
|
*/
|
2012-07-17 06:29:30 -06:00
|
|
|
if (!th->rst) {
|
|
|
|
if (th->syn)
|
|
|
|
goto syn_challenge;
|
2015-02-06 14:04:40 -07:00
|
|
|
if (!tcp_oow_rate_limited(sock_net(sk), skb,
|
|
|
|
LINUX_MIB_TCPACKSKIPPEDSEQ,
|
|
|
|
&tp->last_oow_ack_time))
|
|
|
|
tcp_send_dupack(sk, skb);
|
tcp: accept RST for rcv_nxt - 1 after receiving a FIN
Using a Mac OSX box as a client connecting to a Linux server, we have found
that when certain applications (such as 'ab'), are abruptly terminated
(via ^C), a FIN is sent followed by a RST packet on tcp connections. The
FIN is accepted by the Linux stack but the RST is sent with the same
sequence number as the FIN, and Linux responds with a challenge ACK per
RFC 5961. The OSX client then sometimes (they are rate-limited) does not
reply with any RST as would be expected on a closed socket.
This results in sockets accumulating on the Linux server left mostly in
the CLOSE_WAIT state, although LAST_ACK and CLOSING are also possible.
This sequence of events can tie up a lot of resources on the Linux server
since there may be a lot of data in write buffers at the time of the RST.
Accepting a RST equal to rcv_nxt - 1, after we have already successfully
processed a FIN, has made a significant difference for us in practice, by
freeing up unneeded resources in a more expedient fashion.
A packetdrill test demonstrating the behavior:
// testing mac osx rst behavior
// Establish a connection
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < S 0:0(0) win 32768 <mss 1460,nop,wscale 10>
0.100 > S. 0:0(0) ack 1 <mss 1460,nop,wscale 5>
0.200 < . 1:1(0) ack 1 win 32768
0.200 accept(3, ..., ...) = 4
// Client closes the connection
0.300 < F. 1:1(0) ack 1 win 32768
// now send rst with same sequence
0.300 < R. 1:1(0) ack 1 win 32768
// make sure we are in TCP_CLOSE
0.400 %{
assert tcpi_state == 7
}%
Signed-off-by: Jason Baron <jbaron@akamai.com>
Cc: Eric Dumazet <edumazet@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-01-17 11:37:19 -07:00
|
|
|
} else if (tcp_reset_check(sk, skb)) {
|
|
|
|
tcp_reset(sk);
|
2012-07-17 06:29:30 -06:00
|
|
|
}
|
2008-08-23 06:10:12 -06:00
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Step 2: check RST bit */
|
|
|
|
if (th->rst) {
|
tcp: accept RST for rcv_nxt - 1 after receiving a FIN
Using a Mac OSX box as a client connecting to a Linux server, we have found
that when certain applications (such as 'ab'), are abruptly terminated
(via ^C), a FIN is sent followed by a RST packet on tcp connections. The
FIN is accepted by the Linux stack but the RST is sent with the same
sequence number as the FIN, and Linux responds with a challenge ACK per
RFC 5961. The OSX client then sometimes (they are rate-limited) does not
reply with any RST as would be expected on a closed socket.
This results in sockets accumulating on the Linux server left mostly in
the CLOSE_WAIT state, although LAST_ACK and CLOSING are also possible.
This sequence of events can tie up a lot of resources on the Linux server
since there may be a lot of data in write buffers at the time of the RST.
Accepting a RST equal to rcv_nxt - 1, after we have already successfully
processed a FIN, has made a significant difference for us in practice, by
freeing up unneeded resources in a more expedient fashion.
A packetdrill test demonstrating the behavior:
// testing mac osx rst behavior
// Establish a connection
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < S 0:0(0) win 32768 <mss 1460,nop,wscale 10>
0.100 > S. 0:0(0) ack 1 <mss 1460,nop,wscale 5>
0.200 < . 1:1(0) ack 1 win 32768
0.200 accept(3, ..., ...) = 4
// Client closes the connection
0.300 < F. 1:1(0) ack 1 win 32768
// now send rst with same sequence
0.300 < R. 1:1(0) ack 1 win 32768
// make sure we are in TCP_CLOSE
0.400 %{
assert tcpi_state == 7
}%
Signed-off-by: Jason Baron <jbaron@akamai.com>
Cc: Eric Dumazet <edumazet@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-01-17 11:37:19 -07:00
|
|
|
/* RFC 5961 3.2 (extend to match against (RCV.NXT - 1) after a
|
|
|
|
* FIN and SACK too if available):
|
|
|
|
* If seq num matches RCV.NXT or (RCV.NXT - 1) after a FIN, or
|
|
|
|
* the right-most SACK block,
|
tcp: accept RST if SEQ matches right edge of right-most SACK block
RFC 5961 advises to only accept RST packets containing a seq number
matching the next expected seq number instead of the whole receive
window in order to avoid spoofing attacks.
However, this situation is not optimal in the case SACK is in use at the
time the RST is sent. I recently run into a scenario in which packet
losses were high while uploading data to a server, and userspace was
willing to frequently terminate connections by sending a RST. In
this case, the ACK sent on the receiver side (rcv_nxt) is frozen waiting
for a lost packet retransmission and SACK blocks are used to let the
client continue uploading data. At some point later on, the client sends
the RST (snd_nxt), which matches the next expected seq number of the
right-most SACK block on the receiver side which is going forward
receiving data.
In this scenario, as RFC 5961 defines, the RST SEQ doesn't match the
frozen main ACK at receiver side and thus gets dropped and a challenge
ACK is sent, which gets usually lost due to network conditions. The main
consequence is that the connection stays alive for a while even if it
made sense to accept the RST. This can get really bad if lots of
connections like this one are created in few seconds, allocating all the
resources of the server easily.
For security reasons, not all SACK blocks are checked (there could be a
big amount of SACK blocks => acceptable SEQ numbers). Furthermore, it
wouldn't make sense to check for RST in blocks other than the right-most
received one because the sender is not expected to be sending new data
after the RST. For simplicity, only up to the 4 most recently updated
SACK blocks (selective_acks[4] field) are compared to find the
right-most block, as usually those are the ones with bigger probability
to contain it.
This patch was tested in a 3.18 kernel and probed to improve the
situation in the scenario described above.
Signed-off-by: Pau Espin Pedrol <pau.espin@tessares.net>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Tested-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-06-07 08:30:34 -06:00
|
|
|
* then
|
2012-07-17 02:13:05 -06:00
|
|
|
* RESET the connection
|
|
|
|
* else
|
|
|
|
* Send a challenge ACK
|
|
|
|
*/
|
tcp: accept RST for rcv_nxt - 1 after receiving a FIN
Using a Mac OSX box as a client connecting to a Linux server, we have found
that when certain applications (such as 'ab'), are abruptly terminated
(via ^C), a FIN is sent followed by a RST packet on tcp connections. The
FIN is accepted by the Linux stack but the RST is sent with the same
sequence number as the FIN, and Linux responds with a challenge ACK per
RFC 5961. The OSX client then sometimes (they are rate-limited) does not
reply with any RST as would be expected on a closed socket.
This results in sockets accumulating on the Linux server left mostly in
the CLOSE_WAIT state, although LAST_ACK and CLOSING are also possible.
This sequence of events can tie up a lot of resources on the Linux server
since there may be a lot of data in write buffers at the time of the RST.
Accepting a RST equal to rcv_nxt - 1, after we have already successfully
processed a FIN, has made a significant difference for us in practice, by
freeing up unneeded resources in a more expedient fashion.
A packetdrill test demonstrating the behavior:
// testing mac osx rst behavior
// Establish a connection
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < S 0:0(0) win 32768 <mss 1460,nop,wscale 10>
0.100 > S. 0:0(0) ack 1 <mss 1460,nop,wscale 5>
0.200 < . 1:1(0) ack 1 win 32768
0.200 accept(3, ..., ...) = 4
// Client closes the connection
0.300 < F. 1:1(0) ack 1 win 32768
// now send rst with same sequence
0.300 < R. 1:1(0) ack 1 win 32768
// make sure we are in TCP_CLOSE
0.400 %{
assert tcpi_state == 7
}%
Signed-off-by: Jason Baron <jbaron@akamai.com>
Cc: Eric Dumazet <edumazet@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-01-17 11:37:19 -07:00
|
|
|
if (TCP_SKB_CB(skb)->seq == tp->rcv_nxt ||
|
|
|
|
tcp_reset_check(sk, skb)) {
|
tcp: accept RST if SEQ matches right edge of right-most SACK block
RFC 5961 advises to only accept RST packets containing a seq number
matching the next expected seq number instead of the whole receive
window in order to avoid spoofing attacks.
However, this situation is not optimal in the case SACK is in use at the
time the RST is sent. I recently run into a scenario in which packet
losses were high while uploading data to a server, and userspace was
willing to frequently terminate connections by sending a RST. In
this case, the ACK sent on the receiver side (rcv_nxt) is frozen waiting
for a lost packet retransmission and SACK blocks are used to let the
client continue uploading data. At some point later on, the client sends
the RST (snd_nxt), which matches the next expected seq number of the
right-most SACK block on the receiver side which is going forward
receiving data.
In this scenario, as RFC 5961 defines, the RST SEQ doesn't match the
frozen main ACK at receiver side and thus gets dropped and a challenge
ACK is sent, which gets usually lost due to network conditions. The main
consequence is that the connection stays alive for a while even if it
made sense to accept the RST. This can get really bad if lots of
connections like this one are created in few seconds, allocating all the
resources of the server easily.
For security reasons, not all SACK blocks are checked (there could be a
big amount of SACK blocks => acceptable SEQ numbers). Furthermore, it
wouldn't make sense to check for RST in blocks other than the right-most
received one because the sender is not expected to be sending new data
after the RST. For simplicity, only up to the 4 most recently updated
SACK blocks (selective_acks[4] field) are compared to find the
right-most block, as usually those are the ones with bigger probability
to contain it.
This patch was tested in a 3.18 kernel and probed to improve the
situation in the scenario described above.
Signed-off-by: Pau Espin Pedrol <pau.espin@tessares.net>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Tested-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-06-07 08:30:34 -06:00
|
|
|
rst_seq_match = true;
|
|
|
|
} else if (tcp_is_sack(tp) && tp->rx_opt.num_sacks > 0) {
|
|
|
|
struct tcp_sack_block *sp = &tp->selective_acks[0];
|
|
|
|
int max_sack = sp[0].end_seq;
|
|
|
|
int this_sack;
|
|
|
|
|
|
|
|
for (this_sack = 1; this_sack < tp->rx_opt.num_sacks;
|
|
|
|
++this_sack) {
|
|
|
|
max_sack = after(sp[this_sack].end_seq,
|
|
|
|
max_sack) ?
|
|
|
|
sp[this_sack].end_seq : max_sack;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (TCP_SKB_CB(skb)->seq == max_sack)
|
|
|
|
rst_seq_match = true;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (rst_seq_match)
|
2012-07-17 02:13:05 -06:00
|
|
|
tcp_reset(sk);
|
net/tcp_fastopen: Disable active side TFO in certain scenarios
Middlebox firewall issues can potentially cause server's data being
blackholed after a successful 3WHS using TFO. Following are the related
reports from Apple:
https://www.nanog.org/sites/default/files/Paasch_Network_Support.pdf
Slide 31 identifies an issue where the client ACK to the server's data
sent during a TFO'd handshake is dropped.
C ---> syn-data ---> S
C <--- syn/ack ----- S
C (accept & write)
C <---- data ------- S
C ----- ACK -> X S
[retry and timeout]
https://www.ietf.org/proceedings/94/slides/slides-94-tcpm-13.pdf
Slide 5 shows a similar situation that the server's data gets dropped
after 3WHS.
C ---- syn-data ---> S
C <--- syn/ack ----- S
C ---- ack --------> S
S (accept & write)
C? X <- data ------ S
[retry and timeout]
This is the worst failure b/c the client can not detect such behavior to
mitigate the situation (such as disabling TFO). Failing to proceed, the
application (e.g., SSL library) may simply timeout and retry with TFO
again, and the process repeats indefinitely.
The proposed solution is to disable active TFO globally under the
following circumstances:
1. client side TFO socket detects out of order FIN
2. client side TFO socket receives out of order RST
We disable active side TFO globally for 1hr at first. Then if it
happens again, we disable it for 2h, then 4h, 8h, ...
And we reset the timeout to 1hr if a client side TFO sockets not opened
on loopback has successfully received data segs from server.
And we examine this condition during close().
The rational behind it is that when such firewall issue happens,
application running on the client should eventually close the socket as
it is not able to get the data it is expecting. Or application running
on the server should close the socket as it is not able to receive any
response from client.
In both cases, out of order FIN or RST will get received on the client
given that the firewall will not block them as no data are in those
frames.
And we want to disable active TFO globally as it helps if the middle box
is very close to the client and most of the connections are likely to
fail.
Also, add a debug sysctl:
tcp_fastopen_blackhole_detect_timeout_sec:
the initial timeout to use when firewall blackhole issue happens.
This can be set and read.
When setting it to 0, it means to disable the active disable logic.
Signed-off-by: Wei Wang <weiwan@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-04-20 15:45:46 -06:00
|
|
|
else {
|
|
|
|
/* Disable TFO if RST is out-of-order
|
|
|
|
* and no data has been received
|
|
|
|
* for current active TFO socket
|
|
|
|
*/
|
|
|
|
if (tp->syn_fastopen && !tp->data_segs_in &&
|
|
|
|
sk->sk_state == TCP_ESTABLISHED)
|
2017-04-20 15:45:47 -06:00
|
|
|
tcp_fastopen_active_disable(sk);
|
2015-02-06 14:04:40 -07:00
|
|
|
tcp_send_challenge_ack(sk, skb);
|
net/tcp_fastopen: Disable active side TFO in certain scenarios
Middlebox firewall issues can potentially cause server's data being
blackholed after a successful 3WHS using TFO. Following are the related
reports from Apple:
https://www.nanog.org/sites/default/files/Paasch_Network_Support.pdf
Slide 31 identifies an issue where the client ACK to the server's data
sent during a TFO'd handshake is dropped.
C ---> syn-data ---> S
C <--- syn/ack ----- S
C (accept & write)
C <---- data ------- S
C ----- ACK -> X S
[retry and timeout]
https://www.ietf.org/proceedings/94/slides/slides-94-tcpm-13.pdf
Slide 5 shows a similar situation that the server's data gets dropped
after 3WHS.
C ---- syn-data ---> S
C <--- syn/ack ----- S
C ---- ack --------> S
S (accept & write)
C? X <- data ------ S
[retry and timeout]
This is the worst failure b/c the client can not detect such behavior to
mitigate the situation (such as disabling TFO). Failing to proceed, the
application (e.g., SSL library) may simply timeout and retry with TFO
again, and the process repeats indefinitely.
The proposed solution is to disable active TFO globally under the
following circumstances:
1. client side TFO socket detects out of order FIN
2. client side TFO socket receives out of order RST
We disable active side TFO globally for 1hr at first. Then if it
happens again, we disable it for 2h, then 4h, 8h, ...
And we reset the timeout to 1hr if a client side TFO sockets not opened
on loopback has successfully received data segs from server.
And we examine this condition during close().
The rational behind it is that when such firewall issue happens,
application running on the client should eventually close the socket as
it is not able to get the data it is expecting. Or application running
on the server should close the socket as it is not able to receive any
response from client.
In both cases, out of order FIN or RST will get received on the client
given that the firewall will not block them as no data are in those
frames.
And we want to disable active TFO globally as it helps if the middle box
is very close to the client and most of the connections are likely to
fail.
Also, add a debug sysctl:
tcp_fastopen_blackhole_detect_timeout_sec:
the initial timeout to use when firewall blackhole issue happens.
This can be set and read.
When setting it to 0, it means to disable the active disable logic.
Signed-off-by: Wei Wang <weiwan@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-04-20 15:45:46 -06:00
|
|
|
}
|
2008-08-23 06:10:12 -06:00
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* step 3: check security and precedence [ignored] */
|
|
|
|
|
2012-07-16 19:41:30 -06:00
|
|
|
/* step 4: Check for a SYN
|
2014-10-30 10:48:08 -06:00
|
|
|
* RFC 5961 4.2 : Send a challenge ack
|
2012-07-16 19:41:30 -06:00
|
|
|
*/
|
|
|
|
if (th->syn) {
|
2012-07-17 06:29:30 -06:00
|
|
|
syn_challenge:
|
2008-08-23 06:10:12 -06:00
|
|
|
if (syn_inerr)
|
2016-04-29 15:16:47 -06:00
|
|
|
TCP_INC_STATS(sock_net(sk), TCP_MIB_INERRS);
|
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPSYNCHALLENGE);
|
2015-02-06 14:04:40 -07:00
|
|
|
tcp_send_challenge_ack(sk, skb);
|
2012-07-16 19:41:30 -06:00
|
|
|
goto discard;
|
2008-08-23 06:10:12 -06:00
|
|
|
}
|
|
|
|
|
2012-07-16 19:41:30 -06:00
|
|
|
return true;
|
2008-08-23 06:10:12 -06:00
|
|
|
|
|
|
|
discard:
|
2016-04-01 09:52:19 -06:00
|
|
|
tcp_drop(sk, skb);
|
2012-07-16 19:41:30 -06:00
|
|
|
return false;
|
2008-08-23 06:10:12 -06:00
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
2007-02-09 07:24:47 -07:00
|
|
|
* TCP receive function for the ESTABLISHED state.
|
2017-08-30 11:24:58 -06:00
|
|
|
*
|
|
|
|
* It is split into a fast path and a slow path. The fast path is
|
|
|
|
* disabled when:
|
|
|
|
* - A zero window was announced from us - zero window probing
|
|
|
|
* is only handled properly in the slow path.
|
|
|
|
* - Out of order segments arrived.
|
|
|
|
* - Urgent data is expected.
|
|
|
|
* - There is no buffer space left
|
|
|
|
* - Unexpected TCP flags/window values/header lengths are received
|
|
|
|
* (detected by checking the TCP header against pred_flags)
|
|
|
|
* - Data is sent in both directions. Fast path only supports pure senders
|
|
|
|
* or pure receivers (this means either the sequence number or the ack
|
|
|
|
* value must stay constant)
|
|
|
|
* - Unexpected TCP option.
|
|
|
|
*
|
|
|
|
* When these conditions are not satisfied it drops into a standard
|
|
|
|
* receive procedure patterned after RFC793 to handle all cases.
|
|
|
|
* The first three cases are guaranteed by proper pred_flags setting,
|
|
|
|
* the rest is checked inline. Fast processing is turned on in
|
|
|
|
* tcp_data_queue when everything is OK.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
2018-05-29 09:27:31 -06:00
|
|
|
void tcp_rcv_established(struct sock *sk, struct sk_buff *skb)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2018-05-29 09:27:31 -06:00
|
|
|
const struct tcphdr *th = (const struct tcphdr *)skb->data;
|
2005-04-16 16:20:36 -06:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2018-05-29 09:27:31 -06:00
|
|
|
unsigned int len = skb->len;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-12-28 19:45:51 -07:00
|
|
|
/* TCP congestion window tracking */
|
|
|
|
trace_tcp_probe(sk, skb);
|
|
|
|
|
2017-05-16 15:00:14 -06:00
|
|
|
tcp_mstamp_refresh(tp);
|
2015-04-03 02:17:26 -06:00
|
|
|
if (unlikely(!sk->sk_rx_dst))
|
2012-08-05 23:09:33 -06:00
|
|
|
inet_csk(sk)->icsk_af_ops->sk_rx_dst_set(sk, skb);
|
2017-08-30 11:24:58 -06:00
|
|
|
/*
|
|
|
|
* Header prediction.
|
|
|
|
* The code loosely follows the one in the famous
|
|
|
|
* "30 instruction TCP receive" Van Jacobson mail.
|
|
|
|
*
|
|
|
|
* Van's trick is to deposit buffers into socket queue
|
|
|
|
* on a device interrupt, to call tcp_recv function
|
|
|
|
* on the receive process context and checksum and copy
|
|
|
|
* the buffer to user space. smart...
|
|
|
|
*
|
|
|
|
* Our current scheme is not silly either but we take the
|
|
|
|
* extra cost of the net_bh soft interrupt processing...
|
|
|
|
* We do checksum and copy also but from device to kernel.
|
|
|
|
*/
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
tp->rx_opt.saw_tstamp = 0;
|
|
|
|
|
2017-08-30 11:24:58 -06:00
|
|
|
/* pred_flags is 0xS?10 << 16 + snd_wnd
|
|
|
|
* if header_prediction is to be made
|
|
|
|
* 'S' will always be tp->tcp_header_len >> 2
|
|
|
|
* '?' will be 0 for the fast path, otherwise pred_flags is 0 to
|
|
|
|
* turn it off (when there are holes in the receive
|
|
|
|
* space for instance)
|
|
|
|
* PSH flag is ignored.
|
|
|
|
*/
|
|
|
|
|
|
|
|
if ((tcp_flag_word(th) & TCP_HP_BITS) == tp->pred_flags &&
|
|
|
|
TCP_SKB_CB(skb)->seq == tp->rcv_nxt &&
|
|
|
|
!after(TCP_SKB_CB(skb)->ack_seq, tp->snd_nxt)) {
|
|
|
|
int tcp_header_len = tp->tcp_header_len;
|
|
|
|
|
|
|
|
/* Timestamp header prediction: tcp_header_len
|
|
|
|
* is automatically equal to th->doff*4 due to pred_flags
|
|
|
|
* match.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* Check timestamp */
|
|
|
|
if (tcp_header_len == sizeof(struct tcphdr) + TCPOLEN_TSTAMP_ALIGNED) {
|
|
|
|
/* No? Slow path! */
|
|
|
|
if (!tcp_parse_aligned_timestamp(tp, th))
|
|
|
|
goto slow_path;
|
|
|
|
|
|
|
|
/* If PAWS failed, check it more carefully in slow path */
|
|
|
|
if ((s32)(tp->rx_opt.rcv_tsval - tp->rx_opt.ts_recent) < 0)
|
|
|
|
goto slow_path;
|
|
|
|
|
|
|
|
/* DO NOT update ts_recent here, if checksum fails
|
|
|
|
* and timestamp was corrupted part, it will result
|
|
|
|
* in a hung connection since we will drop all
|
|
|
|
* future packets due to the PAWS test.
|
|
|
|
*/
|
|
|
|
}
|
|
|
|
|
|
|
|
if (len <= tcp_header_len) {
|
|
|
|
/* Bulk data transfer: sender */
|
|
|
|
if (len == tcp_header_len) {
|
|
|
|
/* Predicted packet is in window by definition.
|
|
|
|
* seq == rcv_nxt and rcv_wup <= rcv_nxt.
|
|
|
|
* Hence, check seq<=rcv_wup reduces to:
|
|
|
|
*/
|
|
|
|
if (tcp_header_len ==
|
|
|
|
(sizeof(struct tcphdr) + TCPOLEN_TSTAMP_ALIGNED) &&
|
|
|
|
tp->rcv_nxt == tp->rcv_wup)
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
|
|
|
|
/* We know that such packets are checksummed
|
|
|
|
* on entry.
|
|
|
|
*/
|
|
|
|
tcp_ack(sk, skb, 0);
|
|
|
|
__kfree_skb(skb);
|
|
|
|
tcp_data_snd_check(sk);
|
2018-06-19 22:42:50 -06:00
|
|
|
/* When receiving pure ack in fast path, update
|
|
|
|
* last ts ecr directly instead of calling
|
|
|
|
* tcp_rcv_rtt_measure_ts()
|
|
|
|
*/
|
|
|
|
tp->rcv_rtt_last_tsecr = tp->rx_opt.rcv_tsecr;
|
2017-08-30 11:24:58 -06:00
|
|
|
return;
|
|
|
|
} else { /* Header too small */
|
|
|
|
TCP_INC_STATS(sock_net(sk), TCP_MIB_INERRS);
|
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
int eaten = 0;
|
|
|
|
bool fragstolen = false;
|
|
|
|
|
|
|
|
if (tcp_checksum_complete(skb))
|
|
|
|
goto csum_error;
|
|
|
|
|
|
|
|
if ((int)skb->truesize > sk->sk_forward_alloc)
|
|
|
|
goto step5;
|
|
|
|
|
|
|
|
/* Predicted packet is in window by definition.
|
|
|
|
* seq == rcv_nxt and rcv_wup <= rcv_nxt.
|
|
|
|
* Hence, check seq<=rcv_wup reduces to:
|
|
|
|
*/
|
|
|
|
if (tcp_header_len ==
|
|
|
|
(sizeof(struct tcphdr) + TCPOLEN_TSTAMP_ALIGNED) &&
|
|
|
|
tp->rcv_nxt == tp->rcv_wup)
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
|
|
|
|
tcp_rcv_rtt_measure_ts(sk, skb);
|
|
|
|
|
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPHPHITS);
|
|
|
|
|
|
|
|
/* Bulk data transfer: receiver */
|
2018-11-26 15:49:12 -07:00
|
|
|
__skb_pull(skb, tcp_header_len);
|
|
|
|
eaten = tcp_queue_rcv(sk, skb, &fragstolen);
|
2017-08-30 11:24:58 -06:00
|
|
|
|
|
|
|
tcp_event_data_recv(sk, skb);
|
|
|
|
|
|
|
|
if (TCP_SKB_CB(skb)->ack_seq != tp->snd_una) {
|
|
|
|
/* Well, only one small jumplet in fast path... */
|
|
|
|
tcp_ack(sk, skb, FLAG_DATA);
|
|
|
|
tcp_data_snd_check(sk);
|
|
|
|
if (!inet_csk_ack_scheduled(sk))
|
|
|
|
goto no_ack;
|
|
|
|
}
|
|
|
|
|
|
|
|
__tcp_ack_snd_check(sk, 0);
|
|
|
|
no_ack:
|
|
|
|
if (eaten)
|
|
|
|
kfree_skb_partial(skb, fragstolen);
|
tcp: avoid extra wakeups for SO_RCVLOWAT users
SO_RCVLOWAT is properly handled in tcp_poll(), so that POLLIN is only
generated when enough bytes are available in receive queue, after
David change (commit c7004482e8dc "tcp: Respect SO_RCVLOWAT in tcp_poll().")
But TCP still calls sk->sk_data_ready() for each chunk added in receive
queue, meaning thread is awaken, and goes back to sleep shortly after.
Tested:
tcp_mmap test program, receiving 32768 MB of data with SO_RCVLOWAT set to 512KB
-> Should get ~2 wakeups (c-switches) per MB, regardless of how many
(tiny or big) packets were received.
High speed (mostly full size GRO packets)
received 32768 MB (100 % mmap'ed) in 8.03112 s, 34.2266 Gbit,
cpu usage user:0.037 sys:1.404, 43.9758 usec per MB, 65497 c-switches
received 32768 MB (99.9954 % mmap'ed) in 7.98453 s, 34.4263 Gbit,
cpu usage user:0.03 sys:1.422, 44.3115 usec per MB, 65485 c-switches
Low speed (sender is ratelimited and sends 1-MSS at a time, so GRO is not helping)
received 22474.5 MB (100 % mmap'ed) in 6015.35 s, 0.0313414 Gbit,
cpu usage user:0.05 sys:1.586, 72.7952 usec per MB, 44950 c-switches
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-04-16 11:33:37 -06:00
|
|
|
tcp_data_ready(sk);
|
2017-08-30 11:24:58 -06:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
slow_path:
|
2016-04-29 15:16:48 -06:00
|
|
|
if (len < (th->doff << 2) || tcp_checksum_complete(skb))
|
2005-04-16 16:20:36 -06:00
|
|
|
goto csum_error;
|
|
|
|
|
tcp: Restore RFC5961-compliant behavior for SYN packets
Commit c3ae62af8e755 ("tcp: should drop incoming frames without ACK
flag set") was created to mitigate a security vulnerability in which a
local attacker is able to inject data into locally-opened sockets by
using TCP protocol statistics in procfs to quickly find the correct
sequence number.
This broke the RFC5961 requirement to send a challenge ACK in response
to spurious RST packets, which was subsequently fixed by commit
7b514a886ba50 ("tcp: accept RST without ACK flag").
Unfortunately, the RFC5961 requirement that spurious SYN packets be
handled in a similar manner remains broken.
RFC5961 section 4 states that:
... the handling of the SYN in the synchronized state SHOULD be
performed as follows:
1) If the SYN bit is set, irrespective of the sequence number, TCP
MUST send an ACK (also referred to as challenge ACK) to the remote
peer:
<SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>
After sending the acknowledgment, TCP MUST drop the unacceptable
segment and stop processing further.
By sending an ACK, the remote peer is challenged to confirm the loss
of the previous connection and the request to start a new connection.
A legitimate peer, after restart, would not have a TCB in the
synchronized state. Thus, when the ACK arrives, the peer should send
a RST segment back with the sequence number derived from the ACK
field that caused the RST.
This RST will confirm that the remote peer has indeed closed the
previous connection. Upon receipt of a valid RST, the local TCP
endpoint MUST terminate its connection. The local TCP endpoint
should then rely on SYN retransmission from the remote end to
re-establish the connection.
This patch lets SYN packets through the discard added in c3ae62af8e755,
so that spurious SYN packets are properly dealt with as per the RFC.
The challenge ACK is sent unconditionally and is rate-limited, so the
original vulnerability is not reintroduced by this patch.
Signed-off-by: Calvin Owens <calvinowens@fb.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-20 16:09:53 -07:00
|
|
|
if (!th->ack && !th->rst && !th->syn)
|
2012-12-26 05:44:34 -07:00
|
|
|
goto discard;
|
|
|
|
|
2017-08-30 11:24:58 -06:00
|
|
|
/*
|
|
|
|
* Standard slow path.
|
|
|
|
*/
|
|
|
|
|
2012-07-16 19:41:30 -06:00
|
|
|
if (!tcp_validate_incoming(sk, skb, th, 1))
|
2013-09-03 13:23:22 -06:00
|
|
|
return;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-08-30 11:24:58 -06:00
|
|
|
step5:
|
|
|
|
if (tcp_ack(sk, skb, FLAG_SLOWPATH | FLAG_UPDATE_TS_RECENT) < 0)
|
2009-03-22 22:49:57 -06:00
|
|
|
goto discard;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2005-08-09 21:10:42 -06:00
|
|
|
tcp_rcv_rtt_measure_ts(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Process urgent data. */
|
|
|
|
tcp_urg(sk, skb, th);
|
|
|
|
|
|
|
|
/* step 7: process the segment text */
|
|
|
|
tcp_data_queue(sk, skb);
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
tcp_data_snd_check(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
tcp_ack_snd_check(sk);
|
2013-09-03 13:23:22 -06:00
|
|
|
return;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
csum_error:
|
2016-04-29 15:16:47 -06:00
|
|
|
TCP_INC_STATS(sock_net(sk), TCP_MIB_CSUMERRORS);
|
|
|
|
TCP_INC_STATS(sock_net(sk), TCP_MIB_INERRS);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
discard:
|
2016-04-01 09:52:19 -06:00
|
|
|
tcp_drop(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2010-07-09 15:22:10 -06:00
|
|
|
EXPORT_SYMBOL(tcp_rcv_established);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2019-04-29 16:46:20 -06:00
|
|
|
void tcp_init_transfer(struct sock *sk, int bpf_op)
|
|
|
|
{
|
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
tcp_mtup_init(sk);
|
|
|
|
icsk->icsk_af_ops->rebuild_header(sk);
|
|
|
|
tcp_init_metrics(sk);
|
|
|
|
|
|
|
|
/* Initialize the congestion window to start the transfer.
|
|
|
|
* Cut cwnd down to 1 per RFC5681 if SYN or SYN-ACK has been
|
|
|
|
* retransmitted. In light of RFC6298 more aggressive 1sec
|
|
|
|
* initRTO, we only reset cwnd when more than 1 SYN/SYN-ACK
|
|
|
|
* retransmission has occurred.
|
|
|
|
*/
|
|
|
|
if (tp->total_retrans > 1 && tp->undo_marker)
|
|
|
|
tp->snd_cwnd = 1;
|
|
|
|
else
|
|
|
|
tp->snd_cwnd = tcp_init_cwnd(tp, __sk_dst_get(sk));
|
|
|
|
tp->snd_cwnd_stamp = tcp_jiffies32;
|
|
|
|
|
|
|
|
tcp_call_bpf(sk, bpf_op, 0, NULL);
|
|
|
|
tcp_init_congestion_control(sk);
|
|
|
|
tcp_init_buffer_space(sk);
|
|
|
|
}
|
|
|
|
|
2012-04-18 21:40:01 -06:00
|
|
|
void tcp_finish_connect(struct sock *sk, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
|
|
|
|
tcp_set_state(sk, TCP_ESTABLISHED);
|
2017-05-16 15:00:07 -06:00
|
|
|
icsk->icsk_ack.lrcvtime = tcp_jiffies32;
|
2012-04-18 21:40:01 -06:00
|
|
|
|
2015-04-03 02:17:27 -06:00
|
|
|
if (skb) {
|
2012-08-05 23:09:33 -06:00
|
|
|
icsk->icsk_af_ops->sk_rx_dst_set(sk, skb);
|
2012-04-18 21:40:01 -06:00
|
|
|
security_inet_conn_established(sk, skb);
|
2018-06-29 22:26:57 -06:00
|
|
|
sk_mark_napi_id(sk, skb);
|
2012-06-19 22:22:05 -06:00
|
|
|
}
|
2012-04-18 21:40:01 -06:00
|
|
|
|
tcp: uniform the set up of sockets after successful connection
Currently in the TCP code, the initialization sequence for cached
metrics, congestion control, BPF, etc, after successful connection
is very inconsistent. This introduces inconsistent bevhavior and is
prone to bugs. The current call sequence is as follows:
(1) for active case (tcp_finish_connect() case):
tcp_mtup_init(sk);
icsk->icsk_af_ops->rebuild_header(sk);
tcp_init_metrics(sk);
tcp_call_bpf(sk, BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB);
tcp_init_congestion_control(sk);
tcp_init_buffer_space(sk);
(2) for passive case (tcp_rcv_state_process() TCP_SYN_RECV case):
icsk->icsk_af_ops->rebuild_header(sk);
tcp_call_bpf(sk, BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB);
tcp_init_congestion_control(sk);
tcp_mtup_init(sk);
tcp_init_buffer_space(sk);
tcp_init_metrics(sk);
(3) for TFO passive case (tcp_fastopen_create_child()):
inet_csk(child)->icsk_af_ops->rebuild_header(child);
tcp_init_congestion_control(child);
tcp_mtup_init(child);
tcp_init_metrics(child);
tcp_call_bpf(child, BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB);
tcp_init_buffer_space(child);
This commit uniforms the above functions to have the following sequence:
tcp_mtup_init(sk);
icsk->icsk_af_ops->rebuild_header(sk);
tcp_init_metrics(sk);
tcp_call_bpf(sk, BPF_SOCK_OPS_ACTIVE/PASSIVE_ESTABLISHED_CB);
tcp_init_congestion_control(sk);
tcp_init_buffer_space(sk);
This sequence is the same as the (1) active case. We pick this sequence
because this order correctly allows BPF to override the settings
including congestion control module and initial cwnd, etc from
the route, and then allows the CC module to see those settings.
Suggested-by: Neal Cardwell <ncardwell@google.com>
Tested-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Wei Wang <weiwan@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-04 11:03:44 -06:00
|
|
|
tcp_init_transfer(sk, BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB);
|
2012-04-18 21:40:01 -06:00
|
|
|
|
|
|
|
/* Prevent spurious tcp_cwnd_restart() on first data
|
|
|
|
* packet.
|
|
|
|
*/
|
2017-05-16 15:00:03 -06:00
|
|
|
tp->lsndtime = tcp_jiffies32;
|
2012-04-18 21:40:01 -06:00
|
|
|
|
|
|
|
if (sock_flag(sk, SOCK_KEEPOPEN))
|
|
|
|
inet_csk_reset_keepalive_timer(sk, keepalive_time_when(tp));
|
2017-08-30 11:24:58 -06:00
|
|
|
|
|
|
|
if (!tp->rx_opt.snd_wscale)
|
|
|
|
__tcp_fast_path_on(tp, tp->snd_wnd);
|
|
|
|
else
|
|
|
|
tp->pred_flags = 0;
|
2012-04-18 21:40:01 -06:00
|
|
|
}
|
|
|
|
|
2012-07-19 00:43:08 -06:00
|
|
|
static bool tcp_rcv_fastopen_synack(struct sock *sk, struct sk_buff *synack,
|
|
|
|
struct tcp_fastopen_cookie *cookie)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2017-10-05 23:21:27 -06:00
|
|
|
struct sk_buff *data = tp->syn_data ? tcp_rtx_queue_head(sk) : NULL;
|
2015-04-06 15:37:27 -06:00
|
|
|
u16 mss = tp->rx_opt.mss_clamp, try_exp = 0;
|
|
|
|
bool syn_drop = false;
|
2012-07-19 00:43:08 -06:00
|
|
|
|
|
|
|
if (mss == tp->rx_opt.user_mss) {
|
|
|
|
struct tcp_options_received opt;
|
|
|
|
|
|
|
|
/* Get original SYNACK MSS value if user MSS sets mss_clamp */
|
|
|
|
tcp_clear_options(&opt);
|
|
|
|
opt.user_mss = opt.mss_clamp = 0;
|
2017-06-07 11:34:36 -06:00
|
|
|
tcp_parse_options(sock_net(sk), synack, &opt, 0, NULL);
|
2012-07-19 00:43:08 -06:00
|
|
|
mss = opt.mss_clamp;
|
|
|
|
}
|
|
|
|
|
2015-04-06 15:37:27 -06:00
|
|
|
if (!tp->syn_fastopen) {
|
|
|
|
/* Ignore an unsolicited cookie */
|
2012-07-19 00:43:11 -06:00
|
|
|
cookie->len = -1;
|
2015-04-06 15:37:27 -06:00
|
|
|
} else if (tp->total_retrans) {
|
|
|
|
/* SYN timed out and the SYN-ACK neither has a cookie nor
|
|
|
|
* acknowledges data. Presumably the remote received only
|
|
|
|
* the retransmitted (regular) SYNs: either the original
|
|
|
|
* SYN-data or the corresponding SYN-ACK was dropped.
|
|
|
|
*/
|
|
|
|
syn_drop = (cookie->len < 0 && data);
|
|
|
|
} else if (cookie->len < 0 && !tp->syn_data) {
|
|
|
|
/* We requested a cookie but didn't get it. If we did not use
|
|
|
|
* the (old) exp opt format then try so next time (try_exp=1).
|
|
|
|
* Otherwise we go back to use the RFC7413 opt (try_exp=2).
|
|
|
|
*/
|
|
|
|
try_exp = tp->syn_fastopen_exp ? 2 : 1;
|
|
|
|
}
|
2012-07-19 00:43:11 -06:00
|
|
|
|
2015-04-06 15:37:27 -06:00
|
|
|
tcp_fastopen_cache_set(sk, mss, cookie, syn_drop, try_exp);
|
2012-07-19 00:43:08 -06:00
|
|
|
|
|
|
|
if (data) { /* Retransmit unacked data in SYN */
|
2017-10-05 23:21:27 -06:00
|
|
|
skb_rbtree_walk_from(data) {
|
|
|
|
if (__tcp_retransmit_skb(sk, data, 1))
|
2012-12-06 01:45:32 -07:00
|
|
|
break;
|
|
|
|
}
|
2012-07-19 00:43:08 -06:00
|
|
|
tcp_rearm_rto(sk);
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk),
|
2016-04-27 17:44:39 -06:00
|
|
|
LINUX_MIB_TCPFASTOPENACTIVEFAIL);
|
2012-07-19 00:43:08 -06:00
|
|
|
return true;
|
|
|
|
}
|
2012-10-19 09:14:44 -06:00
|
|
|
tp->syn_data_acked = tp->syn_data;
|
2018-04-18 00:18:46 -06:00
|
|
|
if (tp->syn_data_acked) {
|
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPFASTOPENACTIVE);
|
|
|
|
/* SYN-data is counted as two separate packets in tcp_ack() */
|
|
|
|
if (tp->delivered > 1)
|
|
|
|
--tp->delivered;
|
|
|
|
}
|
2016-02-01 22:03:07 -07:00
|
|
|
|
|
|
|
tcp_fastopen_add_skb(sk, synack);
|
|
|
|
|
2012-07-19 00:43:08 -06:00
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2017-10-25 03:01:45 -06:00
|
|
|
static void smc_check_reset_syn(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
#if IS_ENABLED(CONFIG_SMC)
|
|
|
|
if (static_branch_unlikely(&tcp_have_smc)) {
|
|
|
|
if (tp->syn_smc && !tp->rx_opt.smc_ok)
|
|
|
|
tp->syn_smc = 0;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2019-04-29 16:46:14 -06:00
|
|
|
static void tcp_try_undo_spurious_syn(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
u32 syn_stamp;
|
|
|
|
|
|
|
|
/* undo_marker is set when SYN or SYNACK times out. The timeout is
|
|
|
|
* spurious if the ACK's timestamp option echo value matches the
|
|
|
|
* original SYN timestamp.
|
|
|
|
*/
|
|
|
|
syn_stamp = tp->retrans_stamp;
|
|
|
|
if (tp->undo_marker && syn_stamp && tp->rx_opt.saw_tstamp &&
|
|
|
|
syn_stamp == tp->rx_opt.rcv_tsecr)
|
|
|
|
tp->undo_marker = 0;
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
|
2015-09-29 08:42:40 -06:00
|
|
|
const struct tcphdr *th)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-12-14 00:26:10 -07:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2009-12-02 11:25:27 -07:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2012-07-19 00:43:08 -06:00
|
|
|
struct tcp_fastopen_cookie foc = { .len = -1 };
|
2009-12-02 11:25:27 -07:00
|
|
|
int saved_clamp = tp->rx_opt.mss_clamp;
|
2017-04-18 10:45:52 -06:00
|
|
|
bool fastopen_fail;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-06-07 11:34:36 -06:00
|
|
|
tcp_parse_options(sock_net(sk), skb, &tp->rx_opt, 0, &foc);
|
2013-08-27 02:21:55 -06:00
|
|
|
if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
|
2013-02-10 22:50:19 -07:00
|
|
|
tp->rx_opt.rcv_tsecr -= tp->tsoffset;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
if (th->ack) {
|
|
|
|
/* rfc793:
|
|
|
|
* "If the state is SYN-SENT then
|
|
|
|
* first check the ACK bit
|
|
|
|
* If the ACK bit is set
|
|
|
|
* If SEG.ACK =< ISS, or SEG.ACK > SND.NXT, send
|
|
|
|
* a reset (unless the RST bit is set, if so drop
|
|
|
|
* the segment and return)"
|
|
|
|
*/
|
2012-07-19 00:43:08 -06:00
|
|
|
if (!after(TCP_SKB_CB(skb)->ack_seq, tp->snd_una) ||
|
|
|
|
after(TCP_SKB_CB(skb)->ack_seq, tp->snd_nxt))
|
2005-04-16 16:20:36 -06:00
|
|
|
goto reset_and_undo;
|
|
|
|
|
|
|
|
if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr &&
|
|
|
|
!between(tp->rx_opt.rcv_tsecr, tp->retrans_stamp,
|
2017-05-16 15:00:14 -06:00
|
|
|
tcp_time_stamp(tp))) {
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk),
|
2016-04-27 17:44:39 -06:00
|
|
|
LINUX_MIB_PAWSACTIVEREJECTED);
|
2005-04-16 16:20:36 -06:00
|
|
|
goto reset_and_undo;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Now ACK is acceptable.
|
|
|
|
*
|
|
|
|
* "If the RST bit is set
|
|
|
|
* If the ACK was acceptable then signal the user "error:
|
|
|
|
* connection reset", drop the segment, enter CLOSED state,
|
|
|
|
* delete TCB, and return."
|
|
|
|
*/
|
|
|
|
|
|
|
|
if (th->rst) {
|
|
|
|
tcp_reset(sk);
|
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* rfc793:
|
|
|
|
* "fifth, if neither of the SYN or RST bits is set then
|
|
|
|
* drop the segment and return."
|
|
|
|
*
|
|
|
|
* See note below!
|
|
|
|
* --ANK(990513)
|
|
|
|
*/
|
|
|
|
if (!th->syn)
|
|
|
|
goto discard_and_undo;
|
|
|
|
|
|
|
|
/* rfc793:
|
|
|
|
* "If the SYN bit is on ...
|
|
|
|
* are acceptable then ...
|
|
|
|
* (our SYN has been ACKed), change the connection
|
|
|
|
* state to ESTABLISHED..."
|
|
|
|
*/
|
|
|
|
|
2014-09-29 05:08:30 -06:00
|
|
|
tcp_ecn_rcv_synack(tp, th);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2012-08-14 07:30:20 -06:00
|
|
|
tcp_init_wl(tp, TCP_SKB_CB(skb)->seq);
|
2019-04-29 16:46:14 -06:00
|
|
|
tcp_try_undo_spurious_syn(sk);
|
2017-08-30 11:24:58 -06:00
|
|
|
tcp_ack(sk, skb, FLAG_SLOWPATH);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Ok.. it's good. Set up sequence numbers and
|
|
|
|
* move to established.
|
|
|
|
*/
|
|
|
|
tp->rcv_nxt = TCP_SKB_CB(skb)->seq + 1;
|
|
|
|
tp->rcv_wup = TCP_SKB_CB(skb)->seq + 1;
|
|
|
|
|
|
|
|
/* RFC1323: The window in SYN & SYN/ACK segments is
|
|
|
|
* never scaled.
|
|
|
|
*/
|
|
|
|
tp->snd_wnd = ntohs(th->window);
|
|
|
|
|
|
|
|
if (!tp->rx_opt.wscale_ok) {
|
|
|
|
tp->rx_opt.snd_wscale = tp->rx_opt.rcv_wscale = 0;
|
|
|
|
tp->window_clamp = min(tp->window_clamp, 65535U);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (tp->rx_opt.saw_tstamp) {
|
|
|
|
tp->rx_opt.tstamp_ok = 1;
|
|
|
|
tp->tcp_header_len =
|
|
|
|
sizeof(struct tcphdr) + TCPOLEN_TSTAMP_ALIGNED;
|
|
|
|
tp->advmss -= TCPOLEN_TSTAMP_ALIGNED;
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
} else {
|
|
|
|
tp->tcp_header_len = sizeof(struct tcphdr);
|
|
|
|
}
|
|
|
|
|
2005-12-14 00:26:10 -07:00
|
|
|
tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
|
2005-04-16 16:20:36 -06:00
|
|
|
tcp_initialize_rcv_mss(sk);
|
|
|
|
|
|
|
|
/* Remember, tcp_poll() does not lock socket!
|
|
|
|
* Change state from SYN-SENT only after copied_seq
|
|
|
|
* is initialized. */
|
|
|
|
tp->copied_seq = tp->rcv_nxt;
|
2009-12-02 11:25:27 -07:00
|
|
|
|
2017-10-25 03:01:45 -06:00
|
|
|
smc_check_reset_syn(tp);
|
|
|
|
|
2006-12-07 01:11:33 -07:00
|
|
|
smp_mb();
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2012-04-18 21:40:01 -06:00
|
|
|
tcp_finish_connect(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-04-18 10:45:52 -06:00
|
|
|
fastopen_fail = (tp->syn_fastopen || tp->syn_data) &&
|
|
|
|
tcp_rcv_fastopen_synack(sk, skb, &foc);
|
2012-07-19 00:43:08 -06:00
|
|
|
|
2017-04-18 10:45:52 -06:00
|
|
|
if (!sock_flag(sk, SOCK_DEAD)) {
|
|
|
|
sk->sk_state_change(sk);
|
|
|
|
sk_wake_async(sk, SOCK_WAKE_IO, POLL_OUT);
|
|
|
|
}
|
|
|
|
if (fastopen_fail)
|
|
|
|
return -1;
|
2005-08-09 21:11:56 -06:00
|
|
|
if (sk->sk_write_pending ||
|
|
|
|
icsk->icsk_accept_queue.rskq_defer_accept ||
|
2019-01-25 11:53:19 -07:00
|
|
|
inet_csk_in_pingpong_mode(sk)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Save one ACK. Data will be ready after
|
|
|
|
* several ticks, if write_pending is set.
|
|
|
|
*
|
|
|
|
* It may be deleted, but with this feature tcpdumps
|
|
|
|
* look so _wonderfully_ clever, that I was not able
|
|
|
|
* to stand against the temptation 8) --ANK
|
|
|
|
*/
|
2005-08-09 21:10:42 -06:00
|
|
|
inet_csk_schedule_ack(sk);
|
2018-05-21 16:08:56 -06:00
|
|
|
tcp_enter_quickack_mode(sk, TCP_MAX_QUICKACKS);
|
2005-08-09 21:11:08 -06:00
|
|
|
inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,
|
|
|
|
TCP_DELACK_MAX, TCP_RTO_MAX);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
discard:
|
2016-04-01 09:52:19 -06:00
|
|
|
tcp_drop(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
return 0;
|
|
|
|
} else {
|
|
|
|
tcp_send_ack(sk);
|
|
|
|
}
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* No ACK in the segment */
|
|
|
|
|
|
|
|
if (th->rst) {
|
|
|
|
/* rfc793:
|
|
|
|
* "If the RST bit is set
|
|
|
|
*
|
|
|
|
* Otherwise (no ACK) drop the segment and return."
|
|
|
|
*/
|
|
|
|
|
|
|
|
goto discard_and_undo;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* PAWS check. */
|
2007-12-31 15:57:14 -07:00
|
|
|
if (tp->rx_opt.ts_recent_stamp && tp->rx_opt.saw_tstamp &&
|
2009-03-14 08:23:03 -06:00
|
|
|
tcp_paws_reject(&tp->rx_opt, 0))
|
2005-04-16 16:20:36 -06:00
|
|
|
goto discard_and_undo;
|
|
|
|
|
|
|
|
if (th->syn) {
|
|
|
|
/* We see SYN without ACK. It is attempt of
|
|
|
|
* simultaneous connect with crossed SYNs.
|
|
|
|
* Particularly, it can be connect to self.
|
|
|
|
*/
|
|
|
|
tcp_set_state(sk, TCP_SYN_RECV);
|
|
|
|
|
|
|
|
if (tp->rx_opt.saw_tstamp) {
|
|
|
|
tp->rx_opt.tstamp_ok = 1;
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
tp->tcp_header_len =
|
|
|
|
sizeof(struct tcphdr) + TCPOLEN_TSTAMP_ALIGNED;
|
|
|
|
} else {
|
|
|
|
tp->tcp_header_len = sizeof(struct tcphdr);
|
|
|
|
}
|
|
|
|
|
|
|
|
tp->rcv_nxt = TCP_SKB_CB(skb)->seq + 1;
|
2015-11-26 09:18:14 -07:00
|
|
|
tp->copied_seq = tp->rcv_nxt;
|
2005-04-16 16:20:36 -06:00
|
|
|
tp->rcv_wup = TCP_SKB_CB(skb)->seq + 1;
|
|
|
|
|
|
|
|
/* RFC1323: The window in SYN & SYN/ACK segments is
|
|
|
|
* never scaled.
|
|
|
|
*/
|
|
|
|
tp->snd_wnd = ntohs(th->window);
|
|
|
|
tp->snd_wl1 = TCP_SKB_CB(skb)->seq;
|
|
|
|
tp->max_window = tp->snd_wnd;
|
|
|
|
|
2014-09-29 05:08:30 -06:00
|
|
|
tcp_ecn_rcv_syn(tp, th);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-03-20 18:53:41 -07:00
|
|
|
tcp_mtup_init(sk);
|
2005-12-14 00:26:10 -07:00
|
|
|
tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
|
2005-04-16 16:20:36 -06:00
|
|
|
tcp_initialize_rcv_mss(sk);
|
|
|
|
|
|
|
|
tcp_send_synack(sk);
|
|
|
|
#if 0
|
|
|
|
/* Note, we could accept data and URG from this segment.
|
2012-08-31 06:29:13 -06:00
|
|
|
* There are no obstacles to make this (except that we must
|
|
|
|
* either change tcp_recvmsg() to prevent it from returning data
|
|
|
|
* before 3WHS completes per RFC793, or employ TCP Fast Open).
|
2005-04-16 16:20:36 -06:00
|
|
|
*
|
|
|
|
* However, if we ignore data in ACKless segments sometimes,
|
|
|
|
* we have no reasons to accept it sometimes.
|
|
|
|
* Also, seems the code doing it in step6 of tcp_rcv_state_process
|
|
|
|
* is not flawless. So, discard packet for sanity.
|
|
|
|
* Uncomment this return to process the data.
|
|
|
|
*/
|
|
|
|
return -1;
|
|
|
|
#else
|
|
|
|
goto discard;
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
/* "fifth, if neither of the SYN or RST bits is set then
|
|
|
|
* drop the segment and return."
|
|
|
|
*/
|
|
|
|
|
|
|
|
discard_and_undo:
|
|
|
|
tcp_clear_options(&tp->rx_opt);
|
|
|
|
tp->rx_opt.mss_clamp = saved_clamp;
|
|
|
|
goto discard;
|
|
|
|
|
|
|
|
reset_and_undo:
|
|
|
|
tcp_clear_options(&tp->rx_opt);
|
|
|
|
tp->rx_opt.mss_clamp = saved_clamp;
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2019-04-29 16:46:19 -06:00
|
|
|
static void tcp_rcv_synrecv_state_fastopen(struct sock *sk)
|
|
|
|
{
|
|
|
|
tcp_try_undo_loss(sk, false);
|
2019-05-13 11:32:05 -06:00
|
|
|
|
|
|
|
/* Reset rtx states to prevent spurious retransmits_timed_out() */
|
|
|
|
tcp_sk(sk)->retrans_stamp = 0;
|
2019-04-29 16:46:19 -06:00
|
|
|
inet_csk(sk)->icsk_retransmits = 0;
|
|
|
|
|
|
|
|
/* Once we leave TCP_SYN_RECV or TCP_FIN_WAIT_1,
|
|
|
|
* we no longer need req so release it.
|
|
|
|
*/
|
|
|
|
reqsk_fastopen_remove(sk, tcp_sk(sk)->fastopen_rsk, false);
|
|
|
|
|
|
|
|
/* Re-arm the timer because data may have been sent out.
|
|
|
|
* This is similar to the regular data transmission case
|
|
|
|
* when new data has just been ack'ed.
|
|
|
|
*
|
|
|
|
* (TFO) - we could try to be more aggressive and
|
|
|
|
* retransmitting any data sooner based on when they
|
|
|
|
* are sent out.
|
|
|
|
*/
|
|
|
|
tcp_rearm_rto(sk);
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* This function implements the receiving procedure of RFC 793 for
|
2007-02-09 07:24:47 -07:00
|
|
|
* all states except ESTABLISHED and TIME_WAIT.
|
2005-04-16 16:20:36 -06:00
|
|
|
* It's called from both tcp_v4_rcv and tcp_v6_rcv and should be
|
|
|
|
* address independent.
|
|
|
|
*/
|
2007-02-09 07:24:47 -07:00
|
|
|
|
2015-09-29 08:42:41 -06:00
|
|
|
int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-12-14 00:15:52 -07:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2015-09-29 08:42:41 -06:00
|
|
|
const struct tcphdr *th = tcp_hdr(skb);
|
2012-08-31 06:29:13 -06:00
|
|
|
struct request_sock *req;
|
2005-04-16 16:20:36 -06:00
|
|
|
int queued = 0;
|
2013-05-24 09:03:54 -06:00
|
|
|
bool acceptable;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
switch (sk->sk_state) {
|
|
|
|
case TCP_CLOSE:
|
|
|
|
goto discard;
|
|
|
|
|
|
|
|
case TCP_LISTEN:
|
2007-03-08 21:45:19 -07:00
|
|
|
if (th->ack)
|
2005-04-16 16:20:36 -06:00
|
|
|
return 1;
|
|
|
|
|
2007-03-08 21:45:19 -07:00
|
|
|
if (th->rst)
|
2005-04-16 16:20:36 -06:00
|
|
|
goto discard;
|
|
|
|
|
2007-03-08 21:45:19 -07:00
|
|
|
if (th->syn) {
|
2011-12-02 16:41:42 -07:00
|
|
|
if (th->fin)
|
|
|
|
goto discard;
|
2017-03-01 09:39:49 -07:00
|
|
|
/* It is possible that we process SYN packets from backlog,
|
2018-10-01 16:02:26 -06:00
|
|
|
* so we need to make sure to disable BH and RCU right there.
|
2017-03-01 09:39:49 -07:00
|
|
|
*/
|
2018-10-01 16:02:26 -06:00
|
|
|
rcu_read_lock();
|
2017-03-01 09:39:49 -07:00
|
|
|
local_bh_disable();
|
|
|
|
acceptable = icsk->icsk_af_ops->conn_request(sk, skb) >= 0;
|
|
|
|
local_bh_enable();
|
2018-10-01 16:02:26 -06:00
|
|
|
rcu_read_unlock();
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2017-03-01 09:39:49 -07:00
|
|
|
if (!acceptable)
|
|
|
|
return 1;
|
2016-04-21 23:13:01 -06:00
|
|
|
consume_skb(skb);
|
2007-01-23 21:15:06 -07:00
|
|
|
return 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
goto discard;
|
|
|
|
|
|
|
|
case TCP_SYN_SENT:
|
2016-04-13 23:05:40 -06:00
|
|
|
tp->rx_opt.saw_tstamp = 0;
|
2017-05-16 15:00:14 -06:00
|
|
|
tcp_mstamp_refresh(tp);
|
2015-09-29 08:42:40 -06:00
|
|
|
queued = tcp_rcv_synsent_state_process(sk, skb, th);
|
2005-04-16 16:20:36 -06:00
|
|
|
if (queued >= 0)
|
|
|
|
return queued;
|
|
|
|
|
|
|
|
/* Do step6 onward by hand. */
|
|
|
|
tcp_urg(sk, skb, th);
|
|
|
|
__kfree_skb(skb);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
tcp_data_snd_check(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2017-05-16 15:00:14 -06:00
|
|
|
tcp_mstamp_refresh(tp);
|
2016-04-13 23:05:40 -06:00
|
|
|
tp->rx_opt.saw_tstamp = 0;
|
2012-08-31 06:29:13 -06:00
|
|
|
req = tp->fastopen_rsk;
|
2015-04-03 02:17:27 -06:00
|
|
|
if (req) {
|
2018-02-13 07:14:12 -07:00
|
|
|
bool req_stolen;
|
|
|
|
|
2012-10-22 05:26:36 -06:00
|
|
|
WARN_ON_ONCE(sk->sk_state != TCP_SYN_RECV &&
|
2012-08-31 06:29:13 -06:00
|
|
|
sk->sk_state != TCP_FIN_WAIT1);
|
|
|
|
|
2018-02-13 07:14:12 -07:00
|
|
|
if (!tcp_check_req(sk, skb, req, true, &req_stolen))
|
2012-08-31 06:29:13 -06:00
|
|
|
goto discard;
|
2012-09-21 22:18:57 -06:00
|
|
|
}
|
2012-12-26 05:44:34 -07:00
|
|
|
|
tcp: Restore RFC5961-compliant behavior for SYN packets
Commit c3ae62af8e755 ("tcp: should drop incoming frames without ACK
flag set") was created to mitigate a security vulnerability in which a
local attacker is able to inject data into locally-opened sockets by
using TCP protocol statistics in procfs to quickly find the correct
sequence number.
This broke the RFC5961 requirement to send a challenge ACK in response
to spurious RST packets, which was subsequently fixed by commit
7b514a886ba50 ("tcp: accept RST without ACK flag").
Unfortunately, the RFC5961 requirement that spurious SYN packets be
handled in a similar manner remains broken.
RFC5961 section 4 states that:
... the handling of the SYN in the synchronized state SHOULD be
performed as follows:
1) If the SYN bit is set, irrespective of the sequence number, TCP
MUST send an ACK (also referred to as challenge ACK) to the remote
peer:
<SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>
After sending the acknowledgment, TCP MUST drop the unacceptable
segment and stop processing further.
By sending an ACK, the remote peer is challenged to confirm the loss
of the previous connection and the request to start a new connection.
A legitimate peer, after restart, would not have a TCB in the
synchronized state. Thus, when the ACK arrives, the peer should send
a RST segment back with the sequence number derived from the ACK
field that caused the RST.
This RST will confirm that the remote peer has indeed closed the
previous connection. Upon receipt of a valid RST, the local TCP
endpoint MUST terminate its connection. The local TCP endpoint
should then rely on SYN retransmission from the remote end to
re-establish the connection.
This patch lets SYN packets through the discard added in c3ae62af8e755,
so that spurious SYN packets are properly dealt with as per the RFC.
The challenge ACK is sent unconditionally and is rate-limited, so the
original vulnerability is not reintroduced by this patch.
Signed-off-by: Calvin Owens <calvinowens@fb.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-20 16:09:53 -07:00
|
|
|
if (!th->ack && !th->rst && !th->syn)
|
2012-12-26 05:44:34 -07:00
|
|
|
goto discard;
|
|
|
|
|
2012-09-21 22:18:57 -06:00
|
|
|
if (!tcp_validate_incoming(sk, skb, th, 0))
|
2012-07-16 19:41:30 -06:00
|
|
|
return 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* step 5: check the ACK field */
|
2017-08-30 11:24:58 -06:00
|
|
|
acceptable = tcp_ack(sk, skb, FLAG_SLOWPATH |
|
|
|
|
FLAG_UPDATE_TS_RECENT |
|
tcp: better validation of received ack sequences
Paul Fiterau Brostean reported :
<quote>
Linux TCP stack we analyze exhibits behavior that seems odd to me.
The scenario is as follows (all packets have empty payloads, no window
scaling, rcv/snd window size should not be a factor):
TEST HARNESS (CLIENT) LINUX SERVER
1. - LISTEN (server listen,
then accepts)
2. - --> <SEQ=100><CTL=SYN> --> SYN-RECEIVED
3. - <-- <SEQ=300><ACK=101><CTL=SYN,ACK> <-- SYN-RECEIVED
4. - --> <SEQ=101><ACK=301><CTL=ACK> --> ESTABLISHED
5. - <-- <SEQ=301><ACK=101><CTL=FIN,ACK> <-- FIN WAIT-1 (server
opts to close the data connection calling "close" on the connection
socket)
6. - --> <SEQ=101><ACK=99999><CTL=FIN,ACK> --> CLOSING (client sends
FIN,ACK with not yet sent acknowledgement number)
7. - <-- <SEQ=302><ACK=102><CTL=ACK> <-- CLOSING (ACK is 102
instead of 101, why?)
... (silence from CLIENT)
8. - <-- <SEQ=301><ACK=102><CTL=FIN,ACK> <-- CLOSING
(retransmission, again ACK is 102)
Now, note that packet 6 while having the expected sequence number,
acknowledges something that wasn't sent by the server. So I would
expect
the packet to maybe prompt an ACK response from the server, and then be
ignored. Yet it is not ignored and actually leads to an increase of the
acknowledgement number in the server's retransmission of the FIN,ACK
packet. The explanation I found is that the FIN in packet 6 was
processed, despite the acknowledgement number being unacceptable.
Further experiments indeed show that the server processes this FIN,
transitioning to CLOSING, then on receiving an ACK for the FIN it had
send in packet 5, the server (or better said connection) transitions
from CLOSING to TIME_WAIT (as signaled by netstat).
</quote>
Indeed, tcp_rcv_state_process() calls tcp_ack() but
does not exploit the @acceptable status but for TCP_SYN_RECV
state.
What we want here is to send a challenge ACK, if not in TCP_SYN_RECV
state. TCP_FIN_WAIT1 state is not the only state we should fix.
Add a FLAG_NO_CHALLENGE_ACK so that tcp_rcv_state_process()
can choose to send a challenge ACK and discard the packet instead
of wrongly change socket state.
With help from Neal Cardwell.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Paul Fiterau Brostean <p.fiterau-brostean@science.ru.nl>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-05-23 16:24:46 -06:00
|
|
|
FLAG_NO_CHALLENGE_ACK) > 0;
|
2012-08-31 06:29:13 -06:00
|
|
|
|
tcp: better validation of received ack sequences
Paul Fiterau Brostean reported :
<quote>
Linux TCP stack we analyze exhibits behavior that seems odd to me.
The scenario is as follows (all packets have empty payloads, no window
scaling, rcv/snd window size should not be a factor):
TEST HARNESS (CLIENT) LINUX SERVER
1. - LISTEN (server listen,
then accepts)
2. - --> <SEQ=100><CTL=SYN> --> SYN-RECEIVED
3. - <-- <SEQ=300><ACK=101><CTL=SYN,ACK> <-- SYN-RECEIVED
4. - --> <SEQ=101><ACK=301><CTL=ACK> --> ESTABLISHED
5. - <-- <SEQ=301><ACK=101><CTL=FIN,ACK> <-- FIN WAIT-1 (server
opts to close the data connection calling "close" on the connection
socket)
6. - --> <SEQ=101><ACK=99999><CTL=FIN,ACK> --> CLOSING (client sends
FIN,ACK with not yet sent acknowledgement number)
7. - <-- <SEQ=302><ACK=102><CTL=ACK> <-- CLOSING (ACK is 102
instead of 101, why?)
... (silence from CLIENT)
8. - <-- <SEQ=301><ACK=102><CTL=FIN,ACK> <-- CLOSING
(retransmission, again ACK is 102)
Now, note that packet 6 while having the expected sequence number,
acknowledges something that wasn't sent by the server. So I would
expect
the packet to maybe prompt an ACK response from the server, and then be
ignored. Yet it is not ignored and actually leads to an increase of the
acknowledgement number in the server's retransmission of the FIN,ACK
packet. The explanation I found is that the FIN in packet 6 was
processed, despite the acknowledgement number being unacceptable.
Further experiments indeed show that the server processes this FIN,
transitioning to CLOSING, then on receiving an ACK for the FIN it had
send in packet 5, the server (or better said connection) transitions
from CLOSING to TIME_WAIT (as signaled by netstat).
</quote>
Indeed, tcp_rcv_state_process() calls tcp_ack() but
does not exploit the @acceptable status but for TCP_SYN_RECV
state.
What we want here is to send a challenge ACK, if not in TCP_SYN_RECV
state. TCP_FIN_WAIT1 state is not the only state we should fix.
Add a FLAG_NO_CHALLENGE_ACK so that tcp_rcv_state_process()
can choose to send a challenge ACK and discard the packet instead
of wrongly change socket state.
With help from Neal Cardwell.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Paul Fiterau Brostean <p.fiterau-brostean@science.ru.nl>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-05-23 16:24:46 -06:00
|
|
|
if (!acceptable) {
|
|
|
|
if (sk->sk_state == TCP_SYN_RECV)
|
|
|
|
return 1; /* send one RST */
|
|
|
|
tcp_send_challenge_ack(sk, skb);
|
|
|
|
goto discard;
|
|
|
|
}
|
2013-05-24 09:03:54 -06:00
|
|
|
switch (sk->sk_state) {
|
|
|
|
case TCP_SYN_RECV:
|
2018-04-18 00:18:46 -06:00
|
|
|
tp->delivered++; /* SYN-ACK delivery isn't tracked in tcp_ack */
|
2015-09-18 12:36:14 -06:00
|
|
|
if (!tp->srtt_us)
|
|
|
|
tcp_synack_rtt_meas(sk, req);
|
|
|
|
|
2013-05-24 12:36:13 -06:00
|
|
|
if (req) {
|
2019-04-29 16:46:19 -06:00
|
|
|
tcp_rcv_synrecv_state_fastopen(sk);
|
2013-05-24 12:36:13 -06:00
|
|
|
} else {
|
2019-04-29 16:46:16 -06:00
|
|
|
tcp_try_undo_spurious_syn(sk);
|
|
|
|
tp->retrans_stamp = 0;
|
tcp: uniform the set up of sockets after successful connection
Currently in the TCP code, the initialization sequence for cached
metrics, congestion control, BPF, etc, after successful connection
is very inconsistent. This introduces inconsistent bevhavior and is
prone to bugs. The current call sequence is as follows:
(1) for active case (tcp_finish_connect() case):
tcp_mtup_init(sk);
icsk->icsk_af_ops->rebuild_header(sk);
tcp_init_metrics(sk);
tcp_call_bpf(sk, BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB);
tcp_init_congestion_control(sk);
tcp_init_buffer_space(sk);
(2) for passive case (tcp_rcv_state_process() TCP_SYN_RECV case):
icsk->icsk_af_ops->rebuild_header(sk);
tcp_call_bpf(sk, BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB);
tcp_init_congestion_control(sk);
tcp_mtup_init(sk);
tcp_init_buffer_space(sk);
tcp_init_metrics(sk);
(3) for TFO passive case (tcp_fastopen_create_child()):
inet_csk(child)->icsk_af_ops->rebuild_header(child);
tcp_init_congestion_control(child);
tcp_mtup_init(child);
tcp_init_metrics(child);
tcp_call_bpf(child, BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB);
tcp_init_buffer_space(child);
This commit uniforms the above functions to have the following sequence:
tcp_mtup_init(sk);
icsk->icsk_af_ops->rebuild_header(sk);
tcp_init_metrics(sk);
tcp_call_bpf(sk, BPF_SOCK_OPS_ACTIVE/PASSIVE_ESTABLISHED_CB);
tcp_init_congestion_control(sk);
tcp_init_buffer_space(sk);
This sequence is the same as the (1) active case. We pick this sequence
because this order correctly allows BPF to override the settings
including congestion control module and initial cwnd, etc from
the route, and then allows the CC module to see those settings.
Suggested-by: Neal Cardwell <ncardwell@google.com>
Tested-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Wei Wang <weiwan@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-10-04 11:03:44 -06:00
|
|
|
tcp_init_transfer(sk, BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB);
|
2013-05-24 12:36:13 -06:00
|
|
|
tp->copied_seq = tp->rcv_nxt;
|
|
|
|
}
|
|
|
|
smp_mb();
|
|
|
|
tcp_set_state(sk, TCP_ESTABLISHED);
|
|
|
|
sk->sk_state_change(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2013-05-24 12:36:13 -06:00
|
|
|
/* Note, that this wakeup is only for marginal crossed SYN case.
|
|
|
|
* Passively open sockets are not waked up, because
|
|
|
|
* sk->sk_sleep == NULL and sk->sk_socket == NULL.
|
|
|
|
*/
|
|
|
|
if (sk->sk_socket)
|
|
|
|
sk_wake_async(sk, SOCK_WAKE_IO, POLL_OUT);
|
|
|
|
|
|
|
|
tp->snd_una = TCP_SKB_CB(skb)->ack_seq;
|
|
|
|
tp->snd_wnd = ntohs(th->window) << tp->rx_opt.snd_wscale;
|
|
|
|
tcp_init_wl(tp, TCP_SKB_CB(skb)->seq);
|
|
|
|
|
|
|
|
if (tp->rx_opt.tstamp_ok)
|
|
|
|
tp->advmss -= TCPOLEN_TSTAMP_ALIGNED;
|
|
|
|
|
2016-09-19 21:39:21 -06:00
|
|
|
if (!inet_csk(sk)->icsk_ca_ops->cong_control)
|
|
|
|
tcp_update_pacing_rate(sk);
|
tcp: initialize passive-side sk_pacing_rate after 3WHS
For passive TCP connections, upon receiving the ACK that completes the
3WHS, make sure we set our pacing rate after we get our first RTT
sample.
On passive TCP connections, when we receive the ACK completing the
3WHS we do not take an RTT sample in tcp_ack(), but rather in
tcp_synack_rtt_meas(). So upon receiving the ACK that completes the
3WHS, tcp_ack() leaves sk_pacing_rate at its initial value.
Originally the initial sk_pacing_rate value was 0, so passive-side
connections defaulted to sysctl_tcp_min_tso_segs (2 segs) in skbuffs
made in the first RTT. With a default initial cwnd of 10 packets, this
happened to be correct for RTTs 5ms or bigger, so it was hard to
see problems in WAN or emulated WAN testing.
Since 7eec4174ff ("pkt_sched: fq: fix non TCP flows pacing"), the
initial sk_pacing_rate is 0xffffffff. So after that change, passive
TCP connections were keeping this value (and using large numbers of
segments per skbuff) until receiving an ACK for data.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-21 13:40:19 -06:00
|
|
|
|
2013-05-24 12:36:13 -06:00
|
|
|
/* Prevent spurious tcp_cwnd_restart() on first data packet */
|
2017-05-16 15:00:03 -06:00
|
|
|
tp->lsndtime = tcp_jiffies32;
|
2013-05-24 12:36:13 -06:00
|
|
|
|
|
|
|
tcp_initialize_rcv_mss(sk);
|
2017-08-30 11:24:58 -06:00
|
|
|
tcp_fast_path_on(tp);
|
2013-05-24 09:03:54 -06:00
|
|
|
break;
|
|
|
|
|
2013-05-24 12:06:58 -06:00
|
|
|
case TCP_FIN_WAIT1: {
|
|
|
|
int tmo;
|
|
|
|
|
2019-04-29 16:46:19 -06:00
|
|
|
if (req)
|
|
|
|
tcp_rcv_synrecv_state_fastopen(sk);
|
|
|
|
|
2013-05-24 12:06:58 -06:00
|
|
|
if (tp->snd_una != tp->write_seq)
|
|
|
|
break;
|
2013-05-24 09:03:54 -06:00
|
|
|
|
2013-05-24 12:06:58 -06:00
|
|
|
tcp_set_state(sk, TCP_FIN_WAIT2);
|
|
|
|
sk->sk_shutdown |= SEND_SHUTDOWN;
|
2013-05-24 09:03:54 -06:00
|
|
|
|
2017-02-06 14:14:14 -07:00
|
|
|
sk_dst_confirm(sk);
|
2013-05-24 09:03:54 -06:00
|
|
|
|
2013-05-24 12:06:58 -06:00
|
|
|
if (!sock_flag(sk, SOCK_DEAD)) {
|
|
|
|
/* Wake up lingering close() */
|
|
|
|
sk->sk_state_change(sk);
|
|
|
|
break;
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
net/tcp_fastopen: Disable active side TFO in certain scenarios
Middlebox firewall issues can potentially cause server's data being
blackholed after a successful 3WHS using TFO. Following are the related
reports from Apple:
https://www.nanog.org/sites/default/files/Paasch_Network_Support.pdf
Slide 31 identifies an issue where the client ACK to the server's data
sent during a TFO'd handshake is dropped.
C ---> syn-data ---> S
C <--- syn/ack ----- S
C (accept & write)
C <---- data ------- S
C ----- ACK -> X S
[retry and timeout]
https://www.ietf.org/proceedings/94/slides/slides-94-tcpm-13.pdf
Slide 5 shows a similar situation that the server's data gets dropped
after 3WHS.
C ---- syn-data ---> S
C <--- syn/ack ----- S
C ---- ack --------> S
S (accept & write)
C? X <- data ------ S
[retry and timeout]
This is the worst failure b/c the client can not detect such behavior to
mitigate the situation (such as disabling TFO). Failing to proceed, the
application (e.g., SSL library) may simply timeout and retry with TFO
again, and the process repeats indefinitely.
The proposed solution is to disable active TFO globally under the
following circumstances:
1. client side TFO socket detects out of order FIN
2. client side TFO socket receives out of order RST
We disable active side TFO globally for 1hr at first. Then if it
happens again, we disable it for 2h, then 4h, 8h, ...
And we reset the timeout to 1hr if a client side TFO sockets not opened
on loopback has successfully received data segs from server.
And we examine this condition during close().
The rational behind it is that when such firewall issue happens,
application running on the client should eventually close the socket as
it is not able to get the data it is expecting. Or application running
on the server should close the socket as it is not able to receive any
response from client.
In both cases, out of order FIN or RST will get received on the client
given that the firewall will not block them as no data are in those
frames.
And we want to disable active TFO globally as it helps if the middle box
is very close to the client and most of the connections are likely to
fail.
Also, add a debug sysctl:
tcp_fastopen_blackhole_detect_timeout_sec:
the initial timeout to use when firewall blackhole issue happens.
This can be set and read.
When setting it to 0, it means to disable the active disable logic.
Signed-off-by: Wei Wang <weiwan@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-04-20 15:45:46 -06:00
|
|
|
if (tp->linger2 < 0) {
|
|
|
|
tcp_done(sk);
|
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
if (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
|
|
|
|
after(TCP_SKB_CB(skb)->end_seq - th->fin, tp->rcv_nxt)) {
|
|
|
|
/* Receive out of order FIN after close() */
|
|
|
|
if (tp->syn_fastopen && th->fin)
|
2017-04-20 15:45:47 -06:00
|
|
|
tcp_fastopen_active_disable(sk);
|
2013-05-24 12:06:58 -06:00
|
|
|
tcp_done(sk);
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
|
2013-05-24 12:06:58 -06:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
tmo = tcp_fin_time(sk);
|
|
|
|
if (tmo > TCP_TIMEWAIT_LEN) {
|
|
|
|
inet_csk_reset_keepalive_timer(sk, tmo - TCP_TIMEWAIT_LEN);
|
|
|
|
} else if (th->fin || sock_owned_by_user(sk)) {
|
|
|
|
/* Bad case. We could lose such FIN otherwise.
|
|
|
|
* It is not a big problem, but it looks confusing
|
|
|
|
* and not so rare event. We still can lose it now,
|
|
|
|
* if it spins in bh_lock_sock(), but it is really
|
|
|
|
* marginal case.
|
|
|
|
*/
|
|
|
|
inet_csk_reset_keepalive_timer(sk, tmo);
|
|
|
|
} else {
|
|
|
|
tcp_time_wait(sk, TCP_FIN_WAIT2, tmo);
|
|
|
|
goto discard;
|
2013-05-24 09:03:54 -06:00
|
|
|
}
|
|
|
|
break;
|
2013-05-24 12:06:58 -06:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2013-05-24 09:03:54 -06:00
|
|
|
case TCP_CLOSING:
|
|
|
|
if (tp->snd_una == tp->write_seq) {
|
|
|
|
tcp_time_wait(sk, TCP_TIME_WAIT, 0);
|
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
break;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2013-05-24 09:03:54 -06:00
|
|
|
case TCP_LAST_ACK:
|
|
|
|
if (tp->snd_una == tp->write_seq) {
|
|
|
|
tcp_update_metrics(sk);
|
|
|
|
tcp_done(sk);
|
|
|
|
goto discard;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2013-05-24 09:03:54 -06:00
|
|
|
break;
|
2012-12-26 05:44:34 -07:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* step 6: check the URG bit */
|
|
|
|
tcp_urg(sk, skb, th);
|
|
|
|
|
|
|
|
/* step 7: process the segment text */
|
|
|
|
switch (sk->sk_state) {
|
|
|
|
case TCP_CLOSE_WAIT:
|
|
|
|
case TCP_CLOSING:
|
|
|
|
case TCP_LAST_ACK:
|
|
|
|
if (!before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt))
|
|
|
|
break;
|
2017-10-16 14:48:55 -06:00
|
|
|
/* fall through */
|
2005-04-16 16:20:36 -06:00
|
|
|
case TCP_FIN_WAIT1:
|
|
|
|
case TCP_FIN_WAIT2:
|
|
|
|
/* RFC 793 says to queue data in these states,
|
2007-02-09 07:24:47 -07:00
|
|
|
* RFC 1122 says we MUST send a reset.
|
2005-04-16 16:20:36 -06:00
|
|
|
* BSD 4.4 also does reset.
|
|
|
|
*/
|
|
|
|
if (sk->sk_shutdown & RCV_SHUTDOWN) {
|
|
|
|
if (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
|
|
|
|
after(TCP_SKB_CB(skb)->end_seq - th->fin, tp->rcv_nxt)) {
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
|
2005-04-16 16:20:36 -06:00
|
|
|
tcp_reset(sk);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
/* Fall through */
|
2007-02-09 07:24:47 -07:00
|
|
|
case TCP_ESTABLISHED:
|
2005-04-16 16:20:36 -06:00
|
|
|
tcp_data_queue(sk, skb);
|
|
|
|
queued = 1;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* tcp_data could move socket to TIME-WAIT */
|
|
|
|
if (sk->sk_state != TCP_CLOSE) {
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-20 23:18:02 -06:00
|
|
|
tcp_data_snd_check(sk);
|
2005-04-16 16:20:36 -06:00
|
|
|
tcp_ack_snd_check(sk);
|
|
|
|
}
|
|
|
|
|
2007-02-09 07:24:47 -07:00
|
|
|
if (!queued) {
|
2005-04-16 16:20:36 -06:00
|
|
|
discard:
|
2016-04-01 09:52:19 -06:00
|
|
|
tcp_drop(sk, skb);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(tcp_rcv_state_process);
|
2014-06-25 08:10:02 -06:00
|
|
|
|
|
|
|
static inline void pr_drop_req(struct request_sock *req, __u16 port, int family)
|
|
|
|
{
|
|
|
|
struct inet_request_sock *ireq = inet_rsk(req);
|
|
|
|
|
|
|
|
if (family == AF_INET)
|
2014-11-11 11:59:17 -07:00
|
|
|
net_dbg_ratelimited("drop open request from %pI4/%u\n",
|
|
|
|
&ireq->ir_rmt_addr, port);
|
2014-06-28 12:20:54 -06:00
|
|
|
#if IS_ENABLED(CONFIG_IPV6)
|
|
|
|
else if (family == AF_INET6)
|
2014-11-11 11:59:17 -07:00
|
|
|
net_dbg_ratelimited("drop open request from %pI6/%u\n",
|
|
|
|
&ireq->ir_v6_rmt_addr, port);
|
2014-06-28 12:20:54 -06:00
|
|
|
#endif
|
2014-06-25 08:10:02 -06:00
|
|
|
}
|
|
|
|
|
2014-09-29 05:08:29 -06:00
|
|
|
/* RFC3168 : 6.1.1 SYN packets must not have ECT/ECN bits set
|
|
|
|
*
|
|
|
|
* If we receive a SYN packet with these bits set, it means a
|
|
|
|
* network is playing bad games with TOS bits. In order to
|
|
|
|
* avoid possible false congestion notifications, we disable
|
2014-10-29 17:05:06 -06:00
|
|
|
* TCP ECN negotiation.
|
2014-09-29 05:08:29 -06:00
|
|
|
*
|
|
|
|
* Exception: tcp_ca wants ECN. This is required for DCTCP
|
net: dctcp: loosen requirement to assert ECT(0) during 3WHS
One deployment requirement of DCTCP is to be able to run
in a DC setting along with TCP traffic. As Glenn Judd's
NSDI'15 paper "Attaining the Promise and Avoiding the Pitfalls
of TCP in the Datacenter" [1] (tba) explains, one way to
solve this on switch side is to split DCTCP and TCP traffic
in two queues per switch port based on the DSCP: one queue
soley intended for DCTCP traffic and one for non-DCTCP traffic.
For the DCTCP queue, there's the marking threshold K as
explained in commit e3118e8359bb ("net: tcp: add DCTCP congestion
control algorithm") for RED marking ECT(0) packets with CE.
For the non-DCTCP queue, there's f.e. a classic tail drop queue.
As already explained in e3118e8359bb, running DCTCP at scale
when not marking SYN/SYN-ACK packets with ECT(0) has severe
consequences as for non-ECT(0) packets, traversing the RED
marking DCTCP queue will result in a severe reduction of
connection probability.
This is due to the DCTCP queue being dominated by ECT(0) traffic
and switches handle non-ECT traffic in the RED marking queue
after passing K as drops, where K is usually a low watermark
in order to leave enough tailroom for bursts. Splitting DCTCP
traffic among several queues (ECN and non-ECN queue) is being
considered a terrible idea in the network community as it
splits single flows across multiple network paths.
Therefore, commit e3118e8359bb implements this on Linux as
ECT(0) marked traffic, as we argue that marking all packets
of a DCTCP flow is the only viable solution and also doesn't
speak against the draft.
However, recently, a DCTCP implementation for FreeBSD hit also
their mainline kernel [2]. In order to let them play well
together with Linux' DCTCP, we would need to loosen the
requirement that ECT(0) has to be asserted during the 3WHS as
not implemented in FreeBSD. This simplifies the ECN test and
lets DCTCP work together with FreeBSD.
Joint work with Daniel Borkmann.
[1] https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/judd
[2] https://github.com/freebsd/freebsd/commit/8ad879445281027858a7fa706d13e458095b595f
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Cc: Glenn Judd <glenn.judd@morganstanley.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-01-30 12:45:20 -07:00
|
|
|
* congestion control: Linux DCTCP asserts ECT on all packets,
|
|
|
|
* including SYN, which is most optimal solution; however,
|
|
|
|
* others, such as FreeBSD do not.
|
2019-04-03 07:49:42 -06:00
|
|
|
*
|
|
|
|
* Exception: At least one of the reserved bits of the TCP header (th->res1) is
|
|
|
|
* set, indicating the use of a future TCP extension (such as AccECN). See
|
|
|
|
* RFC8311 §4.3 which updates RFC3168 to allow the development of such
|
|
|
|
* extensions.
|
2014-09-29 05:08:29 -06:00
|
|
|
*/
|
|
|
|
static void tcp_ecn_create_request(struct request_sock *req,
|
|
|
|
const struct sk_buff *skb,
|
net: allow setting ecn via routing table
This patch allows to set ECN on a per-route basis in case the sysctl
tcp_ecn is not set to 1. In other words, when ECN is set for specific
routes, it provides a tcp_ecn=1 behaviour for that route while the rest
of the stack acts according to the global settings.
One can use 'ip route change dev $dev $net features ecn' to toggle this.
Having a more fine-grained per-route setting can be beneficial for various
reasons, for example, 1) within data centers, or 2) local ISPs may deploy
ECN support for their own video/streaming services [1], etc.
There was a recent measurement study/paper [2] which scanned the Alexa's
publicly available top million websites list from a vantage point in US,
Europe and Asia:
Half of the Alexa list will now happily use ECN (tcp_ecn=2, most likely
blamed to commit 255cac91c3 ("tcp: extend ECN sysctl to allow server-side
only ECN") ;)); the break in connectivity on-path was found is about
1 in 10,000 cases. Timeouts rather than receiving back RSTs were much
more common in the negotiation phase (and mostly seen in the Alexa
middle band, ranks around 50k-150k): from 12-thousand hosts on which
there _may_ be ECN-linked connection failures, only 79 failed with RST
when _not_ failing with RST when ECN is not requested.
It's unclear though, how much equipment in the wild actually marks CE
when buffers start to fill up.
We thought about a fallback to non-ECN for retransmitted SYNs as another
global option (which could perhaps one day be made default), but as Eric
points out, there's much more work needed to detect broken middleboxes.
Two examples Eric mentioned are buggy firewalls that accept only a single
SYN per flow, and middleboxes that successfully let an ECN flow establish,
but later mark CE for all packets (so cwnd converges to 1).
[1] http://www.ietf.org/proceedings/89/slides/slides-89-tsvarea-1.pdf, p.15
[2] http://ecn.ethz.ch/
Joint work with Daniel Borkmann.
Reference: http://thread.gmane.org/gmane.linux.network/335797
Suggested-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Daniel Borkmann <dborkman@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-03 09:35:03 -07:00
|
|
|
const struct sock *listen_sk,
|
|
|
|
const struct dst_entry *dst)
|
2014-09-29 05:08:29 -06:00
|
|
|
{
|
|
|
|
const struct tcphdr *th = tcp_hdr(skb);
|
|
|
|
const struct net *net = sock_net(listen_sk);
|
|
|
|
bool th_ecn = th->ece && th->cwr;
|
net: dctcp: loosen requirement to assert ECT(0) during 3WHS
One deployment requirement of DCTCP is to be able to run
in a DC setting along with TCP traffic. As Glenn Judd's
NSDI'15 paper "Attaining the Promise and Avoiding the Pitfalls
of TCP in the Datacenter" [1] (tba) explains, one way to
solve this on switch side is to split DCTCP and TCP traffic
in two queues per switch port based on the DSCP: one queue
soley intended for DCTCP traffic and one for non-DCTCP traffic.
For the DCTCP queue, there's the marking threshold K as
explained in commit e3118e8359bb ("net: tcp: add DCTCP congestion
control algorithm") for RED marking ECT(0) packets with CE.
For the non-DCTCP queue, there's f.e. a classic tail drop queue.
As already explained in e3118e8359bb, running DCTCP at scale
when not marking SYN/SYN-ACK packets with ECT(0) has severe
consequences as for non-ECT(0) packets, traversing the RED
marking DCTCP queue will result in a severe reduction of
connection probability.
This is due to the DCTCP queue being dominated by ECT(0) traffic
and switches handle non-ECT traffic in the RED marking queue
after passing K as drops, where K is usually a low watermark
in order to leave enough tailroom for bursts. Splitting DCTCP
traffic among several queues (ECN and non-ECN queue) is being
considered a terrible idea in the network community as it
splits single flows across multiple network paths.
Therefore, commit e3118e8359bb implements this on Linux as
ECT(0) marked traffic, as we argue that marking all packets
of a DCTCP flow is the only viable solution and also doesn't
speak against the draft.
However, recently, a DCTCP implementation for FreeBSD hit also
their mainline kernel [2]. In order to let them play well
together with Linux' DCTCP, we would need to loosen the
requirement that ECT(0) has to be asserted during the 3WHS as
not implemented in FreeBSD. This simplifies the ECN test and
lets DCTCP work together with FreeBSD.
Joint work with Daniel Borkmann.
[1] https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/judd
[2] https://github.com/freebsd/freebsd/commit/8ad879445281027858a7fa706d13e458095b595f
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Cc: Glenn Judd <glenn.judd@morganstanley.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-01-30 12:45:20 -07:00
|
|
|
bool ect, ecn_ok;
|
tcp: use dctcp if enabled on the route to the initiator
Currently, the following case doesn't use DCTCP, even if it should:
A responder has f.e. Cubic as system wide default, but for a specific
route to the initiating host, DCTCP is being set in RTAX_CC_ALGO. The
initiating host then uses DCTCP as congestion control, but since the
initiator sets ECT(0), tcp_ecn_create_request() doesn't set ecn_ok,
and we have to fall back to Reno after 3WHS completes.
We were thinking on how to solve this in a minimal, non-intrusive
way without bloating tcp_ecn_create_request() needlessly: lets cache
the CA ecn option flag in RTAX_FEATURES. In other words, when ECT(0)
is set on the SYN packet, set ecn_ok=1 iff route RTAX_FEATURES
contains the unexposed (internal-only) DST_FEATURE_ECN_CA. This allows
to only do a single metric feature lookup inside tcp_ecn_create_request().
Joint work with Florian Westphal.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-08-31 07:58:47 -06:00
|
|
|
u32 ecn_ok_dst;
|
2014-09-29 05:08:29 -06:00
|
|
|
|
|
|
|
if (!th_ecn)
|
|
|
|
return;
|
|
|
|
|
|
|
|
ect = !INET_ECN_is_not_ect(TCP_SKB_CB(skb)->ip_dsfield);
|
tcp: use dctcp if enabled on the route to the initiator
Currently, the following case doesn't use DCTCP, even if it should:
A responder has f.e. Cubic as system wide default, but for a specific
route to the initiating host, DCTCP is being set in RTAX_CC_ALGO. The
initiating host then uses DCTCP as congestion control, but since the
initiator sets ECT(0), tcp_ecn_create_request() doesn't set ecn_ok,
and we have to fall back to Reno after 3WHS completes.
We were thinking on how to solve this in a minimal, non-intrusive
way without bloating tcp_ecn_create_request() needlessly: lets cache
the CA ecn option flag in RTAX_FEATURES. In other words, when ECT(0)
is set on the SYN packet, set ecn_ok=1 iff route RTAX_FEATURES
contains the unexposed (internal-only) DST_FEATURE_ECN_CA. This allows
to only do a single metric feature lookup inside tcp_ecn_create_request().
Joint work with Florian Westphal.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-08-31 07:58:47 -06:00
|
|
|
ecn_ok_dst = dst_feature(dst, DST_FEATURE_ECN_MASK);
|
|
|
|
ecn_ok = net->ipv4.sysctl_tcp_ecn || ecn_ok_dst;
|
2014-09-29 05:08:29 -06:00
|
|
|
|
2019-04-03 07:49:42 -06:00
|
|
|
if (((!ect || th->res1) && ecn_ok) || tcp_ca_needs_ecn(listen_sk) ||
|
2017-06-30 21:02:49 -06:00
|
|
|
(ecn_ok_dst & DST_FEATURE_ECN_CA) ||
|
|
|
|
tcp_bpf_ca_needs_ecn((struct sock *)req))
|
2014-09-29 05:08:29 -06:00
|
|
|
inet_rsk(req)->ecn_ok = 1;
|
|
|
|
}
|
|
|
|
|
2015-03-16 22:06:19 -06:00
|
|
|
static void tcp_openreq_init(struct request_sock *req,
|
|
|
|
const struct tcp_options_received *rx_opt,
|
|
|
|
struct sk_buff *skb, const struct sock *sk)
|
|
|
|
{
|
|
|
|
struct inet_request_sock *ireq = inet_rsk(req);
|
|
|
|
|
2015-10-08 20:33:23 -06:00
|
|
|
req->rsk_rcv_wnd = 0; /* So that tcp_send_synack() knows! */
|
2015-03-16 22:06:19 -06:00
|
|
|
req->cookie_ts = 0;
|
|
|
|
tcp_rsk(req)->rcv_isn = TCP_SKB_CB(skb)->seq;
|
|
|
|
tcp_rsk(req)->rcv_nxt = TCP_SKB_CB(skb)->seq + 1;
|
2019-04-29 16:46:15 -06:00
|
|
|
tcp_rsk(req)->snt_synack = 0;
|
2015-03-16 22:06:19 -06:00
|
|
|
tcp_rsk(req)->last_oow_ack_time = 0;
|
|
|
|
req->mss = rx_opt->mss_clamp;
|
|
|
|
req->ts_recent = rx_opt->saw_tstamp ? rx_opt->rcv_tsval : 0;
|
|
|
|
ireq->tstamp_ok = rx_opt->tstamp_ok;
|
|
|
|
ireq->sack_ok = rx_opt->sack_ok;
|
|
|
|
ireq->snd_wscale = rx_opt->snd_wscale;
|
|
|
|
ireq->wscale_ok = rx_opt->wscale_ok;
|
|
|
|
ireq->acked = 0;
|
|
|
|
ireq->ecn_ok = 0;
|
|
|
|
ireq->ir_rmt_port = tcp_hdr(skb)->source;
|
|
|
|
ireq->ir_num = ntohs(tcp_hdr(skb)->dest);
|
|
|
|
ireq->ir_mark = inet_request_mark(sk, skb);
|
2017-10-25 03:01:45 -06:00
|
|
|
#if IS_ENABLED(CONFIG_SMC)
|
|
|
|
ireq->smc_ok = rx_opt->smc_ok;
|
|
|
|
#endif
|
2015-03-16 22:06:19 -06:00
|
|
|
}
|
|
|
|
|
2015-03-17 19:32:27 -06:00
|
|
|
struct request_sock *inet_reqsk_alloc(const struct request_sock_ops *ops,
|
2015-10-04 22:08:11 -06:00
|
|
|
struct sock *sk_listener,
|
|
|
|
bool attach_listener)
|
2015-03-17 19:32:27 -06:00
|
|
|
{
|
2015-10-04 22:08:11 -06:00
|
|
|
struct request_sock *req = reqsk_alloc(ops, sk_listener,
|
|
|
|
attach_listener);
|
2015-03-17 19:32:27 -06:00
|
|
|
|
|
|
|
if (req) {
|
|
|
|
struct inet_request_sock *ireq = inet_rsk(req);
|
|
|
|
|
2017-10-20 10:04:13 -06:00
|
|
|
ireq->ireq_opt = NULL;
|
2016-06-27 13:05:28 -06:00
|
|
|
#if IS_ENABLED(CONFIG_IPV6)
|
|
|
|
ireq->pktopts = NULL;
|
|
|
|
#endif
|
2018-04-24 09:07:45 -06:00
|
|
|
atomic64_set(&ireq->ir_cookie, 0);
|
2015-03-17 19:32:27 -06:00
|
|
|
ireq->ireq_state = TCP_NEW_SYN_RECV;
|
|
|
|
write_pnet(&ireq->ireq_net, sock_net(sk_listener));
|
2015-03-24 22:45:56 -06:00
|
|
|
ireq->ireq_family = sk_listener->sk_family;
|
2015-03-17 19:32:27 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
return req;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(inet_reqsk_alloc);
|
|
|
|
|
2015-03-25 16:08:47 -06:00
|
|
|
/*
|
|
|
|
* Return true if a syncookie should be sent
|
|
|
|
*/
|
2015-09-29 08:42:51 -06:00
|
|
|
static bool tcp_syn_flood_action(const struct sock *sk,
|
2015-03-25 16:08:47 -06:00
|
|
|
const struct sk_buff *skb,
|
|
|
|
const char *proto)
|
|
|
|
{
|
2015-10-02 12:43:25 -06:00
|
|
|
struct request_sock_queue *queue = &inet_csk(sk)->icsk_accept_queue;
|
2015-03-25 16:08:47 -06:00
|
|
|
const char *msg = "Dropping request";
|
|
|
|
bool want_cookie = false;
|
2016-02-03 00:46:51 -07:00
|
|
|
struct net *net = sock_net(sk);
|
2015-03-25 16:08:47 -06:00
|
|
|
|
|
|
|
#ifdef CONFIG_SYN_COOKIES
|
2016-02-03 00:46:51 -07:00
|
|
|
if (net->ipv4.sysctl_tcp_syncookies) {
|
2015-03-25 16:08:47 -06:00
|
|
|
msg = "Sending cookies";
|
|
|
|
want_cookie = true;
|
2016-04-27 17:44:39 -06:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPREQQFULLDOCOOKIES);
|
2015-03-25 16:08:47 -06:00
|
|
|
} else
|
|
|
|
#endif
|
2016-04-27 17:44:39 -06:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPREQQFULLDROP);
|
2015-03-25 16:08:47 -06:00
|
|
|
|
2015-10-02 12:43:25 -06:00
|
|
|
if (!queue->synflood_warned &&
|
2016-02-03 00:46:51 -07:00
|
|
|
net->ipv4.sysctl_tcp_syncookies != 2 &&
|
2015-10-02 12:43:25 -06:00
|
|
|
xchg(&queue->synflood_warned, 1) == 0)
|
2018-09-09 17:12:12 -06:00
|
|
|
net_info_ratelimited("%s: Possible SYN flooding on port %d. %s. Check SNMP counters.\n",
|
|
|
|
proto, ntohs(tcp_hdr(skb)->dest), msg);
|
2015-09-29 08:42:51 -06:00
|
|
|
|
2015-03-25 16:08:47 -06:00
|
|
|
return want_cookie;
|
|
|
|
}
|
|
|
|
|
2015-05-03 22:34:46 -06:00
|
|
|
static void tcp_reqsk_record_syn(const struct sock *sk,
|
|
|
|
struct request_sock *req,
|
|
|
|
const struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
if (tcp_sk(sk)->save_syn) {
|
|
|
|
u32 len = skb_network_header_len(skb) + tcp_hdrlen(skb);
|
|
|
|
u32 *copy;
|
|
|
|
|
|
|
|
copy = kmalloc(len + sizeof(u32), GFP_ATOMIC);
|
|
|
|
if (copy) {
|
|
|
|
copy[0] = len;
|
|
|
|
memcpy(©[1], skb_network_header(skb), len);
|
|
|
|
req->saved_syn = copy;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2014-06-25 08:10:02 -06:00
|
|
|
int tcp_conn_request(struct request_sock_ops *rsk_ops,
|
|
|
|
const struct tcp_request_sock_ops *af_ops,
|
|
|
|
struct sock *sk, struct sk_buff *skb)
|
|
|
|
{
|
2015-09-24 18:16:05 -06:00
|
|
|
struct tcp_fastopen_cookie foc = { .len = -1 };
|
|
|
|
__u32 isn = TCP_SKB_CB(skb)->tcp_tw_isn;
|
2014-06-25 08:10:02 -06:00
|
|
|
struct tcp_options_received tmp_opt;
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2016-02-03 00:46:51 -07:00
|
|
|
struct net *net = sock_net(sk);
|
2015-09-24 18:16:05 -06:00
|
|
|
struct sock *fastopen_sk = NULL;
|
|
|
|
struct request_sock *req;
|
|
|
|
bool want_cookie = false;
|
2017-08-29 16:16:01 -06:00
|
|
|
struct dst_entry *dst;
|
2014-06-25 08:10:02 -06:00
|
|
|
struct flowi fl;
|
|
|
|
|
|
|
|
/* TW buckets are converted to open requests without
|
|
|
|
* limitations, they conserve resources and peer is
|
|
|
|
* evidently real one.
|
|
|
|
*/
|
2016-02-03 00:46:51 -07:00
|
|
|
if ((net->ipv4.sysctl_tcp_syncookies == 2 ||
|
2014-06-25 08:10:02 -06:00
|
|
|
inet_csk_reqsk_queue_is_full(sk)) && !isn) {
|
|
|
|
want_cookie = tcp_syn_flood_action(sk, skb, rsk_ops->slab_name);
|
|
|
|
if (!want_cookie)
|
|
|
|
goto drop;
|
|
|
|
}
|
|
|
|
|
2016-10-26 10:27:57 -06:00
|
|
|
if (sk_acceptq_is_full(sk)) {
|
2016-04-29 15:16:47 -06:00
|
|
|
NET_INC_STATS(sock_net(sk), LINUX_MIB_LISTENOVERFLOWS);
|
2014-06-25 08:10:02 -06:00
|
|
|
goto drop;
|
|
|
|
}
|
|
|
|
|
2015-10-04 22:08:11 -06:00
|
|
|
req = inet_reqsk_alloc(rsk_ops, sk, !want_cookie);
|
2014-06-25 08:10:02 -06:00
|
|
|
if (!req)
|
|
|
|
goto drop;
|
|
|
|
|
|
|
|
tcp_rsk(req)->af_specific = af_ops;
|
2016-12-01 03:32:06 -07:00
|
|
|
tcp_rsk(req)->ts_off = 0;
|
2014-06-25 08:10:02 -06:00
|
|
|
|
|
|
|
tcp_clear_options(&tmp_opt);
|
|
|
|
tmp_opt.mss_clamp = af_ops->mss_clamp;
|
|
|
|
tmp_opt.user_mss = tp->rx_opt.user_mss;
|
2017-06-07 11:34:36 -06:00
|
|
|
tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0,
|
|
|
|
want_cookie ? NULL : &foc);
|
2014-06-25 08:10:02 -06:00
|
|
|
|
|
|
|
if (want_cookie && !tmp_opt.saw_tstamp)
|
|
|
|
tcp_clear_options(&tmp_opt);
|
|
|
|
|
2018-03-23 04:05:45 -06:00
|
|
|
if (IS_ENABLED(CONFIG_SMC) && want_cookie)
|
|
|
|
tmp_opt.smc_ok = 0;
|
|
|
|
|
2014-06-25 08:10:02 -06:00
|
|
|
tmp_opt.tstamp_ok = tmp_opt.saw_tstamp;
|
|
|
|
tcp_openreq_init(req, &tmp_opt, skb, sk);
|
2016-09-23 03:27:42 -06:00
|
|
|
inet_rsk(req)->no_srccheck = inet_sk(sk)->transparent;
|
2014-06-25 08:10:02 -06:00
|
|
|
|
2015-03-13 16:51:10 -06:00
|
|
|
/* Note: tcp_v6_init_req() might override ir_iif for link locals */
|
2015-12-16 14:20:44 -07:00
|
|
|
inet_rsk(req)->ir_iif = inet_request_bound_dev_if(sk, skb);
|
2015-03-13 16:51:10 -06:00
|
|
|
|
2014-06-25 08:10:02 -06:00
|
|
|
af_ops->init_req(req, sk, skb);
|
|
|
|
|
|
|
|
if (security_inet_conn_request(sk, skb, req))
|
|
|
|
goto drop_and_free;
|
|
|
|
|
2017-05-05 07:56:54 -06:00
|
|
|
if (tmp_opt.tstamp_ok)
|
2017-06-07 11:34:39 -06:00
|
|
|
tcp_rsk(req)->ts_off = af_ops->init_ts_off(net, skb);
|
2016-12-01 03:32:06 -07:00
|
|
|
|
2017-08-22 00:33:48 -06:00
|
|
|
dst = af_ops->route_req(sk, &fl, req);
|
|
|
|
if (!dst)
|
|
|
|
goto drop_and_free;
|
|
|
|
|
net: allow setting ecn via routing table
This patch allows to set ECN on a per-route basis in case the sysctl
tcp_ecn is not set to 1. In other words, when ECN is set for specific
routes, it provides a tcp_ecn=1 behaviour for that route while the rest
of the stack acts according to the global settings.
One can use 'ip route change dev $dev $net features ecn' to toggle this.
Having a more fine-grained per-route setting can be beneficial for various
reasons, for example, 1) within data centers, or 2) local ISPs may deploy
ECN support for their own video/streaming services [1], etc.
There was a recent measurement study/paper [2] which scanned the Alexa's
publicly available top million websites list from a vantage point in US,
Europe and Asia:
Half of the Alexa list will now happily use ECN (tcp_ecn=2, most likely
blamed to commit 255cac91c3 ("tcp: extend ECN sysctl to allow server-side
only ECN") ;)); the break in connectivity on-path was found is about
1 in 10,000 cases. Timeouts rather than receiving back RSTs were much
more common in the negotiation phase (and mostly seen in the Alexa
middle band, ranks around 50k-150k): from 12-thousand hosts on which
there _may_ be ECN-linked connection failures, only 79 failed with RST
when _not_ failing with RST when ECN is not requested.
It's unclear though, how much equipment in the wild actually marks CE
when buffers start to fill up.
We thought about a fallback to non-ECN for retransmitted SYNs as another
global option (which could perhaps one day be made default), but as Eric
points out, there's much more work needed to detect broken middleboxes.
Two examples Eric mentioned are buggy firewalls that accept only a single
SYN per flow, and middleboxes that successfully let an ECN flow establish,
but later mark CE for all packets (so cwnd converges to 1).
[1] http://www.ietf.org/proceedings/89/slides/slides-89-tsvarea-1.pdf, p.15
[2] http://ecn.ethz.ch/
Joint work with Daniel Borkmann.
Reference: http://thread.gmane.org/gmane.linux.network/335797
Suggested-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Daniel Borkmann <dborkman@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-03 09:35:03 -07:00
|
|
|
if (!want_cookie && !isn) {
|
2014-06-25 08:10:02 -06:00
|
|
|
/* Kill the following clause, if you dislike this way. */
|
2017-03-15 14:30:46 -06:00
|
|
|
if (!net->ipv4.sysctl_tcp_syncookies &&
|
|
|
|
(net->ipv4.sysctl_max_syn_backlog - inet_csk_reqsk_queue_len(sk) <
|
|
|
|
(net->ipv4.sysctl_max_syn_backlog >> 2)) &&
|
|
|
|
!tcp_peer_is_proven(req, dst)) {
|
2014-06-25 08:10:02 -06:00
|
|
|
/* Without syncookies last quarter of
|
|
|
|
* backlog is filled with destinations,
|
|
|
|
* proven to be alive.
|
|
|
|
* It means that we continue to communicate
|
|
|
|
* to destinations, already remembered
|
|
|
|
* to the moment of synflood.
|
|
|
|
*/
|
|
|
|
pr_drop_req(req, ntohs(tcp_hdr(skb)->source),
|
|
|
|
rsk_ops->family);
|
|
|
|
goto drop_and_release;
|
|
|
|
}
|
|
|
|
|
2017-05-05 07:56:54 -06:00
|
|
|
isn = af_ops->init_seq(skb);
|
2014-06-25 08:10:02 -06:00
|
|
|
}
|
|
|
|
|
net: allow setting ecn via routing table
This patch allows to set ECN on a per-route basis in case the sysctl
tcp_ecn is not set to 1. In other words, when ECN is set for specific
routes, it provides a tcp_ecn=1 behaviour for that route while the rest
of the stack acts according to the global settings.
One can use 'ip route change dev $dev $net features ecn' to toggle this.
Having a more fine-grained per-route setting can be beneficial for various
reasons, for example, 1) within data centers, or 2) local ISPs may deploy
ECN support for their own video/streaming services [1], etc.
There was a recent measurement study/paper [2] which scanned the Alexa's
publicly available top million websites list from a vantage point in US,
Europe and Asia:
Half of the Alexa list will now happily use ECN (tcp_ecn=2, most likely
blamed to commit 255cac91c3 ("tcp: extend ECN sysctl to allow server-side
only ECN") ;)); the break in connectivity on-path was found is about
1 in 10,000 cases. Timeouts rather than receiving back RSTs were much
more common in the negotiation phase (and mostly seen in the Alexa
middle band, ranks around 50k-150k): from 12-thousand hosts on which
there _may_ be ECN-linked connection failures, only 79 failed with RST
when _not_ failing with RST when ECN is not requested.
It's unclear though, how much equipment in the wild actually marks CE
when buffers start to fill up.
We thought about a fallback to non-ECN for retransmitted SYNs as another
global option (which could perhaps one day be made default), but as Eric
points out, there's much more work needed to detect broken middleboxes.
Two examples Eric mentioned are buggy firewalls that accept only a single
SYN per flow, and middleboxes that successfully let an ECN flow establish,
but later mark CE for all packets (so cwnd converges to 1).
[1] http://www.ietf.org/proceedings/89/slides/slides-89-tsvarea-1.pdf, p.15
[2] http://ecn.ethz.ch/
Joint work with Daniel Borkmann.
Reference: http://thread.gmane.org/gmane.linux.network/335797
Suggested-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Daniel Borkmann <dborkman@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-03 09:35:03 -07:00
|
|
|
tcp_ecn_create_request(req, skb, sk, dst);
|
|
|
|
|
|
|
|
if (want_cookie) {
|
|
|
|
isn = cookie_init_sequence(af_ops, sk, skb, &req->mss);
|
|
|
|
req->cookie_ts = tmp_opt.tstamp_ok;
|
|
|
|
if (!tmp_opt.tstamp_ok)
|
|
|
|
inet_rsk(req)->ecn_ok = 0;
|
|
|
|
}
|
|
|
|
|
2014-06-25 08:10:02 -06:00
|
|
|
tcp_rsk(req)->snt_isn = isn;
|
2015-09-15 16:24:20 -06:00
|
|
|
tcp_rsk(req)->txhash = net_tx_rndhash();
|
2014-06-25 08:10:02 -06:00
|
|
|
tcp_openreq_init_rwin(req, sk, dst);
|
2018-06-29 22:26:57 -06:00
|
|
|
sk_rx_queue_set(req_to_sk(req), skb);
|
2015-10-02 12:43:35 -06:00
|
|
|
if (!want_cookie) {
|
|
|
|
tcp_reqsk_record_syn(sk, req, skb);
|
2017-10-23 14:22:23 -06:00
|
|
|
fastopen_sk = tcp_try_fastopen(sk, skb, req, &foc, dst);
|
2015-10-02 12:43:35 -06:00
|
|
|
}
|
2015-09-24 18:16:05 -06:00
|
|
|
if (fastopen_sk) {
|
2015-10-02 12:43:35 -06:00
|
|
|
af_ops->send_synack(fastopen_sk, dst, &fl, req,
|
2016-04-13 23:05:39 -06:00
|
|
|
&foc, TCP_SYNACK_FASTOPEN);
|
2015-10-04 22:08:07 -06:00
|
|
|
/* Add the child socket directly into the accept queue */
|
2019-03-08 14:09:47 -07:00
|
|
|
if (!inet_csk_reqsk_queue_add(sk, req, fastopen_sk)) {
|
|
|
|
reqsk_fastopen_remove(fastopen_sk, req, false);
|
|
|
|
bh_unlock_sock(fastopen_sk);
|
|
|
|
sock_put(fastopen_sk);
|
2019-03-19 09:05:44 -06:00
|
|
|
goto drop_and_free;
|
2019-03-08 14:09:47 -07:00
|
|
|
}
|
2015-10-04 22:08:07 -06:00
|
|
|
sk->sk_data_ready(sk);
|
|
|
|
bh_unlock_sock(fastopen_sk);
|
2015-09-24 18:16:05 -06:00
|
|
|
sock_put(fastopen_sk);
|
|
|
|
} else {
|
2015-03-17 19:32:29 -06:00
|
|
|
tcp_rsk(req)->tfo_listener = false;
|
2015-10-02 12:43:35 -06:00
|
|
|
if (!want_cookie)
|
2017-06-30 21:02:42 -06:00
|
|
|
inet_csk_reqsk_queue_hash_add(sk, req,
|
|
|
|
tcp_timeout_init((struct sock *)req));
|
2016-04-13 23:05:39 -06:00
|
|
|
af_ops->send_synack(sk, dst, &fl, req, &foc,
|
|
|
|
!want_cookie ? TCP_SYNACK_NORMAL :
|
|
|
|
TCP_SYNACK_COOKIE);
|
2016-04-01 09:52:20 -06:00
|
|
|
if (want_cookie) {
|
|
|
|
reqsk_free(req);
|
|
|
|
return 0;
|
|
|
|
}
|
2014-06-25 08:10:02 -06:00
|
|
|
}
|
2015-10-02 12:43:35 -06:00
|
|
|
reqsk_put(req);
|
2014-06-25 08:10:02 -06:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
drop_and_release:
|
|
|
|
dst_release(dst);
|
|
|
|
drop_and_free:
|
2019-03-19 09:05:44 -06:00
|
|
|
__reqsk_free(req);
|
2014-06-25 08:10:02 -06:00
|
|
|
drop:
|
2016-04-01 09:52:20 -06:00
|
|
|
tcp_listendrop(sk);
|
2014-06-25 08:10:02 -06:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(tcp_conn_request);
|