Re: [e2e] query reg improving TCP performane

Hi,

2040 bytes seem to be really a very low buffer size for any router with FIFO buffer management scheme. With this buffer size, the router would not be able to keep more than 2 packets of size 1500 bytes in its buffer. Alternate possibility is 2040 packets.

Regards,

Anil

On Tue, 10 Jul 2007, query wrote:

> Hi All,
>
> I was able to identify the size of the buffer in the Router's interface
> . It was found to be 2040
> bytes and the buffer management scheme used is FIFO .
> Then based on this statement by Lachlan , I did the following tunings.
>
> ""
> The general rule-of-thumb for Reno is that the send buffer should be
> at least twice the bandwidth*RTT. For BIC is is probably reduced to
> about 120% of the BDP (because it reduces its window by a smaller
> factor when there is a loss). The receive buffer should still be at
> least equal to the BDP plus the router buffer.
> ""
> The tunings are as follows.
>
> Send buffer = BDP + 120 % of BDP
> = 921600 + 184320
> = 1105920 bytes
>
> Receive buffer = BDP + Router's buffer size
> = 921600 + 2040
> = 923640 bytes
>
> After that I tried the same earlier experiments with iperf . I got a
> average throughput of 66
> Mbits/s .
>
> Next , I tried some more tuning . I increased the the send buffer
> size to twice of BDP.
> The receive buffer is same as earlier. This is according to what
> Lachlan suggested for TCP
> Reno.
> Based on these tunings , I got a throughput of average 78 Mbits/s
> using iperf. More improvement
> but not fully utilise the link capacity.
>
> But if I tune the window size to twice the size of BDP , I got a
> average throughput of
> around 88 Mbits/sec which I feel very much O.K for a 100 MBits/sec link
> . But it can be further
> improved. Also , earlier, I have written in this mailing list that when
> I tuned the window size to twice
> of BDP , I was getting a throughput of 95 Mbits /s . That I was
> referring to the maximum
> throughput .
> For all these experiments, I was using BIC.
>
> I also tried with the following tunings as suggested . But I didnot get
> any difference.
>
> /sbin/ifconfig eth0 txqueuelen 1000
> /sbin/sysctl -w net.ipv4.tcp_adv_win_scale=3
>
>
> So , based on the result of all these experiments , I reach the
> following conclusion.
>
> The receive buffer should be at least equal to the BDP plus the router
> buffer .
> The send buffer should be 20% more than BDP if you are using BIC.
> These tunings will probably try to utilise the maximum link capacity
> provided
> the buffer size in the intermediate router is equal to the BDP.
> This I cannot prove practically because it is not possible fot me to
> increase the buffer
> size to the size of BDP , because the network is not under my
> administration.
>
> But since in most cases the network is under the control of ISP , so it
> might not
> be possible for end users to know the size of the router's interface.
> In that case , the end to end send and receive buffer size should be
> atleast
> equal to twice the size of BDP to obtain maximum throughput. This
> statement
> was reflected by my findings.
>
> It will be helpfull to me if everybody give there views on my
> conclusion. If you
> have any contradiction , please write.
>
> With Thanks and Regards
> zaman
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On 7/6/07, query <query.cdac@gmail.com> wrote:
> >
> >
> > Thanks a lot Andrew . It helped me to understand and I feel that my
> > Tunings are O.K.
> >
> > > I was doing some Bandwidth measurement test on a 100 mbs link with a
> > > RTT
> > > > of about 70ms.
> > > > Based on that, I calculated the BDP as follows.
> > > >
> > > > BDP = Bandwidth * RTT
> > > > = 921600 bytes
> > > > I did the following adjustments. I increased the above calculated
> > > BDP by
> > > > nearly
> > > > half of the value . The TCP settings now look like this.
> > > >
> > > > /proc/sys/net/core/rmem_max 175636
> > > > /proc/sys/net/core/wmem_max 175636
> > > > /proc/sys/net/ipv4/tcp_rmem 4096 87380
> > > > 175636
> > > > /proc/sys/net/ipv4/tcp_wmem 4096 87380
> > > > 175636
> > > >
> > > > After these settings , I find the link utilisation to be nearly 95
> > > mbs.
> > > >
> > > > According to many papers that I read , I found that the BDP should
> > > be
> > > > equal
> > > > to the product of Bandwidth * RTT .
> > >
> > > The papers probably said that *router* buffers need to equal the
> > > bandwidth*RTT. You are adjusting the sender/receiver buffers. These
> > > need to be significantly larger, as you have found.
> >
> >
> > The papers or rather articles are talking of sender
> > and receiver
> > buffers . Here is one such link where I find it.
> > http://www.psc.edu/networking/projects/tcptune/ .
> >
> >
> >
> > In order to allow retransmissions, the sender buffer needs to be able
> > > to store all "packets in flight", which include both those in the in
> > > the router buffers and those "on the wire" (that is, in the nominal
> > > RTT of the link).
> > >
> > > In order to be able to provide in-order delivery, the receiver buffer
> > > needs to be able to hold even more packets. If a packet is lost, it
> > > will receive an entire RTT (plus router buffer) worth of data before
> > > the first retransmission of that packet will arrive. If the first
> > > retransmission is also lost, then it will need to store yet another
> > > RTT worth of data.
> > >
> > > The general rule-of-thumb for Reno is that the send buffer should be
> > > at least twice the bandwidth*RTT. For BIC is is probably reduced to
> > > about 120% of the BDP (because it reduces its window by a smaller
> > > factor when there is a loss). The receive buffer should still be at
> > > least equal to the BDP plus the router buffer.
> >
> >
> > What I understand from your reply, is that It is not necessary that
> > TCP Window should be
> > equal to BDP in all cases . Had the router buffer size is equal to BDP
> > , then I think I
> > should equal link utilisation equal to the capacity of the link .
> > Since , In Internet it will not be possible to know the router buffer
> > size , so the best thing one
> > can do , is to make the TCP window size twice to BDP as you have
> > suggested.
> >
> > I am finding another problem. The UDP transmission rate on that link is
> > decreased. I changed
> > to the default settings , but it is showing the exact readings after
> > tuning. It seems it is reading some fixed value from something and
> > based on that it is transferring data .
> >
> > The readings are like this..........
> >
> > iperf -u -c 192.168.60.62 -t 300 -l 1460 -i 2
> > ------------------------------------------------------------
> > Client connecting to 192.168.60.62, UDP port 5001
> > Sending 1460 byte datagrams
> > UDP buffer size: 108 KByte (default)
> > ------------------------------------------------------------
> > [ 3] local 10.128.0.2 port 32785 connected with 192.168.60.62 port 5001
> > [ ID] Interval Transfer Bandwidth
> > [ 3] -0.0- 2.0 sec 257 KBytes 1.05 Mbits/sec
> > [ 3] 2.0- 4.0 sec 257 KBytes 1.05 Mbits/sec
> > [ 3] 4.0- 6.0 sec 255 KBytes 1.05 Mbits/sec
> > [ 3] 6.0- 8.0 sec 257 KBytes 1.05 Mbits/sec
> > [ 3] 8.0-10.0 sec 255 KBytes 1.05 Mbits/sec
> > [ 3] 10.0-12.0 sec 257 KBytes 1.05 Mbits/sec
> > [ 3] 12.0-14.0 sec 255 KBytes 1.05 Mbits/sec
> > [ 3] 14.0-16.0 sec 257 KBytes 1.05 Mbits/sec
> > [ 3] 16.0-18.0 sec 257 KBytes 1.05 Mbits/sec
> >
> > The result is for the following tuning.
> > net.core.rmem_default = 110592
> > net.core.wmem_default = 110592
> >
> > After that I changed the tuning to
> > net.core.rmem_default = 196608
> > net.core.wmem_default = 196608
> >
> > The readings for the tuning is like this...
> > iperf -u -c 192.168.60.62 -t 300 -l 1460 -i 2
> > ------------------------------------------------------------
> > Client connecting to 192.168.60.62, UDP port 5001
> > Sending 1460 byte datagrams
> > UDP buffer size: 192 KByte (default)
> > ------------------------------------------------------------
> > [ 3] local 10.128.0.2 port 32785 connected with 192.168.60.62 port 5001
> > [ ID] Interval Transfer Bandwidth
> > [ 3] -0.0- 2.0 sec 257 KBytes 1.05 Mbits/sec
> > [ 3] 2.0- 4.0 sec 257 KBytes 1.05 Mbits/sec
> > [ 3] 4.0- 6.0 sec 255 KBytes 1.05 Mbits/sec
> > [ 3] 6.0- 8.0 sec 257 KBytes 1.05 Mbits/sec
> > [ 3] 8.0-10.0 sec 255 KBytes 1.05 Mbits/sec
> > [ 3] 10.0-12.0 sec 257 KBytes 1.05 Mbits/sec
> > [ 3] 12.0-14.0 sec 255 KBytes 1.05 Mbits/sec
> >
> > Kindly please help me to rectify the problem. It is the same link on
> > which I
> > performed the TCp test.
> >
> > regards
> > zaman
> >
> > >
> > >
> >
> >
> >
> >
> >
> >
> > I hope this help,
> > > Lachlan
> > >
> > > --
> > > Lachlan Andrew Dept of Computer Science, Caltech
> > > 1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA
> > > Phone: +1 (626) 395-8820 Fax: +1 (626) 568-3603
> > >
> >
> >
>
Received on Tue Jul 10 13:52:09 2007