Internet2 Land Speed Record in single and multiple IPv4 TCP stream

20th, Feb.2006

8.8049Gbps on 30,000km (actually 32372 km)connection from Tokyo through Seattle to Amsterdam, through Amsterdam to Chicago, through Chicago to Tokyo Latency 500ms RTT

Network Configuration

treceroute from ntt-x366-1c to ntt-x366-2c (Tokyo - Chicago - Amsterdam - Seattle - Tokyo)

traceroute to ntt-x366-2c (192.31.116.3), 30 hops max, 46 byte packets
 1  gs4k-chi-3300 (192.31.116.35)  163.331 ms  164.806 ms  164.651 ms
 2  gs4k-ams-359 (192.31.116.66)  268.008 ms  267.482 ms  267.488 ms
 3  ni40g-sea-655 (192.31.116.98)  407.143 ms  407.145 ms  407.143 ms
 4  ntt-x366-2c (192.31.116.3)  503.209 ms  503.207 ms  503.196 ms

treceroute from ntt-x366-2c to ntt-x366-1c (Tokyo - Seattle - Amsterdam - Chicago - Tokyo)

traceroute to ntt-x366-1c (192.31.116.36), 30 hops max, 46 byte packets
 1  ni40g-sea-654 (192.31.116.2)  96.279 ms  96.268 ms  96.259 ms
 2  gs4k-ams-655 (192.31.116.97)  239.712 ms  239.519 ms  239.542 ms
 3  gs4k-chi-359 (192.31.116.65)  344.898 ms  343.971 ms  343.723 ms
 4  ntt-x366-1c (192.31.116.36)  503.203 ms  503.203 ms  503.215 ms

Internet2 TCP Single stream Land Speed Record measurement

System

Server configuration

CPU: Four Intel Xeon MP, 3.66GHz
IBM x366 Server
Memory: 32GB PC3200 DDR2
OS: Linux kernel 2.6.12
Disk: 440GB disk (SAS)(disk speed not essential for performance)
Network interface card

Chelsio T310-X (10GBASE-SR)
PCI-X 2.0 I/O bus connection

Technical points

(1) IPv4 packets
(2) 9014B Jumbo frame
(3) single stream TCP
(4) All the trans continental connections are WAN PHY based.
(5) Use of PCI-X V2.0 gives performance more than PCI-X bandwidth
(6) Very high utilization factor of available network bandwidth by precise pacing mechanism in network adaptors

Target Performance of single stream TCP on 30,000(actual 32,372) km circuit

8.8049 Gbps (TCP payload), standard frame -> 264,147Terabit meter / second (Internet2 LSR approved Single/Multi TCP Stream record)

Internet2 Land Speed Record Page

Results

Traffic Graphs

dump of trial

index.html LSR20060220-network.ppt lsr_scripts.tar.gz sub.html index.html~ lsr-8.8049g-rx.txt packet-acklog.txt tcpdump-rx.bin LSR20060220A.doc packet-log.txt tcpdump-rx.txt LSR20060220.doc lsr_scripts sub.files tcpdump-tx.bin

These traces are submitted traial non-drop version(first 100 packets)

sender side log

receiver side log

tcpdump format(sender side)

tcpdump format(receiver side)

Name of Institutions and persons

The University of Tokyo

Kei Hiraki
Mary Inaba
Junji Tamatsukuri
Yutaka Sugawara
Takeshi Yoshino
Katsushi Inagami

WIDE project

Akira Kato
Hiroshi Esaki
Yuji Sekiya
Seiichi Yamamoto

Chelsio Communications

Felix Marti
Wael Noureddine

JGN2

Asako Toyoda
Kazuhiko Nakamura
Katsushi Kobayashi
Yoshitaka Hattori
Jin Tanaka
Shinya Hori

Northwest Giga Pop

Jan Eveleth
Bill Mar
Beth Garrison

NTT Communications

Katsuyuki Hasebe
Miyuki Neo
Yu Inada
Mitsuo Murakami

APAN

Kazunori Konishi

Fujitsu Computer Technologies LTD

Ryutaro Kurusu
Masakazu Sakamoto
Yuki Furukawa
Yukichi Ikuta

IEEAF
CANARIE

Rene Hatem
Damir Pobric
Thomas Tam

StarLight

Linda Winkler
Jake Sallee

SURFnet

Kees Neggers
Erik-Jan Bos

SARA

Pieter de Boer

University of Amsterdam

Cees de Laat
Paola Grosso
Freek Dijkstra
Hans Blom

Supported by

Special Coordination Fund for Promoting Science and Technology, Ministr y of Education, Culture, Sports, Science and Technology, Japan
Foundry Networks,
Bussan Networks,
Alaxala Networks,
Hitachi,
CISCO systems, Inc.

Whole Connection World Map

Data Reservoir / GRAPE-DR project, University of Tokyo