Category Archives: Testing

Test 1 100 rules

Send 8G files from sdn-a1 to sdn-b1 and sdn-a2 to sdn-b3 (b2 is still AWOL) while monitoring the rate of change of the SNMP counters for ports in switches OF-1 and OF-2 in from sdn-a1, between the switches, and out to sdn-b1 and the same for sdn-a2 -> b3.  The SNMP sampling rate is once/10 seconds, and the difference to the counters is calculated in the spreadsheet.
The files were sent starting at 14:39:06.

During that test, 100 OpenFlow rules per second were pushed, starting at 14:53:36.

These graphs are in order, from sdn-a1 and a2 to switch of-1 to switch of-2 to sdn-b1 and b3.

40 sends of a 1G file

1G-sdn-b1-in1G-sdn-b1-out1G-of-2_56-in1G-of-2_56-out1G-of-3_56-in1G-of-3_56-out       1G-sdn-c1_in 1G-sdn-c1_out1G-of-2-cpu-40-1G-sends1G-of-3-cpu-40-1G-sends
These graphs show the number of packets sent through the switches during 40 repeats of sending a one GB file (1073741824 bits).  The series starts at timestamp 14:36:44.
The graphs are named and labeled according to their position on the switches – for example, the port that holds sdn-B1’s connection is labeled sdn-b1. There are both in and out datasets for each port.  The “in” dataset shows information coming in to the switch from outside of it, and the “out” dataset shows information being sent elsewhere in the switch for processing.

There are also two graphs of CPU load (taken as a 4-second average), one for each switch.

POX webservice/JSON issue

While trying to enable the openflow.webservice.py and web.webcore.py modules from pox on RedHat 6.4 Linux… I kept receiving the following error…

[root@sdn-A4 Python-2.7.3]# python -u /opt/pox/pox.py --verbose forwarding.l2_pairs web.webcore openflow.webservice
POX 0.2.0 (carp) / Copyright 2011-2013 James McCauley, et al.
Traceback (most recent call last):
 File "/opt/pox/pox/boot.py", line 91, in do_import2
  __import__(name, level=0)
 File "/opt/pox/pox/openflow/webservice.py", line 50, in <module>
from pox.openflow.of_json import *
 File "/opt/pox/pox/openflow/of_json.py", line 111
 _unfix_map = {k:_unfix_null for k in of.ofp_match_data.keys()}
                                  ^
SyntaxError: invalid syntax
Could not import module: openflow.webservice

It was working fine on my MacOSx with Mavericks, but was getting the above error on Red Hat. After looking at the version of pox on both systems… I saw no difference. The only difference was Red Hat was running Python 2.6.6 and MacOSx was running 2.7.3.

The fix? Upgrade Python, BUT do not replace version 2.6.6 in the system path because YUM depends on it.

Run the following commands…

mkdir /opt/Python-273
cd /opt/Python-273
wget http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz
tar -xvzf Python-2.7.3.tgz
yum install gcc
cd Python-2.7.3
./configure
make altinstall
alias python273="/opt/Python-273/Python-2.7.3/python"

Now that python 2.7.3 is installed… just make sure you run POX with the new version alias command.

python273 pox.py

Single Rack SDN Environment

Attached is a PDF of a visio diagram of the single rack layout.

A couple of things about the diagram-

One blade per chassis is on the 10G “control network” – A1 and B1 to be used as controllers\ and C1 as the monitor/SNMP poller.

The blade chassis are connected via 1G to the control network as well (console/ssh access)

Initial configuration will not connect the data networks to the production network

Later testing needs to be done to run load over the OpenFlow VRF

In addition to our current ongoing activity around load testing (both traffic and rules) we need to start work on testing the general use cases.

Use Case #1 – Expressway #1

  • Normal traffic will flow over the A links between switches
  • Traffic from A2 to B2 will be routed over the B link between switch #1 and switch #2

This  will require a fairly simple rule – but it could be based on MAC address, source/dest IP, port, …

I think we should load up traffic on the A link and then show that we can independently load up traffic on the B path.  We should also plan to put rule set updates/stress on the servers as well.

Use Case #2 – Expressway #2A

This is similar to above but traffic has to flow through two switches

  • Traffic from A2 to C2 will flow over the B paths

Use Case #3 – Expressway #2B

This is similar to #1 – but the path is between switch #1 and switch #3 and bypasses switch #2

  • Traffic from A2 to C2 will flow over the C path

Visio-SDN Use Case Mapping – Single Rack – 12-04-13

POX openflow NEC flow test Nov 30th, 2013

POX openflow NEC flow test

Lesson learned:

1. In order for the NEC switch to start see a new controller you must disable openflow on the specified vlan then reenable it after configuring the new openflow controller

Example…

VLAN 213
openflow-id 0 #disables openflow
no openflow controller 1
openflow controller 1 address 10.138.32.3 DATA
openflow-id 1 #enables openflow again on the vlan

2. to see what ports are openflow enabled run the “show openflow 1” command
PortList Status State Config Current Advertised Supported Peer
57 e 0x200 0x2 0x140 0x0 0x0 0x0
58 e 0x200 0x2 0x140 0x0 0x0 0x0
59 e 0x200 0x2 0x140 0x0 0x0 0x0
60 e 0x200 0x2 0x140 0x0 0x0 0x0

Then you can manually add one port.
NEC Tests

Test1 – Layer 2 flows – Add 10 flows per second, max of 1000 flows, statically adding to port 57
Results – The rate didn’t appear to be an issue, but there is a max of 750 flows L2

ERROR:openflow.of_01:[74-99-75-81-b3-00|213 1] OpenFlow Error:
[74-99-75-81-b3-00|213 1] Error: header:
[74-99-75-81-b3-00|213 1] Error: version: 1
[74-99-75-81-b3-00|213 1] Error: type: 1 (OFPT_ERROR)
[74-99-75-81-b3-00|213 1] Error: length: 76
[74-99-75-81-b3-00|213 1] Error: xid: 1005
[74-99-75-81-b3-00|213 1] Error: type: OFPET_FLOW_MOD_FAILED (3)
[74-99-75-81-b3-00|213 1] Error: code: OFPFMFC_ALL_TABLES_FULL (0)
[74-99-75-81-b3-00|213 1] Error: datalen: 64
[74-99-75-81-b3-00|213 1] Error: 0000: 01 0e 00 50 00 00 03 ed 00 3f ff f7 ff ff 00 00 …P…..?……
[74-99-75-81-b3-00|213 1] Error: 0010: 00 00 00 00 01 23 45 36 a0 de 00 00 00 00 00 00 …..#E6……..
[74-99-75-81-b3-00|213 1] Error: 0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
[74-99-75-81-b3-00|213 1] Error: 0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80 00 …………….

Test2 – Layer 2 flows – Add 10 flows per second, max of 1000 flows, statically adding from range for ports 57 to 60
Results – same as test1 – The rate didn’t appear to be an issue, but there is a max of 750 flows L2

ERROR:openflow.of_01:[74-99-75-81-b3-00|213 1] OpenFlow Error:
[74-99-75-81-b3-00|213 1] Error: header:
[74-99-75-81-b3-00|213 1] Error: version: 1
[74-99-75-81-b3-00|213 1] Error: type: 1 (OFPT_ERROR)
[74-99-75-81-b3-00|213 1] Error: length: 76
[74-99-75-81-b3-00|213 1] Error: xid: 974
[74-99-75-81-b3-00|213 1] Error: type: OFPET_FLOW_MOD_FAILED (3)
[74-99-75-81-b3-00|213 1] Error: code: OFPFMFC_ALL_TABLES_FULL (0)
[74-99-75-81-b3-00|213 1] Error: datalen: 64
[74-99-75-81-b3-00|213 1] Error: 0000: 01 0e 00 50 00 00 03 ce 00 3f ff f7 ff ff 00 00 …P…..?……
[74-99-75-81-b3-00|213 1] Error: 0010: 00 00 00 00 01 23 45 50 21 c7 00 00 00 00 00 00 …..#EP!…….
[74-99-75-81-b3-00|213 1] Error: 0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
[74-99-75-81-b3-00|213 1] Error: 0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80 00 …………….
Test3 – Layer 2 flows – Add 50 flows per second, max of 749 flows, statically adding from range for ports 57 to 60
Results – The rate didn’t appear to be an issue, this time there were no errors

Apparently the NEC does state that it only supports 750 12 tuple flows, but supports over 80K layer2 flows.

Mininet, Pox, and Generating Flows per Second

Development for testing Layer2 flows per second was successful! The goals were the following:

  • get a controller to generate X number of flows (currently L2 flows) to see how many flows a switch could handle
  • 2nd, get a controller to generate X number of flows per second

Both were successful thanks to mininet. http://mininet.org/

Mininet creates virtual hosts, controllers, and switches in an instant. This was developed by people that wanted create a tool to help with openflow/SDN development.

It was pretty easy to get setup by just following their directions on their website. Once I downloaded the mininet VM, installed it… it was up and running.

The entire testing setup consisted:

  • a Mininet VM, which had an openflow switch and two hosts
  • a POX openflow controller on my local machine

All I had to do to get mininet to connect to the controller on my local machine was to point it to my actual network interface that go me outside of my machine on to our production network.

sudo mn --controller=remote,ip=10.180.0.207,port=6633

I tried from mininet to have it use my loopback address and the nat’d gateway address, but neither worked until I used my local computers prodcuction network NIC.

Once I did that, mininet connected to my machines POX controller…

INFO:openflow.of_01:[00-00-00-00-00-01 1] connected
DEBUG:forwarding.ryan_l2:Switch 00-00-00-00-00-01 has come up.

Here is the code that is used for Flows per second testing…

http://sites.duke.edu/dukesdn/2013/11/06/pox-l2-flow-generator-code/ 

The default for the code is 100 flows total, 10 per second. This can be changed by edit a couple of the numbers commented in the code.

In mininet, in order to see the flows you type the following…

mininet> dpctl dump-flows
*** s1 ------------------------------------------------------------------------
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=31.849s, table=0, n_packets=0, n_bytes=0, idle_age=31, dl_dst=01:23:45:d6:52:a3 actions=output:2
 cookie=0x0, duration=34.973s, table=0, n_packets=0, n_bytes=0, idle_age=34, dl_dst=01:23:45:98:8d:f4 actions=output:2
 cookie=0x0, duration=33.935s, table=0, n_packets=0, n_bytes=0, idle_age=33, dl_dst=01:23:45:2f:8d:55 actions=output:4
 cookie=0x0, duration=36.006s, table=0, n_packets=0, n_bytes=0, idle_age=36, dl_dst=01:23:45:d5:1a:2e actions=output:4
 cookie=0x0, duration=32.899s, table=0, n_packets=0, n_bytes=0, idle_age=32, dl_dst=01:23:45:db:89:8d actions=output:4

Since each line is a flow (except the first line) we can just use the wc (word count) command to see the number…

The output will be:

# of lines, # of words, and # of characters

mininet> dpctl dump-flows | wc
*** s1 ------------------------------------------------------------------------
 101 803 12107

So we have 101 lines, or 100 flows.

perfSONAR Test Results – 10/28/13 to 10/31/13

Jianan Wang ran a series of perfSONAR runs that included runs that overlapped their timing and runs that were done with no overlap.

The three blade chassis, each containing four blade servers, are connected to three independent OpenFlow switches.

Connections between blades SDN09/SDN10/SDN11/SDN12 to SDN05/SDN06/SDN07/SDN08 traverse two OpenFlow switches

Connections between blades SDN09/SDN10/SDN11/SDN12 to SDN01/SDN02/SDN03/SDN04 traverse three OpenFlow switches

As expected, the runs with no overlap showed approximately 9 Gbps of available bandwidth for traffic that ran through two switches, while the runs that overlapped (across three servers) showed approximately 3 Gbps of available bandwidth

A reduction of 1 Gbps was seen in traffic that flowed across the additional switch for the unblocked test.

SDN09 Rates

As a reminder, here is an overview of the server topology

SDN Topology Image

Here is a shorter time window that shows the variation between blocking and non-blocking perfSONAR tests (note that this is for SDN10 connections to SDN01 and SDN05).

SDN10 Rates

perfSONAR Tests – 9/03/13 – Bandwidth Testing Between Blade Chassis

 
perfSONAR test from SDN-09 to SDN-01, SDN-02, and SDN-12

Jianan Wang ran a series of pefSONAR tests from approximately 12:00 AM on 9/3/13 to 4:00 AM on 9/4/13.

These test were run from SDN-09 (in NB154) to SDN-01 and SDN-02 in NB-145 (DSCR).

The results are shown below:

perfSONAR test from SDN-09 to SDN-01, SDN-02, and SDN-12

As expected, the performance that stayed ‘in chassis’ from SDN-09 to SDN-12 was constant at about 9 GBPS.  The performance achieved for inter chassis communication that is required to go through the core network on the openflow VRF showed significant variability.  Post testing it was determined that testing from machines in the DSCR to the OIT NAS heads in NB154 was responsible for the performance degradation during the day.  Given that SDN-01 and SDN-02 are in the same chassis we are not certain of the cause of the difference in performance between them during the period of midnight to 10 AM, but they appeared to converge to between 8 and 8.5 Gbps after midnight on the 3rd.

Testing Processes

A series of bandwidth tests have been performed using the existing production network in order to understand baseline performance capabilities as well as confirming the impact that large file transfers have on the network.

It was assumed that a 10G connected server would not be able to impact the core 20G Duke network.  However, we have seen multiple impacts on the Duke network due to baseline testing for this project.

perfSONAR tests have been shown to be NON intrusive, but large, dedicated load cannons that do sustained file transfers have highlighted capacity issues in the core Duke academic campus network.

PerfSonar Tests – 9/16/13 – Bandwidth Testing Between Blade Chassis

Below is a summary of three different perfSONAR tests that were run.

Tests were run from blades in three chassis to three blades in each of the chassis.

 

First sets of tests between server chassis to blade SDN-11 – for a picture of the network layout see:

http://sites.duke.edu/dukesdn/files/2013/09/SDN-Network-Layout-Pre-Open-Flow.pdf

perfSONAR bandwidth test to SDN-11

perfSONAR bandwidth test to SDN-11

perfSONAR test to SDN-06

perfSONAR test to SDN-06

Note significantly reduced bandwidth on connection from SDN-11 to SDN-06 – investingation underway to determine root cause.

 

perfSONAR test to SDN-01

perfSONAR test to SDN-01