As noted in an earlier perfSONAR article we are using perfSONAR to monitor overall performance of connections to the campus core network and among nodes in different locations on the campus network. In addition, we are using perfSONAR to identify problems with those connections.
For example, after enabling the 10Gbps connection of the BioSci building to the new core, we saw some strange behavior as shown in the graph below:
After the network upgrade was completed, one direction of traffic flow showed the expected improvements, but the reverse direction was poor. The performance of the path from phy-perfsonar-02.phy.duke.edu -> fitz-perfsonar-03.oit.duke.edu did not improve as the Fitz East data center was migrated to the new core nor when the traffic was white listed at the IPS. The problem was fixed in mid June when an interface on one of the core switches was found to have issues. Traffic then greatly improved for the path from Physics to Fitz East and the traffic paths were symmetrical. Traffic stayed consistent between the two paths until late July 2014 (7/23/14) when the traffic from Physics to Fitz East again showed a degradation when compared with the opposite direction. On 8/12/14, updates to the connection of the core network to the Internet were made and performance again became consistent between inbound and outbound traffic between Fitz and Physics (detail shown below):
We are instituting a regular review of a number of perfSONAR graphs by Duke’s Service Operations Center (SOC) in order to catch these issues early. A simple ping of the path will work and so our normal monitoring services (link monitoring or ping monitoring) may not be sufficient to determine problems which only appear under load. iPerf tests that are part of perfSONAR appear the best way to reliably monitor links for their usable and available bandwidth.
The perfSONAR monitoring of traffic between Fitz East and Physics seems to be a good barometer of the health of connections between the campus core and building networks. A live view of this can be found at fitz-personar-03 (NetID login required).
