Lab Devices with Unexpected Connection Attempts
While performing exploratory data analysis on the MISTRAL flow records, the analysis team examined outbound traffic from the lab devices grouped by the source ips, destination subnets, and destination ports to identify a baseline for normal traffic per device. One source IP in particular stood out for the unexpected destination ranges and destination ports. The unexpected traffic included:
- multicast traffic to 244.0.0.0/24 on port 5353
- multicast traffic to 244.0.0.0/24 on port 5355
- unused private IP traffic to 192.168.1.0/24 on port 59421
- multicast traffic to 229.111.112.0/24 on port 3071
- multicast traffic to 239.255.255.0/24 on port 1900
- multicast traffic to 239.255.255.0/24 on port 3702
- traffic to a commercial ISP on port 59421
- traffic to a range of external addresses on port 5938
The connections also included known or expect traffic including web traffic associated with Operating System updates and the campus mandated external logging tools.
The device had been mislabeled in the server inventory which complicated identification, but with further investigation the device was determined to be a windows server running proprietary software from a vendor, and used to interface with and control microscopy hardware used in the lab’s research. The traffic to destination per 5938 was determined to be team viewer software used by the vendor to maintain the equipment remotely.
As a result of the findings, it was recommended that the vendor be consulted to determine if the multicast traffic was needed for the normal functions of the microscopes, and if not to determine if the multicast features could be disabled.
NFS Connection Spikes
One of the exercises to identify normal traffic patterns conducted by the analysis team involved graphing connection counts grouped by destination ports over time. During this process a sharp spike in the number of connections to TCP port 2049 was observed. Several other destination ports also showed spikes in the connection counts, notable UDP 53 and 111.
Further investigation showed that the source IPs for the connections were lab machines that typically had a small number of long lasting NFS connections using port 2049 to central fileservers where research data is stored. These long running connections usually were maintained for hours or days at a time. As an artifact of the data collection process, long connections are logged every 30 minutes and the amount of data transferred to and from the file servers can be tracked based on the packet and byte counts.
During the time period of the spike in connections, very different characteristics were observed for the NFS connections however. Rather than a small number of long lasting connections, most but not all of the source IPs showed a large number of very short connections, usually lasting a fraction of a second. Each of the short connections showed small total byte counts, and a large percentage of them showed only packets and bytes transferred from the source IP with no response from the destination IPs, indicating that these were incomplete connections that did not receive any response from the destinations.
The analysis team theorized that the spike in NFS connection resulted from an unknown issue with the file servers that had broken the existing NFS connections and was preventing the clients from successfully reconnecting. The large number of small NFS connections in rapid succession could have resulted from the clients’ failing attempts to reconnect, and was consistent with the known behavior of NFS clients under similar circumstances.
The analysis team later learned that the theory was correct — there had been an NFS outage affecting some but not all of the NFS file servers. The known start time for the outage was a short time after the beginning of the traffic spike observed in the MISTRAL data, suggesting that the MISTRAL data might be a useful source of monitoring data for the health of the NFS connections going forward.
DNS Traffic Spikes
While charting the MISTRAL network traffic data by destination IP over time, the analysis team found a series of unexpected spikes in traffic intended for destination port udp 53, which is usually used for DNS requests. The destination IPs were legitimate campus DNS servers, and based on the normal traffic details, the source IPs were servers that were configured to use the campus DNS servers. The traffic spikes reoccurred daily with an average periodicity of 24 hours, with a 5 to 10 minute variability.
The traffic originally consisted of a single large spike daily in the early afternoons, but with continued observation resolved into two spikes beginning just before 2pm and just after 3pm. Each spike lasted between 5 and 7 minutes and varied from between 1200 and 5500 connections per minute between all of the IPs. Individual IPs showed a varied start time and a similar large initial spike in the first minute followed by a smaller number of connections on the subsequent few minutes until the DNS request counts returned to normal.
The data analysis team initially theorized that some other network traffic was causing the sudden increase in DNS requests, but no increases in traffic volumes were observed for other destination ports at related times. Examining the DNS query logs for a selected sample of the source IPs showed that most of the increase in DNS requests were reverse lookups for IPs belonging to the campus vulnerability scanners. The MISTRAL collection mechanism deliberately excludes the scanner traffic to limit the volume of data collected, but the secondary effects from the scans such as the DNS requests resulting from them are still included. The analysis team examined the possibility of excluding the DNS traffic resulting from the scans as well as the scans themselves, but determined that there was no easy way to differentiate the requests resulting from the scans from normal DNS requests resulting from daily operation.
