Duke Science DMZ Overview
1. Background
ESnet within the US Department of Energy (DoE) developed the Science DMZ architecture [1] to help facilitate the transfer of research data by minimizing unnecessary friction. Although the ESnet architecture [2] shows a parallel network in support of the Science DMZ within their documentation, that is not required, and Duke is currently transitioning from a more traditional architecture to a Software-Defined Science DMZ architecture.
2. Current Science DMZ Architecture at Duke
Duke has resilient 100G connections to the North Carolina Research and Education Network (NCREN) operated by MCNC. These connections provide connectivity to Internet2 as well as commodity ISPs. Moreover, these circuits are virtualized and support 802.1Q VLAN tags for connectivity to other research institutions via MCNC/NCREN and Internet2 AL2S. The internet-edge routers provide external connectivity to general resources at Duke via multiple tiers of security appliances including a perimeter firewall supporting Intrusion Prevention System (IPS) services and an interior firewall cluster as illustrated in Fig 2.1:
Figure 2.1 – Duke Single-Line Network Architecture
Although the multi-tier cybersecurity architecture provides flexibility, the appliances within the green and blue layers can introduce friction, latency, jitter and impose other constraints that can limit the transfer of large volumes of research data. To address this, Duke has deployed a traditional Science DMZ architecture as shown in Fig 2.2:
Figure 2.2 – Current Science DMZ Architecture at Duke
Within the current Duke Science DMZ architecture, two hypervisor hosts are directly attached to an edge router with Mellanox ConnectX-5 NICs supporting 40/100G links. Today, these two hypervisors are each directly attached to the edge with 40G transceivers and can transition to 100G with a change in optics if needed in the future. This approach places the hypervisor hosts outside of the perimeter IPS and interior firewalls. Moreover, these hypervisor hosts are directly attached to internal data center switches at 40G with direct access to storage located within the Duke Compute Cluster (DCC) Virtual Routing and Forwarding (VRF) instance. Today, the hypervisors host Globus VMs to facilitate efficient transfer of large research data sets with friction-free access to the Internet/Internet2 and storage within DCC. Other VMs can be provisioned on these hypervisors as needed to take advantage of the Science DMZ architecture.
In addition to the hypervisors, we also have a bare-metal Linux host attached directly to an edge router at 10G hosting a containerized version of PerfSONAR [3]. The 10G NIC on the host leverages PCIe SR-IOV virtual functions and presents the PerfSONAR container with a hardware-driven virtual NIC thereby eliminating any bottlenecks associated with software-based virtual NICs and virtual switches typically used within container hosts.
3. Next-Generation Science DMZ Architecture at Duke
In 2019, Duke developed an NSF grant proposal called “Archipelago” [4] that was awarded with $1M in funding to develop and deploy a hybrid multi-node campus Software-Defined Network (SDN) architecture in support of agility and enhanced cybersecurity [5]. One aspect of Archipelago involved exploring state-of-the-art SDN switches as well as Smart Network Interface Cards (SmartNICs) [6]. As a part of the development of the next generation production network architecture at Duke, we realized that some of the products evaluated in Archipelago could provide significant flexibility and cost savings in the production network shown in Fig 2.1. We use the salmon colored “Rapid Response SDN” layer to block unwanted traffic, without introducing friction to permitted traffic, before it arrives at the perimeter IPS. Duke developed software to control this layer via a web interface (as shown in Fig. 2.3) as well as via a REST API that the IT Security Office (ITSO) makes heavy use of.
Figure 2.3 – Perimeter Rapid Response (Black Hole++) Block Interface
Referring to Fig. 2.1, within the green perimeter IPS and blue firewall functional blocks, we leverage dedicated SDN switches to provide horizontal scalability, bypass and service chaining as shown in Fig. 2.3.
Figure 2.4 – IPS and Firewall Bypass and Service Chaining
Duke also developed software to program the SDN switches supporting the IPS functional block with bypass capabilities. The IPS bypass functionality permits ITSO to selectively bypass the IPS appliances as needed as shown in Fig. 2.4. When the request is submitted, the SDN policies are automatically deployed to bypass the IPS.
In the blue-colored firewall functional blocks shown in Fig. 2.1, we also leverage the architecture outlined in Fig. 2.3 but have deployed two pairs of active/standby interior firewalls. The SDN switches support delivering a subset of traffic to each pair of firewalls allowing us to reduce costs of the firewall layer and scale horizontally. Although we have not yet implemented the SDN-driven firewall bypass capabilities, as we have done with the IPS, these features are on our development roadmap and will be a key part of our future Science DMZ architecture described in §3. We also leverage the SDN switches within the firewall functional block for service chaining as shown in Fig. 2.5:
Figure 2.5 – SDN-Driven Service Chaining
As shown within Fig 2.5, we can steer different types of traffic through different tiers of appliances as needed. As an example, for “outlands”, associated with student instructional computing, we can force traffic to flow through the perimeter IPS even though it originates within Duke before it then hits the interior firewall functional block. The service-chaining functionality is defined via policy and not physical cabling topologies as handled in legacy environments.
By combining the use of SDN-driven bypass and service chaining our future Science DMZ architecture at Duke will be virtualized to permit Science DMZ applications to be hosted in a variety of locations without requiring special physical connectivity to a parallel network or to have direct connectivity to the edge router as shown in Fig. 2.6:
Figure 2.6 – SDN-Driven Friction Bypass
The architecture shown in Fig. 2.6 will provide enhanced protection to Virtual Science DMZ applications via the Rapid Response SDN layer which will permit undesirable traffic to be blocked via API without introducing friction to desirable traffic. This is a significant advantage over our current architecture in Fig. 2.2 where static ACLs or host-based policies are required. Moreover, we are currently developing an SDN-driven monitoring fabric and horizontally scalable sensor array as a part of our NSF-funded MISTRAL [7] project at Duke. We envisage developing a closed-loop friction-free feedback system in support of dynamic control of research data flows with our SDN-driven Science DMZ.
3. References
Dart, Eli, et al. “The science dmz: A network design pattern for data-intensive science.” Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. 2013.
ESnet Science DMZ Architecture: https://fasterdata.es.net/science-dmz/science-dmz-architecture/
PerfSONAR: https://www.perfsonar.net/
Archipelago: https://sites.duke.edu/archipelago (NSF OAC 1925550)
Brockelsby, William, and Rudra Dutta. “Archipelago: A Hybrid Multi-Node Campus SDN Architecture.” 2023 26th Conference on Innovation in Clouds, Internet and Networks and Workshops (ICIN). IEEE, 2023.
Brockelsby, William, and Rudra Dutta. “Performance implications of problem decomposition approaches for SDN pipelines.” GLOBECOM 2020-2020 IEEE Global Communications Conference. IEEE, 2020.
MISTRAL: https://sites.duke.edu/mistral/ (NSF OAC 2319864)
