The Open Science Data Federation (OSDF) is a method to share data and also a method to provide the Open Science Grid (OSG) with tools to cache data close to where the compute is occurring.
FAST Research Storage is deploying OSDF’s Pelican software suite on a Kubernetes (k8s) cluster deployed in Duke’s High Performance Computing (HPC) network. The HPC network is in private address space, but Duke’s Science DMZ allows bypass of latency inducing network security tools (both firewalls and Intrusion Prevention Systems(IPS)) using SDN bypass networks. See Duke’s Science DMZ and FAST Research Storage
Rancher (Rancher) was used to deploy a k8s cluster that was used to install and configure Pelican. Access to FAST Research Storage via CephFS is facilitated by Ceph-CSI. Duke can provide access to both cache data for OSG users in the region but also share Duke sourced data with the larger research world.
