Econ Research Computing Cluster Documentation

The systems described in this document constitute a set of Linux-based resources whose goal is to facilitate the computational requirements of faculty members in the Economics Department or affiliated Social Sciences Research groups. This document describes how to access and begin to use this collection of systems. The systems consist of three major components:

Storage
Interactive nodes
Batch nodes

These systems are the NFS/SLURM backed research systems.

Super Short Version

Use ssh to connect to login.econ.duke.edu
Jobs can be run interactively with the following executables:
- matlab
- stata
- R
- Any other generally available RHEL application
Jobs can be submitted to the cluster with the following submission scripts:
- matbatch
- statabatch
- Rbatch
- Custom scripts can be written to be used with sbatch
Our batch queue management system is
SLURM
A snapshot of cluster stats and performance can be found
here.

Notable Software

The OS version of the software is Scientific Linux version 6. This is binary compatible and contains the same packages as Red Hat Enterprise Linux 6. Any software generally available from RHEL can be installed easily upon request.

Additionally, the following software packages are available:

Matlab version 2018-b
Stata version 14.1 (SE and MP)
R version 3.6.0

Older versions of some software may be available under /econ/sw. For example, /econ/sw/matlab/R2014b is available in that path.

Other software not available through RPM package management will be installed under /econ/sw into a directory named after the package.

How to Use the Economics Cluster

Connect to the cluster using SSH and run your jobs interactively or submit them to the batch system using the provided scripts.

Requesting Access

The username for the Economics Research Computing Cluster is your University NetID. The password is different from your NetID password. If you do not know your password, please contact ECS at
helpdesk@econ.duke.edu.

All faculty have accounts on this system.
Master’s and PhD students should request access by emailing helpdesk@econ.duke.edu.
Guest access may be provided for others outside of the department. Contact helpdesk@econ.duke.edu for more information.

How to Log In

The Economics Research Computing Cluster currently has one front-end node (login.econ.duke.edu), also referred to as a login node, front-end node, or interactive node. This system allows jobs to be run interactively and to submit jobs to batch nodes.

To access faculty.econ.duke.edu, use an SSH client to connect. Further details for OS X and Windows are available below.

How to Log in from OS X – More Detail

All versions of OS X have access to an application called Terminal, which provides command line access to the interactive node. It is possible to run applications like Matlab and Stata with their graphical interfaces, but later versions of OS X require the installation of XQuartz to enable the graphical user interface.

XQuartz can be found here.

Assuming X11/XQuartz is installed, follow these steps to access the interactive cluster:

Run Applications > Utilities > XQuartz.app
Right-click on the XQuartz icon in the dock and select Applications > Terminal.
In this xterm window, ssh into the Linux system of your choice using the -X argument (secure X11 forwarding):
ssh -X -Y NetID@login.econ.duke.edu

Once you are logged into the Econ Cluster, you can run the GUI program of your choice, such as Matlab.
Example: matlab

How to Log in from Windows – More Detail

For Windows, you will need a terminal application. To utilize the GUI interfaces of applications such as Matlab, Stata, and SAS, you will need to install an X11 emulator.

Accessing the interactive Linux servers from a computer running Windows requires PuTTY, available for free on the
Duke Software site.

After installing PuTTY:

Open PuTTY and enter login.econ.duke.edu as the hostname.
To enable X11 forwarding, expand the “SSH” section in the sidebar, click “X11”, and check “Enable X11 forwarding”.
Click “Open” to start the session and log in with your NetID and cluster password.
Once logged in, you can run GUI programs (if you have an X11 server such as Xming or VcXsrv running on your Windows machine).

Managing Files on the Cluster

Use standard Linux commands (ls, cp, mv, rm, etc.) to manage your files on the cluster.

Transferring Files to the Cluster

You can transfer files using scp, sftp, or graphical clients such as WinSCP (Windows) or Cyberduck (Mac).

Transferring Files From an OS X Client

Open Terminal.
Use scp or sftp to transfer files. Example:
scp localfile NetID@login.econ.duke.edu:/path/to/destination/

Transferring Files From a Windows Client

Download and install WinSCP or FileZilla.
Connect to login.econ.duke.edu using your NetID and cluster password.
Drag and drop files between your computer and the cluster.

Managing Jobs

Use SLURM commands to manage your jobs. Common commands include:

squeue – View jobs in the queue
scancel – Cancel a job
scontrol – Control jobs and nodes

Submitting Jobs

Submit jobs using sbatch with your job script. Example:

sbatch myjobscript.sh

Checking on Jobs

Use squeue to check the status of your jobs.

Deleting Jobs

Use scancel <jobid> to delete a job.

Some Useful Linux Commands

ls – List files
cd – Change directory
cp – Copy files
mv – Move/rename files
rm – Remove files
cat – View file contents
less – View file contents page by page
man – Show manual for a command

Other Useful Bits

Remember to log out when you are done: exit
Check disk usage with du and df
Use module avail and module load to manage software modules

Terms and Definitions

SSH: Secure Shell, a protocol for secure remote login and other secure network services over an insecure network.
SLURM: Simple Linux Utility for Resource Management, a workload manager for Linux clusters.
Batch Node: A compute node where jobs are run non-interactively.
Interactive Node: A node where users can log in and run jobs interactively.