Cluster Operation
2 minute read
Login to HPCC
ssh username@cluster.hpcc.ucr.edu
Create a Cluster
hpcc_cloud create <NameForYourCluster>
Show status of a Cluster
hpcc_cloud status <NameForYourCluster>
Output:
Status: cfncluster-new-cluster - CREATE_COMPLETE
Output:"MasterPublicIP"="52.52.227.148"
Output:"MasterPrivateIP"="172.31.24.51"
Output:"GangliaPublicURL"="http://52.52.227.148/ganglia/"
Output:"GangliaPrivateURL"="http://172.31.24.51/ganglia/"
Note the “MasterPublicIP” Address from the output. Use this IP Address when connecting to the cluster via “ssh” or uploading and downloading via “scp”.
Show running Clusters
hcpp_cloud list
Delete cluster
hpcc_cloud delete <NameForYourCluster>
Connecting to your cluster
Note - /path/to/your/key-file.pem = where you saved your AWS account key file
MasterPublicIP = Master Public IP address from the cluster status
ssh -i /path/to/your/key-file.pem centos@<MasterPublicIP>
Uploading data to your cluster
This will transfer the local files to your home directory on the cluster.
scp -i /path/to/your/key-file.pem <local-files-to-copy> centos@<MasterPublicIP>:.
Downloading data/results
This would be called from the HPCC cluster and it will download the specified remote files to your current directory.
scp -i /path/to/your/key-file.pem centos@<MasterPublicIP>:./<files-to-download> .
Running a job on your cluster
This will show all the steps needed to create a cluster and run a simple batch job
1. Start a new cluster
hpcc_cloud create new-cluster
2. Get the IP address of the new cluster
Once the cluster build is complete you will be presented with the status informatiom. Note the “MasterPublicIP” Address from the output. Use this IP Address when connecting to the cluster via “ssh” or uploading and downloading via “scp”.
Output:
hpcc_cloud create new-cluster
Beginning cluster creation for cluster: new-cluster
Creating stack named: cfncluster-new-cluster
Status: cfncluster-new-cluster - CREATE_COMPLETE
Output:"MasterPublicIP"="52.52.227.148"
Output:"MasterPrivateIP"="172.31.24.51"
Output:"GangliaPublicURL"="http://52.52.227.148/ganglia/"
Output:"GangliaPrivateURL"="http://172.31.24.51/ganglia/"
Find the “MasterPublicIP”
3. Upload your input files and slurm submission script
This will transfer the local files to your home directory on the cluster.
scp -i /path/to/your/key-file.pem <local-files-to-copy> centos@<MasterPublicIP>:.
4. SSH to your new cluster
ssh -i /path/to/your/key-file.pem centos@<MasterPublicIP>
5. Submit your job to the cluster
sbatch slurm-submission-script.sh
6. Monitor your job
squeue
7. Download results
This command would be called from the HPCC cluster and it will download the specified remote files to your current directory.
scp -i /path/to/your/key-file.pem centos@<MasterPublicIP>:./<results-to-download> .
8. Delete cluster
hpcc_cloud delete new-cluster