Page tree
Skip to end of metadata
Go to start of metadata

Sol is Lehigh's newest Linux cluster replacing Corona and other ancillary Level 2 resources. Following our tradition of naming high performance computing clusters after stars or celestial phenomena, Sol is named after the nearest star.

Sol is a heterogeneous cluster launched on Oct 1, 2016 with a total of 34 nodes, 26 are Condo investments by two CAS faculty. All nodes provide 500GB scratch storage for running jobs and are interconnected with 2:1 oversubscribed EDR (100Gbps) Infiniband fabric. In Fall 2018, a new Ceph storage cluster was installed that provides a 11TB CephFS global scratch space for storing temporary data for 7 days after completion of jobs.

Upgrades by Condo Investments

  • In Jan. 2017, each of the 25 Condo nodes were upgraded to include two GTX 1080 GPU cards.
  • In 2017, Condo Investments from RCEAS and CBE faculty added 22 nodes and 16 nVIDIA GTX 1080 GPU cards.
  • In 2018, Condo Investments from RCEAS and CAS faculty added 24 nodes and 48 nVIDIA RTX 2080 TI GPU cards.
  • In Mar. 2019, Condo Investments from RCEAS faculty added 1 node.

As of Mar. 2019

Processor Type

Number of Nodes

Number of CPUs

Number of GPUs

CPU Memory (GB)

GPU Memory (GB)

CPU TFLOPs

GPU TFLOPs

Annual SUs

2.3 GHz E5-2650v3

9

180

10

1152

80

5.76

2.57

1,576,800

2.3 GHz E5-2670v3

33

792

62

4224

496

25.344

15.934

6,937,920

2.2 GHz E5-2650v4

14

336


896


9.6768


2,943,360

2.6 GHz E5-2640v3

1

16


512


0.5632


140,160

2.3 GHz Gold 6140

24

864

48

4608

528

39.744

18.392

7,568,640


81

2188

120

11392

1104

82.816

36.130

19,166,880

System Configuration

 1 Interactive node
  • Two 2.3GHz 10-core Intel Xeon E5-2650 v3, 25M Cache
  • 128GB 2133MHz RAM
  • 1TB HDD
  • 10 GbE and 1 GbE network interface
  • CentOS 7.x
 9 Haswell GPU nodes
  • Two 2.3GHz 10-core Intel Xeon E5-2650 v3, 25M Cache
  • 128GB 2133MHz RAM
  • 1TB HDD
  • 100 Gb/s EDR Infiniband network interface
  • 10 GbE and 1 GbE network interface
  • 8 nodes with one EVGA Geforce GTX 1080 PCIE 8GB GDDR5
  • 1 node with two EVGA Geforce GTX 1080 PCIE 8GB GDDR5
  • CentOS 7.x 
 33 Haswell GPU nodes
  • Two 2.3GHz 12-core Intel Xeon E5-2670 v3, 30M Cache
  • 128GB 2133MHz RAM
  • 1TB HDD
  • 100 Gb/s EDR Infiniband network interface
  • 10 GbE and 1 GbE network interface
  • 4 nodes with one EVGA Geforce GTX 1080 PCIE 8GB GDDR5
  • 29 nodes with two EVGA Geforce GTX 1080 PCIE 8GB GDDR5
  • CentOS 7.x
 14 Broadwell Compute nodes
  • Two 2.2GHz 12-core Intel Xeon E5-2650 v4, 30M Cache
  • 64GB 2133MHz RAM
  • 1TB HDD
  • 100 Gb/s EDR Infiniband network interface
  • 10 GbE and 1 GbE network interface
  • CentOS 7.x
 1 Haswell BigMem node
  • Two 2.6GHz 8-core Intel Xeon E5-2640 v3, 20M Cache
  • 512GB 2400MHz RAM
  • 1TB HDD
  • 100 Gb/s EDR Infiniband network interface
  • 10 GbE and 1 GbE network interface
  • CentOS 7.x
 12 Skylake Compute nodes
  • Two 2.3GHz 18-core Intel Xeon Gold 6140, 24.75M Cache
  • 192GB 2666MHz RAM
  • 1TB HDD
  • 100 Gb/s EDR Infiniband network interface
  • 10 GbE and 1 GbE network interface
  • CentOS 7.x
 12 Skylake Quad GPU nodes
  • Two 2.3GHz 18-core Intel Xeon Gold 6140, 24.75M Cache
  • 192GB 2666MHz RAM
  • 1TB HDD
  • 100 Gb/s EDR Infiniband network interface
  • 10 GbE and 1 GbE network interface
  • Four ASUS Geforce RTX 2080 TI PCIE 11GB GDDR6
  • CentOS 7.x

Intel XEON processors AVX2 and AVX512 frequencies


Condo Investments

Sol is designed to be expanded via the Condo Investment program. Faculty, Departments or Colleges can invest in Sol by purchasing nodes that will increase the overall capacity. Investors will be provided with an annual access equivalent to the amount of computing core hours or service units (SU) corresponding to their investment that can be shared with collaborators (students, postdocs and other faculty) for a 5 year period (or length of hardware warranty). Sol users, Condo and Hotel Investors, can utilize all available nodes provided their allocations have not been consumed.

Current Investors

  1. Dimitrios Vavylonis, Department of Physics: 1 20-core compute node

    • Annual allocation: 175,200 SUs

  2. Wonpil Im, Department of Biological Sciences:

    • 25 24-core compute node with 2 GTX 1080 cards per node (5,256,000 SUs)

    • 12 36-core compute nodes with 4 RTX 2080 cards per node (3,784,320 SUs)

    • Total Annual allocation: 9,040,320 SUs

  3. Anand Jagota, Department of Chemical Engineering: 1 24-core compute node

    • Annual allocation: 210,240 SUs

  4. Brian Chen, Department of Computer Science and Engineering: 1 24-core compute node

    • Annual allocation: 210,240 SUs

  5. Edmund Webb III & Alparslan Oztekin, Department of Mechanical Engineering and Mechanics: 6 24-core compute node

    • Annual allocation: 1,261,440 SUs

  6. Jeetain Mittal & Srinivas Rangarajan, Department of Chemical Engineering: 13 24-core Broadwell based compute node and 16 GTX 1080 cards

    • Annual allocation: 2,733,120 SUs

  7. Seth Richards-Shubik, Department of Economics

    • Annual allocation: 140,160 SUs

  8. Ganesh Balasubramanian, Department of Mechanical Engineering and Mechanics: 7 36-core Skylake based compute node

    • Annual allocation: 2,207,520 SUs

  9. Department of Industrial and Systems Engineering: 2 36-core Skylake based compute node

    • Annual allocation: 630,720 SUs

  10. Lisa Fredin, Department of Chemistry: 2 36-core Skylake based compute node

    • Annual allocation: 630,720 SUs

  11. Paolo Bocchini, Department of Civil and Environmental Engineering: 1 24-core Broadwell based compute node

    • Annual Allocation: 210,240 SUs

  12. Hannah Dailey, Department of Mechanical Engineering and Mechanics: 1 36-core Skylake based compute node

    • Annual allocation: 315,360 SUs

Accounts

A Principal Investigator can request accounts for his/her users for $50/user/year. Each account is provided with 150GB home storage quota. Every user will need to have an active allocation to use Sol. Sharing of accounts is explicitly forbidden and will result in forfeiture of accounts.

If additional storage is required, a PI can purchase a Ceph Storage project for his/her group based on a 5 year purchase. PIs who purchase a Ceph Storage project, minimum $375/TB, can opt for using their Ceph space for home directories and have their account fees waived for the life of the Ceph project. More details of Ceph Pricing is available here .


Allocations

Principal Investigators, who are not Condo Investors, called Hotel Investors henceforth, will be able to purchase computing time, if available, on an annual basis. Computing time equivalent to 8 20-core compute nodes or 1,400,000 SUs is available for general use on an annual basis. The cost per SU is fixed at 1¢ with a minimum annual buy-in of 50,000 SUs with increments of 10,000 SUs.

  1. Minimum Annual Allocation (50K SUs): $500

  2. 10K Increments: $100

The allocation cycle is fixed at one year and unused allocations (the minimum and any increments purchased during that cycle) will not rollover to the next allocation cycle nor be refunded. The initial allocation cycle will begin on Oct 1, 2016 and end on Sep. 30, 2017. Implementing a rolling allocation cycle for Hotel Investors i.e. allocation cycle begins the day you initiated a purchase rather than a rigid start date (Oct 1.) is a work in progress.

Condo Investors who utilize their allocated cycles before the cycle ends can purchase additional 10K increments, if available, and are not subject to the minimum allocation purchase. However, any 10K increments purchased must be expended before the next allocation cycle begins. There are no refunds or rollovers of SUs from one allocation cycle to the next.

For example, a PI who estimates requiring 75K SUs annually will need to purchase 80K SUs for $800 annually. Suppose the PI only consumes 60K SUs during the allocation cycle, the unused 20K SUs will not rollover to the next allocation cycle, nor will the PI be reimbursed for the unused SUs. Alternatively, the PI can purchase the minimum 50K SUs for $500 and purchase additional increments of 10K whenever his/her allocation balance is low. The PI should consider that the availability of 10K increments for purchase is not guaranteed.


Logging into Sol

Sol can be accessed via SSH using a SSH Client. Linux and Mac users can login to Sol by entering the following command in a terminal:

ssh username@sol.cc.lehigh.edu

If you are off campus, then there are two options

  1. Start a vpn session and then login to Sol using the ssh command above
  2. Use ssh gateway as a jump host first and then login to Sol using the above ssh command on the ssh gateway prompt. If your ssh is from the latest version of openssh, then you can use the following command
ssh -J username@ssh.cc.lehigh.edu username@sol.cc.lehigh.edu

If you are using the ssh gateway, you might want to add the following to your ${HOME}/.ssh/config file on your local system

  Host *ssh
  HostName ssh.cc.lehigh.edu
  Port 22
# This is an example - replace alp514 with your Lehigh ID
  User alp514

  Host *sol
  HostName sol.cc.lehigh.edu
  Port 22
  User <LehighID>
  ProxyCommand ssh ssh nc %h %p

to simplify the ssh and scp (for file transfer) command. You will be prompted for your password twice - first for ssh and then for sol

ssh sol
scp sol:<path to source directory>/filename <path to destination directory>/filename 

If you are using public key authentication, please add a passphrase to your key. Passwordless authentication is a security risk. Use ssh-agent and ssh-add to manage your public keys. See https://kb.iu.edu/d/aeww for details.

Windows users will need to install a SSH Client to access Sol. Lehigh Research Computing recommends MobaXterm since it can be configured to use the SSH Gateway as jump host. DUO Authentication is activated for faculty and staff on the SSH Gateway. If a window pops up for password enter your Lehigh password. The second pop up is for DUO, it only says DUO Login. Enter 1 for Push to DUO or 2 for call to registered phone. 


  • No labels