His nth Bow: Hartree Summer School 2014: High Performance Computing

The Hartree Centre is an STFC centre for scientific computing, home to the UK's fastest academic super computer - Blue Joule an IBM Bluegene/Q, an amazing visualisation suite, and support for all HPC applications in research and industry.

The Course

Prof. Jack Dongarra (Uni. Tennessee, Uni. Manchester, Oak Ridge) lead the teaching staff of the week long event along with Dr. Kirk Jordan (IBM), the school was capably organised by Terry Hewitt, Damian Jones and Dave Cable from the STFC. The whole event was 3 weeks - the first on visualisation and the last on big data, though I only attended the middle week on HPC.

The week involved talks and practical sessions on; general HPC (Jack Dongarra), parallel programming with X10 (Olivier Tardieu, IBM), OpenMP (Christian Terboven, RWTH Aachen), MPI (Chris Wareing, Uni. Leeds), numerical linear algebra (Julien Langou, Uni. Colorado), HPC architectures (Kirk Jordan), computational steering (John Brooke, Uni. Manchester), visualisation (David Duke, Uni. Leeds), and big data (Chris Williams, IBM).

David Duke from the University of Leeds giving a talk on visualisation

Although the course was largely attended by computer scientists, there were also many other scientists, including those from physics, engineering, chemistry, and business.

The weather was surprisingly good over the week, breaks were spent on the canal bank.

The Bridgewater canal running alongside Daresbury Laboratory

Blue Joule

The Hartree Centre is home to the UK's 3rd most powerful supercomputer ( #14 top500 list 39, June 2012, #23 top500 list 43, June 2014). We visited Blue Joule and Blue Wonder. Blue Joule is a 1.2 Petaflop/s 131,072 core IBM BlueGene/Q. Blue Wonder is a 158.7 TeraFlop 8192 core iDataplex.

The ~21 Petabytes of tape storage for Blue Joule taken from here

The rooms are temperature controlled and full of cameras. Some of the racks are water cooled, others are not. The systems contained a lot of varied hardware, including; FPGAs, Xeon Phis, standard nodes, large amounts of memory (tape, HDD, and flash), switches, controllers, power supplies and plenty of cabling. We were told that the system is the largest USB hub in the UK (at >2048 USB connections). Blue Joule is also one of the greenest supercomputers in the world.

Blue Wonder taken from here

Blue Joule taken from here

Visualisation Suite

Hartree has a rather large stereographic 3D power screen for visualisation of scientific data. Some useful features include the ability to observe simulations (running on Blue Joule) in real time, allowing a user to change simulation variables if necessary. For example if something has gone beyond a realistic physical limit, a job can be stopped (saving money, time, and power), or variables can be changed to make the most of remaining resources.

3D Visualisation Suite taken from here

Information on the visualisation opportunities available at the Hartree Centre can be found here.

Interesting Facts

OpenMP and X10 (developed by IBM for the past ~decade) are mainly designed to implement task based code on many core machines, whereas MPI is designed to allow users to work across many nodes or machines. OpenMP is a model for shared memory machines.
Although the top HPC machines (top500) may cost over 200 million dollars each, the cost of power, maintenance, cooling etc is often much more. Running costs can mean that the total cost of buying, running, and maintaining a top500 machine for ~5 years can be over $1 billion.
After the HPC system lifetime (~5 years) is over, the machines are often scrapped (literally), the silicon isn't reused and gold is often stripped from the machine boards.
The power use of a processor scales like it's frequency cubed (f^3), which is why chip manufacturers move into increased transistor density rather than increasing the frequency when improving processor performance.

Slide delivered by Intel, taken from this article

Intel are soon to release a standalone Xeon Phi CPU called Knights Landing, which will deliver 3 TeraFlops, and as it is does not have to be used as a coprocessor via PCI-Express, it should offer massive performance gains over existing 'accelerators' such as GPUs and existing Xeon Phi (Knights Corner), which are bottlenecked by the PCIe bus.
Power consumption is the primary design constraint for future HPC system design.

Adventure Time

Myself and David Bruton decided to walk from Daresbury Laboratories to the Britannia Daresbury Park hotel. There was some confusion when we encountered an abandoned bridge.