Computing
NSF/S.Lidstrom
All major particle physics experiments are also complex computing projects, with data being collected, sorted, and sent around the world for analysis. With three of four WIPAC projects located at the geographic South Pole, there are unique challenges to data processing.
Data production, filtering, and simulation for the IceCube project take place at the University of Wisconsin-Madison and DESY-Zeuthen. At UW-Madison, WIPAC manages computing facilities at 222 West Washington, the Wisconsin Institutes for Discovery, and Chamberlin Hall, home of the Department of Physics.
NSF/F.Descamps
South Pole Infrastructure
At the South Pole, the IceCube Lab is the first stop for raw data collected from IceCube, ARA and DM-Ice. One terabyte of data comes off the detectors daily and is filtered down to 105 gigabytes for satellite transmission to the north. The raw data is copied to tapes which are shipped up from the Pole every year.
The IceCube Lab holds around 150 servers for processing data. One hundred of those are custom built servers called “DOM hubs.” There is one DOM hub for each string of detectors in the IceCube/IceTop configuration. The other fifty are Dell PowerEdge servers.
NSF/IceCube
Data Production and Simulation
Basic, level one data is the filtered stream that comes from the South Pole. Level two cuts provide the basic reconstruction of upgoing muons and good reconstructions of downgoing muons. Further levels of data filters are set by working groups within project collaborations. Each group has specific parameters for direction and energy level as they look for specific neutrino data.
In addition to managing information from the detector, data simulations are regularly produced. While analyzing data, physicists use simulations to help them understand what they are seeing. Neutrino event simulations are almost impossible to tell from actual event data and are revisited as researchers analyze data.
NSF/L.Norris
GZK9000 GPUs
IceCube event simulation requires a lot of processing power. To keep up with the needs of our researchers, in December 2011, WIPAC began using a cluster of general purpose graphics processing units, or GPGPUs.
The cluster, called GZK9000, is housed at the Wisconsin Institutes for Discovery and contains 48 NVIDIA Tesla M2070 GPUs. The GPUs enable researchers to produce simulations more quickly by eliminating the photonics lookup tables that were once needed to construct particle showers in the detector. In addition, as we learn more about ice properties, we can easily adjust the simulations to incorporate new information about how light will scatter.
Using GPUs to process images also enables us to connect to larger computing networks, such as the Wisconsin-based Center for High Throughput Computing and GLOW resources, as well as the international Open Science Grid, because of the reduced usage of photonics lookup tables.