User of the Month December 2013: Joel Brownstein

I am pleased to honor Joel Brownstein from the University of Utah as the Globus User of the Month for December 2013.

Professor Joel Brownstein is the data archivist of the Sloan Digital Sky Survey-IV (SDSS-IV), which is the continuation of the SDSS-I, -II, and -III surveys.

The SDSS-I, -II, and -III surveys have together amassed hundreds of terabytes of data from millions of stars throughout the Milky Way, and distant galaxies and quasars that allow us to study the structure and evolution of galaxies, the nature of dark energy and dark matter, and the cosmological history of the universe.

Joel and Professor Adam Bolton (the SDSS-IV data director) have developed a Python API to leverage the power of Globus services, and to transfer daily telescope observations to the University of Utah, where they have a large cluster to run their various data reduction pipelines. They are leveraging the script-friendly Globus command-line interface via cli.globus.org within their Python API. Joel has moved more than 70 TB of data so far.

"My experience with Globus has made moving large amounts of astrophysical data faster than rsync," said Joel. "I particularly appreciate the command-line interface for automating transfers by means of simple-to-write Python scripts to push smaller amounts of data for our daily telescope operations."

"We are planning to enlarge our usage of Globus services by adding public and collaboration-specific endpoints," Joel added.

They are planning to distribute their results to the collaboration (which will be immediately available) and to the public (via major data releases). They have set up multiple dedicated Data Transfer Nodes (DTNs) and run Globus Connect Servers on them, which are available as three different logical endpoints on Globus for their users: one authenticated read-write endpoint, one authenticated read-only endpoint, and one anonymous read-only endpoint. The authenticated endpoints are for the users in the collaboration and the anonymous endpoint is for the public.

"I particularly appreciate the web interface for manual transfers and special cases, and the CLI for scriptable transfers," Joel said. "For our use case, the most important feature is the integrity of the data. For this, we rely upon the ability to preserve timestamps, file integrity verification after transfer, and the occasional checksumming."

Congratulations Joel! We would like to thank you for being a Globus user, and for providing valuable feedback and requesting interesting and useful new features.

Joel and Adam gratefully acknowledge the support and resources from the Center for High Performance Computing at the University of Utah.