Today, I had an epiphany on generating a report inside of the Illinois Campus Cluster (ICC) with R through rmarkdown. For a long time, I’ve wanted to avoid downloading data from the cluster to my personal computer and, then, running a report generation. Alas, my priorities never aligned with solving this problem as downloading data was quick inside of a University building; but, with COVID-19, my internet is no longer Mazda’s “zoom zoom” fast. I would likely wager downloading the same amount of data is equivalent to placing it on a flash drive and shipping it overnight via FedEx.
The epiphany was simple:
What if we had pandoc on the cluster?
For those who aren’t familiar with pandoc, the software serves as a universal document convert or in more relatable terms it is the “swiss-arm knife” for moving between different document formats. Alas, pandoc is built ontop of Haskell, which wasn’t available on the cluster. So, under usual operating principles on the cluster, I would have to build pandoc from source; though that would require setting up a Haskell environment or so I thought…
But, wait… There’s a binary! From pandoc’s linux section, there is a binary package for amd64 arhitecture that is standalone with both pandoc and pandoc-citeproc. Both binaries are statically linked and have no dynamic dependencies or dependencies on external data files. Huzzah! Let’s try out the binary…
Dynamic retrieval script
First, we want to always get the latest version of a software release from GitHub. With a quick Google, we land at Hanwen Wu’s One Liner to Download the Latest Release from Github Repo gist. Though, we removed the wget to allow for a more targeted pipe into tar.
# Determine latest version from GitHub
LATEST_RELEASE_URL=$(curl -s https://api.github.com/repos/jgm/pandoc/releases/latest | grep "browser_download_url.*amd64.tar.gz" | cut -d : -f 2,3 | tr -d \")Next, we’ll need to retrieve the binary name, download the file, and unpack to our local binary location.
# Destination directory (bin will be created inside)
DESTDIR=~/project-stat/
# Retrieve filename
PANDOC_FILENAME="${LATEST_RELEASE_URL##*/}"
# Download the latest pandoc version
wget -q ${LATEST_RELEASE_URL}
# Unpack into $DESTDIR/bin
tar xvzf ${PANDOC_FILENAME} --strip-components 1 -C ${DESTDIR}From there, we need to append onto the PATH variable the location of where the binary can be found. To do so, place in ~/.bashrc:
export PATH="~/project-stat/bin:${PATH}"Replace ~/project-stat with the appropriate directory.
Then, open up R and trigger the render of the report using rmarkdown:
rmarkdown::render("path/to/RmarkdownFile.Rmd")Once the report is generated, the next step would be to send e-mail with it attached at the end of the simulation.
Fin
In short, this post showed how to dynamically retrieve the latest version of pandoc from GitHub, extract it into a local bin directory on the cluster, and, then, how to include the bin directory to be recognized by R to generate documents using rmarkdown.
Acknowledgements
Special thanks to
NCSA’s Weddie Jackson, who has been my go-to person for getting the R version on the cluster upgraded ever so often and who triggered the epiphany with an e-mail. (He bumped the cluster version to R 4.0.0 as well.)