I’m proud to have been associated with the August 2012 landing of the Curiosity rover on Mars. Since 1983 I’ve been working for and then contracting to NASA’s Jet Propulsion Laboratory (JPL), and I work with IT infrastructure on providing web services to groups in the lab that want to use us. In February we were contacted by the group responsible for mars.jpl.nasa.gov and asked to help with the EDL (Entry-Descent-Landing) event. The existing servers at JPL, and the JPL network, could not handle the projected load of millions of people around the globe hammering the service.
We have an expert on Amazon Web Services (AWS) on our team, and he created an architecture that’s already been published by Amazon, so I won’t reiterate it here. (They’re mostly right; there are some differences and nuances they didn’t capture.)
Perl helped us tremendously during this process, mostly with gathering, troubleshooting, and reporting data. When things weren’t running smoothly, we needed to analyze data such as web server logs, netstat output, process listings, and the quickest way to crunch that data on the fly was with Perl one-liners, like this monster:
perl -MSocket -lane '$F[-1] =~ /^([d.]+)/ or next; $h{scalar gethostbyaddr(inet_aton($1),AF_INET)}{$F[6]}++; END{ for $x ( keys %h ) { $h{$x}{$_} > 1 and print sprintf( "%4d ", $h{$x}{$_} ) . "$x - $_" for sort { $h{$x}{$b} <=> $h{$x}{$a} } keys %{$h{$x}} } }' access_log
That generates a ranked list of hits by server and URI path from an access log (in practice, a subset of one). That would be a horrendous piece of code to be put in a program! But this is what one-liners are for. Command history recall and editing comes in handy, but I was able to type that in one go, because I’ve done component idioms of that command (in particular the concordance idiom – counting occurrences in a hash) so often. A later version included caching for the DNS lookup. If we wanted to do this on anything like it on a regular basis then I would have written a program instead, but this was my quick answer to a one-off question that needed to be answered in less than a minute.
Perl was helpful at higher levels as well. When we wanted custom reports not provided by the AWS dashboard, a quick search of CPAN turned up the Net::Amazon::EC2 module, and a few minutes later, a monitor was born, one that continues to be used by the team today. We are gathering metrics in Splunk and I expect to be turning to WWW::Splunk shortly.
The engineers and staff who designed, built, and flew the Curiosity rover are the real heroes of this endeavor, of course, and always have been (Google for “Real programmers don’t eat quiche” and look for the mention of JPL in that decades-old document). I’m just happy to have helped spread the news of their tremendous exploits and discoveries to the rest of the world. It was among the most exciting tasks I’ve ever undertaken using Perl.