O’Reilly School of Technology Author has a Close Encounter on Mars with the Rover Curiosity

I’m proud to have been associated with the August 2012 landing of the Curiosity rover on Mars. Since 1983 I’ve been working for and then contracting to NASA’s Jet Propulsion Laboratory (JPL), and I work with IT infrastructure on providing web services to groups in the lab that want to use us. In February we were contacted by the group responsible for mars.jpl.nasa.gov and asked to help with the EDL (Entry-Descent-Landing) event.  The existing servers at JPL, and the JPL network, could not handle the projected load of millions of people around the globe hammering the service.

We have an expert on Amazon Web Services (AWS) on our team, and he created an architecture that’s already been published by Amazon, so I won’t reiterate it here. (They’re mostly right; there are some differences and nuances they didn’t capture.)

Perl helped us tremendously during this process, mostly with gathering, troubleshooting, and reporting data. When things weren’t running smoothly, we needed to analyze data such as web server logs, netstat output, process listings, and the quickest way to crunch that data on the fly was with Perl one-liners, like this monster:

perl -MSocket -lane '$F[-1] =~ /^([d.]+)/ or next; $h{scalar gethostbyaddr(inet_aton($1),AF_INET)}{$F[6]}++; END{ for $x ( keys %h ) { $h{$x}{$_} > 1 and print sprintf( "%4d ", $h{$x}{$_} ) . "$x - $_" for sort { $h{$x}{$b} <=> $h{$x}{$a} } keys %{$h{$x}} } }' access_log

That generates a ranked list of hits by server and URI path from an access log (in practice, a subset of one). That would be a horrendous piece of code to be put in a program! But this is what one-liners are for. Command history recall and editing comes in handy, but I was able to type that in one go, because I’ve done component idioms of that command (in particular the concordance idiom – counting occurrences in a hash) so often. A later version included caching for the DNS lookup. If we wanted to do this on anything like it on a regular basis then I would have written a program instead, but this was my quick  answer to a one-off question that needed to be answered in less than a minute.

Perl was helpful at higher levels as well. When we wanted custom reports not provided by the AWS dashboard, a quick search of CPAN turned up the Net::Amazon::EC2 module, and a few minutes later, a monitor was born, one that continues to be used by the team today.  We are gathering metrics in Splunk and I expect to be turning to WWW::Splunk shortly.

The engineers and staff who designed, built, and flew the Curiosity rover are the real heroes of this endeavor, of course, and always have been (Google for “Real programmers don’t eat quiche” and look for the mention of JPL in that decades-old document). I’m just happy to have helped spread the news of their tremendous exploits and discoveries to the rest of the world. It was among the most exciting tasks I’ve ever undertaken using Perl.

 

  • http://twitter.com/sbollig Scott Bollig

    My brain hurts just looking at that code.  Did I hear correctly that you typed this “on the go,” meaning from your head to the keyboard?  I am in your Perl 2 class now, is this what I have to look forward to in Perl 3 or 4?  Yikes! ;-)  (Nice to hear that Perl is doing this great work.)

  • Gonzalo Gasca

    Hi Peter, is great to see how Perl and your knowledge had an impact which is reflected in the real advance of technology specifically for this milestone for human race. I have completed your classes Perl 1 and 2. And is inspiring what can be done with Perl. I remember the moon landing decades ago using TV and now Curiosity in this Internet era. thanks for sharing the experience