Hypomnemata

Actually, since we don’t currently have a public blog or wiki, I thought it’d be nice to collect a bunch of Project Bacchus stuff in one place.

We started Project Bacchus in late October, 2009.  My friend Shannon approached me with a question: “What do you think about sending a camera to the edge of space?”  My answer was quick: “That’s why I got my ham radio license!”  We pulled in a few more people, and a plan emerged through weekly meetings, at which we always have beer. Always.

Our project was kept quiet for a while, in part to give ourselves room to fail (which we did), and in part to keep it coherent. The more people you have, the more ideas/goals you have, the less you can focus. By keeping it to just five guys, we stayed on track, though we did probably bite off too much for our first few launches.

The biggest challenge has actually been scheduling flights, with weather a close second. Everyone wanted to have the whole team present at our first big launch, which we did. Getting there was a challenge, especially in December: between the holidays and the rain, we didn’t have a single launch window. We’ve since moved to a much nimbler group, able and willing to fly with only three guys in the field. That’s worked out really well for us, as you’ll see.

Without further ado, a quick recap of most of Project Bacchus up to the present:

Bacchus I: 2009, November 21; abject failure.

My apartment, the night before Bacchus I

Adam has a flickr set of our attempt here, 20091121 Project Bacchus I.  Note that all these pictures were taken on November 21.  The ones in my apartment were just after midnight, the rest were at about 7am, an hour and a half drive from San Francisco.  This is on top of an all-nighter on Thursday by Michael and myself, which resulted in a nice, solid payload, but slightly frazzled nerves.

Bacchus I's Flight Computer: Spaghetti (Ted Scharff)

That’s a great shot of the flight computer’s (totally spaghetti) guts by Ted; he’s got more electronics porn in his Bacchus I Gallery.  There’s other documentary evidence of this trip, but it’s mostly depressing, so I’ll pretend it doesn’t exist.  On the positive side, I got an inadvertently free margarita at brunch after our failure.

Bacchus I, which never left the ground (Ted Scharff)

In summary, we learned a lot from our first failure.

Bacchus II: 2010, January 10; payload lost at launch. RECOVERED! 2010, May 29

The balloon, tethered between our cars (Adam Fritzler)

Adam’s the most photo-happy member of the group (but Michael and Ted are both pretty photo-happy themselves), so I’ll link to his set first again: 20100111 Bacchus II Launch.  This launch was in a valley that’s socked in with fog, underneath otherwise crystal-clear skies.  Launching through the fog was technically against FAA regulations, but there was no behavioral difference between us launching in clear skys and through the fog: pilots couldn’t have seen our balloon until the same altitude either way.

(L to R) Ted, myself, Michael, and Shannon (hidden) rig Bacchus II (Adam)

But maybe the FAA regs aren’t there for the pilots, because we immediately lost visual contact with the payload, and the ham radio rig only got three check-ins (up to 9,000′!) before it went silent.  We had a backup cell phone in the payload as well, but it never checked in.  The assumption is a catastrophic failure shortly after launch, but we’ll never know.  For now, the official story is pterodactyls.

Update, 2010, May 31 Bacchus II was found! A rancher moving cattle down to summer grazing land came across Bacchus II dangling from a tree. It was in a remote valley with no cell phone coverage, but the phone seemed to be in okay shape. The current theory is that the radio failed shortly after launch. We’ll know more in the next week or two–the call came in two days before Bacchus VI, which was retrieved from the top of a peak in the foothills (technically a mountain), which was yesterday. Needless to say, we’re a little overwhelmed between the post-processing of mission data from VI, post-processing of the data from II, and making time to post-mortem the failure of II’s hardware. Regardless, welcome home, prodigal probe!

Bacchus II: It's About Quality™ (Ted Scharff)

Again, we learned a lot from our failure here. For instance, our new fill method is nice and elegant; the fill method used in Bacchus II… less than elegant. There’s a lot of good photos in Ted’s Bacchus II Picasa Gallery.

Bacchus III: 2010, February 14; payload retrieved.

A camera, dangling from a balloon, took this.

After throwing away several hundreds of dollars of electronics on Bacchus II, we went back to basics for Bacchus III.  One tortilla warmer, one cell phone, and one camera.  We didn’t put on a chute, since we had a very light payload and a giant streamer, which worked out really well.  A couple hours after launch, we came upon Bacchus III lying, undignified, in a fallow field, nary a scratch on it.  The telemetry from the phone’s GPS shows it coming down at about 40mph, which is a little zippy, but not dangerous with as much foam as we had.

Bacchus III Away Team: Shannon, Adam, myself, Michael (Michael Toren)

As part of our “lean and nimble” goal for this launch, we went with just four of the five team members.  Ted had other plans that weekend, so we ran off without him.  We did have his hiking GPS, though, which, once we figured out the UI, was invaluable.

Bacchus III, on the ground (Michael Toren)

We retrieved the payload and got a bunch of photos off of it.  Adam went through and edited them down to the best ones: 20100214 Bacchus III (payload camera).  We thank him greatly for that, as it’s kind of a huge pain, and he’s really good at it.

Based on the data we collected here, we’ve been able to understand the dynamics of the system, predict future behavior, and better plan future flights.  We’re working towards a payload for Bacchus IV in a couple weeks, which is part of why I was reviewing battery holders in the last post.

Bacchus Kite Tests: 2010, February 20; way fun.

Myself, Ted, and Michael rig a "test" payload (Adam Fritzler)

To test some of our descent gear, we decided to use a kite as a flying platform.  We had a few things to test this way, but only really got to one of them this time.  Mostly, this was a good excuse to go out into the park and drop stuff from kites.  It’s really, really fun, try it sometime.  Also, it gave me an excuse to buy a 200′ measuring tape, which is probably one of the more specialized pieces of equipment I own.

Mission accomplished! (Ted Scharff)

After we finished our actual goals (breaking the hell out of some styrofoam), we went ahead and did something silly: dropped the camera, recording video, from 100′ with a Wal-Mart bag as a parachute.  It’s really, incredibly disorienting, but damned fun. Video as soon as wordpress stops being stupid about it.

And that’s it. You’re now caught up on Project Bacchus!

Short version: the Roku Channel SDK License includes language that prohibits GPL-bearing applications.  Given that their device runs a GPL’d OS with a bunch of other OSS infrastructure, this is disappointing, and F/OSS authors should be careful in accepting the Roku SDK’s terms of service.

If you read my old blog, you may remember that I found a way to  Download the Roku Firmware a while ago. It was clear that one could load custom firmware via a man-in-the-middle technique. In my case, I wanted to place my ripped movies into the Netflix Play-it-Now queue. It was technically feasible, but sort of a pain (it required reversing the Netflix protocol, which required adding custom CA certs to the image, and and and…), so I dropped it.

Fast forward to now-ish, with the introduction of the Channel Store on Roku.  I just noticed Roku’s Channel Developer SDK in an email announcement they sent out.  Exciting!  With a little elbow grease, I could watch my ripped DVDs, use YouTube on my TV (Google TechTalks and MIT OpenCourseWare!), or maybe even have a nice weather channel.  I was so stoked, I got out of bed an hour earlier than usual to look into it.  No, really.

Part of my excitement was knowing it would be familiar.  Having seen their firmware, I know the Roku is a Linux box, running X windows, and using QT for its UI.  All of this is Open Source Software, and I’ve used it all before: this is great!  I can bring my former experience to bear, and a lot of the Open Source stuff I’ve been using in the past should integrate pretty flawlessly.

I went and started signing up for the process.  Before downloading the SDK, though, you have to agree to The Roku Channel Developer Agreement. Instead of blindly clicking, I decided to read it.

5.A.iii:   Subject to the Grace Period, Your Channel Application must at all times: … not contain any open source code or other restricted code that could require Roku to publicly post or display any third party notices or any modifications to such code.

Recently @edrabbit had a great idea: “I want to install a cache of balloons and confetti over my desk for when I achieve inbox zero.”

It was brilliant, so I kludged together the first step: a ruby script that takes an action when you hit inbox zero.  Here’s the code, if you want to play along at home.  You’ll need to edit your username and password; if you’re on Linux, “gnome-open” is probably the right thing to have there.  On OS X, you’ll want “open.”  If you’re on Windows, you have your own little set of special problems.

(Note that this program exits 0 when you’ve got inbox zero, and 1 otherwise, so you can easily put it into a shell script.  In addition to the exec() command in the script, this could be used to trigger a program that pings an arduino to unlatch a box of balloons and confetti.  For my sake, I’ll be thrilled to have “Peanut Butter Jelly Time” pop up when I hit inbox zero.)

#!/usr/bin/env ruby
# You need to add your username and password below!
require 'net/imap'

config = {
  'server' => "imap.gmail.com",
  'port' => 993,
  'username' => "josh.myer@gmail.com",
  'password' => "",
  'action' => "gnome-open http://bit.ly/pbj_time",
}

imap = Net::IMAP.new(config['server'], config['port'], true)
imap.login(config['username'], config['password'])
imap.select('INBOX')
n = 0
imap.search(["NOT", "DELETED"]).each do |message_id|
  n += 1
end

imap.logout()
imap.disconnect()

puts "Your inbox contains #{n} not-deleted messages."

if n == 0
  puts "SUCCESS!  INBOX ZERO ACHIEVED!"
  exec(config['action'])
  exit 0 # superfluous, I know
else
  exit 1
end

I’ve recently been working my way through Write Yourself a Scheme in 48 Hours. This is a guided tutorial through implementing a Scheme interpreter in Haskell. It’s been really fun, and a nice change from my current contract (which involves a lot of screwdriver turning and other sysadmin duties).

That said, there is one big change that I’d like to see made in the text: build it with unit tests! I’m not a total test-driven disciple, but I do believe they’re a fantastic tool, and this is a great place to use them.

Parsing is incredibly test-friendly: you have an input string, and an expected output parse tree. It’s totally deterministic, with no random numbers, no ambiguities, and no pesky humans. There are a lot of interacting components, especially in terms of nesting, which are hard to keep in mind when manually “testing” things.

In addition, the “here’s code, extend it to do X” pattern used in the text benefits from having comprehensive tests. In working the homework exercises, you’re going to refactor big chunks of code. As you do so, you should be worried about breaking the hell out of your existing code (if you’re not worried, you’re either brilliant, inexperienced, or inattentive). There’s no better way to assuage those worries than unit tests, and, as you find broken things, adding regression tests. It gets even better if you add code coverage, guaranteeing that you’re exercising all codepaths via your tests.

Another wrinkle that makes you want tests: as you work through the book, you merge your existing code into the next snippet of example code. This is often an error-prone process, and having tests to catch mistakes is great. This is the reason I broke down and refactored the code for testing. I started getting into the interesting stuff at the end of chapter two, and suddenly my strings weren’t parsing properly. Upon examination, I found I’d merged my code into the next sample code incorrectly, and left my parseString implementation behind.

So, with all that in mind, let’s make Write Yourself a Scheme more test-friendly. The first step is to factor the parser implementation out into its own module. This is really easy: we create WYAS.hs, containing everything but main, and change the module line to:

module WYAS where

We’ll also need to extend LispVal a little bit, too. We want to be able to compare them, so we’ll add deriving(Show, Eq) immediately after the definition:

data LispVal = Atom String
             | List [LispVal]
             | DottedList [LispVal] LispVal
             | Number Integer
             | Float Float
             | String String
             | Char Char
             | Bool Bool
             deriving (Show, Eq)

Don’t worry about what “deriving()” means here; it’s just some Haskell typeclass voodoo that lets us now show (Char 'x') and have it do something useful, as well as allowing us to compare two LispVals for equality. After adding deriving(Show), you can also make readExpr‘s Right case return the actual parse tree as a string:

    Right val -> "Found value: " ++ show val

Handy, no?

Then, we add main.hs, containing:

module Main where
import WYAS
import System.Environment

main :: IO ()
main = do args <- getArgs
    putStrLn (readExpr (args !! 0))

Next, let’s compile this, and make sure we get something that acts like our old parser:

ghc --make -o main main.hs
./main 42
./main "#t"

Once we’re convinced that’s all working, save it and commit it to your version control. (You are using version control, right? svn, git, cvs, hg… they’re all great, and totally worth using here!)

Now, we add a test framework. First and foremost: make sure you have HUnit installed (on ubuntu and debian, it should be ‘apt-get install libghc6-hunit-dev libghc6-hunit-doc‘).

Then, we’ll get to using HUnit to write unit tests. I don’t really want to explain HUnit in too much detail, in part because I still don’t fully grok it. Suffice it to say, the following works, and gets good results. If you have better options, please leave them in the comments, as this is one of those things that could always use refinement.

For my purposes, I’m only really interested in running tests on each parse type. That is: I want to put together a bunch of strings representing possible Number inputs, along with their parses, and run them one after the other. I’d like to have a string to tell me the name of the grouping, but, beyond that, I’m not too picky on how much debug output I get. When an input string fails (and it will), I can use the ./main binary to examine the output parse. So, I’m going to write a small templating facility in my test setup. I’ll define a ParseTestSuite, which is a String identifying the group with a list of (String, LispVal) pairs. The String is the input, and the LispVal is the expected parse. Once I have that, add some code around it to turn them into HUnit’s internal Test representation. At the very end, we collect all these Tests, and run them. Here’s the resultant test.hs:

module Main where
import WYAS
import Test.HUnit
import Text.ParserCombinators.Parsec hiding (spaces)

-- A ParseTestSuite is a name, then a list of strings, and their expected parses
-- For instance:
--   ParseTestSuite "testNumbers" [("32", Number 32), ("3.4", Float 3.4), ..]
--
data ParseTestSuite = ParseTestSuite String [(String, LispVal)] deriving (Show)

doParse :: String -> LispVal

doParse input = case parse parseExpr "lisp" input of
    Left err -> Atom ("No Match " ++ show err)
    Right val -> val

-- Convert a ParseTestSuite into an HUnit test suite.
-- This requires extracting the boolean assert, then turning
-- it into a test case.
suiteEntryToAssert :: String -> LispVal -> Assertion
suiteEntryToAssert s p =
assertBool s (p == doParse s)

parseTestSuiteToTest :: ParseTestSuite -> Test
parseTestSuiteToTest (ParseTestSuite name tests) =
    TestLabel name bodies_t where
    bodies_t = TestList $ map (\(s,p) -> TestLabel s (TestCase (suiteEntryToAssert s p))) tests

tests_Number :: ParseTestSuite
tests_Number = ParseTestSuite "tests_Number"
    [
    ("32", Number 32),
    ("3.2", Float 3.2),
    ("0", Number 0),
    ("#d1", Number 1),
    ("#b10", Number 2),
    ("#x10", Number 16),
    ("#o10", Number      ]

tests_Bool :: ParseTestSuite
tests_Bool = ParseTestSuite "tests_Bool"
    [
    ("#t", Bool True),
    ("#f", Bool False)
    ]

-- lots of other tests elided

all_tests :: [ParseTestSuite]
all_tests = [tests_Number, tests_Bool, tests_Atom, tests_String, tests_Char, tests_List, tests_DottedList, tests_Quoted]

main :: IO ()
main = do
    runTestTT $ TestList (map parseTestSuiteToTest all_tests)
    return ()

Then, we compile this:

ghc --make -o test test.hs
./test

and its output:

0] jbm@density:~/src/wyas/ch2 $ ./test
Cases: 22  Tried: 22  Errors: 0  Failures: 0
0] jbm@density:~/src/wyas/ch2 $

From here, it’s just a matter of adding more of these ParseTestSuite entries and inserting it into all_tests. As we do this, we gain more confidence in the parser we’re building in this process.

There’s one more key consideration in confidence, though: code coverage. Unit tests are great, but they might not test all your code, leaving bugs lurking for some poor soul to find later. The best way to ensure this is to run your unit tests under a coverage tool. GHC comes with one of these, handily enough: hpc.

As with all coverage tools, we need to recompile the program with some new options. And, like many of them, we need to clean up the project a bit first. So, in the name of being lazy, I wrote a Makefile:

# NB: you're going to get this mangled.
# The things under the target names MUST be tabs, not spaces.
# The joys of ancient Unix programs.
#
all: test main

coverage: clean WYAS.hs test.hs
        ghc --make -fhpc -o test test.hs
        ./test
        hpc markup --include=WYAS ./test

test: WYAS.hs test.hs
        ghc --make -o test test.hs
        ./test

main: WYAS.hs main.hs
        ghc --make -o main main.hs

clean:
        rm -f *.tix *.hi *.o main test

distclean: clean
        rm -f *.html

This lets us run ‘make coverage‘ and get an HTML file showing the coverage of WYAS.hs:

0] jbm@density:~/src/wyas/ch2 $ make coverage
rm -f *.tix *.hi *.o main test
ghc --make -fhpc -o test test.hs
[1 of 2] Compiling WYAS             ( WYAS.hs, WYAS.o )
[2 of 2] Compiling Main             ( test.hs, test.o )
Linking test ...
./test
Cases: 22  Tried: 22  Errors: 0  Failures: 0
hpc markup --include=WYAS ./test
Writing: WYAS.hs.html
Writing: hpc_index.html
Writing: hpc_index_fun.html
Writing: hpc_index_alt.html
Writing: hpc_index_exp.html

To get a quick view of how we’re doing, we can use ‘hpc report‘:

0] jbm@density:~/src/wyas/ch2 $ hpc report test.tix
91% expressions used (416/457)
100% boolean coverage (1/1)
100% guards (0/0)
100% 'if' conditions (1/1)
100% qualifiers (0/0)
62% alternatives used (22/35)
100% local declarations used (6/6)
96% top-level declarations used (32/33)
0] jbm@density:~/src/wyas/ch2 $ hpc report test.tix  WYAS
91% expressions used (233/256)
100% boolean coverage (1/1)
100% guards (0/0)
100% 'if' conditions (1/1)
100% qualifiers (0/0)
63% alternatives used (21/33)
100% local declarations used (5/5)
95% top-level declarations used (19/20)
0] jbm@density:~/src/wyas/ch2 $

As you can see, my “alternatives used” is rather low. Looking at the highlighted output, I need to finish up the tests around my R5RS character decoding implementation. Time to go in and add more cases to my unit tests!

I hope this post helps explain the benefits of testing for this project, and helps you convert your copy of Write Yourself a Scheme into something more test-driven. If it saves you time, or you have any other ways to help the rest of us save some time, leave a comment. WYAS is a great resource, and I’d love to collect all the time-savings we can and fold them into the original text.

Here’s a quick hack I keep meaning to release in a meaningful way: a morse code decoder in arduino.  Hook up a button to pin 2 (like you would for the button examples), and then watch the serial monitor.  If you key in SOS (…/—/…), it will light up the LED on pin 13.  My intention here was to use this to buzzy myself into my apartment building’s door, but I still haven’t gotten around to actually putting it inside the box yet(!).

Give it a go, and see what you think of it.  I’d also like to hook it up to an LCD panel and make a proper morse training device out of it, but that’s for another day, I think.

Leave comments here if you make any use of this, I’m keen on seeing what happens with it!

73 de KJ6ANM

Here’s the file: arduino_morse_decoder.pde(.txt to make things happier).

My iPhone is gradually becoming a universal remote control, and I love it.  This is the first part of that: how to hook a generic GUI-creation tool on the iPhone up to run arbitrary commands on my Linux box.

One of the peacetime dividends of Perceptron is an awareness of OSC.  OSC is Open Sound Control, a standard for sending, well, sound controls over the network.  If you’re familiar with MIDI, it’s basically MIDI over the network, but with twice the resolution.  Technically, they seem to have made lots of really good design decisions, and it’s been around long enough to have mature tools around it.

Basically, OSC allows you to have volume sliders, arbitrary buttons (play, pause, turn on lights, etc), knobs: just about anything you could use to control music can be put into OSC.  This is great, because it allows a designer to abstract out the interface from their application, and just plug in different physical widgets to implement “volume slider 1″ or “play drum B.”  Or, just as easily, hook those up to software, so they can have a GUI to interface to humans, or pure software control, with a software robot hitting virtual buttons.

While poking around one day, I thought to myself, “Wait, someone has to have done an OSC GUI for the iPhone!”  And, sure enough, they had: OSCRemote.  It’s great fun to play with: you can add little sliders, move buttons around, and even use the accelerometers.  It’s a delight, especially if you’re used to creating GUIs in code.

However, at the end of the day, I’m still a command-line Unix guy.  How do I bridge that gap?  With a control daemon, of course!  There’s a nice C library that implements OSC (liblo) using callbacks; add a trivial config file parser and an exec() implementation, and voila: I can now run set unix commands by hitting buttons on my iPhone.  It’s incredibly configurable and general, while staying pretty straightforward.  Now I can start and stop music, jump tracks, and nudge volume up and down from anywhere in my apartment, using just my phone.

You can get the code for this at oscremoted (creative name, eh?).  It needs to get a proper homepage on my domain; until then, it’s just a tarball with a freshmeat page.  I’m curious if anyone has ideas for refinements or improvements; comment if you do!