DARPA sets speed thresholds for walking over rough ground
The Learning Locomotion programme put six teams on the same LittleDog quadruped and measured them by number: each phase required clearing higher obstacles at higher speed, against thresholds DARPA fixed in advance.
Why it matters
This is a rare case for the era where walking capability had to be measured rather than shown, and against thresholds set by the funder rather than the authors. Identical hardware across all six teams made their results comparable — the same thing shared datasets and standard rigs would do fifteen years later.
All teams were given the same LittleDog robot, built for DARPA by Boston Dynamics, with a motion-capture system, so that they could concentrate on locomotion rather than hardware. Boston Dynamics, in the abstract of its own paper on the robot (International Journal of Robotics Research, 2011), calls it a common platform across the teams that allowed direct comparison of results, and speaks of a fleet of 12 robots; the paper itself was not read. The programme began in November 2005 and had one 15-month phase and two of 12 months. The thresholds were fixed rather than relative, so in principle every team or none could advance. By the 2007 article of the programme's DARPA manager and colleagues, the first phase required crossing obstacles 4.8 centimetres high at an average of 1.2 centimetres per second; the second and third, 4.2 and 7.2 centimetres per second over 7.8 and 10.8 centimetres. The Carnegie Mellon team confirms the final pair: 10.8 centimetres at more than 7.2 centimetres per second. The thesis of an MIT team member, of September 2008, prints the same table and the way of scoring: the best two of three blind trials on each of seven classes of terrain, with an unfinished run counted as zero speed. Scientific American in 2008 gave slightly different figures and named the teams: Carnegie Mellon, the Florida Institute for Human and Machine Cognition, MIT, Stanford, the University of Southern California and the University of Pennsylvania; the engineering school of the University of Southern California names the same six. For the third phase, from the second half of 2008, DARPA planned to keep three of the six teams. That school reported on 23 October 2009 that after three years of research only four teams could cope with DARPA's final tests, and that its own was the only one to exceed the target speed on five of six test boards, averaging 8.7 centimetres per second; this is a team's account of its own result. The Stanford team's author manuscript of 21 June 2010 says that six teams took part in the first two phases and five in the last, so, by that team's count, five teams rather than the three planned went into the last phase. The University of Southern California (USC) team's 2010 conference paper, on its final submission to the programme, says that five other teams also took part, six with itself, but not how many were in the last phase. The sources therefore do not say whether those four teams were out of the five of the last phase or out of six. The same manuscript repeats the threshold pairs; says that in the second and third phases the final measurements ran not on the boards the teams had had access to beforehand but on others, informally of the same classes; and that at the time of writing detailed results for all teams had not been officially released, so its authors compare themselves with the others only briefly: by their account Stanford's average speed was the highest of the six teams in the second phase and the second highest in the third, though other teams crossed more terrain types above the threshold speed. The Average rows of the team's own tables print 6.3 centimetres per second for the second phase (threshold 4.2) and 9.7 for the third (threshold 7.2); these too are the team's own account. An editorial in the programme's special issue of the same journal (February 2011), written by three guest editors from USC, the Florida Institute for Human and Machine Cognition and MIT, institutions of three of the teams, rather than by DARPA, says that the programme supported six teams in the United States for a total of 36 months, that the control software was compared on identical hardware and in independent off-site testing, and that, in the editors' summing-up, forward speed rose from around 1 to approximately 20 centimetres per second and obstacle height from about 2 to 10.8 centimetres; the passage read does not say whose speed that is, and it is not a threshold (the third-phase threshold is 7.2). The editorial does not say how many teams met the thresholds; its 36 months differ from the 39 that the three phases of the 2007 article add up to, and no source read gives the month in which the last phase ended: the 2024 survey gives only the year 2009 for the end of the programme. A 2024 survey, citing a 2011 report by Pippine and co-authors, likewise says that at the end of the programme in 2009 four of six teams finished with satisfactory performance. What the record does not claim: that it is known how many teams met the final thresholds. Here "four of six" rests on the USC school's news item and the 2024 survey; the programme's own overview, which the survey cites (Pippine, Hackett and Watson, in the same journal, 30(2), pp. 141-144), was not read: it is behind a paywall, its public abstract gives only the start of the programme in November 2005, and the editorial of the same issue says it includes an overview of the test results.