The August 2022 issue of IEEE Spectrum is here!
IEEE websites place cookies on your device to give you the best user experience. By using our websites, you agree to the placement of these cookies. To learn more, read our Privacy Policy.
Copenhagen Suborbitals is crowdfunding its crewed rocket
Copenhagen Suborbitals volunteers are building a crewed rocket on nights and weekends. The team includes [from left] Mads Stenfatt, Martin Hedegaard Petersen, Jørgen Skyt , Carsten Olsen, and Anna Olsen .
It was one of the prettiest sights I have ever seen: our homemade rocket floating down from the sky, slowed by a white-and-orange parachute that I had worked on during many nights at the dining room table. The 6.7-meter-tall Nexø II rocket was powered by a bipropellant engine designed and constructed by the Copenhagen Suborbitals team. The engine mixed ethanol and liquid oxygen together to produce a thrust of 5 kilonewtons, and the rocket soared to a height of 6,500 meters. Even more important, it came back down in one piece.
That successful mission in August 2018 was a huge step toward our goal of sending an amateur astronaut to the edge of space aboard one of our DIY rockets. We're now building the Spica rocket to fulfill that mission, and we hope to launch a crewed rocket about 10 years from now.
Copenhagen Suborbitals is the world's only crowdsourced crewed spaceflight program, funded to the tune of almost US $100,000 per year by hundreds of generous donors around the world. Our project is staffed by a motley crew of volunteers who have a wide variety of day jobs. We have plenty of engineers, as well as people like me, a pricing manager with a skydiving hobby. I'm also one of three candidates for the astronaut position.
We're in a new era of spaceflight: The national space agencies are no longer the only game in town, and space is becoming more accessible. Rockets built by commercial players like Blue Origin are now bringing private citizens into orbit. That said, Blue Origin, SpaceX, and Virgin Galactic are all backed by billionaires with enormous resources, and they have all expressed intentions to sell flights for hundreds of thousands to millions of dollars. Copenhagen Suborbitals has a very different vision. We believe that spaceflight should be available to anyone who's willing to put in the time and effort.
Copenhagen Suborbitals was founded in 2008 by a self-taught engineer and a space architect who had previously worked for NASA . From the beginning, the mission was clear: crewed spaceflight. B oth founders left the organization in 2014, but by then the project had about 50 volunteers and plenty of momentum.
The group took as its founding principle that the challenges involved in building a crewed spacecraft on the cheap are all engineering problems that can be solved, one at a time, by a diligent team of smart and dedicated people. When people ask me why we're doing this, I sometimes answer, "Because we can."
Volunteers use a tank of argon gas [left] to fill a tube within which engine elements are fused together. The team recently manufactured a fuel tank for the Spica rocket [right] in their workshop.
Our goal is to reach the Kármán line, which defines the boundary between Earth's atmosphere and outer space, 100 kilometers above sea level. The astronaut who reaches that altitude will have several minutes of silence and weightlessness after the engines cut off and will enjoy a breathtaking view. But it won't be an easy ride. During the descent, the capsule will experience external temperatures of 400 °C and g-forces of 3.5 as it hurtles through the air at speeds of up to 3,500 kilometers per hour.
I joined the group in 2011, after the organization had already moved from a maker space inside a decommissioned ferry to a hangar near the Copenhagen waterfront. Earlier that year, I had watched Copenhagen Suborbital's first launch, in which the HEAT-1X rocket took off from a mobile launch platform in the Baltic Sea—but unfortunately crash-landed in the ocean when most of its parachutes failed to deploy. I brought to the organization some basic knowledge of sports parachutes gained during my years of skydiving, which I hoped would translate into helpful skills.
The team's next milestone came in 2013, when we successfully launched the Sapphire rocket, our first rocket to include guidance and navigation systems. Its navigation computer used a 3-axis accelerometer and a 3-axis gyroscope to keep track of its location, and its thrust-control system kept the rocket on the correct trajectory by moving four servo-mounted copper jet vanes that were inserted into the exhaust assembly.
We believe that spaceflight should be available to anyone who's willing to put in the time and effort.
The HEAT-1X and the Sapphire rockets were fueled with a combination of solid polyurethane and liquid oxygen. We were keen to develop a bipropellant rocket engine that mixed liquid ethanol and liquid oxygen, because such liquid-propellant engines are both efficient and powerful. The HEAT-2X rocket, scheduled to launch in late 2014, was meant to demonstrate that technology. Unfortunately, its engine went up in flames, literally, in a static test firing some weeks before the scheduled launch. That test was supposed to be a controlled 90-second burn; instead, because of a welding error, much of the ethanol gushed into the combustion chamber in just a few seconds, resulting in a massive conflagration. I was standing a few hundred meters away, and even from that distance I felt the heat on my face.
The HEAT-2X rocket's engine was rendered inoperable, and the mission was canceled. While it was a major disappointment, we learned some valuable lessons. Until then, we'd been basing our designs on our existing capabilities—the tools in our workshop and the people on the project. The failure forced us to take a step back and consider what new technologies and skills we would need to master to reach our end goal. That rethinking led us to design the relatively small Nexø I and Nexø II rockets to demonstrate key technologies such as the parachute system, the bipropellant engine, and the pressure regulation assembly for the tanks.
For the Nexø II launch in August 2018, our launch site was 30 k m east of Bornholm, Denmark's easternmost island, in a part of the Baltic Sea used by the Danish navy for military exercises. W e left Bornholm's Nexø harbor at 1 a . m . to reach the designated patch of ocean in time for a 9 a.m. launch , the time approved by Swedish air traffic control. (While our boats were in international waters, Sweden has oversight of the airspace above that part of the Baltic Sea.) Many of our crew members had spent the entire previous day testing the rocket's various systems and got no sleep before the launch . We were running on coffee .
When the Nexø II blasted off, separating neatly from the launch tower, we all cheered. The rocket continued on its trajectory, jettisoning its nose cone when it reached its apogee of 6,500 meters, and sending telemetry data back to our mission control ship all the while. As it began to descend, it first deployed its ballute, a balloon-like parachute used to stabilize spacecraft at high altitudes, and then deployed its main parachute, which brought it gently down to the ocean waves.
In 2018, the Nexø II rocket launched successfully [left] and returned safely to the Baltic Sea [right].
The launch brought us one step closer to mastering the logistics of launching and landing at sea. For this launch, we were also testing our ability to predict the rocket's path. I created a model that estimated a splashdown 4.2 km east of the launch platform; it actually landed 4.0 km to the east. This controlled water landing—our first under a fully inflated parachute—was an important proof of concept for us, since a soft landing is an absolute imperative for any crewed mission.
This past April, the team tested its new fuel injectors in a static engine test. Carsten Olsen
The Nexø II's engine, which we called the BPM5, was one of the few components we hadn't machined entirely in our workshop; a Danish company made the most complicated engine parts. But when those parts arrived in our workshop shortly before the launch date, we realized that the exhaust nozzle was a little bit misshapen. We didn't have time to order a new part, so one of our volunteers, Jacob Larsen, used a sledgehammer to pound it into shape. The engine didn't look pretty—we nicknamed it the Franken-Engine—but it worked. Since the Nexø II's flight, we've test-fired that engine more than 30 times, sometimes pushing it beyond its design limits, but we haven't killed it yet.
The Spica astronaut's 15-minute ride to the stars will be the product of more than two decades of work.
That mission also demonstrated our new dynamic pressure regulation (DPR) system, which helped us control the flow of fuel into the combustion chamber. The Nexø I had used a simpler system called pressure blowdown, in which the fuel tanks were one-third filled with pressurized gas to drive the liquid fuel into the chamber. With DPR, the tanks are filled to capacity with fuel and linked by a set of control valves to a separate tank of helium gas under high pressure. That setup lets us regulate the amount of helium gas flowing into the tanks to push fuel into the combustion chamber, enabling us to program in different amounts of thrust at different points during the rocket's flight.
The 2018 Nexø II mission proved that our design and technology were fundamentally sound. It was time to start working on the human-rated Spica rocket.
Copenhagen Suborbitals hopes to send an astronaut aloft in its Spica rocket in about a decade. Caspar Stanley
With its crew capsule, the Spica rocketwill measure 13 meters high and will have a gross liftoff weight of 4, 000 kilograms, of which 2 , 600 k g will be fuel . It will be, by a significant margin, the largest rocket ever built by amateurs.
The Spica rocket will use the BPM100 engine, which the team is currently manufacturing. Thomas Pedersen
Its engine, the 100-kN BPM100, uses technologies we mastered for the BPM5, with a few improvements. Like the prior design, it uses regenerative cooling in which some of the propellant passes through channels around the combustion chamber to limit the engine's temperature. To push fuel into the chamber, it uses a combination of the simple pressure blowdown method in the first phase of flight and the DPR system, which gives us finer control over the rocket's thrust. The engine parts will be stainless steel, and we hope to make most of them ourselves out of rolled sheet metal. The trickiest part, the double-curved "throat" section that connects the combustion chamber to the exhaust nozzle, requires computer-controlled machining equipment that we don't have. Luckily, we have good industry contacts who can help out.
One major change was the switch from the Nexø II's showerhead-style fuel injector to a coaxial-swirl fuel injector. The showerhead injector had about 200 very small fuel channels. It was tough to manufacture, because if something went wrong when we were making one of those channels—say, the drill got stuck—we had to throw the whole thing away. In a coaxial-swirl injector, the liquid fuels come into the chamber as two rotating liquid sheets, and as the sheets collide, they're atomized to create a propellant that combusts. Our swirl injector uses about 150 swirler elements, which are assembled into one structure. This modular design should be easier to manufacture and test for quality assurance.
The BPM100 engine will replace an old showerhead-style fuel injector [right] with a coaxial-swirl injector [left], which will be easier to manufacture.Thomas Pedersen
In April of this year, we ran static tests of several types of injectors. We first did a trial with a well-understood showerhead injector to establish a baseline, then tested brass swirl injectors made by traditional machine milling as well as steel swirl injectors made by 3D printing. We were satisfied overall with the performance of both swirl injectors, and we're still analyzing the data to determine which functioned better. However, we did see some combustion instability—namely, some oscillation in the flames between the injector and the engine's throat, a potentially dangerous phenomenon. We have a good idea of the cause of these oscillations, and we're confident that a few design tweaks can solve the problem.
Volunteer Jacob Larsen holds a brass fuel injector that performed well in a 2021 engine test.Carsten Olsen
We'll soon commence building a full-scale BPM100 engine, which will ultimately incorporate a new guidance system for the rocket. Our prior rockets, within their engines' exhaust nozzles, had metal vanes that we would move to change the angle of thrust. But those vanes generated drag within the exhaust stream and reduced effective thrust by about 10 percent. The new design has gimbals that swivel the entire engine back and forth to control the thrust vector. As further support for our belief that tough engineering problems can be solved by smart and dedicated people, our gimbal system was designed and tested by a 21-year-old undergraduate student from the Netherlands named Jop Nijenhuis, who used the gimbal design as his thesis project (for which he got the highest possible grade).
We're using the same guidance, navigation, and control (GNC) computers that we used in the Nexø rockets. One new challenge is the crew capsule; once the capsule separates from the rocket, we'll have to control each part on its own to bring them both back down to Earth in the desired orientation. When separation occurs, the GNC computers for the two components will need to understand that the parameters for optimal flight have changed. But from a software point of view, that's a minor problem compared to those we've solved already.
Bianca Diana works on a drone she's using to test a new guidance system for the Spica rocket.Carsten Olsen
My specialty is parachute design. I've worked on the ballute, which will inflate at an altitude of 70 km to slow the crewed capsule during its high-speed initial descent, and the main parachutes, which will inflate when the capsule is 4 km above the ocean. We've tested both types by having skydivers jump out of planes with the parachutes, most recently in a 2019 test of the ballute. The pandemic forced us to pause our parachute testing, but we should resume soon.
For the parachute that will deploy from the Spica's booster rocket, the team tested a small prototype of a ribbon parachute.Mads Stenfatt
For the drogue parachute that will deploy from the booster rocket, my first prototype was based on a design called Supersonic X, which is a parachute that looks somewhat like a flying onion and is very easy to make. However, I reluctantly switched to ribbon parachutes, which have been more thoroughly tested in high-stress situations and found to be more stable and robust. I say "reluctantly" because I knew how much work it would be to assemble such a device. I first made a 1.24-meter-diameter parachute that had 27 ribbons going across 12 panels, each attached in three places. So on that small prototype, I had to sew 972 connections. A full-scale version will have 7,920 connection points. I'm trying to keep an open mind about this challenge, but I also wouldn't object if further testing shows the Supersonic X design to be sufficient for our purposes.
We've tested twocrew capsules in past missions: the Tycho Brahe in 2011 and the Tycho Deep Space in 2012. The next-generation Spica crew capsulewon't be spacious, but it will be big enough to hold a single astronaut, who will remain seated for the 15 minute s of flight (and for two hours of preflight checks). The first spacecraft we're building is a heavy steel "boilerplate" capsule, a basic prototype that we're using to arrive at a practical layout and design. We'll also use this model to test hatch design, overall resistance to pressure and vacuum, and the aerodynamics and hydrodynamics of the shape, as we want the capsule to splash down into the sea with minimal shock to the astronaut inside. Once we're happy with the boilerplate design, we'll make the lightweight flight version.
Copenhagen Suborbitals currently has three astronaut candidates for its first flight: from left, Mads Stenfatt, Anna Olsen, and Carsten Olsen. Mads Stenfatt
Three members of the Copenhagen Suborbitals team are currently candidates to be the astronaut in our first crewed mission—me, Carsten Olsen, and his daughter, Anna Olsen. We all understand and accept the risks involved in flying into space on a homemade rocket. In our day-to-day operations, we astronaut candidates don't receive any special treatment or training. Our one extra responsibility thus far has been sitting in the crew capsule's seat to check its dimensions. Since our first crewed flight is still a decade away, the candidate list may well change. As for me, I think there's considerable glory in just being part of the mission and helping to build the rocket that will bring the first amateur astronaut into space. Whether or not I end up being that astronaut, I'll forever be proud of our achievements.
The astronaut will go to space inside a small crew capsule on the Spica rocket. The astronaut will remain seated for the 15-minute flight (and for the 2-hour flight check before). Carsten Brandt
People may wonder how we get by on a shoestring budget of about $100,000 a year—particularly when they learn that half of our income goes to paying rent on our workshop. We keep costs down by buying standard off-the-shelf parts as much as possible, and when we need custom designs, we're lucky to work with companies that give us generous discounts to support our project. We launch from international waters, so we don't have to pay a launch facility. When we travel to Bornholm for our launches, each volunteer pays his or her own way, and we stay in a sports club near the harbor, sleeping on mats on the floor and showering in the changing rooms. I sometimes joke that our budget is about one-tenth what NASA spends on coffee. Yet it may well be enough to do the job.
We had intended to launch Spica for the first time in the summer of 2021, but our schedule was delayed by the COVID-19 pandemic, which closed our workshop for many months. Now we're hoping for a test launch in the summer of 2022, when conditions on the Baltic Sea will be relatively tame. For this preliminary test of Spica, we'll fill the fuel tanks only partway and will aim to send the rocket to a height of around 30 to 50 km.
If that flight is a success, in the next test, Spica will carry more fuel and soar higher. If the 2022 flight fails, we'll figure out what went wrong, fix the problems, and try again. It's remarkable to think that the Spica astronaut's eventual 15-minute ride to the stars will be the product of more than two decades of work. But we know our supporters are counting down until the historic day when an amateur astronaut will climb aboard a homemade rocket and wave goodbye to Earth, ready to take a giant leap for DIY-kind.
One reason that Copenhagen Suborbitals has advanced quite slowly toward its ultimate goal of crewed spaceflight is our focus on safety. We test our components extensively; for example, we tested the engine that powered the 2016 Nexø I rocket about 30 times before the launch.
When we plan and execute launches, our bible is a safety manual from the Wallops Flight Facility, part of NASA's Goddard Space Flight Center. Before each launch, we run simulations of the flight profile to ensure there's no risk of harm to our crew, our boats, and any other people or property. We launch from the sea to further reduce the chance that our rockets will damage anyone or anything.
We recognize that our human-rated spacecraft, the Spica rocket and crew capsule, must meet a higher bar for safety than anything we've built before. But we must be honest about our situation: If we set the bar too high, we'll never finish the project. We can't afford to test our systems to the extent that commercial companies do (that's why we'll never sell rides on our rockets). Each astronaut candidate understands these risks. Speaking as one of those candidates, I'd feel confident enough to climb aboard if each of my friends who worked on the rocket can look me in the eyes and say, "Yes, we're ready."
This article appears in the December 2021 print issue as "The First Crowdfunded Astronaut."
Mads Stenfatt first contacted Copenhagen Suborbitals with some constructive criticism. In 2011, while looking at photos of the DIY rocketeers' latest rocket launch, he had noticed a camera mounted close to the parachute apparatus. Stenfatt sent an email detailing his concern—namely, that a parachute's lines could easily get tangled around the camera. "The answer I got was essentially, 'If you can do better, come join us and do it yourself,' " he remembers. That's how he became a volunteer with the world's only crowdfunded crewed spaceflight program.
As an amateur skydiver, Stenfatt knew the basic mechanics of parachute packing and deployment. He started helping Copenhagen Suborbitals design and pack parachutes, and a few years later he took over the job of sewing the chutes as well. He had never used a sewing machine before, but he learned quickly over nights and weekends at his dining room table.
One of his favorite projects was the design of a high-altitude parachute for the Nexø II rocket, launched in 2018. While working on a prototype and puzzling over the design of the air intakes, he found himself on a Danish sewing website looking at brassiere components. He decided to use bra underwires to stiffen the air intakes and keep them open, which worked quite well. Though he eventually went in a different design direction, the episode is a classic example of the Copenhagen Suborbitals ethos: Gather inspiration and resources from wherever you find them to get the job done.
Today, Stenfatt serves as lead parachute designer, frequent spokesperson, and astronaut candidate. He also continues to skydive in his spare time, with hundreds of jumps to his name. Having ample experience zooming down through the sky, he's intently curious about what it would feel like to go the other direction.
Mads Stenfatt is leading the development of the parachute recovery systems for Copenhagen Suborbitals' crowdfunded space capsule and booster rocket. Mads designs and sews the parachutes from the comfort of his living room, and tests them with the help of his connections in the skydiving community."
The company, says the company—but other interpretations persist
Mark Harris is an investigative science and technology reporter based in Seattle, with a particular interest in robotics, transportation, green technologies, and medical devices. He’s on Twitter at @meharris and email at mark(at)meharris(dot)com. Email or DM for Signal number for sensitive/encrypted messaging.
A Tesla user charges his Model S in Burbank, Calif.
On 29 September 2020, a masked man entered a branch of the Wells Fargo bank in Washington, D.C., and handed the teller a note: “This is a robbery. Act calm give me all hundreds.” The teller complied. The man then fled the bank and jumped into a gray Tesla Model S. This was one of three bank robberies the man attempted the same day.
When FBI agents began investigating, they reviewed Washington, D.C.’s District Department of Transportation camera footage, and spotted a Tesla matching the getaway vehicle’s description. The license plate on that car showed that it was registered to Exelorate Enterprises LLC, the parent company of Steer EV—a D.C.-based monthly vehicle-subscription service.
Agents served a subpoena on Steer EV for the renter’s billing and contact details. Steer EV provided those—and also voluntarily supplied historical GPS data for the vehicle. The data showed the car driving between, and parking at, each bank at the time of the heists. The renter was arrested and, in September, sentenced to four years in prison.
“If an entity is collecting, retaining, [and] sharing historical location data on an individualized level, it’s extraordinarily difficult to de-identify that, verging on impossible.” —John Verdi, Future of Privacy Forum
In this case, the GPS data likely came from a device Steer EV itself installed in the vehicle (neither Steer nor Tesla responded to interview requests). However, according to researchers, Tesla is potentially in a position to provide similar GPS tracks for many of its 3 million customers.
For Teslas built since mid-2017, “every time you drive, it records the whole track of where you drive, the GPS coordinates and certain other metrics for every mile driven,” says Green, a Tesla owner who has reverse engineered the company’s Autopilot data collection. “They say that they are anonymizing the trigger results,” but, he says, “you could probably match everything to a single person if you wanted to.”
Each of these trip logs, and other data “snapshots” captured by the Autopilot system that include images and video, is stripped of its identifying VIN and given a temporary, random ID number when it is uploaded to Tesla, says Green. However, he notes, that temporary ID can persist for days or weeks, connecting all the uploads made during that time.
Elon Musk, CEO of Tesla MotorsMark Mahaney/Redux
Given that some trip logs will also likely record journeys between a driver’s home, school, or place of work, guaranteeing complete anonymity is unrealistic, says John Verdi, senior vice president of policy at the Future of Privacy Forum: “If an entity is collecting, retaining, [and] sharing historical location data on an individualized level, it’s extraordinarily difficult to de-identify that, verging on impossible.”
Tesla, like all other automakers, has a policy that spells out what it can and cannot do with the data it gets from customers’ vehicles, including location information. This states that while the company does not sell customer and vehicle data, it can share that data with service providers, business partners, affiliates, some authorized third parties, and government entities according to the law.
Owners can buy a special kit for US $1,400 that allows them to access data on their own car's event data recorder, but this represents just a tiny subset of the data the company collects, and is related only to crashes. Owners living in California and Europe benefit from legislation that means Tesla will provide access to more data generated by their vehicles, although not the Autopilot snapshots and trip logs that are supposedly anonymized.
Once governments realize that a company possesses such a trove of information, it could be only a matter of time before they seek access to it. “If the data exists…and in particular exists in the domain of somebody who’s not the subject of those data, it’s much more likely that a government will eventually get access to them in some way,” says Bryant Walker Smith, an associate professor in the schools of law and engineering at the University of South Carolina.
“Individuals ought to think about their cars more like they think about their cellphones.” —John Verdi, Future of Privacy Forum
This is not necessarily a terrible thing, Walker says, who suggests that such rich data could unlock valuable insights into which roads or intersections are dangerous. The wealth of data could also surface subtle problems in the vehicles themselves.
In many ways, the data genie is already out of the bottle, according to Verdi. “Individuals ought to think about their cars more like they think about their cellphones,” he says. “The auto industry has a lot to learn from the ways that mobile-phone operating systems handle data permissions…. Both iOS and Android have made great strides in recent years in empowering consumers when it comes to data collection, data disclosure, and data use.”
Tesla permits owners to control some data sharing, including Autopilot and road segment analytics. If they want to opt out of data collection completely, they can ask Tesla to disable the vehicle’s connectivity altogether. However, this would mean losing features such as remote services, Internet radio, voice commands, and Web browser functionality, and even safety-related over-the-air updates.
Green says he is not aware of anyone who has successfully undergone this nuclear option. The only real way to know you’ve prevented data sharing, he says, is to “go to a repair place and ask them to remove the modem out of the car.”
Tesla almost certainly has the biggest empire of customer and vehicle data among automakers. It also appears to be the most aggressive in using that data to develop its automated driving systems, and to protect its reputation in the courts of law and public opinion, even to the detriment of some of its customers.
But while the world’s most valuable automaker dominates the discussion around connected cars, others are not far behind. Elon Musk’s insight—to embrace the data-driven world that our other digital devices already inhabit—is rapidly becoming the industry standard. When our cars become as powerful and convenient as our phones, it is hardly surprising that they suffer the same challenges around surveillance, privacy, and accountability.
But can a fire-hose approach solve self-driving’s biggest problems?
Mark Harris is an investigative science and technology reporter based in Seattle, with a particular interest in robotics, transportation, green technologies, and medical devices. He’s on Twitter at @meharris and email at mark(at)meharris(dot)com. Email or DM for Signal number for sensitive/encrypted messaging.
In 2019, Elon Musk stood up at a Tesla day devoted to automated driving and said, “Essentially everyone’s training the network all the time, is what it amounts to. Whether Autopilot’s on or off, the network is being trained.”
Tesla’s suite of assistive and semi-autonomous technologies, collectively known as Autopilot, is among the most widely deployed—and undeniably the most controversial—driver-assistance systems on the road today. While many drivers love it, using it for a combined total of more than 5 billion kilometers, the technology has been involved in hundreds of crashes, some of them fatal, and is currently the subject of a comprehensive investigation by the National Highway Traffic Safety Administration.
This second story—in IEEE Spectrum’s series of three on Tesla’s empire of data (story 1; story 3)—focuses on how Autopilot rests on a foundation of data harvested from the company’s own customers. Although the company’s approach has unparalleled scope and includes impressive technological innovations, it also faces particular challenges—not least of which is Musk’s decision to widely deploy the misleadingly named Full Self-Driving feature as a largely untested beta.
“Right now, automated vehicles are one to two magnitudes below human drivers in terms of safety performance.” —Henry Liu, Mcity
Most companies working on automated driving rely on a small fleet of highly instrumented test vehicles, festooned with high-resolution cameras, radars, and laser-ranging lidar devices. Some of these have been estimated to generate 750 megabytes of sensor data every second, providing a rich seam of training data for neural networks and other machine-learning systems to improve their driving skills.
Such systems have now effectively solved the task of everyday driving, including for a multitude of road users, different weather conditions, and road types, says Henry Liu, director of Mcity, a public-private mobility research partnership at the University of Michigan.
“But right now, automated vehicles are one to two magnitudes below human drivers in terms of safety performance,” says Liu. “And that’s because current automated vehicles can’t handle the curse of rarity: low-frequency, long-tail, safety-critical events that they just don’t see enough to know how to handle.” Think of a deer suddenly springing into the road, or a slick of spilled fuel.
Tesla’s bold bet is that its own customers can provide the long tail of data needed to boost self-driving cars to superhuman levels of safety. Above and beyond their contractual obligations, many are happy to do so—seeing themselves as willing participants in the development of technology that they have been told will one day soon allow them to simply sit back and enjoy being driven by the car itself.
For a start, the routing information for every trip undertaken in a recent model Autopilot-equipped Tesla is shared with the company—see the the previous installment in this series. But Tesla’s data effort goes far beyond navigation.
In autonomypresentations over the past few years, Musk and Tesla’s then-head of AI, Andrej Karpathy, detailed the company’s approach, including its so-called Shadow Mode.
In Shadow Mode, operating on Tesla vehicles since 2016, if the car’s Autopilot computer is not controlling the car, it is simulating the driving process in parallel with the human driver. When its own predictions do not match the driver’s behavior, this might trigger the recording of a short “snapshot” of the car’s cameras, speed, acceleration, and other parameters for later uploading to Tesla. Snapshots are also triggered when a Tesla crashes.
After the snapshots are uploaded, a team may review them to identify human actions that the system should try to imitate, and input them as training data for its neural networks. Or they may notice that the system is failing, for instance, to properly identify road signs obscured by trees.
In that case, engineers can train a detector designed specifically for this scenario and download it to some or all Tesla vehicles. “We can beam it down to the fleet, and we can ask the fleet to please apply this detector on top of everything else you’re doing,” said Karpathy in 2020. If that detector thinks it spots such a road sign, it will capture images from the car’s cameras for later uploading,
His team would quickly receive thousands of images, which they would use to iterate the detector, and eventually roll it out to all production vehicles. “I’m not exactly sure how you build out a data set like this without the fleet,” said Karpathy. An amateur Tesla hacker who tweets using the pseudonym Green told Spectrum that he identified over 900 Autopilot test campaigns, before the company stopped numbering them in 2019.
For all the promise of Tesla’s fleet learning, Autopilot has yet to prove that it can drive as safely as a human, let alone be trusted to operate a vehicle without supervision.
Liu is bullish on Tesla’s approach to leveraging its ever-growing consumer base. “I don’t think a small…fleet will ever be able to handle these [rare] situations,” he says. “But even with these shadow drivers—and if you deploy millions of these fleet vehicles, that’s a very, very large data collection—I don’t know whether Tesla is fully utilizing them because there’s no public information really available.”
One obstacle is the sheer cost. Karpathy admitted that having a large team to assess and label images and video was expensive and said that Tesla was working on detectors that can train themselves on video clips captured in Autopilot snapshots. In June, the company duly laid off 195 people working on data annotation at a Bay Area office.
While the Autopilot does seem to have improved over the years, with Tesla allowing its operation on more roads and in more situations, serious and fatal accidents are still occurring. These may or may not have purely technical causes. Certainly, some drivers seem to be overestimating the system’s capabilities or are either accidentally or deliberately failing to supervise it sufficiently.
Other experts are worried that Tesla’s approach has more fundamental flaws. “The vast majority of the world generally believes that you’re never going to get the same level of safety with a camera-only system that you will based on a system that includes lidar,” says Dr. Matthew Weed, senior director of product management at Luminar, a company that manufacturers advanced lidar systems.
He points out that Tesla’s Shadow Mode only captures a small fraction of each car’s driving time. “When it comes to safety, the whole thing is about…your unknown unknowns,” he says. “What are the things that I don’t even know about that will cause my system to fail? Those are really difficult to ascertain in a bulk fleet” that is down-selecting data.
For all the promise of Tesla’s fleet learning and the enthusiastic support of many of its customers, Autopilot has yet to prove that it can drive as safely as a human, let alone be trusted to operate a vehicle without supervision. And there are other difficulties looming. Andrej Karpathy left Tesla in mid-July, while the company continues to face the damaging possibility of NHTSA issuing a recall for Autopilot in the United States. This would be a terrible PR (and possibly economic) blow for the company but would likely not halt its harvesting of customer data to improve the system, nor prevent its continued deployment overseas.
Tesla’s use of fleet vehicle data to develop Autopilot echoes the user-fueled rise of Internet giants like Google, YouTube, and Facebook. The more its customers drive, so Musk’s story goes, the better the system performs.
But just as tech companies have had to come to terms with their complicated relationships with data, so Tesla is beginning to see a backlash. Why does the company charge US $12,000 for a so-called “full self-driving” capability that is utterly reliant on its customers’ data? How much control do drivers have over data extracted from their daily journeys? And what happens when other entities, from companies to the government, seek access to it? These are the themes for our third story.
High-performance computing (HPC) has historically been available primarily to governments, research institutions, and a few very large corporations for modeling, simulation, and forecasting applications. As HPC platforms are being deployed in the cloud for shared services, high-performance computing is becoming much more accessible, and its use is benefiting organizations of all sizes. Increasing investment in the industrial internet of things (IIoT), artificial intelligence (AI), and electronic design automation (EDA) and silicon IP for engineering development are a few factors that are driving increased use of high-performance computing systems. In general, increasingly complex models for big data processing, simulation, and forecasting are driving a need for more compute power and greater storage capacity & performance.
This white paper highlights how different storage technologies can maximize the efficiency and effectiveness of HPC systems while providing high capacity and low latency storage, and minimizing network bandwidth and power consumption.