Before astronaut Scott Kelly ended his year in space, he accomplished an unprecedented technological feat. He called mission control using Skype and streamed his first-person perspective through an augmented reality headset that NASA sent to the International Space Station in December.
"We messed around with it for like two hours and immediately I sensed this is a capability we could use right now," Kelly said during a news conference in March.
The Microsoft HoloLens that Kelly used is just one tool being tinkered with at the Jet Propulsion Laboratory in La Cañada Flintridge that might change space exploration. JPL's Ops Lab, a team of roughly a dozen, is experimenting with virtual and augmented reality technologies to allow NASA to take direct control of its robotic explorers, to enable humans to see distant worlds with their own eyes and to teach astronauts how to perform complex tasks on the fly.
This isn't a technology relegated to the distant future; these innovations are happening today, said Jeff Norris, the founder and leader of the Ops Lab, during a presentation at the Vision Summit in Hollywood last month.
"Imagine a spacecraft designer studying full-scale holograms of a spacecraft years before a piece of metal is cut," he said. "They could discover and correct problems before they could endanger a launch or a mission."
Norris said he sees a future where augmented displays are integrated into spacesuit helmets. The technology will play an important role in NASA's mission to send humans to Mars by the 2030s, he said.
"We think we're going to use it to design the spacecraft that takes the astronauts to Mars; we think we're going to use these technologies to assist astronauts on board the spacecraft on the way and when they arrive, to increase their autonomy so they can perform tasks that they need to on Mars without having to be helped as much on the ground," Norris said. "That's how we think all these things are coming together to enable the next chapter in humanity's exploration of space."
In Kelly's case, his Skype test showed that scientists on Earth and the astronauts at the space station could connect live using the HoloLens, despite the station's limited connection to the Internet.
The HoloLens, Microsoft's foray into augmented reality, is the backbone of two projects at JPL called Sidekick and OnSight. Other experiments use Microsoft's Kinect and the Oculus Rift, according to NASA officials.
'SIDEKICK' MAKES SPACE TASKS EASIER
The HoloLens headset overlays projections on top of users' surroundings to "augment" their vision. NASA hopes to use the consumer technology through Project SideKick to create interactive guides for astronauts. The headset, through Skype, even lets experts on Earth walk astronauts through unfamiliar tasks. They can draw and share objects to the astronaut's field of vision, such as an arrow to direct the headset wearer's eyes or numbers to show the order to perform each step.
NASA wants to use standalone interactive manuals for the headset that use holographic animations, according to Victor Luo, a senior technical lead and human interfaces engineer at JPL. It would potentially replace bulky paper manuals used on the station today.
"As we walk through the procedures, the application is hand-holding us, showing us animations and diagrams, everything we need to know as we're doing it," Luo said at the Vision Summit.
Luo tested the display in an underwater space station analog in the Atlantic Ocean. Tasks expected to last an entire afternoon were completed in less than an hour, he said. The team certified the HoloLens for space in NASA's Weightless Wonder, a jet that climbs and dives to reduce gravity.
Initially they expected Sidekick to reach the space station this summer, but a rocket explosion delayed its delivery until December. Still, Kelly was able to perform some initial tests before his return to Earth on March 1.
It's not just astronauts who benefit from the technology, however.
TAKING MARS INTO THE OFFICE
A steady stream of images flow in every day from Mars. They come from rovers and orbiters mapping out the planet from the ground and above.
OnSight combines the data from NASA's robotic explorers to create a virtual map of Mars. Scientists can use OnSight to virtually meet up on the Red Planet and even plot out movements for NASA's rovers.
"We want to bring the surface of Mars into their offices," Norris said. "Let them explore the Red Planet as geologists have explored Earth."
Studying images from a computer screen lacks the depth of seeing it with your own eyes, Norris said. The Ops Lab tested early versions of OnSight by giving a headset to the science teams from the Curiosity and Opportunity rover missions, then asking them to complete a task using data sent back by Curiosity. They compared the results against a control group that used the standard operational tools for the rover.
"What we found was there is a dramatic, measurable and statistically significant effect on their understanding of the vicinity of the data acquired by the Mars Rover when they were wearing a head mounted display," Norris said.
Even with zero experience with the headsets, scientists performed as well as, or better, than their counterparts using the existing tools, according to a graph Norris showed during his presentation.
A pilot group is now using the headgear to operate Curiosity, according to JPL. OnSight recently helped Abigail Fraeman, a Curiosity science team member at JPL, and her team identify a point they would like to study between two Martian rock formations.
"OnSight makes the whole process of analyzing data feel a lot more natural to me," Fraeman said in a statement. "It really gives me the sense that I'm in the field when I put it on. Thinking about Martian geology is a lot more intuitive when I can stand in the scene and walk around the way I would if I were in the field."
SEE MARS FOR YOURSELF
NASA and JPL have unveiled "Destination: Mars," an exhibit opening at the Kennedy Space Center in Florida this summer, that will let guests visit Mars using an adaptation of OnSight.
The tour across several sites on Mars, reconstructed using real imagery from Curiosity, is guided by holographic versions of astronaut Buzz Aldrin and Curiosity Rover driver Erisa Hines.
"This experience lets the public explore Mars in an entirely new way. To walk through the exact landscape that Curiosity is roving across puts its achievements and discoveries into beautiful context," said Doug Ellison, visualization producer at JPL.
Separately, NASA is working with developers to create a free video game, "The Mars 2030 Experience," using the Unreal Engine 4 for consumer virtual reality headsets, according to an announcement on Epic Games' website.