On a Monday morning in downtown Baltimore, in a second-floor lab at the Johns Hopkins Hospital, a bottlenose dolphin named Bandit is having the time of his life.
He’s rolling playfully in the water. He’s munching fish with a satisfied crunch. He’s breaching the surface once he’s fortified, leaping and twirling in the air with acrobatic power and grace. And when he plunges back into the cerulean water, he chirps and whistles and clicks, then looks for more fish.
Could this be the smiling new face of conservation?
Bandit is a physical simulation. His remarkably lifelike movements, from the twitch of his tail to the depths of his dives, are happening on a monitor. They’re being controlled by a computer scientist named Omar Ahmad, first with a mouse, then with a robot arm, and finally, on a touch-screen tablet, with a finger swipe. Learning to “think” like Bandit and move him properly—like a real dolphin thinks and moves—is tricky, but Ahmad is a practiced hand.
Later this summer, in a series of clinical trials, post-stroke patients will get a chance to “be” Bandit themselves. If the immersive, dynamic experience helps them recover lost motor function faster than the repetitive exercises of conventional treatment, it could signal a paradigm shift in rehabilitative therapy.
Bandit’s back story is just as novel. The virtual dolphin is the result of an unlikely consortium—of scientists and animators, software engineers and animal-intelligence experts—finding common cause in the science of movement.
Perhaps most surprising of all, Bandit may soon point the way forward on another front far removed from medicine: wildlife conservation.
How It All Began
“We met and started talking, and it was love at first sight,” says John W. Krakauer.
The Johns Hopkins professor of neuroscience and neurology is describing the fateful spring day in 2011 when he met Ahmad and the other future members of the Kata Project (“kata” is a Japanese term for practiced movement).
“I’d been wondering for a long time: Where does the cognitive stop and the motor begin?” says Krakauer, who co-founded Kata with Ahmad and directs the Hopkins neurology department’s Brain, Learning, Animation, and Movement (BLAM) lab. In reality, “they’re inseparable: Movement is cognition. And by studying movement and what it does to people—why they love it, why they’re devastated when they lose it—we can learn so much more about the brain. So I realized we needed new tasks to tackle this fundamental insight.”
Krakauer decided that video games were the best way to study the various motor tasks “that had everything we all love about movement.” But the games he’d seen were lacking a crucial feature: They didn’t require the person playing to directly control a character onscreen and, using the player’s own cognitive intention, make it move in a realistic way.
So Krakauer began looking for forward-thinking gamer-scientists to work with. His search led him to Hopkins’s Homewood campus and Ahmad, who had recently collaborated with 3-D sound engineer Jeff Anderson on a simulated game. Though Ahmad calls it “terrible” today, the game featured a highly realistic bird modeled, uniquely, on biomechanics—a marked contrast to the artificial-looking characters in most other games.
In fact, the bird was so realistic that when it died during the course of the game, angry letters poured in from people who’d played it. That’s when Ahmad knew he was on the cusp of something big.
“I used to work with Disney,” he says, “and one of the things animators there are concerned with is how you make a character realistic. Why are Mickey Mouse or Donald Duck real to a child? The answer is what we call emotional reality, which is determined by ‘squash and stretch’—the elasticity of a character’s movement. And that’s what I was trying to apply to games.”
When video gaming exploded commercially in the late 1990s, he says, people thought that the latest technology would deepen the sense of emotional reality.
“But the reverse happened,” says Ahmad. “When you play games today, there’s still a sense that the characters aren’t alive—that the physics of them aren’t real. That’s because most two- and three-dimensional animation is basically just an expensive, stitched-together flipbook. And the mind picks up on that artificiality. The ‘illusion of life,’ as we call it, is destroyed.”
To try to preserve the illusion in his next game, Ahmad enlisted Promit Roy, a software engineer, and Kat McNally, an animator and illustrator with a rare knack for swiftly translating scientific concepts into visual forms.
When Krakauer met Ahmad and his team, he realized their prototypes were also addressing cognition and motor control, albeit from a different angle.
“When I saw what they were doing, it led to a eureka moment about what we could do together,” he says. “Especially if we shifted the focus from interacting with the animal to actually controlling it,” which would spark the highest mental action.
When they made that shift, says Ahmad, “the light bulb turned on. Promit built a very powerful physics engine that let us go light-years beyond that original bird.”
Building a Better Dolphin
Once the multidisciplinary team was in place, the members—augmented by robotics specialist Kevin Olds—started looking for the perfect animal to base their work on.
“I chose a dolphin,” says Ahmad, “because of the simple intention in its beautiful, complex movements. Dolphins do an incredible amount of real-time computation—calculating their lift, their drag, the friction and hydrodynamics of the water—that they’re not conscious of. And the acrobatics of their jump, which they trigger with a flick of their head, is just the coolest thing.
“Of course, all of that breaks down if you produce a virtual animal that’s not endearing too. So we found the perfect artist in Kat. She has an innate ability to put that quality into shapes.”
As Roy explains, “Most movies and games today are done using expensive motion-capture technology. But the characters they wind up with are always kind of creepy looking. So we don’t directly capture numerical data and try to match that to what we’re doing. We take tons of video and make sketches, then internalize—aesthetically and artistically—how the creature moves. We’re trying to re-create it with an artistic eye rather than a mathematical algorithm.”
To bring out what Krakauer calls “the idealized, essential ‘dolphin-ness’ of a dolphin’s movement,” Roy says that “you have to exaggerate and accentuate certain things about how it moves. If you replicate exactly what it does in the real world, it feels weird in a digital world.”
As the team began to spend hours analyzing dolphin movements, Ahmad realized something was still missing. To design a virtual animal “with true emotional reality,” they’d need to truly understand its emotions.
So he began reading a book called The Dolphin in the Mirror: Exploring Dolphin Minds and Saving Dolphin Lives. And he soon learned that its author, Hunter College psychology professor Diana Reiss (also a National Geographic grantee and explorer-in-residence), was an expert on animal cognition and communication—and, as luck would have it, a good friend of Krakauer’s. Reiss soon signed on to Kata as a consultant.
“People have been mesmerized by the movements of dolphins since the time of the ancient Greeks,” she says. “They’re so synchronous with each other. And watching them you feel in synchrony with them, because we live vicariously by watching skilled movement.”
Reiss in turn introduced the Kata team to Sue Hunter, director of animal programs at the National Aquarium in Baltimore.
“After we met Sue, this whole thing really took off in terms of our study of the animal,” says Ahmad. “Her insights were so important in guiding how we viewed the dolphins and their emotional nature. And watching her and the other trainers was really important to see [that interspecies] bonding.”
Hunter also granted Ahmad and McNally virtually unlimited access to a viewing area between the aquarium’s three dolphin tanks. That gave them a good look at the three dolphins that would serve as the composite model for Bandit.
Connection, Compassion … Conservation?
From Greek and Hindu myths to heraldic symbols, from Flipper to The Hitchhiker’s Guide to the Galaxy, dolphins are famously intelligent, charismatic, and photogenic. Yet 20,000-plus are legally slaughtered each year in Japanese waters. And many thousands more perish annually after becoming entangled in fishing nets.
As the debate over keeping dolphins and whales in captivity has heated up, the National Aquarium has phased out its dolphin shows and put an indefinite moratorium on breeding programs. Last month the aquarium’s CEO, John Racanelli, said the current colony of eight may eventually be retired to the first ever (still theoretical) seaside sanctuary. (See: “Q&A: National Aquarium CEO Discusses Dolphin’s Retirement.”)
On a June afternoon at the aquarium, in the Dolphin Discovery exhibit—where visitors can watch the animals, which thrive on continual stimulation, playing with “enrichment” toys like hula hoops, soccer balls, and footballs—two young male dolphins are cavorting in a 24-foot-deep pool.
Beau is eight years old, Foster is five. As they train and play—leaping ballistically out of the water, chasing each other around the pool, slapping the surface with their tails, gulping down fish tossed by trainers, cheekily splashing the spectators they see lingering closest to the glass—they look for all the world like Bandit in his virtual sea.
That’s because McNally based her sketches and animations on this pair, along with a five-year-old female named Bayley. And Anderson designed the game’s binaural audio (recording with two microphones to produce 3-D sound) using recordings of their above-water and underwater sounds.
That audio-visual verisimilitude is important for more than just a realistic gaming experience. It may be the key to inspiring empathy and compassion—and, by extension, conservation.
After “being” Bandit in the game, a player can’t help but gain a new appreciation of the species—including how hard it is to be a dolphin. Ahmad has experienced it himself.
“I spent hundreds of hours looking at these animals,” he said. “Then I’d go to the lab and, with the game, try to control every subtlety—try to re-create it. When I would come back to the aquarium, the actual animal took on a whole different reality.
“Now I understand what I have to be aware of for Bandit to kick his tail in a certain way. The jump in the game is hard to do well at first. But you soon realize you have to time it properly. And you have a motor connection to the animal, which you get to know so well through the course of the game. At the same time, you begin to understand that Bandit has a personality, just like real dolphins do.”
To that end, Krakauer sees the game as a sort of stealth campaign for conservation.
“If you fool people into falling in love with animals for implicit reasons,” he says, “then you start finding out what you can do to protect [the animals]. For too long we’ve put the cart before the horse. People don’t need facts and data first. They need the things they love. And that’s movement—everyone loves to see something move beautifully, like a dolphin does in the water.
“We think this is the way to get kids in Japan to vote on [conservation issues]. Do you think they’re going to countenance their grandparents slaughtering all these animals [after they play the game]? Or fishing the whales to extinction? If this game can instill in them a sympathy and an empathy, it will make it hard for them to be clueless.”
In other words, as Aristotle once said, “To perceive is to suffer.” Of course, the philosopher also said, “For the things we have to learn before we can do them, we learn by doing them.”
The Kata crew seems to have taken that to heart.
“This game is a way of linking people to dolphins,” says Reiss, “of learning about them and building empathy. It puts you in the context of their environment. That’s what’s critical here—you’re entering their world and feeling what it’s like to be a dolphin. And you don’t just feel like you’re controlling the dolphin. You become it.”
The hope is that virtual empathy will, in time, lead to widespread real-world protection.
“Dolphins are a keystone species,” says Reiss. “A conservation goal here at the aquarium is to protect these animals and their environment. But dolphins are still getting hammered in the wild, whether by nets or hunts. We need to have global protection for this species. It’s a big goal, but this is a great first step.”
Time will tell whether people who play the game agree. Kata’s first purpose in designing it was for use as a medical therapeutic tool—an application that Krakauer hopes will eventually fund itself. But later this summer Max & Haley, Ahmad and Co.’s commercial arm, will publicly release the game as an app for iPads and iPhones. The rollout is important, says Ahmad, because so much is riding on it.
“We want to make games that I’d give my own son to play,” he says. “The hyper-violence that’s in some of these games—we absolutely have to move away from that. But to do that we have to offer an experience that’s more compelling. We’re not against action. Nature is violent. But it has to be in a certain context.
“Aligning this game with conservation could have a very positive cultural impact, which is what we’re trying to do. This could be a lot bigger than just a video game being released. It could be a new way of understanding these majestic creatures. And that’s really what this game is all about—building a bridge between our world and theirs.”
Source: National Geographic