Seeing data with your tongue.

April 19, 2007

The last issue of Wired magazine had this fascinating article about how researchers are taking advantage of the plasticity of the brain to use technology to either enhance human senses or to even create totally new ones. For example, they have the story of a person who used a “feelSpace belt” to acquire a sixth sense of direction, like the one that birds have to always know how to fly north or south.

For six weird weeks in the fall of 2004, Udo Wächter had an unerring sense of direction. Every morning after he got out of the shower, Wächter, a sysadmin at the University of Osnabrück in Germany, put on a wide beige belt lined with 13 vibrating pads — the same weight-and-gear modules that make a cell phone judder. On the outside of the belt were a power supply and a sensor that detected Earth’s magnetic field. Whichever buzzer was pointing north would go off. Constantly.

“It was slightly strange at first,” Wächter says, “though on the bike, it was great.” He started to become more aware of the peregrinations he had to make while trying to reach a destination. “I finally understood just how much roads actually wind,” he says. He learned to deal with the stares he got in the library, his belt humming like a distant chain saw. Deep into the experiment, Wächter says, “I suddenly realized that my perception had shifted. I had some kind of internal map of the city in my head. I could always find my way home. Eventually, I felt I couldn’t get lost, even in a completely new place.”

Another fascinating example describes how some researchers are using a device that basically hooks up a camera to a set of tiny electrodes that are attached to the tongue. Visual images are interpreted as electric current against the tongue. When people put on this device and have their vision totally blocked, they can quickly learn to “see” with their tongue. It’s pretty amazing stuff.

This is all based on the fact that our brain has the ability to shift resources around. If the visual cortex is damaged or doesn’t receive information for some reason, the brain can refocus a different part of it to start handling processing of visual information. This is why rehabilitation after a stroke or other major brain damage can restore whatever senses were compromised.

This is very promising when it comes to aiding those who are impaired in some way or even creating “enhanced” humans who can, for example, find their way around no matter where they are. But it also hold promise in a different area: enabling humans to use their senses to understand complex data sets.

Humans have a very hard time interpreting multidimensional data. Give someone a two dimensional table (rows by columns) and they can easily understand the data and make decisions on them. But add a third dimension or a fourth or fifth and it becomes very hard to represent the data in a way that humans can work with. That’s because we depend only on vision to see the data and then on our brain’s capacity to crunch multiple numbers simultaneously, which is a notoriously small capacity.

So, how about bringing in more senses? How about allowing multidimensional data to be perceived not only by sight but also by touch, taste, or even new senses, such as that of direction? Basically I’m talking about creating virtual reality environments where the environment is not a realistic replica of the real world but a representation of a multidimensional data space.

This is science fiction stuff, I know. But if a blind person can go mountain biking simply by clicking his or her tongue and using a sonar-like sense called echolocation (true story), why can’t I see, hear, touch, and taste my way through a complex data space to find solutions or make optimal decisions?