Viewing entries in
Assistive Technologies

Giving Prosthetic Limbs The Sense Of Touch

When I was a child I dreamt about one day becoming a biomedical engineer. And, somewhere in my mother's attic lies the remnants of that dream in the form of school papers and crayon drawings of the limbs, heads, and torsos that I one day hoped to design, engineer, and implant on human beings and animals. Aside from making me the creepiest kid in the neighborhood this also provided me with a special way of thinking about the interconnections and relationships between machines and people. That dream ultimately evaporated in college when I chose to start monetizing my software development hobby instead of completing my inorganic chemistry studies.

Today however actual biomedical engineers have come one step close to the giving a sense of touch to prosthetics for humans. Existing robotic prostheses have limited motor control, provide no sensory feedback and can be uncomfortable to wear. In an effort to make a prosthesis that moves like a normal hand, researchers at the University of Michigan (U-M) have bioengineered a scaffold that is placed over severed nerve endings like a sleeve and could improve the function of prosthetic hands and possibly restore the sense of touch for injured patients.


To overcome the limitations of existing prostheses, the U-M researchers realized a better nerve interface was needed to control the upper extremity prostheses. So they created what they called an “artificial neuromuscular junction” composed of muscle cells and a nano-sized polymer placed on a biological scaffold. Neuromuscular junctions are the body's own nerve-muscle connections that enable the brain to control muscle movement.

When a hand is amputated, the nerve endings in the arm continue to sprout branches, growing a mass of nerve fibers that send flawed signals back to the brain. The bioengineered scaffold was placed over the severed nerve endings like a sleeve. The muscle cells on the scaffold and in the body bonded and the body's native nerve sprouts fed electrical impulses into the tissue, creating a stable nerve-muscle connection.

In laboratory rats, the bioengineered interface relayed both motor and sensory electrical impulses and created a target for the nerve endings to grow properly. This indicates that the interface may not only improve fine motor control of prostheses, but can also relay sensory perceptions such as touch and temperature back to the brain. Laboratory rats with the interface responded to tickling of feet with appropriate motor signals to move the limb.


The research project, which was funded by the Department of Defense, arose from a need for better prosthetic devices for troops wounded in Afghanistan and Iraq. The DoD and the Army have already provided $4.5 million in grants to support the research. Meanwhile, the University of Michigan research team has submitted a proposal to the Defense Advance Research Project Agency (DARPA) to begin testing the bioengineered interface in humans in three years.

Moving Towards An Open Singularity

Recently, I had a dialogue with some colleagues (Tina and RJ), about technology and the future. The focus of our discussion was the Metaverse and The Singularity. Although, my colleagues were unfamiliar with these exact terms. I believe the dialog important enough to want to share some thoughts about that discussion and the singularity prior to the Singularity Summit (which is happening in NYC on October 3-4). And I encourage anyone reading this to attend.

 

Yes, this post is long, but worthwhile, if for no other reason than to share the ideas of The Singularity and the Metaverse as well some new thoughts I had on those subjects.

 

So, the conversation with my colleagues when like this (paraphrasing):

 

- "What happens when.. virtual worlds meet geospatial maps of the planet?"

- "When simulations get real and life and business go virtual?"

- "When you use a virtual Earth to navigate the physical Earth, and your avatar becomes your online agent?"

-- "What happens then," I said, "is called the Metaverse."

I recall an observation made by polio vaccine pioneer Dr. Jonas Salk. He said that the most important question we can ask of ourselves is, "are we being good ancestors?"

 

This is a particularly relevant question for those of us that will be attending the Singularity Summit this year. In our work, in our policies, in our choices, in the alternatives that we open and those that we close, are we being good ancestors? Our actions, our lives have consequences, and we must realize that it is incumbent upon us to ask if the consequences we're bringing about are desirable.

 

This question was a big part of the conversation with my colleagues. Although, that is not an easy question to answer, in part because it can be an uncomfortable examination. But this question becomes especially challenging when we recognize that even small choices matter. It's not just the multi-billion dollar projects and unmistakably world-altering ideas that will change the lives of our descendants. Sometimes, perhaps most of the time, profound consequences can arise from the most prosaic of topics.

 

Which is why I'm going to write a bit here about video games.

 

Well, not just video games, but video games and camera phones (which many of my readers know - I happen to know quite a bit about), and Google Earth and the myriad day-to-day technologies that, individually, may attract momentary notice, but in combination, may actually offer us a new way of grappling with the world. And just might, along the way, help to shape the potential for a safe Singularity.

 

In the Metaverse Roadmap Overview the authors sketch out four scenarios of how a combination of forces driving the development of immersive, richly connected information technologies may play out over the next decade. But what has struck me more recently about the roadmap scenarios is that the four worlds could also represent four pathways to a Singularity. Not just in terms of the technologies, but—more importantly—in terms of the social and cultural choices we make while building those technologies.

 

The four metaverse worlds emerged from a relatively commonplace scenario structure. The authors arrayed two spectra of possibility against each other, thereby offering four outcomes. Analysts sometimes refer to this as the "four-box" method, and it's a simple way of forcing yourself to think through different possibilities.

 

This is probably the right spot to insert this notion: scenarios are not predictions, they're provocations. They're ways of describing different future possibilities not to demonstrate what will happen, but to suggest what could happen. They offer a way to test out strategies and assumptions—what would the world look like if we undertook a given action in these four futures?

 

To construct the scenario set the authors selected two themes likely to shape the ways in which the Metaverse unfolds: the spectrum of technologies and applications ranging from augmentation tools that add new capabilities to simulation systems that model new worlds; and the spectrum ranging from intimate technologies, those that focus on identity and the individual, to external technologies, those that provide information about and control over the world around you. These two spectra collide and contrast to produce four scenarios.

 

The first, Virtual Worlds, emerges from the combination of Simulation and Intimate technologies. These are immersive representations of an environment, one where the user has a presence within that reality, typically as an avatar of some sort. Today, this means World of Warcraft, Second Life, PlayStation Home and the like.

 

Over the course of the Virtual Worlds scenario, we'd see the continued growth and increased sophistication of immersive networked environments, allowing more and more people to spend substantial amounts of time engaged in meaningful ways online. The ultimate manifestation of this scenario would be a world in which the vast majority of people spend essentially all of their work and play time in virtual settings, whether because the digital worlds are supremely compelling and seductive, or because the real world has suffered widespread environmental and economic collapse.

 

The next scenario, Mirror Worlds, comes from the intersection of Simulation and Externally-focused technologies. These are information-enhanced virtual models or “reflections” of the physical world, usually embracing maps and geo-locative sensors. Google Earth is probably the canonical present-day version of an early Mirror World.

 

While undoubtedly appealing to many individuals, in my view, the real power of the Mirror World setting falls to institutions and organizations seeking to have a more complete, accurate and nuanced understanding of the world's transactions and underlying systems. The capabilities of Mirror World systems is enhanced by a proliferation of sensors and remote data gathering, giving these distributed information platforms a global context. Geospatial, environmental and economic patterns could be easily represented and analyzed. Undoubtedly, political debates would arise over just who does, and does not, get access to these models and databases.

 

Thirdly, Augmented Reality looks at the collision of Augmentation and External technologies. Such tools would enhance the external physical world for the individual, through the use of location-aware systems and interfaces that process and layer networked information on top of our everyday perceptions.

 

Augmented Reality makes use of the same kinds of distributed information and sensory systems as Mirror Worlds, but does so in a much more granular, personal way. The AR world is much more interested in depth than in flows: the history of a given product on a store shelf; the name of the person waving at you down the street (along with her social network connections and reputation score); the comments and recommendations left by friends at a particular coffee shop, or bar, or bookstore. This world is almost vibrating with information, and is likely to spawn as many efforts to produce viable filtering tools as there are projects to assign and recognize new data sources.

 

Lastly, we have Lifelogging, which brings together Augmentation and Intimate technologies. Here, the systems record and report the states and life histories of objects and users, enhancing observation, recall, and communication. I've sometimes discussed one version of this as the "participatory panopticon."

Here, the observation tools of an Augmented Reality world get turned inward, serving as an adjunct memory. Lifelogging systems are less apt to be attuned to the digital comments left at a bar than to the spoken words of the person at the table next to you. These tools would be used to capture both the practical and the ephemeral, like where you left your car in the lot and what it was that made your spouse laugh so much. Such systems have obvious political implications, such as catching a candidate's gaffe or a bureaucrat's corruption. But they also have significant personal implications: what does the world look like when we know that everything we say or do is likely to be recorded?

 

This underscores a deep concern that crosses the boundaries of all four scenarios: trust.

 

"Trust" encompasses a variety of key issues: protecting privacy and being safely visible; information and transaction security; and, critically, honesty and transparency. It wouldn't take much effort to turn all four of these scenarios into dystopias. The common element of the malevolent versions of these societies would be easy to spot: widely divergent levels of control over and access to information, especially personal information. The ultimate importance of these scenarios isn't just the technologies they describe, but the societies that they create.

 

So what do these tell us about a Singularity?

 

Across the four Metaverse scenarios, we can see a variety of ways in which the addition of an intelligent system would enhance the audience's experience. Dumb non-player characters and repetitive bots in virtual worlds, for example, might be replaced by virtual people essentially indistinguishable from characters controlled by human users. Efforts to make sense of the massive flows of information in a Mirror World setting would be enormously enhanced with the assistance of sophisticated machine analyst. Augmented Reality environments would thrive with truly intelligent agent systems, knowing what to filter and what to emphasize. In a lifelogging world, an intelligent companion in one's mobile or wearable system would be needed in order to figure out how to index and catalog memories in a personally meaningful way; it's likely that such a system would need to learn how to emulate your own thought processes, becoming a virtual shadow.

 

None of these systems would truly need to be self-aware, self-modifying intelligent machines—but in time, each could lead to that point.

 

But if the potential benefits of these scenarist worlds would be enhanced with intelligent information technology, so too would the dangers. Unfortunately, avoiding dystopian outcomes is a challenge that may be trickier than some may expect—and is one with direct implications for all of our hopes and efforts for bringing about a future that would benefit human civilization, not end it.

 

It starts with a basic premise: software is a human construction. That's obvious when considering code written by hand over empty pizza boxes and stacks of paper coffee cups. But even the closest process we have to entirely computer-crafted software—emergent, evolutionary code—still betrays the presence of a human maker: evolutionary algorithms may have produced the final software, and may even have done so in ways that remain opaque to human observers, but the goals of the evolutionary process, and the selection mechanism that drives the digital evolution towards these goals, are quite clearly of human origin.

 

To put it bluntly, software, like all technologies, is inherently political. Even the most disruptive technologies, the innovations and ideas that can utterly transform society, carry with them the legacies of past decisions, the culture and history of the societies that spawned them. Code inevitably reflects the choices, biases and desires of its creators.

 

This will often be unambiguous and visible, as with digital rights management. It can also be subtle, as with operating system routines written to benefit one application over its competitors (I know some of you reading this are old enough to remember "DOS isn't done 'til Lotus won't run"). Sometimes, code may be written to reflect an even more dubious bias, as with the allegations of voting machines intentionally designed to make election-hacking easy for those in the know. Much of the time, however, the inclusion of software elements reflecting the choices, biases and desires of its creators will be utterly unconscious, the result of what the coders deem obviously right.

 

We can imagine parallel examples of the ways in which metaverse technologies could be shaped by deeply-embedded cultural and political forces: the obvious, such as lifelogging systems that know to not record digitally-watermarked background music and television; the subtle, such as augmented reality filters that give added visibility to sponsors, and make competitors harder to see; the malicious, such as mirror world networks that accelerate the rupture between the information haves and have-nots—or, perhaps more correctly, between the users and the used; and, again and again, the unintended-but-consequential, such as virtual world environments that make it impossible to build an avatar that reflects your real or desired appearance, offering only virtual bodies sprung from the fevered imagination of perpetual adolescents.

 

So too with what we today talk about as a "singularity." The degree to which human software engineers actually get their hands dirty with the nuts & bolts of AI code is secondary to the basic condition that humans will guide the technology's development, making the choices as to which characteristics should be encouraged, which should be suppressed or ignored, and which ones signify that "progress" has been made. Whatever the degree to which post-singularity intelligences would be able to reshape their own minds, we have to remember that the first generation will be our creations, built with interests and abilities based upon our choices, biases and desires.

 

This isn't intrinsically bad; emerging digital minds that reflect the interests of their human creators is a lever that gives us a real chance to make sure that a "singularity" ultimately benefits us. But it holds a real risk. Not that people won't know that there's a bias: we've lived long enough with software bugs and so-called "computer errors" to know not to put complete trust in the pronouncements of what may seem to be digital oracles. The risk comes from not being able to see what that bias might be.

 

Many of us rightly worry about what might happen with "Metaverse" systems that analyze our life logs, that monitor our every step and word, that track our behavior online so as to offer us the safest possible society—or best possible spam. Imagine the risks associated with trusting that when the creators of emerging self- aware systems say that they have our best interests in mind, they mean the same thing by that phrase that we do.

 

For me, the solution is clear. Trust depends upon transparency. Transparency, in turn, requires openness.

 

We need an Open Singularity.

 

At minimum, this means expanding the conversation about the shape that a singularity might take beyond a self-selected group of technologists and philosophers. An "open access" singularity, if you will. Ray Kurzweil's books and lectures are a solid first step, but the public discourse around the singularity concept needs to reflect a wider diversity of opinion and perspective.

 

If the singularity is as likely and as globally, utterly transformative as many here believe, it would be profoundly unethical to make it happen without including all of the stakeholders in the process—and we are all stakeholders in the future.

 

World-altering decisions made without taking our vast array of interests into account are intrinsically flawed, likely fatally so. They would become catalysts for conflicts, potentially even the triggers for some of the "existential threats" that may arise from transformative technologies. Moreover, working to bring in diverse interests has to happen as early in the process as possible. Balancing and managing a global diversity of needs won't be easy, but it will be impossible if democratization is thought of as a bolt-on addition at the end.

 

Democracy is a messy process. It requires give-and-take, and an acknowledgement that efficiency is less important than participation.

 

We may not have an answer now as to how to do this, how to democratize the singularity. If this is the case—and I suspect that it is—then we have added work ahead of us. The people who have embraced the possibility of a singularity should be working at least as hard on making possible a global inclusion of interests as they do on making the singularity itself happen. All of the talk of "friendly AI" and "positive singularities" will be meaningless if the only people who get to decide what that means are the few hundred who read and understand this blog posting.

 

My preferred pathway would be to "open source" the singularity, to bring in the eyes and minds of millions of collaborators to examine and co-create the relevant software and models, seeking out flaws and making the code more broadly reflective of a variety of interests. Such a proposal is not without risks. Accidents will happen, and there will always be those few who wish to do others harm. But the same is true in a world of proprietary interests and abundant secrecy, and those are precisely the conditions that can make effective responses to looming disasters difficult. With an open approach, you have millions of people who know how dangerous technologies work, know the risks that they hold, and are committed to helping to detect, defend and respond to crises. That these are, in Bill Joy's term, "knowledge-enabled" dangers means that knowledge also enables our defense; knowledge, in turn, grows faster as it becomes more widespread. This is not simply speculation; we've seen time and again, from digital security to the global response to influenza, that open access to information-laden risks ultimately makes them more manageable.

 

The Metaverse Roadmap offers a glimpse of what the next decade might hold, but does so recognizing that the futures it describes are not end-points, but transitions. The choices we make today about commonplace tools and everyday technologies will shape what's possible, and what's imaginable, with the generations of technologies to come. If the singularity is in fact near, the fundamental tools of information, collaboration and access will be our best hope for making it happen in a way that spreads its benefits and minimizes its dangers—in short, making it happen in a way that lets us be good ancestors.

 

If we're willing to try, we can create a future, a singularity, that's wise, democratic and sustainable—a future that's open. Open as in transparent. Open as in participatory. Open as in available to all. Open as in filled with an abundance of options.

 

The shape of tomorrow remains in our grasp, and will be determined by the choices we make today. Choose wisely.

Helping The Blind To "See"

Those who know me know that I have a keen interest in assistive technologies for people with physical and cognitive disabilities. This interest came about as a child as my grandmother Madeline "Madge" Snyder brailled hundreds of books for the Library of Congress and started the now famous "Books On Tape" program for those with visual impairments.

My interest then grew significantly when I had the opportunity to produce the interactive and motion graphics components of the 2000 Paralympic Games is Sydney, Australia. braille.jpgSince then, I have tried to keep current with the latest advances in technologies for the disabled. To that end, I stumbled across an exciting licensing agreement for two new technologies that will help bring affordable graphic reading systems to the blind and visually impaired. The systems give physical dimension to electronic images in the same way that Braille makes words readable.

The Braille system, which incidentally, was based on a method of communication originally developed by Charles Barbier for Napoleon's soldiers, was devised by Frenchman Louis Braille in 1821. As we know, braille allows vision impaired people to read and write using characters made up of raised dots. Braille has been used the same way as a system for almost two centuries.  But, these new technologies could mark a significant change in the way the blind are able to “see” in that they incorporate images, rather than words and numbers.

One of the new systems, a tactile graphic display device and fingertip graphic reader, were developed by researchers at the National Institute for Standards and Technology (NIST). The tactile graphic display for localized sensory stimulation, was created using an array of about 100 small, very closely spaced (1/10 of a millimeter apart), actuator points set against a user’s fingertip. To “see” a computer graphic with this technology, a blind or visually impaired person moves the device-tipped finger across a surface like a computer mouse to scan an image in computer memory. The computer sends a signal to the display device and moves the actuators against the skin to “translate” the pattern, replicating the sensation of the finger moving over the pattern being displayed. With additional development, the technology could possibly be used to make fingertip tactile graphics practical for virtual reality systems or give a detailed sense of touch to robotic control (teleoperation), and space suit gloves.

The second technology, introduced as a prototype in 2002, conveys scanned illustrations, map outlines or other graphical images to the fingertips, and can translate images displayed on Internet Web pages or in electronic books. It uses refreshable tactile graphic display technology, allowing a person to feel a succession of images on a reusable surface. The machine uses about 3,600 small pins that can be raised in any pattern, and then locked into place to hold the pattern for reading. The actuator points then can be withdrawn and reset in a new pattern, allowing the tactile reading to continue through a variety of images.

If the devices look familiar to you, it’s because inspiration came from a “bed of nails” toy found in many novelty stores. If you haven't seen it, the toy allows you to press your hand or face or an object onto the back of the nails and they raise up to create an "image" of that object. Watching the pins in the toy depress under fingers and then return to their original state started the researchers thinking about how the principle could be applied to electronic signals. NIST recently signed a non-exclusive license for commercialization of its two tactile graphic display technologies with ELIA Life Technology which may soon see the two products become commercially available.