Neuromotus:
Improving Lives of Amputees
How Augmented Reality helps to avoid phantom limb pain
Using open-source software, freeware games and a webcam, a new treatment developed in Sweden successfully tackles phantom limb pain. Read More
Max Ortiz Catalan
Imagine your right hand is firmly closed in a fist, and that it has been that way for more than half your life. Your fingers feel cramped and stiff, the muscle strain makes your wrist hurt, a sharp pain pervades your lower arm. All you want to do is release the fist and relax your hand. But you can't, because your lower arm is not there anymore – it was amputated 48 years ago.
Such is the story of Ture Johansson from Gothenburg, who lost his limb in a car accident and has suffered from phantom limb pain (PLP) ever since, as do 70% of all amputees. Ture tried almost everything to find relief from PLP: traditional mirror therapy, acupuncture, a variety of pharmaceutical drugs, and self-suggestive hypnosis. But all these treatments failed to address the symptoms the 72-year-old was experiencing.
Ture had to deal with excruciating pain on a daily basis that frequently preventing him from sleeping. Then, after half a century, a new treatment developed in Sweden helped him overcome this agonizing condition.
mind-control of phantom limbs
Neuromotus is a promising new therapy for all those who suffer from PLP. By creating a virtual and augmented reality, it tricks the brain into believing that a limb is there, responding to motor commands predicted by neural decoding algorithms, although in reality the limb is no longer present. A joint effort between Chalmers University of Technology and the medical company Integrum, which specializes in the field of titanium implants for amputees, the novel treatment combines neuroscientific research and technical knowledge.
The treatment approach is primarily based on the theory of cortical reorganisation. When a person loses a limb, the nerves to the brain do not receive the same kind of signals they used to. The nerves to the muscles that were amputated are no longer in use as well. This causes the brain to re-wire itself and it is believed that this rewiring is the cause of PLP.
Neuromotus tries to restore the brain to its original state by having people use the nerves for their muscles again. Interesting enough, patients often report sore muscles after the first treatment, similar to what they would experience it after a hard workout.
In the lab, Dr Max Ortiz Catalan and his research team, Morten Bak Kristoffersen, Alejandra Zepeda E., and Rannveig A. Gudmundsdottir, attach electrodes to the stump that is left of Ture’s arm. When Ture enters the augmented reality of Neuromotus, he sees himself on a screen in real time, captured by a conventional webcam. Except that his body has a new addition: a virtual limb that reacts to myoelectric muscle signals transmitted by the electrodes. This is how Ture's body controls the arm on the screen.First he practices simple movements such as opening and closing his hand or extending his elbow. Although the muscles that are usually responsible for these movements were lost a long time ago, Ture's brain and nervous system are able to produce muscle synergies that are distinctive enough for Neuromotus to decode and translate them into the simulated motor control of the virtual hand.
Patients play pain away
Of course this takes some practice, but step by step Ture takes control of his phantom elbow, lower arm, wrist and fingers. "In the beginning it was very hard to activate the muscles because I had not used them for a very long time. But now I can even move individual fingers. It's very funny," he describes the process in an interview.
© Max Ortiz Catalan After he has acquired the basic skills to control the virtual replacement of his phantom limb, Max Ortiz Catalan suggests a very exciting way to further refine them: by playing the racing game Trackmania Nations Forever. Engaging in this playful exercise, Ture learns to reactivate areas in the brain related to the motor control of the missing limb. By providing visual feedback, Neuromotus makes the brain believe that there is actually a limb that responds to its commands – leading to a steady decrease in PLP with every session.
"I always thought that augmented reality would be a really compelling way for patients to train," says Max, highlighting the value of interactive games for challenging and keeping patients entertained and motivated during treatment. "I was lucky to find Morten, who specializes in the development of augmented reality environments. He found the idea interesting and agreed to do it as a university project. Similarly, Ale and Rannveig have been of great help in supporting the electronic and clinical fronts, which are fundamental for such a project."
Great improvements after 15 weeks
As he learned to control his virtual limb with increasing precision, Ture's pain level gradually declined. 10 weeks into treatment, moments where the pain was almost absent began to occur, and completely pain-free periods set in a couple of sessions later. Although he steers his car solely using phantom movements, Ture performed no worse in the racing game than other Trackmania players after 15 weeks. Now that he has taken control of his phantom limb, Ture is able to release the fist that bothered him for 48 years – even outside the virtual reality created by Neuromotus.
"The pain-free periods are something almost new to me and it is an extremely pleasant sensation", says Ture, who especially enjoys that he now work in the garden without anticipating intense pain afterwards. He also no longer wakes up at night due to PLP. "My husband can live 10 years more than I expected, as pain now plays a less important role in his life, and those close to him can see it," his wife remarks happily.
Open-source and suitable for home-use
At the beginning of Ture's treatment, the motion of the phantom limb was predicted by BioPatRec, an open-source platform initially developed for advanced prosthetic control strategies based on pattern recognition algorithms. However, BioPatRec requires a Matlab license and is only suitable to be used by an exceedingly tech-savvy clientele. The Neuromotus software is easier to use, and clinical staff can perform the PLP treatment themselves after very brief initial training.
Neuromotus features many more advantages as well that put it in pole position in the race for the PLP treatment of the future. Besides being cheap and effective, it works for bilateral amputees too, which traditional mirror-box therapy does not. Since the technology has been combined into a user-friendly package, it can also be administered by the patients themselves, making Neuromotus suitable for use at home.
Progress doesn’t stop here
In addition to his work on Neuromotus, Max has developed the osseointegrated human-machine gateway (OHMG), a new type of artificial limb that is directly connected to the patient's muscles, nerves and bone. During simulation sessions in the lab, signals from the brain are registered by the limb, and signals from the artificial limb are sent back to the brain. When it is touched, patients register sensations as though the limb were real.
Patients can even go skiing.
The direct connection between the patient's neuromuscular system and the artificial limb proves beneficial in many situations. For example, traditional prosthesis don't work very well in winter, because when it is cold the patient's skin gets very dry and electrodes do not transmit the signals accurately. This problem is overcome by implanted electrodes accessed via osseointegration, which means that patients can even go skiing.
© Max Ortiz Catalan 
© Max Ortiz Catalan 
© Max Ortiz Catalan Another advantage of osseointegration is that patients don't need to use a socket, which is often painful, sweaty and uncomfortable, and has been compared to wearing a shoe that does not fit. Also, a socket constrains the patient’s arm, so they cannot move as freely as they would like to.
"Our first patient has had the system for almost two years, and we'll be performing more implants on patients this year," reveals Max, who originally imagined Neuromotus as a training system for OHMG prosthesis users. "Until our work, all research experiments had been confined to the lab. Because of our osseointegrated interface, we can actually provide patients with implanted neuromuscular interfaces to control their prosthesis at home."
"One of the goals with the virtual environments is to train future patients who are to be fitted with the OHMG prior to surgery, so that they have maximum control of the prosthetic. Our current work involves developing a piece of hardware that does the pattern recognition that the computer currently does, and that is small enough to fit between the prosthetic and the OHMG," Morten explains further.
Obviously, perfecting the OHMG is no easy task. "It has to be said that the software is only one part of the treatment, the hardware to receive the signals from the patient is also very important, and the way treatment is conducted is obviously the most important. But it doesn't stop there. Finding partners in clinics who are willing to spend their own time (with no payment) exploring this treatment is also fundamental. You can't convince clinicians to join you in such a venture without doing your homework, so there are a lot of things that were needed in order to move this project forward", says Max, who has been working on advanced prosthetic control for 5 years now.
A pain-free future
At the moment, the people behind Neuromotus cannot tell if Ture's pain level would stay low permanently if treatment were terminated. As he has suffered from severe PLP for 48 years, they think it unethical to test the long-term effects of the treatment on him. Instead, Ture has been provided with a stand-alone system that he uses to enter augmented reality and further improve his motor control at home anytime he wants to.
Because the software as well as the virtual reality environment and all the source code necessary for motion prediction are available for free, other researchers all over the world can further develop the treatment and apply it to other purposes. In the future, the system might also be used to treat stroke or spinal cord injury patients. "I believe that myoelectric control will be a big buzz in a few years and it would be awesome if BioPatRec became the main development platform for that movement," says Morten. "My hopes are that I can continue to develop systems for many years to come that can improve the lives of people who suffer from a condition. Hopefully, these systems will involve some nice new technologies that allow new modes of interaction, as I have a big interest in human-computer interaction", he explains his plans for the future.
Meanwhile, Ture already enjoys his pain-free life achieved by regaining control over his phantom limb with the help of state-of-the-art technology developed by Max’s group. "It's fantastic after all these years. And I hope that it will work very well for other people too", he says.
Neuromotus was awarded with the European Youth Award in the category "Healthy Life" in 2014. The European Youth Award is a pan-European contest to select and honour "Digital Creativy for Social Good!".
Like to learn more? Watch the Videos!
Phantom limb pain (PLP) treatment based in VR, AR, gaming, and MPR 
Natural Control of Artificial Limbs through an Osseointegrated Implant 
Bionic limbs integrated to bone, nerves, and muscles | Max Ortiz Catalan | TEDxGöteborg
