A bright green avatar stands motionless on the TV screen, mirroring the person standing ten feet away. Above the avatar, a digital Block “M” flag hangs attached to a digital brick wall. This is not the newest video game. This is the newest generation of rehabilitation.
For the past ten years, the Digital Media Commons (DMC), housed in the Duderstadt Center, has lent its technological prowess to Michigan Medicine, together creating a mechanism to aid recovery for chronic stroke survivors.
The technology used– dubbed “Green Man”– uses Microsoft Xbox Kinect technology to mirror the realtime movements of patients to a blank avatar, a green-screen-color man, on the television screen in front of them. The goal of the technology is to mirror the non-paretic limbs, or the limbs unaffected from a stroke, to show a body making symmetrical movements even if the patient in front of them is performing asymmetrically.
“It’s helpful, to fool our nervous system into thinking that one side of the body that is impaired is moving and being utilized appropriately,” said Dr. Mary Roberts, a site supervisor of the Canton Therapy Services through Michigan Medicine, and a clinician for the last 18 years. “The patients know they have a limb there already, it’s just not moving the best. And so we married mirror imaging and action observation therapy together using [extended] reality.”
Roberts is the clinician that brought the idea to the DMC back in 2016. It was her idea to pair mirror image therapy, a therapy that includes mirroring a stroke patient’s non-paretic side, and action observation therapy, a therapy that includes copying another’s movements. Both strategies are found to be successful for stroke patients, though when paired, it can often be costly. The Green Man was created to answer the need for a tool that uses the marriage of the two strategies in an easy-to-implement way for Michigan Medicine stroke patients.
“It was a brainchild of myself and one of our other therapists, Hendrika Lietz,” said Roberts. “What if we could show stroke patients what they’re doing on a screen, but show them how they can move their [paretic] limb perfectly without any massive monetary investment. A lot of that technology is there, but it’s [money] we don’t have to spend in our small clinic. How can we make something that is similar, [and] obviously targeted to stroke patients?”
From there, Roberts asked around and found the DMC, reaching out for their help in getting a Xbox Kinect up and running to implement a clinical trial.
The project came to Ted Hall, Advanced Visualization Specialist in the Visualization Studio at the Duderstadt Center, and came at a good time. Four years prior, an EECS student named Rachael Miller was interested in learning the fundamentals of application development for the Kinect and began to work on the Kinect and the virtual-reality application at the time. When Roberts approached the DMC for the Green Man, it was upon Miller’s basis the program was made.
“[Miller] did a lot of the early work,” said Hall. “She helped create a scene with great big boxes, so that she could come in with a Kinect and knock these boxes down.”
The scene with boxes, though no boxes now live in the “Green Man” world, served an important initial purpose. The interaction between body movement and the boxes helped create the physics for the program.
“You see your body, and see yourself touch and interact with this thing using your hands, not using a game controller or anything,” said Hall. “You can step into that immersive space, and you can see the stack of boxes. If you punched it, it would fall down, right? And so that was [Miller’s] goal, and we got that to work.”
What made the physics work was something called colliders, data that was driving the invisible physics. The original Green Man was more a stickman, with the data points at bones and joints creating the physics.
“We were rendering like a capsule for each bone, basically,” said Hall. “So it’s like a stack of capsules, very stick-man-ish.”
From the original stick-figure, the skin mesh was added on. The skin? The bright green-screen color patients have come to know and love. From then on, the name Green Man began and has stayed ever since.
A year after their first connection, Roberts emailed Hall asking to implement a game or way to make the technology more interactive. Per her request, Hall added a University-proud mechanic to the program, a Block “M” flag. Hall also added physics collision detection to the scene to trigger audible rewards (“Let’s Go Blue!”) for touching the flag. The height of the flag is also adjustable, allowing the doctors to manipulate the practice for whatever suits the patient’s needs.
“In terms of neuroplasticity, there are ten principles that we look into and one of them is salance, or how meaningful something is instead of just going through the motions,” said Roberts. “One of [the activities] was where we had the patient stand up and try to bat the Michigan flag like you’re batting flag when the guys run through the tunnel on the football field.”
With “Let’s Go Blue!” playing after the flag is hit, it can make the attempts more rewarding for the patients. Additionally with this feature, there is bio-feedback that can make patients more engaged and competitive with themselves in the rehabilitation process.
“When you get the target, there’s some bio-feedback, some information saying, ‘oh, we hit [the flag] because the song goes on afterwards,’” said Roberts. “It’s nice because, for example, if a patient’s left hand is the one that’s impaired and they’re mostly batting with their right, they are still seeing their left hand bat. It’s cool because you can kind of see [patients] really trying with both hands to get up there as much as they can, they’re really engaged.”
By the end of June 2017, Hall sent Roberts the final Green Man deliverable for patient use and her clinical study, which began over the summer. Her clinical study involved three rounds of testing: pre-testing, in-progress assessment, and post-testing patients who completed 30-minutes of conventional physical therapy and an additional 30-minutes of utilizing the Green Man. The results of the retention test were quite interesting, with significant results being noticed after three months’ rest.
“We brought the ones that could come back three months later and did retention tests, which was really neat to see because we noticed that there was some retention in terms of their gait speed, which was cool because we didn’t work with gait at all in the Green Man,” said Roberts. “There was actually a significant change with gait speed and five times sit-to-stand in terms of our data. Once we were done collecting data, then we started to work toward getting things presented.”
After gathering the data and months of edits, the team had their final submission for technology enhanced real-time action observation therapy (TERTAOT) in survivors with chronic stroke published in 2021.
“It’s engaging,” said Roberts. “It’s salient. [Patients] were super excited about it because then even after the session was over, they had that little bit of carryover, which was cool.”
Roberts shares a success she has had with the Green Man, excited about even the little wins this technology could have.
“One of my patients never used his arm for anything, and was 16 years poststroke,” said Roberts. “He came in day one, did the activities that we had him do, and the next day he came back, he’s like, ‘Mary, check this out.’ And he actually did a shoulder abduction. He’s like, ‘I can get deodorant under my arm.’ I’m like, ‘Yeah!’ You know, just those little wins, right? We were seeing some really good improvements just from day one and on. It was really neat to see.”
Roberts would like to thank the team who got the project as far as it is, and who have made it last for the past decade: “William Farr, Hendrika Lietz, and Alyssa Portelli, Min-Hui Huang, and then obviously to Ted Hall for everything that he’s done to just get us to the point of implementing that initial brainstorm.”
Roberts and her team’s research findings are published in Physiotherapy Theory and Practice Vol. 38.
Photo Credits: Duderstadt Center Visualization Studio
Article by Emma Powell