The University of Michigan Extended Reality Initiative: Embracing the virtual future

Story re-published courtesy of Microsoft.

Operating rooms are crammed with hundreds of thousands of dollars worth of medical equipment. 

Heavy, powerful machines, stacked like stone sculptures around the operating table, display crucial information about the patient. Surgical teams dance around enormous booms that hold multiple screens and all the accompanying wires that connect these monoliths to power. Even state-of-the-art displays require surgeons to look up from their patients to find information, pausing their work to crane their necks to the right, to the left. It’s a difficult ‘dance’ for a delicate job.

Luckily, students at the University of Michigan may have developed an effective means of streaming multiple flows of patient information through one lightweight, ergonomically efficient tool: a Microsoft HoloLens, worn right on the surgeon’s head. 

The idea for this cost-and-space saving extended reality application came from the classroom of David Chesney, Ph.D., a lecturer in the Department of Computer Science of Engineering. Together with Dr. Ron Hirschl and Dr. Marcus Jarboe, David and his students built a HoloLens application that could, if widely adopted, save hospitals hundreds of thousands of dollars while making surgery more ergonomic and effective. 

“With the HoloLens,” says Ron, who heads the U-M Section of Pediatric Surgery, “what you have is a monitor on your face, with data that is right in front of you that you can manipulate. You could bring in two different data streams, you can move those data streams around, scroll through the CT and so on, simply using hand gestures to manipulate them: to make them bigger, smaller, pull them closer when I need them or away when I don’t.”

This is just one of hundreds of ways extended reality is shaping the future, and the University of Michigan is leading the way.

The University of Michigan Extended Reality Initiative 

Faculty from an array of disciplines planted the seeds for this advancement several years ago, when they convened to discuss the potential of teaching, learning, and researching with extended reality at the University of Michigan.

The term “extended reality” (XR) refers to technology that, to varying degrees, alters the users’ perception of the world: virtual reality (VR), augmented reality (AR), and mixed reality (MR). Virtual reality immerses users in an entirely virtual world through a headset. Augmented reality enhances a user’s existing environment with digital inputs such as graphics or, in Ron’s case, medical data. Mixed reality (MR), as the term implies, can be a combination of VR and AR, connecting an immersive digital experience with real-world items, such as converting real-world items into completely different objects within the MR environment. 

In 2016, as part of the Microsoft HoloLens Seeding Program, Microsoft provided researchers at 17 North American universities with 180 Microsoft HoloLens Commercial Suite AR devices, 35 of which ultimately landed at the University of Michigan. Researchers from all over campus put the devices to work immediately, in disciplines as disparate as kinesiology, art and design, medicine, business, engineering, and more. 

In September of 2019, the University of Michigan officially kicked off the UM Extended Reality Initiative, a campus-wide campaign housed in the University of Michigan Center for Academic Innovation which seeks to integrate XR into residential and online curricula through the creation of public/private partnerships to develop new XR educational technology. 

Proponents of the initiative see XR not just as a neat technology but as a professional skill that students should be able to acquire before graduation—a new and vital way to train inquisitive students for the workforce of the future. “While there are many questions to explore, we see the potential of XR technologies — augmented, virtual, and mixed reality — to fundamentally change the way we teach and learn,” says Jeremy Nelson, Director for the U-M XR Initiative.

“I see us as a service center for the university. My vision is that any faculty that wanted to use XR would consult with us, determine their learning goals and objectives, identify the proper technology stack, discover existing content—if it exists—or create original content, and then we will help them build out their projects.” The initiative includes funding for two full-time developers, and supplemental student developers and designers, to help with design, development, 3D modeling, and more. 

Improving people’s understanding of XR at Michigan and beyond

University of Michigan School of Information Assistant Professor Michael Nebeling leads the Information Interaction Lab, where he conducts research, teaches courses, and mentors both students and faculty in XR. 

As the 2020 XR Faculty Innovator-In-Residence, Michael was instrumental in laying the groundwork for the XR Initiative. He partners closely with Microsoft XR experts and attended the Microsoft Faculty Summit in 2019.

In an example of how XR is evolving within the U-M academic ecosystem, one of Michael’s recent studies examined how XR could be used in crisis simulation. Before XR took a foothold at the university, crisis simulation research involved placing subjects in front of television screens showing war movies and then evaluating them through task completion after the screenings. Michael and his team created a multi-user HoloLens crisis simulation, with a toolkit to collect and visualize the data collected: the mixed-reality analytics toolkit. With the proper XR research tools, researchers can generate more comprehensive data and explore areas of understanding previously out of reach.

This kind of process-improvement and tools-focused work is the heart of Michael’s efforts. His toolkits for XR design and evaluation help drive solid research and innovative teaching.

“We’ve worked on extended reality courses for both U-M and for Coursera, as well as tools for the design and evaluation of XR applications, and we’ve had a lot of involvement across Michigan in various projects,” says Michael.

In detailing how the Initiative is funding six pilot projects in its initial round of XR funding, Michael shares, “With the U-M XR Initiative, we now have a production team that will help put together workflows to guide instructors and help them identify good topics that will benefit from XR.”

Michael concludes, “This is really ambitious, but a major benefit is that both more faculty and students will be trained in the use of XR technologies, and this will help transform teaching and learning.”

Training students for careers in extended reality

A major part of the XR Initiative is the Interdisciplinary Graduate Certificate Program in Augmented and Virtual Reality, one of the first official XR graduate programs in the country, which launched in January 2020. U-M faculty developed the certificate program to formalize academic XR offerings at the university. Anson Ho, Microsoft Education Program Manager, worked with U-M on XR for the past year. He served as an advisor for the certificate program, reviewing the goals and curriculum as they were developed, and he’s spoken at student-led XR conferences, such as XR Midwest 2020. 

In addition to the U-M faculty, the certificate program was shaped by major players in the XR industry including HP, Lenovo, Facebook, and Disney, the last of which articulated a clear need for graduates with XR skill sets. 

“The main benefit of the graduate program is that students learn how to develop for mixed reality and how to actually apply XR into a meaningful use-case, whether it’s education, medicine, dentistry, or film. The focus is really on developing practical skills,” explains Anson. 

“This has been a four-year collaboration between U-M and Microsoft, and we’re very excited about the graduate certificate and expansion of XR work across the various U-M schools and colleges,” says Kent Foster, Director, Microsoft University Relations.

As part of the U-M XR Initiative, and to kick off the brand-new certificate program, the university organized a series of monthly guest speakers to educate and inspire faculty across all 19 schools and colleges. The guest speaker series was meant to culminate in the XR at Michigan Summit, for which Microsoft was the Platinum Sponsor. The XR Summit is currently postponed until October 2020.  

Excited students blaze the path into the virtual future

Sophie Stellmach, a senior scientist working on the Microsoft HoloLens team, has seen firsthand how students embrace XR technology. She was a guest lecturer for one of the XR certificate program’s very first classes.

“It’s always very interesting when you bring these devices to the classroom because you have students with very diverse backgrounds, and they all have their own unique ideas about how this new technology could be used,” she says.

“Working with students is so exciting and exhilarating because you get these new ideas and it’s not just like being in some office. I love opportunities to reach out and work with students.”

David Chesney understands the effervescent potential of putting novel technology in engineering students’ curious hands. He’s active within the XR Initiative, but David was guiding students as they explored and innovated with XR technology for years before the initiative launched. He knows what students are capable of and delights in sharing their work.

Of the students’ surgical HoloLens application, David says: “Ron and Marcus were just blown away by how good this thing possibly could be. It’s basically a senior capstone project, so it needs refining, but it also very clearly showed the potential for the use of this technology.”

As one of the collaborating surgeons, Marcus was even more enthusiastic. 

“Holy bananas are those students smart. This application is incredible, and it’s so close to being ready, it just needs a little streamlining and then the cost savings, wow. I have a pretty good sense of when something comes along that we’ve been waiting for, and this is it.”