Sensory Processing – in a Virtual Kodak Hall

a binaural microphone set up with a dummy head.

Rochester researchers will harness the immersive power of virtual reality to study how the brain processes light and sound.

A cross-disciplinary team of researchers from the University of Rochester is collaborating on a project to use virtual reality (VR) to study how humans combine and process light and sound. The first project will be a study of multisensory integration in autism, motivated by prior work showing that children with autism have atypical multisensory processing.

The project was initially conceived by Shui’er Han, a postdoctoral research associate, and Victoire Alleluia Shenge ’19, ’20 (T5), a lab manager, in the lab of Duje Tadin, a professor of brain and cognitive sciences.

“Most people in my world—including most of my work—conduct experiments using artificial types of stimuli, far from the natural world,” Tadin says. “Our goal is to do multisensory research not using beeps and flashes, but real sounds and virtual reality objects presented in realistically looking VR rooms.”

UR students working on the project are looking at information on a laptop with Kodak Hall in the background.
Members of the team begin the setup for audio and visual data collection. From left to right are Shui’er Han, a postdoctoral research fellow in Duje Tadin’s lab; brain and cognitive sciences major Betty Wu ’23; computer science and business major and e5 student Haochen Zeng ’23, who works in River Campus Libraries’s Studio X; and Victoire Alleluia Shenge ’19, ’20 (Take Five), who earned her degree in brain and cognitive sciences and is a manager in Tadin’s lab.

A cognitive scientist, a historian, and an electrical engineer walk into a room . . .

Tadin’s partners in the study include Emily Knight, an incoming associate professor of pediatrics, who is an expert on brain development and multisensory processing in autism. But in creating the virtual reality environment the study participants will use—a virtual version of Kodak Hall at Eastman Theatre in downtown Rochester—Tadin formed collaborations well outside his discipline.

Faculty members working on this initial step in the research project include Ming-Lun Lee, an associate professor of electrical and computer engineering, and Michael Jarvis, an associate professor of history. Several graduate and undergraduate students are also participating.

Many of the tools they’ll use come from River Campus Libraries—in particular, Studio X, the University’s hub for extended reality projects, as well as the Digital Scholarship department. Emily Sherwood, director of Studio X and Digital Scholarship, is leading the effort to actually construct the virtual replica of Kodak Hall.

The group recently gathered in the storied performance space to collect the audio and visual data that Studio X will rely on. University photographer J. Adam Fenster followed along to document the group’s work.

Read more.

Voices of XR: John Haddick

Voices of XR speaker John Haddick.

ThinkReality:  Helping Build the Enterprise Metaverse

Voices of XR speaker John Haddick.

John Haddick is a Distinguished Engineer with the Lenovo Intelligent Devices Group, which focuses on developing technologies for the ThinkReality commercial AR/VR portfolio of solutions. As the CTO of ThinkReality, John is a leading strategist for AR/VR product design and development for the company.

Prior to joining Lenovo, John was the CTO of the Osterhout Design Group, a company pioneering AR technologies, where he was in charge of new product innovations and long range product strategy across the company.

In this talk, he will describe Lenovo’s investment and development of commercial AR/VR solutions. From the ThinkReality software, one of the industry’s first cloud and device agnostic platforms, to the award-winning AR smart glasses, Lenovo is making it easier for enterprises to deploy and scale AR/VR.

The Voices of XR speaker series is made possible by Kathy McMorran Murray and the National Science Foundation (NSF) Research Traineeship (NRT) program as part of the Interdisciplinary Graduate Training in the Science, Technology, and Applications of Augmented and Virtual Reality at the University of Rochester (#1922591).

Where: Studio X & Zoom
When: Friday, April 29th from 1 to 2pm EDT
Register: bit.ly/VoXRJohnHaddick


graphic for XR speaker series entitled Voices of XR. On the bottom, is an illustration of a person in a headset reaching with neon geometric squares in the background. On the top is text that reads: "Voices of XR: A Studio X Speaker Series." On top of that, is the Studio X and River Campus Libraries wordmarks.

Voices of XR is a Studio X speaker series. Speakers are scholars, artists, and extended reality professionals who discuss their work with immersive technologies across disciplines and industries. All talks are free and open to the general public.

Voices of XR: Sylvia Xueni Pan

speaker, Sylvia Xueni Pan.

Virtual Humans in VR

speaker, Sylvia Xueni Pan.

Dr. Sylvia Xueni Pan is a Senior Lecturer in VR at Goldsmiths, University of London. She has a BEng in Computer Science from Beihang University, an MSc in Graphics and PhD in Virtual Reality from UCL (funded by the EPSRC). Before coming to Goldsmiths, she worked as a research fellow in Computer Science at UCL and at the same University’s Institute of Cognitive Neuroscience (ICN). Her research interest is the use of virtual reality (VR) as a medium for real-time social interaction, in particular in the application areas of training and therapy. Dr Pan has developed a unique interdisciplinary research profile, with journal and conference publications on both VR technology and social neuroscience. Her work has been featured on the BBC, in New Scientist magazine and the Wall Street Journal. Her 2017 Coursera VR specialisation attracted over 100,000 registered learners globally. She co-leads the Goldsmiths Social, Empathic, and Embodied VR lab (SeeVR Lab) and the MA/MSc in Virtual and Augmented Reality programme at Goldsmiths Computing.

The Voices of XR speaker series is made possible by Kathy McMorran Murray and the National Science Foundation (NSF) Research Traineeship (NRT) program as part of the Interdisciplinary Graduate Training in the Science, Technology, and Applications of Augmented and Virtual Reality at the University of Rochester (#1922591).

Where: Studio X & Zoom
When: Friday, April 22nd from 1 to 2pm EST
Register: bit.ly/VoXRSylviaPan

Recording

Event recording available for UR faculty, students, and staff with NetID and password: bit.ly/XRSylviaPan


graphic for XR speaker series entitled Voices of XR. On the bottom, is an illustration of a person in a headset reaching with neon geometric squares in the background. On the top is text that reads: "Voices of XR: A Studio X Speaker Series." On top of that, is the Studio X and River Campus Libraries wordmarks.

Voices of XR is a Studio X speaker series. Speakers are scholars, artists, and extended reality professionals who discuss their work with immersive technologies across disciplines and industries. All talks are free and open to the general public.

XR Game Night

promotional graphic for XR game night. Shows some illustrations of people in VR headsets.

Take a break from studying and unwind at XR Game Night at Studio X! The night will begin with a brief headset tutorial, and you can reserve the headsets after the event to keep playing later. We will have snacks, beats, and games to relax, have fun, and vibe with!

promotional graphic for XR game night. Shows some illustrations of people in VR headsets.

Instructor: Nefle Nesli Oruç
Where: Studio X, Carlson Library First Floor
When: Wednesday, 4/20 @7:30pm
Register: bit.ly/VRGameNight

Unity Flapper Bird Pre-Work

This process should take around 15 minutes depending on your download speed.

Step 1: Download and Install the Unity Hub (available on Mac and PC)

  1. Download the Unity Hub here.
  2. From your Downloads folder, double-click on the Unity Hub Setup file to begin the installation.
  3. Agree to Unity Terms of Service and follow the instructions to install Unity Hub.
  4. Open Unity Hub for the first time.

Step 2: Download and Install Unity Version 2021.1.16f1

There are many different versions of the Unity software, as its developed and released over time. During this workshop, we’ll be using version 2021.1.16f1.

  1. In the Installs tab, click Add to add a new Unity version.
  2. Choose 2021.1.16f1.
  3. Choose to install Visual Studio (for Mac or PC) and Android Build Support. (very important!)
  4. Accept any necessary terms and conditions and begin installation.

Step 3: Create a Unity ID

  1. From the Account menu in the Unity Hub, click to sign in.
  2. If you already have an account, sign in. Otherwise, you can create a new Unity ID.

You may receive a notice about needing to activate a license. You can do this under the settings menu (click the cog in the upper-right corner). From there, you can click “License Management” in the menu and choose a personal license.

Select “Unity Personal” and “I don’t use Unity in a professional capacity.”

Step 4: Download the Workshop Project

Download this file. Extract the file.

Step 5: Watch this Short Tutorial

This video will walk you through some key Unity concepts that will make the workshop more useful for you.

Is Extended Reality Shaping the Future of Academic Libraries? This Dean Thinks So.

Studio X salon area. Shows students sitting or standing near the entrance of Studio X.
Mary Ann Mavrinac, vice provost and dean of the University of Rochester Libraries, shares insight into how the campus community directed the development of Studio X, the library’s new extended reality hub featuring advanced technology and expert training 

“I don’t believe in ‘if you build it, they will come.’ You can build something, but they won’t come if you don’t know what your users want,” Mavrinac said. It’s the guiding principle she and her team followed throughout the ideation and planning of the library’s new high-tech hub, Studio X. Located on the first floor of the Carlson Science and Engineering Library, the 3,000 SF space allows students and faculty to participate in immersive learning experiences.

Equipped with technology that supports virtual reality (VR), augmented reality (AR) and everything between (extended reality or XR), Studio X allows researchers to perform tasks such as visualizing large data sets and safely experimenting with hazardous materials by creating a virtual environment. Studio X broadens the range of possibilities for discovery and instruction, but what makes it truly special is its source of inspiration. CannonDesign collaborated with the university to design a facility that the campus community not only requested but also intimately shaped. From inception to completion, student and faculty preferences were integrated with expert knowledge to deliver a space tailored to serve the entire campus community.

We spoke with Dean Mavrinac to learn more about the process and impact of the project. She wanted to underscore that the success, to date, of Studio X is a team effort, much of it led by Digital Scholarship and Studio X director, Emily Sherwood.

Studio X salon area. Shows students sitting or standing near the entrance of Studio X.

There aren’t many academic libraries that offer a space like Studio X. What is it, and how did the project begin?

The project began in fall 2017 when Lauren Di Monte joined our team and learned from the faculty that there was a lot of research activity in extended reality and other immersive technologies. We thought it was something the library could get involved in since we had close to 50 researchers engaged in XR technologies. So, we set out to better understand that landscape and how the researchers would engage with any initiative we developed, whether it was a space or specialized expertise. We knew a generic cave wouldn’t work for them, so we thought about what we may be able to do to help them tackle specific research questions. As it turned out, we pivoted to a space and service that would provide an easy on-ramp to those less familiar with these technologies and related needs.

Today, Studio X is a collaborative hub for extended reality where students and faculty are immersed in learning and teaching in ways that just aren’t possible without advanced technology. It’s a high-tech space that allows exploration, experimentation and experience that truly brings education to life.

What was the goal of Studio X? Who is it for?

The overall goal was to offer physical space, a program, services, technology and expertise that students and faculty needed—and expertise was really big. The user research told us that they wanted a space and experts in the space to teach them how to use and apply the technologies. We approached this goal by providing an on-ramp that made it easy for people to gain access to and experience with XR technologies.

Whether a person is an advanced researcher or a novice user, we’re good at helping people feel comfortable to explore their questions. The library is an interdisciplinary crossroads at the university, so it could be someone studying history, biomedical engineering, neuroscience, religion, ethics—whatever it is—if they’re interested in using XR technologies, we provide the support they need to feel welcome.

Read the full interview.

Voices of XR: Scott McEldowney

voices of xr speaker, Scott McEldowney.

“Optics of Virtual Reality”

voices of xr speaker, Scott McEldowney.

Scott McEldowney, Ph.D. is a Director, Research Scientist for Reality Labs Research, Meta in Redmond, WA. His research spans areas from novel imaging systems to exploration of emerging concepts in near to eye display. Scott was the founding member of the optics research team within Reality Labs. He built and lead a team of researchers conducting advanced research in augmented and virtual reality display systems.  Prior to Reality Labs Research, Scott was a Principal Optical Engineer for Microsoft incubating new hardware technologies. While at Microsoft, Scott helped develop and launch new 3D sensing platforms (Kinect) and was a core member of the team that developed the optic architecture for the first version of Hololens. Prior to Microsoft, Scott worked for 18 years in the Advanced Optical Technology division of JDSU developing optical components and systems for consumer electronics, medical instrumentation, and aerospace. Scott holds a MS degree in Mechanical Engineering and PhD in Optical Sciences from the College of Optical Sciences at the University of Arizona.

This presentation will give an overview of Reality Labs (RL) Research, Meta and introduce many of the key elements necessary to build a successful augmented reality (AR) or virtual reality (VR) system, focusing on optics and displays. He will present recent advances at RL Research in near-to-eye display systems in aspects such as resolution, field of view, image quality, form factor, accommodation. Making progress requires not just uncovering new technological building blocks but also developing perceptual test vehicles to establish the benefits, limitations, and requirements for future VR and AR optics and displays. 

The Voices of XR speaker series  is made possible by Kathy McMorran Murray and the National Science Foundation (NSF) Research Traineeship (NRT) program as part of the Interdisciplinary Graduate Training in the Science, Technology, and Applications of Augmented and Virtual Reality at the University of Rochester (#1922591).

Where: In person in Studio X & Virtual on Zoom
When: Friday, April 15th from 1 to 2pm EST
Register: bit.ly/VoXR415


graphic for XR speaker series entitled Voices of XR. On the bottom, is an illustration of a person in a headset reaching with neon geometric squares in the background. On the top is text that reads: "Voices of XR: A Studio X Speaker Series." On top of that, is the Studio X and River Campus Libraries wordmarks.Voices of XR is a Studio X speaker series. Speakers are scholars, artists, and extended reality professionals who discuss their work with immersive technologies across disciplines and industries. All talks are free and open to the general public.

Build Your Own VR Flappy Bird in Unity

promotional image for workshop. Shows a person in a VR headset playing flapping bird. Text reads, "Create Your Own VR Flappy Bird"

RESCHEDULED!!!

Metaverse? Waste of time. VR Flappy Bird wherein you are the bird? That’s more like it. Come join us in learning what the future of VR truly looks like in our step-by-step workshop on how to make your own VR Flappy Bird game in Unity, a real-time creation platform! Unity is the industry-standard engine behind all kinds of experiences such as Pokémon Go, Beat Saber, and the new Lion King movie. 

Join Studio X, UR’s hub for immersive technologies, and learn more about the digital world of extended reality (XR). All levels welcome. No experience necessary!

promotional image for workshop. Shows a person in a VR headset playing flapping bird. Text reads, "Create Your Own VR Flappy Bird"

Note: In order to participate, you will need to complete the pre-workshop instructions, which will be sent by email prior to the event. Need assistance with this process? Ask for help on the Studio X Discord (Quick Questions Channel). 

Instructor: Muhammed El-Sayed, Liam O’Leary, Hao Zeng
Where: Studio X, Carlson Library First Floor
When: Wednesday, 4/6 from 6 to 7:30pm
Register: bit.ly/VRFlappyBird

Voices of XR: David Chalmers

speaker David Chalmers.

“From the Matrix to the Metaverse”

speaker David Chalmers.

David Chalmers is a leading philosopher of consciousness and philosophy of the mind. He is Professor of Philosophy and Neural Science at New York University. He is the Co-Director of the Center for Mind, Brain, and Consciousness. He is also Honorary Professor of Philosophy at the Australian National University and Co-Director of the PhilPapers Foundation. He studies the foundations of cognitive science, physics, and technology, as well as philosophy of language, metaphysics, and epistemology, and many other areas.

He will discuss his new book, Reality+, which focuses on the philosophical implications of virtual and augmented reality. He argues that virtual worlds are not second-class worlds and that we can live a meaningful life in virtual reality.

The Voices of XR speaker series  is made possible by Kathy McMorran Murray and the National Science Foundation (NSF) Research Traineeship (NRT) program as part of the Interdisciplinary Graduate Training in the Science, Technology, and Applications of Augmented and Virtual Reality at the University of Rochester (#1922591).

Where: In person in Studio X & Virtual on Zoom
When: Wednesday, April 6th from 12 to 1pm EST
Register: bit.ly/VoXRDavidChalmers

Recording

Event recording available for UR faculty, students, and staff with NetID and password: bit.ly/XRDavidChalmers


graphic for XR speaker series entitled Voices of XR. On the bottom, is an illustration of a person in a headset reaching with neon geometric squares in the background. On the top is text that reads: "Voices of XR: A Studio X Speaker Series." On top of that, is the Studio X and River Campus Libraries wordmarks.

Voices of XR is a Studio X speaker series. Speakers are scholars, artists, and extended reality professionals who discuss their work with immersive technologies across disciplines and industries. All talks are free and open to the general public.

Voices of XR: Muhsinah Morris

speaker Muhsinah Morris, Ph.D.

VR Pioneer in the Higher Education Metaverse

speaker Muhsinah Morris, Ph.D.

Muhsinah L. Holmes Morris, Ph.D., is the Director of “Morehouse in the Metaverse,” a VR Project at Morehouse College and Assistant Professor of Education for the Morehouse Center for Excellence in Education (ECEE). She is an alumna of Clark Atlanta University (BS) and Emory University (MS, PhD). She teaches her chemistry courses in VR on Morehouse’s digital twin campus created by VictoryXR on the Engage Platform.

Her research encompasses working in the Morehouse Makerspace Exploration Center, 3D Printing Specialized Laboratory Equipment for those with Autism and other Developmental Disorders. She is the PI of the Morris Research and Innovation Group, where they research and develop technologically innovative solutions for those with autism. Dr. Morris won the 2021 Vulcan Teaching Award of Excellence and launched her Advanced Inorganic Chemistry course in virtual reality in the spring of 2021 in the digital twin campus. She is a pioneer in this space as the chemistry content in VR is limited to non-existent. Wife to a gamer and mom to five sons, she is an inventive autism mom, a volunteer advocacy ambassador, National Community Advisory Council member, and grant review committee member for Autism Speaks. Her purpose in life is to create inclusivity in STEAAM (science, technology, engineering, arts, agriculture, and math). She believes that VR provides a pathway for creating that inclusion through immersive education, vocational rehabilitation services, and therapeutic experiences.

The Voices of XR speaker series  is made possible by Kathy McMorran Murray and the National Science Foundation (NSF) Research Traineeship (NRT) program as part of the Interdisciplinary Graduate Training in the Science, Technology, and Applications of Augmented and Virtual Reality at the University of Rochester (#1922591).

Where: In person in Studio X & Virtual on Zoom
When: Friday, March 25th from 1 to 2pm EST
Register: bit.ly/VoXRMuhsinahMorris

Recording


graphic for XR speaker series entitled Voices of XR. On the bottom, is an illustration of a person in a headset reaching with neon geometric squares in the background. On the top is text that reads: "Voices of XR: A Studio X Speaker Series." On top of that, is the Studio X and River Campus Libraries wordmarks.

Voices of XR is a Studio X speaker series. Speakers are scholars, artists, and extended reality professionals who discuss their work with immersive technologies across disciplines and industries. All talks are free and open to the general public.