Sensory Processing – in a Virtual Kodak Hall

a binaural microphone set up with a dummy head.

Rochester researchers will harness the immersive power of virtual reality to study how the brain processes light and sound.

A cross-disciplinary team of researchers from the University of Rochester is collaborating on a project to use virtual reality (VR) to study how humans combine and process light and sound. The first project will be a study of multisensory integration in autism, motivated by prior work showing that children with autism have atypical multisensory processing.

The project was initially conceived by Shui’er Han, a postdoctoral research associate, and Victoire Alleluia Shenge ’19, ’20 (T5), a lab manager, in the lab of Duje Tadin, a professor of brain and cognitive sciences.

“Most people in my world—including most of my work—conduct experiments using artificial types of stimuli, far from the natural world,” Tadin says. “Our goal is to do multisensory research not using beeps and flashes, but real sounds and virtual reality objects presented in realistically looking VR rooms.”

UR students working on the project are looking at information on a laptop with Kodak Hall in the background.
Members of the team begin the setup for audio and visual data collection. From left to right are Shui’er Han, a postdoctoral research fellow in Duje Tadin’s lab; brain and cognitive sciences major Betty Wu ’23; computer science and business major and e5 student Haochen Zeng ’23, who works in River Campus Libraries’s Studio X; and Victoire Alleluia Shenge ’19, ’20 (Take Five), who earned her degree in brain and cognitive sciences and is a manager in Tadin’s lab.

A cognitive scientist, a historian, and an electrical engineer walk into a room . . .

Tadin’s partners in the study include Emily Knight, an incoming associate professor of pediatrics, who is an expert on brain development and multisensory processing in autism. But in creating the virtual reality environment the study participants will use—a virtual version of Kodak Hall at Eastman Theatre in downtown Rochester—Tadin formed collaborations well outside his discipline.

Faculty members working on this initial step in the research project include Ming-Lun Lee, an associate professor of electrical and computer engineering, and Michael Jarvis, an associate professor of history. Several graduate and undergraduate students are also participating.

Many of the tools they’ll use come from River Campus Libraries—in particular, Studio X, the University’s hub for extended reality projects, as well as the Digital Scholarship department. Emily Sherwood, director of Studio X and Digital Scholarship, is leading the effort to actually construct the virtual replica of Kodak Hall.

The group recently gathered in the storied performance space to collect the audio and visual data that Studio X will rely on. University photographer J. Adam Fenster followed along to document the group’s work.

Read more.

Teaching Blender

Emma Schaale, Karp Library Fellow

Karp Library Fellows Experiences with Studio X

Emma Schaale, Karp Library Fellow

By Emma Schaale
Karp Library Fellow, Public Programs Coordinator for Studio X
June 2020 – October 2020
Area of Focus: Varies between considering branding for Studio X, creating promotional materials, and conceptualizing workshop for Studio X’s fall programming

For the past month, I’ve been working on an exciting project… a Blender workshop for Studio X, which took place on Wednesday, September 23rd, 2020! This came about as a collaborative effort between Studio X and Tinkerspace to create introductory programming on immersive technologies for the UR community.

We first prioritized workshops for Unity, a video game development environment, due to faculty and student interest in this platform. This was not my area of expertise, as I primarily work in Unreal Engine 4, a similar but different development tool. Luckily, Sebastian Jakymiw, the Karp Library Fellow Immersive Technologies Developer for Studio X, has worked extensively with Unity and was able to take on the lead instruction for those workshops. However, I participated in the conversations between him; Meaghan Moody, Studio X’s Immersive Technologies Librarian; and Allegra Tennis, STEM Librarian.

Because students and faculty are also interested in 3D content creation, Meaghan and I started brainstorming a workshop focused on 3D modeling. Having completed recent coursework (CSC131: Recreational Graphics) for 3D modeling as well as having created my own personal gaming projects, I felt comfortable teaching a workshop on Blender, which is a popular, free, and open-source 3D creation software.

One of the most important considerations we had in our conversations is how to brand our programming. This is crucial for building an image and personality for Studio X, so that a community can form around it as well as pique students’ interest. We wanted these workshops to grab students’ attention and spark their creativity. Immersive technologies are still very new and might seem complicated to students. For example, if you’ve never heard of Blender before, an “Intro to Blender” workshop may seem irrelevant or uninteresting to you. While a “Create a 3D Animal with Blender” workshop might just catch your eye.

To prepare for this workshop, my first step was to scout YouTube for existing 3D modeling tutorials. Some of my favorites are by Grant Abbitt, who has hundreds of tutorials and workshops posted to his channel. I was inspired, in particular, by this video on creating low-poly animals and decided to use to use this as the basis for the workshop.

While working on the outline for my workshop, I considered how to make the process more interactive. During a YouTube video, the user can pause or speed up the instruction, but for a live session, I would need to ensure that my students have time to see everything on the screen. For new users, Blender’s user interface can be especially overwhelming–especially when you’re learning it virtually!

This was also mentioned during a practice workshop, which I ran for Meaghan, Allegra, and Sebastian. Allegra advised that I should take more time to make sure attendees can follow my cursor on the screen. I related to this frustration based on my own experiences learning new technologies in the classroom, so it was intuitive to incorporate this.

screenshot of Blender interface with a 3D modeled cat from the workshop test run.
Screenshot from the Blender practice workshop

During this practice run, it was great to hear that my trial participants found my tutorial to be a great introduction. It’s a difficult software to teach to first-time users, but I was satisfied to hear that they had a fun time learning.

During the actual workshop, I had quite a fun experience introducing the class to Blender’s user interface. Before diving in though, I described how this platform is used for all kinds of projects such as animated films, video games, and virtual reality experiences. I also broke down some basic terminology and concepts. For example, I explained that a mesh is composed of vertices, edges, and faces.

I also explained that Blender’s user interface can seem even more intimidating than the user interface of software like Adobe Premiere Pro or Photoshop. I demonstrated how to perform basics edits in Object and Edit Mode, how to trace along a reference image using extrusions and edits in Edit Mode, how to apply color to the model, and finally, how to export the model.

screenshot of my desktop from the workshop. On the left, Blender is open, and the author is editing a model. On the right, are attendees Zoom windows.
Screenshot from the official workshop on August 23, 2020

Perhaps the most important advice I received was from Allegra, who suggested I really slow down my instruction. At times, I felt like I was waiting hours in between steps while hovering my mouse over a button or demonstrating keyboard shortcuts. Yet, a couple of times, students asked me to repeat these actions. I learned that going slowly is crucial while teaching a complex software like Blender, especially in a virtual teaching environment.

My previous experiences teaching at an all girl’s tech camp in 2018 and as a Publicity Manager for the UR Game Dev Club have made me a much more comfortable instructor. Certainly, having these experiences made me more understanding of the ways others learn and how to make it a comfortable learning environment. While I would say my Blender workshop went well, I’ve had many failures in the past that I have had to learn from. I was even comfortable joking around, which would have been unimaginable a couple of years ago!

Overall, I enjoyed myself during this entire process. I love teaching things I care about, and building this workshop from the ground up was a wonderful experience. I was supported by Studio X, Meaghan, and Allegra, who made me feel more comfortable and confident teaching to my peers. So thank you to Studio X for making this possible!

Are you interested in learning Blender? Check out my workshop slides! Stay tuned for more workshops, and here are some of my favorite resources in the meantime:

Student Survey Findings

During the month of July 2020, Studio X staff administered a survey to undergraduate and graduate students at the University of Rochester to better understand their needs and interests regarding immersive technologies. These findings were used to inform Studio X’s fall 2020 pilot programming. As of the close of the survey in late July 2020, 58 students responded to the survey.

Q1: Please indicate your familiarity with the following tools:

On average, a majority of students are unfamiliar or have only heard of common XR tools and platforms. A minority of students have experienced deeper engagement by either creating something or teaching others.

Q2: Do you own a VR headset?

Few students own VR headsets. It may be reasonably assumed that they remain cost-prohibitive.

Q3: Given your experience with XR, how might you anticipate needing support to develop an XR project?

Q4: Which skill or tool are you most interested in learning?

Q5: What XR platforms or projects interest you? Can you imagine using XR to enhance your own research and learning? If so, how?

In general, students responses were very specific, but a majority of them fall under the STEM disciplines. For example, students envisioned engaging with XR for physical reasoning, machine learning, electrical design, 3D audio, etc.

Many students also expressed uncertainty about how they might use XR technologies but were very enthusiastic about developing a new skillset and gaining hands-on experience.

I’m not really sure, but I want to learn more about it so I can think about how to use it.

This response stood out, in particular, because it underscores the value of exposure not only to the tools and methods but to real examples and use cases. New XR users require opportunities to make and experiment but also to see how others are leveraging these technologies to grasp the possibilities.

Q6: Is there anything else we should keep in mind?

Students’ concerns, questions, and suggestions

icon of a shield with cross in the center of it

COVID- 19
Provide virtual workshops & prioritize hygiene and sanitation of equipment

icon of a teacher and students.

Beginner Friendly Options
Demystify XR & cover the basics of tools and methods

icon of a lightbulb hovering over an open box.

Keep an Open Mind
Provide equal access for all students & create “noneducational” opportunities

icon to represent collaboration.

Involve us!
“I would love to help!” & create peer-teaching opportunities

icon of a star.

Stand Out from the Crowd
Campus is over programmed & distinguish your space and services from Rettner

icon of a wrench.

Practical Advice
VR cables are annoying and Carlson can be out of the way for students

Samuel Henderson, XR Specialist

Samuel Henderson, XR Specialist standing in front of some foliage.
Samuel Henderson, XR Specialist standing in front of some foliage.

Samuel is a sophomore majoring in Computer Science who enjoys creating apps and games using Unity Engine. He is an international student from Zimbabwe. His hobbies include playing chess, fishing, hiking, playing video games, and he makes a delicious naan bread.