Work proceeding on low-cost body trackers

Sean Jasin, an Electrical Engineering major, along with his fellow student Luke Liu, has been working on a new, low-cost solution for full body tracking in VR. This will help us to integreate more of the body into our movement explorations with the movement games in Master Dancer.

This video short shows the data coming from the sensor into the computer via WiFi. The cable that connects the breadboard to the computer is there simply to provide power, which eventually would be supplied via a small li-po battery:

https://youtube.com/shorts/pq5Mw_jIMEY

Master Dancer at DRHA 2023

Professor Toenjes took the Master Dancer demo app as it stands now to the Digital Research in Humanities and the Arts 2023 conference at the University of Turin, Italy. The team worked very hard to get the app working in time for the conference in early September. The app was presented every day at the conference for two hours, and happy to say, it ran flawlessly the whole time!

Performing Cultural Heritage in the Digital Present

A person using a VR headset in the library at the University of Torino, Italy.

Fig.1 – A person using a VR headset in the library at the University of Turin, Italy during the 2023 DRHA conference.

The title of the conference was “Performing Cultural Heritage in the Digital Present.” The Master Dancer app fit perfectly into this theme, as Master Dancer focuses on the dancer Loie Fuller as its main character. The app is delivered to the headset via a cable from a PC, and the University of Turin generously provided a powerful laptop for me to use.

Below is a video of the current version of the app as seen through the headset. We are currently in discussion with Dan Cermak and his team of student developers at the ST/UDIO to work on another revision to the app based upon feedback received while watching people engage with this version. This will be detailed in another post…

Master Dancer VR project underway

With the help of a grant from the University of Illinois Research Board, U of I Dance Professor John Toenjes has been able to work with a talented team of students to begin creating a Virtual Reality dance adventure game tentatively called “Master Dancer.” Not only will the game itself be used to reach out to other communities, but the technologies themselves used in developing this game are also going to be used to reach out to the community around the university. For example, Professor Toenjes is partnering with the HipHop Express project to take the cutting-edge motion capture suits used to create dance animations into underserved communities to expose them to, and excite them about, the possibilities of new technologies for cultural expression. This project is supported by the U of I Research Board, the Grainger Engineering Library IDEA Lab, and the Siebel Center for Design.

A playlist of videos of the project is viewable by clicking HERE.

Detailed Description of the Master Dancer Project

Much is being written about the centrality of computer games to contemporary human experience. In Reality is Broken,[1] author Jane McGonigal presents staggering statistics of the number of game players in contemporary life: even in 2011, 3 billion hours per week were spent gaming world-wide. As she writes, “Games, in the twenty-first century, will be a primary platform for enabling the future…[our] neurological and physiological systems—our attention systems, our reward center, our motivations systems, our emotion and memory centers—are fully activated by gameplay.”[2] Gamification of dance and the other performing arts can promote deeper involvement, understanding and appreciation through direct interaction with these art forms, and gives the creator new ways to directly affect the audience’s attitudes and feelings towards issues being dealt with in contemporary performance.

In order create art that is also a game, the artist must ask themselves: what goal can I construct for my performers to reach? How do I construct rules that make the game compelling and give it structure? What is the feedback system or systems that the performers and/or audience will rely upon to feel like the art is progressing along a line of inquiry? Lastly, how do I get the audience to want to participate?

A technology that is akin to live performance is Virtual Reality, which has an immersive quality like the experience of being in a theater with the house lights out and the theater brightly lit. But VR has the interactive capabilities of a game controller, which makes it a ripe technology for just the kind of game/art integration described above. Therefore, I have embarked on a quest to make art/games in Virtual Reality.

Description of “Master Dancer”

Master Dancer is just starting to be under development. The goals of the Master Dancer game are: 1) to instruct users in some basic movement energies and then to get them to use those new skills to move in their own creative way, and 2) to be exposed to some historical dance information. The movement energies are based upon Laban Movement Analysis principles, but they are more generalized than the eight effort actions. (The video example below still uses the Laban term “slash,” but the finished version will remove that in lieu of the more general term “directed forceful movement.”)

The basic movement energies are taught through movement games that are scored and these scores compared to those of other users. The motivation to continue in the game is in the story, as follows. The famous dancer Loïe Fuller (1862-1928) is found in an old, decrepit theater. She asks the user to help her restore the theater and her dancing abilities by becoming a better dancer. This is accomplished through training in the movement energies by playing action games. After the user finishes each training game in a “training theater,” they re-enter the theater lobby to find the Loie is a little younger, and the theater a little brighter. They are asked to dance a creative improvisation based upon the skill they just learned, and this dance is recorded in movement trails—a “living 3D sculpture.” This living sculpture will then appear on the lobby pedestal (seen behind Loïe in the opening), an elaborate sculpture eventually constructed of movement trails of different colors and types based upon the energies of the different movements required to master the dance skill games. Once all the dance skills are mastered, a final dance with a re-invigorated Loïe in the fully restored theater is the reward.

In the lobby, playing on the jukeboxes at the entrance to each game, are videos of dance types that are made up primarily of that dance energy, as well as short biopics of dancers or dances that are based largely on that type of movement energy. This is in service of the goal of imparting historical knowledge through playing this game.

The two movement energies that have games currently in later stages of development are “directed forceful movement” and “light, quick movement.” The supplemental video shows the onboarding of the game, the theater (with a mannequin avatar at the moment, to be replaced soon with a Loie-like avatar) lobby and the beginnings of a greeting by Loie, and the game-in-progress that is designed to get users moving with a directed, forceful movement. What is not shown in the video is the actual avatar of Loie, the second game of “light, quick movement,” and the “live sculpture” that each user makes by the trails of their movements in virtual space.


[1] McGonigal, Jane Reality is Broken: How Games Make Us Better and How They Can Change the World. NY: Penguin Books 2011

[2] McGonigal, p. 30

LAIT is now moving into Virtual Reality

Starting in the Spring of 2021, LAIT Director Professor John Toenjes began a new chapter by starting to work on a Virtual Reality/360-degree video dance adventure. This adventure was developed in the Unity3D engine and shot with dancers in Prof Toenjes’ “ChoreoLab” class at the University of Illinois.

It is still under development, but some of the content can be viewed here. This is the final dance of the project, shot in 360. You can watch it with a headset, or scroll around with a mouse.

We will be posting more material soon that describes the entire project.
#360video #LAIT

Lait to be part of new IPRH Research cluster

Mosho will be used to support a new Illinois Program for Research in the Humanities “Research Cluster” by LAIT founder John Toenjes and University of Illinois Mechanical Engineering Professor Amy Laviers who runs the RAD lab. The title of the cluster is “Understanding Movement Style and Social Interactions Through Participatory Performance: Development of An Interactive Performance Concept for Research On Style Preferences.” The idea is to create a theatrical performance where the audience will participate in interacting with each other and with robots, to research movement styles and how those styles are affected by different types of interaction with other entities, whether human or mechanical.

We are just in the brainstorming and planning stages for this performance. So we held a “mixer” for faculty and students interested in this topic, and who might be interested in getting involved. About 10 people showed up, with a range of interests and talents. We had a wonderful discussion about what this performance might be, and many of the attendees were interested in helping out. We have a range of talents, from coding to story writing to hardware design to music. We’re excited to work with this team and hope to involve more people as well.

“LAIT” is now “Mosho”

Featured

The LAIT app is now called “Mosho” which stands for “Mobile Show.” This name change comes about as the LAIT application system has evolved and been re-written to the extent that it is time to christen it with a better name. The feature list is quite extensive now, and it is now being used by other dance and theater companies. For example, Erica Mott and Dancers used Mosho in their production of “Mycelial” at the 2018 Bates Dance Festival, and the Know Theater of Cincinnati used it in their production of Ada & the Machine in April 2018.

We also have a beautiful new logo, designed by Ran Tao. You can search the Google Play Store and Apple App Store for Mosho the next time you attend a Mosho-enhanced performance.

Alternate Reality performance at UC-Irvine

INTERFACE: Alternate Reality is a new work-in-progress that had its first incubation period during a three-week residency as a 21C Seed project at the Institute for 21st Century Creativity at the UC-Irvine Claire Trevor School of the Arts. This work builds upon our earlier works which use the LAIT mobile application system for live performance, Critical Mass and Public Figure, both of which seek to use the mobile phone as a means of increasing audience engagement with dance as an art form and theatrical experience. Alternate Reality is a work that incorporates game culture and game thinking into the conception of live dance performance. It seeks to find ways to modernize dance performance to be in line with contemporary culture that is heavily influenced by first-person user interaction and competitiveness and the goal-orientated thinking which that promotes. Many people, particularly young audiences, are used to taking an active role in the direction and outcomes of their entertainment activities. They’re used to using game controllers that help them navigate game structures. With the capabilities of our LAIT-based app, now called “Mosho”, we have been able to craft a live-action game structure that incorporates contemporary dance.

We crafted a plot where a ship (undefined: could be a spaceship, or a sailing vessel) had hit a maelstrom which scattered her crew into many different places and times, and now they all were existing resulting in alternate realities. The captain of the ship was looking to reunite them, and enlisted the help of the audience “players” to locate them and bring them all back to one reality. To reunite the crew, the players had to solve various puzzles that would yield clues, that would lead players to discover keywords by scanning AR target images. These words were then used to complete phrases about “reality” which had come from three famous personages: Albert Einstein, John Lennon, and the film director Tim Burton. The audience used the Mosho app to find the various locations of the crew, to solve the puzzles that were found at each game level, to scan the AR target images to uncover the keywords, and then text those keywords to the captain.

Below is video documentation of the work-in-progress showing. For even more in-depth writing about this production, visit Prof. Toenjes’ new CV site page about Alternate Reality.

Student reactions to Critical Mass performance

My colleage at the University of Illinois shared her beginning dance student responses to Critical Mass. They are quite enlightening as to the effect that this technology has on young audiences. Read on…

• The cell phone app was the most interesting piece of technology used in my opinion. I had never heard of dancer’s using a phone app to guide the performance and to allow you to act and react to the story line. I thought in some ways it could be distracting because of all of the sounds and images that popped up throughout the show BUT at the same time forced the audience to pay attention. I think many times people wander off into their cell phones during performances which is very annoying. What I noticed was most of the audience was drawn in because of the use of technology. They wanted to see what would happen during the performance, they wanted to see how the app reacted, and they could almost have cared less about their Facebook, twitter, Instagram and other modes of social media. Overall, I think the phone app was a success for that particular dance and improved the audience’s interest… I would have also liked to see more perspectives on social media. One area of interest for me, as a psychology major, is how much social media influences psychopathology. How has depression, anxiety, bi polar, Cyclothymia, dysthymia etc. rates increased with social media and what does that mean for our society? Perhaps I would have added a dancer that viewed social media as a means of creating illness not wellness. — Allyson B.

• The concept of social media and nuclear energy seemed fairly clear to me, or at least much more clear than the remaining 3 acts, most of which offered very little explanation. The first act had a lot of ideas that were executed extremely well, and the inclusion of the app really helped drive the point home. The audience choosing their favorite duet is more than just interesting in the sense that it engages the audience, it also seems to be referencing the judgmental nature of social media and showing us what people go through in order to be liked…It was a super cool performance though, and I really like what you came up with. I agree that the use of technology made the performance much more engaging, or at least novel. I remember much more of this performance than the other three in the show, even though it was the first one. — Nick O.

• The nuclear metaphor relating a radioactive meltdown to the breaking point or critical mass of Internet consumption of an idea or concept was quite astonishing. At all times I was drawn to the fabulous lighting, unique stage design, and thought-provoking monologues coming from my phone. To say that it was innovative does not do it justice. I am so completely blown away by the performance that I want to take the time to praise in explicitly in the first part of my reflection. I had never seen a dance performance like this, but it is safe to say that I am now a lifelong fan of contemporary dance because of ‘Critical Mass’. Going back to the concept being transferred to the stage, I appreciated the towering, almost god-like presence of the cube. The humor that was sprinkled in intermittently over the course of the show allowed for the critical mass metaphor to be presented without being too ‘overindulgent’ is the word that comes to mind. Without the humor, the good and bad of Internet consumption of social media domination would have been more difficult to ascertain.
The aesthetics of the performance, to me, we the electronic lighting that simulated the type of light that comes from a smartphone. The cube itself, flashing various funny picture and nuclear images, presented a strange but accessible format by which the audience received the information. The app, although it crashed many times during the performance, forced me to become fully immersed in the show. The initial immersion, in addition, primed me to pay closer attention to the following shows, elevating my enjoyment of them as well. The wonderful thing about the show, I feel, is that it is a good introduction to contemporary dance. — Jake P.

• Critical mass combined cell phone with the dance. The cellphone app was directly connected to the screen on the cube that people can post their photos and messages onto it. Another use of the app is to vote for the dance that audience want to see. Also we can hear the dancer’s thoughts about the theme of the dance –social media. The cube on the stage was also very novel, it had a door to let the dancers hide inside and come out, and each side of the cube had screens connected to the app. When the cube rotated, people can see different parts on the app such as images or dancer’s interviews. Critical mass to me is like a nuclear center, and the dancers were like the splits of the center. The dancers wore security clothes to represent the danger of the nuclear, and the app fully represented the social media. This dance is very unique and I think it has a bright future. Cell phones, ipads are inseparable parts in life, the combination of these technologies with dance could be the main stream in the future. — Jiayue S.

• The only piece that felt truly alive for my money was the first performance, “Critical Mass”. Prof. Toenjes’ piece offered an experience that was thought-provoking and consistently engaging throughout its entirety. Its paranoid setting was brought to life through the use of the costumes on the dancers that looked like they were in a radioactive area, through the echoes of glitch and static that would erupt at seemingly random intervals from phones in the audience, and from the chaos of having so many different elements happening at once, very reminiscent of the Mark Morris dances where the stage was filled with performers. — Matt N.

LAIT-based “CM2=E” app at the West Hollywood Dance Festival

LAIT was used at the 2017 West Hollywood Dance Festival to deliver interactive program notes to the audience. The app took the place of paper programs and displayed the title of the dance, the dance company, and other information about the dance that the choreographers wanted the audience to know. The audience could move freely through the program because LAIT now has quite functional buttons and menus that make it operate more like a typical app on a smartphone. This lets the app function on its own without the need for the stage manager to trigger cues all the time. This was a very successful use of LAIT in a different way for live performance.

Here are a few screenshots to give you an idea of how it worked:

WeHo program app opening screen

WeHo program app opening screen

WeHo screen 2

WeHo feature overview

WeHo program overview

WeHo program overview – interactive navigation of the program

WeHo program app dance detail

WeHo program app – dance detail