Renowned dance artist and Loïe Fuller expert Jody Sperling is visiting this week, teaching workshops and helping with the “Master Dancer” VR app development. We have been working on real-time control of avatars in Unreal Engine, mocap for “AI Loïe, and kicking the tires on the real-time AI video generator programmed by Jake Metz at the SCIM lab in the UIUC Engineering Library. Exciting work!
Experimenting with AI generated video in SCIM lab.
Sean Jasin, an Electrical Engineering major, along with his fellow student Luke Liu, has been working on a new, low-cost solution for full body tracking in VR. This will help us to integreate more of the body into our movement explorations with the movement games in Master Dancer.
This video short shows the data coming from the sensor into the computer via WiFi. The cable that connects the breadboard to the computer is there simply to provide power, which eventually would be supplied via a small li-po battery:
Professor Toenjes took the Master Dancer demo app as it stands now to the Digital Research in Humanities and the Arts 2023 conference at the University of Turin, Italy. The team worked very hard to get the app working in time for the conference in early September. The app was presented every day at the conference for two hours, and happy to say, it ran flawlessly the whole time!
Fig.1 – A person using a VR headset in the library at the University of Turin, Italy during the 2023 DRHA conference.
The title of the conference was “Performing Cultural Heritage in the Digital Present.” The Master Dancer app fit perfectly into this theme, as Master Dancer focuses on the dancer Loie Fuller as its main character. The app is delivered to the headset via a cable from a PC, and the University of Turin generously provided a powerful laptop for me to use.
Below is a video of the current version of the app as seen through the headset. We are currently in discussion with Dan Cermak and his team of student developers at the ST/UDIO to work on another revision to the app based upon feedback received while watching people engage with this version. This will be detailed in another post…
With the help of a grant from the University of Illinois Research Board, U of I Dance Professor John Toenjes has been able to work with a talented team of students to begin creating a Virtual Reality dance adventure game tentatively called “Master Dancer.” Not only will the game itself be used to reach out to other communities, but the technologies themselves used in developing this game are also going to be used to reach out to the community around the university. For example, Professor Toenjes is partnering with the HipHop Express project to take the cutting-edge motion capture suits used to create dance animations into underserved communities to expose them to, and excite them about, the possibilities of new technologies for cultural expression. This project is supported by the U of I Research Board, the Grainger Engineering Library IDEA Lab, and the Siebel Center for Design.
A playlist of videos of the project is viewable by clicking HERE.
Detailed Description of the Master Dancer Project
Much is being written about the centrality of computer games to contemporary human experience. In Reality is Broken,[1] author Jane McGonigal presents staggering statistics of the number of game players in contemporary life: even in 2011, 3 billion hours per week were spent gaming world-wide. As she writes, “Games, in the twenty-first century, will be a primary platform for enabling the future…[our] neurological and physiological systems—our attention systems, our reward center, our motivations systems, our emotion and memory centers—are fully activated by gameplay.”[2] Gamification of dance and the other performing arts can promote deeper involvement, understanding and appreciation through direct interaction with these art forms, and gives the creator new ways to directly affect the audience’s attitudes and feelings towards issues being dealt with in contemporary performance.
In order create art that is also a game, the artist must ask themselves: what goal can I construct for my performers to reach? How do I construct rules that make the game compelling and give it structure? What is the feedback system or systems that the performers and/or audience will rely upon to feel like the art is progressing along a line of inquiry? Lastly, how do I get the audience to want to participate?
A technology that is akin to live performance is Virtual Reality, which has an immersive quality like the experience of being in a theater with the house lights out and the theater brightly lit. But VR has the interactive capabilities of a game controller, which makes it a ripe technology for just the kind of game/art integration described above. Therefore, I have embarked on a quest to make art/games in Virtual Reality.
Description of “Master Dancer”
Master Dancer is just starting to be under development. The goals of the Master Dancer game are: 1) to instruct users in some basic movement energies and then to get them to use those new skills to move in their own creative way, and 2) to be exposed to some historical dance information. The movement energies are based upon Laban Movement Analysis principles, but they are more generalized than the eight effort actions. (The video example below still uses the Laban term “slash,” but the finished version will remove that in lieu of the more general term “directed forceful movement.”)
The basic movement energies are taught through movement games that are scored and these scores compared to those of other users. The motivation to continue in the game is in the story, as follows. The famous dancer Loïe Fuller (1862-1928) is found in an old, decrepit theater. She asks the user to help her restore the theater and her dancing abilities by becoming a better dancer. This is accomplished through training in the movement energies by playing action games. After the user finishes each training game in a “training theater,” they re-enter the theater lobby to find the Loie is a little younger, and the theater a little brighter. They are asked to dance a creative improvisation based upon the skill they just learned, and this dance is recorded in movement trails—a “living 3D sculpture.” This living sculpture will then appear on the lobby pedestal (seen behind Loïe in the opening), an elaborate sculpture eventually constructed of movement trails of different colors and types based upon the energies of the different movements required to master the dance skill games. Once all the dance skills are mastered, a final dance with a re-invigorated Loïe in the fully restored theater is the reward.
In the lobby, playing on the jukeboxes at the entrance to each game, are videos of dance types that are made up primarily of that dance energy, as well as short biopics of dancers or dances that are based largely on that type of movement energy. This is in service of the goal of imparting historical knowledge through playing this game.
The two movement energies that have games currently in later stages of development are “directed forceful movement” and “light, quick movement.” The supplemental video shows the onboarding of the game, the theater (with a mannequin avatar at the moment, to be replaced soon with a Loie-like avatar) lobby and the beginnings of a greeting by Loie, and the game-in-progress that is designed to get users moving with a directed, forceful movement. What is not shown in the video is the actual avatar of Loie, the second game of “light, quick movement,” and the “live sculpture” that each user makes by the trails of their movements in virtual space.
[1] McGonigal, Jane Reality is Broken: How Games Make Us Better and How They Can Change the World. NY: Penguin Books 2011
[2] McGonigal, p. 30
The LAIT app is now called “Mosho” which stands for “Mobile Show.” This name change comes about as the LAIT application system has evolved and been re-written to the extent that it is time to christen it with a better name. The feature list is quite extensive now, and it is now being used by other dance and theater companies. For example, Erica Mott and Dancers used Mosho in their production of “Mycelial” at the 2018 Bates Dance Festival, and the Know Theater of Cincinnati used it in their production of Ada & the Machine in April 2018.
We also have a beautiful new logo, designed by Ran Tao. You can search the Google Play Store and Apple App Store for Mosho the next time you attend a Mosho-enhanced performance.
INTERFACE: Alternate Reality is a new work-in-progress that had its first incubation period during a three-week residency as a 21C Seed project at the Institute for 21st Century Creativity at the UC-Irvine Claire Trevor School of the Arts. This work builds upon our earlier works which use the LAIT mobile application system for live performance, Critical Mass and Public Figure, both of which seek to use the mobile phone as a means of increasing audience engagement with dance as an art form and theatrical experience. Alternate Reality is a work that incorporates game culture and game thinking into the conception of live dance performance. It seeks to find ways to modernize dance performance to be in line with contemporary culture that is heavily influenced by first-person user interaction and competitiveness and the goal-orientated thinking which that promotes. Many people, particularly young audiences, are used to taking an active role in the direction and outcomes of their entertainment activities. They’re used to using game controllers that help them navigate game structures. With the capabilities of our LAIT-based app, now called “Mosho”, we have been able to craft a live-action game structure that incorporates contemporary dance.
We crafted a plot where a ship (undefined: could be a spaceship, or a sailing vessel) had hit a maelstrom which scattered her crew into many different places and times, and now they all were existing resulting in alternate realities. The captain of the ship was looking to reunite them, and enlisted the help of the audience “players” to locate them and bring them all back to one reality. To reunite the crew, the players had to solve various puzzles that would yield clues, that would lead players to discover keywords by scanning AR target images. These words were then used to complete phrases about “reality” which had come from three famous personages: Albert Einstein, John Lennon, and the film director Tim Burton. The audience used the Mosho app to find the various locations of the crew, to solve the puzzles that were found at each game level, to scan the AR target images to uncover the keywords, and then text those keywords to the captain.
Below is video documentation of the work-in-progress showing. For even more in-depth writing about this production, visit Prof. Toenjes’ new CV site page about Alternate Reality.
I have just updated the video of how the cellphones performed during the Critical Mass shows. This gives you a great insight into how the system works, and how it is integrated into the performance. Check it out!
LAIT was used at the 2017 West Hollywood Dance Festival to deliver interactive program notes to the audience. The app took the place of paper programs and displayed the title of the dance, the dance company, and other information about the dance that the choreographers wanted the audience to know. The audience could move freely through the program because LAIT now has quite functional buttons and menus that make it operate more like a typical app on a smartphone. This lets the app function on its own without the need for the stage manager to trigger cues all the time. This was a very successful use of LAIT in a different way for live performance.
Here are a few screenshots to give you an idea of how it worked:
WeHo program app opening screen
WeHo feature overview
WeHo program overview – interactive navigation of the program
WeHo program app – dance detail
Dancers rehearsing in front of the new cube, under construction in the background
We are hard at work on the premiere of the dance Critical Mass, which will occur on February 2-4 at the February Dance concert at the Krannert Center for the Performing Arts. We have a cast of eight U of I dancers, and Chad Michael Hall of Multiplex Dance came back from Los Angeles to work with us once again on the conception and choreography. We are having to make the work for both audience who will be coming onstage as well as audience that will sit in the seats like a traditional theater, so we’ve have to redo some of the staging. But the LAIT app is going to be central to this show, and showing off its many new capabilities! Tony Reimer, our programmer, has been hard at work, and he ROCKS!!
Click for more photos of the rehearsal process!
We started a new collaboration with Chicago-based choreographer and director Erica Mott. The work is entitled Mycelial: Networked Bodies, which uses the incredible life-giving and communicative capabilities of underground networks of forest mushrooms as a metaphor for the interconnectedness of various fractured communities. To that end, we are working on having mobile devices running LAIT be able to be assigned to different groups, and to have those groups respond to specific messages in real time. Watch the video of the first week’s workshop: