Plot Twist is an immersive storytelling experience that lets you step into a story as the main character, determining how it unfolds by physically enacting your choices. Inspired by the ancient walls of CIID's home in Bergamo, walls that have witnessed countless stories over centuries, the team used generative AI to let passersby imagine and immerse themselves in tales of the past.
To begin, one approaches a pulsing orb projected on the wall. Medieval music (of the story’s time period) grows louder as they near. When they touch the orb, the music swells, and the story begins. A shadow-like projection of themselves appears on the ancient stone, shifting color to reflect the mood of the story as it progresses, and the background music of the story continues to evolve to support the tone of the story. An audio narration unfolds in second person, immersing the participant as the main character. When the story reaches a branching point, the participant chooses between two actions by physically performing them. For example, slink into the shadows on the left, or raise a sword and fight off the oncomers. Each decision shapes the tone and direction of the next story told by these walls for the next participant.
The project was built using TouchDesigner with MediaPipe for gesture detection, translating body movements into story decisions and generating the projected particle-shadow image. The narrative is generated using Google's Gemini API, creating branching tales that evolve with each playthrough, voiced through Gemini's text-to-speech. The music responds too, shifting between suspenseful, melancholic, or triumphant tones powered by Google's experimental Lyria AI.
This was the team's first experience with TouchDesigner. Through it, the team learned how simple gestures like stepping sideways or raising an arm can translate into rich narrative actions like escaping into shadows or drawing a sword. The project became an exploration of cutting-edge generative AI transformed into an embodied human experience.
