Margaret Minsky [email protected]
NYU Shanghai
Oliver Steele [email protected]
Tiled video-conferencing environments such as Zoom have become a “home” for many of us since 2020. Further, we often conference from our personal spaces such as offices in kitchens or living rooms, sometimes showing and sometimes hiding them. Many people conference in a seated position near the camera, and we see only a face or small portion of each others’ embodied selves, missing nonverbal cues or the proxemics and social impact of sitting together, walking together, or dancing together.
The PoseShare platform allows us move our bodies (in the form of video, avatars, or visualizations) to visit each other’s homes in the space defined by a Zoom conference or classroom. We interrogate the possibilities of this visitation through a student project in creating avatars (”spirits”), creating a UI for moving to other participants’ tile homes, and performing a choreographed dance sequence for an online audience.
Each Zoom participant’s body is inserted into a pipeline through BlazePose (pose recognition system) in p5.js to create avatars whose movements are tracked. The avatars are reinserted into Zoom through OBS. Spatial coherence is maintained using Zoom’s “follow host’s video order” feature. Choreography is a combination of a notated, cued dance score along with manual UI control for avatar inter-tile movement, and pose recognition elements programmed in p5.js. The platform can be run on average, non-gaming laptops and enables interactive inter-tile visiting in video conferencing.
A rehearsal of a PoseShare performance. During the rehearsal, the “spirits” are solid-color representations tracking the participant’s pose. For reference, wireframe skeletons are also overlayed onto the participant videos in their home tiles. The actual performance used a volumetric rendition for “spirits” based on metaballs.
Videoconferencing technology has become capable of displaying multiple streams of video in real-time to enable us to support meetings, classrooms, and social events. Through clever use of props and direction, theatre directors and dancers have been able to present webinar-style performances that give the illusion of direct interaction between the embodied participants. However, the participant video streams could not be composited in real-time, nor could the pose of a participant directly affect another tile. PoseShare extends the capability of videoconferencing by permitting the body poses (or direct video stream) of a participant (visitor) to be composited into any other participants’ home tile. PoseShare further supports programming interactions between the visual representation of the visitor and visitee, and the creation of UI cues or gesture recognition to move the visitor to various tiles. As a consequence of these capabilities, PoseShare encourages full-body movement in videoconferencing applications.
.png)
Diagram 1: Each participant has a station: a computer and a camera (such as a webcam). All the stations are simultaneously connected to two servers: a Zoom server to mediate video and audio, and our PoseShare server that shares information about the spatial arrangement of tiles, and broadcasts each participant’s dynamic body poses to other participants.
.png)
Diagram 2
The participant station runs three application programs (Diagram 2). Our PoseShare client application captures and broadcasts participant poses. It uses the pose of the local participant and the position and pose of other participants to synthesize an image. The image is overlaid onto video that is sent from this station. The Zoom client application captures, broadcasts, and presents video and audio. The OBS Studio streaming program mediates between the PoseShare client application and the Zoom client application, by presenting the PoseShare synthetic image as a virtual camera to Zoom.
The PoseShare server broadcasts participant poses. It is also responsible for the broadcasting the stage (the spatial arrangement of Zoom participant tiles), the scene (information about which participants are active, and how poses should be rendered), and blocking (which participant should be shown in which tiles). This last feature is used to implement the visit feature, where a participant can cause their spirit to appear in other participant’s tiles.
.png)
Diagram 3: An instance of the PoseShare client application runs on each participant station. This consists of a local server (on the left), that transfers pose information from the local web client to the PoseShare server (not pictured), and stage, scene and blocking information from the PoseShare server to the local web client; and a local web client, that runs as a page inside the Chrome web browser.
PoseShare’s capabilities were demonstrated through a dance performance project co-created with students in NYU Shanghai’s Spring 2022 Class “Introduction to Movement Practices”. The students programmed “spirit” appearance and behavior, and had input on a movement vocabulary for moving between tile homes. The vocabulary of four distinct dance movements was designed to encourage whole body movement with a range of relationships to ground, posture, and energy. The students designed a choreographic score representation for the dance performance:
Choreography included unison group movement, duet, counterpoint, and solos.
“Spirits” are visual interpretations, programmed by students in p5, of the body pose processed through BlazePose. The “spirit” can be programmed in PoseShare to in a specified tile.
Student testing a prototype of a metaball-based “spirit”