MoSculp: Interactive Visualization of Shape and Time

Xiuming Zhang 1   Tali Dekel 2   Tianfan Xue 2   Andrew Owens 3   Qiurui He 1   Jiajun Wu 1   Stefanie Mueller 1   William T. Freeman 1, 2

1 MIT CSAIL 2 Google Research 3 UC Berkeley
     

Abstract

We present a system that visualizes complex human motion via 3D motion sculptures---a representation that conveys the 3D structure swept by a human body as it moves through space. Our system computes a motion sculpture from an input video, and then embeds it back into the scene in a 3D-aware fashion. The user may also explore the sculpture directly in 3D or physically print it. Our interactive interface allows users to customize the sculpture design, for example, by selecting materials and lighting conditions.

To provide this end-to-end workflow, we introduce an algorithm that estimates a human’s 3D geometry over time from a set of 2D images, and develop a 3D-aware image-based rendering approach that inserts the sculpture back into the original video. By automating the process, our system takes motion sculpture creation out of the realm of professional artists, and makes it applicable to a wide range of existing video material.

By conveying 3D information to users, motion sculptures reveal space-time motion information that is difficult to perceive with the naked eye, and allow viewers to interpret how different parts of the object interact over time. We validate the effectiveness of motion sculptures with user studies, finding that our visualizations are more informative about motion than existing stroboscopic and space-time visualization methods.

Our MoSculp system transforms a video (a) into a motion sculpture, i.e., the 3D path traced by the human while moving through space. Our motion sculptures can be virtually inserted back into the original video (b), rendered in a synthetic scene (c), and physically 3D printed (d). Users can interactively customize their design, e.g., by changing the sculpture material and lighting.

Paper

MoSculp: Interactive Visualization of Shape and Time
Xiuming Zhang, Tali Dekel, Tianfan Xue, Andrew Owens, Qiurui He, Jiajun Wu, Stefanie Mueller, William T. Freeman
ACM Symposium on User Interface Software and Technology (UIST) 2018
paper (hi-res)  /   paper (arXiv)  /   supplemental (coming soon...)

BibTeX
@inproceedings{zhang2018mosculp,
    author    = {Zhang, Xiuming and Dekel, Tali and Xue, Tianfan and Owens, Andrew and He, Qiurui and Wu, Jiajun and Mueller, Stefanie, and Freeman, William T.},
    title     = {{MoSculp: Interactive Visualization of Shape and Time}},
    booktitle = {Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology},
    series    = {UIST '18},
    year      = {2018},
    location  = {Berlin, Germany},
    publisher = {ACM},
    address   = {New York, NY, USA}
}

Talk

Slides

.pptx (486 MB)  /   .pdf (104 MB)  /   compressed .pdf (6 MB)

Press Coverage

MITCreating 3-D-printed "motion sculptures" from 2-D videos (featured as the 9/19 MIT homepage)
BBCCreating 3D sculptures from 2D video
ForbesThese Researchers Turned 2D Videos Into 3D Motion Sculptures
ACMCreating 3D-Printed 'Motion Sculptures' From 2D Videos

Code

Videos

Supplemental Video


Other Sources

News Video

Courtesy of Tom Buehler (MIT CSAIL)

Example Results

Our motion sculptures can be embedded back into the source video or rendered in a synthetic scene. Note that the depth ordering between the moving human and the sculpture is preserved.

Customizing Rendering of Motion Sculptures

In the following example, you can customize the sculpture design by hovering your mouse over the desired sculpture material. Our user interface (linked above) allows further customization, such as changing the lighting settings, sculpture transperancy, number of human snapshots, sculpture material, and rendering background.

Wood  Leather  Metal  Marble  Tarp  Tiles

3D Printing Motion Sculptures

Since we have estiamted the full 3D geometry of the human, we can create a physical copy of the motion by 3D printing the motion sculpture.

Video Courtesy of Tom Buehler (MIT CSAIL)