CraGL: Computational Reality, Creativity and Graphics Lab

Video Recoloring via Spatial-Temporal Geometric Palettes

Zheng-Jun Du, Kai-Xiang Lei, Kun Xu, Jianchao Tan, Yotam Gingold
ACM Transactions on Graphics (TOG) 40(4). To be presented at SIGGRAPH North America 2021.

Paper: PDF, 600dpi images (20 MB)

Code: GitHub

Supplementary material: User Study statistics (32 KB PDF)

Frames of a recolored video and slices of its polyhedral palette. Frames of a recolored video and slices of its polyhedral palette.

Abstract:

Color correction and color grading are important steps in film production. Recent palette-based approaches to image recoloring have shown that a small set of representative colors provide an intuitive set of handles for color adjustment. However, a single, static palette cannot represent the time-varying colors in a video. We introduce a spatial-temporal geometry-based approach to video recoloring. Specifically, its core is a 4D skew polytope with a few vertices that approximately encloses the video pixels in color and time, which implicitly defines time-varying palettes through slicing of the 4D skew polytope at specific time values. Our geometric palette is compact, descriptive, and provides a correspondence between colors throughout the video, including topological changes when colors merge or split. Experiments show that our method produces natural, artifact-free recoloring.

Video (5 minutes, contains audio) MP4 (62 MB):

Presentation (4-minutes) Recording (MP4, 28 MB) | PowerPoint (96 MB) | PDF (5 MB) | PDF with notes (4 MB):

Presentation (17-minutes) Recording (MP4, 85 MB) | PowerPoint (85 MB) | PDF (6 MB) | PDF with notes (8 MB):

BibTeX (ACM page): 

@article{Du:2021:VRS,
 author    = {Du, Zheng-Jun and Lei, Kai-Xiang and Xu, Kun and Tan, Jianchao and Gingold, Yotam},
 title     = {Video Recoloring via Spatial-Temporal Geometric Palettes},
 journal   = {ACM Transactions on Graphics (TOG)},
 volume    = {40},
 number    = {4},
 year      = {2021},
 month     = {jul},
 articleno = {150},
 numpages  = {16},
 doi       = {10.1145/3450626.3459675},
 keywords  = {Palette, Color, Grading, Painting, Image, Video, Spatial-temporal, Recoloring, Layer, RGB}
}

This work was supported by the National Natural Science Foundation of China (Project Numbers: 61822204, 61521002, 61863031), the United States National Science Foundation (IIS-1453018), and a gift from Adobe Systems Inc.