Decomposing Time-Lapse Paintings into Layers

Jianchao Tan, Marek Dvorožňák, Daniel Sýkora, Yotam Gingold
ACM Transactions on Graphics (TOG) 34(4). Also in Proceedings of SIGGRAPH North America 2015. Selected for the Highlights of SIGGRAPH session at FMX 2016.

Paper: PDF (28 MB) | PDF 600dpi (5 MB)

Presentation: Keynote 6 (142M) | PDF (53M) | M4V (166M)

Videos and other supplementary materials: Zip (130 MB)

Code: GitHub

A time lapse video of an apple; changing its color under its spots; erasing its spots. Time lapse video © Marcello Barenghi.


The creation of a painting, in the physical world or digitally, is a process that occurs over time. Later strokes cover earlier strokes, and strokes painted at a similar time are likely to be part of the same object. In the final painting, this temporal history is lost, and a static arrangement of color is all that remains. The rich literature for interacting with image editing history cannot be used. To enable these interactions, we present a set of techniques to decompose a time lapse video of a painting (defined generally to include pencils, markers, etc.) into a sequence of translucent “stroke” images. We present translucency-maximizing solutions for recovering physical (Kubelka and Munk layering) or digital (Porter and Duff “over” blending operation) paint parameters from before/after image pairs. We also present a pipeline for processing real-world videos of paintings capable of handling long-term occlusions, such as the painter’s hand and its shadow, color shifts, and noise.

Time lapse video © Matyáš Veselý.

BibTeX (or see the ACM Digital Library entry):

 author    = {Tan, Jianchao and Dvoro\v{z}\v{n}\'{a}k, Marek and S\'{y}kora, Daniel and Gingold, Yotam},
 title     = {Decomposing Time-Lapse Paintings into Layers},
 journal   = {ACM Transactions on Graphics (TOG)},
 volume    = {34},
 number    = {4},
 month     = jul,
 year      = {2015},
 articleno = {61},
 pages     = {61:1--61:10},
 numpages  = {10},
 url       = {},
 doi       = {10.1145/2766960},
 publisher = {ACM Press},
 address   = {New York, NY, USA}

Copyrighted artwork: Marcello Barenghi (yellowapple, rose, eye, egg, cube, cola, candy), Matyáš Veselý (graffiti), Will Kemp (lemon), Dani Jones (scrooge), and Sycra Yasin (woman).

Funding: Authors Tan and Gingold were supported in part by the United States National Science Foundation (IIS-1451198 and IIS-1453018) and a Google research award. Authors Dvorožňák and Sýkora were supported by the Technology Agency of the Czech Republic under the research program TE01020415 (V3C – Visual Computing Competence Center), by the Czech Science Foundation under research program P202/12/2413 (OPALIS), and by the Grant Agency of the Czech Technical University in Prague, grant No. SGS13/214/OHK3/3T/13 (Research of Progressive Computer Graphics Methods).