A Benchmark for Rough Sketch Cleanup

Paper: PDF, 600dpi images (10 MB) | PDF, full size images (30 MB)

Benchmark: explore | regenerate (GitHub)

Supplementary material: instructions for cleanup artists (3 MB ZIP) | perceptual study data (2 MB ZIP)

Above: Rough sketches collected from the wild. Below: professionally cleaned and vectorized ground truth.

Abstract:

Sketching is a foundational step in the design process. Decades of sketch processing research have produced algorithms for 3D shape interpretation, beautification, animation generation, colorization, etc. However, there is a mismatch between sketches created in the wild and the clean, sketch-like input required by these algorithms, preventing their adoption in practice. The recent flurry of sketch vectorization, simplification, and cleanup algorithms could be used to bridge this gap. However, they differ wildly in the assumptions they make on the input and output sketches. We present the first benchmark to evaluate and focus sketch cleanup research. Our dataset consists of 281 sketches obtained in the wild and a curated subset of 101 sketches. For this curated subset along with 40 sketches from previous work, we commissioned manual vectorizations and multiple ground truth cleaned versions by professional artists. The sketches span artistic and technical categories and were created by a variety of artists with different styles. Most sketches have Creative Commons licenses; the rest permit academic use. Our benchmark's metrics measure the similarity of automatically cleaned rough sketches to artist-created ground truth; the ambiguity and messiness of rough sketches; and low-level properties of the output parameterized curves. Our evaluation identifies shortcomings among state-of-the-art cleanup algorithms and discusses open problems for future research.

40-second Fast Forward MP4 (10 MB):

15-minute Presentation MP4 (40 MB) | Keynote (100 MB) | PDF (70 MB) | exported to PowerPoint (60 MB):

BibTeX (approximate):

@article{Yan:2020:ABR,
 author    = {Yan, Chuan and Vanderhaeghe, David and Gingold, Yotam},
 title     = {A Benchmark for Rough Sketch Cleanup},
 journal   = {ACM Transactions on Graphics (TOG)},
 volume    = {39},
 number    = {6},
 year      = {2020},
 month     = nov,
 articleno = {163},
 numpages  = {14},
 issn      = {0730-0301},
 url       = {https://doi.org/10.1145/3414685.3417784},
 doi       = {10.1145/3414685.3417784},
 keywords  = {sketch, drawing, design, cleanup, beautification, dataset, benchmark}
}

Artists: We are grateful to all the artists who generously contributed their rough sketches to our dataset. We acknowledge the artists whose hard work created the ground truth data: Branislav Mirkovic, Santiago Rial, Diego Barrionuevo, Ge Jin, Jonathan Velasco, Liliya Larsen, and Maria Fiddler.

Funding: Authors Yan and Gingold were supported by the United States National Science Foundation (IIS-1453018), a Google research award, and a gift from Adobe Systems Inc. Author Gingold is grateful to Adobe for supporting him during his sabbatical, during which much of the work was carried out. Author Vanderhaeghe was funded by "Investissements d'Avenir" LabEx CIMI (ANR-11-LABEX-0040) and project Structures (ANR-19-CE38-0009-01).