A Scaffold-Based Tool for Product Design Variations in Virtual Reality

We summarize several key observations about existing practice that underpin our design. The design of our tool was inspired by previous research into product design sketching [16, 17, 19, 20, 24, 35, 38], knowledge from design textbooks [13], personal communication with industrial designers, and our previous experience as researchers in this domain. Sketching plays a crucial role in the design process. Designers create numerous design variations. The process narrows until they are iterating on a small number of specific designs [13]. The open-ended interviews we conducted with professional designers as part of our Evaluation (Section 6.2) corroborated this. Prior to introducing our tool, the interviews discussed existing product design workflows and the role of design variations. P4: “I try to create as many proposals and barrier proposals as possible just to give a very big idea of what I can do and what the possibilities of designing are. And once we already start to get into one, maybe one design path or one design identity, I try to start to reduce the amount of proposals just to get a more linear idea.” P5 described a similar process: “We start the conceptualization process in which we make the conceptual design or the analytic drawing and come up with all the ideas...After we settle on the design, we start making the non-functional 3D model and visualize our idea and analyze it and see if there is another modification...until we reach the final design.” Design sketches are often created as analytic drawings. (Designers sometimes use the terms analytical drawing, two-point perspective, or conceptualization sketch.) Analytic drawing begins with scaffolds, followed by shape curves, and then details are added (Figure 2). If the page becomes too cluttered during this process, designers may trace their drawing onto a clean sheet of paper, taking the opportunity to reduce clutter (typically by omitting some of the scaffolds). Our system is designed to facilitate design variations of an existing analytic drawing.

Motivation. Design is an iterative process, in which ideas are conceptualized (i.e., visualized as an analytic drawing), evaluated, and modified. In this non-linear process, sketches must be redrawn as they cannot easily be edited to, for example, adjust the reclining angle of a chair. We argue that the scaffolds created for an analytic drawing, which aid the designer in drawing accurately, also provide an opportunity to aid the designer when editing.

In analytic drawing, shape curves are drawn in relation to the scaffolds. Detail strokes are drawn in relation to shape curves. Our interface is based on the idea that the scaffolds form a natural set of handles with which to manipulate a shape and explore design variations. Our interface takes a 3D (rather than 2D) analytic drawing as input. 3D analytic drawings can be obtained automatically by lifting a 2D drawing into 3D [16, 18, 35]; drawn natively in 3D [38]; or generated automatically from segmented meshes [20] or CAD sequences [19]. This provides a fixed set of scaffold and shape curves for the design exploration. (Our system does not support adding or removing them. We focus on the design variation aspect of workflows. Integration into other tools is future work.) We implemented our interface for a Virtual Reality headset and controllers. Users interact with the shape using 3D direct manipulation.

After loading a 3D analytic drawing (Figure 1 a), users select a set of scaffolds by touching them. The selection is accompanied by haptic feedback and visual highlighting (Figure 1 b's red scaffolds). Touching a selected scaffold deselects it. (A trigger deselects all.) To move the selected scaffolds, the user holds the grip button. Users can translate, rotate, and scale the selection freely in 3D (Figure 3). Upon release, the selected scaffold lines snap to a set of aesthetic constraints.

Users can also draw detail curves freely in 3D (Figure 1 e). Shape and detail curves are automatically updated whenever scaffolds are manipulated (Figure 1 f,g). The interface provides a toggle to show and hide “clutter,” which we define as scaffold lines whose endpoints lie on other scaffold lines.

Choice of VR. We implemented our user interface in VR because VR controllers allow the user to directly manipulate scaffolds and draw detail strokes in 3D. At its core, 3D shape editing is a 6 degree-of-freedom (6DOF) input problem. 3D editing in VR avoids the need for complicated 2D-to-3D lifting or projection techniques. Some product designers have already embraced VR-related techniques in their daily lives, as evidenced by our previous collaborations and interviews with product designers (P1 from Section 6) who use GravitySketch in their routines. Additionally, we have learned from design school faculty (A1, A2 from Section 6) that their school design labs are equipped with VR headsets for student exploration. Some students use the headsets quite frequently to complete studio design work. VR allows us to create a very simple user interface based on direct manipulation. At the same time, users have many degrees of freedom in 3D and struggle with precision. Our algorithm considers user accuracy and helps reduce human input error. The benefits of VR were validated in our expert study (Section 6).

Relevance Beyond VR. Our algorithms are relevant even in non-VR scenarios. A desktop interface with hand tracking input (e.g., Leap Motion) or other 6DOF input (Omni Phantom) would have similar limitations. Even precise 2D manipulations of scaffold lines (e.g., from an interface like Blender) needs our algorithms from Section 4.1 to break irrelevant constraints and find new relevant ones. Even 2D detail strokes lifted or projected into 3D need our algorithms from Section 4.3 to deform smoothly as scaffolds are manipulated. Shape curves need our algorithms from Section 4.2 to deform.

In the course of analytic drawing, designers draw construction lines that serve as scaffolds for the later shape curves that depict the object's surface features. The scaffold lines have unambiguous relationships, e.g., coincident, perpendicular, parallel, same length. The shape curves intersect the scaffold lines, often tangentially. Our observation is that the scaffold lines offer a natural set of handles for shape editing. The relationships provide a natural set of constraints to preserve so that the shape remains aesthetic. Yet those same constraints raise challenges when editing. (1) For example, shared-endpoint, same-length, perpendicular, and parallel constraints can bind a set of scaffold lines to form a cube. These constraints lock the scaffold lines, allowing only similarity transformations (translation, rotation, and uniform scale). To allow users to manipulate scaffold lines, we must find a new set of relevant constraints. (2) Scaffold lines may be attached to points along other scaffold lines. What manipulations should we allow for such scaffold lines? (3) Some shape strokes do not intersect scaffold lines tangentially. How should they deform when the scaffolds change? (4) Detail strokes are drawn in space. How should they deform?

Scaffold lines often have perpendicular, parallel, or equal length constraints. When a user makes an edit, some of these constraints will be broken, while new ones may emerge. To optimize the scaffolds, we remove the broken constraints and add the new emerging ones. This is the fundamental idea behind our scaffold optimization approach.

To begin, suppose that the user loads a square scaffold into the scene (Figure 6 (a)). These lines have the following constraints:

• Equal length: L₀ = L₁ = L₂ = L₃
  
• Parallel: L₀∥L₂, L₁∥L₃
  
• Perpendicular: L₀⊥L₁, L₀⊥L₃, L₁⊥L₂
  

If the user grabs L₁ and moves it to the right, L₂ and L₃ will also change because the lines share endpoints. Additionally, human input error is inevitable, so even if the user wants to move L₁ strictly to the right, there may be some deviation. This movement breaks some constraints:

• Length: L₀ ≠ L₁, L₀ ≠ L₃, and even L₀ ≠ L₂ due to the input noise
  
• Parallel: L₁∦L₃
  
• Perpendicular: $L_0 \not\perp L_1$, $L_0 \not\perp L_3$, $L_1 \not\perp L_2$
  

However, the user may only intended to move the line to the right. Some of the above constraints broken due to unintentional input noise. Therefore, we detect constraints again for the changed lines with a tolerance. For two lines with directions v_a and v_b, we calculate the clamped angle $\theta = \cos ^{-1} \frac{| v_a \cdot v_b | }{|v_a| |v_b|}$. If θ ≤ 6°, we detect a parallel constraint. If θ ≥ 85°, we detect a perpendicular constraint. If the length ratio of two lines is within $5\%$, we detect an equal length constraint. We only detect equal length constraints between the changed lines. The thresholds were determined experimentally and can be tuned by the user.

So for the case where L₁ moved to the right (Figure 6), we might detect the following new constraints:

• Equal length: L₀ = L₂
  
• Parallel: L₁∥L₃
  
• Perpendicular: L₁⊥L₂
  

The entire process can be summarized as follows:

• Initially, the scaffold lines have a set of constraints C_old, including perpendicular, parallel, and equal length.
  
• When the user makes an edit, some of the constraints in C_old might be broken due to input noise or intentional changes. These broken constraints are denoted as C_broken.
  
• If the user moves a line, we detect the constraints again for the changed lines with tolerance to account for input noise. The detected constraints are denoted as C_new.
  
• We calculate the set of constraints C_opt by applying the formula C_opt = C_old − C_broken + C_new.
  
• However, the constraints in C_opt might conflict with each other. Therefore, we use iteratively re-weighted least squares (IRLS) to select a set of satisfiable constraints from C_opt that best fits the scaffold lines [38]. The optimized scaffold lines are then determined based on the constraints in the selected set.
  

The user can manipulate a half-constrained scaffold line the same way as a free scaffold line. This includes translation, rotation, and scaling. The optimization process remains unchanged. However, there is a key difference for the half-constrained line: one of its endpoints is connected to a tick point on another line. Consequently, after optimization, we project this point back onto the existing line to preserve scaffold topology. Fully-constrained lines are represented as two tick points on existing lines. Users have the option to select tick points and slide them along the attached lines to make adjustments.

We also support the selection and editing of multiple scaffolds. This feature allows the user to select and edit specific parts of the scaffolded sketch more easily. For instance, if the user wants to move the face of a cube, they can select all the lines that form the face and move them simultaneously. Likewise, if the user wants to uniformly scale the entire cube, they can select all the lines and scale them together.

Inspired by ScaffoldSketch [38], we use piece-wise cubic Bézier curves to represent shape curves. (Shape curves can also be straight lines.) The Bézier information is given by the first and last control points, referred to as curve key points. These lie on scaffold lines as either ticks or vertices. Another piece of information is given by the first and last control points’ tangent directions, referred to as key directions. Key directions are often, but not always, tangent to the scaffold lines they are attached to (Figure 7). Real world examples can be found in Figure 8.

Since the curve key points are already ticks or vertices, they are automatically updated whenever the scaffold lines are edited. However, this is not sufficient as we also need to update the tangent directions to ensure that the curve looks smooth and natural. If the tangent directions are not updated accordingly, the resulting curve may appear distorted or unnatural. See Figure 9 for an example.

When the tangent direction of a curve is aligned with the direction of the scaffold line it is attached to, it is natural for the direction to follow the changes in the scaffold line direction. However, this approach does not address the case when the tangent is not aligned with the scaffold lines. In such cases, we need to consider how nearby lines can influence the direction of the non-aligned curve.

Suppose we have a curve key point p with key tangent direction d_p, and the scaffold has n lines with directions d₁, d₂, ⋅⋅⋅, d_n. Note that d_i = ±d_j could happen. We can write d_p as a linear combination of the scaffold directions:

To identify the basis for direction d_p, we first perform a global sort on all scaffold lines and then iterate over them in sorted order. Once we successfully identify a basis, we stop iterating. Otherwise, we select the next line as a candidate and continue. If the candidate line direction is redundant (i.e., it can be spanned by the current chosen lines), we discard it. We also discard candidate lines that are almost parallel (< 15°) to the currently chosen lines.

We use two sorting criteria for the scaffold lines to ensure they are close to both point p and the curve c that p lies on. First, we calculate the Euclidean distance between all the scaffold lines and the point p, and sort the lines based on this distance. We break ties with a secondary criterion, the point-point distance between a scaffold line's points and points on the curve c. This secondary criterion prefers scaffold lines that are closer to the curve c.

A concrete example can be seen in Figure 10. This BFS-like algorithm allows us to express any curve key direction as a sparse linear combination of nearby line directions. Now, if the user makes an edit, the positions of vertices and ticks will be updated, and consequently, the scaffold lines, which are represented by indices of points, will also be updated, along with their directions. Shape curves will be updated accordingly, and move along with the scaffolds. To ensure that shape curves remain fair, we optimize shape curves’ tangent magnitudes to minimize variation of curvature [38].

Detail strokes, such as small buttons, seams, or electric cords, do not require high accuracy and therefore do not need scaffolds. Some elements, such as logos, text, or texture-like elements, may be difficult or even impossible to scaffold. However, these strokes are an important part of design exploration.

To enable detail strokes to move along with the scaffolds and shape curves, we represent them as weighted combinations of scaffold endpoints. Given a new detail stroke, we first fit a cubic B-spline representation to smooth out any input noise and discontinuities in the soon-to-be-computed weights (Figure 11). Similar in spirit to expressing the shape curve tangents as a linear combination of scaffold lines, we express the B-spline control points as a linear combination of scaffold points as follows.

Given a single control point q, and n scaffold points p₁, p₂, ⋅⋅⋅, p_n, we can write q = ∑_iw_ip_i subject to ∑_iw_i = 1, where w_i is the weight controlling how much the control point moves when the scaffold point p_i moves. This is a type of generalized Barycentric coordinate system. Control points, unlike directions, require that the chosen weights sum to one. This equation is trivially solvable if any four of the n points form a non-degenerate tetrahedron in 3D, which is always true for non-planar scaffolds.

To choose intuitive weights from among the infinite solutions, we desire nearby scaffold points to have higher influence. Additionally, we prefer to represent a control point with relatively few scaffold points, meaning that most scaffold points should have zero influence (weights). An example can be seen in Figure 12. Unlike shape curve tangents (Section 4.2), detail curves are not attached to scaffolds.

We minimize the following objective to obtain detail curve weights:

This approach finds a relevant set of scaffold points and weights. When moving the scaffolds, the control points move accordingly, and the detail strokes along with them (Figure 13).

The As-Locally-Uniform-As-Possible (ALUP) approach was recently demonstrated for structure-aware editing of 2D illustrations [1]. Their algorithm is based on the idea that edits should try to uniformly scale and avoid rotating shape features. Although our algorithms are entirely different, and 2D illustrations lack user scaffolds, our scaffold-based editing approach provides similar functionality in 3D, resulting in intuitive, shape-preserving edits. The additional scaffold under the bottle's lid in Figure 17(c) provides more precise control over the neck. This is similar to ALUP's $w_i^d$ parameter (Figure 12 in [1]). In addition, our system supports 3D shapes, whereas ALUP is limited to 2D.

We received initial feedback on our tool from three product designers with whom we have collaborated in the past. (Full comments can be seen in our supplementary materials.) A1 and A2 are both professors in industrial design departments, A1 having more than 25 years of teaching design and design-related experience, and A2 with more than 10 years of design experience. A3 recently graduated with an industrial design degree.

They viewed a video walkthrough demonstration of our tool. All three were enthusiastic about the tool and believe it could be used in teaching product design as well as in their everyday workflow. A1’s initial response was, “I can see it being so useful. Can I integrate it into my design studio curriculum? Can I share this with my [colleagues]?” Both A1 and A2 suggested that the tool could be employed for form study, enabling the creation of different variants from a base form. A1 remarked, “The system works in a very similar manner to the way an industrial designer would think about constructing and then modulating form. It feels very natural and more importantly provides results that are anticipated and expected rather than the software throwing a curve ball at you. It can be a powerful tool for teaching students the basics of form modulation as well and I feel it will result in students graduating the class with a much better understanding of form development.” A2 commented, “What really excites me initially is the manipulation of the scaffolding/construction lines that directly translate to the form lines. This would be great for quickly doing form studies and iteration based upon scale, and could allow users to really understand what a form study actually is without redrawing things over and over.” A3 commented on how our tool could be integrated into his workflow. “I can definitely see this being used for quick ideations and creating multiple ideation ideas from one idea. I can also see this being helpful for understanding how to prototype your ideas after you are done with ideation.”

A2 suggested an direction for future work: “I wonder if you make a shift for the detail lines and shape curves to include more construction lines—more like a mesh or grid that can be tuned by the user depending on the desired outcome.” We expand upon the idea of automatically adding scaffold lines in Section 7, where we discuss implications for design and future work.

Following guidelines and best practices [15, 25, 32], we conducted a walkthrough demonstration and open-ended interview [25]. We believe this shifts the focus away from training participants to use the software but in favor of determining the utility or value of our tool.

We recruited 5 professional product designers (P1–P5) from the UpWork platform. They were globally distributed and had 3–8 (average 4.3) years of post-education professional design experience (Table 1). Interviews were conducted remotely and lasted between 22 and 38 minutes (average 27). During the interviews, designers sometimes demonstrated their workflows via screen sharing. The protocol was approved by our university ethics board. The interview script and full transcripts can be found in the supplemental materials.

After discussing traditional design workflows and the role of design variations (Section 3.1), we performed a walkthrough demonstration of our tool via a pre-recorded video and allowed designers to view and rotate a gallery of 3D analytic drawings and edited results created with our tool. We then asked a series of questions about the design space supported by our tool, whether the tool can be incorporated into their design processes, and features or ideas they wish our tool had.

Once the interviews were complete, we conducted a thematic analysis [9, 27] on the interview scripts. We discuss the resulting themes below. Where appropriate, we also mention the preliminary and hands-on users’ agreement with these themes.

Design Variations & Iterations. Overall, P1–P5 expressed similar views and attitudes to A1–A3 from our preliminary feedback. Because the nature of the design process includes making variations and iterations, they agreed that our tool would benefit their workflow. P1: “This is a great tool for iterations like quick line iterations”. P2: “It's more of a fluid approach to design...It removes that step [redrawing multiple times] and we can go faster.” P4: “You could definitely use the software...to refine the the basic lines of the product.”

Detail Strokes. Designers (P2, P3, U1) appreciated the ability to add detail strokes as a way to perform small, low-friction updates on a design. This validated our motivation for this functionality. P2: “Even when you're in a meeting with the client, you can easily just adjust some details, and it would be clear for both persons.” P3: “The details part can help me create variants that would be really useful...It's easier for me to have the shell of the shoe here, with the analytic design of it, and add rough details, using the sketching tool and seeing what different variations I can create out of them and having them in front of me in 3D in such an easy manner would be such time saving for such a project.”

Simple User Interface. Designers (P2, P3, P5, U1) appreciated the simplicity (focus and directness) of our tool's user interface, which allows grabbing and drawing directly in 3D. This was validated by our hands-on feedback user, who learned to smoothly use the system in minutes. P2: “Do not over-complicate it and over-add commands and options like other software. It's nice to have software that just has one function. And that's it.” P3: “The interface is such a simple and easy interface that won't be hard for any new users or experienced users in the analytical design.” P5: “Just move your hand to do the variations....It is an awesome idea.”

Time Saving & Tracing. Designers (P2, P3, P4, U1) appreciated the time-saving potential of our tool by eliminating the need to redraw (similar to A2’s observation). P2: “It takes a part that is redrawing multiple times, it removes those steps, and we can go faster...You also remove that pain to draw and draw and draw.” P3: “It won't take you any time at all to do this [with your tool], and it will be much easier than creating an analytical design and then trying to create another one.” P4 drew an analogy with traditional tracing practice. “It's like, if we were working with this transparent paper...and you will be working over the same sketch all the time....I draw this basic design, and I want to recreate some small iterations....[Your tool] would be much faster than actually working or sketching the 3 or 10 or 20 proposals...over a sketch.”

Snapping. P1 has extensive experience with VR sketching using GravitySketch. He appreciated that our optimization helps eliminate human input errors, “because moving things with [one's] hands is completely random.”

Topology. Some designers (P1, P3, P5) appreciated that the scaffold manipulation maintained the topology of the drawings. P1: “That would be super useful to have a clean topology...and that's missing in GravitySketch.” (P1 was referring to maintaining the planarity of curves.) P3: “The handles are segmented for each part making it really easy and detailed for you to control each line and manipulate it in the way that you want.” P5: “All the variations follow the same restrictions you made in the template.” Although not discussed by designers, the topology maintenance limits the edits that can be performed (Section 7).

We validated the methods in Section 6.2 and investigated its usability (e.g., whether the software functionality is discoverable and consistent) by recruiting a remote product designer (U1) with VR experience to use our interface. We modified the protocol to have the user run our software. The user was first guided through the software's functionality and then allowed time for open-ended exploration on a number of template models. The session lasted 1 hour. The user edited a bowl, chair, shoe, and car model. (The VR interface connected to our compute server over the Internet; there was no noticeable delay.) Variations created by U1 can be seen in Figure 18.

U1’s feedback was generally in agreement with the walkthrough demonstration participants (A1–A3 and P1–P5) regarding workflow suitability and the beneficial role our system could play: “I'm doing something right now that would really benefit from this.” As a user, U1 also provided experiential feedback. U1 was generally pleased with the deformations of the shape and detail strokes in response to their scaffold deformations. U1 appreciated that detail strokes followed scaffold drags in real-time, a feature enabled by our linear blend weights (Section 4.3): “That's pretty cool. I like how it travels with [the scaffolds].” U1 was comfortable in our system after 2 to 3 minutes: “The idea of the interface works really well, like being able to move individual handles or scale or move them in conjunction is really smart.”

U1 remarked that it would be useful to have the ability to fork a model and display variations side-by-side or in layers. U1 described this functionality in terms of bookmarking, copying, and detail stroke layers that can be toggled on and off.

U1 also commented on usability issues to address in the future. First, U1 was in a constrained physical space, and so desired functionality to reposition the shape without physically moving around the space. Second, U1 struggled to make small edits due to fixed thresholds. Since the parameters were determined experimentally and are related to the scale of the model, we believe this can be alleviated by introducing a “zoom” feature with egocentric thresholds, similar to the approach in ScaffoldSketch [38]. Finally, U1 had suggestions to avoid accidentally selecting scaffolds.

Product design is a highly specialized discipline which greatly reduces the size of potential participants versus the general population. In our study recruitment efforts, we were unable to locate professionals within several hours of our geographic location. Product designers with VR experience or equipment are even rarer. We considered a remote study with experts. However, due to the client-server structure of our tool, setting up a remote environment is extremely challenging. Users need to be product design experts, have a Meta Quest in developer mode, have a Meta developer account, have Android APK-related knowledge to install the software, and command line knowledge to run the server. These factors made a hands-on expert study challenging for us to perform. Our evaluation goals and these expert population constraints suggested a utility study [15, 25, 32] as our primary evaluation modality. We leave a more extensive evaluation, including hands on testing with more professional designers, as important future work.