"SIGGRAPH hates systems papers," I've heard frustrated researchers say.
During a SIGGRAPH PC meeting, I've heard a committee member disparagingly comment: "I don't believe there is novelty here, maybe it's a good systems paper."
And from a recent SIGGRAPH post-rebuttal discussion post: "This is clearly a systems paper rather than a research contribution."
This is not an issue of whether systems research has a place in the graphics community. (It does!) Rather, these comments suggest that both graphics papers authors and reviewers hold a misunderstanding of the intellectual value of graphics systems research. Understanding the key principles and values of systems research thinking is important to system designers that wish to communicate their findings and for reviewers evaluating those results. Improving this understanding will make the graphics community better equipped to evaluate and disseminate its valuable systems results.
With this article, I hope to contribute to our shared understanding of (graphics) systems research principles. I suspect that most computer graphics researchers, even those that do not explicitly aim to create new systems, can benefit from applying systems thinking principles to their work.
This article is not an attempt to provide a comprehensive guide to writing systems papers, for which there are many excellent takes. For example, I recommend "How (and How Not) to Write a Good Systems Paper" [Levin and Redell 1983].
Advances in graphics systems have long played an important part of SIGGRAPH. In fact, systems work has historically had disproportionate impact, not only on the progress of computer graphics, but on a broad range of technical fields. It is easy to make an impressive list of examples.
In recent years we've seen papers at SIGGRAPH that describe the HDR photo processing pipeline in modern smartphone cameras [Hasinoff 16][Wadhwa 18], practical VR video generation pipelines [Andersen 16], the design decisions behind the implementation of the widely used OpenVDB library [Museph 13], and that of systems for large-scale video processing [Poms 18]. SIGGRAPH 2016 had an entire technical papers session on domain specific languages (DSLs) for graphics and simulation, and in 2018 an entire special issue of TOG was dedicated to the design of modern production rendering systems.
Papers that focus on systems contributions are in the minority at SIGGRAPH, but they are frequently featured. Without question these efforts have carved out an important place in the graphics community, as well as in the broader tech world.
Ask researchers throughout computer graphics to name defining characteristics of their favorite papers. Regardless of area, I suspect the lists would be similar. Great papers teach us something we didn't know about our field: either they contain an idea that we had never thought about, make us think differently (or more clearly) about a concept we thought we knew, or introduce a hard-to-create artifact (or tool) that is useful to making future progress. We tend to call these "contributions", a term I've increasingly come to appreciate as it embodies what a good paper does for the field. For example:
The reader learned something new:
Perhaps less surprising, but generally useful to the progress of others:
As with all good research papers, the intellectual contribution of a systems paper is much more in the ideas and wisdom the paper contains, and much less about the specific artifact (the implementation) it describes.
However, those not accustomed to assessing systems research can struggle to identify its intellectual contributions because they are often not written down explicitly in the form of an algorithm or equation. Conversely, it is common for paper authors, having spent considerable time implementing and engineering a system (and maybe even deploying it with success to users), to erroneously think that simply describing the system's capabilities and implementation constitutes a "good" systems paper.
In some areas of computer graphics, it is common for problem inputs and outputs to be well defined. The challenge is that the problem itself is hard to solve! In contrast, articulating and defining the right problem to solve is a critical part of systems research --- often this is the hardest part. Good systems thinking requires an architect to internalize a complex set of (potentially conflicting) design goals, design constraints, and potential solution strategies. Therefore, in order to assess the contribution of a systems paper, it is essential to clearly understand these problem requirements.
Problem characterization requires a system architect to spend considerable time understanding an application domain and to develop knowledge of hardware, software, human factors, and algorithmic techniques in play. Therefore, the work put into clearly establishing and articulating system requirements is itself part of the intellectual contribution of a graphics systems paper.
In other words, good systems papers often put forth an argument of the flavor:
We are looking to achieve the goals A, B, C, under the constraints X, Y, Z. No system existed to do so when we set out on our efforts (otherwise we probably would have used it!), so from our experiences building systems in this area, we can distill these experiences into a specific set of system requirements. These requirements may not be obvious to readers who have not spent considerable time thinking about or building applications in this space.
This setup is critical for several reasons:
Let's take a look at some examples.
Example 1: Section 1 of the Reyes paper dedicated an entire column of text to Pixar's need to handle unlimited scene complexity, which they defined in terms of scene geometry and texture data. The column also established global lighting effects as an important non-goal, since the authors' experiences at Pixar suggested that global effects could often be reasonably approximated with local computations using texture mapping.
Example 2: A decade later Bill Mark articulated a complex set of design goals for Cg. To improve developer productivity, there was a need to raise the level of abstraction for programming early GPU hardware. Doing so was challenging because developers expected the performance transparency of low-level abstractions (the reason to use GPUs was performance!) and GPU vendors wanted flexibility to rapidly evolve GPU hardware implementations. These goals were quite different from those facing the creators of prior shading languages for offline renderers and were ultimately addressed with an approach that was basically "C for graphics hardware", instead of a graphics-specific shading language like RSL.
Example 3: Google's recent HDR+ paper expresses the system requirements as a set of guiding principles for determining what algorithms were candidates for inclusion in an modern smartphone camera application: be run fast enough to provide immediate feedback, do not require human tweaking, and never produce output that looks worse than a traditional photo.
Example 4: In Yong He's recent work on the Slang shader programming language, problem requirements were established via a detailed background section (Section 2) that describes how the goal of maintaining modular shader code was at odds with the goal of writing high performance shaders in modern game engines. This section served to articulate goals and also discuss design alternatives.
Given stated goals and constraints, a systems paper will often propose a formulation of a problem that facilitates meeting these requirements. In other words, many systems papers make the argument:
It is useful to think about the problem in terms of these structures (e.g., using these abstractions, these representations, or by factoring the problem into these modules), because when one does, there are compelling benefits.
Benefits might take the form of: improved system performance (or scaling), enhanced programmer productivity, greater system extensibility/generality, or the ability to provide new application-level capabilities that have not been possible before.
Identifying useful problem structure often forms the central intellectual idea of a systems paper. As in other areas of computer graphics, elegant conceptual insights can be summarized in a few sentences. For example:
Given a set of requirements, a systems architect is usually faced with a variety of solution strategies. For example, a performance requirement could be solved through algorithmic innovation, through the design of new specialized hardware, or both (modifying an existing algorithm to better map to existing parallel hardware). Alternatively, the path to better performance might best go through narrowing system scope to a smaller domain of tasks. A productivity goal might be achieved through the design of new programming abstractions, which might be realized as a new domain specific language, or via a library implemented in an existing system.
As a result, a systems paper author must identify the key choices made in architecting their system, and elaborate on the rationale for these design decisions. Doing so typically involves discussing the palette of potential alternatives, and providing an argument for why the choices made are a preferred way to meet the stated requirements and design goals. It is not sufficient to merely describe the path that was taken without saying why was deemed a good one.
Discussion of key design decisions provides wisdom and guidance for future system implementors. By understanding the rationale for a system designer's choices, a reader will be able to better determine which decisions made in the paper may be applicable to their own requirements.
While reflecting on their design decisions, researchers should consider the following:
Differentiate key decisions from implementation details. A systems architect should clearly indicate which decisions they consider to be carefully thought out decisions that are central to the system's design (contributions of the paper they want to be given credit for) and which decisions were made "just to get something working". When less critical decisions need to be mentioned for completeness of exposition, it is useful to clarify that "algorithm X was used, but the decision was not deemed fundamental and many other equally good options are likely possible."
Make an argument for what "should" be done, not what "can" be done. In other words, good systems architects strive to argue why the decisions made are preferred solutions to meet the specified systems goals. One common property of a preferred solution is that it is a simple one. Therefore, a common question when assessing the quality of design decisions is whether the prescribed solutions are the simplest possible approach to achieving the desired goals. Presenting a more elaborate method can often be tempting for students, but a well architected system will eschew complexity that is not fundamental to meeting goals.
Identify cross-cutting issues. Many important design decisions are informed by considering cross-cutting issues that only reveal themselves when building an end-to-end system. For example, if only a single design goal or single aspect of the system was considered, there might be multiple viable solutions. However, it is often the case that a system architect's design decisions are motivated by end-to-end view of the problem. For example:
Discussion of cross-cutting issues is an important aspect of systems thinking. (it is less common in methods-centric research). Cross-cutting and end-to-end issues are often the reason why more sophisticated techniques from the research community may be less desirable for use in effective systems. Often, new methods are (wisely) developed under simplifying assumptions that help facilitate exploration of new techniques. Systems thinking must consider the complete picture of the context in which techniques are used in practice.
If a paper clearly describes a system's goals and constraints, as well as articulates key system design decisions, then the strategy for evaluating the system is to provide evidence that the described decisions were responsible for meeting the stated goals.
Particularly when a system's evaluation focuses on performance, it is tempting to compare the proposed system's end-to-end performance against that of competing alternative systems. While such an evaluation demonstrates that performance goals were met, it is equally (and sometimes more) important to conduct experiments that specifically assess the benefit of key optimizations and design decisions. Evaluation of why success was achieved is necessary to verify that the central claims of the paper are sound. Failing to perform this evaluation leaves open the possibility that the success of the system is due to other factors. (e.g., high-quality software engineering), than the proposed key ideas.
SIGGRAPH PC meetings often begin with a discussion about the importance of promoting "new ideas" and there is broad agreement that the metrics for evaluation change when they introduce a new topic area or break new ground. For example, systems that introduce new experiences or new ways of thinking about data (e.g., Phototourism [Snavely 2006] or AverageExplorer [Zhu 2014]) have no obvious numerical comparison to prior systems. Evaluating this type of work typically involves demonstrating the new experiences or insights now possible given the new capability. (In these cases, the choice of methods was "good" insofar as they enabled the new capability to be achieved.)
When assessing the merit of a systems paper, it is important for reviewers to consider the extent to which the system introduces new capability to the field, and what the implications of these new capabilities are. While new interaction techniques, such as the two examples cited above, are often easier to identify as providing new capabilities, it can also be the case that dramatic improvements in performance, scale of operation, or programmer productivity transport the field into a "new capability" regime. For example, the ability to provide real-time performance for the first time (enabling interactive applications or human-in-the-loop experiences), or the ability to write applications that leverage increasingly large image databases, could be considered new capabilities if it was previously difficult or impossible for programmers to attempt these tasks.
In these cases, reviewers should be mindful about the value of extensive quantitative comparison to prior systems or methods because prior systems that meet the stated constraints may not exist. Similarly, "user studies" might be less valuable than understanding the extent to which a system allowed its authors (who might be practicing experts in a domain) to perform tasks that had never been performed by the community before. When apples-to-apples evaluations are not realistic to provide, responsibility lies with paper authors to make a clear argument for why the provided evaluation is sufficient to lend scientific credibility to the proposed ideas, and on reviewers to carefully consider the implications of the proposed work. Requests for lengthy numerical evaluation should not be used as a substitute for author/reviewer thought and judgment.
I hope this article has highlighted the depth of thought required for good systems research and good systems paper writing. Architecting good systems is a challenging, thoughtful task that involves understanding a complex set of factors, balancing conflicting goals and constraints, assessing a myriad of potential solutions to produce a single working system, and measuring the effects of these ideas.
The approach "we have a result, now just write it up" rarely turns out well when writing a systems paper. Since there is typically not a new proof, equation, or algorithm pseudocode to point to as an explicitly identifiable contribution, the intellectual value in systems work is conveyed through careful exposition that documents wisdom gained from the design process. Personally, I find the act of writing to be a valuable mechanism to achieve clarity about what my work has accomplished. As I attempt to make a case for a system's design, more alternatives and evaluation questions come to mind. (Are we sure we can't take this feature out and get the same result? How do we know we really need this?)
On the flip side, reviewing a systems paper requires considerable thought and judgment. The reviewer must assess their level of agreement with the stated goals, requirements and design decisions. They must measure the value of the services afforded by these design decisions and consider their utility and significance to users. Last, they must determine if there is evidence the proposed decisions were actually responsible for the outcomes. Since the true test of good systems work lies in whether the ideas achieve adoption by the broader community over time, a reviewer must employ their own taste and experiences to make predictions about the likelihood this will occur and the amount of wisdom they have gained.
I wish everyone good luck with future graphics systems work!
Addendum: I've added a few tips for graphics conference papers chairs, papers committee members, and papers sorters.
Acknowledgments: Thanks to Andrew Adams, Maneesh Agrawala, Fredo Durand, Bill Mark, Morgan McGuire, Jonathan Ragan-Kelley, Matt Pharr, Peter-Pike Sloan, and Jean Yang for helpful feedback.