Pandemic Prototyping and Augmenting the Art Museum
AbstractNorton Art+ is a new augmented-reality (AR) iPad app developed by the Norton Museum of Art in partnership with exhibition designer Local Projects. The experience invites visitors to explore creative and educational content around six key contemporary artworks in the museum’s collection through unique AR interactions. It goes beyond simply adding a digital layer of didactic material over the artworks by empowering visitors to create in the style of the artists whose work they're observing. Concept development began in 2019, before COVID-19 was of widespread concern, and it was decided early in the process that the experience would be designed exclusively for use at the museum, and would run on iPads provided to guests by the Norton (and would not be available through the App Store). Community and stakeholder input was critical to the development process, and as the realities of the pandemic set in midway through the project timeline, it became clear that initial plans for in-person prototyping and usability testing at the museum would require a rethink. In this paper, we'll describe our approach to adapting the development, prototyping, and usability testing process for Norton Art+ across a distributed team of museum professionals, exhibition designers, and community members. We will also explore how Norton Art+'s unique approach to artwork-specific interactions sparks visitors' interest in the visual and process-driven dimensions of an artist’s work. Finally, we will investigate how this experience can build on two pieces of relevant scholarship from Professor Gal Zauberman of the Yale School of Management, with whom we are planning future studies.
Keywords: Augmented Reality, Exhibition Design, Engagement, Education, Art, Interactive
Norton Art+ is a new augmented-reality (AR) iPad app developed by the Norton Museum of Art in partnership with exhibition designer Local Projects.
The experience invites visitors to explore creative and educational content around six key contemporary artworks in the museum’s collection through unique AR interactions. It goes beyond simply adding a digital layer of didactic material over the artworks by empowering visitors to create in the style of the artists whose work they’re observing.
Concept development began in 2019, before COVID-19 was of widespread concern, and it was decided early in the process that the experience would be designed exclusively for use at the museum, and would run on iPads provided to guests by the Norton (and would not be available through the App Store).
There are a number of factors in favor of this approach: Siting the experience in the museum maximizes engagement with the actual artworks that are the subjects of the app, and allows for interactions that reference the context, composition, and scale of the works specifically and directly — and creates a draw to increase visitorship. Relying on in-museum distribution of iPads spares visitors the trouble of downloading and installing another app, and allows for the use of improved AR techniques that leverage hardware only available on specific device models.
Community and stakeholder input was critical to the development process, and as the realities of the pandemic set in midway through the project timeline, it became clear that initial plans for in-person prototyping and usability testing at the museum would require a rethink.
In this paper, we’ll describe our approach to adapting the development, prototyping, and usability testing process for Norton Art+ across a distributed team of museum professionals, exhibition designers, and community members. We will also explore how Norton Art+’s unique approach to artwork-specific interactions sparks visitors’ interest in the visual and process-driven dimensions of an artist’s work. Finally, we will investigate how this experience can build on two pieces of relevant scholarship from Professor Gal Zauberman of the Yale School of Management, with whom we are planning future studies.
Project Overview and Objectives
In June of 2019, the Norton Museum of Art in West Palm Beach, Florida, received a major grant from the Anna-Maria and Stephen Kellen Foundation to create experiences showcasing contemporary works.
The project donor directed the project team to explore the potential and promise of augmented reality to create new ways for young visitors and families to discover and get excited about the contemporary art collection at the Norton.
The Norton partnered with Local Projects to design an experience blending didactic information about the artist and their process with fun, interactive experiences that put creative power in the hands of museum visitors.
The project was designed to maintain interest for the museum’s core audience groups of families and older adults, while creating new entry points into art for young people. To further broaden the audience and community reach, the app and its content management system were architected to support additional languages — and includes English and Spanish at launch. Accessibility was also a priority, and accomplished in large part by developing the application as a “native” iOS app, allowing good interoperation with the accessibility features built into Apple’s platform.
Educational objectives included deepening visitors’ experience of looking at art, while teaching visitors about the artists, their inspirations, and/or their processes. We also wanted to inspire visitors to learn more after the experience.
Operationally, the app would need to integrate effectively with museum tours, and include some takeaway content to support post-visit engagement and marketing opportunities. This was ultimately realized through a “portfolio” feature which makes it easy to save and email the virtual artworks visitors create in the app along their journey through the museum.
The project plan originally allocated six months from concept to launch, but due to delays related to COVID-19, the engagement ended up stretching to 13 months — and ultimately launched to the public on January 2 2021.
The Norton Art+ approach breaks with a number of precedents around how AR apps can work in museums. Taken together, these help define a conceptual framework for the project that’s playful, interpretive, and inclusive.
First, there’s the matter of depth versus breadth: Conventional approaches (including some of Local Projects’ past designs) work well as an “information lens” to address a large number of art objects (100+) with relatively templatized content. For example, every artwork might receive an overlay with related content, or with specific points of interest within the work, visually highlighted for visitors’ consideration. In this approach, the core interaction is the same for each work, and only the didactic content in the template changes.
For Norton Art+, we wanted to go beyond the traditional “lens” approach to AR to create deep, context-specific interactions for a smaller number of artworks (six at launch). A significant aspect of each artwork is identified and defines a unique interaction specific to that work.
Second, we wanted to move from consumption to creation — in-depth information about each work would still be available in the app, but the entry point is the opportunity to explore something about or create something related to the artwork. This could mean exploring materiality (for example, “crinkling” the foil in Pae White’s Eikón), activating the kinetics of a work (for example, taking Oldenburg and Van Bruggen’s iconic typewriter eraser for a drive), or exploring process (for example, dropping blobs of virtual molten glass for addition to Rob Wynne’s I Remember Ceramic Castles, Mermaids & Japanese Bridges…).
Third, we wanted to elevate visitor creations by putting on virtual exhibitions of their works. In some of the interactions in Norton Art+, you have the opportunity to “install” the virtual work you’ve created onto the walls of the Museum or in the sculpture garden outside in AR alongside the physical artworks. Other visitors’ virtual works are also visible in this AR layer, creating a sense of shared creation and allowing the otherwise hallowed walls of the art museum seem a bit more accessible to young visitors.
With these priorities in hand, the first major challenge of the project was identifying a list of contemporary works to consider for inclusion in Norton Art+. The Norton’s curatorial and education teams identified a dozen key works in the collection that held the most educational value and provided the broadest potential for interactive affordances.
From there, we went to work coming up with unique AR interactions for each artwork, ultimately generating more than 30. With this portfolio of possibilities in hand, we wanted to involve the community in whittling down our surfeit of ideas to the best six concepts.
For this, we turned to the Norton’s Teen Advisory Squad (TASQ), a team of 15 high school students of different ethnicities and backgrounds from all over Palm Beach County who collaborate with the Norton to discuss and design art programming for teens. We pitched the ten artworks and 30+ ideas over Zoom to the teens and listened to their feedback. We also asked them to rank their favorite interactions, as seen in the chart below.
Some clear winners emerged in this process, and the teens’ insightful feedback was incorporated into the final curation of artworks and interactions for the app.
Artist approval was the final step, which yielded the final six artworks and interactions:
- Pae White’s Eikón (2019)
This interaction invites visitors to recreate Pae White’s large-scale tapestry, which was commissioned by the Norton specifically for its Ruth and Carl Shapiro Great Hall. The tapestry cleverly uses trompe l’oeil to give the impression of a three-dimensional “crinkled” piece of metallic foil, and was originally conceived as a distorted reflection of the room itself and the large window opposite the artwork. Visitors point the iPad at the artwork to smooth it out into a reflective mirror in AR, showing a virtual reflection of the room and its window. Visitors then use their fingers to crinkle the smooth mirror into an abstraction resembling the original artwork. Once satisfied with their creation, visitors can save a snapshot of their crinkled creation in-situ in the Great Hall.
- Claes Oldenburg and Coosje van Bruggen’s Typewriter Eraser, Scale X (1999)
This interaction activates the iconic sculpture in the Norton’s entrance plaza. Visitors can “drive” a miniature AR version of the typewriter eraser, which erases virtual letters as it traverses the reflecting pool beneath the physical sculpture. With each letter erased, the AR eraser grows in size, until it matches the monumental scale of the artwork. This playfully explores the sense of kinetics and motion intrinsic to the work, and hints at the original functionality of an object that’s sure to be mysterious to younger generations. The visitor’s “drive” is captured in a time lapse video, which can be shared as an animated GIF.
- Nick Cave’s Soundsuit (2010)
This interaction leverages the”body tracking” functionality long associated with the Microsoft Kinect sensor, which has recently become available on mobile devices. Visitors within the iPads’ field of view are transformed into moving three-dimensional soundsuits, obscuring the visitor and suggesting the notions of masked identity present in Cave’s work. The app allows visitors to take a short video clip of a friend being transformed into a soundsuit, which can be shared as an animated GIF.
- Danh Vo’s We the People (detail) (2011)
Augmented reality is used in this work to reveal its conceptual origins. At a glance, this artwork would appear to the uninitiated as an abstract copper sculpture — in fact, it’s a 1:1 scale section of the Statue of Liberty, and part of a series of sculptures by Vo which are installed around the world. Scanning the artwork with an iPad reveals a small version of the Statue of Liberty, and invites users to “pick up” Vo’s sculpture and fit it like a puzzle piece into the virtual statue. This unlocks the ability to expand the virtual statue into dozens of three-dimensional pieces, some of which are other works by Vo from his We the People series. Visitors can then capture and share a snapshot of their favorite piece, or photograph a friend inside the expanded field of virtual Statue of Liberty fragments.
- Ugo Rondinone’s MOONRISE. east. november (2006) and MOONRISE. east. july (2006)
This interaction explores two large sculptures of anthropomorphic faces in the Norton’s sculpture garden. Visitors create a new virtual sculpture, using a special filter that creates similarly abstracted, whimsical faces based on their own facial expression, texture it in the style of the artworks, and then install it alongside other visitors’ works in a virtual exhibition in the sculpture garden. At the end of the experience, visitors capture and share a snapshot of their virtual work in the sculpture garden.
- Rob Wynne’s I Remember Ceramic Castles, Mermaids & Japanese Bridges… (2018)
This interaction focuses on a large-scale work commissioned by the Norton for its three-story Muriel and Ralph Saltzman Stairway. The piece consists of thousands of hand-poured glass pieces, inspired by the waves. Visitors pour virtual molten glass from a ladle in AR to create their own unique globule of glass, mirroring the process used by Wynne to create the work. Users can then pick up and choose where to install their “bubble” within Wynne’s composition, and also see bubbles created and installed by other visitors. Visitors can share a snapshot of the user’s bubble in-situ in the stairway.
Each of these six interactions also acts as a gateway to an “Artwork Info” screen, incorporating copy written by the Norton team highlighting interesting aspects of the artist’s process and the artwork’s themes. This screen also connects the featured artwork to other pieces in the Norton’s collection.
Though these concepts were identified early in the project, the exact interactions described above reflect their final instantiations. These emerged through an iterative process of prototyping, design development, software development, and testing as described in the following sections. Throughout these phases, the team remained committed to investing in radically different modes of interaction for each artwork. Instead of a one-size-fits-all paradigm, each of the six experiences and its visitor takeaway was tailored to the artwork.
Before finalizing the visual design and layout of the app, we built a prototype representing the “minimum viable product” implementation of the six interactions. This pared-down version of the experience helped us to evaluate the technical feasibility and experiential viability of the approaches — e.g. what is it like to “drive” an avatar in augmented reality, how should you steer? What does direct (but virtual) manipulation of an artwork hanging on the wall feel like on-screen, and what should happen if you pan the iPad away from the artwork?
Prototype demonstrations and findings were shared between Local Projects and the Norton over Zoom, and also via screen-capture videos.
Though these functional explorations were valuable, limited access to the artworks on account of the pandemic meant that certain aspects of the experience were not readily prototyped remotely. For example, we would have done well to spend some more time prototyping the nuances of how experiences are activated. When you are in front of an artwork, how should the visitor initiate the interaction? We wanted this process to be as seamless as possible.
In eras past, a QR code, a dedicated piece of signage, or a two-dimensional object like a painting might be used to kick off interactions when placed in view of the tablet or phone. We wanted to take a more seamless approach, leveraging recent improvements in three-dimensional object recognition to trigger interactions with sculptural pieces like Typewriter Eraser, Scale X. We briefly tested this capability using everyday objects readily accessible during lockdown (e.g. chairs, bookcases), and under these test conditions it seemed likely to work well. However, much later in the project the object-detection approach proved more difficult than anticipated after arriving at the museum for final installation, and some last-minute revisions and user-experience compromises were necessary to ensure reliable activation of the interactions.
Design Development proceeded concurrent to the prototyping efforts. The app’s visual design and structure was aligned with the Norton’s style guidelines. Norton and Local Projects’ content teams collaborated to identify themes and develop copy for the “Artwork Info” screens and other aspects of the user-interface copy.
Across the design, balancing the didactic with the experiential presented a challenge. To balance project resources, we opted to diversify the interactions (experiential) and templatize the artwork information (didactics). Also, wherever possible, didactic information was embedded in the structure of the interaction, for example, the We the People (detail) interaction reveals the wider context of the work in a collection when the statue expands to reveal other pieces in the series. Likewise, the I Remember Ceramic Castles, Mermaids & Japanese Bridges… interaction communicates information about the artist’s materials and process.
The concepts were informed by an understanding (and some optimism) about techniques that have recently become possible in the AR space. The Rondinone interaction’s high-fidelity facial expression tracking was improved by the depth sensors used on the front of iPads and iPhones to support Apple’s FaceID feature.
It is always challenging to set the right level of technical ambition for a project with a development schedule that is likely to straddle versions of the target device (in this case, the iPad Pro). Too much optimism can expose a project to risk if the future hardware fails to improve as anticipated, but too tame an approach can leave experiential value on the table.
For the most part, optimism payed off: Interactions involving tracking surfaces — as the app needs to do while you’re installing a glass bubble in the Wynne’s piece, for example — received a technical windfall when Apple introduced a new iPad incorporating a LiDAR sensor on the back midway through the project. This sensor greatly improves the tablet’s ability to map its environment, and provides more stable AR content that feels “locked into” the world.
Making the most of these recent hardware improvements meant leveraging Apple’s native software development frameworks and tools. This approach is also central crafting the app for maximum performance and accessibility, and the project enjoyed a major advantage by limiting the final deployment to a single hardware specification and to devices owned and controlled by the museum.
Limited opportunities to test the experience directly with the artworks during development presented some challenges for such a context-dependent set of interactives. Due to COVID-19-related travel constraints, a number of critical development tasks were delayed until the installation phase of the project. These included scaling and aligning AR content relative to the physical artworks, capturing three-dimensional object scans for use in artwork activation, and verification that artwork activation is working as designed.
Towards the end of the software development process, the team prepared a second demonstration and feedback session with Norton’s TASQ teens. We presented screen-capture videos of the app over Zoom, and then solicited feedback from each TASQ member. Their feedback was extremely thoughtful and informed a number of design decisions — including the approach to watermarking images and videos created in the app with the Norton logo, and how “steering” behaves in the Typewriter Eraser, Scale X interactive.
Usability testing is a critical step in the development of a new experience. Under normal circumstances, we would want to be on-site to observe testers while they interact with the app for the first time, and we would want this to take place in the exact context for which the app was designed. Since travel was ill-advised in the COVID-19 era, and social distancing remained advisable for the majority of the project’s timeline, neither of these typical testing requirements were able to be realized.
Despite these challenges, we still wanted to engage community members in the testing process, so we assembled a “usability testing kit” to use at home. The kit consists of an iPad preloaded with a work-in-progress version of the Norton Art+ app, plus a ~24 page printed guide. We did not want to bias the testers with too much explanatory information, so the guide tried to provide just enough background and contextual information to make up for the fact that testers weren’t going to be in the museum, standing in front of the artwork. For each artwork, the guide included a snapshot of the artwork in-situ at the museum, a short collection of questions specific to that interaction.
In the museum, each interactive is “activated” by pointing your iPad at the work of art, so as an at-home alternative we created a version of the app with a menu that allows testers to manually activate an art interaction in the absence of the artwork.
The kits were distributed to a number of Norton staff families and community members, who provided invaluable feedback about the app — particularly identifying instructional text that was unclear, and interactions that were too complex or too simple.
In the course of building this project, we were influenced by the scholarship of Professor Gal Zauberman of the Yale School of Management and his team. His 2016 investigation into how taking digital photos increases attention and heightens positive or negative impressions of an experience (Barasch, Diehl, Zauberman, 2016) helped frame our AR app not just as a medium for play, but also as a photographic lens that frames a visitor’s perception of the artwork, and challenges them to train their eye in the observation and interpretation of the artwork.
We also drew from Zauberman’s experiments on how visitors might report back on the overall impression of an experience based upon the duration and segmenting of timed experiences (Ariely, Zauberman, 2000). Our experience is broken up spatially, no experience is alike, and there is no reinforcement of a pattern or theme beyond that of play and immersion in different artistic processes. In this way, we hope visitors can enjoy and interpret each experience discretely, preventing the influence of macro patterns from influence any particular experience, and giving each visitor “recovery time” to prevent fatigue or sensory overload.
The Local Projects team is now in conversation with the research team at the Yale School of Management, consisting of professors and PhD students at the marketing department, to propose a study utilizing the Norton Art + app. The Yale team is interested in engaging with the Norton Museum of Art to conduct a study around two areas of interest using Norton Art+:
- How hybrid digital and physical experiences (e.g., AR), like the one delivered in Norton Art+, can generate enjoyment, memory, and engagement with the institution. As an initial hypothesis, the Yale team suggests that potential visitors will mis-forecast the value AR adds to the museum experience. A series of surveys, conducted prior to and following museum visits, will compare the anticipated and actual enjoyment from engaging with artworks with or without the Norton Art+ app.
- How digital pre-visit experiences can create a measurable increase in anticipation, curiosity, and desire to visit the Norton Museum of Art, as well as other outcomes. While testing how pre-visit experiences affect actual visits to the museum can be challenging during the current health crisis, we believe that insights from testing pre-visit stimulus can nevertheless be useful going forward. Therefore, the Yale team proposes measuring prospective visitors’ reaction to:
- Exposure to the Norton Art+ app case study on Norton.org
- Engaging with parts of the Norton Art+ app that may serve as previews of the museum experience, such as the MOONRISE face filter or the Soundsuit body filter.
The team is now developing a more specific proposal for the number of study participants, outcomes and measures of interest, and plans for study deployment. We hope and look forward to reporting back to the MuseWeb community next year with the findings of this study.
Norton Art+ was made possible by the generosity of the Anna-Maria and Stephen Kellen Foundation. The authors would like to thank the Norton’s Teen Advisory Squad (TASQ) for their thoughtful feedback at two critical moments in the project’s development. We would also like to express gratitude to the Norton staff families and community members who participated in the take-hope user testing process and contributed their feedback.
K. Diehl, G. Zauberman, and A. Barasch (2016) “How Taking Photos Increases Enjoyment of Experiences.” Journal of Personality and Social Psychology, Vol. 111, No. 2, pp. 119–140. Also available at https://internet.psych.wisc.edu/wp-content/uploads/532-Master/532-UnitPages/Unit-09/Diehl_JPSP_2016.pdf
D. Ariely and G. Zauberman (2000) “On the Making of an Experience: The Effects of Breaking and Combining Experiences on their Overall Evaluation.” Journal of Behavioral Decision Making, 13: 219-232. Also available at https://erationality.media.mit.edu/materials/Hedonic_Calculus/expseg.pdf
Millstein, Ben, Mika, Eric, Gregory, Meredith and Millstein, Ben. "Pandemic Prototyping and Augmenting the Art Museum." MW21: MW 2021. Published January 16, 2021. Consulted .