Photogrammetry of Complex, Metallic, And Mirror Like Objects – The Process, Trials, and Success of Charlie Parker’s Alto Saxophone
AbstractLearn how to capture difficult subjects with a full 3D production workflow from the Smithsonian Digitization Program Office (DPO) and The National Museum of African American History and Culture's (NMAAHC) Digitization Team, including the planning, data capture, post-production optimization, photogrammetric processing software, 3D editing tools, and the use of the Smithsonian 3D online viewer, for Charlie Parker's alto saxophone.
Keywords: Photogrammetry, 3D Processing, Online Interactive, Data Capture, Documentation, Workflows
Photogrammetry of Complex, Metallic, And Mirror Like Objects – The Process, Trials, and Success of Charlie Parker’s Alto Saxophone
Traditionally, photogrammetry fails to provide a robust capture method to faithfully reproduce surface properties for metallic and highly reflective objects. So how does one approach the photogrammetric data capture and subsequent digital processing for objects that challenge the standard workflows for photogrammetry? And what tools are needed to accurately prepare and visualize the subsequent 3D model for an online interactive?
In collaboration with the Smithsonian Digitization Program Office (DPO), The National Museum of African American History and Culture’s (NMAAHC) Digitization Team would like to present the process, trials, and successes that resulted in the published 3D representation of Charlie Parker’s alto saxophone. This highlight of the collection posed an almost impossible task for photogrammetry due to its intricate, metallic, and mirror like surface properties.
The How-to Session will walk through the workflow and processes for NMAAHC and DPO’s approach to such a complicated object. This includes the digital imaging capture process, post-production, and the necessary 3D processing steps to produce the 3D model. As well as a detailed look into the 3D editing, automated 3D derivative pipeline, and successful online publication using the Smithsonian’s Voyager online viewer.
Joseph Aaron Campbell
The first stage to any photogrammetry project is to get to know your subject. An in-person evaluation of the subject is a critical step in a process that can easily become complicated if the proper planning and preparation are not taken. When assessing the Alto Saxophone owned and played by Charlie Parker, the following key measurements, inspections, and analysis were made of the object.
First, physical measurements were taken to aid in the preparation of the imaging studio setup and lighting, but also for calculating the required Depth of Field for the available imaging system.
Second, the object’s material and physical surface properties were examined to determine if any aspect of the object will cause complications for the photogrammetry process. In this example, the bell of the saxophone along with other parts have highly reflective, metallic, and mirror-like qualities, whereas the majority of the body is a brushed metal semi-reflective surface. Knowing that the reflective areas will need to be handled differently is critical to effectively planning and creating a successful capture event.
Third, what else can we determine about the object when in its proximity? What is the condition of the object? Does it need additional support structures to safely handle or image? Is the object clean or has it not been given a proper review and assessment by conservation staff? In our case, the Alto Saxophone was initially viewed prior to conservation treatment. This was, in fact, to the benefit of the photogrammetry process as the mirror like surfaces had a small amount of grime which provided a minimal amount of detail to work with. Thus, the photogrammetry for the Alto Saxophone was performed before the object was cleaned.
Knowing the attributes and physical character of the Alto Saxophone allowed for an appropriate plan and approach to be devised for successful outcome.
With the assessment and evaluation of the object comes the second stage of any photogrammetry project, Which is to create a written plan and description that provides how and when the project will progress, who the stakeholders are, and what equipment or software is needed to complete the project.
To start, determine which personnel are directly involved. For example, who is on the imaging team? Will there be any collections management staff on hand handle the object? Knowing who to communicate to and when only serves to bolster the success of the capture event as well as helps generate a greater sense of support or buy-in for the project.
Knowing the key personnel, define exactly how the capture event will unfold and define the overall logistics of the event. This includes answering many questions about the timeline of events and dates for the project, clarifying object handling procedures, transportation of equipment, as well as any other action that may impact the success of the capture event prior and during the project.
When working on the Alto Saxophone it became apparent that many of these steps had been overlooked from when the project was requested and leading up to production. From that misstep many small mistakes were made that had large impacts on the timeline of the project. One such example: the Alto Saxophone was assembled incorrectly before imaging. This had a major impact on the post-production and introduced many delays to the delivery of the final 3D model.
Once the big picture has taken shape, what remains is to describe each subsequent imaging flight pass the camera or imaging device will make, and the equipment and resources needed to complete each imaging pass. The Alto Saxophone required a special “room” built to control the highly reflective parts. To develop a controlled environment for imaging, questions you may find yourself asking are: What equipment is needed? What modifications to the imaging space are required to achieve this? How is the lighting equipment going to be positioned and modified to support this controlled environment?
Before finalizing any plans for the capture device, imaging space, and lighting setup, a critical action to perform is calculating the Depth of Field requirements. Depth of Field (DOF) is the measurement of the distance from the nearest point of focus to the farthest point of focus attributed by the combination of the imaging device’s sensor size, the lens focal point distance, and the chosen lens aperture. For photogrammetry, your goal is to produce an imaging setup where the full length and dimensions of the subject are consistently in focus as a result of an appropriate depth of field. For the Alto Saxophone, a Depth of Field calculator was used to define the distance of the camera to the saxophone. Then the subsequent studio and lighting setup was finalized using that calculation as a basis.
Following the written plan and having a familiarity with the subject matter will make the imaging and data capture run smoothly. Entering the third stage of a photogrammetry project, for each planned imaging pass, the most important aspect of the imaging stage is in fact consistency. Which means that at every level of process and motion, there is a level of control that extends and encompasses the entire data set. For the Alto Saxophone, this included consistent camera and lens settings, exposure and lighting, distance from camera to subject, as well as capture patterns.
Combine consistency with the appropriate imaging passes and the capture event becomes quite simple. For the Alto Saxophone, there were multiple imaging passes that followed three main capture patterns. The first pattern utilized the 35mm lens, a turn table, a fixed camera distance from the saxophone, and four distinct camera heights and angles. The second pattern was a flight path utilizing the 50mm lens where the camera was moved from left to right parallel to the saxophone at a fixed distance. Between each subsequent repetition of this pattern the height and angle of the camera was changed to adjust the overlap and coverage of the resulting image data. The third pattern utilized the 100mm lens and was completely hand held. This pattern’s intent was to capture as much of the complicated details, etchings, and surface details as possible.
When the capture process is complete, the fourth stage for a photogrammetry project is converting the RAW capture data into a format the photogrammetric software can understand. For the Alto Saxophone, a standard DSLR camera was used so the data was formatted from camera RAW files into rasterized TIFF images using a software named RawTherapee.
When performing this conversion, it is important to take into consideration the exposure and aspects of the image data. Because the saxophone contained highly reflective surfaces this naturally increased the localized contrast of specific surface areas and more importantly introduced a variability between images, caused by the specular highlights moving along with the camera angle and position. To counter this variance between images, and further the consistency of the exposure between images, I applied a reduction in overall contrast and an increase to the L* curve’s lightness channel. Although this action made the images unpleasant to the human eye, it helped optimize the data for the photogrammetric software.
This is a great question I am currently still working on. When preparing image data, how do we best optimize the rasterized TIFF images for the various algorithms that generate the 3D model?
For the fifth stage of the photogrammetry project, the converted capture data is imported into the photogrammetric software and processed. This can often be a simple process from start to finish. However, due to the physical nature and surface properties of the saxophone, there were multiple challenges to overcome when processing the data. Then there were minor mistakes made during imaging and during the assembly of the Alto Saxophone that further complicated the required actions to produce a quality 3D model.
Using Reality Capture as the photogrammetry software, I opted for a component workflow supported by a series of manually placed control points to overcome the mistakes made during and before image capture. The component workflow consisted of processing each side of the saxophone separately, and then exporting each separate part as components, only to join the two components as whole point cloud.
The control points were used because the mouthpiece and neck of the saxophone shifted sometime during imaging. This created misalignment for specific chunks of the image data. However, by manually placing the control points on specific features of interest, Reality Capture was the successful in bringing the remaining data together.
Normally, such a highly reflective object would immediately pose problems for the photogrammetry software. However, because the studio setup, lighting, and imaging passes took into account the surface and material properties of the saxophone, those problems were majorly subdued.
One additional layer to this specific project is stage six. Due to the saxophone being assembled incorrectly before image capture, the resulting 3D model from the initial component workflow had to be modified to fix this digitally. To do this effectively, the 3D model’s Diffuse texture was generated within Reality Capture for a high-resolution version of the model. That initial 3D model was exported and then imported into the 3D editing software Blender. Within Blender, several standard processes of filling holes in the mesh and removing stray 3D data were performed. Once ready, the Mouthpiece was then split off of the main 3D model, rotated, and then re-joined with the main 3D model.
Once completed, several 3D texture maps were baked (generated) using the high-res mesh and a decimated lower resolution mesh. This included the Normal Map and the Ambient Occlusion Map. The final action was to use the Texture Paint feature in Blender to manually create a Specular Map.
The final stage in the photogrammetry project is delivery. In this case, the lower resolution 3D model was sent to the Digitization Program Office (DPO) to be processed and prepared for the Smithsonian Voyager online 3D viewer.
The DPO has a semi-automated pipeline that converts large high resolution 3D data sets into derivatives useful to end users. In the case of the Alto Saxophone, the data was converted to a web-ready 3D surface model with the following steps:
Decimation: The model was originally over one million triangles. Our standard for web-ready models that can be reliably viewed across devices is one-hundred and fifty thousand triangles. We used the open-source software Meshlab and the Quadric Edge Collapse technique to decimate to the necessary size while maintaining the integrity and detail of the mesh surface.
Unwrap: The decimated mesh is then assigned texture coordinates to each of its vertices, or “unwrapped”, so that textures can be appropriately mapped to it. We use a commercial software called RizomUV to do this assignment while making efficient use of the texture space.
Texture Bake: The textures assigned to the high-resolution mesh must then be transferred accurately to the low-resolution version. This involves projecting (or “baking”) the images from one geometry to the other so that the images can be reconfigured to match the texture coordinate layout of the simpler geometry. For this process we use the freely available application xNormal. During this stage we also generate and “bake” normal and ambient occlusion maps so that we retain this high level of detail for the final product.
Texture Resize: Along with controlling the complexity of a web-ready mesh, we must also make sure that the textures being used are a reasonable size for use with modern display devices. Our standard for high-resolution web models is 4k by 4k pixels. We also create medium (2k) and low (1k) versions for less capable hardware. This step can be done with any image processing software.
Conversion: We then convert the model from its original format (OBJ in this case) to a compact and web-accessible format. Our choice for this is a Draco compressed GLB, the binary form of the glTF web standard for 3D. We use our in-house open-source application Meshsmith to perform this conversion, though other applications can be used.
Prep for Display: The final stage of the post-processing pipeline is to generate a scene file for our web viewing platform Voyager. This scene file allows us to set properties like lighting, model orientation, camera position, and add educational components like annotations and tours. Important to the Alto Saxophone scene was the addition of an environmental reflection map so that the final display would include photo-realistic reflections that highlight the material properties captured in the initial steps of this project.
Alto Saxophone Owned and Played by Charlie Parker, last updated August 1st, 2020, Available
DPO Cook – 3D Model/Geometry/Texture Processing Server Github repository, last updated Monday, February 8th, 2021, Available
DPO Voyager – 3D Explorer and Tool Suite Github repository: last updated Monday, February 4th, 2021, Available
Campbell, Joseph and Cope, Jamie. "Photogrammetry of Complex, Metallic, And Mirror Like Objects – The Process, Trials, and Success of Charlie Parker’s Alto Saxophone." MW21: MW 2021. Published March 11, 2021. Consulted .