Feature Films VFX
The Thing – The Fantastic Four: First Steps
Technical Innovation
Masquerade3
Digital Domain
digitaldomain.com
Masquerade3 is a new facial motion capture system for helmet-mounted cameras (HMC) with impressive efficiency gains from its predecessor. It is a proprietary system offered as a service and currently not available as a licensed technology. It was first used in Marvel Studios' Fantastic Four: First Steps, at an unprecedented scale, processing roughly 60 hours of performance capture across three main characters (to contrast with 31 minutes required for Thanos in Avengers: Infinity Wars using the previous system). That work was done on the raw footage material, before the mocap selects were done and irrespective of which vendor would be assigned for the VFX work. This shift meant that previz, postviz, and VFX vendors could access the data at any stage of production without requiring additional solves later in the show.
Typically, facial motion capture is about tracking the features on the face, like the markers or pore details, and turning them into animated 3D faces. That is a process that requires setup time on set and is very sensitive to precise camera calibration, lighting conditions, and footage quality, resulting in an expensive workflow with a significant overhead for manual corrections. As a result, the processing of the material is reduced to the minimum necessary and is typically done only after the mocap select is complete, resulting in production bottlenecks. Instead, we use a holistic approach, and train actor-specific machine learning models to process large volumes of data, which allows for the scalability just mentioned and bypasses the limitations of the traditional methods at comparable quality level.
Masquerade3 uses machine learning to train a model on the actor's footage, such that it becomes a specialist on recognizing this person's facial expressions, even with different appearances, such as unshaved beard, and in a wide range of lighting conditions. That is done using a semi-supervised training technique developed by the research team, which learns how to map the actor's expressions to a 3D character in an automatic manner. As a result, it eliminates the need for markers on performers’ faces, with greater efficiency on set, reduced stress on the actors, and broader creativity. This approach also enables robust and flexible facial motion retargeting to CG facial rigs with high quality animation and easy integration in VFX workflows.
In Fantastic Four: First Steps, the plan was originally to use Masquerade3 to process the material for The Thing. Soon after the initial tests, it became clear that the solution could easily extend to other characters, so additionally, it processed material for the Herald and Galactus, demonstrating that the system could deliver faithful human-like performances as well as more stylized characters.
Masquerade3 has advanced the way movies are made with impactful benefits for production, actors, directors, and the studios (clients), witnessed during Fantastic Four: First Steps:
Masquerade3 ensures the actor’s performance is not limited in any way. This means the actor can perform in the most natural environment, live on stage, without painted markers on the face nor having to shave the beard, interacting with other actors, in sets with dynamic lighting whilst still capturing a facial performance at the fidelity of a seated 4D rig.
The setup time on stage is reduced to a minimum, without the need for camera calibration and marker placements, largely simplifying the recording sessions.
The system is very stable and is not compromised by common issues affecting traditional approaches that could lead to unnatural character expressions or deformations that change the CG character identity, typically only detected at later stages in production.
The automated results speed up the VFX workflow providing early on inputs to all departments before even the final edit is defined, avoiding common production bottlenecks.
The method can benefit from actor-specific 4D data when available but it also works without it, and instead, leverages the automatic machine learning retargeting method for puppeteering a facial rig with standard animation controllers, ideal for VFX work. Indeed, the data processed for Fantastic Four: First Steps utilized direct retargeting from actors' performances to their respective CG characters.
Masquerade3 was successfully applied in Fantastic Four: First Steps and helped all vendors and Marvel access results at very early stages. We believe it represents a fundamental paradigm change for facial motion capture, bolstering flexibility, robustness, and efficiency.
Masquerade 3 streamlines production and ushers in a new era of efficiency and creativity in VFX.
SOFTWARE
Masquerade3 - Digital Domain proprietary tool