All of it Now Uses Sony’s Spatial Reality Display as a Visualization Tool for Virtual Production

All of it Now (AOIN) is a team of technologists and content creators based in Los Angeles, California.  They provide digital content and immersive experiences – from design, previsualization (previs) and virtual location scouting to augmented reality/extended reality (AR/XR) consultation and camera tracking data – for brands, agencies, entertainers, and events.  One of their most recent projects was creating an AR-enabled presentation of Disney’s Encanto which ran at the Hollywood Bowl, an outdoor venue in Los Angeles.  In order to bring an authentic experience of this magnitude to life, AOIN relied on Sony’s Spatial Reality Display (SRD), which enables highly realistic, three-dimensional content without the use of special glasses or VR headsets.

AOIN became aware of the SRD after seeing it showcased in Sony’s booth at the National Association of Broadcasters (NAB) Show.  Berto Mora, AOIN’s Chief Technology Officer, relayed, “Many of the people we work with in the industry and in the virtual production world told us we had to check out Sony’s new display.  It was one of the coolest things I’ve seen.”

The main selling points for Mora are the Spatial Reality Display’s integration with Unreal Engine, its capacity to showcase and visualize 3D content without wearing glasses, the ability to use it for virtual production, and the opportunity to showcase massive content in a realistic way from a tabletop device.  Shortly after receiving a demo, AOIN bought a unit to use in-house.

Danny Firpo, cofounder of AOIN, used the SRD to unlock new forms of creativity and help streamline their pre-production workflow.  Speaking of the Encanto show at the Hollywood Bowl, he explained, “First, we used the Spatial Reality Display to help visualize the Hollywood Bowl venue.  Additionally, because the content is motivated by projection, many of the effects used in the AR broadcast were particle-based.  Sony’s display provided visuals in both a 2D and a 3D space, enabling a better understanding of the paths that the particles were taking, which allowed us to spatialize and contextualize them.”

Beyond a better understanding of the particles, the SRD was also beneficial in finding camera positions.  Firpo continued, “It was really helpful to see the Bowl in a 3D space and use that for identifying camera positions and cable paths, collaboratively.  That 3D environment served as a vital pre-production tool.”  He added, “Using this display provided us with some special moments.  We coordinated closely with the director of photography and individual camera operators to show them the path in Unreal within the Hollywood Bowl, which was helpful for planning the cameras’ paths with the jib.  We were able to see dramatic plans as a result of seeing the camera movements in the 3D space.”

Nicole Plaza, AOIN’s Executive Producer, expanded on the abilities that the Spatial Reality Display offers.  She said, “Augmented reality is new for a lot of people, so having a tool that allows directors and others to visualize it and gives them an understanding of what we’re doing is important.  It allows them to be immersed in that 3D space and understand our work and workflows more quickly.”

Mora added, “We work in a lot of 3D spaces and 3D worlds through the engines, specifically Unreal Engine.  It’s difficult to convey what you’re looking at through a flat, two-dimensional screen.  The beautiful thing about the SRD is that you can now visualize that content in 3D and better understand it, instead of asking someone to imagine it in 3D.  With that comes the ability to play with depth and the perception of 3D with the human eye.  And that all makes the content more valuable.”

Firpo also appreciates the presentation abilities that the SRD offers to customers.  “The display allows us to literally be able to look around and navigate the spaces intuitively.  It creates a transparent technology layer for those who aren’t used to Unreal environments.  In addition, the Spatial Reality Display is very easy for us and for clients to use.  You don’t have to use any buttons to navigate it.  And at the end of the day, everyone appreciates the technology and loves seeing the content.”

Speaking of clients, AOIN has received positive responses to the SRD from creatives.  “Production designers often deal with the 2D space, so for them to see their designs in a 3D space before a final version is completed is helpful to provide context and spatial reference.  To also see those AR effects interacting with production design pieces is a very special thing for a lot of these production teams and we’ve received some really great feedback.  Beyond just seeing the final output or mockups, we can provide insight into the whole process,” explained Firpo.

He added, “The great part is, a lot of the artists that work on the content also get to go on-site and see it live.  I think it’s really fulfilling, and it creates a positive reinforcement that artists can see their work first in a 2D space, then the 3D space with the Spatial Reality Display during previs and pre-production and then finally live on camera during the broadcast.”

The Spatial Reality Display is comprised of several innovative features including a real-time rendering algorithm, a micro-optical lens, easy integration and a proprietary high-speed vision sensor.  Mora shared that one of his favorite aspects of the display is the eye-sensing technology.  “The gaze tracking of your eyes lets you know this isn’t a conventional display.  The real-time perspective shift is a huge selling point that makes it much more interesting.  This feature complements XR environments nicely.”

Similarly, AOIN appreciates the ability to view their content unencumbered by glasses.  “The display looks more realistic and presents content more accurately when you’re creating it.  It provides a different effect when compared to wearing glasses,” Mora explained.  Firpo added, “As somebody who already wears glasses every day, there’s a fatigue layer and added friction.  The weight of the glasses on your face limits the amount of time you can use them.  It can also cause strain on your neck, and glasses can hurt your face and make you sweat.  There’s also the potential for dirt and smudges to present themselves, which can degrade the viewing experience and affect the image quality.”

Firpo explained that his and his clients’ familiarity and experience with the Sony brand also contributed to AOIN’s decision to choose the SRD.  He said, “The Sony brand is trusted by us, our clients, producers, and camera operators.  Using a Sony display adds a level of confidence, as well as compatibility since we’re often using Sony’s cameras on-site.  To have a Sony pipeline that uses multiple different products like the VENICE and Spatial Reality Display gives us peace of mind that the color reproduction is correct and accurate.”

As AOIN continues their use of the Spatial Reality Display they’re excited by the opportunities it presents not only for preproduction, previs and AR but the additional potential it will provide for in-camera virtual effects for longer-form television and film, as well as virtual production – areas where the company, and the industry, are rapidly expanding.  With the promise that digital characters and metahuman interactions hold, the possibilities are endless.  Sony’s SRD will continue to help AOIN socialize these concepts to their clients in a highly realistic and impressive fashion as they continue to experiment with technology and push the boundaries of visualization.