When the COVID-19 pandemic arrived at the start of the year, it wiped out live events almost overnight. In the intervening months one thing has remained clear – we still crave those “goosebump” moments experienced during a live show just as much now as we ever did before. The world of virtual production is helping to fill the void and give us something just as thrilling.
While virtual production methods were already harnessed by a few companies in the live arena – with some using interactive ‘smart stages’ – from March 2020 onwards, they’ve become essential. Those already in the virtual studio space shifted up a gear, while others looked at their business models and realised that they, too, could migrate into this new arena.
Notting Hill Carnival, American Idol, the Emmy Awards, and Monsters of Rock Cruise (MORC) are among the hundreds of shows that have used VP this year to host real and often live performances in virtual surroundings. With technologies such as High Dynamic Range (HDR) cementing virtual production as a viable tool for immersive storytelling, viewers can now enter a virtual world limited only by their imagination.
So, what if a team experienced in live events wants to offer a new virtual space? It’s not a simple case of taking existing kit and rearranging it to make a virtual studio. The world of live events can look to the film industry where, in recent years, virtual production and technological developments have broadened the spectrum of what is visually possible with LED video environments. LED screens with high-quality LED processing are being used as replacements for traditional green screens, enabling content creators to capture both live action and CGI in-camera together, greatly streamlining post-production.
As with all on-set technologies, the choice of equipment has a huge bearing on usability and the quality of the result. The live events industry is very familiar with using LED screens as a backdrop for a live show, but now the LED screens must provide a plausible environment with high performance and a sense of depth when captured on camera, so that the audience at home believes and feels drawn into a 3D space.
For virtual production, the choice of LED panels and processor is typically considered far in advance of a shoot, and there are several important decisions to be made. For instance, pixel pitch needs to be carefully considered; it needs to be fine enough to minimise the chances of seeing moiré patterns, based on the planned shots and lenses, but not so fine that it drives up the price unnecessarily.
LED professionals would all agree that selecting the correct processing power is equally important as choosing the right LED panels, since the latter are tightly integrated with a particular processing system, with electronics built into each panel paired with a central LED processor that controls the whole screen and feeds it a video signal. This means it is important to select the correct processing system from the start, as it cannot be changed later if issues arise. As well as being responsible for the entire video pipeline within the screen, the processing system includes the user interface for configuring the overall appearance of the LED screen, ensuring it appears completely seamless, scales the content properly and shows precisely correct colours with no colour casts.
Typically, virtual production environments run LED panels at lower brightness levels than might be used for a live event. Running a panel designed to output thousands of nits at a tiny fraction of that level can result in banding and colour-casts, so it’s important to specify an LED processor that can compensate. Very few processing systems can achieve the performance required to deliver artefact-free images on camera, especially at low brightness, as most of these systems were designed for much less demanding applications like low-cost digital signage.
A camera test at an early stage is highly recommended to build confidence for both the creative and technical teams that the combination of LED panels and LED processing can deliver the desired performance. Testing should check for issues like rolling black bars, flickering, banding or poor colour quality.
The moiré effect is also a common problem when LED screens are too close to the camera or in too sharp focus, requiring careful lens and depth of field choices. Camera tests must be done using the same camera type as will later be used on set and trying out the same types of shots that will be needed. Similarly, the correct processing system must be used during these tests. All cameras and LED screens have their own characteristics, and it is important to confirm that the proposed pairing of equipment works well together.
SET UP FOR SUCCESS
A system will always be let down by the weakest link in the chain, so when pairing cameras and lenses, it is important to pair good LED panels with a premium LED processing system.
Selecting the correct processing system can give you a great deal of flexibility later on-set. This is crucial to enable delivery of the necessary image quality in what is often a rapidly changing environment.
Premium LED processing systems offer a full range of features such as the ability to freely drive the screen at any desired framerate, from 24fps up to high frame rates such as 144fps. Matching this to the camera framerate helps avoid motion artefacts, especially when shooting at high framerate or over-cranking the camera for slow motion shots. It is often also necessary to lock the screen refresh to the precise camera timing, and this can be achieved via genlock, with phase offset controls essential to avoid black bars appearing on modern rolling-shutter cameras.
Cameras see colour differently to the human eye and can pick up on colour casts or other distracting artefacts that destroy the intended illusion, so it is vital to have a full range of nuanced colour correction controls within the processing system. This ensures any unwelcome surprises, that only become apparent once on-set, can be addressed immediately so that colours are portrayed accurately on-camera. Making these adjustments directly within the display system ensures the best possible image quality can be maintained, as the display system operates internally at higher bit depth than the incoming video signal.
To date, almost all LED screens have only been SDR-capable, but now some are also able to display HDR (High Dynamic Range) content. HDR is particularly interesting for virtual production use. Compared to SDR, it offers much larger contrast ratios and colour gamuts that more closely mimic what we see in the real world, and what cameras are capable of capturing. This ultimately provides a much more lifelike final image when the LED screen is being shot on camera, translating to an improvement in image quality and realism, and offering more flexibility in grading than if using a ‘flatter-looking’ SDR screen.
Coupled with a processing system that can achieve accurate colours even for wide colour gamuts such as Rec.2020, HDR video content – whether pre-rendered or generated in real-time – can specify precise absolute brightness and colour values for every pixel, giving much better control over the display, and much more predictable results once the screen is integrated with other on-set elements, additional lighting, and even LED panels of different types or from different vendors.
Achieving genuine HDR performance on LED screens has proven surprisingly elusive until recently because traditional LED panel calibration methods limit the brightness and colour gamut of the LEDs to match the capabilities of that batch of LEDs. Recently, new ‘dynamic’ calibration techniques have been developed to overcome this limitation and deliver the necessary performance gains to achieve genuine HDR output.
As well as being the only way to deliver genuine HDR performance, dynamic calibration technology enables the LED screen to be reconfigured instantly to achieve different colour or brightness targets. This maintains colour accuracy and uniformity, while unlocking the level of flexibility needed in a professional environment to ensure a perfect image can be captured first time, every time.
THE NEW REALITY OF LIVE EVENTS
In the absence of traditional live concerts, livestreaming is becoming a vital component of keeping the event industry going, with state-of-the-art virtual production technology offering an engaging and immersive way of delivering content and live performances to an audience.
The era of digital storytelling is here, and while there is no doubt that the live event industry is a demanding environment for top-quality visual backgrounds, the unforgiving scrutiny of a studio environment is even more demanding on the capabilities of the LED screens used, demanding lifelike, HDR-quality content for the best performance on camera.
With this collision of the physical and digital realms, event companies are now expected to be able to deliver stunning visuals for any discipline, with the flexibility to adapt to many different setups.
Whether an event is taking place with a live audience in a physical environment with a giant LED screen as a backdrop or livestreamed from a virtual studio with a number of LED screens on all sides, those designing the show want to ensure that their audience is delighted and engaged the whole way through. We might not currently be able to have the thrill of being in a crowd, but we can still thrill with dynamic, creative new worlds captured on camera.
By carefully coupling the right LED screen with the right LED processing, companies are now able to bring concerts, plays and more to life, with technologies like Brompton HDR able to future-proof both physically and digitally delivered events, bringing those ‘goosebump’ moments to audiences, irrespective of the medium.
This article originally appeared in issue #256 of TPi, which you can read here.
