07. Live FX - VP Checklist

Expand / Collapse
 

07. Live FX - VP Checklist


General

Virtual Production projects come with its own set of challenges, including time and budget constraints. This document provides a checklist for Live FX usage to prepare for a project and to diagnose potential issues during a shoot, to help to prevent and if needed to deal with unforeseen circumstances effectively. Although the focus of this list is on LED volume shooting, it is also applicable to green-screen projects.

Preparation

View the Live FX tutorial videos and read the user guide. Live FX is build as part of the Assimilate Product Suite, with a long tradition in (post)production. It is packed with functionality and is a toolbox for a wide range of workflows: nodal and layer compositing, colorgrading, rendering, editing. All of which might come into play in a virtual production. The best guard against unforeseen circumstances is flexibility.

Create the LED volume model in the Stage Manager. The Stage Manager is the heart of the Live FX LED volume projection. Create a model of your LED walls. If you have a non-standard shaped wall, ensure that you have a correct .obj model file. Use the model view (eye-icon button) option to check the UV-mappings of the .obj files. Note that you can do this on any system / laptop. The model setup is stored in the stageconfigs file in the c:\programDATA\assimilator\settings folder and can be copied from one system to system. Do note that the paths to the .obj files need to maintained.

LED Volume Mappings. The Stage Manager again. Create and test the mappings for the Live FX output to the LED wall processors. This should be done well ahead of a shoot. It is something you likely probably have to do only once for a LED volume but it might take some time and possibly require adjusting the mappings in the LED processor. Use the Preview option to validate the mappings.

Nvidia Mosaic. If you are outputting directly from the GPU to (multiple) processors, using the Dual Head as opposed to Video IO, then you probably need to use an Nvidia mosaic setup. Make sure you are familiar with setting up this mosaic. It is know that in some cases you might have to re-create the mosaic on the spot.

Camera tracking system. Ensure that the available tracking system is supported in Live FX: if not directly then most tracking systems also allow to output using the FreeD protocol. Check if the tracker uses on the same network as the primary network adapter of the Live FX system (see below on multiple network adapters). Check if the tracking system is calibrated or needs to be calibrated using the Live FX tools to determine the origin-point of the set (using an Aruco marker). Once the origin position is known, then ensure that the LED walls in the Stage Manager model are positioned relative to that point. If the tracking system needs to be calibrated with the Live FX tools, then schedule enough time for this ahead of the actual shoot, especially if you have not done this before.

Camera specs. Familiarize yourself with the camera is that is going to be used. What is framerate that it is shooting – to make sure the LED wall (processor) and Live FX operate in the same rate. Normally the camera and the LED wall (processor) are genlocked. Unless you are using multiple Live FX systems to drive large wall volumes, Live FX itself does not need to be genlocked. What are the specs of the lens that will be used: what is the focal length, which it important to determine the frustum projection on the LED wall or the green-screen background. In some cases the camera tracking system includes the lens calibration to determine the effective focal length (and lens distortion), in other cases you might have to use Live FX to do this calibration. In both cases, store the lens settings in a camera Profile in Live FX so that it can be easily used when creating a projection setup of live capture.

Color(space). Live FX handles all color management to ensure that the image is send in the correct format to the LED wall processor. In some cases a specific external LUT might be required. Ensure that Live FX supports the LUT format and that it is known what input-colorspace the LUT is expecting so that you can apply it the correct way in Live FX. Potentially use OpenVpCal to do a LED wall – camera calibration. Live FX wraps the OpenVpCal calibration application and can apply the results directly as display LUT.

Network. Changes are that Live FX is operating in one or more networks on set: main computer network with storage, camera tracking, IBL/DMX network for Stage Lights. Check what the network will be and the consequences for Live FX. Not all supported camera trackers can run in a separate network and need to run in the primary network the Live FX system is connected to. Detecting ArtNet nodes in a non-primary network might also be problematic – potentially switch to ArtNet broadcast. WIFI networks usually do not prove the same performance or speed as a wired alternative – this might be a problem when e.g. sending very large amounts of DMX data, sending/capturing high resolution (NDI) video streams.

Source Media. Ensure that the media that is used for projection of the LED wall (or green-screen background for that matter) is properly prepared. If the media was recorded in a different framerate than that it is to be used, consider re-rendering it ahead of time at the correct framerate using a quality re-timer, rather than just playing it in a different framerate and risking skipping / repeating frames.

  • Ensure that the media can playback in realtime. If not, e.g. due to very high resolution or non-optimized format, consider transcoding it ahead of time to a more suited format for high resolutions like NotchLC.
  • Similar aspects apply when the projected media comes from an Unreal scene: ensure it plays back in the same framerate as Live FX (and the LED wall processor). If performance prevents realtime playback consider running on a lower resolution and upscaling the image in Live FX. Or consider playback of the scene from a separate system.
  • Select the proper projection model with the media that you are using: frustum, planar, spherical. Try creating the projection setup composition ahead of time and familiarize yourself with composition layout. Optionally use the Stage Manager to verify the projection and get a preview of what to expect on the actual LED volume.
  • Color grade the media ahead of time rather than having to do a full grade on the spot. You can easily create different versions ahead of time that you can present and do a final tweak on, on the shoot day itself. When the grade has a substantial performance impact (e.g. using plug-ins, heavy blur on high resolution images), then consider rendering the graded media out ahead of time and use that instead of the raw media.
  • When using an Unreal scene, consider a packaged project to make it easier to manage with a smaller footprint then when using it from the editor. Do not forget to ensure that the Live Link in the packaged project is enabled and started by default.
  • In a frustum-projection Live FX uses by default the inner frustum image also for the outer frustum. Consider if you require special footage and/or grading for the outer frustum image for reflections on the scene. Live FX can play and grade separate media for the outer frustum.

Image Based Lighting / DMX. In your light-plan include the DMX universes and channel mappings required for Live FX to use, as well as which protocol: ArtNet / sACN. Note that the number of ArtNet universes from Live FX is limited to 16. sACN does not have that restriction.

  • Prepare channel mappings for the fixtures used – possibly using GDTF or Open Fixture Library templates. Once you have the correct mapping for the fixture/mode, it is easily replicated when setting to create the full setup during pre-light.
  • Determine the source for the color sampling for the fixtures / image based lighting. This will depend on the full setup: is Stage Light for IBL the same Live FX system that also provides the LED wall projection or green-screen background, or does it get a live feed from another (Live FX/UE/other) system – possibly through SDI/NDI video capture? When running both functions in a single system, you still might have to add a media stream to the projection setup to get the correct images for the lights: if the lights are in front or to the side of the scene then you might not want to sample form the background image. Ensure you extend any prepared projection setup with the additional media.
  • Having both projection / background rendering and dmx light control on a single system is often very doable from a performance point of view. However, both tasks are often performed by different people and having more than one person operate the Live FX interface is less optimal. There are different ways to tackle this problem. Use a second system and either duplicate the media or stream content from the projection / background system to the stage light system. Alternatively, prepare the lights setup as much as possible on forehand and use the http interface to tweak fixtures in Stage Lights remotely. Use and external lighting console and use Stage Lights purely for sampling colors / images that are send to the console.

Set-Extensions and recording. When planning for doing a set-extension or doing a recording in combination with an LED volume projection there is a performance aspect to consider and to test ahead of the shoot when doing all from the same system. Test if the system is able to capture a live stream and create the set-extension and/or record it to disk, without interfering with its primary role of projection media in realtime on the LED volume. Potentially consider moving the recording and/or set-extension to a separate system.

Backup. The exact backup infrastructure and procedures are beyond the scope of this document, but just making sure all media, stage manager setup, Live FX project including all camera profiles and fixture definitions are on a backup disk seems live a relative easy step that can save lots of time in case of disaster.

Diagnose

No image or incorrect part of the image showing on the LED wall

  • Does the Live FX startup screen show on the LED wall? If not then start by checking the physical connection. Ensure that the Dual Head option is enabled in the system settings when outputting directly from the GPU. When outputting through Video IO, open the Video IO panel and check if the devices are enabled as well as the output channels and the correct format is selected.
  • Does it not show an image when just playing back a clip but not when creating a projection setup? The issue is most likely with the projection mappings in the Stage Manager or the mapping in the LED processor. Check the Stage Manager and use the Preview option to check how that looks. In case you have multiple LED walls defined, check the Channel Controller (Displays tab) to see it the correct channel is send to the correct display. Check the LED processor to see the image it is getting. Then check if that image is mapped correctly to the LED wall.
  • When using Frustum or Planar projection, check the position of the camera as well as the size and position of the planes as defined in the projection node. Easiest is to open the Stage Manager panel while in the Player. Is the camera visible in the model? If not, ensure that the camera on the root node is active (Camera menu). Is the camera in the expected location in the model (not directly in front / behind the LED wall)? Check the Camera menu if the camera is tied to a tracker or appropriate position values are set. Are the planes visible (for Planar projection)? Navigate to the projection node and adjust the size and position of the plane used for the projection.

Choppy playback or tearing

  • When going out directly from the GPU (Dual Head), ensure that the Vertical Sync option in enabled in the Player – Settings – General menu.
  • Ensure that all parts of the pipeline operate in the same framerate: Source media / UE scene, Live FX Player, LED wall processor (,Genlock, camera). Ensure that if Video IO is enabled in Live FX that it also uses the same framerate. If enabled, Video IO will by default override the playback speed of any media or player setting. Disable Video IO if not used as the main output or, if you need Video IO but on a different framerate – adjust the Sync option in the Player-Settings-General menu.
  • Check the Performance monitor by using quick key CTRL+F1. Is the stutter visible there as well as on the main UI screen? If not then the stutter is likely causes further downstream, in the LED processor.
  • Does the stutter show on regular intervals or at the same position in the media? Double check the framerate settings again. Check if at the stutter position no additional grade (animations) are initiated.
  • Does the same stutter show up with other source media – preferably more lightweight? If not, then this might be a performance issue – see below.

Playback Performance

  • To see more details on the actual playback performance, open the performance monitor with quick key CTRL+F1. If this shows a straight line but not at the expected framerate then check the framerate of the root node in the player (Player-Media menu). If you entered the Player with a timeline, then the framerate of the timeline is used for all shots, rather than the framerate of the individual shots. Check the timeline framerate in the Construct. If the framerate is still seems locked to a different value, check if no Video IO device (SDI/NDI) is enabled on that framerate – see the Video IO panel which can be opened form the Player – Settings menu.
  • If the performance monitor does not show a straight line / regular pattern and is not realtime, then likely the system cannot process all the media fast enough. Then you need to find out what the bottleneck might be. Ensure that all other functions than playback are off: recording, Stage Lights, Audio.
  • Try playing back the media without any projection or even outputting to Dual Head or Video IO. If this is not realtime, consider a different codec/format or lower resolution (and use scaling by the projection node).
  • Check the Taskmanager to see if a bottleneck is obvious: is the CPU or GPU performance maxed out? A maxed out CPU often indicates a lot of decoding time. Consider an alternative format. A maxed out GPU indicates a lot of grading time. Consider rendering the graded shot and use that in the projection. More often though the bottleneck might be with reading from disk or transferring data fast enough from main memory to the GPU. In these cases consider using lower resolution media, which is scaled up on the GPU. Alternatively, if possible use faster hardware or consider using an extra system. That latter option brings with it some more complexity because you need to sync both systems.

Display Flashing and/or Showing White

  • If you are using multiple GPU's and the Main display is not on the first GPU or the dual head is on a different GPU than the main interface - then ensure that the NVidia settings are correct: Open the NVIDIA control panel, go to “Manage 3D settings”, scroll down to the “OpenGl Rendering GPU” setting, set that to the GPU used for the main UI.

Contact Assimilate Support

  • Ensure that you have a full description of the issue and include a log file of Live FX of the relevant session. If the issues are performance related, also try running a Profile with the media that gives the issue. You can run a profile from the Player-Settings-Profile menu.


Rate this Article:


Details
Last Modified:Saturday, July 20, 2024
Type: INFO
Options