This document described the details of the setup presented in this video: A Live FX - Unity workflow
1. Setup your Unity project
First, setup a Unity project with the scene content that you want to use. Ensure that the project contains these 2 plugins:
Next to these two plug-ins you also need to the add the Live FX – Unity component. Download the LiveFX_OSC_Camera_Control.cs file from here and link to it in the project.
Create an OSC connection object. You can add this into your project through the Assets menu / Create / ScriptableObject / OSC Jack / Connection.
2. Setup your Unity camera
You require a camera in your Unity scene. Ensure to activate the Physical Camera option. This will enable your Unity camera to use sensor width / height as well as the focal length which can be live linked with the Live FX camera. In steps:
- Select the Unity camera that you want to live link. Add the sample Live FX component to your camera or drag/drop the Live FX prefab from the Live FX folder onto the selected camera.
- After adding the Live FX component, set the port number for communication between Live FX and Unity. Select the Live FX component in the Inspector panel and set the desired port number in the "Port" field. By default, port 9002 is used.
- Add a Spout sender to share the Unity camera image with Live FX on the GPU. Add the Klak Spout component onto your camera and set the target to the camera.
After these steps, start Play mode in Unity and then switch to the Live FX app.
3. Setup your Live FX composition
- In Live FX, create a new Basic Composition by clicking on the "Live Setup" option on the main toolbar in the Construct. Currently you do need to select a Video IO capture channel for the setup. In future version you will be able to directly select the GPU share as live input. Further, in the video example the Intel RealSense T265 is used as a 3D camera tracker. However, you can use any compatible camera or tracker to achieve similar results. Start the Live composition in the player.
- The next step is to add the Spout capture node. As mentioned, in the future you will be able to select the Spout share directly in the Live Setup panel. For now, create a layer and click the “Add Plug-in” option to open the Plug-in browser. There select the Spout GPU Capture and click the Add to Layer. You should now see the Unity image being captured.
- To send the camera transform, lens, and sensor data - activate it OSC Sender into Live Links panel. You can open this panel from the Live FX menu. After activating the OSC Sender, ensure that the port number to use is the same as you set in Unity (9002 by default) and then click Connect. All camera tracking data that is captured and from the external tracker and linked in the camera menu is now forwarded to Unity and applied to the Unity camera. So effectively you now control the Unity camera from the external tracker.
4. Look Development
Next, start adding effects and grading to create the desired look for the plate.
- Apply Effects. One effect you can start with is the Highlight Bloom effect, which adds a glow or bloom to the highlights in the image. To add the effect, apply it from the Plug-in browser on its own layer.
- Another effect you can use is the Lens (un)Distort, which can add lens distortion or more creative effects to your image and can be adjusted in real time.
- Color correction. Live FX offers a complete set of grading tools to adjust the colors and tones in your image. From standard lift, gamma, gain to vectors and curves, using unlimited layers of grading. Possibly use the blend modes of the layers to combine the multiple layers.
- Matchbox shaders are advanced shader-effects and behave similar as regular plug-ins. Matchbox shaders are not part of the default Live FX installation but a library with a big selection of effects can be installed (for free) from the startup screen of Live FX.
5. Record and instant playback review
One of the biggest benefits of using Live FX and Unity together is that all the processing is done on the GPU. This means that you can easily preview your effects in real-time without having to wait for time-consuming renders.
- Record your scene as a ProRes file at any time. Once you setup your composition to your liking, press the Rec button. Note that in the Live FX menu, in the Record tab, you can set the recording format: ProRes H264 or Dnx.
- When you stop recording, the recording is loaded into the version stack and ready for playback and review, simply click it.
Overall, using Assimilate Live FX and Unity together provides a powerful and flexible workflow for creating unique and dynamic visual effects. With Live FX's advanced tools and Unity's real-time processing capabilities, you can easily create complex effects and review them in real-time.