4.6 Visual Effects: Tracking
A little background on noise and how it affects tracking
Noise has a huge impact on tracking, especially with film, because the random appearance of film dye clouds will be different for each frame of the film. As the camera moves through the scene the object will move around on the image sensor. On top of this, the film dye, or ‘pixel’ is also moving around on the image plane as well.
For the Arri D21 the pixels on the sensor stay stationary as the camera moves around; this makes the D21 (in theory) easier to track. The noise from the D21 should not have too much of an impact on the tracking ability.
However, for the Vision3 500T 35mm film there are 3 color layers, each with different noise characteristics that could affect the track and have a serious impact on the tracking software. For this reason we will try several tracking methods and see how each color layer impacts the track.
Each camera will go through:
A track with default settings (also known as a luminance track)
Red channel track
Green channel track
Blue channel track (probably will have the most impact)
Because the scope of this project is focused more on the cameras’s abilities rather than the software’s abilities, the tracking nodes in Nuke were left on basic settings.
Tracking
Sharpness, obviously, has an impact on the ability to track footage. Looking at the MTF plots from the Sharpness post, film has an unstable falloff of sharpness. Which in turn, could cause problems for the tracking software; and comparing the film plots with the D21 plots, digital should have an overall better track.
D21 and Film Noise
Macbeth Color Chart Patches
The best place for footage to be, is somewhere in between a sharp and blurry image. Sharpness really helps the tracking software to stay on track with whatever mark has been chosen; however, if the image is too sharp, it can increase noise which can cause the tracker to jump around and track the noise rather than the tracking object.
Sometimes it is good practice for the artist to put a slight blur on a shot to reduce noise so that the tracker can stay on point.
Noise
Table IV
Looking at the MTF curves and the noise calculations from both of the cameras, a couple decisions can be made about how well a track could hold on this type of footage.
This is not to say that the numerical difference between the two camera systems will give a better track by a certain amount, but for the most part the noise from the camera is going to have a larger impact on the track than the D21 footage.
Look at the difference in the red channel (14.64); the film has a chance to track better than the D21. However, this is only taking half of the problem into consideration. Another influence on tracking is sharpness, or the MTF of the camera system. Looking at the MTF curves of each camera, it can clearly be seen that the film system is not as good as the D21; however, for tracking this actually proves beneficial to the film!
The soft image from the film helps keep the trackers relatively on point.
The markers were tracked at the center of each tennis ball. Colored dots, representing each color (with black as the luminance channel), were applied to the tracking data within Nuke. The dots were shifted so that the black, luminance dots were located in the center of the ball, red on the left, green on the top and blue on the right.
Fig. 19-1. D21 Track
Fig. 19-2. Film Track
Notice how most of the channel trackers have fallen off point
The black dot on the third ball of the D21 track has shifted off center representing that the luminance tracker shifted during the tracking process; whereas all of the dots on the third ball of the Film track have completely been displaced!
Remember, this is just a basic track. The tracking can be fine tuned, but this is not the purpose of this test.
During the shot, the C-stand with the marker on it fell slightly; and even though the red dot looks like it stayed with the ball, it should be located more towards the top of the ball. This shows how the noise and grain within the film, is affecting the tracking ability!
Perhaps the reason why the green dot drifted off the tracker can be attributed to the fact that this was shot on a green screen using yellow tennis balls. So it is expected that the green channel would not have the most successful track that well. In fact, the Red channel for the D21 footage tracked the best, it was both quick and the most accurate.
Look at the Film track [Fig. 19-2. Film Track], the tracker has drifted from the ball in this frame. The tennis ball fell down about a foot, and the tracker should be able to detect this change but it does not. After several different attempts the tracker just does not want to follow its mark. This leads to the question of whether or not the software is really tracking the marker or if it is tracking a clump of grainy noise.
This frame is one of the more exaggerated frames in the sequence, but it does show how far the tracker can drift off point due to noise. For large motion picture features, the film goes through several weeks of noise reduction; which our film did not have time for this process. As mentioned earlier, the softness of focus from the lens actually helps keep the trackers fairly on point. About 9 of the tracker dots stay on their mark fairly well, and this shows how much a soft shot can help for tracking a simple shot.
3D Tracking
Noise can also affect 3D tracking. It may not have such a huge impact as it does with normal 2D tracking; but it depends on which software is being used. Some tracking software uses predictive geometry to not just track a scene, but actually calculate and recreate the objects within it. **
Noise can give a small amount of jitter to mess up a 3D solve. The bigger problem with 3D tracking is lens distortion, and because an older lens was used on the film camera, this could have contributed more to the problem rather than noise. This is why when looking at both of the 3D scene recreations, the film camera could not reproduce the scene, even with large adjustments.
The picture on the left resembles colored points in the scene. C-stands, tennis balls, the net, and the Macbeth Chart can be seen; whereas in the film track, points are produced that do not resemble the scene.
