Wednesday, 16 May 2012

Tutorial: Camera Solving for Moving Objects


We've had a few requests to make companion written tutorials alongside our video tutorials.  
So for those who prefer reading step-by-step instructions, rather than examples where you have to hit pause all the time, this is for you.

When doing camera solves in mocha v3, the most common workflow is to track planar areas in your shot that are not moving, so you can define how the camera is moving relative to the non-moving objects in the shot.
This is fine for when you need to insert non-moving or animated objects that are relative to your camera, but sometimes you are looking to replace objects that are moving in the shot, or attach things to them.
Mocha handles this process by still solving the camera how you normally would with static areas, but then you can track any moving planes and export those planes based on your camera solve.  

In the surfing shot below we want to add a particle stream to the surf board.  We need to track the non-moving background first and then get 3D surface data for the surfboard itself.

The footage we are using is courtesy of Rubberball footage supplied by
You can grab a small sample from this footage to follow along here.

Let's take a look at how this is achieved.

The Process

First of all you want to track your static (non-moving) objects to define a camera.  You will need to define more than one plane for moving (xyz) cameras, and at least 1 plane for cameras that are fixed, but can Pan, Tilt and Zoom (PTZ).

To reiterate, "static" in this sense does not mean a locked-off shot with no motion.  Static means anything in the shot that is not moving, such as a floor or wall.  To solve for a camera we need to determine how that camera is moving relative to stationary objects in the shot.
Where possible, it is good to get as much information as you can (without overdoing it).
In this shot, we are using 2 portions of the sky using the Add Spline to Layer tool to create a second shape in the layer.

001 - Draw shapes in static planar areas
Before we track we also need to animate the shape to avoid the movement of the man and the surfboard.  This is necessary because we are going to do an unlinked track.  We could also just mask out the man and the surfboard by drawing layers over the sky-tracking layer, but animating the shape is faster in this instance.

002 - Adjust shapes to avoid foreground motion
So we can have the splines in the layer not moving while we track, we can set our Link to Track option to None.  This means that as the footage plays back, anything passing underneath our spline "Search Areas" will be tracked, without having to move the spline as well.

This is very useful when we have long pans and don't want to keep animating the shapes back on screen.

003 - Choose "None" from Link to Track so that the splines don't move.
We can then set our parameters and start tracking the sky.  I have gone with:
  • Motion: Check Translation, Scale and Rotation
  • Min % Pixels Used: 90% (The higher the number the more accurate the track, but the slower the tracking)
  • Input Channel: Auto Channel 
004 - Tracking Parameters

I always recommend having the grid on while you track so you can see if the track is drifting, but for tracks like this you can also turn on the Stabilize viewer in the view controls so that the area you are tracking stays still.

005 - Stabilize view (Not the same as the stabilize module)
Once you have finished tracking the static areas of the scene and you have determined you have a good track, you can then move to solving the shot.
In this case I know it is a Pan, Tilt, Zoom (PTZ) shot, so I have chosen that option, along with zooming checked to cover some of the subtle zoom in the shot.

006 - Solving for PTZ with Zooming turned on

Once we have solved the camera, we can export it out for the scene, but if we want to get the surfboard, we need to track that first.
We do this by drawing a shape around some texture detail on the surfboard.

007- Drawing shapes on the surfboard
We also draw a new shape in the layer with the Add Spline to Layer tool to cover the front area of the surfboard so we have as much detail as possible.  Again, we could just draw one shape and then mask out the arm, but you can do it either way.

008 - Adding another shape to the surfboard layer
Just so we can see how the track is going, you can then move the corners the surface to line up with the general position of the surfboard.  This is a useful visual aid when tracking.

009 - Setting up the surface so the grid lines up with the surfboard
We can then set up the parameters for the track.  This is a Perspective track, so we want to make sure that is turned on:
  • Motion: Check Translation, Scale, Rotation, Shear and Perspective
  • Min % Pixels Used: 90% (The higher the number the more accurate the track, but the slower the tracking)
  • Input Channel: Auto Channel 

010 - Surfboard tracking parameters

You can then start tracking!

Once it is tracked you can immediately switch back to the camera solver module.
The great thing is once you have solved the static camera from the first step, you don't need to resolve to export data for moving objects.  That all happens when you actually export.

In this case we are exporting out to After Effects.  If you want to use a different application you can export out to FBX if you have mocha Pro.

011 - Exporting to After Effects
Over in After Effects you need to go to the Edit menu and choose Paste mocha camera.  This option will only appear if you have installed the mocha 3D track importer for AE plugin from our website.  The plugin is free when you buy mocha AE V3 or mocha Pro V3 and it available here:

012 - Paste mocha camera option in the Edit menu in After Effects

 Once pasted, you will get a camera and 5 nulls; 1 null for each corner of your surface and a central null.  If the nulls are too large you can easily scale them down by just selecting them all and changing a single value in one of the nulls.

013 - The resulting nulls in After Effects

The great thing about being able to track surfaces like this is that you can adjust the surface relative to the same plane, and get different positions on your plane.  In this case, we want to centre our surface nulls around the tip of the surfboard, so we can shift the surface inside mocha accordingly.

014 - Moving the surface to a different area on the surfboard
Then when we export the 3d data again and paste back into After Effects, we get the nulls in the same position.

015 - The new nulls in the place where we want them

Trapcode Particular

The rest of this tutorial is related to linking a Trapcode Particular emitter up to the central mocha null, so if you just wanted to know how to track the moving objects you're done!  
Carry on for seeing how I use Red Giant's great particle tool to make a trail linked to the 3d data.

First of all we want to add a new solid.  Either press CMD/Ctrl+Y to bring up the solid window or go to the menu and choose Layer | New > Solid.
Make sure the new solid is the same size as the composition and hit OK.

016 - New solid!  It's like a meme, only for After Effects.

Then we want to apply the Trapcode Particular effect from the menu at Effect | Trapcode > Particular.  You could also choose another default particle system if you like, and similar processes would apply, but Particular has a lot of great controls.

You can choose to leave velocity on, but since the surfboard is moving, you can get a nice trail effect if you set velocity to zero.  Everything else can be left at default values until you want to tweak them.  See the settings below.

017 - Set velocity to zero

Now comes the mildly tricky part.  You can watch the video  if you need further clarification:
  1. Go to your layer panel and select mocha null 4 in your layers (or which ever null you want to drive your motion) and press P to show its position information
  2. Select your solid and navigate to the individual properties of the Particular effect applied to it
  3. Locate Position XY under the Emitter section of the Particular effect
  4. Alt-Click the Position XY stopwatch to open the expressions area
  5. Using the pick whip (See image below) drag from the Position XY in Particular to the Position value of your target mocha null.
018 - Pick whip dragging from Position XY to mocha null position
This will generate an expression in the Position XY value of the emitter.  It will look similar to this:
temp = thisComp.layer("mocha null 4").transform.position;
[temp[0], temp[1]]
This is just telling After Effects to assign the X, Y and Z values of the mocha null position to the value temp and then only use the first two values of it (X = temp[0] Y = temp[1]).
 Copy this expression for the next step using CMD/CTRL+C.

019 - Copy the expression create by pick whip
Next, alt-click the Position Z value in the Emitter section of Particular, and paste the expression you copied into the available expression field.

Rather than keeping the original expression however, remove this line:
[temp[0], temp[1]]

And replace it with this:

This expression does exactly the same thing as the previous expression, only this time it is only grabbing the Z value from the null position. There are other ways to code this, but this suits the purpose fine.

020 - Modifying Position Z expression to only get Z value from the mocha null
 If everything has worked correctly you should get a lovely trail of particles from the emitter, following along exactly in 3d space, driven by your tracked null from mocha.

021 - Whee!

Please check out the video at the start of the tutorial to see the complete run-through, and if you have any questions, feel free to ask in the comments, or on the forum at:

Wednesday, 9 May 2012

Your Camera Solver Questions: Answered.

Photo of mocha V3's release at NAB
by Michele Yamazaki of Tool Farm
Since the release of our camera solver in mocha V3, we have been getting tons of enthusiasm and buzz from you guys, and we're so pleased for your continued, awesome support. We got lots of upgrade orders the day we released mocha V3, so we know how excited you guys are about the brand new features and that excitement tends to fuel tons of questions!

I posted something very much like this in our imagineer systems forums and over on Tool Farm's forums, and Tool Farm even had a pretty cool blog post about our camera tracker... but I want to expand on this idea here. We've been getting a lot of questions from a lot of you about what mocha's planar solver is, and how is compares to all other camera trackers. I wanted to take a moment to expand upon what our solver is, how it works, and how it compares to camera trackers in general.

So what's the difference between mocha V3's camera solver and any of the other camera trackers on the market right now?

Image: scottchan /
The difference is that our camera solver solves for the planes you track in mocha and builds a camera based off that. Just like we’re not a point or feature tracker, neither is our camera solver a camera tracker. 

mocha v3 creates a 3D camera solve based on planar tracking, while, for instance, many 3D feature trackers requires the user to define good and bad data for the tracker in order to complete a camera tracking solve. Using planar data means no more "X"s wobbling across your screen and no more telling your tracker "no."

Because mocha is using the planar tracker to solve, mocha can steadfastly track better than any feature tracking in any other software because our incredible planar tracker has the ability to track and solve footage with motion blur, footage that goes offscreen, etc. Common difficult tracks such as tracing around reflections or obscured tracking areas are things that the planar tracker can solve easily. And you tell mocha where to look, you dont ask it to track everything and then adjust proper features for ages. 

Another difference is that mocha v3 will solve a virtual camera and “select” 3D planes and that mocha v3 exports 3D nulls instead of a point cloud. Now, when I say our camera solver doesn’t solve a point cloud I want to point out that it will give you nulls that represent the planar surface that you can use as “a point cloud” in programs like nuke even though they just mark where your planar surface is at all four corner and the center point. 

The bottom line is that mocha doesn’t solve for “the” camera like a camera tracker does, the ultimate goal we have in our camera solver being to put objects anywhere in the 3D scene or putting volumetrics or particles in the scene. Our camera solver solves for a camera relative to the planes you have tracked.

How can we get the most out of mocha V3's camera solver?

Use large shapes for the best tracks.
With our camera solver, usually less is more. Find two or three really good NON-co-planar planes to track and then hit solve. You will be able to put a 3d object into the scene relative to one of those planes. 

The point being that you usually don’t need a huge camera solve to put a 3D object into a scene, as there are numerous solves you can get with a 3D camera tracker anyway. You just need a reference point and a camera that works for the element you want to drop into the shot.

Things to look out for are just the normal things you need to look out for with our planar tracker, avoid tracking reflections, occlusions, slow moving shadows, etc. And understand that you don’t need tons of data for the solve, you just need to track one good plane for a Pan/Tilt/Zoom camera, and two non-co-planar planes for parallax cameras. As in, you can’t track two shapes on the same wall and expect to get a good parallax solve.

For example: if you need to rebuild your entire set in 3D, our camera solver might not be the best option, but if you need to do a simple set extension for a wall or a ground or both, we can totally do that. mocha’s 3D solver does not give you the entire camera space, only select planes. This could be very useful for many match moves, set extensions, particle effects, etc.

We want you guys to get up to speed fast on our new camera solver so I even released a new video about how to solve difficult planar tracks this week.

Tracking Difficult Shots with mocha Pro v3's Camera Solver from Imagineer Systems on Vimeo.

And Martin has also put together an excellent tutorial on the camera solver as well that he will expand on in another blog post later this week!

mocha v3 tutorial: Camera Solving for Moving Objects from Imagineer Systems on Vimeo.

Are you trying to compete with other camera trackers?

We’re not trying to replace conventional camera trackers, we're not even trying to compete with them. In fact, for blurry shots or other hard to track shots mocha can help camera trackers by putting in new, sharp planar information or grids where bad information used to be. That is if you have a need for THE camera the shot was shot with and want to help your camera tracker figure the shot out. How's that for being a team player?

How can I try out the new camera solver in mocha V3?

You can purchase mocha Pro here.
It is probably best to download mocha Pro v3, which is now shipping, and activate the 15 day trial to do some tests and compare for yourselves. Both mocha AE and mocha Pro products are similar but they do different things depending on what kind of work you do. mocha Pro has tons of time saving features that aren't present in mocha AE, but they're both based on the same amazing planar tracking technology.

I always recommend upgrading to mocha Pro for the dedicated generalist or the artist who doesn't have time to mess with time consuming corner pins and roto paint.

You can purchase mocha AE here.

mocha AE V3 is also very useful for roto and 2D tracking. Upgrade from the mocha AE bundle version is $195

Hope that helps answer some common questions for you guys! Let me know if you have any questions.

Stay tuned to our blog for more this week from our Product Manager, Martin Brennand!