Tuesday, 6 March 2012

How Mocha Thinks


It was snowing in Chicago
I recently returned from a trip to Chicago with Imagineer’s head of sales, Bryan Milne, where we had the always-valuable opportunity to visit with a ton of studios. We also had the opportunity to present to the Chicago Final Cut Pro Users Group (soon to become the Chicago Creative Professional Users Group), hosted by the lovely Sue Lawson.

During the meeting, we generated a lot of interest with our presentation, which was focused on helping artists save time in their workflow. What was interesting, however, was we kept running into the same questions from artists who were still struggling with our workflow. What I quickly realized was all of those issues could be solved with a better understanding of the workflow in mocha. I travel all over North America as an Imagineer Systems Product Specialist, and in my travels, I am noticing a trend...

Every time I meet artists in person, I get these ‘epiphany moments’ from our users - some of whom have been using mocha for years - about how our software works. And what I realized is that many people don’t fully understand how mocha’s Planar Tracker actually thinks. Some don’t quite jive with the spline-to-track relationship, and none of these artists realize that all of these issues can be fixed with a simple visit.

I thought to myself, “how do I get this message out to as many people as possible?” So I decided to make an in-depth tutorial on it with lots of visual aids and clear explanations. And so I am in that process. The tutorial is in the works and being polished as we speak, but I couldn’t hold back anymore and wanted to tide you over with a much needed blog update that covers off some of the common questions I run into and attempts to breakdown some of the answers. So here goes!

How mocha Thinks

What do I do with this, anyway?
What is mocha used for?

mocha is an intelligent Planar Tracking utility for post production and VFX professionals that
combines advanced roto, motion tracking, object removal, insert and stabilization features into a single desktop application.

So what does that mean? mocha has this amazing technology called Planar Tracking, which is a 2D tracker that streamlines tasks like tracking, rotoscoping, match moving, and screen replacements for most shots... taking tedious frame by frame work out of your workflow so you can have a life outside of work.

Time is the most valuable commodity in life.

To that end, mocha is for EVERY Day use, not just a fire extinguisher that sits in the cabinet, waiting for that emergency! Using mocha first will ensure that you will never have to scramble for an emergency tracker to save your butt at the last second. It’s all about having the right tools in your toolbox to get the work done - right from the start of the project.

Tracking and rotoscoping are part of almost every visual effects, editing, or motion graphics project-finishing, post-broadcast... all of these professions use it. Point and feature trackers are most commonly used, but to get good point or feature tracks requires a mix of experience and luck. You often have to color correct and tweak images to get “meatier” data for traditional trackers. If the point or feature being tracked exits frame, you get into offset tracking, which presents its own set of challenges. If it all fails, you are into hand tracking, which is time consuming and very hard to get accurate. mocha tracks patterns of data across the shot through blur and partial occlusions, significantly decreasing the amount of experience and luck you need to get the data you need.

Mocha is a Planar Tracker
So what is a Planar Tracker, and what the heck do you mean by a plane?

So here’s where I tell you what mocha is and isn’t.

mocha tracks planes - that is - a pattern of pixels moving relative to one another in a scene. It is not a point or feature tracker. A plane is any flat surface having only two dimensions, such as a table top, a wall, or a television screen. But it can also be the front of someone’s face, the top of someones chest, and you can even cheat those more “rounded” planes into good workable tracks, such as in tops of arms and torso shapes. Planes provide much more detail about an object’s translation, rotation and scaling than is possible with point-based tracking tools, mainly because we are calculating the relative motion of pixels in the pattern of the plane. Even as an object leaves and enters a frame, there is usually enough information for the Planar Tracker to maintain a solid track of the object, because relative motion applies whether you are tracking all or part of the plane. When you work with mocha, you will need to look for planes in the clip. More specifically, you will need to look for planes that coincide with movements you want to track. Like pictures hanging on a texture-less wall, or the frame around a computer screen.

I’m moving my spline around, why isn’t my track moving with it?

This is one of the biggest misconceptions about mocha that I’ve encountered. In mocha, EVERYTHING is a child of the track, with the exception of manual track mode where the surface drives the track. When I say everything is a child of the track, I mean it! Even the splines that define the area the track is looking are driven by the movement of the track. The surface tool and grid both follow the track. And you can even parent other finer roto shapes to the track, but moving the spline around does not move the track around, except where it defines another surface to track mid track.
Link layers to any track! Or none!

You can only track one surface at a time, so if you’re tracking a ground plane, feel free to move the tracking spline around on the ground to good areas of visibility. But watch your track and see what it does, the track won’t move anywhere if you move the splines. It will just continue to track the ground, doing something point tracking veterans might think of as "offset tracking" as part of mocha's basic workflow.

You can also un-parent tracks from splines. Another great way to track objects like ground planes that are moving off-screen is to unlink the track from itself. In the link to track drop down menu, the default of your spline should always be linked to a track of “itself.” But you can link it to “none” and the spline will stay in one place and read everything that moves underneath it like a scanner.

What are the surface and the grid tools for?

The surface tool is the little blue square in the top of your tool bar. This surface tool defines what will be your “corner pin” for whatever compositing program you need to export to (mocha AE exports only to After Effects, mocha Pro exports to everything). I like to turn my surface tool on when I track so that I can use my grid tool.

The grid tool is right next to the surface tool, and the grid tool is only an artists visual tool to check and see what the track is doing while you’re tracking. It’s used to check your work after the track is finished. When I track, I align my surface tool to the plane I am tracking, which warps the grid tool in space appropriately so that I can know immediately as I track if my data is good or bad. This is very helpful if I am tracking a long shot, because I can stop the track where it gets bad and then move my spline to a new area to track that might give me better tracking data.

What does align surface do?

In mocha, align surface is what we use to warp corner pins appropriately for After Effects. Often, to get best results in certain compositing programs, you will need to align the surface tool to the entire height and width of the shot to avoid any unusual warping in your composite. That is all align surface does. You just go to the frame that you need mocha to “align to” and hit align surface, and mocha will put the corners of your surface perfectly at all four corners of your clip on that frame and then warp according to your tracking data for the rest of the tracked shot.

Is layer order important?

Always define foreground and background layers for removes!
Layer order is very important in mocha, particularly regarding the remove tool and the background to foreground relationship that the remove tool needs in order to get a good remove. When I say background and foreground layer, I mean that layer order is user defined and layer names are set by the user as well. You, as the artist, have to define what background and foreground objects are. mocha will treat objects at the top of a layer pile as closest to the camera, and objects at the bottom of the layer pile as furthest from the camera. This means that mocha holds out top shapes you have defined from bottom shapes, so you never have to create holdout mattes if you track objects from the foreground of the scene to the background of the scene. This becomes particularly important in removes, where you must define a foreground element to be removed and you must track a larger background area for mocha to pull pixels from to replace that object.

What you are telling mocha is: look in the background you have defined to replace the foreground object you have defined. If you do not define a foreground object and you do not track a background on a separate layer that is BELOW the foreground object, mocha will not make a cleanplate for you. If you are trying to remove a blemish on someones skin or text off the side of an object, you can’t see behind the “foreground” object, so you have to make a cleanplate for mocha to use to see “behind” that foreground object.

So those are just a few of the many ‘epiphanies’ I’ll be covering in my video. I hope this breakdown can tide you over for a little while, while I finish it up, but I am confident that I can share some more epiphany moments with you soon! As we gear up for NAB next month, stay tuned for more news on the Blog. We have some exciting things coming!

Cheers!
Mary



2 comments:

  1. Nice teaser article, Mary! Will await your full tutorial. A clarification - if splines are children of the track, there should be no need to move a spline mid-track (outside of "except where it defines another surface to track mid track"). Am I correct? In that case, why allow an original spline to be updated on a different frame after it is initially created?

    ReplyDelete
    Replies
    1. Hi there!

      Actually, the need to move the spline is so that you can move the spline around occlusions, animated shadows, reflections, or any bits of bad data you don't want the tracker to pick up so long as the spline is on the same (or very similar) plane. Or if the plane starts to go offscreen and you have nothing left to track so you have to pick another section of good data.

      We allow the spline to be updated in order to perform these sorts of bad data avoidance processes I just talked about OR to rotoscope.

      In order to be able to rotoscope well you can't have the spline move the exact same way as the tracking spline every time, and the reason for that is because often you have to move the tracking spline around to get good tracking data first and then attach your finer roto shapes to that solid track afterwards, rather than roto and track at the same time.

      If you mean why do we allow you to see what the spline is doing, we do that so you can see what mocha is looking at while it is tracking. We also do this by letting you turn the surface and grid tool on as well. To really get an example of what I mean, try tracking a plane (like a ground plane on a hand held camera shot) and moving the spline around, but turn the grid and surface tool on as well and align them to the plane you are trying to track. Watch that track progress smoothly (represented by the surface and grid tool) as the splines moves all across the plane you are trying to track.

      Let me know if that clarifies the question!

      Cheers,
      Mary

      Delete