The Art & Science of Movie Special Effects

Developed by researchers at the National Science Foundation's (NSF) Integrated Media Systems Center (IMSC), the Fastrack technology has helped specialists at Academy Award-winning special effects studio Rhythm & Hues drastically reduce production time for such films as X-Men 2, Daredevil, and the upcoming Dr. Seuss' 'The Cat in the Hat.'

"Fastrack is capable of tracking hundreds of features from one frame to another with sub-pixel accuracy in only a few seconds on a standard personal computer," said Eugene Vendrovsky, principal graphics scientist at Rhythm & Hues in Los Angeles, Calif. With the software, effects artists can process roughly 40 percent of movie shots without having to provide extensive input to the computer.

"This is a huge productivity leap for us," said Vendrovsky. "We are almost twice as productive thanks to Fastrack."

The software tracks motion between each frame of film to carefully wed "real" objects, such as an actor, with computer-generated special effects, such as a supersonic jet, a flying car or a raging river.

"At a broader level, special effects involve superimposing something synthetic onto something real," said Fastrack co-developer Ulrich Neumann, associate professor of computer science and Director of IMSC, an NSF Engineering Research Center at the University of Southern California. "The difficult part is to get the motion exact so the objects move correctly relative to each other," he added.

IMSC develops new multimedia technologies for entertainment, security, communications, and education applications-from advanced 10.2 channel sound systems for movie theaters to three-dimensional surveillance technology for airport security.

"The research at IMSC brings engineering, art, mathematics, psychology, and computer science together," said Mary Harper, the NSF program officer who oversees support of the center and its research, education and industrial collaboration programs. "The breakthroughs coming out of IMSC affect everyday life," she added, "engineering that directly impacts education, business and, of course, entertainment."

Neumann and IMSC assistant professor Suya You developed their tracking technology with cinema and training video applications in mind. The researchers used a set of mathematical algorithms to determine which features in a scene provide the best frame of reference for a computer to track.

Reference points, such as corners on a doorframe or a stop sign on street, can be inherent in a scene or they can be fiducials, objects such as large plastic balls, that substitute for a soon-to-be-animated character during filming. The software tracks the reference points, helping the computer glue every digital component-such as an animated character-to each frame of film. Neumann and You originally described their breakthrough approach in the March 1999 issue of IEEE Transactions on Multimedia.

"You create a lot of image layers and superimpose them later," said Neumann. "The layers can be real people and objects in a studio or synthetic, digital graphics, but they are all interleaved in the final image."

Rhythm & Hues bought the right to use the technology from IMSC in 2002, named the software Fastrack, and continually modifies it for use in films requiring complex special effects.

Until the advent of computer graphics technology in the 1980s, a team of animators had to draw many effects onto the film one frame at a time-much like cartoonists-or shoot miniature models frame by frame.

Computers have enabled machines to take over many of the hand-drawn tasks, although effects artists can spend many hours or days smoothly matching a computer graphic to background film.

If the frame of reference moves, even digital processes can be difficult and time-consuming. For example, in X-Men 2, a camera pans around aircraft flying through numerous tornadoes, all in front of the backdrop of a sunset. The camera view of the digital tornadoes has to match exactly with the imagery of the aircraft and the pilot's motions in the filmed world.

Fastrack can do the initial, difficult matching, processing each frame in just seconds. A person then performs final edits and adjustments to tweak the film into a finished product.

For a given scene, the first step is to film the live set and any objects, such as a street with people. Then, Fastrack software analyzes the film and tracks camera motion and staging, saving production time early in the filming process.

Effects artists next create necessary digital elements-for example, swirling tornadoes-animate them, and add any other effects, such as explosions. In the final step, computer tools combine the film and computer elements and transfer the scene from digital data onto film.

"Someone starts the software, looks over the results, and cleans up the shot," said Vendrovsky, "in a process that now takes a couple of hours instead of couple of days."

In the future, Rhythm & Hues hopes to further modify Fastrack for "flexible body tracking," where effects artists superimpose a digital character over an actor in a special body suit.