January 2, 2007
and what does it mean for Final Cut Studio users?
You've probably heard the term XML and never been able to quite pin down what it means, because every time you see the term it seems to mean something different.
This really isn't a surprise because "XML" is really just an umbrella term for a way of presenting information in a document that's both computer readable and (in theory) human readable. There are many different types of XML file. Even in Final Cut Studio there are several different "XML" formats in use in addition to the obvious exporting or importing of Final Cut Pro XML. Motion's Project Files are XML files, despite the ".motn" suffix. All .plist preference files are also XML files.
You can think of XML as if it was humanity. There is only one humanity, but within humanity there are many languages. Similarly, there's only one XML format but there are many, many different languages of XML.
What is XML?
Skip ahead to the next section if you don't really care about what an XML file is, just how it can be used.
Figure 1 - I've extracted the first part of a Final Cut Pro XML Export; a Motion Project File and a .plist Preference file. The first line tells us, and a computer reading it, that this is, indeed, an XML file.
Looking at Figure 1, there's probably something just a little bit familiar about the way it looks if you've done much HTML work for the web.
As it turns out the "M" and "L" in HTML and XML stand for the same thing: Markup Language. As in "HyperText Markup Language" and "eXtensible Markup Language". One "marks up" text for formatting; the other is an extensible markup language for data. HTML is limited to the display functions whereas XML, because it's extensible, does not have limits to the type of data that can be marked up.
Both HTML and XML mark up using tags. HTML marks up blocks of text to tell the browser how to display it. XML marks up blocks of data with tags that tell us what type of information or data is stored.
In HTML you have your words and paragraphs and the tags that tell the browser how to display the text as "marked up." (Marking up was an editor's job in newspapers or magazines before any of this technology stole the name.) For example, in the HTML "code" you might see something like this:
<p> at the <b>beginning</b> of a paragraph </p>
There are two sets of tags in this example: the pair of 'p' tags tell the browser to display all the text between the opening <p> and closing </p> tags as a paragraph, and the pair of 'b' tags tell the browser to display the word "beginning" in bold.
Likewise in an XML file there are tags but these tags tell the reader or computer what type of information is between the tags. Instead of paragraphs we might have <timecode> at the start of a Timecode entry, and </timecode> at the end of a Timecode entry. Like the <b> </b> tags in the paragraph above, all XML tags can nest as well. This means we can have this piece of XML code that gives the Timecode details for a Clip (out of a Final Cut Pro XML Export.)
<timecode>
<rate>
<ntsc>TRUE</ntsc>
<timebase>30</timebase>
</rate>
<string>01:00:00;00</string>
<frame>107892</frame>
<source>source</source>
<displayformat>DF</displayformat>
</timecode>
Here the tags are to tell us that this is some information about Timecode. Everything between the beginning <Timecode> tag and the ending </Timecode> tag is information about the Timecode. Within the <rate> </rate> tags (remember these are Timecode rates because it is nested inside the <timecode> tag pair) we can read that the Timecode is NTSC and has a timebase of 30.
Remember that any nested information is totally subservient to the tags that surround it. Therefore we don't have to say "Timecode frame rate" for a tag because that's implicit in having <rate> nested inside <Timecode>. In this example <timecode> has five properties, one of which is <rate> and <rate> has two properties: NTSC and 30.
XML tags define the type of data that is between the tags. An EDL file, which also carries data about clips and edits (but a much older format) contains only data. Figure Unless you know in advance what each item in each line in Figure XX means, you can't infer it. With XML that data would be explicitly described by being marked up with tags.
Different Tags for different XML formats
If we were to look inside a Motion Project File we'd see different tags for different data being handled within a Motion Project File, like <scenesetting> or <parameter>.
This ability to be used for many different purposes is why XML is referred to as extensible: it can be extended to any new purpose without having to pass through a standards body. This is also why each XML document starts off by first telling us that it is an XML document (first line of the examples in Figure 1) but then continues with what language its speaking.
In Figure 1, you can see that the second line in each example is a declaration of what type of XML file it is - !Doctype - and the third line contains the version of that type of document. This is where the XML is declaring what language it's speaking so that "listeners" (human and computer) know how to understand what it says. For example, Motion projects are an "ozxmlscene" and at version 1.5 of "ozxmlscene". Final Cut Pro's XML format is "xmeml" and (as of Final Cut Pro 5.1.2) is at version 3. The type and version information is particularly important to the computer when it uses the XML information, because in order for any language to be useful, the listener has to understand the language, or find an interpreter. In this case an XML interpreter is called a "parser" because it breaks out, or parses, the information in the file. We could create a 'mega-parser' but since each use of XML is really quite specific, it's easier to use a simple parser and declare the language variation right at the start of the file itself.
While Final Cut Pro and Motion both produce XML, Final Cut Pro's XML parser only speaks "xmeml" and doesn't have a clue how to interpret "ozxmlscene" from Motion. Final Cut Pro interprets the Motion file as a QuickTime Clip instead. The reason why Motion uses its own suffix, .motn, instead of .xml, is so that it can be interpreted appropriately in each context: in Motion, it's a project file, while in a QuickTime application, like Final Cut Pro, that same file is interpreted as a QuickTime movie and a Final Cut Pro clip. Similarly a Final Cut Pro XML file isn't going to make a useful .plist file because the language doesn't contain the same vocabulary.
So, when we talk about XML, we really need to be a little more specific about the type of XML file we're talking about. As well as the examples I've already mentioned, you'll find that your iPhoto Album data is stored as XML files but with a vocabulary and language optimized for still image data. Likewise your iTunes Playlists are stored in an XML format that uses an iTunes-specific language and vocabulary. iMovie, iDVD and the Desktop Pattern Preference Pane are able to open iPhoto albums and iTunes music libraries by reading and interpreting the XML files. Final Cut Pro could, in theory, be written to also open iPhoto albums and iTunes Playlists but they're not as relevant for professional production. Podcast feed files are another common use of XML with, yes, yet another XML language with a vocabulary optimized for creating feedsfor audio or video files. Do a search on your hard drive and see how many .xml files there are, let alone all the other XML files hiding as something else, like .plist or .motn (among others).
XML compared with AAF
We've already learnt that, while both XML and EDL files are presented as plain text, the XML file includes tags to identify each piece of data in the file while the EDL file just has the data with no clues as to what number is a start Timecode or end Timecode or duration in seconds and frames. EDL was succeeded by OMF and subsequently by AAF - the Advanced Authoring Format that was designed to be the Esperanto of human languages: the common exchange format between editing systems. Unlike XML, the AAF format is not human readable because it is a "binary" format (meaning it's like computer code), so while an XML file can be opened in any text editor and browsed and edited, a binary file like AAF cannot be so easily read or edited. In general computer terms, a completely open standard is best.
So, can I do with XML in Final Cut Studio?
Final Cut Pro XML by itself isn't a whole lot of use because it's designed as a way of expanding Final Cut Pro's integration with other applications and tools.
As Apple says in its developer documentation about XML:
"The door is now open wide for developers who want to create applications that are fully integrated with Final Cut Pro, or to build customized post-production pipelines that can save clients significant time and money."
It lets developers get into the guts of a Final Cut Pro project and expose the details for other purposes because every aspect of a project is exposed, unlike what can happen with the Advanced Authoring Format - AAF - interchange between applications. For example, Avid uses AAF for interchange between its applications, as does Adobe, but lots of the useful information is kept "dark", that is, proprietary, and only Avid applications can access that Avid proprietary data. Likewise only Adobe applications can access the Adobe proprietary part. Final Cut Pro XML exposes everything in the project, bin or sequence to any external application.
This has allowed Automatic Duck to create "translator" modules to take Final Cut Pro XML and convert it to OMF and AAF formats for use in Avid, Premiere Pro, After Effects, Combustion and Pro Tools and in some cases bring project information from those applications to Final Cut Pro. Spherico have created a number of utility tools that take Final Cut Pro XML data and converts it to other useful formats or converts external data into something Final Cut Pro can use. Likewise Digital Heaven have leveraged the Motion XML project format to enable Automotion automation. More on these uses further into the article.
Final Cut Pro XML in Final Cut Studio
As mentioned, while Motion Project Files are an XML format, they are opened directly into Final Cut Pro as a QuickTime Clip. They are not using the Final Cut Pro XML format. So, by itself about all we can use Final Cut Pro XML for, is as a way of "going back" to an earlier version of Final Cut Pro.
As I'm sure you know, we cannot open a Final Cut Pro 5 project in Final Cut Pro 4.5, or earlier - at least not directly. However, export from Final Cut Pro 5 as Final Cut Pro XML version 1 and then that XML version of the project file can be imported into Final Cut Pro 4.5 or 4. Only version 1 of Final Cut Pro XML is readable by Final Cut Pro 4.x so we have to use that for compatibility. So, although Final Cut Pro does not have a "Save As earlier version" export it does seem we'll always be able to go back at least one version using XML.
The Final Cut Studio makes one other use of Final Cut Pro XML. Cinema Tools can export records from a Cinema Tools database as Final Cut Pro XML so that Final Cut Pro can import and create clips that link back to the Cinema Tools database. This means you can import a Telecine log, convert it to a Cinema Tools database in Cinema Tools and carry all the information forward directly to a Final Cut Pro project using Final Cut Pro XML.
Project Interchange
The next most common way that Final Cut Pro XML gets used is to take project information from Final Cut Pro to another editing, compositing or audio application. Automatic Duck have been the leaders in this field. Automatic Duck's range of tools massage Final Cut Pro XML into:
- "Avid-Compatible" OMF compositions from Final Cut Pro;
- OMFs with embedded audio for Pro Tools¨; or
- AAF files for Quantel iQ or eQ 3.5. or Premiere Pro.
Other Automatic Duck tools allow Final Cut Pro exports to be imported into Autodesk Combustion and Adobe After Effects as well as allow Avid OMF compositions to be imported into Final Cut Pro. Because the information is revealed in the Final Cut Pro XML, it can be converted to these other formats to really smooth workflow. Automatic Duck's power comes at a price, but the smoothing of workflow between systems is worth it.
Increasingly applications that want to work with Final Cut Pro are providing native XML support in the applications. For example Pyramix - a digital audio workstation from Switzerland's Merging Technologies - will be the first PC-based audio workstation to natively support Final Cut Pro XML to import and export clip gain, fades, effect definitions, envelopes, etc necessary to work smoothly between Payramix, Vcube and Final Cut Pro. Automatic Duck's Pro Import FCP version 2 and higher supports the import of OMF from Pro Tools with all layers intact in Final Cut Pro. Autodesk supports Final Cut Pro XML import into its smoke, fire and lustre systems without third party tools. In fact Autodesk joined forces with Automatic Duck to import Avid AAF files into Smoke using technology to seamlessly turn AAF into Final Cut Pro XML which goes into Smoke.
AutoMotion
Digital Heaven have created a very innovative use for Motion XML in their AutoMotion application. AutoMotion takes one or more Motion Projects and uses them as a template, merging information from a database-like table with the project template(s) to create, for example, all the lower third titles for a project almost instantly.
This is possible because a Motion project is structured as an XML file so it can be read, interpreted and then copies can made that are mostly the same but have changed text in the parts of the file where the actual words in a title are stored. Titles created with AutoMotion can be edited faster than you could re-launch Motion and edit one title.
AutoMotion and what it does, tends to cause my head to explode when I think about it. A Motion project file is an XML text file, so it can be manipulated by AutoMotion, but a Motion Project file, thanks to a plug-in in your QuickTime folder (inside Library), is also a QuickTime movie file (unrendered) so Final Cut Pro "sees" the Motion Project as a QuickTime clip it can import, trim and render. It's both a dessert topping and a floor wax!
But wait, there's more!
Apple's goal was to open up Final Cut Pro to third parties and open it has become.
Almost every clip management tool, from Lumiére HD, to CatDV to P2 Log and DV Log all export Final Cut Pro XML from their clip management tools direct to Final Cut Pro projects. There's no need to import each clip or clips separately or as a group. In fact these days, any logging tool designed for the Macintosh market, better have Final Cut Pro XML export to be competitive.
Another developer that has been very active in creating XML-based tools for Final Cut Pro users is Andreas Kiel of Spherico. Andreas has created literally dozens of tools that use XML to manipulate, extend or support Final Cut Pro.
XML2Text is a simple, but very versatile, tool that extracts the XML information and allows the editor to save this information in different formats. It can also be used to extract Marker or Clip information from the Sequence XML and manipulate it before export. It has many uses but just one of them is to convert Clip Markers to Sequence Markers or to print Markers with notes. It can also export Marker information to other MPEG-2 encoders other than Compressor or to extract the text of Titles within the Sequence and a lot more.
TitleExchange Lite/Pro is a tool that will get your title information out of Final Cut Pro into other applications; to get titles into Final Cut Pro from another source or create a QuickTime with your Final Cut Pro subtitles without the need to render.
SoundCount is a fast, easy way to calculate the total playtime of commercial sound in a Sequence or Project by analyzing the Final Cut Pro XML file to create time tables.
rotateClips shows just how Final Cut Pro XML can make life easier. Designed for a single task, rotateClips takes a Bin, Sequence or individual clips and flips or flops the clips to correct for 35 mm adapter inversion.
Spherico has several other tools that will speed workflow, including the ability to take an iTunes Playlist and convert it to Final Cut Pro clips linked to the source media; BWF2XML which takes metadata from Broadcast Wave Format (BWF) files into Final Cut Pro; and more. Check out the website - chances are Andreas has a solution for a problem you've been having, powered by Final Cut Pro XML.
xm|Edit's slogan of "Work Less, Play More" talks to the advantages of Final Cut Pro XML over editing-specific formats like EDL or AAF. xm|Edit's primary application, Traffic, takes Final Cut Pro XML and manipulates it in a node environment (like Shake) and processes it. Traffic is very powerful, allowing users to apply logic to Sequences and manipulating the clips accordingly.
With Traffic, editors can do multiple versions of their sequences in seconds, merging a base template sequence with dozens (or hundreds) of station-specific or region-specific endings. Another use might be to select clips from a Sequence based on any criteria in the clip logged data and build sequences based on the selection. For example taking all the green screen shots and isolating them into a Sequence separate from the shots that don't need effects processing.
Into the Future
Apple are obviously committed to making Final Cut Pro as open as it possibly can be as evidenced by the additions they "slipped" into the release of Final Cut Pro 5.1.2. In 5.1.2 they added support for Version 3 of Final Cut Pro XML, for Apple Events and for the interpretation of QuickTime movie metadata. Even without these new features, there is still unfulfilled potential from the current implementation.
In the article at Apple revealing Final Cut Pro XML for developers - a surprisingly easy-to-read article for a developer article - there are suggested uses that just have never seen the light of day from anyone. One example talks of an editor who creates a motion path in Adobe Illustrator and uses an AppleScript to extract the image information from the Illustrator file and translate it into Final Cut Pro XML. That would add a degree of control over Motion Paths we can't easily get in Final Cut Pro directly. The developer article describes a whole list of applications "that support XML, and then to consider how they might interact with Final Cut Pro." As we've already learnt, however, support for XML does not immediately guarantee support for Final Cut Pro XML. Usually some sort of translation software would need to be created to massage the XML output from Keynote or Omni Outliner to turn it into something useful for Final Cut Pro! In fact it's highly unlikely that those applications would produce anything useful for Final Cut Pro as they lack information about media, clips, sequences or any metadata useful for Final Cut Pro.
Database integration
One really logical application for Final Cut Pro XML export (and import) that hasn't been tapped fully yet, is to be able to export from all projects to a central database that keeps track of all clips and media across projects. The database would provide a browsable, searchable index and provide for searching, sorting and exporting selects back to Final Cut Pro for inclusion in new projects. This type of workflow could also include "transcription" fields so that longer transcriptions can be integrated with Final Cut Pro, perhaps as an extended version of Sergei Franklin's Final Log with more detailed two-way communication with Final Cut Pro. Perhaps we'll see some of this type of integration from Apple itself, as an outcome of their recent purchase of Proximity's Artbox software.
Script-based editing
One tool that Final Cut Pro has lacked, compared with Avid Media Composer, is Script-based editing. In script-based editing a script, or documentary transcription, can be aligned to the audio and video in the project. When working this way, we could search for a word phrase, or sentence from the script and find the matching audio and video.
Version Tracking
As we move into a world that requires one edit for cinema or broadcast release, a different edit into webisodes for the Internet and yet another for cell phone release, tracking versions is a growing problem. This is in addition to the normal versioning any project goes through. With Final Cut Pro XML the production company could export XML routinely (say daily) and kept track of all the versions of all the Sequences that were ever created in the Project.
This would allow an editor to go back much further in the history of the project to retrieve an earlier version than they could with the Final Cut Pro Vault.
Automated workflows
The one area where Final Cut Pro XML is going to have a big impact is on automated workflows.. For example, Kent Bye got very excited about integrating Final Cut Pro with the Open Source content management system Drupal using XML. In an article from late 2005 he outlines a process for collaborative filmmaking using Drupal and Final Cut Pro XML.
Unfortunately for automation, Final Cut Pro does not currently include support for AppleScript or Automator, but it appears that support is likely in the future because in Final Cut Pro 5.1.2 Apple added Apple Event support. Apple Event support is a necessary first step to adding AppleScript or Automator support. Support for AppleScript or Automator, means that we can automate workflows into and out of Final Cut Pro using XML to communicate the details of the project.
And into the future!
The availability of Final Cut Pro XML has been, and will continue to be, an important part of Final Cut Pro's success. By providing an open way for 3rd parties to get information from, and to, Final Cut Pro projects, Apple can concentrate on offering core features in Final Cut Pro with 3rd party developers to create and integrate the types of tools we've talked about here.
Apple are providing more ways in and out of Final Cut Pro and more "hooks" for developers to extend and expand Final Cut Pro's capabilities. As I noted above, Final Cut Pro XML version 3 support was included with Final Cut Pro 5.1.2 along with the aforementioned Apple Event support. Final Cut Pro XML version 3 supports another new feature where Final Cut Pro caches metadata from the QuickTime file. Most commonly we'd see metadata like the author and copyright details but there is a lot of metadata that can be contained in a QuickTime movie and Final Cut Pro will now cache that into the project.
How will that be used? Who knows, it's a new feature but Apple have created a demonstration application MovieAssembler. MovieAssembler uses these new features to create a "watch folder". When any new material is added to the folder, MovieAssembler is watching for it and imports it to Final Cut Pro automatically. Even better, MovieAssember passes the metadata from the QuickTime movie to Final Cut Pro. Metadata that tells Final Cut Pro what Sequence to edit the Clip into, and what time into the sequence to edit it at!
However workflows evolve in the future, having an open, accessible and well documented interchange format will continue to be a big plus for Final Cut Pro and a useful tool for editors.
Philip Hodgetts is a technologist, editor, industry pundit, podcasting veteran and specialist in new distribution
systems. His weekly show "Creative Planet's Digital Production BuZZ"
has become a reference point for Digital Production, Post Production
and Distribution technology and business. He is well known in Final
Cut Pro circles as an expert in digital workflow based around Final
Cut Pro.
