All human beings—even famous movie directors like Federico Fellini—have a finite lifespan. But can their talent live on (and continue to create) with artificial intelligence? Maybe so.
Campari Red Diaries: Fellini ForwardCampari Red Diaries: Fellini Forward, a short film and behind-the-scenes documentary, premieres at the Venice Film Festival on Sept. 7, and will be featured at the New York Film Festival before an on-demand release in select markets.
While there are three humans with directorial credits—Zackary Canepari and Drea Cooper for the documentary, and Maximilian Niemann on the short film—the post-human creative force on this movie is AI, masterminded by innovation production studio UNIT9. We spoke with Marc D’Souza, Production Director at UNIT9, to find out more.
Finding Fellini’s Flair
The question behind Campari Red Diaries: Fellini Forward is whether AI can be trained to not just imitate Fellini’s oeuvre, but extend it into new and original work. We haven’t seen the film or documentary, as it wasn’t made available to the press ahead of its premiere, but the film is set in Rome and explores Fellini’s life, work, and dreams as if with Fellini’s own flair. The accompanying documentary gives the backstory on how the AI learned to direct it.
“We used AI algorithms on a selection of Fellini’s past movies and scripts in order to extract a rich data set and establish patterns that could ascertain Fellini’s signature creative decisions,” D’Souza says. “Natural language processing, computer vision, and image processing were used in order to extract the data. Shot type, scene cuts, camera movement, objects per frame, facial emotion and action/dialogue syntax and structure were all analyzed, as well as scenes and locations.”
Fellini has a highly recognizable style—ideal for a pattern-seeking intelligence—and the director was prolific, so there was a lot of material for the AI to ingest. A giant of twentieth century art cinema, Fellini came of age under the trauma of Italian post-war fascism, and was fascinated by the subconscious, keeping dream journals for decades to understand what drove him, before transposing the weirdest and most wonderful scenes onto celluloid.
Supervising the AI
To ensure a “control” for the project, the global production team assembled under the watchful eye of Francesca Fabbri Fellini. She’s Fellini’s niece and the founder of Fellini MagazineFellini Magazine, which protects his legacy for future generations. Surviving members of Fellini’s original crew also consulted on the project, to counsel the AI what was (or was not) “Felliniesque.”
“As the ones who knew the director’s spirit best, they were able to validate the AI’s vision so that the final output could be perceived as being truly Felliniesque,” D’Souza points out.
University of Oxford Mathematics Professor Marcus du Sautoy, and Dr. Emily L. Spratt, a Princeton University art historian and data science expert, were called on for academic supervision. But how did the production team teach the AI to direct like Fellini?
“AI had two main roles within the process,” D’Souza says. “The first was to analyze Fellini’s past movies and scripts to extract what makes them Felliniesque. The second was to then use this data to generate suggestions for the creation of this new short film infused with the flavor of Fellini. Millions of frames were investigated. For instance, to study the camera movement, just one element of many, we analyzed nearly 3 million frames and spent nearly 260 hours on extracting the data.”
Meet the Synthetic Scriptwriter
But before any frames were captured in the virtual can, they needed a script.
UNIT9 created a custom tool to handle original script generation, building on OpenAI’s GPT-2 (they would have used GPT-3, but it wasn’t out in 2019 when the project started development). This meant the AI settled down to “read” every script Fellini ever wrote, breaking it down into style, context, nomenclature, juxtapositions of concepts, recurring themes/motifs, and so on.
Then the AI composed an entirely new script, which is really cool—unless you make your living from writing spec scripts in Hollywood.
Inside the AI Toolkit
What was under the hood of the tech setup?
UNIT9 deployed some existing off-the-shelf algorithmic-building tools (TensorFlow, PyTorch) but also created several new ones from scratch. Rather than deploy a generative adversarial network (GAN), where two competing neural nets duke it out for the “correct” output, UNIT9 decided to use a supervised learning approach. This gave them more control over the process, and allowed them to work on multiple challenges at the same time, D’Souza explains.
“To identify and classify different cinematic shot types, we used the Shot-Type-Classifier Model for ResNet-50 network; Mask R-CNN for object detection, and YOLOv3 and DeepSort for real-time multi-person tracking,” D’Souza says. “We also brought in Keras, the Python deep learning API for emotion detection within a live camera feed.
“We developed a key tool for ‘pre-visualization,'” says D’Souza, “which generated a sequence of camera shots and cuts, then assessed them against Fellini’s visual language style, before we committed to shooting them. This AI model at the core of the tool, with the Fellini shot prediction model, was developed by us at UNIT9 from scratch, then integrated into Epic’s Unreal Engine as a plug-in.”
In film parlance, the AI became a second unit assistant director and cinematographer.
Off to the Venice Film Festival
In the end, did the short film resemble the great Federico Fellini’s work? It must have, as Francesca Fabbri Fellini, protector of his legacy, has given it her blessing.
Campari Red Diaries: Fellini Forward will now head to its premiere at the Venice Film Festival. As the UNIT9 team disassembles the servers and starts on the next project, D’Souza is satisfied that the final result is proof that AI is getting smarter and smarter every day.
“What I personally love about this project is that it really sparks debate over the future of cinema and the role that AI will play in it,” D’Souza tells us. “Plus part of the beauty of using AI is that so much is possible—things we as humans may not have even thought of yet.”
