L’Institut du Monde Arabe accueille les témoignages de guerre présentés en réalité virtuelle.
A quoi servent les images sur la guerre et comment informer aujourd’hui ? Comment raconter la guerre ? Karim Ben Khelifa, reporter photo se pose ces questions tout le temps. Il y a répondu par des publications dans les médias, des expos photos. Il cherchait à aller plus loin, et toucher le public au plus intime. Non par voyeurisme ou goût du sensationnel. Il voudrait juste qu’on se souvienne de la paroles des combattants, ceux qui tuent et sont tués.
Avec France Télévisions Nouvelles Ecritures et Camera Lucida, Emissive, le MIT et l’ONF au Canada, il a donc conçu un dispositif en réalité virtuelle, avec casque et en déambulation. La guerre par ceux qui la font vraiment, plutôt que par ceux qui la pensent ou l’observent.
Ici Karim Ben Khelifa fait le choix d’une relation directe, personnelle, quasi intime entre le visiteur et les témoins de son reportage. Le public est donc amener à s’informer différemment.
L’expérience
L’expérience est troublante. On est au contact de soldats, miliciens et autres hommes armés au quatre coins de la planète.
Pour autant vous n’entendrez pas les balles siffler à vos oreilles. Les hommes se matérialisent face à vous, debout, reconstitués en 3D, et ils s’expriment très posément.
Le visiteur de ce dispositif qui s’étend sur 300 mètres carré, déambule et change de pièce pour chaque conflit évoqué : Israël-Palestine, Congo, Salvador.
Les soldats s’adressent à lui directement, le suivent du regard, il ne peut échapper à leur discours qu’en s’éloignant. Chaque visiteur vit une expérience différente, et en fonction de ses réactions durant le parcours, la conclusion varie. C’est un face à face, entre les combattants, et un face à face entre vous et eux. Ça dure 50 minutes, et les gens restent captifs… comme au cinéma.
Un journalisme engagé pour la paix ? The Enemy propose une représentation de la guerre qui montre l’humain derrière le tueur, l’enfant derrière le milicien.
Karim Ben Khelifa défend ici l’idée d’un journalisme qui touchera son public plus sûrement que des images trash diffusées à la télé ou sur le net. Il cherche la paix plus que les raisons de la guerre.
Son dispositif interroge la pratique journalistique que les producteurs qualifient d’ « oeuvre d’innovation documentaire et humaniste ». On verra au fil du temps si d’autres producteurs se lancent dans ce type de diffusion, et si le public adopte cette forme d’information.
Après l’Institut du Monde Arabe The Enemy sera montré au Canada mais aussi en Israël. Karim Ben Khelifa espère le montrer au Congo, ou au Salvador, les pays dont sont originaires ces témoins.
CAMBRIDGE, Mass. — Sitting in a conference room at a hotel near the Massachusetts Institute of Technology here, I slip on large headphones and an Oculus Rift virtual reality headset and wriggle into the straps of a backpack, weighed down with a computer and a battery. It feels as if I were getting ready for a spacewalk or a deep-sea dive.
CAMBRIDGE, Mass. — Sitting in a conference room at a hotel near the Massachusetts Institute of Technology here, I slip on large headphones and an Oculus Rift virtual reality headset and wriggle into the straps of a backpack, weighed down with a computer and a battery. It feels as if I were getting ready for a spacewalk or a deep-sea dive.
“The Enemy,” underway at M.I.T., is the result of a collaboration between Mr. Ben Khelifa and Fox Harrell, an associate professor of digital media.
The work grows out of more than half a century of collaborations between the world of art and the worlds of science and technology, spurred by pioneers like Experiments in Art and Technology, begun in 1967 by the Bell Labs engineers Billy Klüver and Fred Waldhauer and the artists Robert Rauschenberg and Robert Whitman. M.I.T. has been at the forefront of such cross-pollination, which has taken off at schools around the world in recent years.
In an interview before I experienced the virtual reality environment, Mr. Ben Khelifa, 44, said the idea emerged from a frustration that deepened over almost 20 years he spent as a photojournalist covering strife — often entrenched, interminable warfare — mostly in the Middle East. “Being a Tunisian growing up in Belgium, I think I always felt like I was wearing two different kinds of shoes,” he said. “In Iraq and Afghanistan, I could sometimes see things about fighters on both sides of the conflict that some other Europeans couldn’t.”
What he saw there was a culture of warfare that often perpetuated itself through misunderstanding and misinformation, with no mechanism for those of opposing sects or political forces to gain a sense of the enemy as a fellow human being. “I began to think, ‘I’m meeting the same people over and over again,’” he said. “I’m seeing people I knew as kids, and now they’re grown-up fighters, in power, fighting the same fight. And you start to think about your work in terms of: ‘Am I helping to change anything? Am I having any impact?’ ” He added: “I thought of myself as a war illustrator. I started calling myself that.”
Over the last two years, as a visiting artist at the university’s Center for Art, Science and Technology, he transformed what he initially conceived of as an unconventional photo and testimonial project involving fighters into a far more unconventional way of hearing and seeing his subjects, hoping to be able to engender a form of empathy beyond the reach of traditional documentary film. He interviewed Mr. Khaled in Gaza and Mr. Peled in Tel Aviv, asking them the same six questions — basic ones like “Who’s your enemy and why?”; “What is peace for you?”; “Have you ever killed one of your enemies?”; “Where do you see yourself in 20 years?”
Then he and a small crew captured three-dimensional scans of the men and photographed them from multiple angles. (He later repeated this process in the Democratic Republic of Congo. He is expanding the project to El Salvador, which is being decimated by gang violence.)
With help from several technology and digital media companies and funding from a collection of prominent foundations, he began to build avatars of his interviewees and ways for them to move and respond inside a virtual world so realistic it makes even a 3-D movie seem like an artifact from the distant past. Mr. Harrell describes it as “long-form journalism in a totally new form.”
“It should have a kind of lush imaginative vitality to it — a kind of lyricism,” he said of the effect that he and Mr. Ben Khelifa have been working to achieve, polishing the project with the help of test viewers since introducing it to mainstream audiences in 2015 at the Tribeca Film Festival. He added: “You have something here you don’t have in any other form of journalism: body language.”
And, indeed, inside the world they have made, the power comes from the feeling of listening to the interviewees speak (you hear Mr. Ben Khelifa’s disembodied voice asking the questions, and the men’s voices answer, overlaid by the voice of an interpreter) as your body viscerally senses a person standing a few feet away from you, his eyes following yours as he talks, his chest rising and falling as he breathes. I listened intently and immediately felt compelled, out of basic politeness, to remain in front of one interviewee until he had finished answering all of his questions before crossing the room to the other man. I could have sworn the 37 minutes that Mr. Ben Khelifa told me I had been inside the world, listening to the first two enemies and then to those from Congo, was no more than 15.
Sofia Ayala, an M.I.T. sophomore, tested the project after I did and emerged — as I did — with a mesmerized flush on her face, a feeling of meeting someone not really there. “It makes it feel so much more personal than just reading about these things online,” she said. “When someone’s right there talking to you, you want to listen.”
While Mr. Ben Khelifa hopes the project will eventually reach large audiences in the way documentary films do now, he said his target audience was “really the next generation of fighters from wherever we are.”
“In many places I’ve been, you’re given your enemy when you’re born,” he said. “You grow up with this ‘other’ always out there. The best we can hope is that the ‘other’ will now be able to come into the same room with you for a while, where you can listen to him, and see him face to face.”
Pinchas Gutter survived a Nazi death camp – and now his story will live on through a hologram that can answer your questions.
Pinchas Gutter goes out of his way to find me biscuits. In a sun-baked living room in his north London home, he opens a packet of Rich Tea, sits down and tells me about the Holocaust.
Gutter was seven years old when the second world war broke out. He lived in the Warsaw ghetto for three and a half years, took part in its uprising, survived six Nazi concentration camps – including the Majdanek extermination camp – and lived through a death march across Germany to Theresienstadt in occupied Czechoslovakia.
“Remembrance is the secret of redemption, while forgetting leads to exile,” he says, quoting the Baal Shem Tov, the founder of Hasidic Judaism.
“For me this whole thing is about not forgetting. Because if you forget, you repeat it over and over again. To me, the great importance of testimony is not to forget what people are capable of.” Sitting opposite him here in Hampstead, meeting his eyes and hearing his voice, the idea of forgetting this extraordinary story seems impossible.
One week later I am staring into Gutter’s eyes again. But these eyes are on a screen in the Alternate Realities strand of Sheffield’s international documentary festival. The version of Gutter projected on the monitor is a prototype for a responsive hologram that will be wheeled into classrooms, lecture halls and museums. The idea is that the audience asks questions and pre-recorded memories from Gutter will respond – much as if you’re talking to the real person. This virtual Gutter meets my gaze and tilts his head when I speak into the microphone.
The project, called New Dimensions in Testimony, was thought up by concept developer Heather Maio, and was made in a collaboration between the University of Southern California’s Institute for Creative Technologies (ICT) and the Shoah Foundation – an organisation dedicated to making and archiving interviews with survivors and witnesses of genocide.
The prototype involved filming extensive interviews with Gutter using an array of cameras and a specialised light stage. The team at ICT then used natural language processing software to help create an interactive version of the video footage, with vocal cues triggering responses from the pool of recorded speech.
Complex algorithms dictate which responses are played back – faintly reminiscent of virtual assistants such as Apple’s Siri and Microsoft’s Cortana, except that while those pieces of software are primed to answer your questions about emails or the weather, New Dimensions in Testimony is based on communicating one man’s experience of genocide.
We don’t consider it an AI. I think of it more like a smart search engine which gives you a conversation-like experience
“We wanted the visitor to experience the discussion and what that means to them,” Maio tells me. “Not the technology that goes around it and makes it work. In fact, we didn’t want them to think there was technology around them at all.”
Maio says 12 Holocaust survivors have so far been interviewed in this manner, adding that there are plans to advance the intelligence of the computer learning – giving the program the ability to track what the recording has said and tailor the order of its responses to fit. Directing the content in this way enhances the sense of talking to a real person, but it also raises questions about authenticity.
When algorithms are choosing the answers and the order in which they are spoken, are the responses those of Gutter or of an AI?
“In a way it’s both,” says David Traum, director for natural language research at the ICT. “We have to distinguish two concepts of response. Everything that is said comes directly from Pinchas. However, the decision of which recording to present as a response to a specific question is made by the technology, based on its training data of how Pinchas and other people have answered questions. In the best cases, the answers are exactly how Pinchas himself would have responded. In some cases, it may seem that he is addressing a somewhat different question, but the response itself is still his.”
Instead of AI, Stephen Smith, executive director of the Shoah Foundation, says we should consider New Dimensions in Testimony as a voice-activated search engine. He emphasises that no new content has been added beyond that in the 25 hours of video footage with Gutter.
“As time goes by and many more users use the content, machine learning will enhance its intelligence – it will understand a wider range of questions as each interaction helps train the system,” he says. “We still don’t consider it an AI, though, because AI assumes the machine is thinking independently. I think of it more like a smart search engine which gives you a conversation-like experience.”
That conversation-like experience is the crux of New Dimensions in Testimony. The project is winning many plaudits including the interactive award and the audience award at the Sheffield documentary festival this week. Smith says audience members have applauded the virtual Gutter, thanked him, argued with him.
“One woman apologised to him for the negligence of the world, telling him she would have willingly traded places with him had she been alive then.”
My personal experience with the project hinged on eye contact. Having Gutter staring at me as he answered my question on the Warsaw uprising, or about whether he still believed in God, was affecting on an instinctive level. Though I was aware that this was an assortment of clips directed by speech recognition software, it was hard not to feel the impression of an interaction in real time. One comparable forthcoming project in this sense is Anne, a VR film of Anne Frank, which will place viewers in a version of the annex where she spent two years hiding from the Nazis.
While New Dimensions in Testimony aims to represent a person’s individual testimony, Anne aims to represent a particular environment.
Both want to keep the memory of the Holocaust alive through interaction, imparting a sense of tangibility on to events rapidly passing from eyewitness testimony. While there are pressing ethical questions – not least the danger of representing real situations as a form of immersive entertainment – previous projects have shown serious scope for documentary use of VR.
Nonny de la Peña’s Project Syria recreates a rocket attack in the Aleppo district of Syria to shocking effect; the National Theatre-produced Home: Aamir is a 360° video account of life in the Calais Jungle camp; Jane Gauntlett’s Dancing with Myself subtly and effectively communicates the experience of living with epilepsy; and the Guardian’s own 6×9 combines VR with more traditional testimony about the effects of solitary confinement.
These projects succeed largely due to their focus on upsetting the viewer’s sense of comfort. Instead of easing you into a cosy virtual playground, these experiences jolt another world into your own. Elsewhere in the Sheffield festival is an example taken to its extreme. The Enemy, an augmented reality piece directed by war photographer Karim Ben Khelifa, involves holding an iPad and seeing Palestinian and Israeli fighters projected on to the surrounding floor. The avatars follow your gaze as you walk around, and speak to you as you approach. The audio responses are all taken from actual interviews, and the 3D figures are modelled on the interviewees.
Once again, the power of the experience comes from the sense of physicality. Like interacting with New Dimensions in Testimony, eye contact and body language communicate something unlike film documentary. And yet, as with the virtual Gutter, there’s also the question about representation. When you make a version of a person that can watch, listen and respond, do you ultimately do more than show a recording of them in a particular moment of history? Do you have a responsibility to show them as a whole person?
“VR invites us to break taboos around how close we get to people, how directly we observe them,” says Mark Atkin, curator of the Alternate Realities programme at the festival. “It has no code of conduct – that might have to come. And it has no rating system – that will come. For now, the honour system seems to be working well but once the technology to create this sits on all of our phones and anyone can manipulate a 3D scan of a human and put it in any environment, a new form of abuse could well emerge. We need to imagine these futures now, in order to be able to deal with them if and when they do happen. »
Whether it is Pinchas Gutter listening or a computer, the most important thing for the man who survived the Holocaust is that the outcome stands as evidence that these events happened. Photograph: USC Shoah Foundation
The team behind New Dimensions in Testimony has been rigorous in its conduct about recording and representing its subject. The responses given are all clear in context, and having spoken to both the real Gutter and the recording, I didn’t feel that he was misrepresented. But when Gutter and his generation are gone, that point of comparison will be harder to define. The Shoah Foundation Institute is aware of its responsibility to portray Gutter truthfully, but when I speak to Maio she accepts that other organisations might not be as meticulous. In Hampstead I ask Pinchas Gutter how it felt to see the virtual version of himself.
“I saw myself sitting there, moving my hands and legs, being a presence and speaking to people,” he tells me. “Not just speaking but answering. In other words, I’m listening. I’m not just speaking, I’m listening.”
Whether it is Gutter listening or a computer, the most important thing for the man who survived the Holocaust is that the outcome stands as evidence that these events happened, and they happened to a real human. Gutter tells me he didn’t talk about the Holocaust at all in the years following the war, but when he did finally speak, he found it cathartic. When he is no longer alive to speak to audiences, he wants that two-way process to carry on.
“It helped me alleviate some of the pain and the difficulty of relating to people,” he says. “I tell them when they speak to me: now you have a piece of me and I have a piece of you. We’re joined together now, in both the good and the bad that can happen.”
Can I give a piece of myself to a hologram? Can it give a piece back?
Perhaps not. Perhaps there’s no way to replicate the power of talking to another body – a body that stands as physical testimony to the events it lived through. Projects such as New Dimensions in Testimony are trying though, and Gutter tells me it’s the closest thing we have.
“To me, this is such a vitally important project. It’s beyond value. There’s almost something holy about it.”
• New Dimensions of Testimony won both the Alternate Realities interactive award and audience award at the 2016 Sheffield Doc Fest
• This article was amended on 21 June 2016 to clarify the quotes of David Traum and Heather Maio.
Emissive, société de production de contenus immersifs, présentait à la presse des premiers extraits de The Enemy, un des premiers documentaires à gros budget pour casques de réalité virtuelle. Une piste future qu’explorent les chaînes de télévision.
France Télévisions est en train de coproduire The Enemy, un ambitieux documentaire journalistique en réalité virtuelle. Un prototype était dévoilé à la presse par la société de production Emissive.
Une expérience journalistique, un récit de guerre susceptible de vous prendre aux tripes. Car la réalité virtuelle permet d’innover dans la narration: le spectateur peut pénétrer directement dans l’événement. Muni d’un casque de réalité virtuelle équipé de capteurs, un joystick de jeu vidéo en main, on évolue dans une pièce, et on écoute les protagonistes d’une guerre. Les scénarios du documentaire portent sur plusieurs conflits au long cours: la guerre Israël-Palestine, le Congo, et un troisième scénario sera consacré au conflit du Salvador ou de la Corée.
A l’origine du projet, le photojournaliste Karim Ben Khelifa, membre invité depuis 2013 du prestigieux Massachussets Institute of Technology. Pour ce documentaire, il a tourné ses interviews sur place, « puis nous avons remodélisé les personnages, pour les intégrer avec leurs témoignages en voix off dans la vidéo », résume Fabien Barratti, cofondateur de Emissive.
Les personnages nous parlent de qui est leur ennemi, pourquoi ils font la guerre et comment ils voient leur avenir. « On est entre deux témoins, deux combattants, on écoute leurs histoires, leurs arguments, à son rythme. L’idée était de créer une rencontre humaine avec des gens que l’on n’a pas l’habitude d’entendre », explique Antonin Lhot, chef de projet web du département Nouvelles écritures France Télévisions.
Un budget de 1,6 million d’euros
« Le projet est un prototype, un test de ce qu’on peut raconter comme histoire à l’ère de la réalité virtuelle. L’installation a vocation à tourner dans les pays concernés, et dans les écoles et lieux publics », poursuit-il. Il faudra attendre fin 2016 pour pouvoir expérimenter le concept.
Assurément, il préfigure de nouvelles formes d’écriture pour les documentaires, alors que les premiers casques de réalité virtuelle (tels le Oculus VR, le Playstation VR de Sony, le Vive de HTC associé à Valve), sont attendus pour le premier semestre 2016.
A strand of the Tribeca Film Festival looks at interactive storytelling, including a project bringing together an Israeli and a Palestinian.
“He’s been to many warzones around the world and got really interested in this idea of empathy and enemies, and how do you humanise your enemy.” Ingrid Kopp, Director of Interactive at the Tribeca Film Institute, is describing one of the projects in her Storyscapes strand.
The Enemy is an installation bringing together an Israeli and a Palestinian as two full-scale human figures. It was created by conflict photographer Karim Ben Khelifa. “I think he got really frustrated by the limits of the photo and the limits of the still image, in some ways, around exploring these ideas,” says Kopp. “So his solution was to use virtual reality as a way to get us to think differently about images and enemies.”
Other projects in Storyscapes include an audio documentary played through headphones to blindfolded listeners and a web series called Do Not Track that incorporates information about each viewer into its episodes. Tom Brook reports on new forms of storytelling emerging at the festival.
At first, if it weren’t for the contraption that makes you look half-alien, half-astronaut, The Enemy, a virtual reality experience on show at the Tribeca Film Festival, would be akin to most exhibitions.
“Fundamentally, as journalists, we’ve always been trying to arouse empathy so that viewers will care for a situation happening miles from their home or to others,”
As you don the awkward oculus rift goggles and the backpack with the umbilical cord that ties you to the operating system, photos taken in the Palestinian Territories appear on opposing sides. As you would when touring a gallery, you approach either wall, focusing your attention on the scene depicted on the first image before moving on to the next. After a few moments, portraits of two adversaries, Abu Khaled, a garrison leader for the Democratic Front for the Liberation of Palestine, and Gilad, an IDF soldier, replace the depictions of everyday struggles. The two men don’t stay inanimate for long. At once, they take shape right in front of you, as sophisticated holograms answering questions about violence and peace.
The virtual conversation mirrors the interactions of an actual one. Come in too close to the simulated fighter, and they’ll draw back. Shift left or right and their gaze will follow you. Take a few step back and the volume of their voice weakens. After a few minutes, once the conversation is over, they fade away, leaving you alone in an empty room. At this point, removing the high-tech gear feels like waking up from an exceptionally vivid dream: you keep a distinct memory of the experience, yet are acutely aware that it was not real.
“I wanted to know what would happen if we took the likeness of the combatants I’ve photographed off the wall and breathed life into them,” says Karim Ben Khelifa, the mastermind behind The Enemy, which received funding and support from the Tribeca Film Institute’s New Media Fund. “How would people react? How would the public engage with them? And what impact would it have on their understanding or on how much they care?”
For the better part of the last decade, the Tunisian photographer has dedicated himself to taking portraits of foes, especially those captive to entrenched conflicts, born with the hatred of the other, such as the Israeli-Palestinian hostilities or the India-Kashmir rift. Ben Khelifa sees it as a way to make the viewer – and, hopefully, bitter rivals – see the human being behind the fighter.
“Fundamentally, as journalists, we’ve always been trying to arouse empathy so that viewers will care for a situation happening miles from their home or to others,” he says. “When I began as a photographer, I wanted to work for the most renowned magazines in order to reach the largest audience and touch the most souls. With time, I grew increasingly frustrated with the lack of results.”
It turned out that three images in a newspaper spread were not enough to bring on change. Nor were a dozen shared online. Nor even fifty or sixty bound in a book or displayed as a show. “The challenge is to devise mechanisms that will grab and hold people’s attention long enough for you to tell a complex story,” he says, two years after he had his first virtual reality experience when he joined the MIT’s Open Documentary Lab.
Virtual reality, with its potential to offer a fully engrossing experience, might just be the way, he believes, though it’s not without obstacles. Producing such an elaborate project is exorbitantly costly and requires a wide set of skills. Over 45 people have worked on The Enemy – some of whom were dedicated to rendering exclusively and with utmost accuracy skin, hair or the slightest movement.
“It is essential to make the presence of the combatant feel as genuine as possible,” says Ben Khelifa. “By reacting to the viewer’s behavior, the hologram is acknowledging his presence, which in turn, creates a cognitive reaction of reciprocity. The hologram appears more real.” So much so, that though participants could walk through the simulated fighter, none of them has done so. During a demonstration in Paris, one woman fled when the combatants appeared, unsure of whether they were going to come after her, and another turned away from Abu Khaled. She wanted to hear what he had to say, but staring the masked man in the eyes made her uneasy.
Since starting the project two years ago and basing himself on the reactions of his users, Ben Khelifa has been tweaking and fine tuning the experience, keeping it as simple and intuitive as possible. “The difficulty is that I can’t fully control how everything unfolds,” he says. “Some of it is in the hand of the user who needs to feel some agency. There’s a fine balance that needs to be achieved between directing the viewer and giving him freedom. I could add an endless amount of sensorial experiences, but that could quickly become overwhelming.”
The prototype for The Enemy is currently on view at the Tribeca Film Festival as part of the Storyscapes programming alongside other interactive experiences. Ben Khelifa hopes that within the next 20 months he will be able to add seven more duos of fighters that will interact with the viewers in different manners. He will also spend that time devising ways to make it more accessible, creating a traveling installation that could welcome more than one person at a time as well as an app version to be experienced at home.
And, he has plans to record with more accuracy the reactions of participants, monitoring their heart rate, keeping track of how close or distant to each combatant they get and noting how attentive they are. “Analyzing people’s responses will give us insights that can help us become more effective storytellers,” he says.
CAMBRIDGE, Mass. — Sitting in a conference room at a hotel near the Massachusetts Institute of Technology here, I slip on large headphones and an Oculus Rift virtual reality headset and wriggle into the straps of a backpack, weighed down with a computer and a battery. It feels as if I were getting ready for a spacewalk or a deep-sea dive.
A strand of the Tribeca Film Festival looks at interactive storytelling, including a project bringing together an Israeli and a Palestinian.
