Virtual reality meets theatre: an accessible experience for deaf and hard-of-hearing people

On 13th November 2024, IULM University (Milan, Italy) hosted an unprecedented event on accessible and inclusive theatre performance. The initiative featured two performances of Family Game VR – a 2020 original multimedia show created and directed by Mimosa Campironi and performed by Alessandro Averone – aimed at an audience of both hearing individuals and people with hearing impairments.

About The Play: The Role of Virtual Reality in Innovating Performing Arts

One of the first examples of mixed reality (MR) theatre worldwide, Family Game VR combines technology with elements of traditional in-person theatre, creating a hybrid format. The unique and innovative nature of this format lies in the combination of a 50-minute-long video –which the audience watches using 360° VR headsets– and a live session during which the audience takes their headsets off and interacts with the actor. The combination of the virtual and the physical part of the show generates a new theatrical dimension where the magic of conventional theatre –the viewers’ physical presence in the same place– merges with an innovative way of experiencing and enjoying the performance, which is therefore both technological and profoundly human.

What is the purpose of virtual reality (VR) in Family Game VR? And what does VR have to do with what the play is about? Campironi uses virtual reality not only as a tool to break the fourth wall and to create an immersive experience, but also as a metaphor for the fragmentation of one’s identity and the many realities one can live in. Inspired by Luigi Pirandello’s works, these themes are the very heart of the play. The plot revolves around a case of mistaken identity, the ambiguity between characters and masks, and the confusion between reality and imagination, leading the audience to question whether what they are witnessing is a real event or just a figment of the characters’ imagination. Such confusion, along with a sense of disorientation, is further enhanced by the fact that a single actor plays all the characters.

Captioning MR Theatre Productions for People with Hearing Impairments: A One-of-A-Kind Experiment

Family Game VR’s innovative and artistic value sparked Professor Francesco Laurenti’s interest: ‘I was very impressed by the challenge posed by the director, one that brings together artistic research and the investigation of VR potential from the early stages of the creative process. And I couldn’t help but add a new challenge – the creation of a new model of theatre captioning, designed to ensure accessibility for deaf and hard-of-hearing people, which, to my knowledge, hadn’t been done yet. As far as I know, ours is one of the first projects of this kind in the world.’

Before delving into the details of the project, it is worth making a few observations on accessible captioning. Accessible captions are not the mere transcription of the dialogue of a play, but a synchronised adaptation of both verbal and non-verbal acoustic information. Meant for deaf and hard-of-hearing audiences, they include ‘every piece of information that depends on the auditory channel’ (Franco and Santiago Araújo 2003, p. 250) and cannot be deduced from the action, such as dialogues, on- and off-screen sound effects, background noise, music, pauses, paralinguistic cues (vocal qualities such as timbre, intonation, tone, volume), etc. Accessible captions should be clearly formulated, use simple language and syntax to facilitate comprehension and reading, adapt idiomatic expressions, avoid ambiguities that could compromise comprehension, be accurate, preserve the integrity of the message and its original intention (Creative Words 2024), and use colours for speaker identification. Often within square brackets, they convey all relevant audio and context information, enhancing accessibility and understanding for people with hearing loss, attention or cognitive deficit, learning difficulties, etc.

Under these premises, the creation of captions for Family Game VR can now be addressed. All acoustic information in the video of the performance was transcribed into the script and put within square brackets (e.g., [voce rotta], [urla], [accento abruzzese]). The text was then divided into segments based on the pauses in the delivery and adapted to the accessibility guidelines and principles of linguistic simplification provided by Mariangela Laurenti (2023, p. 43); these included writing short sentences, using the subject-verb-object (SVO) word order, preferring active over passive verb forms, adding subjects in implied subject situations, in addition to avoiding foreign, idiomatic and metaphorical expressions when possible. Once the linguistic adaptation process was completed, the captions related to the 360° VR video were converted into an .srt file and burnt into the video using Adobe Premiere, while the ones related to the live performance were transcribed into a PowerPoint presentation, with each slide designed to function as a caption during the live projection.

In the early stages of the project, the possibility of including a sign language interpreter in virtual reality headsets was also taken into account. However, after pondering on it, such an option turned out not to be viable, as hearing-impaired viewers might struggle to follow both the actor and the interpreter’s performance and would have to choose whether to focus on one or the other. This would only result in a partial enjoyment of the performance, as well as difficulties in understanding it. Nevertheless, videos in Italian Sign Language (LIS) were included in the theatre programmes that audience members were given as soon as they walked into the room where the event took place.

Accessible Captioning: A Joint Effort

Captioning a play to make it accessible to people with hearing impairments is a challenging task, especially when it combines filmed and live parts, as in the case of Family Game VR. The key to successfully approaching this task lies in working within multidisciplinary teams to address various aspects, like captioning rules, accessibility and technological matters.

To achieve high-quality accessible captions, IULM brought technical, theatrical, and linguistic expertise together in collaboration with several partners: the project was carried out by the International Center for Research on Collaborative Translation IULM, in association with IULM University, Centro EURESIS IULM, KIT Italia, Fondazione Gualandi a favore dei sordi and Istituto dei Sordi di Torino. The project involved Francesco Laurenti, Allegra Mancini, Claudia Pavignano, Lisa Vecchia (International Center for Research on Collaborative Translation IULM); Marta Muscariello (Centro EURESIS IULM); Donatella Codonesu (KIT Italia); Lorenzo Mosna (IULM University). Many thanks to Chiara Sanfelici, Mariangela Laurenti and Martina Biasin (Fondazione Gualandi a favore dei sordi) for their valuable feedback on captions, and to Enrico Dolza and Nicola Della Maggiora (Istituto dei Sordi di Torino) for the creation of the videos in LIS.

To sum up, Family Game VR represents an evolution in the field of experimental theatre, as well as a significant step towards greater equality, cultural accessibility, and the inclusion of people with hearing impairments in recreational activities such as theatre performance. The result is an experience that overcomes physical, sensory, and communication barriers, giving everyone the opportunity to experience art and culture in an authentic and engaging way.

Credits to Family Game VR

Written and directed by Mimosa Campironi

With Alessandro Averone

Lighting design Massimo Galardini

Scenic and costume design Paola Castrignanò

Make-up artist Bruna Calvaresi 

Music by Bertrand

Produced by Teatro Metastasio di Prato and 369gradi

Executive production GoldVR

Selected text Italian and American Playwrights Project 2020/22

References

Creative Words, ‘Audiovisual Content: Closed Captioning and SDH’, Translation & Localization, Creative Words website, 15 July 2024, accessed 8 December 2024, https://creative-words.com/en/audiovisual-content-closed-captioning-and-sdh/.

Creative Words, ‘Audiovisual Translation for The Deaf and Hard-of-Hearing: How It Works and Why It Is Important’, Translation & Localization, Creative Words website, 28 July 2024, accessed 8 December 2024, https://creative-words.com/en/audiovisual-translation-for-the-deaf-and-hard-of-hearing-how-it-works-and-why-it-is-important/.

Franco, E. P. C., and Santiago Araújo, V. L., ‘Reading Television: Checking Deaf People’s Reactions to Closed Subtitling in Fortaleza, Brazil’, in: Gambier, Y. (ed.), Screen translation, St. Jerome Publishing, Manchester, 2003, pp. 249–267.

IULM, ‘Teatro, realtà virtuale e accessibilità’, IULM University website, 8 November 2024, accessed 24 February 2025, https://www.iulm.it/it/news-ed-eventi/news/teatro-realta-virtuale-accessibilita.

Laurenti, M., ‘Adattamento dei testi’, in: Mazzeo, L. (ed.), ACCESs, Una nuova visione dell’accessibilità museale, Un progetto di FIADDA Emilia-Romagna per l’inclusione delle persone sorde, Ante Quem, Bologna, 2023, pp. 31-44.