Nippon Telegraph and Telephone: Inclusive sport experience “Watch goalball by ear” by reproducing the sound field as if the sound source were there.


~ For the first time in history, the visually impaired and the able-bodied can watch a goalball game together using only sound, as students from a school for the blind demonstrated. ??

NTT Corporation (NTT; Headquarters: Chiyoda-ku, Tokyo; Chairman: Jun Sawada) announced that it is creating a very realistic sound-only sports viewing experience that can be enjoyed by both visually impaired and able-bodied people. In addition to using “highly realistic sound image localization technology” which reproduces a sound field as if the sound source were there, which has been developed in the past, NTT has attempted to reproduce the sound that players selectively perceive. from various sounds using the “inclusive design method” which involves goalball players and visually impaired people in the design. It is the first time in history that a very realistic sports game has been created using sound without relying on sight.
On December 17, 2021, in cooperation with the city of Yokohama, we organized a demonstration for the students of the Yokohama City Special Support School for the Visually Impaired (hereinafter referred to as “School for the Blind”) to discover a sports game with sound (“Watch goalball by ear”).

1. Origins

NTT demonstrated “ultra-realistic communication technology: Kirari!” in the biggest sporting event in the world in 2021, applying it to the sailing (* 1), badminton (* 2) and marathon (* 3) events to provide a very realistic experience.
In addition, with the aim of realizing a symbiotic society, we entered into a joint project agreement with the city of Yokohama on November 30, 2020, and discussed together through workshops and other means the means by which the the visually impaired and the able-bodied may like to watch sports together (* 4).
The results of this research are being applied to the “2021 Japan Para Goalball Tournament” to be held in December 2021 to implement a new spectator experience program that reproduces the sound of players and ball movements on the competition field. in three dimensions. In this program, 15 elementary school students and 13 teachers from a school for the blind in Yokohama city experimented and demonstrated this system.

2. Overview of implementation

Until now, the common method of watching games for the visually impaired has been through live radio and television broadcasts. However, with verbal comments, it was difficult to convey the incredible throwing speed of top players, which can be achieved in 0.5 seconds, and the differences in the variety of throws, such as bouncing and spinning throws, this which makes it difficult to feel the intensity of the game as it is.
To solve this problem, we applied the “High Reality Sound Image Localization Technology” developed by NTT to sports games, and reproduced the acoustic space experienced by players on the field in three dimensions by reproducing sound that reflects the positions of the players and the ball. Watching a match using highly realistic sound image localization technology creates the sound space that players hear in the “orientation zone” (the area where players are defending and throwing) of a field. goalball competition.
A total of 100 speakers are placed in the front and rear rows of a pitch-like experiment area (4.5m long x 7.5m wide), and controlling the audio output from each speaker Using this technology, the pitch sound is synthesized to reflect the position information of the players and the ball (Fig. 1).
It is the first time in history that a sports match can be watched with a high level of realism thanks to sound without relying on sight.

Fig 1: Image of the “Watch Goalball by Ear” experiment

:

NTT Yokosuka Research and Development Center 1-1 Hikarino-oka, Yokosuka Town, Kanagawa Prefecture

:

December 17, 2021 (Friday)

:

Japanese Para-Sports Association (JPSA), Yokohama City, Yokohama City Special Support School for the Visually Impaired

3. Technical points

(1) Highly realistic sound image localization technology

The “Highly realistic sound image localization technology” used in this program is an acoustic reproduction technology which reproduces the spatial and physical waves of sound on the basis of a physical model (* 5). The key point of this technology is a large number of loudspeakers (speaker array) densely arranged in a straight line (Fig. 2). By adjusting the playback timing and strength of the sound emitted from each speaker to create a sound focal point at an arbitrary position, the sound field is reproduced as if the sound source existed there.

Fig 2: Loudspeaker network

we explain it concretely with the help of Fig. 3 and Fig. 4. When a sound source is placed in the red circle, the sound propagates in concentric circles (Fig. 3). On the other hand, when the timing and power of the sound are adjusted and played back from the speaker array using this technology, the sound waves are synthesized and a circle similar to that when the sound source is placed in the red circle in front of the speaker array can be drawn (Fig. 4). This allows the sound to be felt as if it is coming out.

Fig 3: How sound travels

Fig 4: Playback from the speaker network

(2) Problems and design of the acoustic space for the visualization of sports with sound

(A) Sound selection for sound image localization

This time, in order to achieve a goal ball game using high-presence sound image localization technology, we used the “inclusive design method” in which goal ball players and visually impaired people participate in the design. (Fig. 5).

Fig 5: Survey using inclusive design methodology.

Among these sounds, players exclude from their awareness the distracting sounds of the opposing team (such as hitting the ground to mask the throwing position) and follow the series of “throwing sounds” such as the player’s running sound with the ball. ball and the background noise of the ball and the ground “like the stroke of a pen” It became clear that we are always looking for the location of the ball by following the Therefore, instead of reproducing the physical acoustic space as It is, we have attempted to reproduce the sounds that players selectively perceive from the various sounds that resonate. In doing so, we aimed to allow people with no competition experience to simulate some of the athletes’ exceptional hearing.

(B) Simplified localization of the sound image

Goalball players can recognize the distance from one end of the goal to the other (9m) in 9 segments of 1m intervals. However, because the spatial resolution of non-athletes is not as good as that of players, it is difficult for them to grasp the position of a pitch course when it is reproduced as is. In this study, we reduced the spatial resolution in the direction of the goal from nine to three (left, center and right) and the rate of recognition of the ball position by the sound of non-athletes was improved.
In addition, we have created “introductory content” for non-sports spectators to give them a lecture on sound localization, so that they can experience sound image localization on various courses. to throw upstream before watching a game. Non-athletes have stated that they have gradually come to understand the pitch course by sound alone. As described above, simplifying the pitch course and creating the introductory content allowed us to create a spectator system that allows non-athletes to experience the sounds players hear during the game.

4. Future projects

NTT will continue to engage with those involved and the local community to research and develop new methods of communication that focus not only on sport, but also on the physical senses.
In addition, we plan to conduct research on methodologies to achieve the well-being of various people by realizing their own way of being through their physical senses and fostering empathy and trust between people, and to propose results for the realization of a symbiotic society.

* 1Ultra-wide 12K composite video live streaming to remote location for all sailing games
https://group.ntt/en/newsrelease/2021/07/01/210701a.html

* 2Live transmission of holographic images of badminton competition
https://group.ntt/en/newsrelease/2021/07/29/210729a.html

* 3Real-time remote cheering of the marathon
https://group.ntt/en/newsrelease/2021/08/07/210807a.html

* 4Workshop in the city of Yokohama
https://www.city.yokohama.lg.jp/city-info/koho-kocho/press/shimin/2021/olypara1203.files/0001_20211202.pdf

* 5Introduction of highly realistic sound image localization technology
https://group.ntt/jp/journal/1710/files/JN20171024.pdf

About NTT Corporation

NTT believes in solving social problems through our business operations by applying technology for good. We help our clients accelerate their growth and innovate for current and new business models. Our services include digital business consulting, technology and managed services for cybersecurity, applications, workplace, cloud, data centers and networks, all supported by our deep industry expertise and innovation. .
As one of the world’s top 5 providers of technology and business solutions, our diverse teams operate in 88 countries and regions and provide services to more than 190 of them. We serve 85% of the Fortune Global 100 companies and thousands of other customers and communities around the world.
For more information on NTT, visit www.global.ntt.


Comments are closed.