Abstract
Conventional methods of real time sound effects in 3D graphical and virtual environments relied upon preparing all the needed samples ahead of time and simply replaying them as needed, or parametrically modifying a basic set of samples using physically based techniques such as the spring-damper simulation and modal analysis/synthesis. In this work, we propose to apply the generative adversarial network (GAN) approach to the problem at hand, with which only one generator is trained to produce the needed sounds fast with perceptually indifferent quality. Otherwise, with the conventional methods, separate and approximate models would be needed to deal with different material properties and contact types, and manage real time performance. We demonstrate our claim by training a GAN (more specifically WaveGAN) with sounds of different drums and synthesizing the sounds on the fly for a virtual drum playing environment. The perceptual test revealed that the subjects could not discern the synthesized sounds from the ground truth nor perceived any noticeable delay upon the corresponding physical event.
| Original language | English |
|---|---|
| Title of host publication | Proceedings - 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2018 |
| Publisher | Institute of Electrical and Electronics Engineers Inc. |
| Pages | 144-148 |
| Number of pages | 5 |
| ISBN (Electronic) | 9781538692691 |
| DOIs | |
| Publication status | Published - 2019 Jan 15 |
| Event | 1st IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2018 - Taichung, Taiwan, Province of China Duration: 2018 Dec 10 → 2018 Dec 12 |
Publication series
| Name | Proceedings - 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2018 |
|---|
Conference
| Conference | 1st IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2018 |
|---|---|
| Country | Taiwan, Province of China |
| City | Taichung |
| Period | 18/12/10 → 18/12/12 |
Fingerprint
Keywords
- Generation of immersive environments and virtual worlds
- Machine learning for multimodal interaction
- Multimodal interaction and experiences in VR/AR
ASJC Scopus subject areas
- Computer Science Applications
- Artificial Intelligence
- Media Technology
Cite this
A perceptual evaluation of generative adversarial network real-time synthesized drum sounds in a virtual environment. / Chang, Minwook; Kim, Youngwon Ryan; Kim, Jeonghyun.
Proceedings - 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2018. Institute of Electrical and Electronics Engineers Inc., 2019. p. 144-148 8613649 (Proceedings - 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2018).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
}
TY - GEN
T1 - A perceptual evaluation of generative adversarial network real-time synthesized drum sounds in a virtual environment
AU - Chang, Minwook
AU - Kim, Youngwon Ryan
AU - Kim, Jeonghyun
PY - 2019/1/15
Y1 - 2019/1/15
N2 - Conventional methods of real time sound effects in 3D graphical and virtual environments relied upon preparing all the needed samples ahead of time and simply replaying them as needed, or parametrically modifying a basic set of samples using physically based techniques such as the spring-damper simulation and modal analysis/synthesis. In this work, we propose to apply the generative adversarial network (GAN) approach to the problem at hand, with which only one generator is trained to produce the needed sounds fast with perceptually indifferent quality. Otherwise, with the conventional methods, separate and approximate models would be needed to deal with different material properties and contact types, and manage real time performance. We demonstrate our claim by training a GAN (more specifically WaveGAN) with sounds of different drums and synthesizing the sounds on the fly for a virtual drum playing environment. The perceptual test revealed that the subjects could not discern the synthesized sounds from the ground truth nor perceived any noticeable delay upon the corresponding physical event.
AB - Conventional methods of real time sound effects in 3D graphical and virtual environments relied upon preparing all the needed samples ahead of time and simply replaying them as needed, or parametrically modifying a basic set of samples using physically based techniques such as the spring-damper simulation and modal analysis/synthesis. In this work, we propose to apply the generative adversarial network (GAN) approach to the problem at hand, with which only one generator is trained to produce the needed sounds fast with perceptually indifferent quality. Otherwise, with the conventional methods, separate and approximate models would be needed to deal with different material properties and contact types, and manage real time performance. We demonstrate our claim by training a GAN (more specifically WaveGAN) with sounds of different drums and synthesizing the sounds on the fly for a virtual drum playing environment. The perceptual test revealed that the subjects could not discern the synthesized sounds from the ground truth nor perceived any noticeable delay upon the corresponding physical event.
KW - Generation of immersive environments and virtual worlds
KW - Machine learning for multimodal interaction
KW - Multimodal interaction and experiences in VR/AR
UR - http://www.scopus.com/inward/record.url?scp=85062185396&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85062185396&partnerID=8YFLogxK
U2 - 10.1109/AIVR.2018.00030
DO - 10.1109/AIVR.2018.00030
M3 - Conference contribution
AN - SCOPUS:85062185396
T3 - Proceedings - 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2018
SP - 144
EP - 148
BT - Proceedings - 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2018
PB - Institute of Electrical and Electronics Engineers Inc.
ER -