An Unsupervised Visual-Only Voice Activity Detection Approach Using Temporal Orofacial Features

DSpace/Manakin Repository

An Unsupervised Visual-Only Voice Activity Detection Approach Using Temporal Orofacial Features

Show full item record

Title: An Unsupervised Visual-Only Voice Activity Detection Approach Using Temporal Orofacial Features
Author(s):
Tao, Fei;
Hansen, John H. L. (UT Dallas);
Busso, Carlos
Item Type: article
Keywords: Show Keywords
Abstract: Detecting the presence or absence of speech is an important step toward building robust speech-based interfaces. While previous studies have made progress on voice activity detection (VAD), the performance of these systems significantly degrades when subjects employ challenging speech modes that deviate from normal acoustic patterns (e.g., whisper speech), or in noisy/adverse conditions. An appealing approach under these conditions is visual voice activity detection (VVAD), which detects speech using features characterizing the orofacial activity. This study proposes an unsupervised approach that relies only on visual features, and, therefore, is insensitive to vocal style or time-varying background noise. This study proposes an unsupervised approach that relies on visual features. We estimate optical flow variance and geometrical features around lips, extracting the short-time zero crossing rates, short-time variances, and delta features over a small temporal window. These variables are fused using principal component analysis (PCA) to obtain a "combo" feature, which displays a bimodal distributions (speech versus silence). A threshold is automatically determine using the expectation-maximization (EM) algorithm. The approach can be easily transformed into a supervised VVAD, if needed. We evaluate the system in neutral and whisper speech. While speech based VADs generally fail to detect speech activity in whisper speech, given its important acoustic differences, the proposed VVAD achieves near 80% accuracy in both neutral and whisper speech, highlighting the benefits of the system. Copyright
Publisher: International Speech and Communication Association
ISSN: 2308-457X
Persistent Link: http://www.isca-speech.org/archive/interspeech_2015/i15_2302.html
http://hdl.handle.net/10735.1/5107
Bibliographic Citation: Tao, F., J. H. L. Hansen, and C. Busso. 2015. "An unsupervised visual-only voice activity detection approach using temporal orofacial features." INTERSPEECH 2015 (16th Annual Conference of the International Speech Communication Association), p. 2302-2306.
Terms of Use: ©2015 ISCA
Sponsors: Funded in part by NSF grant #IIS-1217104

Files in this item

Files Size Format View
JECS-3626-4709.67.pdf 1003.Kb PDF View/Open Article

This item appears in the following Collection(s)


Show full item record