Overview of Research

My research interests lie in machine learning techniques and their applications to Behavioral Signal Processing domain, and aims to employ artificial intelligence techniques to model and understand human behaviors through acoustic and lexical features.

We build computational models for human-centered behavioral understanding system, including dynamic modeling for couples’ interactions, DNN based behavior recognition system.

Along the way, we have actively collaborated with researchers in diverse disciplines, including computer science, psychology and electrical engineering.

Current Projects

DNN based behavioral recognition system

alt text 
  • Speech acoustic features can provide complementary information of human behavioral descriptions. However, behavior recognition from speech remains a challenging task since it is difficult to find generalizable and representative features because of noisy and high-dimensional data, especially when data is limited and annotated coarsely and subjectively.

  • We propose a Sparsely-Connected and Disjointly-Trained DNN (SD-DNN) framework to deal with limited data. First, we break the acoustic feature set into subsets and train multiple distinct classifiers. Then, the hidden layers of these classifiers become parts of a deeper network that integrates all feature streams. The overall system allows for full connectivity while limiting the number of parameters trained at any time and allows convergence possible with even limited data.

  • We can also provide domain experts with human behavioral change trajectory, showing the viability towards live behavior annotations.


  • Haoqi Li, Brian Baucom, Panayiotis Georgiou. “Sparsely Connected and Disjointly Trained Deep Neural Networks for Low Resource Behavioral Annotation: Acoustic Classification in Couples’ Therapy”, submitted to Interspeech 2016.

Audio to Behavior processing pipeline

  • Audio to Behavioral understanding system, including VAD, speaker diarization, acoustic feature extraction, ASR and behavior recognition.