Academic & Research Interests
Current Projects
Embodied Conversational Agents that build Rapport With their Users
I'm working on developing theoretical and formal models of how conversation relates to and affects human-human and human-computer rapport. Right now this research involves experiments with human participants investigating this relationship, including the analysis of language and nonverbal behavior that may be involved in coordination and grounding between pairs of participants of varying level of rapport. Ultimately, this empirical research is aimed at developing computational models of these phenomena. I'm also involved in research on building such systems for mobile devices, in part with help and funding from Motorola.
Generating Coordinated Language & Iconic Gestures
I'm also working on the problem of planning coordinated natural
language utterances with novel iconic
(pictorial/depictive/representative/etc.) and deictic (pointing)
gestures for embodied conversational agents (ECAs). ECAs are
conversational agents with animated bodies, or systems that allow
people to interact with computers using spoken language and non-verbal
communication. The ECA I'm currently working on is named NUMACK.
An intrinsically interdisciplinary project, this work includes both the development and engineering of computational models along with the collection and analysis of empirical data to inform and motivate the design of these models. The NUMACK architecture employs a range of models and techniques, including:
- A sentence planner that incrementally constructs natural language utterances based on an LTAG grammar and an evolving model of context & the world, and coordinates the structure of multimodal utterances based on their intended interpretation.
- A gesture planner that generates novel iconic gestures, via a generative, unification-based formalism for combining morphological gesture features based on visuo-spatial that features they resemble.
- A kinematic, human body model capable of processing high-level (XML) gesture descriptions, executing them using an motor planning model based on findings from human movement science and neurophysiology, and the coordination of these movements with synthesized speech.
- An experimental dialogue manager based on the Information State approach to dialogue management, allowing for coordination of a host of input and output agents used as multimodal input and output modules.
We have also collected and are currently in the process of analyzing a large video corpus of people giving directions around Northwestern University. Employing established methodologies and developing new techniques for the annotation and analysis of discourse and gesture, we are looking for patterns in the form and meaning of gestures in context. For more information on this research and our international project team, have a look at our Publications and our NUMACK project description.
Past Projects
In September 2003, I completed an MSc in Artificial Intelligence at Edinburgh. I specialized in Natural Language Engineering. I took courses in natural language processing techniques (generation, processing, etc.), theoretical and computational linguistics, probabilistic and statistical techniques in artificial intelligence, machine learning, etc. (Complete List). My masters thesis project looked at generation of planned multi-clausal and multi-sentential discourse (content or text planning) by extending the SPUD natural language generation system to deal with DLTAG, a formalism for representing discourse structure. It was supervised by Johanna Moore and Bonnie Webber My undergrad thesis, supervised by Matthew Stone at Rutgers University also used SPUD to explore syntactic and semantic representations of verbs and events for Natural Language Generation.
General Research Interests
My current research interests fall under the general areas of natural language technology, computational linguistics and human computer interaction. More specifically (but in no particular order):
- dialogue systems & discourse
- human computer interaction (HCI), esp. multimodal HCI
- natural language generation & understanding
- embodied conversational agents
- computational pragmatics, semantics & lexical semantics
- knowledge representation
- mixing symbolic and machine learning approaches in language technology
- non-verbal communication, interpersonal communication
In coursework and research, I've explored all of these areas.
I'm also interested in User Interface (UI) design,
theoretical linguistics, psycholinguistics, sociolinguistics,
philosophy (philosophy of mind, ph. of language, ph. of psychology,
social and political philosophy, etc. . .), graphic design, music, art,
cognition and cognitive science more generally.
What
I think Natural Language Technology, AI and
Cognitive Science
are all about: A Non-Technical Explanation