Simulation-based language understanding

Together with colleagues in the NTL group, I have been exploring the hypothesis that language exploits many of the same structures used for action, perception, imagination, memory and other neurally grounded processes. This idea brings together previous work in the group on embodied representations (image schemas, x-schemas, etc.) and the assumptions of constructional and cognitive approaches to grammar.

The main structures and processes of our model of language understanding (as described in Bergen and Chang, 2005) are shown above.

  • Linguistic knowledge is represented using constructions (form-meaning mappings), which are expressed using the Embodied Construction Grammar formalism.
  • The analysis process takes an utterance and draws on linguistic knowledge, world knowledge and the current communicative context to produce a semantic specification. The semspec may also undergo a resolution process to further specify any relevant referents.
  • The semantic specification provides parameters for a dynamic simulation using active embodied structures; the meaning of the utterance consists of the simulation and the inferences it produces.
The basic language understanding model has been extended in several directions, including:
  • John Bryant's dissertation work describes a robust, incremental probabilistic construction analyzer that can handle omissions.
  • Eva Mok and I have proposed models of the situational and discourse context, necessary for contextual resolution as well as our models of language acquisition.

Simulation semantics

The simulation-based approach to language understanding provides an attractive means of coping with the open-ended, context-dependent nature of language: constructions need only specify simulation parameters, allowing features of the current context and of richer embodied and world knowledge to influence the result of any particular simulation. A number of phenomena have been addressed:
  • Srini Narayanan's work on modeling metaphor and aspect served as a starting point for simulative-based inference; see here for an overview of how x-schemas are used to model aspectual inference and coercion.
  • Simulation semantics has been applied to model frame-based representation issues, as exemplified by the Commercial Transaction frame. In collaboration with members of the FrameNet project, we have shown how ECG can be used to bridge the gap between semantically tagged FrameNet corpora and the richer inferential mechanisms provided by simulation.

Representative publications

See also language understanding papers and aspect page.