Original at

04/19/2000

Multiplying Meaning: Literacy in a Multimedia World by Lemke, J. L.

ERIC ABSTRACT: As material objects, texts are as much the product of visual semiotic codes as of linguistic ones. And throughout history, verbal texts have been combined with nonverbal, visual modes of presenting information, taking a stance toward information and readers, and organizing parts into wholes. The major challenge to creating multimodal texts in the near future will be a lack of multimedial literacy. A more fundamental understanding of existing cultural conventions in communities for combining verbal and nonverbal elements in multimedial texts is needed. To understand how meaning is made simultaneously in several semiotic modalities, common features of all semiotic systems must be identified, i.e., the presentational, the orientational, and the organizational features. Scientific and technical texts have long preserved a tradition of incorporating nonverbal visual-graphic elements as integral and normal parts of their genres. What it means to "read" a text of this kind depends on the literacy practice involved; that is, on the cultural activity as part of which meaning is being made with this text.

Multiplying Meaning: Visual and Verbal Semiotics in Scientific text by J.L. LEMKE

Multimedia Semiotics

Scientific research articles and other genres of formal scientific communication in print rely heavily on the use of visual representations such as graphs, tables, diagrams, and drawings as well as mathematical expressions. How are these symbolic presentations integrated with those made through normally textualized verbal language? How do we make meaning with such multimedia texts? What specific kinds of meanings have these multimedia genres evolved to help us make?

This report on my current research-in-progress sketches a theoretical framework for investigating these questions and communicate some very preliminary findings. I will argue that human communication normally deploys the resources of multiple semiotic systems and combines them according to essentially functional principles.

Scientific communication in particular seeks to make meanings that overflow the preponderantly typological principles of linguistic semantics and require their integration with the more topological modalities of visual semiotics and their extension through the hybrid resources of quantitative mathematics. Also, the report includes results of two preliminary surveys of the types and frequencies of non-textual presentations in formal scientific print communication, and offers some semiotic analyses of the functional (presentational, orientational, and organizational) integration of text, tables, graphs, diagrams and drawings in these multimedia genres.

Social semiotics seeks to explicate how we make meaning with all the resources at our disposal: linguistic, pictorial, gestural, musical, choreographic, and most generally actional. But we never make meaning with language alone.

In multimedia genres, meanings made with each functional resource in each semiotic modality can modulate meanings of each kind in each other semiotic modality, thus multiplying the set of possible meanings that can be made (and so also the specificity of any particular meaning made against the background of this larger set of possibilities).

This combinatorial semiotic principle provides not just a theoretical framework, but an analytical engine for investigating multimedia semiotics.

Language, and typological modes of semiosis generally, have evolved to work in partnership with other, often more topologically grounded, semiotic systems. It is my hope that by studying scientific and computer hypermedia genres, we can explore the powerful mysteries of that partnership in order to better understand just how we make sense of our world and ourselves.