Coh-Metrix version 3.0 indices

 

Table of contents

I. General overview
II. Overview of Coh-Metrix indices (output file)

III. Indices in the Coh-Metrix 3.0 output file

 

1. Descriptive

2. Text Easability Principal Component Scores

3. Referential Cohesion

4. LSA

5. Lexical Diversity

6. Connectives

7. Situation Model

8.Syntactic Complexity

9.Syntactic Pattern Density

10. Word Information

11. Readability

 

IV.References

V.Example Text and Output

 

 

 

I. General Overview

 

Coh-Metrix is a computational tool that produces indices of the linguistic and discourse representations of a text. These values can be used in many different ways to investigate the cohesion of the explicit text and the coherence of the mental representation of the text. Our definition of cohesion consists of characteristics of the explicit text that play some role in helping the reader mentally connect ideas in the text (Graesser, McNamara, & Louwerse, 2003). The definition of coherence is the subject of much debate. Theoretically, the coherence of a text is defined by the interaction between linguistic representations and knowledge representations. When we put the spotlight on the text, however, coherence can be defined as characteristics of the text (i.e., aspects of cohesion) that are likely to contribute to the coherence of the mental representation.  Coh-Metrix provides indices of such cohesion characteristics.

1. Preliminary information

This document contains a description of 108 indices incorporated into the website Coh-Metrix version 3.0. These descriptions are intended to be succinct specifications for people who want to work on this version of Coh-Metrix. More theoretical information on the Coh-Metrix indices and architecture is reported in Graesser, McNamara, Louwerse and Cai (2004) and Graesser and McNamara (2011). For each level of indices, example scores are provided.  However, it is important to note that even small changes in texts can lead to large changes in Coh-Metrix output. It is also important to note that the scores are often subject to the output of third party parsers, lexicons, and word frequency databases, all of which are outside of the control of Coh-Metrix.

2. Coh-Metrix concepts

Some definitions of key concepts are needed to specify the algorithms that underlie the indices in Coh-Metrix 3.0.  We define these concepts in this section.

 

Adjacent versus All sentences

 

Adjacent sentences are successive sentences in a span of text.  For example, if a span of text has 4 sentences, then the adjacent sentences would be sentences 1-2, 2-3, and 3-4. In contrast, all sentences are all possible pairs of sentences: 1-2, 2-3, 3-4, 1-3, 1-4, and 2-4. A span of text may be defined in different ways for different purposes, but there is a distinction between paragraph spans and the span of an entire document. The adjacent sentences in Coh-Metrix 3.0 ignore junctures between paragraphs. 

 

 

Weighted versus Unweighted distances between sentences

 

 

This distinction is pervasive in a more advanced version of Coh-Metrix (1.2), but not in most indices used in Coh-Metrix 3.0. When the distance between sentences is weighted, the weight between two sentences decreases the further they are apart in the text. All sentence pairs have equal weight when distances are unweighted. With rare exception, distances between sentences are unweighted in Coh-Metrix 3.0.

 

 

Incidence scores versus Ratio scores

 

An incidence score is the number of classified units per 1000 words. For example, the incidence score for pronouns would compute the number of words that are classified as pronouns for a span of 1000 words. It is equivalent to what some researchers call rates or density scores. In contrast, a ratio score is a relative measure that compares the incidence of one class of units to the incidence of another class of units. For example, a pronoun ratio is the incidence of pronouns divided by the incidence of noun-phrases. Ratio scores compare two different metrics (classes of units) whereas an incidence score applies to only one metric.

 

Repetition score

A repetition score is computed on sequences of text units that are classified into categories.  This score is the proportion of adjacent pairs of units in the sequence that are in the same category.  If there are N units in a sequence, there are (N-1) adjacent pairs.  The number of adjacent pairs in the same category is divided by N-1. For example, we have computed the repetition score for a sequence of categories A, B, and C:

 

Category sequence: A BB B C A A C C B B B B A C C

Adjacency repetition0 1 1 0 0 1 0 1 0 1 1 1 0 0 1

The repetition score for this sequence is 8 / 15.

 

 

II. Overview of Coh-Metrix 3.0 Indices

 

This Appendix provides the list of indices in Coh-Metrix Version 3.0. The first column provides the label that appears in the output in the current version. The second column provides the label used in prior versions of Coh-Metrix. The third column provides a short description of the index.

 

 

Table 1: Indices in the Coh-Metrix 3.0 Output File

 

 

 

Title Title
Genre Genre
Source Source
UserCode UserCode
LSASpace LSASpace
Date Date
Time Time
  Label in Version 3.x Label in Version 2.x Description
Descriptive
1 DESPC READNP Paragraph count, number of paragraphs
2 DESSC
READNS Sentence count, number of sentences
3 DESWC READNW Word count, number of words
4 DESPL READAPL Paragraph length, number of sentences, mean
5 DESPLd n/a Paragraph length, number of sentences, standard deviation
6 DESSL READASL Sentence length, number of words, mean
7 DESSLd n/a Sentence length, number of words, standard deviation
8 DESWLsy READASW Word length, number of syllables, mean
9 DESWLsyd n/a Word length, number of syllables, standard deviation
10 DESWLlt n/a Word length, number of letters, mean
11 DESWLltd n/a Word length, number of letters, standard deviation
Text Easability Principal Component Scores
12 PCNARz n/a Text Easability PC Narrativity, z score
13 PCNARp n/a Text Easability PC Narrativity, percentile
14 PCSYNz n/a Text Easability PC Syntactic simplicity, z score
15 PCSYNp n/a Text Easability PC Syntactic simplicity, percentile
16 PCCNCz n/a Text Easability PC Word concreteness, z score
17 PCCNCp n/a Text Easability PC Word concreteness, percentile
18 PCREFz n/a Text Easability PC Referential cohesion, z score
19 PCREFp n/a Text Easability PC Referential cohesion, percentile
20 PCDCz n/a Text Easability PC Deep cohesion, z score
21 PCDCp n/a Text Easability PC Deep cohesion, percentile
22 PCVERBz n/a Text Easability PC Verb cohesion, z score
23 PCVERBp n/a Text Easability PC Verb cohesion, percentile
24 PCCONNz n/a Text Easability PC Connectivity, z score
25 PCCONNp n/a Text Easability PC Connectivity, percentile
26 PCTEMPz n/a Text Easability PC Temporality, z score
27 PCTEMPp n/a Text Easability PC Temporality, percentile
Referential Cohesion
28 CRFNO1 CRFBN1um Noun overlap, adjacent sentences, binary, mean
29 CRFAO1 CRFBA1um Argument overlap, adjacent sentences, binary, mean
30 CRFSO1 CRFBS1um Stem overlap, adjacent sentences, binary, mean
31 CRFNOa CRFBNaum Noun overlap, all sentences, binary, mean
32 CRFAOa CRFBAaum Argument overlap, all sentences, binary, mean
33 CRFSOa CRFBSaum Stem overlap, all sentences, binary, mean
34 CRFCWO1 CRFPC1um Content word overlap, adjacent sentences, proportional, mean
35 CRFCWO1d n/a Content word overlap, adjacent sentences, proportional, standard deviation
36 CRFCWOa CRFPCaum Content word overlap, all sentences, proportional, mean
37 CRFCWOad n/a Content word overlap, all sentences, proportional, standard deviation
38 CRFANP1 CREFP1u Anaphor overlap, adjacent sentences
39 CRFANPa CREFPau Anaphor overlap, all sentences
LSA
40 LSASS1 LSAassa LSA overlap, adjacent sentences, mean
41 LSASS1d LSAassd LSA overlap, adjacent sentences, standard deviation
42 LSASSp LSApssa LSA overlap, all sentences in paragraph, mean
43 LSASSpd LSApssd LSA overlap, all sentences in paragraph, standard deviation
44 LSAPP1 LSAppa LSA overlap, adjacent paragraphs, mean
45 LSAPP1d LSAppd LSA overlap, adjacent paragraphs, standard deviation
46 LSAGN LSAGN LSA given/new, sentences, mean
47 LSAGNd n/a LSA given/new, sentences, standard deviation
Lexical Diversity
48 LDTTRc TYPTOKc Lexical diversity, type-token ratio, content word lemmas
49 LDTTRa n/a Lexical diversity, type-token ratio, all words
50 LDMTLDa LEXDIVTD Lexical diversity, MTLD, all words
51 LDVOCDa LEXDIVVD Lexical diversity, VOCD, all words
Connectives
52 CNCAll CONi All connectives incidence
53 CNCCaus CONCAUSi Causal connectives incidence
54 CNCLogic CONLOGi Logical connectives incidence
55 CNCADC CONADVCONi Adversative and contrastive connectives incidence
56 CNCTemp CONTEMPi Temporal connectives incidence
57 CNCTempx CONTEMPEXi Expanded temporal connectives incidence
58 CNCAdd CONADDi Additive connectives incidence
59 CNCPos n/a Positive connectives incidence
60 CNCNeg n/a Negative connectives incidence
Situation Model
61 SMCAUSv CAUSV Causal verb incidence
62 SMCAUSvp CAUSVP Causal verbs and causal particles incidence
63 SMINTEp INTEi Intentional verbs incidence
64 SMCAUSr CAUSC Ratio of casual particles to causal verbs
65 SMINTEr INTEC Ratio of intentional particles to intentional verbs
66 SMCAUSlsa CAUSLSA LSA verb overlap
67 SMCAUSwn CAUSWN WordNet verb overlap
68 SMTEMP TEMPta Temporal cohesion, tense and aspect repetition, mean
Syntactic Complexity
69 SYNLE SYNLE Left embeddedness, words before main verb, mean
70 SYNNP SYNNP Number of modifiers per noun phrase, mean
71 SYNMEDpos MEDwtm Minimal Edit Distance, part of speech
72 SYNMEDwrd MEDawm Minimal Edit Distance, all words
73 SYNMEDlem MEDalm Minimal Edit Distance, lemmas
74 SYNSTRUTa STRUTa Sentence syntax similarity, adjacent sentences, mean.
75 SYNSTRUTt STRUTt Sentence syntax similarity, all combinations, across paragraphs, mean
Syntactic Pattern Density
76 DRNP n/a Noun phrase density, incidence
77 DRVP n/a Verb phrase density, incidence
78 DRAP n/a Adverbial phrase density, incidence
79 DRPP n/a Preposition phrase density, incidence
80 DRPVAL AGLSPSVi Agentless passive voice density, incidence
81 DRNEG DENNEGi Negation density, incidence 
82 DRGERUND GERUNDi Gerund density, incidence
83 DRINF INFi Infinitive density, incidence
Word Information
84 WRDNOUN NOUNi Noun incidence
85 WRDVERB VERBi Verb incidence
86 WRDADJ ADJi Adjective incidence
87 WRDADV ADVi Adverb incidence
88 WRDPRO DENPRPi Pronoun incidence
89 WRDPRP1s n/a First person singular pronoun incidence
90 WRDPRP1p n/a First person plural pronoun incidence
91 WRDPRP2 PRO2i Second person pronoun incidence
92 WRDPRP3s n/a Third person singular pronoun incidence
93 WRDPRP3p n/a Third person plural pronoun incidence
94 WRDFRQc FRCLacwm CELEX word frequency for content words, mean
95 WRDFRQa FRCLaewm CELEX Log frequency for all words, mean
96 WRDFRQmc FRCLmcsm CELEX Log minimum frequency for content words, mean
97 WRDAOAc WRDAacwm Age of acquisition for content words, mean
98 WRDFAMc WRDFacwm Familiarity for content words, mean
99 WRDCNCc WRDCacwm Concreteness for content words, mean
100 WRDIMGc WRDIacwm Imagability for content words, mean
101 WRDMEAc WRDMacwm Meaningfulness, Colorado norms, content words, mean
102 WRDPOLc POLm Polysemy for content words, mean
103 WRDHYPn HYNOUNaw Hypernymy for nouns, mean
104 WRDHYPv HYVERBaw Hypernymy for verbs, mean
105 WRDHYPnv HYPm Hypernymy for nouns and verbs, mean
Readability
106 RDFRE READFRE Flesch Reading Ease
107 RDFKGL READFKGL Flesch-Kincaid Grade Level
108 RDL2 L2 Coh-Metrix L2 Readability

 

 

III. Indices in the Coh-Metrix 3.0 output file

 

The indices in Coh-Metrix 3.0 are categorized into eleven groups: (1) Descriptive, (2) Text Easability Principal Component Scores, (3) Referential Cohesion, (4) LSA, (5) Lexical Diversity, (6) Connectives, (7) Situation Model, (8) Syntactic Complexity, (9) Syntactic Pattern Density, (10) Word Information, and (11) Readability.

1. Descriptive Indices

Coh-Metrix provides descriptive indices to help the user to check the Coh-Metrix output (e.g., to make sure that the numbers make sense) and also to interpret patterns of data. The extracted indices include those listed below. In the output for the current version of Coh-Metrix (Version 3.0) all of these indices are preceded by DES to designate that they are descriptive measures.

Number of paragraphs (DESPC). (index 01)

 

This is the total number of paragraphs in the text. Paragraphs are simply delimited by a hard return.

 

 

Number of sentences (DESSC). (index 02)

 

This is the total number of sentences in the text. Sentences are identified by the OpenNLP sentence splitter (http://opennlp.sourceforge.net/projects.html).

 

 

Number of words (DESWC). (index 03)

 

This is the total number of words in the text. Words are calculated using the output from the Charniak parser. For each sentence, the Charniak parser generates a parse tree with part of speech (POS) tags for clauses, phrases, words and punctuations. The elements on the leaves of a parse tree are tagged words or punctuations. In Coh-Metrix, words are taken from the leaves of the sentence parse trees.

 

 

Mean length of paragraphs (DESPL). (index 04)

 

This is the average number of sentences in each paragraph within the text. Longer paragraphs may be more difficult to process.

 

 

Standard deviation of the mean length of paragraphs (DESPLd). (index 05)

 

This is the standard deviation of the measure for the mean length of paragraphs within the text. In the output, d is used at the end of the name of the indices to designate that it is a standard deviation. A large standard deviation indicates that the text has large variation in terms of the lengths of its paragraphs, such that it may have some very short and some very long paragraphs. The presence of headers in a short text can increase values on this measure.

 

 

Mean number of words (length) of sentences in (DESSL). (index 06)

 

This is the average number of words in each sentence within the text, where a word is anything that is tagged as a part-of-speech by the Charniak parser. Sentences with more words may have more complex syntax and may be more difficult to process. While this is a descriptive measure, this also provides one commonly used proxy for syntactic complexity. However, Coh-Metrix provides additional more precise measures of syntactic complexity discussed later in this chapter.

 

 

Standard deviation of the mean length of sentences (DESSLd). (index 07)

 

This is the standard deviation of the measure for the mean length of sentences within the text. A large standard deviation indicates that the text has large variation in terms of the lengths of its sentences, such that it may have some very short and some very long sentences. The presence of headers in a short text may impact this measure. Narrative text may also have variations in sentence length as authors move from short character utterances to long descriptions of scenes.

 

 

Mean number of syllables (length) in words (DESWLsy). (index 08)

 

Coh-Metrix calculates the average number of syllables in all of the words in the text. Shorter words are easier to read and the estimate of word length serves as a common proxy for word frequency.

 

 

Standard deviation of the mean number of syllables in words (DESWLsyd). (index 09)

 

This is the standard deviation of the measure for the mean number of syllables in the words within the text. A large standard deviation indicates that the text has large variation in terms of the lengths of its words, such that it may have both short and long words.

 

 

Mean number of letters (length) in words (DESWLlt). (index 10)

 

This is the average number of letters for all of the words in the text. Longer words tend to be lower in frequency or familiarity to a reader.

 

 

Standard deviation of the mean number of letter in words (DESWLltd). (index 11)

 

This is the standard deviation of the measure for the mean number of letters in the words within the text. A large standard deviation indicates that the text has large variation in terms of the lengths of its words, such that it may have both short and long words.

 

 

2. Text Easability Principal Component Scores

 

Recent work has culminated in the development of the Coh-Metrix Easability components (Graesser, McNamara, & Kulikowich, 2011). These components provide a more complete picture of text ease (and difficulty) that emerge from the linguistic characteristics of texts. The Easability components provided by Coh-Metrix go beyond traditional readability measures by providing metrics of text characteristics on multiple levels of language and discourse. Moreover, they are well-aligned with theories of text and discourse comprehension (e.g., Graesser, Singer, & Trabasso, 1994; Graesser & McNamara, 2011; Kintsch, 1998; McNamara & Magliano, 2009).

 

Narrativity: PCNARz, PCNARp(index 12, 13)

Narrative text tells a story, with characters, events, places, and things that are familiar to the reader. Narrative is closely affiliated with everyday, oral conversation. This robust component is highly affiliated with word familiarity, world knowledge, and oral language. Non-narrative texts on less familiar topics lie at the opposite end of the continuum.


Syntactic Simplicity: PCSYNz, PCSYNp(index 14, 15)

This component reflects the degree to which the sentences in the text contain fewer words and use simpler, familiar syntactic structures, which are less challenging to process. At the opposite end of the continuum are texts that contain sentences with more words and use complex, unfamiliar syntactic structures.


Word Concreteness: PCCNCz, PCCNCp(index 16, 17)

Texts that contain content words that are concrete, meaningful, and evoke mental images are easier to process and understand. Abstract words represent concepts that are difficult to represent visually. Texts that contain more abstract words are more challenging to understand.


Referential Cohesion: PCREFz, PCREFp (index 18, 19)

A text with high referential cohesion contains words and ideas that overlap across sentences and the entire text, forming explicit threads that connect the text for the reader. Low cohesion text is typically more difficult to process because there are fewer connections that tie the ideas together for the reader.


Deep Cohesion: PCDCz, PCDCp (index 20, 21)

This dimension reflects the degree to which the text contains causal and intentional connectives when there are causal and logical relationships within the text. These connectives help the reader to form a more coherent and deeper understanding of the causal events, processes, and actions in the text. When a text contains many relationships but does not contain those connectives, then the reader must infer the relationships between the ideas in the text. If the text is high in deep cohesion, then those relationships and global cohesion are more explicit.



Verb Cohesion: PCVERBz, PCVERBp (index 22, 23)

This component reflects the degree to which there are overlapping verbs in the text. When there are repeated verbs, the text likely includes a more coherent event structure that will facilitate and enhance situation model understanding. This component score is likely to be more relevant for texts intended for younger readers and for narrative texts (McNamara, Graessar, &Louwerse, 2012).


Connectivity: PCCONNz, PCCONNp (index 24, 25)

This component reflects the degree to which the text contains explicit adversative, additive, and comparative connectives to express relations in the text. This component reflects the number of logical relations in the text that are explicitly conveyed. This score is likely to be related to the reader’s deeper understanding of the relations in the text.



Temporality: PCTEMPz, PCTEMPp (index 26, 27)

Texts that contain more cues about temporality and that have more consistent temporality (i.e., tense, aspect) are easier to process and understand. In addition, temporal cohesion contributes to the reader’s situation model level understanding of the events in the text.

 

 

3. Referential Cohesion

Referential cohesion refers to overlap in content words between local sentences, or co-reference. In the output for the current version of Coh-Metrix (Version 3.0), all of these indices are preceded by CRF to designate that they are co-reference measures. Co-reference is a linguistic cue that can aid readers in making connections between propositions, clauses, and sentences in their textbase understanding (Halliday & Hasan, 1976; McNamara & Kintsch, 1996). Coh-Metrix measures for referential cohesion vary along two dimensions. First, the indices vary from local to more global. Local cohesion is measured by assessing the overlap between consecutive, adjacent sentences, whereas global cohesion is assessed by measuring the overlap between all of the sentences in a paragraph or text. Additional information about the co-reference measures are provided below.

 

Noun overlap (CRFNO1 and CRFNOa). (index 28, 31)  

These are measures of local and global overlap between sentences in terms of nouns. Adjacent noun overlap (CRFNO1) represents the average number of sentences in the text that have noun overlap from one sentence back to the previous sentence. Among the co-reference measures, it is the most strict, in the sense that the noun must match exactly, in form and plurality.

Whereas local overlap considers only adjacent sentences, global overlap (CRFNOa) considers the overlap of each sentence with every other sentence. As shown in Table 4.1, just over 50 percent of the adjacent sentences contained an overlapping noun, and 43 percent of the sentence pairs in the text contained an overlapping noun when comparing all of the sentences (global overlap).

 

Argument overlap (CRFAO1 and CRFAOa). (index 29, 32)

These local and global overlap measures are similar to noun overlap measures, but include overlap between sentences in terms of nouns and pronouns. Argument overlap occurs when there is overlap between a noun in one sentence and the same noun (in singular or plural form) in another sentence; it also occurs when there are matching personal pronouns between two sentences (e.g., he/he). The term argument is used in a linguistic sense, where noun/pronoun arguments are contrasted with verb/adjective predicates (Kintsch & Van Dijk, 1978). Consider argument overlap for the science passage in Table 4.1 in the second column. Note that in comparison to noun overlap, it is less strict because it considers the overlap for example between cells and cell. Argument and stem overlap would also include overlap between pronouns, such as it to it, or he to he, which noun overlap does not include.

 

Stem overlap (CRFSO1, CRFSOa). (index 30, 33)

These two local and global overlap measures relax the noun constraint held by the noun and argument overlap measures. A noun in one sentence is matched with a content word (i.e., nouns, verbs, adjectives, adverbs) in a previous sentence that shares a common lemma (e.g., tree/treed; mouse/mousey; price/priced). Notably, the outcome for stem and argument overlap in Table 4.1 were identical; however, this will not always be the case.

 

Content word overlap (CRFCWO1, CRFCWO1d, CRFCWOa, CRFCWOad) . (index 34, 35, 36, 37)

This measure considers the proportion of explicit content words that overlap between pairs of sentences. For example, if a sentence pair has fewer words and two words overlap, the proportion is greater than if a pair has many words and two words overlap. This measure includes both local (CRFCWO1) and global (CRFCWOa) indices, and also includes their standard deviations (CRFCWO1d, CRFCWOad). In the example provided in Table 4.1, the content word overlap both locally and globally was lower than that estimated by the binary overlap scores. This measure may be particularly useful when the lengths of the sentences in the text are a principal concern.

 

Anaphor overlap (CRFANP1, CRFANPa) (index 38,39)

This measure considers the anphor overlap between pairs of sentences. A pair of sentences has an anphor overlap if the later sentence contains a pronoun that refers to a pronoun or noun in the earlier sentence. The score for each pair of sentences is binary, i.e., 0 or 1. The measure of the text is the average of the pair scores. This measure includes both local (CRFANP1) and global (CRFANPa) indice.

 

4. Latent Semantic Analysis

Latent Semantic Analysis (LSA; Landauer et al., 2007) provides measures of semantic overlap between sentences or between paragraphs. Coh-Metrix 3.0 provides eight LSA indices. Each of these measures varies from 0 (low cohesion) to 1 (high cohesion).

 

LSA sentence adjacent: LSASS1(index 40)

 

This index computes mean LSA cosines for adjacent, sentence-to-sentence (abbreviated as "ass") units. This measures how conceptually similar each sentence is to the next sentence.

 

Example:

 

Text 1: The field was full of lush, green grass. The horses grazed peacefully. The young children played with kites. The women occasionally looked up, but only occasionally. A warm summer breeze blew and everyone, for once, was almost happy.

 

Text 2: The field was full of lush, green grass. An elephant is a large animal. No-one appreciates being lied to. What are we going to have for dinner tonight?

 

In the example texts printed above, Text 1 records much higher LSA scores than Text 2. The words in Text 1 tend to be thematically related to a pleasant day in an idyllic park scene: green, grass, children, playing, summer, breeze, kites, andhappy, In contrast, the sentences in Text 2 tend to be unrelated.

 

 

LSASS1d (index 41)

 

This index computes standard deviation of LSA cosines for adjacent, sentence-to-sentence (abbreviated as "ass") units. This measures how consistent adjacent sentences are overlaped semantically.

 

LSA sentence all: LSASSp (index 42)

 

Like LSA sentence adjacent (LSAassa), this index computes mean LSA cosines.  However, for this index all sentence combinations are considered, not just adjacent sentences. LSApssa computes how conceptually similar each sentence is to every other sentence in the text.

 

 

LSASSpd (index 43)

 

This index computes the standard deviation of LSA cosine of all sentence pairs within paragraphs.


LSAPP1 (index 44)

 

This index computes the mean of the LSA cosines between adjacent paragraphs.

 

 

LSAPP1d (index 45)

 

This index is the standard deviation of LSA cosinces between adjacent paragraphs.

 

 

LSAGN (index 46)

 

This is the avarage givenness of each sentence.

 

 

LSAGNd (index 47)

 

This is the standard deviation of giveness of each sentence.

 

 

5. Lexical Diversity

Lexical diversity refers to the variety of unique words (types) that occur in a text in relation to the total number of words (tokens). When the number of word types is equal to the total number of words (tokens), then all of the words are different. In that case, lexical diversity is at a maximum, and the text is likely to be either very low in cohesion or very short. A high number of different words in a text indicates that new words need to be integrated into the discourse context. By contrast, lexical diversity is lower (and cohesion is higher) when more words are used multiple times across the text.

 

 

Type-token ratio: LDTTRc (index 48)

 

Type-token ratio (TTR) (Templin, 1957) is the number of unique words (called types) divided by the number of tokens of these words. Each unique word in a text is considered a word type. Each instance of a particular word is a token. For example, if the word dog appears in the text 7 times, its type value is 1, whereas its token value is 7. When the type-token ratio approaches 1, each word occurs only once in the text; comprehension should be comparatively difficult because many unique words need to be decoded and integrated with the discourse context. As the type-token ratio decreases, words are repeated many times in the text, which should increase the ease and speed of text processing. Type-token ratios are computed for content words, but not function words. TTR scores are most valuable when texts of similar lengths are compared.

 

Example:

 

Cytokinesis, the second stage of cell division, begins to occur before mitosis is complete (usually during telophase) and continues after the nuclei of the daughter cells are completely formed. The preliminary steps of cytokinesis occur during the growth interphases (called the G phases) of the cell cycle.

 

In these sentences (taken from the text reprinted later in the help facility), the TTR for content words is 0.933. Words such as stage only occur once, but words like cytokinesis and cell appear more than once. Coh-Metrix uses lexeme versions in its calculation rather than lemma or stem versions; for example, cell is considered different from cells.

 

 

LDTTRa (index 49)

 

Type token ratio for all words.

 

 

LDMTLDa (index 50)

 

MTLD lexcical diversity measure for all words.

 

 

LDVOCDa (index 51)

 

VOC lexical diversity measure for all words.

 

 

6. Connectives

 

Connectives play an important role in the creation of cohesive links between ideas and clauses and provide clues about text organization (Cain & Nash, 2011; Crismore, Markkanen, & Steffensen, 1993; Longo, 1994; Sanders & Noordman, 2000; van de Kopple, 1985). Coh-Metrix provides an incidence score (occurrence per 1000 words) for all connectives (CNCAll) as well as different types of connectives. Indices are provided on five general classes of connectives (Halliday & Hasan, 1976; Louwerse, 2001): causal (CNCCaus; because, so), logical (CNCLogic; and, or), adversative/contrastive (CNCADC; although, whereas), temporal (CNCTemp, CNCTempx; first, until), and additive (CNCAdd; and, moreover). In addition, there is a distinction between positive connectives (CNCPos; also, moreover) and negative connectives (CNCNeg; however, but).

 

All connectives: CNCAll (index 52)

 

This is the incidence of all connectives.

 

Causal Connectives: CNCCaus (index 53)

This is the incidence score of causal connectives. Among the various types of connectives, only causal connectives (CNCCaus) discriminated between the high and low cohesion texts, presumably because the researchers who created the texts primarily manipulated causal cohesion and not additive, temporal, or clarification connectives.

CNCLogic (index 54)

This is the incidence score of logic connectives.

 

CNCADC (index 55)

This is the incidence score of adversative/contrastive connectives.

 

CNCTemp (index 56)

This is the incidence score of temporal connectives.

 

CNCTempx (index 57)

This is the incidence score of extended temporal connectives.

 

CNCAdd (index 58)

This is the incidence score of additive connectives.

 

CNCPos (index 59)

This is the incidence score of positive connectives.

 

CNCNeg (index 60)

This is the incidence score of negative connectives.

 

 

7. Situation Model

The expression situation model has been used by researchers in discourse processing and cognitive science to refer to the level of mental representation for a text that involves much more than the explicit words (van Dijk & Kintsch, 1983; Graesser & McNamara, 2011; Graesser, Singer, & Trabasso, 1994; Kintsch, 1998; Zwaan & Radvansky, 1998). Some researchers have described the situational model in terms of the features that are present in the comprehender’s mental representation when a given context is activated (e.g., Singer & Leon, 2007). For example, with episodes in narrative text, the situation model would include the plot. In an informational text about the circulatory system, the situation model might convey the flow of the blood.

 

SMCAUSv (index 61)

This is the incidence score of causal verbs.

 

Causal content: SMCAUSvp (index 62)  

This is the incidence of causal verbs and causal particles in text.

Intentional content: SMINTEp (index 63)  

This is the incidence of intentional actions, events, and particles (per thousand words).

Causal cohesion: SMCAUSr (index 64)  

This is a ratio of causal particles (P) to causal verbs (V). The denominator is incremented by the value of 1 to handle the rare case when there are 0 causal verbs in the text. Cohesion suffers when the text has many causal verbs (signifying events and actions) but few causal particles that signal how the events and actions are connected.

To see the list of causal particles click here

 

Intentional cohesion: SMINTEr (index 65)

 

This is the ratio of intentional particles to intentional actions/events. 

 

SMCAUSlsa (index 66)

This is the LSA overlap between verbs.

 

SMCAUSwn (index 67)

This is the WordNet overlap between verbs.

 

Temporal cohesion: SMTEMP (index 68)

This is the repetition score for tense and aspect.  The repetition score for tense is averaged with the repetition score for aspect. 

8. Syntactic Complexity

 

Theories of syntax assign words to part-of-speech categories (e.g., nouns, verbs, adjectives, conjunctions), group words into phrases or constituents (noun-phrases, verb-phrases, prepositional-phrases, clauses), and construct syntactic tree structures for sentences. For example, some sentences are short and have a simple syntax that follow an actor-action-object syntactic pattern, have few if any embedded clauses, and have an active rather than passive voice. Some sentences have complex, embedded syntax that potentially places heavier demands on working memory. The syntax in text tends to be easier to process when there are shorter sentences, few words before the main verb of the main clause, and few words per noun-phase.

 

 

Words before main verb: SYNLE (index 69)

 

This is the mean number of words before the main verb of the main clause in sentences. This is a good index of working memory load. 

 

 

Modifiers per NP: SYNNP (index 70)

 

 

  This is the mean number of modifiers per noun-phrase.  

 

SYNMEDpos (index 71)

 

This the minimum editorial distance score for part of speech tags.

 

 

SYNMEDwrd (index 72)

 

This is the minimum editorial distance score for words.

 

 

SYNMEDlem (index 73)

 

This is the minimun editorial distance score for lemmas.

 

 

Syntactic structure similarity adjacent: SYNSTRUTa (index 74)

 

This is the proportion of intersection tree nodes between all adjacent sentences.        

 

 

Syntactic structure similarity all 01: SYNSTRUTt (index 75)

 

This is the proportion of intersection tree nodes between all sentences and across paragraphs.      

 

 

 

9. Syntactic Pattern Density

 

Syntactic complexity is also informed by the density of particular syntactic patterns, word types, and phrase types. Coh-Metrix provides information on the incidence of noun phrases (DRNP, verb phrases (DRVP), adverbial phrases (DRAP), and prepositions (DRPP). The relative density of each of these can be expected to affect processing difficulty of text, particularly with respect to other features in a text. For example, if a text has a higher noun and verb phrase incidence, it is more likely to be informationally dense with complex syntax.

 

DRNP (index 76)

This is the incidence score of noun phrases.


DRVP (index 77)

This is the incidence score of verb phrases.


DRAP (index 78)

This is the incidence score of adverbial phrases.


DRPP (index 79)

This is the incidence score of preposition phrases.


DRPVAL (index 80)

This is the incidence score of agentless passive voice forms.


Negations: DRNEG (index 81)  

This is the incidence score for negation expressions.


DRGERUND (index 82)

This is the incidence score of gerunds.


DRINF (index 83)


This is the incidence score of infinitives.

 

 

 

10. Word Information

 

Word information refers to the idea that each word is assigned a syntactic part-of-speech category thus, syntactic categories are segregated into content words (e.g., nouns, verbs, adjectives, adverbs) and function words (e.g., prepositions, determiners, pronouns). Many words can be assigned to multiple syntactic categories. For example, the word “bank”can be a noun (“river bank”), a verb (“don’t bank on it”), or an adjective (‘bank shot”). Coh-Metrix assigns only one part-of-speech category to each word on the basis of its syntactic context. In addition, Coh-Metrix computes word frequency scores and psychological ratings.

 

 

WRDNOUN (index 84)

 

This is the incidence score of nouns.

 


WRDVERB (index 85)

 

This is the incidence score of verbs.

 


WRDADJ (index 86)

 

This is the incidence score of adjectives.

 


WRDADV (index 87)

 

This is the incidence score of adverbs.

 


Personal pronoun: WRDPRO (index 88)

 

This is the number of personal pronouns per 1000 words. A high density of pronouns can create referential cohesion problems if the reader does not know what the pronouns refer to.

 

Example:

 

Paul told John that he wanted to help him out.

 

The words he and him in this sentence are both pronouns, leading to a density score of 200. The pronouns, however, are ambiguous as we do not know which pronoun refers to which person.


WRDPRP1s (index 89)

 

This is the incidence score of pronouns, first person, single form.

 


WRDPRP1p (index 90)

 

This is the incidence score of pronouns, first peron, plural form.

 


WRDPRP2 (index 91)

 

This is the incidence score of pronouns, second person.

 


WRDPRP3s (index 92)

 

This is the incidence score of pronouns, third person, single form.

 


WRDPRP3p (index 93)

 

This is the incidence score of pronouns, third person, plural form.

 


WRDFRQc (index 94)

 

This is the average word frequency for content words.

 


WRDFRQa (index 95)

 

This is the average word frequency for all words.


WRDFRQmc (index 96)

This is the average minimum word frequency in sentences.

 

 

Age of acquisition (WRDAOAc). (index 97)

 

Coh-Metrix includes the age of acquisition norms from MRC which were compiled by Gilhooly and Logie (1980) for 1903 unique words. The c at the end of the index name indicates that it is calculated for the average ratings for content words in a text. Age of acquisition reflects the notion that some words appear in children’s language earlier than others. Words such as cortex, dogma, and matrix (AOA= 700) have higher age-of-acquisition scores than words such as milk, smile, and pony (AOA =202). Words with higher age-of-acquisition scores denote spoken words that are learned later by children.

 

Familiarity (WRDFAMc). (index 98)

This is a rating of how familiar a word seems to an adult. Sentences with more familiar words are words that are processed more quickly. MRC provides ratings for 3488 unique words. Coh-Metrix provides the average ratings for content words in a text. Raters for familiarity provided ratings using a 7-point scale, with 1 being assigned to words that they never had seen and 7 to words that they had seen very often (nearly every day). The ratings were multiplied by 100 and rounded to integers.

For example, the words milk (588), smile (594), and pony (524) have an average Familiarity of 569 compared to the words cornet (364), dogma (328), andmanus (113), which have an average Familiarity of 268. Words with very high Familiarity include mother (632) and water (641), compared to calix (124) and witan (110).

 

Concreteness (WRDCNCc). (index 99)

This is an index of how concrete or non-abstract a word is. Words that are more concrete are those things you can hear, taste, or touch. MRC provides ratings for 4293 unique words. Coh-Metrix provides the average ratings for content words in a text. Words that score low on the concreteness scale include protocol (264) and difference (270) compared to box (597) and ball (615).

 

Imagability (WRDIMGc). (index 100)

An index of how easy it is to construct a mental image of the word is also provided in the merged ratings of the MRC, which provides ratings for 4825 words. Coh-Metrix provides the average ratings for content words in a text. Examples of low imagery words are reason (285), dogma (327), and overtone (268) compared to words with high imagery such as bracelet (606) and hammer (618).

 

Meaningfulness (WRDMEAc). (index 101)

These are the meaningfulness ratings from a corpus developed in Colorado by Toglia and Battig (1978). MRC provides ratings for 2627 words. Coh-Metrix provides the average ratings for content words in a text. An example of meaningful word is people (612) as compared to abbess (218). Words with higher meaningfulness scores are highly associated with other words (e.g., people), whereas a low meaningfulness score indicates that the word is weakly associated with other words.

 

Polysemy (WRDPOLc). (index 102)

Polysemy refers to the number of senses (core meanings) of a word. For example, the word bank has at least two senses, one referring to a building or institution for depositing money and the other referring to the side of a river. Coh-Metrix provides average polysemy for content words in a text. Polysemy relations in WordNet are based on synsets (i.e., groups of related lexical items), which are used to represent similar concepts but distinguish between synonyms and word senses (Miller et al., 1990). These synsets allow for the differentiation of senses and provide a basis for examining the number of senses associated with a word. Coh-Metrix reports the mean WordNet polysemy values for all content words in a text. Word polysemy is considered to be indicative of text ambiguity because the more senses a word contains relates to the potential for a greater number of lexical interpretations. However, more frequent words also tend to have more meanings, and so higher values of polysemy in a text may be reflective of the presence of higher frequency words.

 

Hypernymy (WRDHYPn, WRDHYPv, WRDHYPnv). (index 103, 104, 105)

Coh-Metrix also uses WordNet to report word hypernymy (i.e., word specificity). In WordNet, each word is located on a hierarchical scale allowing for the measurement of the number of subordinate words below and superordinate words above the target word. Thus, entity, as a possible hypernym for the noun chair, would be assigned the nu1mber 1. All other possible hyponyms of entity as it relates to the concept of a chair (e.g., object, furniture, seat, chair, camp chair, folding chair) would receive higher values (see also Chapter 2). Similar values are assigned for verbs (e.g., hightail, run, travel). As a result, a lower value reflects an overall use of less specific words, while a higher value reflects an overall use of more specific words. Coh-Metrix provides estimates of hypernymy for nouns (WRDHYPn), verbs (WRDHYPv), and a combination of both nouns and verbs (WRDHYPnv).

 

11. Readability

 

The traditional method of assessing texts on difficulty consists of various readability formulas. More than 40 readability formulas have been developed over the years (Klare, 1974-1975). The most common formulas are the Flesch Reading Ease Score and the Flesch Kincaid Grade Level.

 

 

Flesch Reading Ease: RDFRE (index 106)

The output of the Flesch Reading Ease formula is a number from 0 to 100, with a higher score indicating easier reading. The average document has a Flesch Reading Ease score between 6 and 70. The formula is provided below:  

READFRE = 206.835 - (1.015 x ASL) - (84.6 x ASW)

 

where:

 

ASL = average sentence length = the number of words divided by the number of sentences. This is the same as READASL.


ASW (comes from CELEX database) = average number of syllables per word = the number of syllables divided by the number of words. This is the same as READASW.

 

 

 

Flesch_Kincaid Grade Level: RDFKGL (index 107)

 

 

This more common Flesch-Kincaid Grade Level formula converts the Reading Ease Score to a U.S. grade-school level. The higher the number, the harder it is to read the text. The grade levels range from 0 to 12.

 

READFKGL = (.39 x ASL) + (11.8 x ASW) - 15.59

 

In general, a text should generally have more than 200 words before the Flesch Reading Ease and Flesch-Kincaid Grade Level scores can successfully be applied.


RDL2 (index 108)

 

T his is the second language readability score.

 

 

IV References

Allen, J. (1995). Natural language understanding. Redwood City, CA: Benjamin/Cummings.
Allen, J. F. (2009). Word Senses, Semantic Roles and Entailment. In 5th International Conference on Generative Approaches to the Lexicon, Pisa, Italy. September 17-19 2009.
Adams, M. J. (1990). Beginning to read: Thinking and learning about print. Cambridge, MA: MIT Press.
Baayen, R. H., Piepenbrock, R., & Gulikers, L. (1995). The CELEX lexical database (CD-ROM). Philadelphia, PA: Linguistic Data Consortium, University of Pennsylvania.
Beck, I. L., McKeown, M. G., Omanson, R. C., & Pople, M. T. (1984). Improving the comprehensibility of stories: The effects of revisions that improve coherence. Reading Research Quarterly, 19, 263-277.
Beck, I. L., McKeown, M. G., Sinatra, G. M., & Loxterman, J. A. (1991). Revising social studies text from a text-processing perspective: Evidence of improved comprehensibility. Reading Research Quarterly, 27, 251-276.
Beck, I. L., McKeown, M. G., Hamilton, R. L., & Kucan, L. (1997). Questioning the Author: An approach for enhancing student engagement with text. Delaware: International Reading Association.
Bell, C., McCarthy, P. M., & McNamara, D. S. (2012). Using LIWC and Coh-Metrix to investigate gender differences in linguistic styles. In P. M. McCarthy & C. Boonthum-Denecke (Eds.), Applied natural language processing and content analysis: Identification, investigation, and resolution (pp. 545-556). Hershey, PA: IGI Global.
Best, R. M., Floyd, R. G., & McNamara, D. S. (2008). Differential competencies contributing to children's comprehension of narrative and expository texts. Reading Psychology, 29, 137-164.
Best, R., Ozuru, Y., & McNamara, D. S. (2004). Self-explaining science texts: Strategies, knowledge, and reading skill. In Y. B. Kafai, W. A. Sandoval, N. Enyedy, A. S. Nixon, & F. Herrera (Eds.), Proceedings of the Sixth International Conference of the Learning Sciences: Embracing Diversity in the Learning Sciences (pp. 89-96). Mahwah, NJ: Erlbaum.
Biber, D. (1988). Variation across Speech and Writing. Cambridge University Press: Cambridge.
Biber, D. (1993). Register Variation and Corpus Design, Computational Linguistics. Cambridge University Press: Cambridge.
Biber, D., Conrad, S., & Reppen, R. (1998). Corpus linguistics: Investigating language structure and use. Cambridge University Press: Cambridge.
Boonthum-Denecke, C., McCarthy, P. M., & Lamkin, T. (Eds.). (2012). Cross-disciplinary advances in applied natural language processing: Issues and approaches. Hershey, PA: IGI Global.
Bormuth, J. R. (1971). Development of standards of readability: Toward a rational criterion of passage performance. Final report, U.S. Office of Education, Project No. 9-0237. Chicago: University of Chicago.
Brill, E. (1995). Transformation-based error-driven learning and natural language processing: A case study in part-of-speech tagging. Computational Linguistics, 21, 543-566.
Britton, B. K., & Gulgoz, S. (1991). Using Kintsch’s computational model to improve instructional text: Effects of repairing inference calls on recall and cognitive structures. Journal of Educational Psychology, 83, 329-345.
Brooks, C., & Warren, R. P. (1972). Modern rhetoric. New York: Harcourt Brace Jovanovich.
Brun, C., Ehrmann M., & Jacquet G., (2007). A hybrid system for named entity metonymy recognition. In Proceedings of the 4th International Workshop on Semantic Evaluations (ACL-SemEval), 23-24. Prague, Czech Republic.
Bruner, J. (1986). Actual minds, possible worlds. Cambridge, MA: Harvard University Press.
Bruss, M., Albers, M., & McNamara, D. S. (2004). Changes in scientific articles over two hundred years: A Coh-Metrix analysis. Proceedings of the ACM 22nd International Conference on Computer Documentation (pp. 104–109). New York: ACM Press.
Cai, Z., McNamara, D.S., Louwerse, M., Hu, X., Rowe, M., & Graesser, A.C. (2004). NLS: Non-latent similarity algorithm. In K. Forbus, D. Gentner & T. Regier (Eds.), Proceedings of the 26th Annual Cognitive Science Society (pp. 180-185). Mahwah, NJ: Erlbaum
Cain, K. &Nash, H. M. 2011. The influence of connectives on young readers’ processing and comprehension of text. Journal of Educational Psychology 103, (2): 429-441.
Carrell, P. (1982). Cohesion is not coherence. TESOL Quarterly, 16, 479-88.
Cataldo, M. G., & Oakhill, J. (2000). The effect of text organization (original vs. scrambled) on readers’ ability to search for information. Journal of Educational Psychology, 92, 791-799.
Charniak, E. (2000). A maximum-entropy-inspired parser. Proceedings of the First Conference on North American Chapter of the Association for Computational Linguistics (pp. 132-139). San Francisco: Morgan Kaufmann Publishers.
Chomsky (1965). Aspects of the theory of syntax. Cambridge: The MIT Press.
Clark, H. H. (1996). Using language. Cambridge: Cambridge University Press.
Clark, H. H., & Clark, E. V. (1977). Psychology and language. New York: Harcourt Brace Jovanovich.
Clark, H. H., & Schaefer, E. F. (1989). Contributing to discourse. Cognitive Science , 13, 259-294.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
Coltheart, M. (1981). The MRC Psycholinguistic Database. Quarterly Journal of Experimental Psychology, 33A, 497-505.
Connor, C. M., Morrison, F. J., Fishman, B. J., Schatschneider, C., & Underwood, P. (2007). THE EARLY YEARS: Algorithm-Guided Individualized Reading Instruction. Science, 315(5811), 464-465.
Conrad, F. G., & Schober, M. F. (2007) (Eds.), Envisioning the survey interview of the future. New York: Wiley.
Crossley, S. A., Allen, D., & McNamara, D. S. (2011). Text readability and intuitive simplification: A comparison of readability formulas. Reading in a Foreign Language, 23, 84-102.
Crossley, S. A., Allen, D., & McNamara, D. S. (2012). Text simplification and comprehensive input: A case for intuitive approach. Language Teaching and Research, 16, 89-108.
Crossley, S. A., Dufty, D. F., McCarthy, P. M., & McNamara, D. S. (2007). Toward a new readability: A mixed model approach. In D.S. McNamara and G. Trafton (Eds.), Proceedings of the 29th annual  conference of the Cognitive Science Society (pp.197-202). Austin, TX: Cognitive Science Society.
Crossley, S. A., Greenfield, J., & McNamara, D. S. (2008). Assessing text readability using psycholinguistic indices. TESOL Quarterly, 42, 475-493.
Crossley, S. A., Louwerse, M., McCarthy, P. M., & McNamara, D. S. (2007). A linguistic analysis of simplified and authentic texts. Modern Language Journal, 91, 15-30.
Crossley, S. A. & McNamara, D. S. (2008). Assessing second language reading texts at the intermediate level: An approximate replication of Crossley, Louwerse, McCarthy, and McNamara (2007). Language Teaching, 41, 229-409.
Crossley, S. A. & McNamara, D. S. (2009). Computational assessment of lexical differences in L1 and L2 writing. Journal of Second Language Writing, 18, 119-135.
Crossley, S. A., & McNamara, D. S. (2010). Cohesion, coherence, and expert evaluations of writing proficiency. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society (pp. 984-989). Austin, TX: Cognitive Science Society.
Crossley, S.A., & McNamara, D.S. (2011). Text coherence and judgments of essay quality: Models of quality and coherence. In L. Carlson, C. Hoelscher, & T.F. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society. (pp. 1236-1231). Austin, TX: Cognitive Science Society. 
Crossley, S. A., & McNamara, D. S. (2011). Understanding expert ratings of essay quality: Coh-Metrix analyses of first and second language writing. International Journal of Continuing Engineering Education and Life, 21, 170-191.
Crossley, S.A., & McNamara, D.S. (2012). Detecting the first language of second language writers using automated indices of cohesion, lexical sophistication, syntactic complexity and conceptual knowledge. In S. Jarvis & S. A. Crossley (Eds.), Approaching language transfer through text classification: Explorations in the detection-based approach (p. 106-126). Bristol, UK: Multilingual Matters.
Crossley, S.A., & McNamara, D.S. (2012b). Interlanguage Talk: A computational analysis of non-native speakers' lexical production and exposure. In P.M. McCarthy & C. Boonthum-Denecke (Eds.), Applied natural language processing and content analysis: Identification, investigation, and resolution (pp. 425-437). Hershey, PA: IGI Global.
Crossley, S. A., McCarthy, P. M. & McNamara, D. S. (2007). Discriminating between second language learning text-types. In D. Wilson & G. Sutcliffe (Eds.), Proceedings of the 20th International Florida Artificial Intelligence Research Society Conference (pp. 205-210). Menlo Park, California: The AAAI Press.
Crossley, S. A., Roscoe, R., Graesser, A., & McNamara, D. S. (2011). Predicting human scores of essay quality using computational indices of linguistic and textual features. In G. Biswas, S. Bull, J. Kay, & A. Mitrovic (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education. (pp. 438-440). Auckland, New Zealand: AIED.
Crossley, S. A., Salsbury, T. McCarthy, P. M., & McNamara, D. S. (2008), LSA as a measure of coherence in second language natural discourse. In V. Sloutsky, B. Love, & K. McRae (Eds.), Proceedings of the 30th annual conference of the Cognitive Science Society (pp.1906-1911). Washington, D.C.: Cognitive Science Society.
Crossley, S. A., Salsbury, T., & McNamara, D. S. (2009). Measuring L2 lexical growth using hypernymic relationships. Language Learning, 59, 307-334.
Crossley, S. A., Salsbury, T., & McNamara, D. S. (2010a). The development of polysemy and frequency use in English second language speakers. Language Learning, 60, 573-605. 
Crossley, S.A., Salsbury, T., & McNamara, D.S. (2010b). The development of semantic relations in second language speakers. A case for Latent Semantic Analysis. Vigo International Journal of Applied Linguistics, 7, 55-74.
Crossley, S. A., Salsbury, T., & McNamara, D. S. (2010c). The role of lexical cohesive devices in triggering negotiations for meaning. Issues in Applied Linguistics, 18, 55-80.
Crossley, S. A., Weston, J., McLain Sullivan, S. T., & McNamara, D. S. (2011). The development of writing proficiency as a function of grade level: A linguistic analysis. Written Communication, 28, 282-311.
Crismore, A., Markkanen, R., & Steffensen, M. S. (1993). Metadiscourse in persuasive writing: a study of texts written by American and finish university students. Written Communication, 39, 39-71.
DARPA (1995). Proceedings of the Sixth Message Understanding Conference (MUC-6). San Francisco: Morgan Kaufman Publishers.
Day, R. S. Comprehension of prescription drug information: Overview of a research program. Proceedings of the American Association for Artificial Intelligence. Argumentation for Consumer Healthcare, 2006. (www.aaai.org)
Dell, G., McKoon, G., & Ratcliff, R. (1983). The activation of antecedent information during the processing of anaphorix reference in reading. Journal of Verbal Learning and Verbal Behavior, 22, 121-132.
Dempsey, K. B., McCarthy, P. M., & McNamara, D. S. (2007). Using phrasal verbs as an index to distinguish text genres. In D. Wilson and G. Sutcliffe (Eds.), Proceedings of the twentieth International Florida Artificial Intelligence Research Society Conference (pp. 217-222). Menlo Park, California: The AAAI Press.
Dufty, D. F., Graesser, A. C., Louwerse, M., & McNamara, D. S. (2006). Assigning grade level to textbooks: Is it just readability? In R. Sun & N. Miyake (Eds.), Proceedings of the 28th Annual Conference of the Cognitive Science Society (pp. 1251-1256). Austin, TX: Cognitive Science Society.
Dufty, D. F., McNamara, D., Louwerse, M., Cai, Z., & Graesser, A. C. (2004). Automatic evaluation of aspects of document quality. In S. Tilley & S. Huang (Eds.), Proceedings of the 22nd Annual International Conference on Design of Communication: the Engineering of Quality Documentation (pp. 14-16). New York: ACM Press.
Duncan, B., & Hall, C. (2009). A coh-metrix analysis of variation among biomedical abstracts. In Florida Artificial Intelligence Research Society Conference (pp. 237- 242) Menlo Park, CA: The AAAI Press.
Duran, N., Bellissens, C., Taylor, R., & McNamara, D. S. (2007). Qualifying text difficulty with automated indices of cohesion and semantics. In D.S. McNamara and G. Trafton (Eds.), Proceedings of the 29th Annual Meeting of the Cognitive Science Society (pp. 233-238). Austin, TX: Cognitive Science Society.
Duran, N. D., Hall, C., McCarthy, P. M., & McNamara, D. S. (2010). The linguistic correlates of conversational deception: Comparing natural language processing technologies. Applied Psycholinguistics
Duran, N. D., McCarthy, P. M., Graesser, A. C., & McNamara, D. S. (2007). Using temporal cohesion to predict temporal coherence in narrative and expository texts. Behavior Research Methods, 39, 212–223.
Duran, N. D., McCarthy, P. M., Graesser, A. C., & McNamara, D. S., (2006). Using Coh-Metrix temporal indices to predict psychological measures of time. In R. Sun & N. Miyake (Eds), Proceedings of the 28th Annual Conference of the Cognitive Science Society (pp. 190-195). Cognitive Science Society.
Duran, N. D., & McNamara, D. S. (2006, July). It’s about time: Discriminating differences in temporality between genres. Poster presented at the 16th Annual Meeting of the Society for Text and Discourse, Minneapolis, MN.
Fellbaum, C. (Ed.). (1998). WordNet: An Electronic Lexical Database [CD-ROM]. Cambridge, MA: MIT Press.
Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32, 221-233.
Freedman, A., & Ian, P. (1980). Writing in the college years: some indices of growth. College Composition and Communication, 31, 311-324.
Garnham, A., Oakhill, J. & Johnson-Laird, P. N. (1982). Referential continuity and the coherence of discourse. Cognition, 11, 29-46.
Gernsbacher, M. A. (1990). Language comprehension as structure building. Hillsdale, NJ: Erlbaum.
Gernsbacher, M. A., & Givón, T. (1995). (Eds.) Coherence in spontaneous text. Amsterdam: Benjamins.
Gildea, D. (2001). Corpus variation and parser performance. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing,(pp. 167–202). Pittsburgh, PA.
Gilhooly, K. L. & Logie, R. H. (1980) Age of Acquisition, imagery, concreteness, familiarity and ambiguity measures for 1944 words. Behavioral Research Methods and Instrumentation 12, 395-427.
Givón, T. 1995.Functionalism and grammar. Philadelphia: John Benjamins.
Graesser, A. C. (1981). Prose comprehension beyond the word. New York: Springer-Verlag.
Graesser, A. C., Cai, Z., Louwerse, M., & Daniel, F. (2006). Question Understanding Aid (QUAID): A web facility that helps survey methodologists improve the comprehensibility of questions. Public Opinion Quarterly, 70, 3–22.
Graesser, A. C., Chipman, P., Haynes, B. C., & Olney, A. (2005). AutoTutor: An intelligent tutoring system with mixed-initiative dialogue. IEEE Transactions in Education, 48, 612–618.
Graesser, A. C., Dowell, N., & Moldovan, C. (2011). A computer’s understanding of literature. Scientific Studies of Literature, 1, 24-33.
Graesser, A. C., Gernsbacher, M. A., & Goldman, S. (Eds.) (2003). Handbook of discourse processes.
Graesser, A. C., & Hemphill, D. (1991). Question answering in the context of scientific mechanisms. Journal of Memory and Language, 30, 186–209.
Graesser, A. C., Hoffman, N. L., & Clark, L. F. (1980). Structural components of reading time. Journal of Verbal Learning and Verbal Behavior, 19, 135-151.
Graesser, A. C., Jeon, M., Cai, Z., & McNamara, D. S. (2008). Automatic analyses of language, discourse, and situation models. In J. Auracher, & W. van Peer (Eds.), New beginnings in literary studies (pp. 72-88). Cambridge, U.K.: Cambridge Scholars Publishing.
Graesser, A. C., Jeon, M., & Dufty, D. (2008). Agent technologies designed to facilitate interactive knowledge construction. Discourse Processes, 45, 298–322.
Graesser, A. C., Jeon, M., Yang, Y., & Cai, Z. (2007). Discourse cohesion in text and tutorial dialogue. Information Design Journal, 15, 199-213.
Graesser, A. C., & McNamara, D. S. (2011). Computational analyses of multilevel discourse comprehension. Topics in Cognitive Science, 3, 371-398.
Graesser, A. C., McNamara, D. S., & Kulikowich, J. (2011). Coh-Metrix: Providing multilevel analyses of text characteristics. Educational Researcher, 40, 223-234.
Graesser, A. C., McNamara, D. S., & Louwerse, M. M (2003). What do readers need to learn in order to process coherence relations in narrative and expository text. In A.P. Sweet and C.E. Snow (Eds.), Rethinking reading comprehension. New York: Guilford Publications.
Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, and Computers, 36, 193–202.
Graesser, A. C., Millis, K. K., & Zwaan, R. A. (1997). Discourse comprehension. Annual Review of Psychology, 48, 163-189.
Graesser, A. C., & Ottati, V. (1996). Why stories? Some evidence, questions, and challenges. In R. S. Wyer (Ed.), Knowledge and memory: The real story (pp. 121–132). Hillsdale, NJ: Erlbaum.
Graesser, A. C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehension. Psychological Review, 101, 371–395.
Greenfield, G. (1999). Classic readability formulas in an EFL context: Are they valid for
Japanese speakers? Unpublished doctoral dissertation, Temple University, Philadelphia,
PA, United States. (University Microfilms No. 99–38670).
Haberlandt, K., & Graesser, A. C. (1985). Component processes in text comprehension and some of their interactions. Journal of Experimental Psychology: General, 114, 357–374.
Halliday, M. A. K. & Hasan, R. (1976). Cohesion in English. London: Longman.
Hall, C., McCarthy, P. M., Lewis, G. A., Lee, D. S., & McNamara, D. S. (2007). A Coh-Metrix assessment of American and English/Welsh Legal English. Coyote Papers: Psycholinguistic and Computational Perspectives. University of Arizona Working Papers in Linguistics, 15, 40-54.
Haviland, S. E., & Clark, H. H. (1974). What’s new? Acquiring new information as a process in comprehension. Journal of Verbal Learning and Verbal Behavior, 13, 515-521.
Healy, S. L., Weintraub, J. D., McCarthy, P. M., Hall, C., & McNamara, D. S. (2009). Assessment of LDAT as a grammatical diversity assessment tool. In C. H. Lane & H. W. Guesgen (Eds.), Proceedings of the 22nd International Florida Artificial Intelligence Research Society (FLAIRS) International Conference (pp. 249-253). Menlo Park, CA: The AAAI Press.
Hempelmann, C. F., Dufty, D., McCarthy, P., Graesser, A. C., Cai, Z., & McNamara, D. S. (2005). Using LSA to automatically identify givenness and newness of noun-phrases in written discourse. In B. Bara (Ed.), Proceedings of the 27th Annual Meeting of the Cognitive Science Society (pp. 941–946). Mahwah, NJ: Erlbaum.
Herskovits, A. (1998). Schematization. In P. Olivier & K. P. Gapp (Eds.), Representation and processing of spatial expressions (pp. 149-162). Mahwah, NJ: Lawrence Erlbaum Associates.
Hu, X., Cai, Z., Louwerse, M. M., A. Olney, P. Penumatsa, A. C. Graesser, & TRG. (2003). A revised algorithm for Latent Semantic Analysis. Proceedings of the 2003 International Joint Conference on Artificial Intelligence (pp. 1489-1491).
Jurafsky, D., & Martin, J. (2008). Speech and language processing. Englewood, NJ: Prentice Hall.
Just, M. A., & Carpenter, P. A. (1971). Comprehension of negation with quantification. Journal of Verbal Learning and Verbal Behavior, 12, 21-31.
Just, M. A., & Carpenter, P. A. (1987). The psychology of reading and language comprehension. Boston: Allyn & Bacon.
Just, M. A., & Carpenter, P. A. (1992). A capacity theory of comprehension: Individual differences in working memory. Psychological Review, 99, 122-149.
Kallet H. (2004) How to write the methods section of a research paper. Respiratory Care Services, 49, 1229–1232.
Kalyuga, S. (2012). Cognitive load aspects of text processing. In C. Boonthum-Denecke, P. McCarthy, & T. Lamkin (Eds.), Cross-Disciplinary Advances in Applied Natural Language Processing: Issues and Approaches (pp. 114-132). Hershey, PA: Information Science Reference
Kamil, M. L., Pearson, D., Moje, E. B., & Afflerbach, P. (Eds.), Handbook of Reading Research, Volume IV. New York: Routledge
Keil, F.C. (1981) Constraints on knowledge and cognitive development. Psych of. Rev., 88, 197-227.
Keenan, J. M., Betjemann, R. S., & Olson, R. K. (2008). Reading comprehension tests vary in the skills they assess: Differential dependence on decoding and oral comprehension. Scientific Studies of Reading, 12, 281–300.
Kieras, D. E. (1978). Good and bad structure in simple paragraphs: Effects on apparent theme, reading time, and recall. Journal of Verbal Learning and Verbal Behavior, 17, 13-28.
Kincaid, J., Fishburne, R., Rogers, R., & Chissom, B. (1975). Derivation of new readability formulas for navy enlisted personnel. Branch Report 8-75. Millington, TN: Chief of Naval Training.
King, M., & Rentel, V. (1979). Toward a theory of early writing development. Research in the Teaching of English, 13, 243-255.
Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge, UK: Cambridge University Press.
Kintsch, W. & Van Djik, T. A. (1978). Toward a model of text comprehension and production. Psychological Review, 85, 363-394.
Kireyev, K., & Landauer, T. (2011). Word maturity: Computational modeling of word knowledge. Proceedings of the Association of Computational Linguistics (pp. 299-308).
Kinnear, P. R., & Gray, C. D. (2008). SPSS 15 made simple. New York: Psychology Press
Klare, G. R. (1974–1975). Assessing readability. Reading Research Quarterly, 10, 62-102.
Klein, W. 1994. Time in Language. London: Routledge.
Koslin, B. I., Zeno, S., & Koslin, S. (1987). The DRP: An effective measure in reading. New York: College Entrance Examination Board.
Lamkin, T. A., & McCarthy, P. M. (2012).The Hierarchy of Detective Fiction. In C. Murray & P. M. McCarthy (Eds.), Proceedings of the 24rd International Florida Artificial Intelligence Research Society Conference (pp. 257-262). Menlo Park, CA: The AAAI Press.
Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104, 211-240.
Landauer, T. K., Laham, D., & Foltz, P. W. (2003). Automatic essay assessment. Assessment in Education: Principles, Policy & Practice, 10, 295-308.
Landauer, T., McNamara, D. S., Dennis, S., & Kintsch, W. (2007) (Eds.). Handbook of Latent Semantic Analysis. Mahwah, NJ: Erlbaum.
Lappin, S., & Leass, H. J. (1994). An algorithm for pronominal coreference resolution. Computational Linguistics, 20, 535-561.
Leahey, T. H., & Harris, R. J. (1997). Learning and cognition (4th ed.). Saddle River, NJ: Prentice Hall.
Leech, N. L., Barrett, K. C., & Morgan, G. A. (2008). SPSS for intermediate statistics: Use and
interpretation. Mahwah, NJ: Lawrence Erlbaum Associates, Inc
Lehnert, W. G. & Ringle, M. H. (Eds.) (1982). Strategies for natural language processing. Hillsdale, NJ: Lawrence Erlbaum.
Lenat, D. B. (1995). CYC: A large-scale investment in knowledge infrastructure. Communication of the ACM, 38, 32-38. 
Lightman, E. J., McCarthy, P. M., Dufty, D. F., & McNamara, D. S. (2007a). The structural organization of high school educational texts. In D. Wilson & G. Sutcliffe (Eds.), Proceedings of the twentieth International Florida Artificial Intelligence Research Society Conference (pp. 235-240). Menlo Park, California: The AAAI Press.
Lightman, E. J., McCarthy, P. M., Dufty, D. F., & McNamara, D. S. (2007b). Using computation text analysis tools to compare the lyrics of suicidal and non-suicidal song-writers. In D. S. McNamara & G. Trafton (Eds.), Proceedings of the 29th Annual Conference of the Cognitive Science Society (pp. 1217–1222). Austin, TX: Cognitive Science Society.
Long, D. L., Oppy, B. J., & Seely, M. R. (1994). Individual differences in the time course of inferential processing, Journal of Experimental Psychology, 20, 1245–1470.
Long, M. & Ross, S. (1993). Modifications that preserve language and content. In M. L. Tickoo (ed.), Simplification: Theory and application. Singapore: SEAMEO Regional Language Center, 29–52.
Longo, B. (1994). The role of metadiscourse in persuasion. Technical Communication, 41, 348–352.
Lorch, R. F., Jr., Lorch, E. P., & Mogan, A. M. (1987). Task effects and individual differences in on-line processing of the topic structure of a text. Discourse Processes, 24, 350-362.
Louwerse, M. M. (2001). An analytic and cognitive parameterization of coherence relations. Cognitive Linguistics, 12, 291-315. Mahwah, NJ: Erlbaum.
Louwerse, M. M., McCarthy, P. M., McNamara, D. S., & Graesser, A. C. (2004). Variation in language and cohesion across written and spoken registers. In K. D. Forbus, D. Gentner, & T. Regier (Eds.), Proceedings of the 26th Annual Meeting of the Cognitive Science Society. Mahwah, NJ: Erlbaum.
Loxterman, J. A., Beck, I. L., & McKeown, M. G. (1994). The effects of thinking aloud during reading on students’ comprehension of more or less coherent text. Reading Research Quarterly, 29, 353-367.
Mann W. C., & Thompson, S. (1988). Rhetorical Structure Theory: Toward a functional theory of text organization. Text, 8, 243-281.Marcu, D. (2000). The theory and practice of discourse parsing and summarization. Cambridge, MA: MIT Press.
Marcus, M., Santorini, B., & Marcinkiewicz, M. (1993). Building a large annoted corpus of English: The Penn Treebank. Computational Linguistics, 19, 313-330.
McCarthy, P.M., & Boonthum, C. (2012) (Eds.), Applied natural language processing and content analysis: Identification, investigation, and resolution. Hershey, PA: IGI Global.
McCarthy, P.M., Briner, S.W., Rus, V., & McNamara, D.S. (2007). Textual signatures: Identifying text-types using latent semantic analysis to measure the cohesion of text structures. In A. Kao, & S. Poteet (Eds.), Natural language processing and text mining (pp. 107-122) . London: Springer-Verlag UK.
McCarthy, P. M., Dufty, D., Hempelman, C., Cai, Z., Graesser, A. C., & McNamara, D. S. (2012). Newness and Givenness of Information : Automated Identification in Written Discourse. In P. M. McCarthy & C. Boonthum (Eds.), Applied natural language processing and content analysis: Identification, investigation, and resolution (pp. 457-478). Hershey, PA: IGI Global.
McCarthy, P. M., Hall, C., Duran N. D., Doiuchi, M., Duncan, B., Fujiwara, Y., & McNamara, D.S., (2009). A computational analysis of journal abstracts written by Japanese, American, and British scientists. The ESPecialist, 30, 141-173.
McCarthy, P. M., & Jarvis, S. (2007). A theoretical and empirical evaluation of vocd. Language Testing. 24, 459-488.
McCarthy, P. M. & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42, 381-392.
McCarthy, P. M., & Jarvis, S. (in press). From intrinsic to extrinsic issues of lexical diversity assessment: An ecological validation study. In S. Jarvis & M. Daller (Eds.), Vocabulary knowledge: Human ratings and automated measures. Amsterdam: Benjamins.
McCarthy, P.M., Lehenbauer, B.M., Hall, C., Duran, N.D., Fujiwara, Y., & McNamara, D.S. (2007). A Coh-Metrix analysis of discourse variation in the texts of Japanese, American, and British Scientists. Foreign Languages for Specific Purposes, 6, 46-77.
McCarthy, P. M., Lewis, G. A., Dufty, D. F., & McNamara, D. S. (2006). Analyzing writing styles with Coh-Metrix. Proceedings of the Florida Artificial Intelligence Research Society International Conference (pp. 764-769), Menlo Park, CA: AAAI Press.
McCarthy, P.M., & McNamara, D.S. (2007). Are seven words all we need? Recognizing genre at the sub-sentential level. In D.S. McNamara and G. Trafton (Eds.), Proceedings of the 29th annual conference of the Cognitive Science Society (pp. 1295-1300). Austin, TX: Cognitive Science Society.
McCarthy, P. M., Myers, J. C., Briner, S. W., Graesser, A. C., & McNamara, D. S. (2009). Are three words all we need? A psychological and computational study of sub-sentential genre recognition. Journal for Language Technology and Computational Linguistics, 24, 23-55.
McCarthy, P. M., Renner, A. M., Duncan, M. G., Duran, N. D., Lightman, E. J., & McNamara, D. S. (2008). Identifying topic sentencehood. Behavior Research and Methods, 40, 647-664.
McCarthy, P. M., Rus, V., Crossley, S. A., Bigham, S. C., Graesser, A. C., & McNamara, D. S. (2007). Assessing entailer with a corpus of natural language. In D. Wilson & G. Sutcliffe (Eds.), Proceedings of the twentieth International Florida Artificial Intelligence Research Society Conference (pp. 247-252). Menlo Park, California: The AAAI Press
McCarthy, P.M., Rus, V., Crossley, S., Graesser, A.C., & McNamara, D.S. (2008). Assessing forward-, reverse-, and average-entailment indices on natural language input from the intelligent tutoring system, iSTART. In D. Wilson & G. Sutcliffe (Eds.), Proceedings of the 21st International Florida Artificial Intelligence Research Society Conference (pp. 165-170).
McCarthy, P. M., Watanabi S, & Lamkin, T. A. (2012). The Gramulator: A Tool to Identify Differential Linguistic Features of Correlative Text Types. In P.M. McCarthy & C. Boonthum-Denecke (Eds.), Applied natural language processing: Identification, investigation, and resolution. Hershey, PA: IGI Global.
McCutchen, D. (1986). Domain knowledge and linguistic knowledge in the development of writing ability. Journal of Memory and Language, 25, 431-444.
McCutchen, D. & Perfetti, C. A. (1982). Coherence and connectedness in the development of discourse production. Text, 2, 113-139.
McNamara, D. S. (1997). Comprehension skill: A knowledge-based account. In M.G. Shafto & P. Langley (Eds.), Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society (pp. 508-513). Hillsdale, NJ: Erlbaum.McNamara, D. S. (2001). Reading both high-coherence and low-coherence texts: Effects of text sequence and prior knowledge. Canadian Journal of Experimental Psychology, 55, 51-62.
McNamara, D. S. (2001). Reading both high and low coherence texts: Effects of text sequence and prior knowledge. Canadian Journal of Experimental Psychology, 55, 51–62.
McNamara, D. S. (2004). SERT: Self-explanation reading training. Discourse Processes, 38, 1–30.
McNamara, D. S. (2011). Computational methods to extract meaning from text and advance theories of human cognition. Topics in Cognitive Science, 2, 1-15.
McNamara, D. S., Cai, Z., & Louwerse, M. M. (2007). Optimizing LSA measures of cohesion. In T. Landauer, D.S. McNamara, S. Dennis, & W. Kintsch (Eds.), Handbook of Latent Semantic Analysis (pp. 379-400). Mahwah, NJ: Erlbaum.
McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2010). Linguistic features of writing quality. Written Communication, 27, 57-86.
McNamara, D. S., Crossley, S. A., & Roscoe, R. D. (in press). Natural language processing in an intelligent writing strategy tutoring system. Behavior Research Methods.
McNamara, D. S. & Dempsey, K. (2011). Reader expectations of question formats and difficulty: Targeting the zone. In M. McCrudden, J. Magliano, & G. Schraw (Eds.), Text relevance and learning from text. Charlotte, NC: Information Age Publishing, Inc.
McNamara, D. S., & Graesser, A. C. (2012). Coh-Metrix: An automated tool for theoretical and applied natural language processing. In P. M. McCarthy & C. Boonthum (Eds.), Applied natural language processing and content analysis: Identification, investigation, and resolution (pp. 188-205). Hershey, PA: IGI Global.
McNamara, D. S., Graesser, A. C., & Louwerse, M. M. (2012). Sources of text difficulty: Across genres and grades. In J. P. Sabatini, E. Albro, & T. O’Reilly (Eds.), Measuring up: Advances in how we assess reading ability (pp. 89-116). Plymouth, UK: Rowman & Littlefield Education.
McNamara, D. S., & Kintsch, W. (1996). Learning from text: Effects of prior knowledge and text coherence. Discourse Processes, 22, 247-287.
McNamara, D. S., Kintsch, E., Songer, N.B., & Kintsch, W. (1996). Are good texts always better? Text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 14, 1-43.
McNamara, D. S., Louwerse, M. M., McCarthy, P. M., & Graesser, A. C. (2010). Coh-Metrix: Capturing linguistic features of cohesion. Discourse Processes, 47, 292-330.
McNamara, D. S., & Magliano, J. P. (2009). Towards a comprehensive model of comprehension. In B. Ross (Ed.). The psychology of learning and motivation (vol.51, pp. 297–384). New York, NY, US: Elsevier Science.
McNamara, D. S., & McDaniel, M. (2004). Suppressing irrelevant information: Knowledge activation or inhibition? Journal of Experimental Psychology: Learning, Memory, & Cognition, 30, 465–482.
McNamara, D. S., Ozuru, Y., & Floyd, R. G. (2011). Comprehension challenges in the fourth grade: The roles of text cohesion, text genre, and readers' prior knowledge. International Electronic Journal of Elementary Education, 4, 229-257.
McNamara, D. S., Ozuru, Y., Graesser, A. C., & Louwerse, M. (2006). Validating Coh-Metrix. In R. Sun & N. Miyake (Eds.), Proceedings of the 28th Annual Conference of the Cognitive Science Society (pp. 573-578). Austin, TX: Cognitive Science Society.
McNamara, D. S., Raine, R., Roscoe, R., Crossley, S., Jackson, G. T., Dai, J., Cai, Z., Renner, A., Brandon, R., Weston, J., Dempsey, K., Lam, D., Sullivan, S., Kim, L., Rus, V., Floyd, R., McCarthy, P. M., & Graesser, A. C. (2012). The Writing-Pal: Natural language algorithms to support intelligent tutoring on writing strategies. In P. M. McCarthy & C. Boonthum (Eds.), Applied natural language processing and content analysis: Identification, investigation, and resolution (pp. 298-311). Hershey, PA: IGI Global.
Meichenbaum, D., & Biemiller, A. (1998). Nurturing independent learners: Helping students take charge of their learning. Cambridge, MA: Brookline Books.
Meyer, B. J. F. (1975). The organization of prose and its effect on memory. New York: Elsevier.
Meyer, B. J. F., & Wijekumar, K. (2007). Web-based tutoring of the structure strategy: Theoretical background, design, and findings. In D.S. McNamara (Ed.), Reading comprehension strategies: Theories, interventions, and technologies (347-375). Mahwah, NJ: Lawrence Erlbaum Associates.
Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., & Miller, K. J. (1990). Introduction to WordNet: An on-line lexical database. Journal of Lexicography, 3, 235-244.
Millis, K., Graesser, A. C., & Haberlandt, K. (1993). The impact of connectives on memory for expository texts. Applied Cognitive Psychology, 7, 317–340.
Millis, K., Magliano, J., Wiemer-Hastings, K., Todaro, S., & McNamara, D.S. (2007). Assessing and improving comprehension with Latent Semantic Analysis. In T. Landauer, D.S. McNamara, S. Dennis, & W. Kintsch (Eds.), Handbook of Latent Semantic Analysis (pp. 207-225). Mahwah, NJ: Erlbaum.
Min, H. C. & McCarthy, P. M. (2010). Identifying varietals in the discourse of American and Korean scientists: A contrastive corpus analysis using the gramulator. In H. W. Guesgen & C. Murray (Eds.), Proceedings of the 23rd International Florida Artificial Intelligence Research Society Conference (pp. 247-252). Menlo Park, CA: The AAAI Press.
Nelson, J., Perfetti, C., Liben, D., & Liben, M. (2012). Measures of text difficulty: Testing their predictive value for grade levels and student performance. New York, NY: Student Achievement Partners.
Oakhill, J. V. (1984). Inferential and memory skills in children’s comprehension of stories, British Journal of Educational Psychology, 54, 31–39.
Oakhill, J., & Cain, K. (2007). Issues of causality in children’s reading comprehension. In K. Cain & J. Oakhill (Eds.), Cognitive bases of children’s language comprehension difficulties. New York: Guilford.
Oakhill, J. V., & Yuill, N. M. (1996). Higher order factors in comprehensive disability: Processes and remediation. In C. Cornoldi & J.V. Oakhill (Eds.), Reading comprehension difficulties: Processes and remediation. (pp. 69-93).Mahwah, NJ: Lawrence Erlbaum Associates Inc.
Oakhill, J., Yuill, N. & Donaldson, M. L. (1990). Understanding of causal expressions in skilled and less skilled text comprehenders. British Journal of Developmental Psychology, 8, 401-410.
O’Brien, E. J., Rizzella, M. L., Albrecht, J. E., & Halleran, J. G. (1998). Updating a situation model: A memory-based text processing view. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, 1200-1210.
O’Reilly, T., Best, R., & McNamara, D. S. (2004). Self-explanation reading training: Effects for low-knowledge readers. In K. Forbus, D. Gentner, & T. Regier (Eds.), Proceedings of the 26th Annual Conference of the Cognitive Science Society (pp. 1053–1058). Mahwah, NJ: Erlbaum.
O’Reilly, T. & McNamara, D. S. (2007). Reversing the reverse cohesion effect: good texts can be better for strategic, high-knowledge readers. Discourse Processes, 43, 121-152.
Ozuru, Y., Best, R., Bell, C., Witherspoon, A., & McNamara, D. S. (2007). Influence of question format and text availability on assessment of expository text comprehension. Cognition & Instruction, 25, 399-438.
Ozuru, Y., Briner, S., Best, R., & McNamara, D. S. (2010). Contributions of self-explanation to comprehension of high and low cohesion texts. Discourse Processes, 47, 641-667.
Ozuru, Y., Dempsey, K., & McNamara, D. S. (2009). Prior knowledge, reading skill, and text cohesion in the comprehension of science texts. Learning and Instruction, 19, 228-242.
Ozuru, Y., Rowe, M., O'Reilly, T., & McNamara, D. S. (2008). Where's the difficulty in standardized reading tests: The passage or the question? Behavior Research Methods. 40, 1001-1015.
Page, E. B. & Petersen, N. S. (1995). The computer moves into essay grading: Updating the ancient test. Phi Delta Kappan, 76, 561-565.
Paivio, A., Yuille, J. C., & Madigan, S. A. (1968). Concreteness, imagery and meaningfulness values for 925 words. Journal of Experimental Psychology Monograph Supplement, 76 (3, Part 2).
Palmer, M., Kingsbury, P., & Gildea, D. (2005). The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31, 71–106.
Pennebaker, W. B. (2011). The secret life of pronouns: What our words say about us. London: Bloomsbury Press.
Pennebaker, J. W., Booth, R. J., & Francis, M. E. (2007). LIWC2007: Linguistic inquiry and word count. Austin, Texas.
Pennebaker, J. W., Francis, M. E., & Booth, R. J. (2001). Linguistic Inquiry and Word Count (LIWC) (Version LIWC2001) [Computer software]. Mahwah, NJ: Erlbaum.
Pentimonti, J. M., Zucker, T. A., Justice, L. M., & Kaderavek, J. N. (2010). Informational text use in preschool classroom read-alouds. The Reading Teacher, 63, 656–665.
Perfetti, C. A. (2007). Reading ability: Lexical quality to comprehension. Scientific Studies of Reading, 11, 357-383.
Pickering, M., & Garrod, S. (2004). Toward a mechanistic psychology of dialogue. Behavioral and Brain Sciences, 27, 169-226.
Popken, R. (1991). A study of topic sentence use in technical writing. The Technical Writing Teacher, 18, 49-58.
Prince, E. F. (1981). Toward a taxonomy of given-new information. In P. Cole (Ed.), Radical pragmatics (pp. 223-255). New York: Academic Press.
Rapp, D. N., van den Broek, P., McMaster, K. L., Kendeou, P., & Espin, C. A. (2007). Higher-order comprehension processes in struggling readers: A perspective for research and intervention.  Scientific Studies of Reading, 11, 289-312.
Rayner, K. (1998) Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372-422.
Rayner, K., Foorman, B., Perfetti, C., Pesetsky, D., & Seidenberg, M. (2001). How psychological science informs the teaching of reading. Psychological Science in the Public Interest, 2(2), 31-74.
Renner, A., McCarthy, P. M., Boonthum-Denecke, C., & McNamara, D. S. (2012). Maximizing ANLP evaluation: Harmonizing flawed input. In P. M. McCarthy & C. Boonthum-Denecke (Eds.), Applied natural language processing and content analysis: Identification, investigation, and resolution. (pp. 438-456). Hershey, PA: IGI Global.
Roscoe, R. D., Crossley, S. A., Weston, J. L., & McNamara, D. S. (2011). Automated assessment of paragraph quality: Introductions, body, and conclusion paragraphs. In R. C. Murray & P. M. McCarthy (Eds.), Proceedings of the 24th International Florida Artificial Intelligence Research Society (FLAIRS) Conference (pp. 281-286). Menlo Park, CA: AAAI Press.
Rosch, E. & Mervis, C. B. (1975). Family resemblances: Studies in the internal structure of categories. Cognitive Psychology, 7, 573-605.
Rowe, M., & McNamara, D.S. (2008). Inhibition needs no negativity: Negativity links in the construction-integration model. In V. Sloutsky, B. Love, & K. McRae (Eds.), Proceedings of the 30th annual conference of the Cognitive Science Society (pp. 1777-1782). Washington, DC: Cognitive Science Society
Rubin, D. C. (1995). Introduction. In D. C. Rubin (Ed.), Remembering our past: Studies in autobiographical memory (pp. 1-15). New York: Cambridge University Press.
Rufenacht, R. M., McCarthy, P. M., Lamkin, T. M., (2011). Fairy tales and ESL texts: An analysis of linguistic features using the gramulator. In R. C. Murray & P. M. McCarthy (Eds.), Proceedings of the 24th International Florida Artificial Intelligence Research Society (FLAIRS) Conference (pp. 287-292). Menlo Park, CA: AAAI Press.
Rus, V. (2004). A first exercise for evaluating logic form identification systems, Proceedings Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text (SENSEVAL-3), at the Association of Computational Linguistics Annual Meeting, July 2004. Barcelona, Spain: ACL.
Rus, V., McCarthy, P. M., McNamara, D. S., & Graesser, A. C. (2008). A study of textual entailment. International Journal on Artificial Intelligence Tools, 17, 659–685.
Sanacore, J. & Palumbo, A. (2009). Understanding the fourth-grade slump: Our point of view. Educational Forum, 73, 67-74.
Sanders, T. J. M., (1997). Semantic and Pragmatic Sources of Coherence: On the categorization of coherence relations in context. Discourse Processes, 24,119-147.
Sanders, T .J. M., Spooren, W. P. M. & Noordman, L. G. M. (1992).Toward a Taxonomy of Coherence Relations. Discourse Processes, 15, 1-35.
Sanders, T. J. M., & Noordman, L. G. M. (2000). The role of coherence relations and their linguistic markers in text processing. Discourse Processes, 29, 37–60.
Sanford, A. J., & Emmott, C. (2012). Mind, brain and narrative. Cambridge, UK: Cambridge University Press.
Schank, R. C., & Abelson, R. P. (1977). Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Hillsdale, NJ: Erlbaum.
Sekine, S., & Grishman, R. (1995). A corpus-based probabilistic grammar with only two nonterminals. Fourth International Workshop on Parsing Technologies (pp. 260-270). Prague/Karlovy Vary, Czech Republic.
Shanahan, T., Kamil, M. L., & Tobin, A. W. (1982). Cloze as a measure of intersentential comprehension. Reading Research Quarterly, 17, 229-255.
Singer, M., & Leon, J. (2007). Psychological studies of higher language processes: Behavioral and empirical approaches. In F. Schmalhofer & C. Perfetti (Eds.), Higher level language processes in the brain: Inference and comprehension processes. Mahwah, NJ.
Singer M., & Ritchot, K. F. M. (1996). The role of working memory capacity and knowledge access in text inference processing. Memory & Cognition;24, 733–743.
Smith, E. E., & Medin, D. L. (1981). Categories and concepts. Cambridge, MA: Harvard University Press.
Snow, C. (2002). Reading for understanding: Toward an R&D program in reading comprehension. Santa Monica, CA: RAND Corporation.
Spivey, M., Joanisse, M., & McRae, K. (Eds.), The Cambridge handbook of psycholinguistics. Cambridge, U.K.: Cambridge University Press.
Stanovich, K. E. (1986). Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly, 21 (4), 360-406.
Stenner, A. J. (2006). Measuring reading comprehension with the Lexile framework. Durham, NC: Metametrics, Inc. presented at the California Comparability Symposium, October 1996. Retrieved January 30, 2006 from http://www.lexile.com/DesktopDefault.aspx?view=re.
Swales, J. (1981). Aspects of article introductions. Birmingham, UK: The University of Aston, Language Studies Unit.
SwaIes, J. (1990). Genre analysis. Cambridge, UK: Cambridge University Press.
Sweet, A. P., & Snow, C. E. (Eds.). (2003). Rethinking reading comprehension. New York: Guilford.
Tannen, D. (1982). Oral and literate strategies in spoken and written narratives. Language, 58, 1-21.
Templin, M. (1957). Certain language skills in children: Their development and interrelationships. Minneapolis, MN: The University of Minnesota Press.
ter Meulen, A. G. B. (1995). Representing Time in Natural Language. The Dynamic Interpretation of Tense and Aspect. Cambridge, Massachusetts, The MIT Press.
Toglia, M. P., & Battig, W. R. (1978). Handbook of semantic word norms. New York: Lawrence Erlbaum.
Tonjes, M. J., Ray, W., & Zintz, M. V. (1999). Integrated Content Literacy. New York: The McGraw-Hill Publishers.
Trabasso, T., & van den Broek, P. (1985). Causal thinking and the representation of narrative events. Journal of Memory and Language, 24, 612-630.
Tweissi, A. I. (1998). The effects of the amount and the type of simplification on foreign language reading comprehension. Reading in a Foreign Language, 11, 191-206.
U.S. Air Force Reserve Officers' Training Corps. (1985). U.S. air power: Key to deterrence. Maxwell Air Force Base, Montgomery, AL: U.S. Air Force.
Van den Broek, P., Rapp, D. N., & Kendeou, P. (2005). Integrating memory-based and constructionist approaches in accounts of reading comprehension. Discourse Processes, 39, 299–316.
Van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic Press.
Vande Kopple, W. J. (1985). Some exploratory discourse on metadiscourse. College Composition and Communication, 36, 82-93.
VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., & Rose, C. P. (2007). When are tutorial dialogues more effective than reading? Cognitive Science, 31, 3–62. 
Weston, J., Crossley, S. A., & McNamara, D. S. (2010). Towards a computational assessment of freewriting quality. In H. W. Guesgen & C. Murray (Eds.), Proceedings of the 23rd International Florida Artificial Intelligence Research Society (FLAIRS) Conference (pp. 283-288). Menlo Park, CA: The AAAI Press.
Williams, P. J. (2007). Literacy in the curriculum: Integrating text structure and content area instruction. In D. S. McNamara (Ed.), Reading comprehension strategies: theories, interventions, and technologies (pp. 199-219). Mahwah, NJ: Erlbaum.
Winograd, T. (1983). Language as a cognitive process: Syntax. Reading, MA: Addison-Wesley.
Wittgenstein, L (1953). Philosophical investigations. Blackwell.
Whitney, P., Ritchie, B. G., & Clark, M. B. (1991). Working-memory capacity and the use of elaborative inferences in text comprehension. Discourse Processes, 14,133–145.
Yano, Y, Long, M., & Ross, S. (1994). Effects of simplified and elaborated texts on foreign language reading comprehension. Language Learning 44 (2), 189-219.
Yuill, N. & Oakhill, J. (1988). Understanding of anaphoric relations in skilled and less skilled comprehenders. British Journal of Psychology, 79, 173-186.
Zipf, G. K. (1949). Human behavior and the principle of least effort. Reading, MA: Addison- Wesley.
Zwaan, R. A. (1994). Effect of genre expectations on text comprehension. Journal of Experimental Psychology: Learning, Memory, Cognition, 20, 920-933.
Zwaan, R. A., Magliano, J. P., & Graesser, A. C. (1995). Dimensions of situation model construction in narrative comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 386–397.
Zwaan, R. A., & Radvansky, G. A. (1998). Situation models in language comprehension and memory. Psychological Bulletin, 123, 162-185.

V. Example Text and Output

 

Example Text            

Coh-Metrix Output for Example Text