Methods · how every figure is computed

Methods

This page is the reference for the analysis pages and the exported workbooks. Any number on the site can be reproduced from it. The pipeline code is in the project's pipeline/ folder and runs in the order given at the end.

1. Texts

1. Every text is a Project Gutenberg plain-text file, identified by ebook number on the work's page. The header, footer and licence block are removed.
2. Footnote markers ([1], ^1, {1}), page markers ({P. 12}), trailing line numbers, printer's asterisk rows and editorial apparatus lines are stripped. Inline glosses printed beside the line (Purves's Chaucer) are moved into stanza notes.
3. Sections and stanzas follow the source's blank lines and headings. Heading normalisation is described in the README (ordinal words to roman numerals, repeated subtitles dropped, roman-numbered single stanzas folded into their poem). No word of the verse is altered.
4. Rhyme schemes are stated as facts only where Saintsbury (1910) or Gummere (1885) give them and the text agrees; elsewhere the scheme shown is detected (section 6).

2. Tokens

A word token is a run of letters with internal apostrophes and hyphens: [A-Za-z'’][A-Za-z'’-]*. Hyphenated compounds are split at the hyphen for syllable counting. Tokens are lower-cased and curly apostrophes normalised for lookup. The end word of a line is its last token after trailing punctuation and quotation marks are removed.

3. Pronunciation

1. The CMU Pronouncing Dictionary, release 0.7b as packaged by NLTK (3.9.2), gives ARPAbet phones with stress digits (0 none, 1 primary, 2 secondary). It records American English.
2. Lookup tries the token itself, then spelling variants in order: 'd as -ed, 'st as -est, letters only, -our as -or, -ise as -ize, -yse as -yze, -re as -er, apostrophe removed. The derivation export records which applied (cmu or variant).
3. A poetic 'd on a base found in the dictionary drops the syllabic -ed only after d or t.
4. Words not found take the heuristic route: vowel groups are counted, a final silent e is dropped (except after l, ee, ye, oe, ie), and a final -ed or -es is non-syllabic unless the stem ends in t, d or a sibilant. Stress for such words is left free.
5. A short list of names and foreign words has fixed syllable counts: 'em (1), 'tis (1), 'twas (1), 'twere (1), adeline (3), alfonso (3), amundeville (4), antonia (3), aurora (3), baba (2), can't (1), don't (1), dudu (2), e'en (1), e'er (1), ev'ry (2), gulbeyaz (3), haidee (2), he'll (1), heav'n (1), i'd (1), i'll (1), i'm (1), i've (1), inez (2), ismail (2), it's (1), jose (2), juan (2), julia (2), lambro (2), lenore (2), let's (1), ne'er (1), o'er (1), pallas (2), raby (2), shan't (1), she'll (1), suwarrow (3), th' (0), that's (1), there's (1), they'll (1), they're (1), we'll (1), we're (1), what's (1), won't (1), you'll (1), you're (1), you've (1).

Coverage. The share of tokens with a dictionary pronunciation is reported for every work as dictionary coverage. It is the first thing to check before trusting a scansion figure; below about 90 percent the stress statistics are weak.

4. Syllables and elision

1. The syllable count of a word is the number of vowel phones in its dictionary entry. Boundaries for display come from pyphen's US English hyphenation patterns, adjusted so the number of pieces equals the count. If hyphenation gives too few pieces, the longest piece is split at a vowel boundary.
2. Where the work has a catalogued metre with a fixed count (iambic pentameter 10, trochaic octameter 16), elision applies when the line runs more than one syllable long. Words in the table below are shortened to the count given, the largest saving first, until the line is within one syllable of the target. The stressed syllable absorbs the following unstressed one (ev-er-y to ev'ry).
3. When the line runs short, a spelled -ed on a word of more than four letters is expanded to a syllable (learn-ed). This is done one word at a time until the count is met.
4. Works without a fixed count (blank verse is counted, free verse and ballads are not forced) receive no elision or expansion.

Elision table

Word and the syllable count it elides to. The list is the pipeline's own; additions are welcome and should cite a scansion handbook.

5. Stress and metre

1. Each syllable is classed one of four ways. S: dictionary primary or secondary stress, polysyllables only. U: dictionary unstressed, polysyllables only. u: a function-word monosyllable, treated as unstressed but not scored. x: any other monosyllable, free. The function-word list is: a ah am an and are as at be been but by did do for from had has have he her him his i if in is it its me my no nor not o of off oh on or our out she so than that the thee their them then these they thine this those thou thy to up us was we were when where while with ye yet you your.
2. Only S and U positions are scored. Monosyllables carry sentence stress that a dictionary cannot know, so they are never counted for or against a reading.
3. Meter fit is the share of S/U positions that agree with an alternating template. The better of the two offsets is taken, since a line may begin on a strong or a weak position. The template rises (weak-strong, iambic) unless the work is catalogued as trochaic, in which case it falls. This is a regularity measure, not a scansion: it says how far the dictionary stresses alone conform, and it will under-read a poet who works against the template deliberately.
4. Where a poem's metre comes from. Four sources, in order (pipeline/metres.py): a published statement about that poem, which for now means the Wikipedia article on the poem where it names the metre in so many words (pipeline/poemmetres.py asks Wikidata for every poem by each poet that has an article, reads it, and keeps the sentence: "Generally, the meter is trochaic octameter"), under the same CC BY-SA attribution as the biographies; a named, dated authority matched poem by poem (Alden 1903); the documented metre of a work famous enough to have one; and otherwise measurement, which is most poems, because no published source names the metre of the ordinary poems in a collected works. Tap the metre on any poem to see which of these it is, whether the scanner agrees and how far, and how to scan that metre. A published metre is shown even where the scanner disagrees, with the scanner's own reading beside it.
5. The reader's stress marks and the scansion game show the scanner's reading of the line in its poem's metre (pipeline/readings.py): the foot and length the poem settled on, with the substitutions the scanner allows, an inverted foot, a spondee, a pyrrhic, a swapped length. A dark mark is a syllable the word itself fixes; a pale mark is one the word left open and the reading filled; a dotted mark is a word's own stress that the reading bends through. Where two readings tie by the scanner's own count and put the beat on different syllables (about three lines in a hundred), the line panel shows the second reading under the first and says how they differ, rather than choosing for the reader; the scansion game never asks about such a line. The rules the scanner reads by, each tested against hand-marked verse before it was kept, are recorded in pipeline/marked/RULES.md.
6. A feminine ending is a line whose last syllable is U or u with more than one syllable in the line.

How far two readers agree

Scanning a line means marking which syllables carry the beat. People who do this for a living do it by hand, and they do not always agree with each other. That matters here because a computer cannot be more right than the people it is checked against: their disagreement is the ceiling. So we measured it. The collection of hand-marked poems the scanner is checked against (Haider 2021, which joins three collections: For Better For Verse, EPG64 and the prosodic corpus) happens to hold 60 lines that were marked twice, in two of the collections. Set side by side, the two markings give the same mark to 95 syllables in every 100, and agree on the whole line 45 times in 60, three lines in four. "Tell that its sculptor well those passions read" is TELL that in one and tell THAT in the other. Sixty lines is a small sample, so "three in four" is rough, and the two copies come from different collections, which makes different hands very likely but does not prove it.

A second, separate sign of the same thing comes from the old handbooks of prosody, whose authors printed their own scansions a century ago (pipeline/marked/, eight of them from Kirkham in 1829 to Schipper in 1910). Ordinary words of two or more syllables have a stress the dictionary fixes, POem not poEM, which no metre can move. On those syllables Schipper agrees with the modern dictionary 95 times in 100 and Saintsbury 86. Same words, different ears: Saintsbury marked what he heard and argued that the length of a syllable mattered as much as its weight. The two kinds of judge also part company on one figure in particular, a line that goes slack over two small words and then carries two stresses together ("of the WIDE WORLD"). The Victorian prosodists mark the small words slack; the modern annotators, like the scanner, put the beat on the small word the metre promotes. A rule that took the prosodists' side made the modern corpus worse at every strength and left the handbooks evenly split, so the scanner reads such lines the modern way.

What this sets is a ceiling. Checked against one team's marks, the scanner gets 93 syllables in 100 right and three lines in four with every mark right; the two teams stand at 95 and three in four with each other. A scanner that agreed with the annotators as well as they agree with each other would score about where this one does, and a figure much above that would mean it had learned one team's habits, not the verse. So where the scanner cannot tell two readings apart, the page shows both rather than choosing, and where the words leave a syllable open it says how many marks are the metre's rather than the word's.

When the marks are withheld

A stress template fitted to free verse, or to a hexameter, produces marks that look like a scansion and are not one. So every passage is classed first, from its own lines, before any mark is shown:

• Measured metre: the work's metre is one the template covers, that is, iambic and trochaic lines of a fixed length. The fixed word stresses land on the template's beats at least 85% of the time, and one line length accounts for at least 38% of lines. A looser pass accepts 80% and 30%. Only here do stress marks show a beat, read as above.
• A metre the template does not measure: dactylic hexameter, alliterative four-stress verse. Stress marks show the dictionary accent of each word of two syllables or more, and nothing on monosyllables; the note names the metre.
• No fixed metre: free verse, mixed forms, and anything that fails the thresholds. Marks, if the reader asks for them, are word accents only, and the page says so.

Rhyme letters and colours appear only where at least 30% of a passage's lines have a rhyme partner. Below that, a note gives the share and no scheme is drawn, because in unrhymed verse a few per cent of lines chime by chance. The same classification governs the analysis page's picture and scheme chart, and the games: no scansion question is ever drawn from a passage without a measured metre. The thresholds are in verseKind in common.js.

Long verse paragraphs, of more than thirty lines, are scanned for rhyme with a four-line window. A line pairs only with a neighbour within four lines, so echoes far apart in a paragraph do not count as a scheme.

Quantity: the Greek and Latin originals

The original texts are scanned by quantity, not stress, by the method a schoolroom uses. It is set out as steps because every step can be checked.

1. Elide a final vowel or -m before a word beginning with a vowel or h-.
2. Split what is left into syllables, one for each vowel or diphthong. In Latin, i and u are consonants at the start of a word before a vowel and between vowels (iactatus, Lauinia); qu and gu are single consonants.
3. Mark a syllable long by position when two consonants follow its vowel. x and z in Latin, and ζ ξ ψ in Greek, count as two. A stop followed by l or r is left open, because the poets treat it either way.
4. Greek marks much of its own quantity: η and ω are long, ε and ο short, the diphthongs long; α, ι and υ are left open. Latin marks none of it, so Latin quantities come from steps 3 and 5 alone.
5. In Greek, a long vowel or diphthong at the end of a word is left open before a word beginning with a vowel, because epic shortens it (correption).
6. Fit the line to the metre. For the hexameter there are thirty-two arrangements of dactyl and spondee across the first five feet; only those giving the right number of syllables are considered, and of those only the ones that put a long where step 3 requires one. Where more than one survives, prefer a dactyl in the fifth foot and fewer spondees early.
7. If nothing fits, try one synizesis in Greek (two vowels of a word read as one) or one open syllable dropped in Latin, and fit again.

Lines that still do not fit are left unscanned rather than forced. The rate is reported per work: the Aeneid 99.1 per cent, the Metamorphoses 99.8, De Rerum Natura 99.3, Juvenal 99.8, the Iliad 74.5, the Odyssey 76.6, Hesiod 79.9. Greek falls short of Latin mainly because of the digamma, proper names and epic licence, which this method does not model. The code is pipeline/classical.py.

6. Rhyme

1. The rhyme key of a word is its phone string from the last primary-stressed vowel to the end, stress digits removed (weary: IH R IY). If no primary stress exists the last stressed vowel of any degree is used.
2. Words outside the dictionary take the key of dictionary words that share their longest ending of three or more letters (suffix analogy); the export marks these in_dictionary = no.
3. The dictionary spells one English sound two ways before r: near, beer and dear are keyed IH R and here, hear and ear are keyed IY R, 210 words against 28. Nothing in English turns on the difference and the two are read as one, which is what lets Kipling’s pint o’ beer rhyme with no red-coats here.
4. Within a stanza, lines are lettered by identity of key in order of first appearance. A line whose key matches no other is given a second chance on its final syllable: the phones from its last vowel to the end, with any unstressed vowel treated as a wildcard. So wantonness finds dress, and temperate finds date. This pass pairs a stressed ending with an unstressed one and never two unstressed ones, because all that survives of those is the consonant after the reduced vowel, which would make every -es and -ies plural in English rhyme with every other. For the same reason the spelling fallback needs two consonants after the last vowel, not one: a lone s is the plural, not a rhyme. The old -y / eye rhyme (civility / tie) is the one vowel-only match allowed, and the wildcard is not admitted to it. A key used once after both passes is shown as -. Verse paragraphs of more than thirty lines are not lettered by this route.
5. The poem completes its own scheme. Everything above reads a stanza alone, and a modern American pronouncing dictionary is a poor witness to how a poet spoke: Emily Brontë rhymed can with wan and west with cast, which the dictionary separates, so two stanzas of a plain ABABCC poem came out gapped — and because letters run in order of first appearance, one gap near the top shifted every letter below it, so the same sound was lettered B in one stanza and A in the next. A stanza is therefore read beside its neighbours. Where a poem plainly keeps a form — at least four stanzas of that length, the form measured whole in at least three of them and in sixty per cent, no rival form holding a quarter, and half the poem’s line endings already lettered — a gapped stanza is completed to it. Four conditions restrain this: the evidence must come from the poem in hand and never from the book around it, the stanza must already have confirmed a rhyme of its own and be no more than half gaps, exactly one of the poem’s forms may fit (a gapped -A-A quatrain fits ABAB and equally fits the ballad’s ABCB, so it is left alone), and two endings the dictionary holds in full may only be joined if they agree in their final consonant — vowels shift across centuries and codas do not, which is what keeps noon from rhyming with mare. A paragraph of couplets is completed the same way when every lettered pair in it is a couplet and none is broken. 2,480 line endings are completed this way across 11 works; each is recorded in the work’s data as inferred rather than measured.
6. Where a handbook scheme is stated for the work and the stanza has the right number of lines, that scheme is recorded. The agreement between it and the detected letters is reported in the build log: 71 to 96 percent across the stated works, the shortfall being slant rhyme, eye rhyme and words outside the dictionary.
7. Rhyme density is the share of line endings whose key recurs within the stanza. Scheme entropy is the Shannon entropy in bits of the distribution of stanza schemes in the work; a single fixed scheme gives 0.
8. Mosaic rhymes, feminine rhymes, triple rhymes, -ation endings, 'd endings, self-rhymes and off-dictionary endings are counted from the keys and spellings as labelled on the analysis page.

7. Drifted rhymes

A pair of end words that the poet rhymes (same letter in the stanza scheme, or adjacent under a stated scheme) whose modern keys differ is listed as drifted. Pairs whose keys differ only by voicing (S/Z, TH/DH, SH/ZH) or by vowel reduction (AH/IH, IY R/IH R) are near-matches and hidden by default. The reference pronunciation is modern American. The site does not yet model the pronunciation of the poet's own time, so a drifted pair may be a historical change, a dialect difference, an eye rhyme, or a slant rhyme. Adding a historical phonology from Ellis's On Early English Pronunciation (1869–89) is planned and would let drift be reported as a change over a stated interval.

8. Alliteration, assonance, caesura

1. A content word is any token not in the function-word list. Its onset is the first phone of its dictionary entry if that phone is a consonant; vowel-initial words are excluded from alliteration. Its nucleus is its primary-stressed vowel phone.
2. A line alliterates if two or more content words share an onset; it assonates if two or more share a nucleus. The rate is the share of lines that do.
3. Baseline. Repetition of sounds happens by chance in any English line. For each section, the content words are shuffled across the section's lines thirty times, keeping each line's word count, and the rate is re-measured. The mean of the shuffles is the expected rate under chance. The ratio observed / expected is the figure to read (1 is chance; The Raven runs 1.35); the z score is (observed − mean) / standard deviation of the shuffles.
4. A caesura is recorded where a line contains strong punctuation (comma, semicolon, colon, exclamation, question mark, dash) before its last character; its position is the syllable count before the mark as a share of the line's syllables, summarised into early, middle and late thirds. Only punctuated pauses are counted; syntactic pauses without punctuation are not.

9. Lexicon

1. Distinctive vocabulary uses the weighted log-odds ratio with an informative Dirichlet prior (Monroe, Colaresi and Quinn, 2008, "Fightin' Words"). The whole library is the background and the prior is scaled to the background counts. Words are ranked by the z-score of the log-odds. Stop words are removed.
2. Rare and archaic words are tokens that appear in Webster's Unabridged Dictionary of 1913 (Project Gutenberg #29765). They must be rare in the library (four uses or fewer across all works), or carry an Obsolete, Archaic or Poetic label with forty uses or fewer.
3. Etymology for the game and the notes is Webster's own etymology field, parsed for the source-language tag.
4. Lexical diversity. The raw type–token ratio is reported but falls with length and should not be compared across works of different size. MTLD (McCarthy and Jarvis, 2010) is the mean length of word runs that keep a type–token ratio above 0.72, averaged over a forward and a backward pass. It is approximately length-independent above a few hundred tokens and is not computed under fifty.
5. The function-word profile lists the twenty most frequent function words per thousand tokens; it is the standard signal in authorship studies (Mosteller and Wallace, 1964; Burrows, 2002).

10. Parts of speech

Tags come from NLTK's averaged perceptron tagger, trained on modern prose. Verse inversion, archaic forms and capitalised nouns lower its accuracy, which on this material is estimated informally at around 90 percent and has not been measured against a hand-tagged sample. Shares are of all tokens; the library average is the mean over works.

11. Themes

Section themes are keyword sets (nature, love, death, and so on) matched against the section's tokens; a theme is assigned when its keywords exceed a fixed rate. This is a finding aid for the library filter, not a measurement, and is not exported.

12. Uncertainty and baselines

• Rates over lines (feminine endings, meter fit, alliteration, assonance, share of lines at the modal length) carry 95 percent percentile bootstrap intervals from 1,000 resamples of lines, random seed 1913. A rate whose interval spans the library average should not be described as different from it.
• Shuffled baselines (section 8) are the chance level for sound repetition; sections with fewer than two lines have none.
• The corpus chart of measures across time uses English originals only, with translations and works composed before 1400 shown in their own table, because a translation's metre belongs to the translator's century.
• Nothing here is a test of significance across works; the intervals are for reading, and comparisons across works of different genre, length and form should be treated as descriptive.

13. Lining up a translation with its original

Where a work is shown beside its original, the two columns are pinned to each other at points where both texts print the same proper name. A translation runs longer or shorter than what it translates — Pope's Iliad I is 781 lines against Homer's 611 — so two columns left to themselves drift apart within a page and are useless by the end of a book.

1. Proper names are taken from both texts: a capitalised word that also appears capitalised away from the start of a line, so that an ordinary word beginning a line is not mistaken for a name.
2. Greek is transliterated letter by letter, with the rough breathing written as h. Both sides are then folded through the spelling changes a classical name goes through on its way into English — ph to f, th to t, k to c, ai and oi to e, doubled letters reduced — and the case ending is dropped. What is left is the stem: Ἀχιλῆος and Achilles both give acile.
3. A stem shared by the two texts, and not common in either, proposes an anchor. Candidates lying far off the diagonal — more than a tenth of the section away from where the ratio of lengths puts them — are dropped.
4. What survives is reduced to the longest run of anchors that keeps both texts moving forward together, then thinned so that no two sit closer than six lines apart. The first and last lines are anchored too, so the columns begin and end level.
5. In the reader, the slack between two anchors is spread evenly over the lines of the shorter side. A line sitting beside a line is therefore close but not exact, and the reader says so; only the lines carrying a tick in the margin are pinned.

17 works, 547 sections, 16,635 anchors: 102 sections lined up stanza by stanza, 228 line by line, 217 by proper name. The anchors are in site/data/works/<work>.align.json and are built by pipeline/align.py. Nothing is inferred from meaning: every anchor is a place where the two texts are printed to the same shape or print the same name, which can be checked by eye.

14. Known limits

• American dictionary pronunciation, no historical phonology: rhyme drift conflates change over time with dialect and slant rhyme.
• Proper names, archaic spellings and foreign words fall to the heuristic route; their stress is unknown and their rhyme keys are by analogy.
• Meter fit measures conformity to one template; it does not scan substitutions, and it says nothing about free verse.
• No gold standard yet: stress, syllable and rhyme accuracy have not been measured against hand-scanned lines. The annotation page draws a stratified sample of lines for hand scansion and exports the annotations so precision and recall can be reported.
• Middle English (Chaucer) and Scots (Burns) are read with a modern dictionary, so their figures are approximate and flagged on the work's page.
• Translations carry the translator's prosody, not the original's.
• A name that does not survive translation gives no anchor: Homer's Achaians are Pope's Greeks, Homer's Hades is Pope's Pluto. A stretch with few names is therefore held level only by the anchors at either end, and a line beside a line there may be out by several lines.

15. Reproducibility

Pipeline order: ingest.py, catalog.py, segfix.py (the segmentation review's findings, applied: sections that are contents pages, prose or a play's dialogue dropped, joined poems split, poems cut at speaker headings rejoined), webster.py, analyze.py, metres.py, readings.py, measures.py, pos.py, corpus.py, commentary.py, plates.py. Python 3.9, NLTK 3.9.2 with cmudict and the averaged perceptron tagger, pyphen. All randomness is seeded (1913). Every figure on the analysis page can be exported with its derivation from the Check the data buttons. The workbook's Derivations sheet lists, for every word of every line, the dictionary source, the phones, the syllable split, the stress classes and whether elision applied. The Method sheet repeats this page in brief.

References: Saintsbury, Historical Manual of English Prosody (1910); Gummere, Handbook of Poetics (1885); Monroe, Colaresi and Quinn, "Fightin' Words," Political Analysis 16 (2008); McCarthy and Jarvis, "MTLD, vocd-D, and HD-D," Behavior Research Methods 42 (2010); Mosteller and Wallace, Inference and Disputed Authorship (1964); Burrows, "Delta," Literary and Linguistic Computing 17 (2002); Ellis, On Early English Pronunciation (1869–89).