Getting Started with NLTK

In [1]:
import nltk
In [1]:
# nltk.download()
In [1]:
from nltk.book import *
*** Introductory Examples for the NLTK Book ***
Loading text1, ..., text9 and sent1, ..., sent9
Type the name of the text or sentence to view it.
Type: 'texts()' or 'sents()' to list the materials.
text1: Moby Dick by Herman Melville 1851
text2: Sense and Sensibility by Jane Austen 1811
text3: The Book of Genesis
text4: Inaugural Address Corpus
text5: Chat Corpus
text6: Monty Python and the Holy Grail
text7: Wall Street Journal
text8: Personals Corpus
text9: The Man Who Was Thursday by G . K . Chesterton 1908
In [2]:
# Test that the data has been installed
from nltk.corpus import brown
brown.words()
Out[2]:
['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', ...]
In [3]:
text1
Out[3]:
<Text: Moby Dick by Herman Melville 1851>
In [4]:
text2
Out[4]:
<Text: Sense and Sensibility by Jane Austen 1811>

Searching Text

In [5]:
# A concordance view shows us every occurrence of a given word, together with some context
text1.concordance("monstrous")
Displaying 11 of 11 matches:
ong the former , one was of a most monstrous size . ... This came towards us , 
ON OF THE PSALMS . " Touching that monstrous bulk of the whale or ork we have r
ll over with a heathenish array of monstrous clubs and spears . Some were thick
d as you gazed , and wondered what monstrous cannibal and savage could ever hav
that has survived the flood ; most monstrous and most mountainous ! That Himmal
they might scout at Moby Dick as a monstrous fable , or still worse and more de
th of Radney .'" CHAPTER 55 Of the Monstrous Pictures of Whales . I shall ere l
ing Scenes . In connexion with the monstrous pictures of whales , I am strongly
ere to enter upon those still more monstrous stories of them which are to be fo
ght have been rummaged out of this monstrous cabinet there is no telling . But 
of Whale - Bones ; for Whales of a monstrous size are oftentimes cast up dead u
In [6]:
# Words that appear in a similar range of contexts
text1.similar("monstrous")
true contemptible christian abundant few part mean careful puzzled
mystifying passing curious loving wise doleful gamesome singular
delightfully perilous fearless
In [7]:
text2.similar("monstrous")
very so exceedingly heartily a as good great extremely remarkably
sweet vast amazingly
In [8]:
# Examine just the contexts that are shared by two or more words
text2.common_contexts(["monstrous", "very"])
a_pretty am_glad a_lucky is_pretty be_glad
In [9]:
import matplotlib.pyplot as plt
%matplotlib inline

# Determine the location of a word in the text
text4.dispersion_plot(["citizens", "democracy", "freedom", "duties", "America"])

Counting Vocabulary

In [10]:
# A 'Token' is the technical name for a sequence of characters
len(text3)
Out[10]:
44764
In [11]:
# Distinct words
# sorted(set(text3))
In [12]:
# Number of distinct words - call these unique items 'Types'
len(set(text3))
Out[12]:
2789
In [13]:
# Lexical richness of the text. i.e number of distinct words is just 6% of the total number of words
len(set(text3)) / len(text3)
Out[13]:
0.06230453042623537
In [14]:
# How often a word occurs in a text
text3.count("smote")
Out[14]:
5
In [15]:
# What percentage of the text is taken up by a specific word
100 * text3.count("smote") / len(text3)
Out[15]:
0.01116968992940756
In [16]:
def lexical_diversity(text):
    return len(set(text)) / len(text)
In [17]:
def percentage(count, total):
    return 100 * count / total
In [18]:
lexical_diversity(text3)
Out[18]:
0.06230453042623537
In [19]:
percentage(text3.count("smote"), len(text3))
Out[19]:
0.01116968992940756

Texts as Lists of Words

Lists

In [20]:
# We will think of a text as nothing more than a sequence of words and punctuation
sent1
Out[20]:
['Call', 'me', 'Ishmael', '.']
In [21]:
sent1.append("Some")
sent1
Out[21]:
['Call', 'me', 'Ishmael', '.', 'Some']
In [22]:
sent4 + sent1
Out[22]:
['Fellow',
 '-',
 'Citizens',
 'of',
 'the',
 'Senate',
 'and',
 'of',
 'the',
 'House',
 'of',
 'Representatives',
 ':',
 'Call',
 'me',
 'Ishmael',
 '.',
 'Some']

Indexing Lists

In [23]:
text4[173]
Out[23]:
'awaken'
In [24]:
text4.index('awaken')
Out[24]:
173
In [25]:
# Slicing
text5[16715:16735]
Out[25]:
['U86',
 'thats',
 'why',
 'something',
 'like',
 'gamefly',
 'is',
 'so',
 'good',
 'because',
 'you',
 'can',
 'actually',
 'play',
 'a',
 'full',
 'game',
 'without',
 'buying',
 'it']
In [26]:
sent = ['word1', 'word2', 'word3', 'word4', 'word5','word6', 'word7', 'word8', 'word9', 'word10']
In [27]:
sent[5:8]
Out[27]:
['word6', 'word7', 'word8']
In [28]:
sent[:3]
Out[28]:
['word1', 'word2', 'word3']
In [29]:
sent[1:9] = ['Second', 'Third']
In [30]:
sent
Out[30]:
['word1', 'Second', 'Third', 'word10']

Variables

In [33]:
my_sent = ['Bravely', 'bold', 'Sir', 'Robin', ',', 'rode','forth', 'from', 'Camelot', '.']
noun_phrase = my_sent[1:4]
print(noun_phrase)

# Remember that capitalized words appear before lowercase words in sorted lists
wOrDs = sorted(noun_phrase)
print(wOrDs)
['bold', 'Sir', 'Robin']
['Robin', 'Sir', 'bold']

Strings

In [40]:
name = 'Monty'
print(name[0])
print(name[:4])
print(name[2:])
print(name[-2:])
print(name * 2)
print(name + '!')
M
Mont
nty
ty
MontyMonty
Monty!
In [36]:
' '.join(['Monty', 'Python'])
Out[36]:
'Monty Python'
In [37]:
'Monty Python'.split()
Out[37]:
['Monty', 'Python']

Computing with Language: Simple Statistics

Frequency Distributions

In [42]:
fdist1 = FreqDist(text1)
fdist1
Out[42]:
FreqDist({',': 18713, 'the': 13721, '.': 6862, 'of': 6536, 'and': 6024, 'a': 4569, 'to': 4542, ';': 4072, 'in': 3916, 'that': 2982, ...})
In [47]:
type(fdist1)
Out[47]:
nltk.probability.FreqDist
In [43]:
fdist1.most_common(25)
Out[43]:
[(',', 18713),
 ('the', 13721),
 ('.', 6862),
 ('of', 6536),
 ('and', 6024),
 ('a', 4569),
 ('to', 4542),
 (';', 4072),
 ('in', 3916),
 ('that', 2982),
 ("'", 2684),
 ('-', 2552),
 ('his', 2459),
 ('it', 2209),
 ('I', 2124),
 ('s', 1739),
 ('is', 1695),
 ('he', 1661),
 ('with', 1659),
 ('was', 1632),
 ('as', 1620),
 ('"', 1478),
 ('all', 1462),
 ('for', 1414),
 ('this', 1280)]
In [46]:
len(text1)
Out[46]:
260819
In [45]:
# proportion of the text is taken up with such words
fdist1.plot(25, cumulative=True)
In [49]:
# words that occur once only, the so-called 'hapaxes'
len(fdist1.hapaxes())
Out[49]:
9002

Fine-grained Selection of Words

The set of all w such that w is an element of V (the vocabulary) and w has property P

{w | w ∈ V & P(w)}

[w for w in V if p(w)]

produces a list, not a set, which means that duplicates are possible

In [64]:
V = set(text1)
long_words = [w for w in V if len(w) > 15]
sorted_long_words = sorted(long_words)

print(sorted_long_words)
['CIRCUMNAVIGATION', 'Physiognomically', 'apprehensiveness', 'cannibalistically', 'characteristically', 'circumnavigating', 'circumnavigation', 'circumnavigations', 'comprehensiveness', 'hermaphroditical', 'indiscriminately', 'indispensableness', 'irresistibleness', 'physiognomically', 'preternaturalness', 'responsibilities', 'simultaneousness', 'subterraneousness', 'supernaturalness', 'superstitiousness', 'uncomfortableness', 'uncompromisedness', 'undiscriminating', 'uninterpenetratingly']
In [67]:
# all words from the chat corpus that are longer than seven characters, that occur more than seven times
fdist5 = FreqDist(text5)
sorted_fdist5 = sorted(w for w in set(text5) if len(w) > 7 and fdist5[w] > 7)

print(sorted_fdist5)
['#14-19teens', '#talkcity_adults', '((((((((((', '........', 'Question', 'actually', 'anything', 'computer', 'cute.-ass', 'everyone', 'football', 'innocent', 'listening', 'remember', 'seriously', 'something', 'together', 'tomorrow', 'watching']

Collocations and Bigrams

A collocation is a sequence of words that occur together unusually often. Thus red wine is a collocation, whereas the wine is not. A characteristic of collocations is that they are resistant to substitution with words that have similar senses; for example, maroon wine sounds definitely odd.

In [52]:
# To get a handle on collocations, we start off by extracting from a text a list of word pairs, also known as bigrams
list(bigrams(['more', 'is', 'said', 'than', 'done']))
Out[52]:
[('more', 'is'), ('is', 'said'), ('said', 'than'), ('than', 'done')]
In [53]:
# collocations are essentially just frequent bigrams
text4.collocations()
United States; fellow citizens; four years; years ago; Federal
Government; General Government; American people; Vice President; Old
World; Almighty God; Fellow citizens; Chief Magistrate; Chief Justice;
God bless; every citizen; Indian tribes; public debt; one another;
foreign nations; political parties

Counting Other Things

In [83]:
# distribution of word lengths in a text

[len(w) for w in text1]
fdist = FreqDist(len(w) for w in text1)
print(fdist)

fdist
# there are only 19 different word lengths
<FreqDist with 19 samples and 260819 outcomes>
Out[83]:
FreqDist({3: 50223, 1: 47933, 4: 42345, 2: 38513, 5: 26597, 6: 17111, 7: 14399, 8: 9966, 9: 6428, 10: 3528, ...})
In [84]:
# how frequent the different lengths of word are (e.g. how many words of length four appear in the text, are there more words of length five than length four, etc).
fdist.most_common()
Out[84]:
[(3, 50223),
 (1, 47933),
 (4, 42345),
 (2, 38513),
 (5, 26597),
 (6, 17111),
 (7, 14399),
 (8, 9966),
 (9, 6428),
 (10, 3528),
 (11, 1873),
 (12, 1053),
 (13, 567),
 (14, 177),
 (15, 70),
 (16, 22),
 (17, 12),
 (18, 1),
 (20, 1)]
In [85]:
fdist.max()
Out[85]:
3
In [86]:
fdist[3]
Out[86]:
50223
In [87]:
fdist.freq(3)
Out[87]:
0.19255882431878046

Functions Defined for NLTK's Frequency Distributions

Example Description
fdist = FreqDist(samples) create a frequency distribution containing the given samples
fdist[sample] += 1 increment the count for this sample
fdist['monstrous'] count of the number of times a given sample occurred
fdist.freq('monstrous') frequency of a given sample
fdist.N() total number of samples
fdist.most_common(n) the n most common samples and their frequencies
for sample in fdist: f iterate over the samples
fdist.max() sample with the greatest count
fdist.tabulate() tabulate the frequency distribution
fdist.plot() graphical plot of the frequency distribution
fdist.plot(cumulative=True) cumulative plot of the frequency distribution
fdist1 = fdist2 update fdist1 with counts from fdist2
fdist1 < fdist2 test if samples in fdist1 occur less frequently than in fdist2

Making Decisions and Taking Control

Conditionals

In [60]:
print(sent7)
['Pierre', 'Vinken', ',', '61', 'years', 'old', ',', 'will', 'join', 'the', 'board', 'as', 'a', 'nonexecutive', 'director', 'Nov.', '29', '.']
In [59]:
print([w for w in sent7 if len(w) < 4])
print([w for w in sent7 if len(w) <= 4])
print([w for w in sent7 if len(w) == 4])
print([w for w in sent7 if len(w) != 4])
[',', '61', 'old', ',', 'the', 'as', 'a', '29', '.']
[',', '61', 'old', ',', 'will', 'join', 'the', 'as', 'a', 'Nov.', '29', '.']
['will', 'join', 'Nov.']
['Pierre', 'Vinken', ',', '61', 'years', 'old', ',', 'the', 'board', 'as', 'a', 'nonexecutive', 'director', '29', '.']
In [68]:
sorted(term for term in set(text4) if 'gnt' in term)
Out[68]:
['Sovereignty', 'sovereignties', 'sovereignty']
In [71]:
# sorted(item for item in set(text6) if item.istitle())
sorted(item for item in set(sent7) if item.isdigit())
Out[71]:
['29', '61']
In [72]:
sorted(wd for wd in set(text3) if wd.istitle() and len(wd) > 10)
Out[72]:
['Abelmizraim',
 'Allonbachuth',
 'Beerlahairoi',
 'Canaanitish',
 'Chedorlaomer',
 'Girgashites',
 'Hazarmaveth',
 'Hazezontamar',
 'Ishmeelites',
 'Jegarsahadutha',
 'Jehovahjireh',
 'Kirjatharba',
 'Melchizedek',
 'Mesopotamia',
 'Peradventure',
 'Philistines',
 'Zaphnathpaaneah']

Operating on Every Element

In [75]:
# [len(w) for w in text1]
# [w.upper() for w in text1]
In [77]:
print(len(text1))
print(len(set(text1)))

# Now that we are not double-counting words like This and this, which differ only in capitalization
print(len(set(word.lower() for word in text1)))

# Eliminate numbers and punctuation from the vocabulary count by filtering out any non-alphabetic item
print(len(set(word.lower() for word in text1 if word.isalpha())))
260819
19317
17231

Looping with Conditions

In [78]:
sent1 = ['Call', 'me', 'Ishmael', '.']
	
for token in sent1:
    if token.islower():
        print(token, 'is a lowercase word')
    elif token.istitle():
        print(token, 'is a titlecase word')
    else:
        print(token, 'is punctuation')
Call is a titlecase word
me is a lowercase word
Ishmael is a titlecase word
. is punctuation
In [79]:
tricky = sorted(w for w in set(text2) if 'cie' in w or 'cei' in w)
for word in tricky:
    print(word, end=' ')
    
# end=' ' tells Python to print a space (not the default newline) after each word
ancient ceiling conceit conceited conceive conscience conscientious conscientiously deceitful deceive deceived deceiving deficiencies deficiency deficient delicacies excellencies fancied insufficiency insufficient legacies perceive perceived perceiving prescience prophecies receipt receive received receiving society species sufficient sufficiently undeceive undeceiving 

Automatic Natural Language Understanding

we are interested in exploiting our knowledge of language and computation by building useful language technologies. Getting a computer to answer the question automatically involves a range of language processing tasks, including information extraction, inference, and summarization, and would need to be carried out on a scale and with a level of robustness that is still beyond our current capabilities.

Word Sense Disambiguation

we want to work out which sense of a word was intended in a given context.

  • serve: help with food or drink; hold an office; put ball into play
  • dish: plate; course of a meal; communications device

In a sentence containing the phrase: he served the dish, you can detect that both serve and dish are being used with their food meanings. It's unlikely that the topic of discussion shifted from sports to crockery in the space of three words.

We automatically disambiguate words using context, exploiting the simple fact that nearby words have closely related meanings.

  • The book by Chesterton (agentive — Chesterton was the author of the book)
  • The cup by the stove (locative — the stove is where the cup is)
  • Submit by Friday (temporal — Friday is the time of the submitting).

The meaning of the italicized word helps us interpret the meaning of by.

  • The lost children were found by the searchers (agentive)
  • The lost children were found by the mountain (locative)
  • The lost children were found by the afternoon (temporal)

Pronoun Resolution

A deeper kind of language understanding is to work out "who did what to whom" — i.e. to detect the subjects and objects of verbs.

Try to determine what was sold, caught, and found (one case is ambiguous):

  • The thieves stole the paintings. They were subsequently sold
  • The thieves stole the paintings. They were subsequently caught
  • The thieves stole the paintings. They were subsequently found

Answering this question involves finding the antecedent of the pronoun they, either thieves or paintings. Computational techniques for tackling this problem include anaphora resolution — identifying what a pronoun or noun phrase refers to — and semantic role labeling — identifying how a noun phrase relates to the verb (as agent, patient, instrument, and so on).

Generating Language Output

If we can automatically solve such problems of language understanding, we will be able to move on to tasks that involve generating language output, such as question answering and machine translation.

In the first case, a machine should be able to answer a user's questions relating to collection of texts:

  • Text: ... The thieves stole the paintings. They were subsequently sold. ...
  • Human: Who or what was sold?
  • Machine: The paintings.

The machine's answer demonstrates that it has correctly worked out that they refers to paintings and not to thieves.

In the second case, the machine should be able to translate the text into another language, accurately conveying the meaning of the original text. In translating the example text into French, we are forced to choose the gender of the pronoun in the second sentence: ils (masculine) if the thieves are found, and elles (feminine) if the paintings are found. Correct translation actually depends on correct understanding of the pronoun.

  • The thieves stole the paintings. They were subsequently found.
  • Les voleurs ont volé les peintures. Ils ont été trouvés plus tard. (the thieves)
  • Les voleurs ont volé les peintures. Elles ont été trouvées plus tard. (the paintings)

In all of these examples, working out the sense of a word, the subject of a verb, and the antecedent of a pronoun are steps in establishing the meaning of a sentence, things we would expect a language understanding system to be able to do.

Machine Translation

For a long time now, machine translation (MT) has been the holy grail of language understanding, ultimately seeking to provide high-quality, idiomatic translation between any pair of languages. Its roots go back to the early days of the Cold War, when the promise of automatic translation led to substantial government sponsorship, and with it, the genesis of NLP itself.

Spoken Dialog Systems

In the history of artificial intelligence, the chief measure of intelligence has been a linguistic one, namely the Turing Test: can a dialogue system, responding to a user's text input, perform so naturally that we cannot distinguish it from a human-generated response? In contrast, today's commercial dialogue systems are very limited, but still perform useful functions in narrowly-defined domains, as we see here:

  • S: How may I help you?
  • U: When is Saving Private Ryan playing?
  • S: For what theater?
  • U: The Paramount theater.
  • S: Saving Private Ryan is not playing at the Paramount theater, but it's playing at the Madison theater at 3:00, 5:30, 8:00, and 10:30.

You could not ask this system to provide driving instructions or details of nearby restaurants unless the required information had already been stored and suitable question-answer pairs had been incorporated into the language processing system.

Observe that this system seems to understand the user's goals: the user asks when a movie is showing and the system correctly determines from this that the user wants to see the movie. This inference seems so obvious that you probably didn't notice it was made, yet a natural language system needs to be endowed with this capability in order to interact naturally. Without it, when asked Do you know when Saving Private Ryan is playing?, a system might unhelpfully respond with a cold Yes. However, the developers of commercial dialogue systems use contextual assumptions and business logic to ensure that the different ways in which a user might express requests or provide information are handled in a way that makes sense for the particular application. So, if you type When is ..., or I want to know when ..., or Can you tell me when ..., simple rules will always yield screening times. This is enough for the system to provide a useful service.

Figure: Simple Pipeline Architecture for a Spoken Dialogue System:

  • Spoken input (top left) is analyzed , words are recognized, sentences are parsed and interpreted in context, application-specific actions take place (top right)
  • a response is planned, realized as a syntactic structure, then to suitably inflected words, and finally to spoken output
  • different types of linguistic knowledge inform each stage of the process.

Dialogue systems give us an opportunity to mention the commonly assumed pipeline for NLP. 5.1 shows the architecture of a simple dialogue system. Along the top of the diagram, moving from left to right, is a "pipeline" of some language understanding components. These map from speech input via syntactic parsing to some kind of meaning representation. Along the middle, moving from right to left, is the reverse pipeline of components for converting concepts to speech. These components make up the dynamic aspects of the system. At the bottom of the diagram are some representative bodies of static information: the repositories of language-related data that the processing components draw on to do their work.

Textual Entailment

The challenge of language understanding has been brought into focus in recent years by a public "shared task" called Recognizing Textual Entailment (RTE). The basic scenario is simple. Suppose you want to find evidence to support the hypothesis: Sandra Goudie was defeated by Max Purnell, and that you have another short text that seems to be relevant, for example, Sandra Goudie was first elected to Parliament in the 2002 elections, narrowly winning the seat of Coromandel by defeating Labour candidate Max Purnell and pushing incumbent Green MP Jeanette Fitzsimons into third place. Does the text provide enough evidence for you to accept the hypothesis? In this particular case, the answer will be "No." You can draw this conclusion easily, but it is very hard to come up with automated methods for making the right decision. The RTE Challenges provide data that allow competitors to develop their systems, but not enough data for "brute force" machine learning techniques (a topic we will cover in chap-data-intensive).

As another illustration of the difficulty of the task, consider the following text-hypothesis pair:

  • Text: David Golinkin is the editor or author of eighteen books, and over 150 responsa, articles, sermons and books
  • Hypothesis: Golinkin has written eighteen books

In order to determine whether the hypothesis is supported by the text, the system needs the following background knowledge: (i) if someone is an author of a book, then he/she has written that book; (ii) if someone is an editor of a book, then he/she has not written (all of) that book; (iii) if someone is editor or author of eighteen books, then one cannot conclude that he/she is author of eighteen books.

Limitations of NLP

Despite the research-led advances in tasks like RTE, natural language systems that have been deployed for real-world applications still cannot perform common-sense reasoning or draw on world knowledge in a general and robust manner. We can wait for these difficult artificial intelligence problems to be solved, but in the meantime it is necessary to live with some severe limitations on the reasoning and knowledge capabilities of natural language systems. Accordingly, right from the beginning, an important goal of NLP research has been to make progress on the difficult task of building technologies that "understand language," using superficial yet powerful techniques instead of unrestricted knowledge and reasoning capabilities.