import nltk
# nltk.download()
from nltk.book import *
# Test that the data has been installed
from nltk.corpus import brown
brown.words()
text1
text2
# A concordance view shows us every occurrence of a given word, together with some context
text1.concordance("monstrous")
# Words that appear in a similar range of contexts
text1.similar("monstrous")
text2.similar("monstrous")
# Examine just the contexts that are shared by two or more words
text2.common_contexts(["monstrous", "very"])
import matplotlib.pyplot as plt
%matplotlib inline
# Determine the location of a word in the text
text4.dispersion_plot(["citizens", "democracy", "freedom", "duties", "America"])
# A 'Token' is the technical name for a sequence of characters
len(text3)
# Distinct words
# sorted(set(text3))
# Number of distinct words - call these unique items 'Types'
len(set(text3))
# Lexical richness of the text. i.e number of distinct words is just 6% of the total number of words
len(set(text3)) / len(text3)
# How often a word occurs in a text
text3.count("smote")
# What percentage of the text is taken up by a specific word
100 * text3.count("smote") / len(text3)
def lexical_diversity(text):
return len(set(text)) / len(text)
def percentage(count, total):
return 100 * count / total
lexical_diversity(text3)
percentage(text3.count("smote"), len(text3))
# We will think of a text as nothing more than a sequence of words and punctuation
sent1
sent1.append("Some")
sent1
sent4 + sent1
text4[173]
text4.index('awaken')
# Slicing
text5[16715:16735]
sent = ['word1', 'word2', 'word3', 'word4', 'word5','word6', 'word7', 'word8', 'word9', 'word10']
sent[5:8]
sent[:3]
sent[1:9] = ['Second', 'Third']
sent
my_sent = ['Bravely', 'bold', 'Sir', 'Robin', ',', 'rode','forth', 'from', 'Camelot', '.']
noun_phrase = my_sent[1:4]
print(noun_phrase)
# Remember that capitalized words appear before lowercase words in sorted lists
wOrDs = sorted(noun_phrase)
print(wOrDs)
name = 'Monty'
print(name[0])
print(name[:4])
print(name[2:])
print(name[-2:])
print(name * 2)
print(name + '!')
' '.join(['Monty', 'Python'])
'Monty Python'.split()
fdist1 = FreqDist(text1)
fdist1
type(fdist1)
fdist1.most_common(25)
len(text1)
# proportion of the text is taken up with such words
fdist1.plot(25, cumulative=True)
# words that occur once only, the so-called 'hapaxes'
len(fdist1.hapaxes())
The set of all w such that w is an element of V (the vocabulary) and w has property P
{w | w ∈ V & P(w)}
[w for w in V if p(w)]
produces a list, not a set, which means that duplicates are possible
V = set(text1)
long_words = [w for w in V if len(w) > 15]
sorted_long_words = sorted(long_words)
print(sorted_long_words)
# all words from the chat corpus that are longer than seven characters, that occur more than seven times
fdist5 = FreqDist(text5)
sorted_fdist5 = sorted(w for w in set(text5) if len(w) > 7 and fdist5[w] > 7)
print(sorted_fdist5)
A collocation is a sequence of words that occur together unusually often. Thus red wine is a collocation, whereas the wine is not. A characteristic of collocations is that they are resistant to substitution with words that have similar senses; for example, maroon wine sounds definitely odd.
# To get a handle on collocations, we start off by extracting from a text a list of word pairs, also known as bigrams
list(bigrams(['more', 'is', 'said', 'than', 'done']))
# collocations are essentially just frequent bigrams
text4.collocations()
# distribution of word lengths in a text
[len(w) for w in text1]
fdist = FreqDist(len(w) for w in text1)
print(fdist)
fdist
# there are only 19 different word lengths
# how frequent the different lengths of word are (e.g. how many words of length four appear in the text, are there more words of length five than length four, etc).
fdist.most_common()
fdist.max()
fdist[3]
fdist.freq(3)
Functions Defined for NLTK's Frequency Distributions
Example | Description |
---|---|
fdist = FreqDist(samples) | create a frequency distribution containing the given samples |
fdist[sample] += 1 | increment the count for this sample |
fdist['monstrous'] | count of the number of times a given sample occurred |
fdist.freq('monstrous') | frequency of a given sample |
fdist.N() | total number of samples |
fdist.most_common(n) | the n most common samples and their frequencies |
for sample in fdist: f | iterate over the samples |
fdist.max() | sample with the greatest count |
fdist.tabulate() | tabulate the frequency distribution |
fdist.plot() | graphical plot of the frequency distribution |
fdist.plot(cumulative=True) | cumulative plot of the frequency distribution |
fdist1 = fdist2 | update fdist1 with counts from fdist2 |
fdist1 < fdist2 | test if samples in fdist1 occur less frequently than in fdist2 |
print(sent7)
print([w for w in sent7 if len(w) < 4])
print([w for w in sent7 if len(w) <= 4])
print([w for w in sent7 if len(w) == 4])
print([w for w in sent7 if len(w) != 4])
sorted(term for term in set(text4) if 'gnt' in term)
# sorted(item for item in set(text6) if item.istitle())
sorted(item for item in set(sent7) if item.isdigit())
sorted(wd for wd in set(text3) if wd.istitle() and len(wd) > 10)
# [len(w) for w in text1]
# [w.upper() for w in text1]
print(len(text1))
print(len(set(text1)))
# Now that we are not double-counting words like This and this, which differ only in capitalization
print(len(set(word.lower() for word in text1)))
# Eliminate numbers and punctuation from the vocabulary count by filtering out any non-alphabetic item
print(len(set(word.lower() for word in text1 if word.isalpha())))
sent1 = ['Call', 'me', 'Ishmael', '.']
for token in sent1:
if token.islower():
print(token, 'is a lowercase word')
elif token.istitle():
print(token, 'is a titlecase word')
else:
print(token, 'is punctuation')
tricky = sorted(w for w in set(text2) if 'cie' in w or 'cei' in w)
for word in tricky:
print(word, end=' ')
# end=' ' tells Python to print a space (not the default newline) after each word
we are interested in exploiting our knowledge of language and computation by building useful language technologies. Getting a computer to answer the question automatically involves a range of language processing tasks, including information extraction, inference, and summarization, and would need to be carried out on a scale and with a level of robustness that is still beyond our current capabilities.
we want to work out which sense of a word was intended in a given context.
In a sentence containing the phrase: he served the dish, you can detect that both serve and dish are being used with their food meanings. It's unlikely that the topic of discussion shifted from sports to crockery in the space of three words.
We automatically disambiguate words using context, exploiting the simple fact that nearby words have closely related meanings.
The meaning of the italicized word helps us interpret the meaning of by.
A deeper kind of language understanding is to work out "who did what to whom" — i.e. to detect the subjects and objects of verbs.
Try to determine what was sold, caught, and found (one case is ambiguous):
Answering this question involves finding the antecedent of the pronoun they, either thieves or paintings. Computational techniques for tackling this problem include anaphora resolution — identifying what a pronoun or noun phrase refers to — and semantic role labeling — identifying how a noun phrase relates to the verb (as agent, patient, instrument, and so on).
If we can automatically solve such problems of language understanding, we will be able to move on to tasks that involve generating language output, such as question answering and machine translation.
In the first case, a machine should be able to answer a user's questions relating to collection of texts:
The machine's answer demonstrates that it has correctly worked out that they refers to paintings and not to thieves.
In the second case, the machine should be able to translate the text into another language, accurately conveying the meaning of the original text. In translating the example text into French, we are forced to choose the gender of the pronoun in the second sentence: ils (masculine) if the thieves are found, and elles (feminine) if the paintings are found. Correct translation actually depends on correct understanding of the pronoun.
In all of these examples, working out the sense of a word, the subject of a verb, and the antecedent of a pronoun are steps in establishing the meaning of a sentence, things we would expect a language understanding system to be able to do.
For a long time now, machine translation (MT) has been the holy grail of language understanding, ultimately seeking to provide high-quality, idiomatic translation between any pair of languages. Its roots go back to the early days of the Cold War, when the promise of automatic translation led to substantial government sponsorship, and with it, the genesis of NLP itself.
In the history of artificial intelligence, the chief measure of intelligence has been a linguistic one, namely the Turing Test: can a dialogue system, responding to a user's text input, perform so naturally that we cannot distinguish it from a human-generated response? In contrast, today's commercial dialogue systems are very limited, but still perform useful functions in narrowly-defined domains, as we see here:
You could not ask this system to provide driving instructions or details of nearby restaurants unless the required information had already been stored and suitable question-answer pairs had been incorporated into the language processing system.
Observe that this system seems to understand the user's goals: the user asks when a movie is showing and the system correctly determines from this that the user wants to see the movie. This inference seems so obvious that you probably didn't notice it was made, yet a natural language system needs to be endowed with this capability in order to interact naturally. Without it, when asked Do you know when Saving Private Ryan is playing?, a system might unhelpfully respond with a cold Yes. However, the developers of commercial dialogue systems use contextual assumptions and business logic to ensure that the different ways in which a user might express requests or provide information are handled in a way that makes sense for the particular application. So, if you type When is ..., or I want to know when ..., or Can you tell me when ..., simple rules will always yield screening times. This is enough for the system to provide a useful service.
Figure: Simple Pipeline Architecture for a Spoken Dialogue System:
Dialogue systems give us an opportunity to mention the commonly assumed pipeline for NLP. 5.1 shows the architecture of a simple dialogue system. Along the top of the diagram, moving from left to right, is a "pipeline" of some language understanding components. These map from speech input via syntactic parsing to some kind of meaning representation. Along the middle, moving from right to left, is the reverse pipeline of components for converting concepts to speech. These components make up the dynamic aspects of the system. At the bottom of the diagram are some representative bodies of static information: the repositories of language-related data that the processing components draw on to do their work.
The challenge of language understanding has been brought into focus in recent years by a public "shared task" called Recognizing Textual Entailment (RTE). The basic scenario is simple. Suppose you want to find evidence to support the hypothesis: Sandra Goudie was defeated by Max Purnell, and that you have another short text that seems to be relevant, for example, Sandra Goudie was first elected to Parliament in the 2002 elections, narrowly winning the seat of Coromandel by defeating Labour candidate Max Purnell and pushing incumbent Green MP Jeanette Fitzsimons into third place. Does the text provide enough evidence for you to accept the hypothesis? In this particular case, the answer will be "No." You can draw this conclusion easily, but it is very hard to come up with automated methods for making the right decision. The RTE Challenges provide data that allow competitors to develop their systems, but not enough data for "brute force" machine learning techniques (a topic we will cover in chap-data-intensive).
As another illustration of the difficulty of the task, consider the following text-hypothesis pair:
In order to determine whether the hypothesis is supported by the text, the system needs the following background knowledge: (i) if someone is an author of a book, then he/she has written that book; (ii) if someone is an editor of a book, then he/she has not written (all of) that book; (iii) if someone is editor or author of eighteen books, then one cannot conclude that he/she is author of eighteen books.
Despite the research-led advances in tasks like RTE, natural language systems that have been deployed for real-world applications still cannot perform common-sense reasoning or draw on world knowledge in a general and robust manner. We can wait for these difficult artificial intelligence problems to be solved, but in the meantime it is necessary to live with some severe limitations on the reasoning and knowledge capabilities of natural language systems. Accordingly, right from the beginning, an important goal of NLP research has been to make progress on the difficult task of building technologies that "understand language," using superficial yet powerful techniques instead of unrestricted knowledge and reasoning capabilities.