Homework 9 Solution

Most students did very well on this assignment. Please take a detailed look at my solution in resources/hmwk/hmwk9.py

1. Use the get_wiki function defined below to download the wikipedia page about Ben Franklin. (4 points)

   def get_wiki(url):
       'Download text from a wikipedia page and return raw text'
       from urllib2 import urlopen, Request
       headers = {'User-Agent': '''Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB;
                   rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4'''}
       req=Request(url=url, headers=headers)
       f = urlopen(req)
       raw = unicode(f.read(),encoding='utf8')
       return(raw)
   

   raw = get_wiki('http://en.wikipedia.org/wiki/Ben_franklin')
2. Wikipedia pages generally have a bunch of references and external links. These almost always occur at the section beginning “See also”. Strip off all text after “See also”. Hint: “See also” actually occurs twice in the document – once in the table of contents, and once as a section heading. You only want to ignore stuff after the section heading. (6 points)
   see_index = raw.rfind('See also')
   text = raw[:see_index]
3. Next, define a function called unknown, which removes any items from
   this set that occur in the Words Corpus (nltk.corpus.words). The function should take a list of tokenized words as input, and return a list of novel words. Hint 1: your code
   will be much faster if you convert the list of words in the Words Corpus into a
   set, and then check whether or not a word is in that set. Hint 2: ignore case and punctuation when checking against the words corpus (but preserve case in your output). Hint 3: Sometimes the nltk word tokenizer does not strip off periods ending sentences. Make sure that none of the words end in a period. (15 points)
   def unknown(tokens):
       from string import punctuation
       words =  nltk.corpus.words.words()
       # convert to lower case and make into a set
       words = set([w.lower() for w in words])
       'returns novel tokens (those not in the words corpus)'
       nopunc = [w.rstrip('.') for w in tokens if w not in punctuation]
       nopunc.append('foo')
       novel = [w for w in nopunc if w.lower() not in words]
       return(novel)
4. Use your unknown function to find novel words in the wikipedia page on Ben Franklin (5 points)
   cleaned = nltk.clean_html(text)
   tokens = nltk.word_tokenize(cleaned)
   novel = unknown(tokens)
5. As with most computational linguistics processes, it is nearly impossible to achieve perfect results. It is clear from browsing through the results that there are a number of “novel” words, which in fact are not novel. Let’s further refine our process. Some of the “novel” words we have found may be numbers, proper names (named entities), acronyms, or words with affixes (including both inflectional and derivational affixes). Let’s try to divide up our novel words into these categories.
   1. Use regular expressions to remove any numbers from the novel words.
      Remember that a number may have commas or a decimal point in it, and may begin
      with a dollar sign or end with a percent sign. Save the result as
      novel_nonum. Hint: when testing your regular expression, it is probaby
      easier to check the result of finding items which are numbers, as opposed to
      checking the result of finding items which are not numbers. (8 points)
      import re
      number_re = r'\$?[0-9,.]+%?$'
      number_match = re.compile(number_re)
      novel_nonum = [w for w in novel if not number_match.match(w)]
   2. Now Use the porter stemmer to stem all the items in novel_nonum, then re-run them through the unknown function, saving the result as as novel_stems (7 points)
      porter = nltk.PorterStemmer()
      stemmed = [porter.stem(w) for w in novel_nonum]
      novel_stems = unknown(stemmed)
   3. Next, find as many proper names from novel_stems as possible, saving the result as proper_names. Note that finding named entities is actually a very difficult problem, and usually involves syntax and semantics. For our purposes however, let’s just use the fact that proper names in English start with a capital letter. Also create a new variable novel_no_proper, which has the proper names removed. (5 points)
      proper_names = [w for w in novel_stems if w[0].isupper()]
      novel_no_proper = [w for w in novel_stems if not w[0].isupper()]
   4. Calculate the percentage of novel tokens in the Ben Franklin wikipedia page, after having excluded number, affixed words, and proper names. (4 points)
      novel_token = len(novel_no_proper) / len(tokens)
   5. Calculate the percentage of novel types in the Ben Franklin wikipedia page, after having excluded number, affixed words, and proper names. (6 points)
      novel_type = len(set(novel_no_proper)) / len(set(tokens))
6. Extra Credit: Find additional ways to remove false positives in our “novel” word list. (3 extra points for each additional way, up to 12 extra points)
   # remove smart quotes, dashes  and other such characters
   novel_no_quotes =  [w for w in novel_no_proper if w.isalpha()]
   # Try to repair stemming process
   # Sometimes the stemmer removes final e's which should be there
   novel_fixed_e = [w for w in novel_no_quotes if w+'e'.lower() not in words]
   # Likewise with 'ate'
   novel_fixed_ate = [w for w in novel_fixed_e if w+'ate'.lower() not in words]
   # Sometimes the stemmer converts y to i when it shouldn't
   novel_fixed_i = [w for w in novel_fixed_ate if re.sub('i$', 'y', w).lower() not in words]