#!/usr/bin/python # coding: utf-8 """ Hard West Turn, 2017-11, for NaNoGenMo, draft/version 0.1 Copyright (c) 2017 Nick Montfort Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. This file is offered as-is, without any warranty. ### THIS VERSION NO LONGER WORKS as of 2018-06-01 because the structure of an English Wikipedia article has changed. ### """ # Typeset this text as well if you are laying out a generated book: # # © Nick Montfort 2017 (DRAFT/NaNoGenMo version) # # The program that generates this book uses text and tabluar data from the English and Simple English Wikipedia which is offered under a CC-BY-SA license, so credit is given to these Wikipedias and the book is offered under the same license. # # This is a computer-generated book, the code to which is offered under a permissive free software license. Download it from nickm.com or another place and generate your own book, formatted however you like, if you wish. Or, study or modify the code, or do whatever else you like with the program: # # http://nickm.com/code/hard_west_turn.py # # This book was written for NaNoGenMo 2017 (National Novel Generation Month), with thanks to Darius Kazemi for founding this monthlong event and thanks to the many brilliant programmer/writers who participate in it: # # https://github.com/dariusk/NaNoGenMo import re from random import choice, shuffle import urllib2 from bs4 import BeautifulSoup from textblob import TextBlob english = 'http://en.wikipedia.org' simple = 'http://simple.wikipedia.org' mass_shootings = english + '/wiki/Mass_shootings_in_the_United_States' html = urllib2.urlopen(mass_shootings).read() soup = BeautifulSoup(html, 'lxml') deadliest = soup.find('span', id='Deadliest_shootings').parent incident = {} for row in deadliest.find_next_sibling('table', class_='wikitable').find_all('tr')[1:]: cells = row.find_all(['td']) try: incident_name = cells[0].text incident_link = cells[0].find('a')['href'] except IndexError: continue incident[incident_name] = incident_link paragraphs = {} full = {} links = [] litany = [] simple_litany = [] degenerate_litany = [] para_frames = [ 'This man was given to thinking of events of national importance.', 'The man thought to himself a good deal.', 'Certain things he had heard and learned resonated in the otherwise routine and still mind of the man.', 'Without outward sign of it, the man sometimes had a swirl of thought.', 'The man went west to escape something, but he did not escape the country or himself.', 'The man went west to find something, not knowing what.', 'There were known knowns familiar to the man, as to anyone who thought.', 'There were known unknown the man of course knew about.', 'The man dreamed at night sometimes, or perhaps all the time, with a sliver of that dream sometimes cutting his consciousness in the morning.', 'The man may never have dreamed.', 'To forget that which had been taken away, the man thought on what he knew and had.', 'The man had regrets; he was trying to see if, through thinking on matters, he could develop something other than regrets.', 'The man took things a day at a time and thought about things a thing at a time.', 'The man remembered a lot of things.', 'The man knew that some things were fake, some facts.', 'The man knew that people said things, sometimes for no reason.', 'Just with what was in his head about the facts of the matter, the man could have written a book.', 'The man had many thoughts, few of them clear.', 'The man knew what he knew.', 'The man had heard things.' ] declarations = [ 'The man always has had a tendency to watch and listen while saying little, a trait which serves him well at times.', 'Although the man does not turn his attention to read a newspaper article in full or to listen to the speech of the television or radio, there is no avoiding the insistent world beyond his current town.', 'The man prefers, when he can, to eat alone in a simple place for working people, facing a window.', 'The man still remembers and imagines.', 'The man remains able to say all the words he needs to carry on with his work, such as it is these days, and with providing for himself.', 'When he is not compelled to do something to survive, the man has been known to whittle.', 'The man carries a packet of remembrances with him which he does not realized has sealed itself shut through pressure on the gummed flap of the envelope.', 'The man believes in opportunity.', 'The man once caught himself tapping his foot when no music was playing.', 'There are no points at which the man sees anything suspicious.', 'The man still knew his place.' ] simple_declarations = [ 'It is a simple time and a great country.', 'Eventually the man is going to get something done that will be really, really good.', 'The man feels it is the morning of his life.', 'The man loves his country and its flag.', 'Freedom is most important for the man.', 'At least the man knows he is free.', 'Success is still somewhere before the man.', 'The man requires nothing.', 'Here, the man can range as he will.', 'If the man could shut it out, the lack of news would be good news.', 'There is nothing for the man to see or say.', 'The man was who he chose to be.' ] with_truck = [ 'He drives off in his truck', 'He gets into his truck', 'His truck carries him', 'His truck still is able to take him along', 'He goes off in his truck' ] no_truck = [ 'He takes a long-distance bus', 'He gathers funds for a bus ticket', 'He manages to hitchhike', 'He finds it is still possible to slip into a freight train', 'He is able to walk and camp out along the way' ] laborer_job = [ 'a locksmith', 'a night watchman', 'a mover', 'a day laborer', 'a scab longshoreman', 'a greeter' ] unpleasant_job = [ 'the cleaner of rough commercial spaces', 'a dishwasher', 'a warehouse picker', 'a collector of recyclables', 'a lookout', 'slaughterhouse worker' ] def all_lowercase(sent, full_text): 'Returns a suitably modified sentence if it seems to have no proper nouns.' if re.search(r'[a-z]', sent) and \ sent[1:] == sent[1:].lower(): sent = re.sub(r'\[.*\]', '', sent) sent = re.sub(r'^\"$', '', sent) sent = re.sub(r'"', '', sent) sent = sent.strip() if len(sent.split()) > 2: first_word = sent.split()[0] lc_pattern = re.compile(' ' + first_word.lower() + ' ') uc_pattern = re.compile(r'[a-z] ' + first_word + ' ') if re.search(r'\:$', sent): return None elif re.search(lc_pattern, full_text): return sent elif not re.search(uc_pattern, full_text): return sent elif first_word == 'I': if sent[-1] == '"': return '"' + sent else: second_word = sent.split()[1] rest = ' '.join(sent.split()[1:])[1:] return second_word[0].upper() + rest for i in incident: article = english + incident[i] html = urllib2.urlopen(article).read() soup = BeautifulSoup(html, 'lxml') content = soup.find('div', id='bodyContent') paragraphs[i] = [] for p in content.find_all('p'): paragraphs[i].append(p.getText()) for a in content.find_all('a'): href = a.get('href') if href is not None: href = href.encode('utf-8') if ':' not in href and re.match(r'/wiki', href): links.append(href) full[i] = content.getText() for i in paragraphs: for part in paragraphs[i]: blob = TextBlob(part) for s in blob.sentences: string = str(s) string = all_lowercase(string, full[i]) if string is not None: litany.append(string) for count, rel_url in sorted(((links.count(e), e) for e in set(links)), reverse=True): if 1 < count < 14: article = simple + rel_url try: html = urllib2.urlopen(article).read() soup = BeautifulSoup(html, 'lxml') content = soup.find('div', id='bodyContent') new_paragraphs = [] for p in content.find_all('p'): new_paragraphs.append(p.getText()) for paragraph in new_paragraphs: blob = TextBlob(paragraph) for s in blob.sentences: string = str(s) string = all_lowercase(string, content.getText()) if string is not None: simple_litany.append(string) except urllib2.HTTPError: continue for string in simple_litany: if ',' in string: degenerate_litany.append(string.split(',')[0] + '.') for string in litany: if ',' in string: degenerate_litany.append(string.split(',')[0] + '.') for string in degenerate_litany: if len(string.split()) < 5: degenerate_litany.append(string[:-1] + ', ' + string.lower()) print 'Hard West Turn' print 'Nick Montfort' print print u'(This book is directly based on incidents of violence in recent American history, which are explicitly related in it.)' print print u'And the LORD said unto Satan, Whence comest thou? Then Satan answered the LORD, and said, From going to and fro in the earth, and from walking up and down in it. And the LORD said unto Satan, Hast thou considered my servant Job, that there is none like him in the earth, a perfect and an upright man, one that feareth God, and escheweth evil? Then Satan answered the LORD, and said, Doth Job fear God for nought? Hast not thou made an hedge about him, and about his house, and about all that he hath on every side? thou hast blessed the work of his hands, and his substance is increased in the land. But put forth thine hand now, and touch all that he hath, and he will curse thee to thy face.' print print u'A man wakes not far from the eastern shore of the great nation to find his family has already departed the house their bank is set to seize from them. He thinks on this with the aid of his little remaining liquor and less food for four days and nights. He packs up, shaking, and sets off to the west.' def print_part(statements, declare, travel, job): 'Prints one of three parts of the book.' shuffle(statements) tenth = len(statements) / 10 next_para = 0 for n in range(10): para = ' ' + choice(para_frames) for j in range(tenth): para += ' ' + statements[next_para + j] para += choice(['', '', '', '', '\n ' + choice(para_frames)]) print para next_para += tenth sentence = choice(declare) declare.remove(sentence) final_sentence = ' ' + sentence + ' ' + choice(travel) if len(job) > 0: final_sentence += ' and he finds work as ' + choice(job) final_sentence += '.' print_part(litany, declarations, with_truck, laborer_job) print print ". . ." print print_part(simple_litany, simple_declarations, no_truck, unpleasant_job) print print ". . ." print for pf in para_frames: if len(pf) > 33: para_frames.remove(pf) print_part(degenerate_litany, ['']*12, no_truck, [])