Understanding How Beautiful Soup Works

Two years ago, when I first grabbed the transcripts of the TED talks, using wget, I relied upon the wisdom and generosity of Padraic C on StackOverflow to help me use Python’s BeautifulSoup library to get the data out of the downloaded HTML files that I wanted. Now that Katherine Kinnaird and I have decided to add talks published since then, and perhaps even go so far as to re-download the entire corpus so that everything is as much the same as possible, it was time for me to understand how BeautifulSoup (hereafter BS4) works for myself.

from bs4 import BeautifulSoup

# NB: no need to read() the file: BS4 does that
thesoup = BeautifulSoup(open("transcript.0.html"), "html5lib")

# Talk metadata is in <meta> tags in the <head>.
# This finds all <meta> tags
metas = thesoup.find_all("meta")

# Let's see what this object is...
print(type(metas))

Output: <class 'bs4.element.ResultSet'>, and we can interact with it as if it were a list. Thus, metas[0] yields: <meta charset="utf-8"/>, which is the first of a long line of <meta tags. (The complete output is at the bottom of this note below under the heading Appendix A.)

type(metas[0]) outputs: <class 'bs4.element.Tag'>. That means we will need to understand how to select items within a BS4 Tag. The items we are interested in are towards the bottom of the result set:

<meta content="Good news in the fight against pancreatic cancer" itemprop="name"/>
<meta content="Anyone who has lost a loved one to pancreatic cancer knows the devastating speed with which it can affect an otherwise healthy person. TED Fellow and biomedical entrepreneur Laura Indolfi is developing a revolutionary way to treat this complex and lethal disease: a drug delivery device that acts as a cage at the site of a tumor, preventing it from spreading and delivering medicine only where it's needed. &quot;We are hoping that one day we can make pancreatic cancer a curable disease,&quot; she says." itemprop="description"/>
<meta content="PT6M3S" itemprop="duration"/>
<meta content="2016-05-17T14:46:20+00:00" itemprop="uploadDate"/>
<meta content="1246654" itemprop="interactionCount"/>
<meta content="Laura Indolfi" itemprop="name"/>

This gives us the slug, the description, the run time, the publication date, the number of hits, and the speaker. So, the question is, how do we navigate the “parse tree” so that we turn up the value of the content attributes when the value of the itemprop attribute is one of the above?

[meta.attrs for meta in metas] returns a list of dictionaries, with each meta its own dictionary. Here is a small sample from the larger list:

{'content': 'PT6M3S', 'itemprop': 'duration'},
{'content': '2016-05-17T14:46:20+00:00', 'itemprop': 'uploadDate'},
{'content': '1246654', 'itemprop': 'interactionCount'},
{'content': 'Laura Indolfi', 'itemprop': 'name'},

What we need to do is identify the dictionary’s position in the list by finding those dictionaries that have the values duration, etc. We then use that position to slice to that dictionary, and get the value associated with content, yes?

It turns out that the best way to do this is built into BS4, though the method was not immediately obvious. One of the answers to the StackOverflow question “Get meta tag content property with BeautifulSoup and Python” suggested the following possibility:

for tag in thesoup.find_all("meta"):
    if tag.get("name", None) == "author":
        speaker = tag.get("content", None)
    if tag.get("itemprop", None) == "duration":
        length = tag.get("content", None)
    if tag.get("itemprop", None) == "uploadDate":
        published = tag.get("content", None)
    if tag.get("itemprop", None) == "interactionCount":
        views = tag.get("content", None)
    if tag.get("itemprop", None) == "description":
        description = tag.get("content", None)

If we ask to see these values with print(speaker, length, published, views, description), we get:

Laura Indolfi PT6M3S 2016-05-17T14:46:20+00:00 1246654 Anyone
who has lost a loved one to pancreatic cancer knows the devastating
speed with which it can affect an otherwise healthy person. TED
Fellow and biomedical entrepreneur Laura Indolfi is developing a
revolutionary way to treat this complex and lethal disease: a drug
delivery device that acts as a cage at the site of a tumor,
preventing it from spreading and delivering medicine only where
it's needed. "We are hoping that one day we can make pancreatic
cancer a curable disease," she says.

Now we need to get the text of the talk out, which is made somewhat difficult by the lack of semantic markup. The start of the text looks like this:

<!-- Transcript text -->
  <div class="Grid Grid--with-gutter d:f@md p-b:4">
    <div class="Grid__cell d:f h:full m-b:.5 m-b:0@md w:12"></div>

    <div class="Grid__cell flx-s:1 p-r:4">

The only reliable thing is the comment tag: there’s also a closing one at the end of the transcript text, so if we can find some way to select all the <p> tags between the two comments, I think we’ll be in good shape.

Appendix A

The output of [print(meta) for meta in metas] is:

<meta charset="utf-8"/>
<meta content="TED Talk Subtitles and Transcript: Anyone who has lost a loved one to pancreatic cancer knows the devastating speed with which it can affect an otherwise healthy person. TED Fellow and biomedical entrepreneur Laura Indolfi is developing a revolutionary way to treat this complex and lethal disease: a drug delivery device that acts as a cage at the site of a tumor, preventing it from spreading and delivering medicine only where it's needed. &quot;We are hoping that one day we can make pancreatic cancer a curable disease,&quot; she says." name="description"/>
<meta content="Laura Indolfi" name="author"/>
<meta content='Transcript of "Good news in the fight against pancreatic cancer"' property="og:title"/>
<meta content="https://pi.tedcdn.com/r/talkstar-photos.s3.amazonaws.com/uploads/70d551c2-1e5c-411e-b926-7d72590f66bb/LauraIndolfi_2016U-embed.jpg?c=1050%2C550&amp;w=1050" property="og:image"/>
<meta content="https://pi.tedcdn.com/r/talkstar-photos.s3.amazonaws.com/uploads/70d551c2-1e5c-411e-b926-7d72590f66bb/LauraIndolfi_2016U-embed.jpg?c=1050%2C550&amp;w=1050" property="og:image:secure_url"/>
<meta content="1050" property="og:image:width"/>
<meta content="550" property="og:image:height"/>
<meta content="article" property="og:type"/>
<meta content="TED, Talks, Themes, Speakers, Technology, Entertainment, Design" name="keywords"/>
<meta content="#E62B1E" name="theme-color"/>
<meta content="True" name="HandheldFriendly"/>
<meta content="320" name="MobileOptimized"/>
<meta content="width=device-width, initial-scale=1.0" name="viewport"/>
<meta content="TED Talks" name="apple-mobile-web-app-title"/>
<meta content="yes" name="apple-mobile-web-app-capable"/>
<meta content="black" name="apple-mobile-web-app-status-bar-style"/>
<meta content="TED Talks" name="application-name"/>
<meta content="https://www.ted.com/browserconfig.xml" name="msapplication-config"/>
<meta content="#000000" name="msapplication-TileColor"/>
<meta content="on" http-equiv="cleartype"/>
<meta content="Laura Indolfi: Good news in the fight against pancreatic cancer" name="title"/>
<meta content="TED Talk Subtitles and Transcript: Anyone who has lost a loved one to pancreatic cancer knows the devastating speed with which it can affect an otherwise healthy person. TED Fellow and biomedical entrepreneur Laura Indolfi is developing a revolutionary way to treat this complex and lethal disease: a drug delivery device that acts as a cage at the site of a tumor, preventing it from spreading and delivering medicine only where it's needed. &quot;We are hoping that one day we can make pancreatic cancer a curable disease,&quot; she says." property="og:description"/>
<meta content="https://www.ted.com/talks/laura_indolfi_good_news_in_the_fight_against_pancreatic_cancer/transcript" property="og:url"/>
<meta content="201021956610141" property="fb:app_id"/>
<meta content="Good news in the fight against pancreatic cancer" itemprop="name"/>
<meta content="Anyone who has lost a loved one to pancreatic cancer knows the devastating speed with which it can affect an otherwise healthy person. TED Fellow and biomedical entrepreneur Laura Indolfi is developing a revolutionary way to treat this complex and lethal disease: a drug delivery device that acts as a cage at the site of a tumor, preventing it from spreading and delivering medicine only where it's needed. &quot;We are hoping that one day we can make pancreatic cancer a curable disease,&quot; she says." itemprop="description"/>
<meta content="PT6M3S" itemprop="duration"/>
<meta content="2016-05-17T14:46:20+00:00" itemprop="uploadDate"/>
<meta content="1246654" itemprop="interactionCount"/>
<meta content="Laura Indolfi" itemprop="name"/>
<meta content="Flash HTML5" itemprop="playerType"/>
<meta content="640" itemprop="width"/>
<meta content="360" itemprop="height"/>

Updating the TEDtalk Corpus

Today’s task is to determine which talks have been added since the initial download of talks in May 2016 both in order to increase the size of the corpus as well as to make sure that our procedures for acquiring and cleaning data have been fully documented. To get started, I read my own documentation for 2016’s effort, which contained a link to the Google Sheet where the data is housed.

I downloaded the Sheet as a CSV file and compared it to the one from May 2016. A couple of issues emerged immediately: First, neither FileMerge, part of Xcode, nor DiffMerge handle files this size very well. Both are slowww. Second, the columns have moved and changed names:

May 2016: 18,id,/,Speaker,Name,Short Summary,Event,Duration,Publish date,
May 2018: Talk ID,public_url,speaker_name,headline,description,event,duration,language,published,tags

For the purpose of comparison, I only need ID and url, so I think I’ll make a copy of both files and trim them to those columns.… On closer inspection, the URLs have changed. In 2016, the URLs were by the ID number — http://www.ted.com/talks/view/id/53 — but now they are a blend of author and title: https://www.ted.com/talks/majora_carter_s_tale_of_urban_renewal. So I’ll go with ID and speaker for comparison but keep the URL so I can use that for the download list.

Kaleidoscope can handle files this size and its viewing options are pretty nice — and it’s affordably priced, if I need to keep it. (It tracks 76 changes, but I can see that some of those changes are simply some internal shifting that the applications doesn’t follow. I’ll take the automagic where I can get it, and work where I don’t.)

Slight change of plan: I duplicated the files and I am going to sort by ID number to see if that helps. Now Kaleidoscop is showing 412 changes. That doesn’t work. Some talks from the earlier CSV do not have IDs.

I considered a number of solutions to the problem, but I returned to the original diff, where Kaleidoscope tracked 76 changes and inspected the diff by hand. Most of the changes were lines shifted up or down, so I resolved those by hand. That left three large blocks of lines at the end that I copied and pasted into a new CSV as well as the following outliers listed by ID and speaker:

- 25, Tan
- 1676, Davis
- 1923, Cameron
+ 2386, n/a, year in ideas
+ 2451, Gerald
+ 2464, Torvalds
- n/a, Evans
+ 925, Shaw

Tomorrow I will see if the wget code still works.

Tedtalk Pie

My friend and colleague Pablo sent this along without realizing it was, in fact, about Tedtalks. (Sometimes you have to worry about mathematicians — but don’t tell them you’re worried.)

Anatomy of a Tedtalk as a Stacked Pie Chart (quality of statistics unknown).