#BEAUTIFUL SOUP PYTHON WEB SCRAPING HOW TO#

find ( 'span', class_ = 'item birthday' ). Similarly, I collect Conor’s birthday, weight, height, nationality, and association (team): birthday = soup. The section of HTML containing this text is shown below: I search soup for the fighter name via the find function using both the html element, span, and the css class name, fn. I reference the name and nickname using the css class in the html. I’ll start with collecting the fighter name and nickname: fighterName = soup. You can see as I move the cursor over the HTML elements they are highlighted in the page.

#BEAUTIFUL SOUP PYTHON WEB SCRAPING HOW TO#

The gif below shows how to view developer tools in Chrome. Ok, so that doesn’t look very useful, but there are a variety of tools to help us sort out what pieces of data we want. The initial contents of soup are shown below (using the soup.prettify() function) Ĭonor 'Notorious' McGregor MMA Stats, Pictures, News, Videos, Biography. from bs4 import BeautifulSoup import urllib import re r = urllib. The first step is to import beautiful soup and then read the web page. So let’s take a look at some of his stats. (Please see References section for links)Ĭonor McGregor is fighting for the UFC Featherweight title at UFC 194 on December 12th against the reigning champ Jose Aldo. NOTE: In case you are looking for a prebuilt solution using Python, there are a couple of Python based Sherdog scapers available on Github. I used this as a template and resource for the examples I provide below. I ran a quick google search for Sherdog web scrapers and found one by Andrew Valish which was written in Node.js.

However Sherdog doesn’t have an API this is where beautiful soup comes in. Specifics about each fight: opponent, result, referee, event, etc.

I’m an avid MMA fan and from time to time I want to check out some data on fighters. So what do you do when you need to grab some data from a website, but there is no API available? Scrape It! Sometimes data comes in a nicely packaged format like a csv or a database table… and sometimes it doesn’t.

Beautiful soup python web scraping