Hello and welcome back.
Let's move right to working with beautiful soup.
In the last video, we used requests to return our HTP and HTML doc information.
We can now use beautiful soup to pass and work with our content to help us scrape.
So to get started, let's first import beautiful soup in order to import beautiful soup, let's give
ourselves some space.
We can use the following from BBB's for import beautiful soup.
We'll give us our beautiful soup library.
I'm also going to add a comment here.
Just we don't have to print the information that we have returned from the last lecture.
We're using our request to get Google Dotcom.
And now we can use the following to get our content and Poza with XML.
And I know I mentioned Al XML as an example, XML is a great option since it's really lightweight,
but there's also a default HTML parser with beautiful soup, or you can use something such as HTML5,
which is recommended to use if you're working with broken HTML.
I mentioned earlier that XML is one of the favorites due to it being so lightweight, but you can really
evaluate which one you use based on your use case or content that you're working with, such as the
broken e-mail content, static, dynamic, etc..
Beautiful Zoop even has a section on this if you're looking for any further information within their
documentation.
So let's take a test of beautiful soup.
Let's do soup equals beautiful soup and we can use our let's do our content.
You can use our don't text.
Let's try our content.
And we also have to pass in the argument of our pastor here, LSM.
Well, let's use that.
Four hour passes and we can run this.
And it has successfully executed we don't have any information retrieved yet because we're not printing
or logging any information and we can use print statements as we've done here, or we can jump directly
into the console.
So let's use Shupe.
We already have superset to our beautiful soup content.
So let's use Soup as an example subtitle.
Let's run it.
And we can see that it's already pulling the title from the page that we're requesting the Google dot
com and we can use supertitles as an element example.
We can also try the following methods such as let's try soup that get underscore text and it will pull
the text from the page and you can also use it to find specific elements or find specific classes.
So let's try soup, dot, find, underscore all and let's check for all elements on the page.
We want to close this and let's run it.
And we can see all of the classes or excuse me, a elements on the page and we can go back to our IDY
or the main part of Spider out of the console, and we could try the following.
Let's use the prettify method.
And prettify in beautiful soup will turn beautiful soups partially into a nicely formatted Unicode string
with a separate line for each tag and each string.
So let's use our print option here.
Print soup prettify and we can run this.
Let's just run this specific selection or current line and we scroll up.
We can find the military, we can find our document prettified.
And this is a really useful method when we're looking through to find specific text.
This is another option compared to using the inspector.
And this is the main point or takeaway.
We've already worked with scrapie.
We're working with beautiful soup now and we see other methods of parsing our document, of crawling
our document, really grabbing the document.
Since we used request to get it.
We're grabbing the e-mail document.
We're transforming, in a sense, all of this information into objects where we can select them using
Python.
We can really work through the tree.
And this is going to be really helpful when you're parsing and working and crawling websites that have
a lot of information, a lot of different classes, a lot of different elements within the tree.
Moreover, if we jump into the beautiful soup documentation really quickly and we can see that with
the beautiful soup library, it makes it really easy to traverse or walk the tree.
And we've mentioned this in the previous section with scrapie, with our HTML doc.
We have to work through the elements to get those specific elements and selectors if we want to return
that structured exact text and beautiful soup makes it really simple with Python to kind of work through
these objects.
And we can see the going up, going sideways, back and forth, the parent elements next siblings.
This is all to help traverse and walk that tree.
So let's take a look at and we could take a look at going up and we could use the parent attribute here
to access the elements parent.
And this is just really helpful.
And it's a simple example of how powerful, beautiful soup is when we're looking to traverse and walk
the tree.
And this is the takeaway.
We always have multiple options.
There's always multiple libraries.
Beautiful soup, in my opinion, makes it really simple to use Python to scrape these objects and return
information in a really structured way.
And it's going to be helpful in the future when we're working on more complex operations.
All right.
So I hope you're starting to see how beautiful soup can iterate through specific elements or select
specific elements such as soup title, soup, final, good text, each one, AP tags, etc..
In the next video, we're going to scrape another example website with beautiful soup to help us become
more familiar with the operations.
I hope you're enjoying this library so far and I will see you in the next lecture.