Welcome back, everyone.
In the last video, we went through basics of beautiful soup, working with the beautiful soup library,
and we're going to do a little exercise to get you more comfortable working with beautiful soup.
But as you can see from the first practical section and this practical section, we can do a lot with
only a few lines of code.
For the purpose of this, we're going to work with the following you are all that you could see passed
in here.
I'll bring it up now.
We could take a look at example.
If we go to the to scrape dotcom website, we can go to the quotes one.
Since we already work with the books website, let's try and take a look at the quotes to scrape.
And I want to scrape for the purpose of this challenge.
Let's try to return the actual quote and let's take a look at the tags as well.
Maybe to help perform an analysis, see how the tags are grouped based on text.
If you're running any type of sentiment analysis, NLP, anything like that, that's not important.
But let's for the purpose of getting comfortable with beautiful super basics, let's try to return those
to specifics, the, quote, text and the text.
So I'm going to jump back in.
Spider, I'm going to run this first.
And I do have a new file here, we were working with one file previously created, you can title it.
I have it as Untitled five right now.
It's not important as long as you can execute it.
Since we already have our response from the quotes to scrape, we can use a little bit of beautiful
magic to do some checks really quick.
So, for example, let's use a little Schoop that title text.
Take a look at the title.
OK, we can see the quotes is great.
This is very useful.
If you're looking to work down the tree, you can kind of experiment, iterate through to look at what
the specific tags, what the elements are returning.
So let's try to pull up the parent name of dry soup.
Since we're already working with this title, let's try the parent name.
OK, we can see in the parent name of our title if we go back to the URL, let's.
Select, actually, we don't need that, we can go into the header and we can see the title, so if
we're specifying the parent name of the title, we can see the parent name is the head here.
And these are just little tricks to help you find these elements to return to specifics we've seen with
scrapie as we were looking to use the XPath and we went to select specific elements here, we can select
specific elements to walk down the tree.
There's always multiple methods, really any tool that's going to help you get those specifics returned,
anything that you want to use, always try to keep it simple you can use.
So let's jump back to the page.
And we wanted to return.
Let's first take a look at quotes.
We want to return the quotes on the page.
We could check the structure of the e-mail document.
And we can see a pattern here, how they were created.
We can see the div class quote is used for the structure of the quotes, always helps to check for those
patterns, for those programming patterns.
When you're creating the HTML document, if you want to assign a class that helps.
It really helps with reusability applying that to each of these so we can try to scrape using the class
quotes.
So let's jump back into Spider.
And we can assign our variable name, let's use the soup method soup, find underscore all, and we
can pass in here our argument for the div class.
So we want let's assign the div tag that we were working with and we know the class so we can set class
equal to quotes.
That's what we had it.
We had equal to quote.
And we can close this and we can run it.
So let me grab this line.
Actually, we can execute the whole thing if we want to execute it again.
And we have our quotes.
If we run the quotes.
Let's take a look.
We have the quotes returned, we can see that the d'Hiv class quotes return, but we're also receiving
the extra information related to the class.
If we're looking for finding all with the div tag, it's actually returning all these divs.
And we want some specifics, we always want the specific text that we're looking to scrape, so I'm
going to try to iterate through quote, so for X in quotes, let's print out the information that we
want we want to find so we can use X don't find.
And let me jump back, or we can look in here and find the quotes, OK, we have a quote, it might
be easier read if I jump back into our chrome.
Inspector, and we can take a look, we have a quote and then we have a spin here, the spin element
with the quote inside it.
OK, so let's jump back in here and we can use the exact fine.
Let's look for the spin so we won't find we have to pass it or spin.
And we could pull the text right after that.
We need the closing because I deleted the one parentheses before and we can select this.
Run election or current line?
And there we have it, we are returning each of the quotes awesome and we can essentially do the same
thing just to keep it simple for this exercise, let's do the same thing.
We're looking for jumping back.
We're looking for the tags.
So let's go back in the spider.
We can just copy this.
Let's assign it to let's use tags, always keeping it simple tags equal super find all it was.
Let me go check.
Let's take a look at the tags.
It was in div tags, OK, just a double check div class tags and then we have the tags assigned here
in the inspector.
So let's go back into Spider one more time and we can change this to tags.
And we can also use I don't know if we're going to need the fine method since we're looking in the inspector
div class tags and the tags are directly within this.
We don't have to deal with this band element like we dealt with previously, so we can just use the
extra text to strip or excuse me, to grab the text directly from the element, let's use text.
And we can run all this.
Oh, no, one second.
We don't want to go.
Always going to make these mistakes just super easy if you're using that quote.
We were talking about the tag, so we want to use the tags class.
That kind of makes sense.
All right.
Now, let's try that again.
Let's go to run selection of current line and execute.
And we have our tags.
Fantastic.
We're just returning the tags.
We're able to print them out.
We're also able to print out just the quotes that we were looking for.
Fantastic, awesome work, guys.
And this is just intended to help you become a little more familiar with beautiful soup, again, we've
kept things very simple.
I like keeping things simple, especially when it comes to Web scraping.
If we're just looking to return this text, we don't really need to go crazy, but we're going to step
it up in the next lecture.
We're going to do introduction to our challenge.
And with the challenge, we're going to be scraping an actual and I want to say this is an actual Web
site.
This is built for practicing scrapie.
We're going to be scraping an actual website with product information on it.
So I will see you in the challenge introduction.