Welcome back, everyone.
In the last video, we were able to successfully go through our page or pages because we were using
the click options to navigate our intended target page to go through the learning paths and select a
learning path that we want to scrape information from.
Now, if we're back in Spider, we're going to resume where we left off so we can get started here by
first.
Let's give ourselves some space because we don't want the driver to quit right away.
Now, since we were able to get rid of the pop up, if we were running into a pop up, we now can scrape
the information that we want.
And this is where we can start using beautiful soup or other scraping languages, but for this we're
sticking with beautiful soup.
So let's implement our beautiful soup scrape and the smell parser so we can set Arpège sauce equal to
driver page sauce and then we can use soup equal.
Two beautiful super in our page source and our pastor, Alex Emelle, since we have this, let's go
visit the scraping information that we want to collect.
We want to collect these items, the text from these as we see here.
And the instructor list, so if we go into our inspector as a first step, always great to check, we
can see the d'Hiv related information.
They are in a div with a class single path heading and we can go through to check the information within
the div.
Ah, right.
I mean, openness, spin class.
All right, we have a spin class here with the class name, we can go to the next one spin class, same
name.
Next one.
All right, so we see a pattern here.
We can use this spin class to retrieve these three items.
Let's go back to Spider and use beautiful suit for this similar process as we use in the previous section.
We can do let's use scrape.
Scrape one gionata, come in here, add passer, and now we can add our scrape information so we get
to scrape one equal to IDOT text, let's iterate through four I Insuk find no.
We want find all open brackets to pass in our spin class.
We're going to use the class here or excuse me, we're actually going to use the tag here and then we're
going to assign our class.
So we could pass any class information for this specific item, let's do a class with parentheses,
note class and we need to give it the class name, it was DSC.
So let's add that he says, see, closing brackets, closing parentheses.
OK, looks good.
So this is for our first item.
We have two more that we want to scrape.
So let's get this out of the way.
We're going to use the same process if possible.
Let's go with scrape two and let's take a look at the information.
Jumping back to the page where an inspector.
And let's take a look at these divs here, single path article.
Single Path article.
This is a key element for paragraph.
Within this single path article content.
And let's use that since we saw the Divx here, let's use the d'Hiv single path article content for
these to retrieve the text that should work.
So let's jump back into Spider.
We're going to use the same setup.
But this was AP Element four paragraph, and we can assign our class and the class name as single path
article.
Let me jump back.
Just want to double check something.
Single path article headings, single path article content.
OK, that's what it is.
Single page article Tashan content to use this specific selector we can go through with Beautifuls who
find all always really useful for scraping.
Lasley Jumping back to the page.
We want to retrieve the instructor information the instructors.
So have Carol and Hadland.
We can try to scrape their names.
Let's take a look.
OK, d'Hiv here.
Let me click on this to go through.
The inspector related information, a header class.
Div class.
Div class instructor list items.
Image content.
Let's open this.
OK, we have a PE class with a name that should be useful.
And we're going to try to scrape the information, let's go with scrape.
Three makes sense when you just grab this, always want to be able to reuse your code, make your life
a lot easier.
So Sufentanil final, we can.
Said it with a PE class and I actually this is a div class, we were not working with the PE, we were
working with a div class.
My apologies.
We are working with a PE for the instructors.
You start working with elements, you start seeing these elements while you're sleeping, all of the
element tags.
So we want a P here and we're working with the class name.
So let's do a class name and that should work.
But.
Let me double check, all right.
That looks good.
So we have our super information, we're giving ourselves some time after clicking that icon.
We're going to be able to retrieve that information, but let's try something when we retrieve it,
because we're going through and scraping and we're signing it to these scrape one scrape to and scrape
three.
But let's also apply it to a data frame just for reusability purposes or just to have something that
we could work with in the future.
It's an option.
You don't need it.
But this is an example to say you're trying to prepare a data set from your scrape.
So let's set our D.F. equal to PD DOT data frame.
And we could do scrape one and we could probably build a list to go through each.
But let's keep this spaghetti code simple and use D.F. Scrape three.
And I want to also don't want to resign the information to the same D.F. every time.
So we want D.F..
Do you have two and three?
We would have three separate data frames here, but I always want to check so I'm going to print MEDEF.
Let's try D.F. to A.F. three.
You try to print our three data frames.
Let's give ourselves some time.
Sleep again.
Let's give ourselves a little more time here just in case we want to perform any other operation.
Let's do time sleep.
And you know, I don't think we'll actually need that, but that's fine.
And let their page load, let our processes go through and then we can use the driver, quit to quit
out.
All right, so now we have our driver quit, let's remove this one, this is from the previous build
that we had that we had that through.
So let's do a little cleaning.
Let's add a little comment here.
Sign DFS and we can print data with our DFS.
We have our sleep and our quitte.
That's fantastic.
We might need to do a little cleaning, which I suspect.
But let's run this and see if we can retrieve our information to solve our challenge.
So let's run this and we should get the window open to go through everything that we have.
Here we go.
We have our first page up and loaded, giving ourselves some time to click on learning paths should
go.
There we go.
We have a learning paths.
It now should click on our engineer path.
Perfect.
If we had to pop up that we had previously, this should exit out of it.
There we go.
We have our pop up.
Assign ourselves more time for the content to load on the page to exit out.
And now our scraping should be running.
Giving ourselves more time and I'm seeing if we can go back in the spider, I'm waiting for this to
close.
But I do see here to the right of the pain, we can see some information printed from our data frames
and there we have it.
Awesome.
We have our three items scraped from the first.
We have our second data frame with the tax information and we have our instructor list.
That's great, fantastic work.
But one extra thing, and we have some species or some breaks that we can see.
And here we can use the following to strip the information.
We can apply to all three.
It's not necessary.
But I'm going to show you as an example how we can clean this up because this will be required to be
cleaned anyway.
If you're working with the data frame, you'll have to strip that text after we have driver quit.
Since we're going to exit the page, we can perform this data manipulation here.
I'm going to add this in and we can talk about it really quickly, just as an additional bonus we can
set.
And it's not necessary for the majority of these data frames that we assign.
We're only retrieving this right now with this text that we can see in the console within Spider.
These lines we can replace and you could specify a rejects.
I don't think it's necessary for this as well as as a parameter within our release.
But this is going to strip those and replace it and clean our data frame.
So let's run this really quickly.
We have our data frames, our data frame, one, two and three.
We're replacing them, assign them to a new data frame and then stacking them together just for visualization
purposes.
You can run these individually.
It's no problem, but let's run it after the fact so we don't have to run the entire script again and
we can run our selection or current line.
We have it.
And let's bring up the clean stack here.
And we could see that we stripped that information, we replaced it and we cleaned up that data, the
text.
Now we can drop or philes nans.
That's not important.
It's also, you know, just a general data frame for reference.
And I don't want you to focus on the specifics at this point, because the main point here is the scraping.
We're looking to go to the page to perform the clicks and to scrape those specifics, to retrieve the
text.
Awesome work.
I hope you guys have learned a lot from this.
I really hope that you have taken it on and tried to perform these operations before going the solution.
It's just a really helpful way to learn, if not if you work through this and your understanding information
as we go through it, that is excellent.
But overall, if you completed this and you have successfully run this algorithm to retrieve the information,
great job, fantastic work and you are scraping data.
All right.
With that, there's a quick little bonus I want to mention in the next video.
So let's wrap this up again.
Fantastic work and I'll see you there.