Welcome back, everyone.
Let's jump right into our challenge solution so we can see how we want to scrape the review accounts
from Amazon for those products, first thing we're going to do, we have our libraries imported.
We have requests, beautiful soup, pandas and time.
So we can start let's set up a class because this is going to be useful.
And I'll explain why when we get done with the end of our challenge solution.
So let's use the following.
Let's define a name for a class.
Let's call it define what do we want to call it.
Let's call it review, count, scrape.
Essentially what we're looking to do here.
And the first thing that we're going to do is pass in the early.
So we want to use our URL equal to.
Let me grab that real quick.
All right.
I have the arrow.
I can go back, I can piece it in, and I'm setting the arrow like this because I'm going to pass in
this variable as an argument with our quest get when we send out the request.
And I mentioned earlier how we want to use headers.
Now, the purpose of this is essentially to emulate a browser.
When we send the GIT request with requests to Amazon, we wanted to look like a browser because we don't
want to run into any issues.
We don't want to set off any flags within the server because this can result in our scraping requests
being denied or blocked.
We are using a browser and we want to emulate that browser and this is why we're going to be using a
header so we can define our header as a following headers equal to.
And let's open this to pass in a dictionary in a second and I'm going to grab the information and we
can see if we visit the following.
Link, we can see a common user agent list, and as I said, we're we're using commands such as get
we can pass in these headers and we're going to be using the Google Chrome one.
We'll use this.
And this specifies the details that we want to emulate, that browser that we're going to be using for
our crawler with beautiful.
So I have grabbed the information that I want to use.
Since we have our headers, we could now send Arget request.
We're going to set are equal to requests, douget, and this is where we pass in our URL and we get
to our headers equal to headers.
It's going to pass in our information to emulate our writers.
Next, we can retrieve the beautiful text we want to pass in our parser as well as you guys have done
already.
A couple of times we can set equal to beautiful soup of text and to pass in our parser.
Elex Emelle, if you're using a different parser, this is where you would use it there.
Fantastic.
And this should look familiar, reusability in code helps big time so we can now print always is a good
option.
Let's print for our I mean, we want to check a look at the status code.
We always want to make sure this status code is correct if we're going to be returning that two hundred
when we send out our request.
And now it's jumped to the page.
Let me go to Amazon best sellers.
We want to grab the totals for each product just to check.
Maybe there's a reason why these products are ranking so high.
Maybe they have a ton of reviews.
This is going to help in any type of analysis.
That's not important for us.
We just want to grab the total to add into a data frame, because this is going to essentially help
down the line.
If you're a data scientist, you're looking to build a data set with information.
This information is going to be useful.
So we want to grab this and we can select it and we can see that the class is going to be assigned.
As a size, small and normal, and we can see it's an element, let me just go over here.
We can see the class sign as well here with an ailment just to double check.
Let's jump back into Spider.
You can call it product total review.
All right, here is where we can get a little more detailed in the sense of we want to iterate, we
want to find the text for all the A's on the page with the class that we specified.
And a useful little trick is we can actually iterate through with a for loop.
We can go through our final.
So let's set it as I go about the text for I for I in soup, Don find we don't want a capital F..
We want lowercase f find all as we've done before, we can specify the element that we want to use.
And we can pass in our class as a DECT here so we can set our class.
Equal to.
A, as we said before, was size small, I think it was, and link normal, let me check it was let's
pull it up.
Yeah, aici small, ayling normal.
All right.
We jump back in here.
So a size small.
And a link normal, since we're passing in both those classes that were assigned, we can just give
us selves a space here and I want to make sure this is closed up correctly.
We have that good hour right now.
We also want to assign it to a data frame.
So let's call our data frame simple enough, D.F. equal peaty dot data frame, and we would pass in
the name we want product total review to create our data frame here.
We can also print it.
Always good to see.
So let's print our def.
And that will bring us to the end of it now, I'm not going to run it just yet because I have time imported
and because we're working with a scraper or a page that is updated every hour.
Ideally, you're not just going to want to scrape this one time.
You're going to want over a period of time.
This is can help for any type of analysis.
So in order to do so, time is really useful for the purpose of this.
We're only going to scrape it twice in a specific amount of time just because, as I have said, always
want to be respectful.
Don't overload the Amazon page with requests is just an easy way to have an issue with your IP or have
the crawler blocked.
So we have RDAF here.
I'm going to give myself some space just so I can add a comment, add timer for a reference and we can
use time sleep.
This is going to crawl every 60 seconds, but we also don't want to just have this and let it sit because
we need to specify how we can end our timer.
We're out and we can specify this.
Let's call it end timer.
And we can set this two time that time, and let's give it an interval to run.
Every six, we want to run for two minutes as we want this to have two minutes every 60 seconds, but
we also need this to break once the timer is done, once it runs those two times so we can use while
time, that time is less then and underscore timer and for the purpose of creating our function here.
And I think I actually called it a class earlier.
My apologies.
I definitely need some more coffee today.
We created our function and this is just useful.
Now, we can actually if we just grab this and we all execute these lines of code, that's fine.
You can run your scraper.
But if we want to use a timer and we want to invoke this, we want to have our function run every so
often, it's always helpful if you create the function for it.
So this is why we created the name of our function so we can specify the review, count, scrape to
run this function and we're going to run it and hopefully we're going to run it now two times.
It's going to run every 60 seconds.
And let's do one more quick check of our code when you sometimes grab this information and pass it and
it's easy to make a small syntax error.
So let's double check the spaghetti code that I have in here and we can see you Earles.
Good.
Actually, that's fine.
We have that passed in request.
I get we have are your headers.
So is good status code.
We can.
Oh Mike Key loves to give me those double spaces and we also have this.
Yeah I definitely need some coffee today that is for sure.
I may add this and let me go back a size small ayling normal.
Let's jump back double check the chrome inspector and we can find our class.
Let's select this.
OK, aici small, a link normal, we jump back into spider.
So small a link yet that will cause an error as well, D.F., our data frame print.
All right.
Only one way to find out if our solution is correct.
Let's run it.
Nancy, easiest way to debug, find out you have an error, run the code, we remove the camera.
I might have removed the comma right then, so let me add that and run it one more time.
And there we have it, we have successfully let's check our status code first is our two hundred.
So it's a success.
But looking really quickly, I see the reviews assigned to the product's fantastic.
So we're grabbing the reviews from the page and we're using the total to account in our data frame.
So maybe WASC nine thousand one hundred fifty.
Let's go back to Chrome nine thousand one hundred forty eight.
So reviews might have been applied six six one six six one three hundred and sixty three thousand is
a lot of reviews, three hundred sixty three thousand.
So we could see that these reviews are changing when we're scraping this information.
It's updated every hour.
Let me jump back one more time.
This one does not have reviews.
OK, so that's an interesting note.
Any of you looking to customize this further?
I would recommend air handling so you could use it, try and accept in Python.
If it's not displaying a specific value or a specific count, maybe you want to mark that as a zero
or market as any type of numerical value.
So you have a reference, OK, this product does not have any reviews assigned to it, but for the most
part, we have all these products with reviews and it could help with any analysis.
The takeaway here is that you were able to define a function, pass in the URL, the information you
want.
You emulated a browser you grabbed in past the HTML document, retrieved it into a data frame, and
with the help of our timer, we ran it twice.
And we can see as I was talking here.
That the timer ended successfully, ran two times, we have our data frame and we return it, it's executed,
it stopped.
If you did not add the timer, if, say, you had a while loop with the function iterating, it might
just continuously iterate without breaking.
You do not want that because that's going to continuously scrape, continuously run use CPU resources.
You don't want that.
And one more quick note, just because I'm going to definitely grab some coffee after this.
I originally had my timer in when I was setting this up.
I had my timer in.
This is just going to daytime.
Not sleep is just going to essentially what it implies.
It's going to sleep.
So it's not run.
But if you have the timer, we can break, we can have the loop end instead of having it run continuously.
So we have these options.
If you are looking to deploy it and say change 60 to however many seconds you want to have it sleep
for before it runs, or if you want to specify the timer to run multiple times.
So say, you know, every 60 seconds you want it to run twice, but you can change these.
You don't want it to run too often.
If it's a daily scrape, you would want to imply that this is just for our practical purpose.
But overall, fantastic job.
I hope you guys have enjoyed this challenge.
If you put together any additional information, any other information that you scraped within your
beautiful shoe project and want to share it, please feel free to share in the Q&A.
And I would be more than happy to take a look in the next section.
We're going to be working with selenium.
Hope you guys are excited and I will see you there.