Hi, everyone, welcome back to our beautiful soup section.
In this video, we're going to be taking a look at our practical challenge set up for beautiful soup.
And we're going to be scraping a very relevant website, a website that has a lot of information pulled
from it, quite often due to its relevance.
And that website is let me put it up, the Amazon.com best sellers.
You can see the URL right here.
First thing I want to know my little legal note.
This project is intended for educational purposes only.
We're only going to scrape this data to see how we can use beautiful soup to work.
In addition, if you take a look at this practical challenge, if you're working on it without seeing
the solution, I will give you the following hint.
Please include headers in your scrape, in your beautiful soup request.
And please do not spam or send a lot of requests because I don't want your IP address to have any issues
or any type of service issue when you're requesting it for Amazon, for Amazon to block or have an issue
with the crawl rate.
Always be respectful of the websites that we're trying to scrape.
Always check the robot's text file as well for guidelines.
And you can see I took a look right here.
I wanted to take a look at best sellers, not beautiful soup.
I want to take a look at the page that we're looking for, the best sellers and I do not see it included
in their robot file.
But always be respectful when you're crawling, do not spam the service, because that always raises
the possibility of a possible ban or a possible suspension.
VIP visiting the website.
These are things you have to take into consideration when you're calling.
OK, let's jump back into Spider just so we can talk about it really quickly.
And we're going to be using the following.
I'm not going to give you too many hints, but we're going to using the following libraries, the requests,
soup, pandas and time.
Time is optional, but you'll see why it makes sense to include that.
And if we go back, the purpose of this and the information that you're trying to retrieve in the practical
challenge, you want to retrieve the number of reviews for the products on the page.
That's the information that we're looking to scrape the number, because maybe there is a correlation,
maybe there's some type of relevance between the number of views and the Amazon best sellers list or
making it to the best sellers list.
Also, this information is updated hourly, so please keep that in mind when we go through the solution.
The information might be different because when I'm putting this together, when you're viewing it,
it's going to be on a different date and this can change.
Just to recap, we're looking to scrape the information, the this information here, the amount of
reviews per product, please do not spam Amazon with requests when you're trying to scrape information
for this project.
This project is for educational purposes only.
And one more last thing with the information we're looking for here with the number of reviews, I would
like you in this challenge to put it into a data frame.
If you can get the results and store in a data frame, that would just be one additional benefit to
this project.
Fantastic.
Let's wrap it up and I'll see you in the challenge solution.