Hi, everyone, welcome back.
As we're going through our lectures and intuition, some of you may be wondering, how do I know that
I can scrape this site?
How is it possible?
Is there a method for us to know if it's acceptable or we have permission and that's when we get into
robots?
But before we get into that, there's actually been a recent court case, or you could say the reason
this happened in 2013 that I would like to show you guys to provide some further context, at least
related to publicly available data.
So let's jump to that real quickly and I'll also add the link to it in the resources of this lecture.
All right, so you have the example article I was referring to here and the main takeaway, we're not
going to get into the details of this article, but I want you to see the following highlighted section
in 2013.
This was a historic moment related to data privacy and data regulation because it's showing that publicly
available and not copyrighted data is, as we see here, fair game for web crawlers.
But commercial use of scrape data is still limited.
OK, so now that's a generalization, but can we obtain more details on seeing if a website can be scraped
or any other information?
You know, is there a request limiting?
And these are things we have to think about.
If we're requesting every time we want to scrape that data, we're going to send that request to the
website.
They may limit it because they don't want their services to be interrupted or spammed with these crawlers.
So there may be, you know, delays or rate limits, et cetera.
But as a general rule, we can look at the robots Stotsky.
Now, what does a robot TSC do summed up, it's going to basically tell our crawlers you're going to
hear me say crawlers or spiders, they can be referred to as scrapie.
It's going to be a spider, but it's going to tell crawlers which pager files that they can't or can't
request from a specific website.
And again, it's mainly to avoid, as we see, overloading your site with requests helps to manage the
traffic overall on a site.
How can we access the robots to see pretty simple, actually, we could do the following.
We can take you, Earl, and you'll see already have a here, this is what's going to come up and you
can see the Earl.
So let's take Google as an example.
Let's go to Google dotcom robots dot texte.
And this would be the example of our domain name, Google.
Any domain you could check if they have the robots text file.
So we will go to robots, see, and this is where you'll see the allowed or disallowed by the user agent,
user agent would be the scraper.
And you can actually see different user agents.
And we have an example.
OK, let's scroll down and we see Google OK, allowed with the following URL for finance.
This is something that will be allowed to be scraped specified in the robot file.
So this is something you can always consult.
And if we scroll down and I apologize if we're scrolling down a little too fast, but I'm looking for
a break in the page just as an example of a different spider or crawler.
And we can see here this is asport.
So this is specifying the disallowed user agent for ASBA on Google.
All right.
But can we take another example for a comparison?
Let's go to Apple.
Robots deoxy and we can see the user agent defined here and we could scroll down slightly.
And we can also see as another user user agent, the Baidu spider.
So overall, you can find if we're talking about using something such as like scrapie or beautiful soup,
we can see it defined in the user agent, the disallowed or allowed services to be scraped.
If you're using any other type of crawler crawler service, you're going to see a defined here if it's
going to be allowed or prevented.
So we can see, again, the bitou spider and we can keep scrolling down and see if there's any others.
OK, so we have another one.
The how spider, if I'm pronouncing that correctly and two more spider is defined down here.
Three more actually.
All right, so let's go back up just a quick example of how you can check or access a site's robot's
text file.
It's a really simple and efficient way of checking.
If you can scrape a specific page on the site and obtain any other information, they may help you understand
the related permissions for crawlers or scrapers art.
So let's wrap things up so you in the next lecture.