Hi, guys, welcome back.
Since we're getting started in our first practical and hands on session with scrapie, I want to build
a spider with you so we can see the general architecture of how these spiders are crawlers work.
We can define it in the following sections.
I mean, we can call this one, let's say section one, just like key takeaways here.
Section one, we want our import statement for the libraries we're working with now with scrapie.
So we'll have Section two with scrapie.
We can start our first spider by building a class.
We also have to pass in.
Our spider subclass, and this is going to help us to find specific attributes, and this is going to
include our name, which we're going to call books.
But first, let's define our first class as first spider, so we will set class.
First Spider, and we need our scrapie spider here, we want to capitalize.
Here we go.
Open that and we can give it a name.
Now, this name is important because this is a name that we're going to crawl with.
So let's call it to keep in the theme of things.
Let's call it books and we're calling it books because of the following you URL is our target.
This is what we're going to scrape.
We want to scrape the following page, the books to scrape dotcom, and this is actually from the website
to scrape dotcom and we can check it out.
It's a web scraping sandbox.
So this is where we can test out our web scrapers with scrapie.
It's a great place to practice and learn.
We're going to focus on the fictional bookstore and with scrapie and for our first spider, let's try
to you URLs.
So we want to use books to scrape.
And we also let's look for science, if you can find it.
OK, here we have it.
We'll grab that in a second.
Let's just jump back into Spider.
We want to set our name so we have our name.
And we also need to define our start, underscore you URLs.
And we're going to have this as a list.
Let's bring that open.
I want to close this.
And for our start, you Earles, we can actually just define are you Earles here and this list is going
to be used by the default implementation of START requests within scrapie and our requests are going
to go out and grab our objects.
So let's define our URLs here.
We want to grab to scrape.
Let me add this, we also want to grab the second as an example, because why do we want to just scrape
one when we can scrape to let's go to books to scrape and grab that we want to put our quotations on
here.
Can't forget those.
And those are the URLs.
Those are our target wells that we want to scrape.
May just fix this.
Now, since we have our new URLs, we want to define our pass method in this.
It's going to handle each request from the URL or your URLs that we have above.
So we can also set our page and filename along with using an option to write or files.
The idea in general to write our files is to write them.
We want to take unstructured data to turn it into structured data that we can work with.
Now, that's not important right now.
We're just working on our first spider.
So let's title this one, section three.
So you have an idea of three main sections.
And I also want you to think while we're working with this, what are we trying to scrape?
What is this scraper going to do?
Just keep that in mind.
Now with Section three.
We can define our pass, and this is going to handle each request from the URL or you URLs, and we
can also set our page and filename along with you, an option to write our files to test information.
Now, that's not really important right now of writing it.
We want to see this information return what we're scraping.
But in the future, we want to take unstructured data and we want to turn it into structured data because
that's going to help us work with this information.
So I want you to keep that in mind, what we're doing in this scraper here, and I'm going to paste
the following in so we can kind of dissect it.
And take a look what's going on.
All right, so as I mentioned, we have our pass method and we have to pass it ourself and response
arguments.
But we also want to use the page set to our response to our split since we're working with the two URLs
and we can use negative indexing with Python, but you can index in slices another way or have multiple
options of the index.
We also want to specify a file name along with adding the string or the formatting option here, and
we have to open it to our filename with the response body.
So we're taking the body of the response from our scraper and this is what we want to use.
But as I mentioned earlier, keep in mind what we're trying to scrape here with this project.
This is just an example, scraper or spider, and we can save this file.
What we want to do here is launch the terminal.
So as an example, let's launch it.
We need to go to our project and we're looking for the spiders folder.
So first, I know it's on my desktop, so I'm going to come to my desktop.
I want to get into I want to take a look into project.
And I have an issue with my Jaquie right now, so you may hear me slamming the key to see if I can get
it.
Project one, let's cut into that.
I want to check I think there's two practical ones the way I set up the folder.
So we want to cwd practical one, practical one.
And I want to go into spiders, into spiders.
And before I make an error, because I'm see working in a room, I have to activate source, activate
web script to work with our specific environment.
OK, so now ready.
This is where we're saving our first spider pi file within spiders.
If you haven't done that already.
Make sure you save it within the spiders.
And if you have to save it there, you can either change the working directory, you can drag and drop
to bring it in.
You want to save your file within these spiders directory.
Now, Scrapie has a command to crawl so we can use scrapie crawl.
And remember, we had books.
That's what we want to title or use for our crawler.
We want to scrapie crawl books and let's let it run.
Fantastic.
We have successfully executed our crawler, it has run, and we could find other useful information
here within the output from our crawler, such as seeing the robot's TSC request count the start time
and other helpful insights.
But I want to take a look in the folder that we just ran.
So we have the following folder.
We can go into Practical One, and this is where we wrote our files to and we can open them.
Remember when I asked to keep in mind what we're trying to scrape here?
So let's open it.
Let's go with the following.
We have the entire page returned that we were trying to scrape, and that's because if we go back into
our spider, we didn't specify anything, any variables, any selectors, Elspeth's, anything like
that to scrape.
We scraped the entire page with our request.
That's what the response is returning.
We're scraping the following you earles.
It's useful because we can actually see what we're working with.
We have successfully tested our spider and we can go through and analyze the information that we have
scraped.
But on the other side we need more structured data.
We can also see that it's not returning images because we're not connected.
We don't have those file parts.
We don't have the access, the images connected to our HTML file right now, which is why it's not displaying
the books, as we can see on the regular pages.
But the main takeaway here is that we're able to scrape the entire page.
So we're pulling data from that page.
And if we jump back into our spider, we have the following three sections.
So in the first section, pretty straightforward.
We need our import statements.
The second section, we have to create a class with scrapie, give it the class name to pass in our
name and our target for the script.
So we want to start you earles here.
We need to assign it a name because that's what we're going to use to crawl.
And then our final section, we have to pass the response.
And if we're writing it to a file for writing it to a CSV or anything like that, this is where it's
going to help structure the data to write the response.
And in the future, we're going to write the response or write the files in the terminal to satisfy
output, but the main take away the structure, the three sections here of a main spider.
Fantastic.
You guys are awesome.
I hope that this helps provide a little bit of insight in how we're going to approach building spiders
with scrapie.
And I will see you in the next video.