Welcome back, everyone, welcome to our Challenge Solution.
In the previous video, we set up our challenge or what we want to scrape, and this is my approach
to scraping it so we can get started.
The first thing we want to do is set up a new file.
So I'm going to say it in our Spiders in our project folder.
Let's go with practical one solution, DCPI We could save the file and now we're ready to get started.
Similar to what we've already covered.
We need to add our import statements.
We need to import scrapie.
And in addition to that, the challenge was to scrape pieces of information, including the category
and in stock where we didn't already have these defined in the items file.
In addition, since we're working with items and we're also working with new items, we will need to
import the items final and we can use the same one that we already imported.
So we'll import it with from practical one items.
And I have to remember my capitalizations on these because I did make a mistake earlier.
We want to import practical one item and sometimes I do like to leave these in.
Just so you see that these mistakes happen, you can remind yourself all the time.
But these mistakes happen when you're coding.
So we have our import statements.
The main first step with scrapie just to hammer down on this is to set up our scrapie class, we need
this to crawl to set the name and start URL's.
So let's define it.
Let's go with class.
Let's name it solution spider for our challenge solution pass in our arguments.
We want scrapie.
Dot Spider.
Just remember the capitalization on Spider.
We have to sign in a name always one of the key parts of this assignment, a name let's call it solution
to many quotes there.
We can return this here.
We can define our start earles.
We can also define this isn't necessary right now.
I'm at a comment here just so it's a reference.
We can use allowed domains and this is to help limit or to specify specific domains that we can crawl.
We can always think and this might come into play in the future.
We can always think.
We have to rate limit, we have to only kraul specific domains that were allowed.
Sometimes our caller, depending on the context, might go through and iterate through specific pages.
We only want to be working with those allowed domains just so we don't break any rules.
We want to stay and abide by the terms for scraping.
But we can have this.
This is just for help to reference for allowed domains so we can also use our start URLs or start URL,
start underscore or else we need to open our list.
And this is we're going to pass in our destination for our spider to crawl.
In the previous challenge, I asked you to pick out four books.
So we're back on the page and we can grab four random books.
And let's take a look through here.
Let's go to our science page.
Go down, I'm going to grab.
These two, let's let's grab the grand design origin of Species and let's also switch it off and get
a different category.
Let's go to travel.
We want to compare and contrast some book prices and some different genres here.
And let's also grab.
OK, this one under the Tuscan Sun, I believe I've heard that before.
Sounds great.
And also, let's grab let's grab this one a year in province.
And if we have any French speakers, I would love if you could correct me on the correct pronunciation
on that.
Much appreciated.
Let's grab those.
I'm going to grab the four links from each.
So I'm just we need to grab the link from each book and I'm going to piece it into our start URL's.
OK, I have these in I want to make sure that we have our commas after each one, these are the words
from each of the books that I chose once we have our start URLs for our scrapie, spider to crawl through,
the next thing that we have to do is to define our pass.
This is going to pass our response.
This is going to go through the information of the response and return the information that we want.
So we have to define our parts in addition to adding my keyboard also to give me double spaces randomly
at times.
We want to also give ourself response arguments here, open this up so we can pass in our item information.
Since we're working with item practical one, we have to define it.
So let's set it to practical one item.
Remember too, we can use this here to double check.
We want to make sure that we're using the correct capitalization.
Let's also scroll down here.
And now this is where we have to do a little customization besides the URLs.
What we have to do here is first to define two new items that we're working with.
We work with the price and title previously.
We also want to add the category and in stock.
So if we visit our items that py file, all right, now we're in our items that py file and we can see
that we're keeping it simple.
It's the same process.
We just have to define two new fields which are the category and the in stock.
This is going to work when we're sending our items.
Let's jump back into Spider.
And let's set our item category.
Item, open brackets category.
Equals to.
Response XPath of the information we want to scrape, and this is where we need to pass in the specifics
to retrieve that information, if we visit the page, let's use the following you URL an example.
We'll jump back to this or we can actually use this one.
Let's open the inspector.
We're looking to scrape the category of the book.
This would be travel so we can see here.
We can click on that, it helps bring it up the reference, so the first thing here we have we're already
seeing that the class assigned to it is breadcrumb, class breadcrumb.
And it's also within an unordered list.
We also want to see let's take a look we have home.
Books, and we're looking for that.
There it is, it's already open travel.
So we're working with an unordered list, and this is also the third item in the list, this would be
almost the third child element within the list, and we want to scrape the text from that.
In order to do so in our XPath, we can use the following, let's set the quotation marks here open.
We have them ready, we can pass in our unordered list, as we just discussed, but we also have to
pass in that class reference equal to breadcrumb.
We have to close this for the closing tag of our list.
Now we have to specify that it is a list item, but it's that third one, remember, we were working
with the third element down, the third list item down.
And finally, we have to scrap the text off of that to add the closing tags on this so we can return
that information.
Don't forget the last.
We already have that in to double check.
We have our quotation marks.
For our XPath here and the final thing that we'd have to do to scrape this is to call the get method.
Now I hope I passes incorrectly.
We're going to find out when we scrape it.
And if there's any errors, we'll come back and correct the XPath.
So we have our category since we already have the price information, not the title.
Since we only used the XPath for that.
I'm going to paste in the price path for the item here just to have that specific, because if we use
the xpac, remember originally when we use it, we passed in the information that was being scraped,
the element tags along with it.
We want the specifics.
So we have our category and our price.
We want to do title.
Let's take on title next.
So same setup item title equal to response XPath Open Quotations.
Now we're ready to pass in our XPath to get started.
Let's jump back to the page.
We can use the inspector again, we want the Under the Tuscan Sun, so let's take a look real quick.
All right.
Again, we have the class here, calm, small six and also product mean.
Keep that in mind because we're working with these two classes.
And it's in a div tag.
This one will be pretty straightforward.
I mean, the last one wasn't overly complicated, but we're looking to read the text from each one.
So let's jump back to our spider.
We can pass in this XPath.
As our d'Hiv we're looking to scrape, we need to add both classes, that's the key thing here.
Class will be equal to.
Column Small six and also product underscore, I mean.
Let's close this.
It was in an H1 here, so we want to see each one can have those closing tags yet and we want to scrape
the text from it.
Text.
To close it with our double quotations here, to wrap that up and cannot forget the get method to pull
the information, the last one, we have to do same process.
We want to check if the item is in stock.
So item.
In stock, in underscore stock equals response XPath, you can grab and just delete the information
from the other ones to pass in the new information.
Let's go back here.
Oh, as I just said, let's make this simple.
Always looking to keep things simple.
We need to just passing in stock and to delete this.
And we're ready to start searching our XPath for the information.
All right, let's check the instant jump back to the page we have in the center here.
And we can see when we open this up, we have our class, the instant availability for our tag.
We can use this to walk down the element.
So let's jump back in the spider.
And one tricky thing here, this actually took a little bit of Googling just to get the right information
to display is the use of normalized space.
Normalized space is a method used to normalize the string to remove any leading or trailing white spaces,
one related to the XPath.
So we want to use the normalized space here.
And now I'm just going to clear this, because it's going to get a little detailed, we have our open
our normalized space since we passed that into strip that information.
We also need to start with our tag.
So similar set up.
We just need to set this up first with our normalized space.
We also cannot forget that it's normalized space needs to have that added in and we can pass in.
We have our class we already saw was the in stock availability.
So let's set our class P.
At class equals in stock.
Availability, closing tag on this closing quotation, excuse me, single quotes, let's close this.
We want to walk down the tree through the items, but for this one for a reference, let's jump back
into there to take a look.
We're right here.
OK, and since this has an icon may jump back in here.
Let's try after the icon to scrape the following node to get the element from it, we want that.
And I have too many quotations, so I need to make sure I have everything correct.
We also cannot forget.
We need our closing quotes, closing parentheses, and Donkin, let me double check this to make sure
I have everything correct, we have in stock normalized space, PE class in stock availability, OK.
Last thing we also want to do, we cannot forget is to return our item once we have our item returned.
Let me just go through this one more time to make sure since this can get pretty busy, if we're trying
to disseminate this information in here, let me spell this out so I can read through them one more
time, make sure we double star, OK?
Yeah, we need to add the following in our class here to start our unordered list.
We also my apologies.
We have to pass in one more quotation marks passing for the breadcrumb.
OK, that one looks good.
Let's double check price price looks good.
Correct quotations.
We have our git check our title.
Same thing using our div.
Person to class is here with our one and last in stock, XPath out with normalized space.
We don't need these since we're starting the pétanque here, we can pass in our class and stock availability.
Correct rotations.
All right.
I'm going to leave these spaced out just for us to run them to make it easier to search through if there's
any error.
Let me bring that back and let's launch a terminal.
The easiest way to find out if we did is correct is to run it.
And let's see what happens now in my terminal already keyed into this by your location so we can run
scrapie.
Crocombe in scrapie, Kerl solution and we want to have an output.
Let's do a challenge.
Challenge, underscore, solution, CSB, and let's run it.
And there we have it, we have successfully executed our spider.
We can check the information to see what was returned in our our spider and it looks good so far.
Let's check in.
We can see, Carol, from this page, we have our category in stock price and title, but we want to
check Garcia's fee just to triple check and be sure if we have the information that we want.
And if I jump into this, ESV, we can see we have our comma separated values, science in Stock five
available The Price and the title of the book.
I can also open another program on Macs, since I think this is a little too easier to visualize compared
to sublime text.
But this is exactly what we want.
We have our challenge solution.
So we have our category in stock.
How many available?
That's what we're retrieving our price and the title.
And the great thing is that we don't have any of those selectors, any of the XPath, that information
that is also included.
Sometimes if you use that relative XPath to it, we have a clean kesby that we're able to use for other
purposes.
Fantastic.
Awesome work, guys.
Great job.
Time to grab a coffee.
Take a quick break and I will see you in the next video.