Hi, everyone, and welcome back.
In the last lecture we went through and built our first spider, we became acquainted with the general
structure of a spider setup.
But what if we ran into a bug or what if we wanted to just test something with scrapie?
And this is where the scrapie shell comes into play.
This is an interactive shell that helps with debugging without needing to run or build a spider.
So it's really useful for testing code, for testing variables such as XPath.
There's a CSFs expressions off a page to see how they're pulled.
Overall, it allows us to interactively test.
So it's definitely an efficient tool to keep in mind.
So let's try it out.
We can launch a terminal for the scrapie show we want to remember, activate your environment source,
activate web screen and let's scrape the page that we're working with.
So let's use books to scrape dotcom, OK, and go back, bring the terminal here and we can run the
scrapie show with the following command.
All we have to do once our environment is activated is run scrapie show and pass in the URL.
So I want to scrape this page.
I'm going to grab that URL and paste it in here and we're going to run it.
And there we have it and we can actually see the information return, such as our scrapie, Kalila item
request, response and settings, this is really useful to see the available scrapie objects.
And the really cool thing is we can actually, instead of passing in the early, we can work with a
local file as well.
But for the purpose of this, we're going to be working with an item from the page.
As I continue to mention, we want to become more specific with the information items that we grab since
we're in the scrapie shell.
Let's test the features of it and see if we can return a specific object from the page and a quick way
of doing this.
Let's try and grab this right here, all products.
So let's go to the inspector within Chrome.
Let's highlight this.
Let's find all products.
But I want to go back, OK?
We have all products right here as an H1, so it's a heading one tag.
And I'm going to do to copy and copy the XPath.
OK, I have the XPath copied.
I want to go back into the scrapie shell.
You can see we're actually in the shell right now and I can run the response command response XPath
since that's what we grabbed we can pass it in.
Just remember to put the quotation here, let's pass it in.
We need to also close it, close our parentheses and we can run the dot, get command.
And this should retrieve exactly what is in each one that we want the all product.
So let's run it.
And there we have it, we have H1 all products.
So that's exactly what we wanted to return.
I highly recommend that you go through, try to grab other objects.
It's a really useful tool to experiment with, to test around, to start playing with the objects on
the page.
Fantastic.
This is the scrapie shell.
So if you grab this information, you can use it to incorporate it into the spider and we're going to
advance to the next lecture.
Great work and see you there.