Hi, everyone.
In the last lecture, we left off working with our second spider where we built an item pipeline to
find items to scrape specific information, but when we return that information, we could see that
we left some of the elements or the tags that were.
In the XPath or in the helmet with our information, if we pull that up, we can see that we had the
class assigned here along with the H1 tag, and we return the information.
This is what we were looking for.
And I mentioned being able to do this later in the course, but I also wanted to touch on it now since,
yes, we can do it with beautiful soup.
I think it'll make more sense when we're working with beautiful soup.
But here in scrapie, we're able to do this in an efficient manner.
And I also want to to introduce a useful method of obtaining this information, working with the XPath.
So when we're working with the XPath, we grab the information from the inspector, which we can pull
up.
For example, we were scraping the information from Sapience and we can copy the XPath.
But in order to retrieve those specific information, we really can traverse the HTML document to select
the specific elements, nodes, children of elements within the H.M. document, parent elements, things
like that.
And to help clarify this, we can take a look at this is for relationship of knowledge with the parent
and children that I was mentioning.
If we think of the HTML as a document or a tree that we can traverse and walk down, we can see the
elements and how we can start selecting them.
We can see some syntax.
And this is for XML.
But just to give you an idea of selecting nodes within the document, we can get some information from
the following page and I'll include these resources in the resources for the lecture.
As I mentioned earlier, we can walk the DOM in a sense where we can traverse down it and really get
to the specifics to extract text from a document.
Overall, I want you to think about and HTML as a tree.
Where we can go through and select the elements that we want to scrape with the document.
All right, from an intuitive level that is helpful, how can we implement this and discover what specific
element that we want to scrape?
We can go back to Spider and more specifically, we can go back into our terminal.
All right, we're back in our terminal and we can use the command scrapie, shell, scrapie, shell.
And let's go with the destination.
Formal request that we want to scrape is here.
We want to use the same one before just to try and extract that specific piece of information.
So let's run the scrapie shell.
Let's grab this.
We are HTTP request.
Let's run this great beashel.
Once it's activated and ready, we can use the following command.
We can use vetch and we need to pass in our.
E-mail one more time, but we need to remember to add the quotation mark and parentheses.
So let's run our fetch.
We can view to test it, let's view our response.
And this should open our HTTP request to the page response.
Now, if we go back in to our scrapie show, we can do the following, if we use the in print response
text, we can print the HTML text or the response text that was retrieved on our request and we could
run that.
And this is just a method of finding elements within the HTML document as we're talking about traversing
or walking the dog, walking the document, working through the tree.
I'm trying to find the top of the document, so I'm just scrolling slowly.
I'm going to grab this.
Copy it and bring it up in a text editor, so I'm just going to bring it up in sublime light weight,
you can use Spider as well, but let's bring it up in Sublime.
I want to open a new window.
And we can piece this in if we jump back to help speed up the process, we can see we're looking for
sapiens.
OK, let's try and grab this or is 54 23.
Let's jump back into our sublime.
Let's search for this.
And we can find the class, the price color.
We can see it's coming from the div tag here and we have our H1 heading.
And this is a way to find those specific information.
Now, you can also go into the inspector and take a look, hover over it, select it, and we can see
the price color as well right here.
The thing is, when we grab this XPath, let's copy.
XPath.
And I'm going to go back to Sublime to show you we could just add it here just for an element we can
we're able to see the d'Hiv and the child elements within the specifying.
The HTML elements are div tag and RP tag for paragraph.
But we can use this class as a method of selecting the text within this.
And by doing so, I'm going to swap this out since we just want to test this price here.
I'm going to comment this out as well and I'm going to add in the following.
This is going to return the specific information from that because we're working with the P or paragraph
element and we found our class.
We can use this to identify the class, the price color class matching this tag.
And we want to print out the text from this.
We have to use the get come in.
So let's see this jump back into our terminal.
I'm going to clear I want to exit out of the scrapie shell.
I need to seed back into our folder.
We can run the command scrapie, grall books to output test.
Let's go a test price CSB.
OK, let's let it run.
We need to jump into our folder now, I have the test price open, we can see that we have the specific
price defined.
We don't have any elements, anything included other than our price.
We have the total here just because we have the title defined in our items, shapefile, so we can remove
it.
But just think about the implications of this.
When you're working with trying to create data frames or data sets, this is going to make your life
a lot easier if you can return that specific information.
So you have the two methods you can take, the more albeit it might be involved this way.
But I think this is a good approach to really see what the HTML structure is made out of on a larger
scale or being able to search within it.
You can go into the inspector and search.
That's absolutely fine.
But when you're working in this window, it might be a little cumbersome sometimes, or maybe you want
to test out the HTML with something.
You want to examine the document, get familiar with the terminal, be able to fetch the request and
then view and print out the response or just some commands that help you debug and find these specifics
within the HTML page that we're looking to scrape.
Awesome.
Hope you found that helpful.
And I'll see you in the next video.