Welcome back, guys, in this video, we're going to be working with some selenium basics.
I think this will be fairly straightforward since we've been through scrapie and beautiful soup, we
can start to see a pattern emerging of how to work with elements within the Diamond browser and the
HTML document.
As a recap, we're going to be using selenium.
Selenium is an automation tool that can be used for web scraping to run an automated browser, grab
information, click links or buttons, pass keys or passwords and much more.
Since we're going to be working with selenium, let's run or talk about some quick tests with selenium.
We've already seen how to pass in the Web driver here and to run our command to get to send that HDB
request to assign a sleep value to delay.
And then we always want to quit out.
You don't have to, but this will keep the Web driver and the browser open.
It's always good when you're done running your scrape.
You always want to sign that quit so we can give ourselves some space for now, just as an example.
And if we see the selenium documentation, just for a quick reference, these are some public methods,
some built in methods where it takes a lot of the work of finding or iterating through the documents.
We can find elements by using the specific paths if we want to find a singular element or if we want
to find multiple elements, we can use the XPath as we've done before.
We can find the class name.
These are methods to be able to grab that specific information from the military, from the structure
to retrieve it.
All right, so we're back in Spider, and we want to test out an action as an example now we can go
to the following page and think about what we can do with selenium.
So we see that there's buttons on the page right now.
And let's use the join us as an option.
So if we go into the inspect and let's click on the join us.
We can copy as an easiest method, you can use class names of selectors and the other options to find
element buy, we can also use beautiful soup as well.
But let's use it XPath to really keep it simple.
If we go back into Spider, we want to pass in our URL that we want to scrape.
So http cds club dot com and now we can use the options from selenium to driver find element by XPath.
Pretty straightforward option here.
We want to assign our quotations for our XPath and we want to add our XPath in here just to make sure
quotations are good.
And then we can actually add an action as dot click and we will be able to click on that button if our
XPath information is correct so we can try it out.
But one more thing I want to do.
I want to give us more time.
So I'm going to assign time sleep.
Let's give ourselves five more seconds here so it doesn't come out right away just for the purpose of
our test.
So let's run this.
Hopefully there's no error thrown and we're good, and now it should come up in five seconds if you
click on the join us button.
And there you have it, we've performed an action with selenium, we're automating this software, this
browser, and there we have a quit out.
So that's perfect.
And that's an example of performing action.
Now, normally, what you would want to do, why selenium can be important is think about pages that
you have to log in to get access to information that maybe it's an internal page that your company's
working with, or maybe the website has restrictions where you need an account to log in and you want
to scrape.
Remember to always check the robots to see, but by doing so, you can click on the sign up button and
then you could pass in your password and keys.
I will add an option here for them.
We're not going to work on the password keys just because I don't want any possibility of sharing.
Sensitive information, if you deploy this to GitHub, just always remember to check your credentials,
we don't want to share them, but it's as simple as if you use find element Biso say it's a username
element by or the class spy.
You can use the driver to find element by ID and then you would you would look for the username.
You could also look for the option if it had a password as well.
Let's so let's close that.
And you could put password as an option.
So you'd be looking for the elements here.
Maybe the class name you would just remember have to check the e-mail structure in order to send the
information.
You could use username, send keys, and then you would hear you'd want to send your username.
And just reusable code here, you would want to send your password to make sense.
So this is a method of automation.
In addition, in the last video, we talked about dynamic pages, most pages now, especially when you're
dealing with things such as e-commerce forums, things like that, you will see the options or they
tend to see options with load more, view more, etc..
As an example, we can see here, if I go into this selector and don't pay attention to the website
or the review, that's not important.
The key concept here is the dynamic content that's generated so we can select this review and if we
go into the spin, we can see the text that's generated from the review.
But if we go back out, we can see the read more option because there's further text available.
Now, we'd be able to use selenium in this instance to click on the read more to generate that text
so we would be able to scrape it.
And this is a very simple explanation.
But we can see this applied to a range of options throughout the Web.
This is something we need to keep in mind.
Also, as another example, say we visit this page in a pop up comes up.
This happens to us all the time.
We would be able to use selenium to exit out of that pop up so we can then focus on scraping the text
on the page.
And that's a powerful thing that we are able to perform with selenium.
Now, this was a really simple explanation of selenium, just to show you what we can do in the sense
of mimicking actions.
Ideally, when you're working with selenium, you will want to pass.
You want to add we have beautiful soup here, have it, use it.
But we want to add beautiful soup to pass the HTML document.
Along with using selenium, so if we think about it, we'd want to click to have that content come up
and then we want to pass information to work through it.
So in the next lecture, we're going to go through our challenge introduction, I'm not going to say
too much more because I don't want to give you much more of a hint.
I think working on this hands on is the best way as you're working through these lectures to be coding
to help reinforce the information.
But we're going to be trying to solve a similar challenge in the sense of having that content loading
and using selenium to load that content, to scrape the information and return it.
So let's wrap things up here and I will see you in the next lecture.