Welcome back, everyone.
In the previous sections, we used beautiful soup to solve our scraping challenge.
In this section, we're going to be working with selenium.
Now, selenium is quite often used as a testing framework or automation tool, but when you apply it
and integrate it within web scraping, it can become such a powerful and versatile tool to have at your
disposal.
As an example, in the last section, we passed in headers to mimic a browser instead of using headers,
we can actually deploy an automated browser that will run through our actions with selenium.
As another example, say you need to pass in a username or password.
You need to click on a button or a load option on the page to scrape information.
This is where selenium is going to become very helpful.
Also, as we've seen from the previous section, we're able to solve our challenges with only a few
lines of code, and this is something that I really want to echo with these tools in libraries that
we're using, such as selenium, it's not necessary.
Deploy 2000 lines of code to solve this.
We're able to scrape the information we want in a few lines of code, keeping everything simple.
So the first thing we're going to want to do is to install Silentium within our environment.
You can go to the Environments tab with the Anakonda navigator.
You can also use kind of the kind of package manager in the terminal or PIP as well.
And you can go to your environment to activate it.
I already have selenium installed, so I'm going to check the install packages as soon as this loads
and I can search for selenium.
If you go to not install on your machine, if it's not installed, you'll use this option and then pass
in selenium and this is what you want to install once you have this installed.
We then need to install the Web driver.
We're going to be using Chrome for the purpose of this section.
You can visit the following URL to download the Chrome driver if you want to check the version of Chrome.
Now, you have installed in Chrome if you go to the Chrome option.
And then click about Google Chrome, this will launch a page where you can see the Chrome version that
you're using.
All right, and here is where it can get tricky a little bit.
This is a pretty simple install, but we need to be working with our path related information on our
systems.
I'm using Mac, so for this, I'll be following these directions.
I will also include the links.
And if you have any issues, please feel free to post in the Q&A and I'll be glad to help resolve any
of the set of information.
We have the zip file and we need to launch a terminal and run the following commands.
We have these pseudo nano paths.
This is going to bring up the nano environment.
If you're running it from a terminal, it's going to look like the following.
So I have this.
Access from my terminal, you will unzip the downloaded file for the Chrome driver and then in our terminal
we can use the following command.
Once we have it, you'd go to your downloads folder and then you would run the following command that
we could see here for our path information.
This is going to set it so we can actually find the chrome driver.
If you're on windows, I can go through the steps here.
You can follow these steps.
It's very similar.
You really want on any OS.
This is the main point of it is to set it in the path information so that we're able to find it so selenium
can find it.
And I can scroll through these, but I'm going to include a link for Windows as well.
And I will also include a link for Linux installations.
So we can cover the three main operating systems here once we have it installed.
I already have this set in my path information, but just to verify it, we can launch Spider and open
a new file.
Once we have our new file created, we can pass in the following lines of code.
We want to import time and we want to import selenium here from selenium, import our Web driver.
Now you can pass in the path related information.
This is an optional argument.
And the cool thing about selenium, in addition to being such a useful tool, is that it works with
many languages.
But we're using Python as we know.
So we're going to pass in our driver equal to Web driver Chrome.
You would include the path if you need to, but you can also just use it.
If it's in your local path, we can assign it the following the web driver eggroll.
We can use Driver Dagget to get the U.
RL that we want.
We want to assign time, sleep time is going to be a very useful library here, because when we're working
with non static pages, if we're working with dynamic HTML pages, we want to be able to let them load
because we want that JavaScript content to be able to come up.
We want to be able to see it.
So it's always good to assign a few seconds to have your page load and be able to see it and then we
want to quit.
So I'm going to run this.
And if it's successful, it should launch Chrome, bring up a page sleep and then it should quit out.
So let's try to.
We have it launching the Crome page, we're giving it.
Time to load.
Here we go, we have the Chrome homepage and it should come out and there we go.
We have successfully ran a test and automated our browser in a really simple form with selenium.
In the next video, we're going to be taking a look at the basics on some options related to using selenium
within the page, a look at the dynamic content as we reference here really quickly.
So let's wrap it up for now.
Grab some coffee and I will see you in the next lecture.