Welcome back.
Welcome back, everyone.
Let's get started.
First thing we want to do is always to import or add our import statements for the purpose of this.
It was optional whether you wanted to use scrapie or beautiful soup for the purpose of this, I'm going
to be using beautiful soup.
My rationale behind that is because I'm going to pass the page and look for the text based on this value,
based on the key.
You can use scrapie for those more specific selectors.
You can use beautiful soup as well.
But I think beautiful soup is situated more beneficially for the purpose of this.
Just because we can pass with our parser the elixir now and we can search based on the text on the page.
As always, there's more than one or there's always multiple ways of solving this, but I want to keep
it simple.
So let's get started.
So the first thing we want to do is, of course, to add our import statements.
So we're going be using requests with beautiful soup to send our request.
My speci, as you're seeing, please give me those double space import requests from for import beautiful
soup.
Import time as well, we've worked with time before, it's great and the purpose of this, it might
not be necessary for our scenario here, but we will want to use it if we're calling multiple pages.
And that would be ideal because you're not just going to be wanting to crawl one page.
Unless, you know, that's where the specific information was released, such as our scenario, if you're
calling multiple pages, you always want to have a crawl rate, you can use time to slow the request
down.
So you're not spamming a service because that's an easy way to get blocked.
Skrappy's actually beneficial in this because it has a default built in crawl rate and you can use this
to either slow it down even more.
But we're only using one euro here.
We're only sending that one request to parts the text.
And that is fine because that won't just be spamming unless you click on it many times.
Please do not do that, because that's another way to spam and send many request to pacemen since we
have our import statements in.
Let's get started with our scraping here.
And this can actually be beneficial if we just use a while loop and set the default to true, because
we can always continuously run through this.
So let's do, while true, have that open, let's say.
Are you Earl?
I have the URL, so I'm just going to piece it in here.
That's the URL that we saw earlier and that's what we're looking to scrape.
If you're looking to add other URLs, you could add this in a list and then iterate through it.
But we're only working with our one you URL, so that's fine for the purpose of this challenge.
In addition, this is going to look similar to our scenario that we worked with in our beautiful soup
section with our headers.
We want to set our headers.
I'm going to open this dict here and I'm going to grab them in the ones we used previously as well in
the beautiful soup section.
So I have them I can add them in as well.
Let me just double check.
OK, that's good.
Just to make sure we're not missing any information, the next thing we're going to want to do is do
we want to send our get request so we can pass in our budget request.
Let's do our equals requests.
Get let's pass in.
Are you earlier and our headers equal to headers makes sense.
OK, now since we have sent our request, we want to pass it so we have our soup equals beautiful soup.
All right.
And we can use our our text for the text on the page and we're going to pass with our parser, our elex
smell fantastic and we're ready to pass.
But what we're doing here is we want to take the text on the page and we want to search for a string.
We're going be looking to find that key based value.
It is a string in the text.
So what we can do is sit and if statement and we can actually use it to less than one where if we find
it one time or more, it's going to set that alert.
If we don't, we can actually set it to sleep to run every X amount of seconds.
Now, we don't want to spam the service more than we have to, so let's pick an arbitrary number.
But for now, let's use and if statement so we can use our if our string in super.
Find we're looking for the key value, because we're assuming what comes after key is some useful information
for the purpose of this scenario.
And as I said, we can set it to negative one here, because if we find it more one time or more, we
want to create that alert.
And it would also probably be helpful if we had it to negative one, since that's what we want, not
equal to one.
If we don't find it, we do not want to spam the server.
We don't want to send an extreme amount of get request.
We want to respect the services so we can use our time, sleep and let's pick an arbitrary number.
Let's use five hundred.
Five hundred seconds is fine.
I think that's enough.
If we do not find anything for the script to run again, it's always going to be checking for the the
key in the text that we're passing.
But we need to continue after continue.
Otherwise we can have with our P&L statement.
And let's convey since we found sensitive information, let's use capital letters because we want to
print an alert like, hey, we might have some serious sense of information here.
Take a look.
We can print that alert.
And this is going to be printed to the console.
And we have to break, of course, to end this.
And this simply enough, is a method of checking the text on the page for a specific value.
Remember, we're looking for a string.
We're looking to find the word key in the text, in the page.
Now, if we run it, we're hoping that there is no issue.
And this comes back matching the key that we're looking.
We want to print the alert for the attention grabbing alert, like, hey, come solve this problem.
There might be sensitive information, so let's run it.
You'll also see that this is not being used.
We will get to that in the next video.
First things first.
Let's test our code and a very straightforward way of scraping this information.
Here we go.
We have our script running and we have our alert.
Awesome to kind of hammer down.
I know I've mentioned it previously.
It's a really simple way of scraping.
We don't need to do too much to solve the solutions that we're working on.
It goes to show the power of some of these libraries where we don't need to do too much.
But at the same time, we are doing a lot.
We're passing information from a page to match specific value.
This is super useful.
I hope you guys are enjoying it.
In the next video, we will take a look at how we can send an email because we won't always be looking
here in the console.
We might want to email a superior the other team.
We might want to send an email to ourselves, to our phone if we're not in front of the computer in
case that information comes up.
Remember, in your cybersecurity analyst role, we need to constantly stay vigilant to keep the company
safe.
OK, let's jump in in the next video.
I'll see you there.