Welcome
Hi. This is Amir Shahzad, and I am so glad that you joined me in this exciting course. In the upcoming modules, you will learn how to automate your penetration testing ideas by using the amazing programming language, Python. We're going to build together a full-functional application that automates most of your daily tasks using the best operating system for hackers, Kali Linux. Your time is expensive, so I'm going to save some discussion for later in this course. Let's get started.
Prerequisites
It is better to understand what you need to know first before you start watching this course. To begin with, you need to have a good understanding of penetration testing in general. If you're hesitating, I advise you to watch my penetration testing course on Pluralsight on the following link, Pluralsight. com/courses/kali-Linux-penetration-testing-ethical-hacking. Also, in this Python course, you're going to see me mentioning some hints that I already discussed previously in my Python course. Feel free to visit the link and try to have an idea about the topic and discussion. Finally, I'm assuming that you have a basic knowledge of the Python programming language. If you don't, then no worries. We have a nice introduction course on Pluralsight that shows you the way on the following link, Pluralsight. com/courses/python-fundamentals. Wait, I'm not done yet. I want to share with you a secret that I use to develop my Python applications. Open your browser and head straight to the following website, www. tutorialspoint. com/python. This is a great place to learn Python and to come back to it later when you want to know about a certain functionality that you want to develop yourself. Alright, now let's jump into the next demo to see this course overview.
Course Overview
In summary, this course is going to cover a lot of interesting topics to automate your penetration testing ideas. I know you're curious and you want to start programming right away, but first, let's see the course workflow, so you would have a better understanding of what you're going to learn. To begin with, we're going to build the infrastructure to understand better the main functionalities of the automation application. After this, you will learn how to automate your reconnaissance executables using Python. Of course, remember that we're going to take advantage of the rich number of applications installed on Kali. Your next lesson will be how to automate an external scan with just a single click. Awesome, right? After the scan phase, the next thing that comes right away into your mind is exploitation and attack automation. Yes, you're going to learn how to build a master. In the final module, you will learn how to automate a website security check. These days, every company has its own website and it's important that our client rest assure that no one will hack his public figure. Now you know what you're going to learn in this course. Let's go to the next demo and have a look at our Kali Linux environment.
Setting up the Environment
Kali Linux is the operating system that we're going to use in this course. Why? Because it contains a rich set of tools for penetration testers, and since Kali is a favorite toy for black hat hackers, it should be your toy as well. Let's get started. First, open your console and type the following comment to know which version of Python is installed on Kali. Now type the keyword python followed by --version. As you can see, the version 2. 7 is installed by default on this operating system. So in this course, we will cover the default version on Kali, which is 2. 7. Now to develop programs in Python, we are going to use an IDE called wingIDE. You're not obliged to use it because this is not a free application. You can use any text editor like gedit or Nano, but you won't have all the bells and whistles that the professional IDE will offer you. To get this application, open your browser and head straight to this website, www. wingware. com and you can download a trial version by clicking on the Download button and selecting the professional version from the menu. Click the Debian Package to download this application. Finally, click on the OK button. I already have this application installed. If you want to learn how it works and how to install it on Kali, head straight to my Python course on the Pluralsight and watch the module, Setting up the Environment to see how it works. To start wingIDE, open your console and type wing5. 1, which is the current version of this application, of course, it depends when you're watching this course. Probably in the future, the version will change.
Summary
You want to watch this course because you already know the importance of the Python language in the penetration testing field. Probably you're sick of counting on automated tools without understanding the concepts, and also, you want to be able to develop your own automated application for the daily tasks. Let's stop for a second because I want to mention an important notice. Please do not use the knowledge acquired in this course without a proper agreement with your client. If you want to test it in your local environment, that's fine. Alright, let's summarize quickly what we're going to cover in this course. First, we're going to build the infrastructure. Second, you will learn how to automate the reconnaissance phase. Then you're going to see how to automate the external scanning with Python. After that, we're going a step further to start exploitation. And finally, you will learn how to test your client website using the automation in Python. I am already excited. Are you? Let's move onto the next module and start programming.
Building the Automation Sandbox
Introduction
It's time to start to dive deep into the programming stuff. In this module, we will build together the infrastructure for our course. This is an important module because you will learn the concepts of Python automation and penetration testing. All the future modules will be based on this one. Alright, let's jump into the next demo to see a detailed overview of this module.
The Automation Application Overview
Let's see a quick overview of what we're going to cover in this module. To begin with, I want you to ask yourself the following question, how do you achieve your daily tasks of penetration testing? The first option. You use the command-line, right? This is where you execute your favorite tools like Nmap or Metasploit. The second option, you use the web browser to open your favorite websites mainly for information gathering. The third and final option is the graphical user interface like Burp Suite. In this course, we are going to automate the first two options, but not the third one because it's practically impossible to do it in Python, unless the application supports it. For example, in the last module, we're going to see how to add plugins to Burp suite to show you at least one example to use a GUI for this purpose. So we're going to build the infrastructure of our mighty application in this module. I'm going to show you the basics of how to automate a console application, and how to open a web browser using Python, and you will see how to save the results to a file as well. At the end, we're going to merge everything together and start refactoring the application to make it clean and mature enough to hold tons of automated tasks. Yes, as I mentioned earlier, it's going to be a master, and guess what, this application is published on GitHub, so everybody would be able to download it and use it. Don't worry, by the end of this course, I will dedicate a demo to show you how to take advantage of this application on GitHub. Alright, let's head straight to the next demo and start automating the terminal commands using Python.
Executing Commands from the Terminal
Most of your time as a penetration tester you probably spend it on the terminal window to test your client security. In this demo, you will see the basics of how to automate this using Python. Later in this course, you will delve deep in the commands that you need for each specific phase. The idea is simple. All you need to know is the proper function in Python that execute commands. After that, you get the results and then print it to the console. I don't want you see me typing, so I already prepared in advance this source code sample. As usual, the entry point is here in the if statement. This is where we call the Main function. In the main function, we just call the execute_commands function to execute the commands. Let's jump into it and see the details. Here I created a dictionary list of commands that I want to execute. The first one is the ls command and the second one is the pwd command and I included some descriptive text for each one. Next, I look into each command to call the open_terminal function. The most important line in this function is here, the subprocess. check_output. First, I'm supplying it with the command that I want to execute and I'm capturing the standard error as well. The shell= to True argument will default the terminal2/been/sh. That's it folks. After this, we return the results and print it to the screen. It's time to see the application in action. I'm going to jump to the terminal window and execute the application. And voila, I just automated the process. I know. This is a basic idea. No big deal here. Later, we will make this baby a mature application. Be patient. Step by step and we'll get there.
Opening the Browser
The second tool that you use frequently in your penetration test daily job is a web browser. The internet contains a rich amount of information that helps penetration testers to achieve their target. Let's see how to automate this scenario. As usual, we start with the main function. Here we call the open_websites function to execution the process. In the open_websites function, first, I declare an array for a couple of websites to test. Remember, this is just a sandbox environment. We go with the easiest scenario. Next, I'm iterating through the website array and for each one I'm using the open_new_tab function to open the browser and add each website to a different tab. Great job, Python. You just made my life easier. Pay attention here that I'm using the sleep function to wait for 2 seconds between the openings of each tab. Why? Because without it the whole thing won't work and you're going to end up with delay issues. Let's see how this application looks in the real world. In my terminal window, I will execute this humble application and the browser should open my favorite websites. Again, this is a basic idea. You will see later more details about how to apply this for penetration testing.
Saving the Results
In this tip, we're going to see how to save the results of the terminal execution. Now what about the web browser? Don't worry. I will show you how to save it manually as well and you don't need Python for this one. Let's start. All the main actions are happening here in the Main function. Here, I'm calling the execute_commands method that we just saw earlier in this module, then I print the results. After this, I'm saving them using the save_results function. Check out here that I'm passing three arguments. The first one is a terminal results text. The second one is a folder name where I want to store the results. And finally, the file name for the report. Let's take a quick look at how we save the results. First, I'm checking if the directory exists before creating it. If it's not created, then I do create a new one using the mkdir function. Next, I set a variable to construct and hold the file name. After this, I use the open function to create the file and use the write function to write the terminal results to it. Let's go back to the main function and see the next step. After saving the results, I use the open_websites function that I assume you know about it by watching the previous demo, right? Now, let's see how the sandbox work all together. As usually, I will open the terminal window to execute the command. Check this out. Here I have the web browser opened my favorite websites. Now if you want to save the results, let's assume this page is something that you want to put in your report. Click on the Menu button and then click on the Print button. After this, click on this Print button as well. In this new dialog box, select the print to file. Now you have the option to save the page as a pdf file. Give it a name and click on the Print button down here. Now you should see it in your Home folder. Let's go back and check if the terminal results were saved in the reports file. Under this new results folder, we can see the reports. txt file. Perfect. Let's open it using gedit and here you go. Our automation results were saved successfully. Now it's time to turn this sandbox into a monster in the next demo of this module.
Putting It All Together
It's time to turn our baby into a monster. As I mentioned earlier, this application architecture should be flexible enough to support thousands of terminal commands. For this reason, we need to make sure to apply best practices in programming. Let's see together the masterpiece in action. At first glance, let's take a look at the source code from the upper level. As you can see, we have three classes. The first one on the top is the Utility class. This object is responsible for all the static and reusable operations in the application. Wait a bit because you're going to see the details in a while. Next, we have the Core class that holds all the core logic of the application. Everything that we saw previously in this module will be included here because this is part of the infrastructure. Finally, we have the main class. This is where we put all the logic that needs to start the application in the right way. For the time being, the application version is 0. 3, and as long as we move on, the version number will increase until we get to the release version 1. 0 in the last module. Alright, let's start. The application begins here inside this if statement. First, I need to make sure to create an instance of the Utility class because this object should be created from the start to the end of the execution lifecycle. Next, I create an instance of the main class to start the application. In this tool, I tried my best to follow the Python community convention. A very private method will be prepended with an _ and every public method will not have it. When the start method is invoked, the application is going to initialize the arguments. I'm assuming you know what this code is, but I'm going to give you a quick brief anyway. First, I'm creating the parser object. Next, I'm adding two arguments that the application should support for the time being. The first one is the company domain name that you want to test and the second one is a DNS test. If the user did not enter the company argument, then the application will print a help message. The usage function is just printing some text to help users to know how to use the application, and in the last line, the exit 0 exit and stop the execution of the application. If everything is good, we can proceed and start the application. Next, we initialize the Core class object and we print a welcome banner using the private print_banner function. Check this out. This method is printing a message and in the second line I'm using the Utility class to print a line separator. Now it's the right moment to understand the idea of the Utility class and why did we create it in the beginning. In summary, every method or variable in this class is shared and reused all over the place like printing a separator line, or changing the color of the test, and so on and so forth. After printing the welcome message, we start processing the argument in the process_arguments function. In this method, I'm checking the terminal window argument. For the time being, to start the project, I'm using the DNS. Later, we will add a lot more as we move along. If the argument supplied is DNS, then we call the pen_test method in the Core class by telling it that this a dns test, and as a second argument, I'm passing the category name, which is a reconnaissance in this case. We're going to see all of the reconnaissance switches in the next module, but for the time being, all you need to understand is how the infrastructure works first. Alright, let's move to the Core class and see the details. In the pen_test function, we start by creating a root folder for the reports. The path structure will follow a pattern, /reports/ the domain name of the client. Now to save the reports file, we create another folder inside this one. To be honest, the easiest way to explain all this is to show you live how the application works. From the terminal window, I will execute the pen_tester automation tool application AKA pat. py, followed by the domain name of the client, and finally, I will specify the dns test, and press Enter. Let's see how everything worked behind the scene. Under the Reports folder, the application created another folder with a client domain name, then it created the Reconnaissance folder, and finally, a file is created for the report itself. There is one more secret in the back application that you didn't see it before yet. All the commands and the websites for DNS are stored in files, not in the source code itself. Why? Because I don't want to hardcode and spoil the source code. I want it to be clean as much as possible. Let's go back to the source code and continue our mission. After creating the root folder, I'm using the get_commands_from_file function to load the commands from a file. Here in this function, I'm checking if the file exists first. Then I iterate for each line independent to the dictionary list of commands. Next, I'm loading the list of websites from a file as well. Before starting, I'm preparing the folder_name and the file_name of the report for the upcoming functions. Finally, I call self_start function. In this method, I start by checking if both the commands and the website list are empty. If not, I proceed further to the next step. Here, I use execute_commands function that we saw previously to execute the commands and save the results to a file. Finally, I open the web browser same as we did in the previous demos and we're officially done. In the upcoming modules, we will start filling up the files with commands and websites and some few changes in the source code.
Summary
This module is the most important one. If you understood everything, then all the upcoming modules will be a piece of cake for you. Let's see quickly what we have developed. First, you saw how to automate a terminal window using Python. After that, you learned how to automate a web browser and you saw how to save the results. Finally, you witnessed the born of the new baby monster, pat. py. In the upcoming modules, we will see how to start using this application for the daily tasks.
Reconnaissance Automation
Introduction
Hi. This is Gus Khawaja. In this module, you're going to learn how to implement the commands for information gathering. If you have ever done penetration testing before, you know how long it takes to conduct this phase. For some people, it's so boring so they skip it without understanding the importance of it. After finishing this module, I'm going to make your life easier. You're going to see how easy it is to use and reuse this application. I'm assuming you know what reconnaissance or information gathering is and that you want to learn Python to be able to automate your penetration testing tasks.
Module Overview
Let's see a quick overview of what you're going to learn in this module. In the previous module, we implemented the infrastructure you saw me using dummy commands for DNS testing. So in this module, you will see the real commands implementation. Next, we will automate files extraction to see if there are any files that exist on the web and probably they contain some juicy information about our target. After that, you're going to learn how to take advantage of the Whois information, then we will use different tools to extract email addresses of our client. Also, not to forget the social media, one of the most important area for penetration testers to extract information. Finally, we will use some website automation to finish our information gathering mission. Before I start, I want to tell you ahead that the site that I will be testing in this module is my blog website. Why? Because I don't want to show leaked information for any company out there during the demo session. Let's start.
DNS
It's time to start implementing the real commands for the DNS automation. Under the PAT directory, go to the Reconnaissance folder and open the dns_commands. txt file. First, I'm using dns recon in two different ways. The first one is a straight DNS Record information. In the next one I'm testing the DNS Transfer. Finally, I'm using fierce to get the DNS information. I find personally that fierce is the best tool ever for DNS testing. Now probably you want to add your own commands. All you have to do is to change this file. No need to change the source code for that. You see, that's a benefit when you develop a good infrastructure in your source code. It brings a lot of advantage in the long run. Speaking about the source code, take a look at the parameter domain that I'm using in this file. This variable will be replaced by the domain name of our target. Soon, I will show you how to implement this in the application infrastructure using Python. Before I close this file and move on, I want you to think for a moment, imagine yourself typing all these commands by hand. Not only that, you have to manage the results manually as well. Too much work, right? Now you see the benefits of this monster that we're trying to build together. Wait, we're not done with DNS automation yet. There's a website part. Let's check what's inside that file. Here, I'm using two websites to collect DNS information. If you feel that you want more, all you have to do is to add your favorite website. In this file, let's revise the source code logic to understand how to execute the DNS commands. In the initialize_arguments function, we receive the dns switch from the command-line, then we call the process_arguments function, which calls the Core class and the pen_test method. In this function, I'm extracting the commands_from_files, and finally, calling the start function to start executing and saving the results. Now it's time to see how I injected the parameter domain in the command and website files. In fact, it's implemented here in this function, inject_parameter_in_file. I'm replacing the domain parameter with the real domain name of the client. In the next line, I'm using the name parameter to extract the company_name and inject it as a file as well. I will be using the company name later during the website automation demo. I know you're asking yourself where are you calling this function? It's here. At the moment, where I'm extracting the command from a file and before sending it to the operating system to execute it, either for the terminal window or the IceWeasel browser. Do you still have a doubt about how this application works? Well you know what they say, a picture is worth a thousand words. I will show you a visual architecture of this application that's worth a million words. First thing first, you use the terminal command to execute the application and you specified the argument that you want to use for your testing needs. Then the main class will use the initialize_arguments function to initialize the application arguments. Next, the application processes the supplied arguments by the user, for example, DNS or emails, depending what you entered in your command-line arguments. After this, the process_arguments function calls the pen_test method in the Core class. This method is just an entry point to this class, then the heart of the application starts executing. At this stage we start by getting the commands from the file and the website pages as well. After that, we execute the commands using the terminal window and save the results eventually using the save_results function. Finally, we open the websites fetched from the file using IceWeasel. These are the main functions that you need to know about for this application. Print this page and keep it with you because I want you easily to follow this course without any doubts about your comprehension of this architecture. Of course, we have other functions, but here we can see the main ones that makes this engine works together. It's time to test the DNS functionality. Let's head to the terminal window and execute it. Python followed by the application name --company followed by your client domain name, and finally use a dns argument. This process is going to take some time to execute, so you need to be patient, especially with the fierce command. It could take like 5 to 10 minutes to brute force all the possible combinations. On my end, I'm fast forwarding this so you won't wait so long. As you saw, I'm using my blog website for testing. The idea is not to show you vulnerable clients. I cannot test any companies without a proper contract, right? And no one is willing to show you guys this information, so I'm volunteering with my own blog. At least, you'll be able to see how it looks like at runtime. Now the application has finished the execution, let's see what it found. DNS recon found an SOA record and multiple NS records, mail exchange, and finally, an A record. Next, DNS recon tried to test for DNS zone transfer and they all failed. It's normal because that's how a secure website should be. Finally, my favorite tool, fierce found an interesting FTP domain for my website and the zone transfer failed as well because it's not enabled. For the websites, the first one Ip Neighbors, they didn't find any results. That makes sense. The next website, the DNSstuff found some public DNS record that I should not be worried about. Let's check out the report file if it's well-saved as it's supposed to. Oh yeah. A nice, clean report saved by Python.
Email Extraction
Extracting email addresses during information gathering helps a lot to figure out the email format that the company use. In addition, it will help to identify some employees that works at the company and much more. Let's jump in and see how to automate this phase. If you parented the architecture workflow that I showed in the last demo, you can guess that my start point is here in the initialize_arguments function. So I added a new email argument parameter according to the workflow that I'm assuming is in front of you. Next, comes what? It's the process_arguments function. Here, as you can see, I added an email switch to check if the pen tester supplied it in the terminal window. If yes, then I tell the core class to proceed with the email test. Do you see how easy it is to change a source code from now on? When you have a clean application source code, the long-term refactoring and the maintenance becomes so minimal, that's why you hear about coding best practices and design patterns. Let's jump and check the commands that we want to use for this test under the emails_commands file text. The first one I'm using is Dmitry to check for email addresses and is a second one I'm using Metasploit, search email collector module to gather email addresses as well. Now what about the websites? In this file, I'm using email-format website to check for email addresses about my target. If you find that you need more, feel free to change this file to your like. Alright, it's time to test it. Open the terminal window and start executing pat. py --company followed by my blog domain name and the emails switch to test. Finally, press Enter. After the execution has finished, check out if your client has some email addresses leaks on the internet, in my case, Dmitry and Metasploit. It didn't find any email addresses and that's what you should be expecting for your client. For the web search, it looks like that my website is secure as well. Good job for me. Until now, I passed all the tests successfully. Finally, let's open the report and see if it works. No doubt we have a nice report saved for us later to check.
Whois
Let's see Whois information is very important in reconnaissance because of the interesting information like email addresses, phone numbers, administrator name, company address, and so on. So we will add it to our application. Let's open the Commands file to see what's inside. Here, I'm using dig and dmitry to extract the whois information of our client. This course assumes that you know these commands. If you don't, check out my Pluralsight course for penetration testing using Kali Linux. Let's take a peek at the web pages part. In this file, I'm using two websites to gather information. The first one is a ewhois. com and the second on is dnsstuff. com. What about the source code? Let's go there and see what's inside. Here in the initialize_arguments function, I added a whois argument and an if statement to support it. That's it. If you watch my previous demos, this should be easy and straightforward. It's time to test this in the terminal window. I will take the previous commands and change it to test the who is information. The execution has finished, so let's check the results output. No personal information that shows here is a Dig results. The Dmitry command is showing that my site is hosted by GoDaddy, but there is no personal information as well. Good job, Gus. I manage personally to secure my website successfully. So should your client. Let's have a look at the web results. The ewhois is not showing any useful information and DNSstuff is showing few useless data that I should not be worried about. I know that the hacker instinct of you is telling you no way, Gus, I want to see some vulnerabilities. It's okay, that's normal. But unfortunately, my blog is secure.
Files, Social Media, and Web Search
Imagine that your client has some file on the internet with credentials inside. That's awesome, right? No, really. This is a high-risk information leakage and should be fixed as soon as possible by your client. In this demo, I will show you the automation of how to look for juicy files, and social media information, and finally some general web search to check if we missed some info at the same time. Let's start. For the source code, I think that you know it by heart at this stage, we add a file argument, social media argument, and a website argument. Next, we add the if statements to support the three arguments. If you don't know what I'm talking about, it means you didn't watch my previous demos. So I invite you to pause and start again this module. Let's open the command file to take a look at the commands for the file search. Here, I'm using the googfile application to search for multiple types of files including Word, PDF, PowerPoint, Text, and Excel. Now for the web, I'm using Google Dorks. Oh yeah. Do you know how painful it is to write this manually in your web browser? I hate it personally. With this automation tool, our task is much, much easier. In the next demo of this module, I will show you how to get even more of these queries. Next, let's check out the commands for the social media. In this file, I'm using theHarvester to do the job and I have no web file for this scenario. Finally, let's open the web search commands. In this file, I'm using theHarvester and Dmitry to collect information from different web sources. It's important to note that theHarvester is not installed by default on Kali Linux, but don't worry about that because I already saved a copy on the main root of this application. Same for goofile. I will open the terminal window to test this scenario. As usual, Python followed by the name of the application then the company domain name, which is my blog, in this case, then the file argument, followed by the socialmedia argument, and finally the websearch argument. The execution will take few moments to execute. Here, we can see that IceWeasel has started. Let's minimize it and go back to the terminal window to check the progress. It's still running. You need to be patient when running this application. If you want, take a cup of coffee. Alright, it looks like that the execution has finished. Let's scroll up and check the output. For the social media test, I don't see any useful information that I should be worried about. Same for the juicy files, which looks clean. Now for the websearch, it's even much more cleaner. So let's check IceWeasel, Google Dorks are charming. No results found. It's clean everywhere. I'm so happy. No active threats on URLVoid. Perfect. Shodan, so clean. Pastebin, no results. Google site search, nothing to worry about. Same for this one. This is just my home page. And finally, no results for Censys. Probably you're thinking that this application is not working. No, it works like a charm. It's my blog that is secure for us.
Google Dorks Extraction
This is going to be an exciting demo. I will show you how to extract the Google hacking database locally, so you would be able to use them or automate them using Python. In this demo, I will walk you through of how to think as a Python programmer. We will build the application from scratch, so you would understand how generally, I find the solution for any problem using Python. The Disneyland for Google Dorks is the Exploit-DB website. Open your browser and click on the Exploit-DB icon. In the home page, you have a link to Google Hacking Database. In this page, scroll down and you will see all the categories for Google queries like Footholds, Sensitive Directories, Vulnerable Files, Vulnerable Servers, and so on. Let's see what happens when I click on a query item. On the Landing Page on the top, I can see the query itself. Next, I have a gray box. It starts with a description. After that, the Google Search query and when the query was submitted. Finally, you have some additional description about the query itself. Take a look carefully at this page. Check out the URL structure. It starts by the domain name, followed by Google Hacking database keyword, then an id number that represents as a query. That's awesome. The Exploit Database made my life easier. So I can expect all the queries from their website. Why? Think carefully. Let me show you what I mean. If I change the id number and increase it by one, voila. I got another query. All I have to do in Python is to construct a loop that increases id number, after that, I will construct a request, then read the response, which downloads a page. Finally, I will save the results to a file. Let's take a look at the architecture overview of the scenario. We start by identifying the range of our loop. Example, you want to extract queries id numbers between 3, 000 and 4, 000. Then you construct the request. After that, you get the response from the server where it's a page id specified in the loop. Next, you extract the query data like the query itself with a description and the date submitted. Finally, you save the results to a file. That's it. Let's go back and start programming this application. I will close the browser and open the terminal window. As usual, I will use wing IDE to write the application. Click on the blue button to open a new file. First thing first, I will create a function and call it extract_google_dorks and I will give it two parameters to specify the range of numbers that I want to download from the site. If you remember from the architecture diagram that I showed you earlier, the first thing on the top is the loop. So let's implement it. Next, I will print a message to show that the process has started and I will print the current query URL. I personally don't like to hardcode URLs, so I will put it in a separate variable. Before I start anything here, it is a better idea to have a try/catch block to avoid the application to crash in case of an error. I print the exception detail. Next, I will sleep the operation for 3 seconds by using the time. sleep function. And don't forget the important time library. If I have five consecutive errors, I want this application to stop immediately because that means there is something wrong. It could be your internet connection, probably the site of Exploit-DB has changed, or is down. Also, I want to make sure to close the file object. And finally, exit the application completely using the exit 0. I initialized the failed_attemps_counter variable at the top of the application and I will initialize a log_file variable later when I save the results. Now let's start to construct the request using the urllib library. First, I will import it. Next, I will create a variable item_request to hold the request object. The request method needs the URL of the page that we want to interact with. Now I can reuse that current URL variable that I just declared for printing the banner. That's the benefit when you have the reflects of good programming practices. You can reuse your code and make it clean instead of repeating yourself with stubby paste. After creating the request object, we need to specify the browser header user agent. I will be using the user agent of IceWeasel for this application. Now the big question is how do I get to use an agent for IceWeasel. That's easy. Open your browser and head to any page, for example, Kali Linux home page. Hit F12 on your keyboard to show the developer mode in IceWeasel. Next, click on the Network tab and reload the page. In this window, click on any request item. On the right side, you will see the user agent value in the header section. Now let's copy it and paste it in wing IDE. Next, I will create the item response variable and use the URL open function to get the response and pass in two parameters. The first one is item_request object variable and the second one is the timeout value for this response. Then I use the read function to read the response page. After getting the page contents, we can start extracting the data. I will be using a library called BeautifulSoup. That was a job. Of course, I need to import it first. I will be using the HTMLParser as well because I will need to use it with the BeautifulSoup library. Let's take a look at the page one more time to analyze how to extract the contents. Open your browser one more time, go to Exploit-DB, and then Google Hacking database, and select any query. On the Landing Page, right-click on the gray box area and select Inspect Element. Check this out. It looks like that this box is a table that will make our easier to extract the data. By using BeautifulSoup library, I will be able to get all the information and save them later to a file. I will copy the extraction part from the clipboard instead of boring you seeing me typing all this. Let's revise this code. First, I'm declaring the BeautifulSoup object and by passing the page contents. Next, I start specifying the scope where I want to focus my extraction, which is the HTML table, then I start with the first td item and declare the HTMLParser object as well. At this stage, I can start extracting the Google Dork query. After that, the data added. And finally, the description for the Google Dork query you can see sometimes how libraries will make our life easier like BeautifulSoup, in this case. It's time to save the results. I will get the piece of code from the clipboard as well for this one. Here, I'm using the right function to save the date_added for the Google Dorks, the description, and the query itself. Then I'm closing the response object. At this moment, I need to declare the variable of the log_file at the top. I will give the fallen name that contains the start_item_number and the last_item_number and I will use the A argument to append the file data. Finally, I will reset the exception counter and print a successful message. If all the pages were downloaded successfully, we close the file object and print a message for this purpose. Before executing the application, we need to develop the main function to start it, right? I'm assuming that you know what I'm doing here in the main function. This is a basic of Python programming. Alright. It's time to test this application. First, make sure it's saved. Give it a name and click on the Save button. Then click on the Debug menu and select Set Current as Main Debug File. After that, click on the run green button up here. No need for any arguments in this case, so leave it empty and click on the OK button. As you can see, we have an exception. The item response is not defined. Let's fix it and start again. Ctrl+S to save the changes and click on the green button one more time. Pay attention at the debugger window down here. It's showing us the progress of the download. I can tell you that this process is so slow, so you have to be patient as usual. It looks like that the execution has finished. Alrighty. Let's check the saved data. Oh, yeah. Here we go. Data is saved in a clean fashion. Pay attention here that the dates are a little bit old. That's because I used the id 900 to a 906. At the moment of this recording, the latest id numbers range is above 4, 000. I will give you the choice to choose your own range.
Summary
I hope that you enjoyed this module, folks. Let's take a quick look of what we've covered until now. First, we implemented the DNS automation, then we tried to extract email addresses. After this, we focused on the Whois information. Next, we looked for files and for the social media information as well. And finally, we automated the browser web search. In the last demo, we saw how to extract the Google Dorks queries using the amazing language Python. Let's head to the next module and continue our automation journey.
Internal Scanning Automation
Introduction
Hi. My name is Gus Khawaja, and I would like to welcome you to this exciting module using Python and Kali Linux. I had so much fun creating this module for you, and I hope you will share the same feeling as well. This module will prepare you to develop an automated scanning using Python to the internal network infrastructure of your client organization. Let's begin.
Module Overview
You saw in the previous modules how we built together this amazing application. We're not done yet, folks. There is more to come. So let's see what we are going to cover. We are going to use the awesomeness of Python to be able to work easily with IP addresses. After we identify the range of IP addresses that we want to target, we start looking for live hosts. Next, comes the part where I take advantage of Python to scan only the hosts that are up and running. I don't want to waste my time while Nmap is scanning useless IP addresses, right? I made my best to show you a new style in this module. For example, in the next demo, we will see together how to create a sandbox as a practical example that you can use in your daily work. Let's start.
IP Integration
I know that we share the same passion, which is penetration testing in general. Imagine that you have an amazing idea and that you want to add it to an existing application like the one that we're doing in this course, pat. py. The problem is that if you start changing the main application itself, you will suffer big time because you have to debug the whole application to know what you did wrong. And finally, you start losing the hope of achieving your goal. It's not your fault. Nobody told you about it before. I will share my experience with you so you would not end in a block state. I want you to achieve your goal in the easiest way as you can. Yeah, but how Gus? The answer is create your own sandbox. What I mean is to create a new Python file and focus only on your amazing idea. In this demo, I have an idea. I want to be able to scan a single IP address or a range of IP addresses, so how do I implement it? Follow me to see how to do it in action. The idea in a sandbox is to be as simple as you can. To start, I will create four variables to hold different types of IP addresses. The first one is an IP range in the format where we have a - that separates the start and the end. The next type that I want to use is a single IP address and I will command to this line for testing purposes. The third type is the subnet mask. For this example, I will use a CIDR form for a class C network. Finally, I want to be able to scan different IP addresses separated by commas. For the first variable, I need to start with an if statement because I know that this type will contain a - as a separator. I will use the split function to split the IP address range. The input_data. split variable will be an array that holds the two values. The first one will be the IP address, so I will store it in a temporary variable, in this case, it's called first_ip_address. The second IP address is not complete. I only have the last part of it, so I will use my Python ninja skills to extract the 10. 0. 0 part from the first IP value using the split function. And lastly, I will concatenate the value split with 120 and the final results will be 10. 0. 0. 120. At this moment, I have to create an IP range object so Python will take care of the iteration of IP addresses. For the sake of simplicity, I will just print the results to the Output window. In order to use the IP range object, I need to import it and I will import the IPNetwork and the IPAddress as well because I will use them later. Now what about the comma-separated IP addresses? That's an easy one. All I need to do is to split it and send it to the loop iteration. Since I'm using the IP address variable in so many places, I will declare a default value here at the top. For the single IP address and the subnet range type, I can use the IPNetwork object that I just imported above to either iterate or print a single id. Take a close look at the source code. Do you see something wrong about it? Right, it needs to be refactored. I want to show you the steps from A to Z because I want you to see yourself how applications are developed in real life. Refactoring, in fact, comes along the way. So let's begin. First, I will create a function called get_ip_addresses that takes the input data as a parameter and it returns a list of IP addresses. Next, I will move the variables and then loop below the function or else I will have an error if I keep them at the top when I call the function get_ip_addresses and not to forget to move the addresses variable inside the function. The final thing remaining in the source code is to call the get_ip_addresses function. Let's see if it works. As usual, make sure that this file is in the current debug and press the green play button to run it. Of course, we need to save the file as well. It works like a charm. The output shows me all the IP addresses printed in the range of 100 to 120. Let's comment the first variable and uncomment the second one, save, and done. It works as well. Next, let's uncomment the third IP subnet class C, save it by pressing Ctrl+S, and run it using the green button. As I expected, the full range from 0 to 255. That's awesome. Finally, let's uncomment the comma-separated IP addresses and again save it and run. And as you can see, the two IP addresses. 1 and. 3 printed. Yes, sir. Our proof of concept is working. Now let's head and implement it in the main application. This is the application at where we have left in the last module. First, let's start by changing the initial argument function. Here, I will add an argument for the IP address -ip or --ip_address and give a help message for this type of argument. Do you remember the architecture image that I showed you in the last module? Now you see its importance. It should make your life easier by mastering the contents of the source code. For the time being, I will copy/paste the function that we did earlier in the bug file and don't forget to copy the libraries as well or you will end up with runtime errors. Imagine that you did all this without a proof of concept project. Take my advice. Make it simple and enjoy your victory.
Identifying Live Hosts
In this demo, you will see how to scan for live hosts using pat. py. In the commands file, you can see that I'm using Nmap to ping scan the ipAddress and the NetBIOS scan as well in the second line. Now pay attention here that I'm using a single ipAddress in the variable, not a range. Why? The answer is I want to be able to have a report per host. That would make my life easier as a penetration tester. Let's see how to implement this in Python. As usual, things start here in the initialize_arguments function. I added a live host argument. You should know what this means if you really watched my previous modules. Next, in the process_arguments function, I added an if statement in case the live host option is supplied by the terminal window. Then I make sure that the pen tester did not forget to supply the IP address option. If he did, I print a usage message to help him go through this step. Below it, I'm using the loop that we created in the bug file in the previous demo. And for each IP address, I call the pen_test method in the Core class. In this way, the Core class will create a report per ipAddress. We're not done yet. I wanted to go even further and create a folder per IP address, so I had to make a little change in the save_results function. Here, I added an if statement to check if this is a scan type by checking the existence of an IP address option in the terminal window. If yes, then I create a variable for the folder name. After that, I create that folder. Finally, I proceed as usual for the rest of this function. Let's test this scenario in action. Open the terminal window and browse to the current application folder and then execute pat. py -c followed by your client name. Next is a livehosts switch. And finally, I am scanning the 10. 0. 0. 1 and. 3 to see if they are alive. Finally, press Enter. Let's check the results. The first IP address. 1 is up, but no NetBIOS scan results. And the second host is done. Let's head straight and open the file report. Look how clean it is, a folder per IP address. That's what I meant by a nice report to check later for reporting purposes.
Port Scanning
It's amazing what we've achieved together so far. Let's take a look at how to implement the port scanning technique in the awesome pen tester automation tool. In the commands file, I'm using my favorite Nmap command, the -A and I'm assuming that you know what this means, right? In case you forgot, the -A means an aggressive scan option. It will execute an operating system scan and a version scan. I'm not done yet. It also executes a script scan and a trace route as well. So powerful. Here, I'm using the T4 timing switch to make the scanner faster. Why? Because I'm scanning an internal network. If I was scanning from the outside, then things will be different. Let's switch to the source code and see the implementation for this scenario. As you can see, I added a new portscan argument in the initialize_arguments function. Next comes the big question. In the process_argument method, I will make some space so you would see things better. In this function, I want to scan only the live hosts, right? I don't want to waste my time scanning useless IP addresses. So first, I'm checking if you already supplied the livehosts option in the terminal window. I don't want to execute my livehosts scanning twice. So after that, I call the port_scan function, which I will show you in a moment. If you only specify the port_scan option in your terminal window, I'm executing the livehosts scan first to identify the live IP addresses. After that, I call the port_scan function. Let's check it out. In the port_scan_live_hosts function, I call the pen_test method in the Core class for each IP address stored in the global variable, LIVE_HOSTS. So where is this coming from, right? The answer is easy. If I scroll up to the Core class, I added few lines of code to the execute commands function, a simple if statement that checks if I'm using the LIVE_HOSTS option. If the answer is yes, then I look for the phrase 1 host up in the results. This will reveal to me if the host is up or down. If it's live, then I add the IP address to the global variable LIVE_HOSTS. Remember that we will use this variable in the next module when we want to exploit machines as well. Now it's the time to see things in action. Open the terminal window and execute pat. py with the port scan options this time. You don't need to specify the livehosts as I showed you earlier in the source code because it will detect it and execute it automatically. After the scan has finished, if you change the output, you can see that the application executes the livehosts first before wasting the time to scan it. Indeed, this machine is up and running. So Nmap will execute the -A switch to aggressively scan my desktop machine that I'm using to record this module. I hope it's not going to blow up in my face while I'm recording this to you guys. It looks that the scan has identified successfully the ports and services and the operating system as well. I love Nmap.
Summary
In this module, you and I saw how to implement the port scanning techniques in the pen tester automation tool, pat. py, and you learned that by using a proof of concept project how simple it is to implement a change in a big application like this. Let's continue our exciting trip to the next module and implement some exploitation logic, my friends, to make our application better than before.
Attack Automation
Introduction
This is the fifth module in this course, and you are going to learn how to implement the attack logic in the pen tester automation tool pat. py. After all the hard work that you did in the previous module to find the live hosts and perform a port scan, you will see that in this module you will celebrate by attacking those vulnerable machines. Let's start.
Module Overview
Having an idea of what you are going to learn in this module will make your life easier. Let's take a quick look on the workflow of this module. In the previous one, you saw how to use pat. py to scan for live hosts. If the machine is up and running, then the application will scan the port numbers and their associated services. In this module, you will learn how to extract those port numbers and use it to scan for vulnerabilities and attack credentials using some sort of a dictionary file. Let's get started.
Ports Extraction
I will use once again a sandbox project to extract the port numbers from scan results. Let's check it out. I know in advance that I will be using in this example regular expressions, so I will start by importing the regular expression library called re. Next, I will create a variable that holds a sample output from a previous Nmap scan. Look closely at this sample. I should be able to extract the port numbers 1723, 23, 53, and 80. Now it's the time to implement the logic of extraction by using the findall function in the regular expression library. In the first argument, I'm using a regular expression rule \d, followed by *, then /tcp. All I'm doing here is telling this function that I won't put extract from the output any digit numbers, which is represented by the \d*, then followed by a/TCP. Finally, I'm printing the output results and I'm expecting to see all the port number stored in the RA. First, save the file to your desired location, then go to the Debug menu and choose Set Current as Main Debug File. Finally, click on the green button to execute the application. Check this out. As expected, we can see all the port numbers extracted in an RA list format. One more thing to note here, sometimes I realize that we have duplicates of port numbers. To avoid this problem, I will add the set function before calling the findall method. This will make sure that I have unique port numbers. Ctrl+S to save it and click on the green button to run. And it works with no doubt.
Vulnerabilities Scan
Let's go and check together how to implement our own vulnerability scanner. For the commands in this example, you can see we have a bunch of them. I added a different command, third port number, which gives me the opportunity to run the scan on each port number found during the port scan phase. Let's jump and check the source code. As usually, things start here in the initialize function. This is the first thing that the penetration tester will enter in his terminal window, the exploit option. If this option is specified, then the next step is to implement the logic in the process_argument function. Here, I'm checking for the exploit argument existence. If yes, then I check if the IP address is supplied. Right after this, I run the livehost scan that we did in the previous module to check if the host is up and running before proceeding further. Next, I scan the ports. And finally, I call the function attack_live_host to start the action. Let's jump into this function and check what it's doing, if we have any live_hosts. If yes, then I look through all the IP addresses and start the attack phase in the Core class. In this class, I made a few changes. The first one is in the start function. As you can see, I added an if statement to check. If it's an exploit call, then I point to the attack machine function to execute the commands. This function starts by extracting the port numbers from the current IP address. Do you remember the port project that we created earlier in this module? Well if you scroll to the execute_commands function, you will see that after a portscan, I extracted the port numbers and stored the results in the global variable, LIVE_HOSTS. Let's go back and see what's next. So for each item, I extract the port number by excluding the /dcp from the value by using the split function. Next, I tried to find if this port number has a command value in the commands file. If yes, then I extract the command. After this, I'll try to get the application name from a dictionary that I created in the Utility class. This is a simple dictionary list that matches the port number with its application_service_name. Now to get it a value for the service name, you need to pass through the get_application_service_name function first and this is what we're doing in our function. After getting the application name, we call the get_terminal_output function to execute the command and we're done. Let's test it in the terminal window. This time, I will scan the metasploitable machine with the IP address, 10. 0. 0. 124, that I have on my network by specifying the exploit option and pressing Enter. Let it run for a while, and as usual, you have to be patient before the whole process finishes up. Of course, I'm speeding the whole process here. It looks like that the scan has finished. So let's check the results. I have to scroll up in this long results list. As you can see, the first thing that the scan has determined is that the machine is up and running. Next, the port scan identified the port numbers that this machine use. There is a bunch of them on the metasploitable machine. After this step, our application will start executing the commands for each port number. The first one is DNS, the second on is SSH, Telnet, SMTP, a long one for SMTP, after that, AGP, or in my registry, X11, SMB, MySQL, it's a rich list, VNC and FTP. I know in advance that this machine has some interesting information on the FTP server. Check this out. It is vulnerable with a back door because of its FTP server version installed and it supports anonymous login as well.
Brute-force Attack
Let's continue and implement the brute force logic for this application. For the commands, I'm using hydra to attack the most popular port numbers like FTP, SSH, Telnet, remote desktop, Microsoft SQL Server, and MySQL. As you can see, I'm using three valuables for each command. The first one is the list of usernames file paths. The second one is a password's dictionary file paths. And the third one is the IP address of the victim machine. So where are these files coming from? Right? Don't look too far. It's in the same folder. Here, we have the passwords. txt file and the users. txt file. For the sake of this demo, I renamed the original passwords file to passwords1 because a dictionary attack takes a really long time to execute. I created a simple dictionary file with few number of passwords, comparing to this huge of list of passwords. Let's jump into the source code and check the implementation of this scenario. In the initialize function, I created a new parameter for the brute force attack. Next, in the process arguments function, I check if this option exists. Then I proceed the same way I did for the exploit step, but with a minor change. By specifying that I'm using the bruteforce technique as a parameter for the attack_live_hosts function. The attack function is still the same that you just saw in the previous demo. No changes required. Now let's move to the Core class and check the changes there. In this class, I had to patch the get_commands_from_file function. Here, I added an if statement that checks for the bruteforce option, then I construct the command in a tricky way. After the split function, I had an issue because the command had another separator. Let's take a quick look at the commands file one more time. I have two colon characters, so the split function will take everything after and stores it in the next RA index. Let's go back to the source code and see how I fixed it. Here, I used the remainder in the RA and appended to the original command. One more fine note change. In the inject_parameter_in_file function here, I added two new entries to handle the paths for the users and the dictionary file. And we're done, folks. Now it's time to test this monster. In the terminal window, I will use pat. py to attack my poor metasploitable machine on the IP address, 10. 0. 0. 124, followed by the bruteforce option. Remember that I created a fake passwords file for this test. The real one contains tons of passwords and it will take few days to attack a single host. Believe me, you don't want to wait that long. The scan has finished, so let's check the program output. As usual, first the livehost detected that this machine is up and running. Next comes support scanning. The program tried to bruteforce the SSH service with no luck. That's normal because we don't have enough passwords to try. Next, the application tried was Telnet, MySQL, and FTP with no luck as well. I know that the hacker instinct is telling you I want to see more passwords cracking. I encourage you to try the real passwords file on the machine that you own, but be prepared this could take days to crack an account. Good luck.
Summary
One more module and we finish this course. Before you jump to the next one, let's have a quick look of what we covered in this module. First, you saw how to extract the port numbers using a sandbox project. Next, we scanned for vulnerabilities. And finally, you saw how to implement the brute-force technique to attack the credentials on the victim machines.
Web Application Security Automation
Introduction
This is it. The last module of this course. Before I start talking about this module, I would like to take the chance to talk about an important subject. I want to congratulate you by being a leader. Are you asking yourself, me a leader? Yes, I'm talking about you. Like Jim Rohn says, leaders are readers. By reading a book or watching a video tutorial like this one and listening to audio books, you are investing in the most expensive thing and it's your brain. Now let's talk about this module. Web Application Security scanning. Generally, when I test a web application, I use the most of the time Burp Suite Pro to fully conduct a dynamic code analysis. But before starting to use Burp, I gather some information using different tools. Here's some techniques that I use personally to gather information before starting to use Burp. First, I check to see if there's a WAF on the production environment. And in fact, this is useless if you're testing on a staging or a dev environment. Second, I test the security of HTTPS hoping that my client uses a secure socket layer to protect his website. Third, I check for the existence of a load balancer that protects the availability of the server. A load balancer is good in case of a denial of service attack. And finally, I conduct a vulnerability assessment just to have an idea, an advance of the web server conditions. In this module, we will automate these four steps before you start using Burp. The web application test is not done yet. Sometimes I conduct an infrastructure test using a port scanner like Nmap that we did earlier in this course and a vulnerability scanner like Nessus or Nexpose. Finally, an important test for web application is static code analysis using Veracode or Checkmarx.
Github Project
This application is officially published on GitHub. Open your browser and go to the following URL, github. com/GusKhawaja/pat. In the home page, you will see the release version 1. 0, and if you scroll down, you will have a good and a simple list of examples of how to use the application. This is going to be the place for future versions as well. I'm assuming you have exercised all the modules in this course and watched them so many times and probably dreamed about this application. A word of wisdom, repetition is the secret of learning. Feel free to add your favorite commands to the list that we created in this course. Test it and try it in your environment, with your colleagues, and probably your friends who share the same passion as you. If you're very excited and want to contribute to this project, I encourage you to contact me using either my website contact page, by email, or Twitter and I will be glad to put your ideas on top of my list. It's important to know that I'm a busy guy. Don't expect this to happen in one night. You need to be patient with me and I will reply to your message as soon as I'm available. Let's go to the next demo and implement the web automation scan using pat. py.
Web Application Security Scanner
At this level, you should be familiar of the application infrastructure and adding a custom functionality should be a piece of cake for you. Let's dive deep into the source code for the implementation of the web application scanner. In the root folder of the application, I created a new folder called Web. That will hold the commands file of this phase. For each scanner, I created a separate file that contains all the commands to accomplish the required tasks. Let's start with the loadbalancer_commands. In this file, I'm using only a single application command, the lbd, followed by the url of the website to test. The second file I created is the one that should test the HTTPS security. Here, I used three application commands to test this phase. The first one is Nmap, the second one is ssl scan, and the third one is sslyze. Let's close this one and open the waf_commands. In this file, the first command I'm using is Nmap and the second one is the Wafw00f application. Finally, for the web vulnerabilities scan, I'm using Nikto to scan the web application for the existence of well-known vulnerabilities. If you think that Nikto is not enough, feel free to add your own favorite commands. Let's jump into the source code and check the changes for the web scanner. I'm sure you know by heart where you should start the changes. It's the initialize argument function. Here, I added five arguments, one for the website url, a second for the waf test, a third for the ssl test, then for the load balancing test, and finally, for the web vulnerabilities test. Speaking of vulnerabilities test, I should warn you that I renamed the exploit option that we created in the previous module to vulns. Why? Because I found that we were doing a vulnerability assessment, so it's better to call it like this, instead of exploit in case this application will automate the exploitation phase after an assessment in future versions. Now in the process_arguments function, I added the condition statement for each case, one for the waf, one for the ssl, one for the load balancer, and finally a condition for the web vulnerability assessment and that's it. I didn't need to make any changes in the Core class, except the renaming of the exploit option that I did in the previous module. Let's jump to the next demo and test this application.
Demo
Let's test the web application scanner. Open your terminal window and start with Python keyword followed by the application name pat. py and don't forget the company name argument followed by the url of the website that you want to test. And in this example, I will use all the commands together. The waf option followed by the ssl, then loadbalance, and finally, the webvulns option. And finally, press Enter. How long this is going to take, I don't know. It depends on your machine and your network speed. Don't forget that this is a live website in production, not a local one. Be prepared for some patience. Go and get a coffee if you want. In my case, I'm speeding the whole process so you don't wait for a long period of time. It looks like that the application has finished the scan. I will scroll up through the long list to check the results. The Nmap scan has detected the existence of a web application firewall. But the wafw00f application did not detect the existence of a waf. Now you understand the importance of executing multiple tools for the certainty of your work results. Don't count on a single tool in your report. For the HTTPS, it looks like that we have a timeout and the reason is probably the hosting provider is blocking my scan that could lead to false negatives. Now for the load balancing part, it looks like that the scanner has found an HTTP load balancer. Finally, check this out. A pretty secure environment like my website will block you from scanning for vulnerabilities. And this is what you should be expecting from your client. Don't take it personally. If you don't find vulnerabilities, it could happen sometimes that your client website is well secure.
Course Summary
The final moments of this course, let's check together what we have covered in this exciting course. At the beginning, we built together the infrastructure of the application. Then we started the implementation of the information gathering automation. After that, we saw how to automate the scanning phase. Next, we automated the vulnerability assessment phase. And finally, in this module, you saw how to automate the web application security scanner. I hope that you enjoyed this course as much as I did and see you in future courses.
0 Comments