Penetration Testing with Kali Linux OffSec

curl https://www.google.com/robots.txt
User-agent: * 
Disallow: /search 
Allow: /search/about 
Allow: /search/static 
Allow: /search/howsearchworks 
Disallow: /sdch 
Disallow: /groups 
Disallow: /index.html? 
Disallow: /? 
Allow: /?hl= 
... 
Listing 104 - https://www.google.com/robots.txt 
Allow
and 
Disallow
are directives for web crawlers indicating pages or directories that “polite” web 
crawlers may or may not access, respectively. In most cases, the listed pages and directories 
may not be interesting, and some may even be invalid. Nevertheless, sitemap files should not be 
overlooked because they may contain clues about the website layout or other interesting 
information, such as yet-unexplored portions of the target. 
8.3.3
Enumerating and Abusing APIs 
In many cases, our penetration test target is an internally-built, closed-source web application that 
is shipped with a number of 
Application Programming Interfaces
(API). These APIs are 
responsible for interacting with the back-end logic and providing a solid backbone of functions to 
the web application. 
A specific type of API named 
Representational State Transfer
(REST) is used for a variety of 
purposes, including authentication. 
In a typical white-box test scenario, we would receive complete API documentation to help us 
fully map the attack surface. However, when performing a black-box test, we’ll need to discover 
the target’s API ourselves. 
354
(Amazon Web Services, Inc. 2022), 
https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request
-custom-headers-behavior 
355
(Sitemaps.org, 2022), https://www.sitemaps.org/ 

---

Penetration Testing with Kali Linux
PWK - Copyright © 2023 OffSec Services Limited. All rights reserved. 
226 
We can use Gobuster features to brute force the API endpoints. In this test scenario, our API 
gateway web server is listening on port 5001 on 192.168.50.16, so we can attempt a directory 
brute force attack. 
API paths are often followed by a version number, resulting in a pattern such as: 
/api_name/v1 
Listing 105 - API Path Naming Convention 
The API name is often quite descriptive about the feature or data it uses to operate, followed 
directly by the version number. 
With this information, let’s try brute forcing the API paths using a wordlist along with the 
pattern
Gobuster feature. We can call this feature by using the -p option and providing a file with patterns. 
For our test, we’ll create a simple pattern file on our Kali system containing the following text: 
{GOBUSTER}/v1 
{GOBUSTER}/v2 
Listing 106 - Gobuster pattern 
In this example, we are using the “{GOBUSTER}” placeholder to match any word from our wordlist, 
which will be appended with the version number. To keep our test simple, we’ll try with only two 
versions. 
We are now ready to enumerate the API with gobuster using the following command: 
kali@kali:~$