One key area of information gather is understanding the directory structure of a server and enumerating each directories files. This is commonly called “directory enumeration” or “web site scanner” or sometimes “directory traversal”.

One easy way to do this is by viewing a web servers directory listing/index. Automatic directory listing/indexing is a web server function that lists all of the files within a requested directory if the normal base file (eg index.php) is not present. If a web server has automatic directory listing/indexing turned on then there is a high probability that its administrator[s] are not security conscious. Typically this is turned off.

So then, how can a pen-tester identify directories and files that are hidden from the public? This is where it gets easy. Typically web administrators run on the assumption that if directory listing is turned off and a particular directory is not indexed by search engines that the data will remain hidden. This is not the case.

The following tools are what I consider to be best-of-class for identifying directories and files. Some of the tools are actually methods that can be executed manually and should be used in conjunction with the tools. I have listed these in the order in which I recommend their use. And I recommend using all of them – as is almost always the case with security testing there is no one single golden tool – multiple tools must be used for each phase of your security testing. They will produce duplicate results but they will also return their own unique results. To be thorough use each tool described in this list in the order in which they are listed:

1. Google Dork.

Description: Google has an almost magical way of finding directories, even those that are hidden. You will be surprised with some of the data that Google stores… you will be at a loss to explain how Google finds your web applications hidden directories.

Usage: Use the simple Google Dork search string: site:yoursite.com and hit enter. The resulting links are specific files and directories that Google has indexed within your yoursite.com domain.

Open a blank spreadsheet and start recording the results. Remember, this is just the information gathering phase of your testing. Do not be tempted to dig in and start testing what you find or you’ll never complete this all too important phase.

2. The Web Archive (web.archive.org)

Description: The Internet Archive is a non-profit that was founded to build an Internet library. You can find snapshots of a web site stored monthly going back to (possibly) the month it was launched. What makes the Web Archive useful? Searching through it can help you find directories and files that were indexed in the past but hidden today.

Usage: http://web.archive.org/web/*/http://yoursite.com/*

This will return a list of all directories and files that the Web Archive has stored.

Record each new directory/file that is currently active (click the links to ensure the files exist today) in your spreadsheet.

3. The robots.txt file

Description: Web site administrators use the /robots.txt file to give instructions about their site to web robots such as Microsoft Bing and Google. It works like this: a robot wants to visit a web site URL, say http://www.yoursite.com/welcome.html. Before it does so, it first checks for the existance of /robots.txt to ensure that it should visit that page. In many cases this file will contains specific directories that web robots should not crawl. In essence this file is a double edged sword; it prevents indexing of directories that should remain hidden but it gives away the names of these directories.

Usage: Browse to http://www.yoursite.com/robots.txt. Look for specific directories that are disallowed. The format will look like this:

User-agent: *
Disallow: /cgi-bin/

In the example above the web site is telling the web robots to ignore the cgi-bin directory but potentially telling you that this directory exists. There is a caveat here – some web site administrators will use this file to create a sort of trap door. Knowing that only hackers will attempt to visit such directories they may put in a ‘fake’ directory, capture your IP address when you visit it and, perhaps, ban you from the site.

4. OWASP DirBuster

Description: This is, hands down, my current favorite file/directory enumeration tool. It is light-years beyond any other I have tested. Its powerful, highly customizable and blazinging flast. There is a warning here – it is possible to bring a web server to its knees with DirBuster by bumping up the number of concurrent threads; it supports up to 100.

DirBuster works in a number of different ways. It comes with many default dictionary files (small, medium, large and upper/lowercase) that can be used to perform ‘list based brute force’. However it can also be used in ‘pure brute force’ mode where it scans for files and directories using a character set (a-z, A-Z, 0-9, %20-_) with settings for minimum and maximum length.

Usage: Enter a target url, select the number of threads and choose the type of scan and click start. DirBuster even gives you an estimated number of hours or days left for each running scan.

5. Intellitamper

Description: This tool has been kicking around for a while but has not been updated in years. An initial warning: versions of it mascarade around on the Internet that include malware/spyware so be sure you find yourself a legitimate copy. That being said Intellitamper should definitely be a part of your enumeration arsenal. Intellitamper uses a dictionary file (like DirBuster) to try to find hidden directories and files.

With some web servers it will return false positives and you will see an enormous list of directories that do not actually exist.

Intellitamper does not perform pure brute force, just list based brute force.

Usage: Enter the URL and you are good to go.

6. Wikto

Description: Wikto is a web server assessment tooland it contains a module called back-end miner which is used to enumerate directories and files. Typically this tool does not return a lot of directories but it will surprise you once in a while; it is definitely worth using on a regular basis.

Usage: Enter a URL and you are off and running.

7. Nikto

Description: Nikto is the unix environment cousin to Wikto. If you are running Windows you can use Nikto by installing Active State Perl.

Usage: “perl nikto.pl -h http://yoursite.com”

8. Tennable Nessus

Description: Tennable Nessus is commercial software and it includes a module for directory enumeration. It typically returns far fewer directories than IntelliTamper or DirBuster but it is still worth running this tool each time you want a comprehensive list of directories/files on web server.

Usage: Launch, start the Nessus Server and enter the URL.

9. Paros Proxy

Description: Paros Proxy is a web application security assessment tool. This tool includes a nice spider function that crawls through a web site and creates a pretty comprehensive list of files and directories. This is based on following links and cannot be considered a brute force tool. The tool itself has not been updated since November of 2004.

That being said Paros Proxy should definitely be in your enumeration toolbelt. It is effective and will surprise you with the things it can find.

Usage: Launch Paros Proxy. Open IE/Firefox and configure a proxy toaccess the internet. By default the HTTP proxy should be 127.0.0.1 on port 81. Next, browse to the URL. When the URL appears in Paros Proxy select it and choose Analyze/Spider.

10. http-dir-enum

Description: Last but not least is the perl script http-dir-enum. This program guesses directory names within a website using a wordlist of potential directory names. It supports Basic Authentication HTTP Keep-alive, proxies, cookies and can save the results in XML format.

Usage: Install Active State Perl. Run this from the command line: “perl http-dir-enum.pl -f directory-names.txt http://yoursite.com”

The ten tools described above, when used in conjunction with one another, provide what I believe to be the most comprehensive method of enumerating directories and files.

Do you know of a tool that should have made this list? Let me know!