Nmap stealth port scanner
*Intro
*Docs
*Download
Security Tools
Good Reading
Security Lists
*Nmap Hackers
*Nmap Dev
*Bugtraq
*Full Disclosure
*Pen Test
*Vuln Dev
*Basics
*More
News
Links
Exploit World
Advertising
About/Contact
Credits


Introduction Documentation Propaganda
Download OS Detection Portability
In The News Related Projects Thanks To
Nmap Version Scanning
By: Fyodor (fyodor@insecure.org)
Written: September 6, 2003; Last modified: $Date: 2003/11/13 00:14:30 $

While Nmap does many things (remote OS detection via TCP/IP fingerprinting, ping sweeps, uptime calculation, protocol scans, etc.), its raison d'être has always been
port scanning. Point Nmap at a remote machine, and it might tell you that ports 25/tcp, 80/tcp, and 53/udp are open. Using its nmap-services database of more than 2,200 "well-known" services, Nmap would explain that those ports probably correspond to a mail server (SMTP), web server (HTTP), and name server (DNS) respectively. This lookup is usually accurate -- the vast majority of daemons listening on port 25 are, in fact, mail servers. But you shouldn't bet your security on this! People can and do run services on strange ports. Perhaps their main web server was already on port 80, so they picked a different port for a staging/test server. Maybe they think hiding a vulnerable service on some obscure port will prevent "evil hackers" from finding it. Even more common lately is that people are choosing ports based not on the service they want to run but based on what will get through the firewall. When ISPs blocked port 80 after major Microsoft IIS worms CodeRed and Nimda, hordes of users responded by moving their personal web servers to different ports. When companies block telnet access due to its horrific security risks, I have seen users simply run telnetd on the secure shell (SSH) port instead.

Even if Nmap is right, and the hypothetical server above is running SMTP, HTTP, and DNS servers, that is not a lot of information. When doing vulnerability assessments of your companies or clients, you really what to know which mail and DNS servers are running, as well as the version number if possible. Having an accurate version number helps dramatically in determining which exploits a server is vulnerable to.

Yet another good reason for determining service/version numbers is that many services share the same port number - making a guess based on the nmap-services table even less accurate. Anyone who has done much scanning knows that you often find services listening on unregistered ports - these are a complete mystery without version detection. In addition, filtered UDP ports often look the same to a simple port scanner as open ports. But if they respond to the service-specific probes sent by Nmap version detection, you know for sure that they are open (and in many cases exactly what is running).

The new Nmap version scanning subsystem tries to answer all these questions by connecting to open ports and interrogating them for this information using probes that the specific services understand. This allows Nmap to give a much more details assessment of what is really running, rather than just what port numbers are open. Here is a real example:

# nmap -A -T4 -F www.insecure.org

Starting nmap 3.40PVT16 ( http://www.insecure.org/nmap/ ) at 2003-09-06 19:49 PDT
Interesting ports on www.insecure.org (205.217.153.53):
(The 1206 ports scanned but not shown below are in state: filtered)
PORT    STATE  SERVICE VERSION
22/tcp  open   ssh     OpenSSH 3.1p1 (protocol 1.99)
25/tcp  open   smtp    Qmail smtpd
53/tcp  open   domain  ISC Bind 9.2.1
80/tcp  open   http    Apache httpd 2.0.39 ((Unix) mod_perl/1.99_07-dev Perl/v5.6.1)
113/tcp closed auth
Device type: general purpose
Running: Linux 2.4.X|2.5.X
OS details: Linux Kernel 2.4.0 - 2.5.20
Uptime 108.307 days (since Wed May 21 12:27:44 2003)

Nmap run completed -- 1 IP address (1 host up) scanned in 34.962 seconds

Now I don't claim that Nmap is the first program to ever implement this sort of port interrogation. Jay Freeman (AKA Saurik) posted an Nmap patch he calls Nmap+V more than three years ago. Even if Nmap+V was rather slow and cryptic at the time, it demonstrated the value of advanced port interrogation. It has improved substantially since then. There is also the excellent THC Amap, and Nessus has limited service detection capabilities. While we could have saved months of work by simply integrating one of these open source implementations, none of them met all of our goals. In particular, the new Nmap version scanning subsystem offers:

  • High speed, parallel operation via non-blocking sockets and a probe/match definition grammar designed for efficient yet powerful implementation.
  • Determines the application name and version number where available -- not just the service protocol.
  • Supports both the TCP and UDP protocols, as well as both textual ASCII and packed binary services.
  • Multi-platform support, including Linux, Windows, Mac OS X, FreeBSD/NetBSD/OpenBSD, Solaris, and all the other platforms Nmap is known to work on.
  • If SSL is detected, Nmap will connect using OpenSSL (if available) and try to determine what service is listening behind the encryption. This allows it to discover services like https, pop3s, imaps, etc. as well as providing version details.
  • If a SunRPC service is discovered, Nmap will launch its brute-force RPC grinder to find the program number, name, and version number.
  • IPv6 is supported, including TCP, UDP, and SSL over TCP.
  • Community contributions - If Nmap gets data back from a service that it doesn't recognize, a "service fingerprint" will be printed along with a submission URL. This system is patterned after the extremely successful Nmap OS Detection fingerprint submission process. New probes and corrections can also be submitted -- please see the How To Contribute section.
  • Comprehensive database - Thanks to a very dedicated beta team, Nmap is releasing with over 300 service signatures! Some of the services already detected include:
    chargen, cvspserver, daytime, domain, echo, exec, finger, font-service, ftp, ftp-proxy, http, http-proxy, hylafax, ident, ident, imap, imaps, ipp, ircbot, ircd, irc-proxy, issrealsecure, landesk-rc, ldap, meetingmaker, microsoft-ds, msrpc, mud, mysql, ncacn_http, ncp, netbios-ns, netbios-ssn, netsaint, netwareip, nntp, nsclient, oracle-tns, pcanywheredata, pop3, pop3s, postgres, printer, qotd, redcarpet, rlogind, rpc, rsync, rtsp, shell, smtp, snpp, spamd, ssc-agent, ssh, ssl, telnet, time, upnp, uucp, vnc, vnc-http, webster, whois, winshell, X11 .
    This list does not include the many specific RPC programs detected, nor most of the SSL-wrapped services (e.g. https, imaps, etc.). We hope that this will grow substantially post-release thanks to community contributions. If Nmap users were able to help each other by submitting this many unique devices and OS versions for TCP/IP fingerprinting, imagine how many unique services you guys have access to!
  • Particular care was taken to implement these functions securely to avoid buffer overflows, format string vulnerabilities, terminal control character printing, etc. I don't want Nmap to be vulnerable to the type of exploits that have been used against other version detection systems. I always assume that the remote servers will send the most malicious data possible.
  • The new system is reasonably well documented - this paper is evidence that we at least try :).

A few people have asked why this functionality is being integrated into Nmap rather than creating a new standalone tool. We believe that service detection should be a core feature of Nmap for these reasons.

Usage / Examples

Before delving into the technical details of how version detection is implemented, here are some examples demonstrating its usage and capabilities. To enable version detection, just add -sV to whatever Nmap flags you normally use. Or use the new '-A' option, which turns on OS detection (-O) too (and may enable other advanced and aggressive features later). It is really that simple. If you were looking for an Nmap feature that is more challenging to use, read our
Idlescan paper instead :). Here is an example:

# nmap -A -T4 -F www.microsoft.com

Starting nmap 3.40PVT16 ( http://www.insecure.org/nmap/ ) at 2003-09-07 01:45 PDT
Interesting ports on 80.67.68.30:
(The 1208 ports scanned but not shown below are in state: closed)
PORT    STATE    SERVICE     VERSION
22/tcp  open     ssh         Akamai-I SSH (protocol 1.5)
80/tcp  open     http        AkamiGHost (Akamai's HTTP Acceleration/Mirror service)
443/tcp open     ssl/http    AkamiGHost (Akamai's HTTP Acceleration/Mirror service)
Device type: general purpose
Running: Linux 2.1.X|2.2.X
OS details: Linux 2.1.19 - 2.2.25
Uptime 22.924 days (since Fri Aug 15 03:34:27 2003)

Nmap run completed -- 1 IP address (1 host up) scanned in 19.223 seconds

This preceding scan demonstrates a couple things. First of all, it is gratifying to see WWW.Microsoft.Com served off one of Akamai's Linux boxes :). More relevant to this paper is that the "service" for port 443 is "ssl/http". That means that service detection first discovered that the port was SSL, then it loaded up OpenSSL and performed service detection again through SSL connections to discover a web server running AkamiGHost behind the encryption. For people unfamiliar with Nmap, -T4 causes Nmap to go faster (more aggressive timing) and -F tells Nmap to scan only ports registered in nmap-services.

And here is longer and more diverse example:

./nmap -A -T4 localhost 

Starting nmap 3.40PVT16 ( http://www.insecure.org/nmap/ ) at 2003-09-07 02:56 PDT
Interesting ports on felix (127.0.0.1):
(The 1640 ports scanned but not shown below are in state: closed)
PORT     STATE SERVICE    VERSION
21/tcp   open  ftp        WU-FTPD wu-2.6.1-20
22/tcp   open  ssh        OpenSSH 3.1p1 (protocol 1.99)
53/tcp   open  domain     ISC Bind 9.2.1
79/tcp   open  finger     Linux fingerd
111/tcp  open  rpcbind    2 (rpc #100000)
443/tcp  open  ssl/http   Apache httpd 2.0.39 ((Unix) mod_perl/1.99_04-dev [cut])
515/tcp  open  printer
631/tcp  open  ipp        CUPS 1.1
953/tcp  open  rndc?
5000/tcp open  ssl/ftp    WU-FTPD wu-2.6.1-20
5001/tcp open  ssl/ssh    OpenSSH 3.1p1 (protocol 1.99)
5002/tcp open  ssl/domain ISC Bind 9.2.1
5003/tcp open  ssl/finger Linux fingerd
6000/tcp open  X11        (access denied)
8000/tcp open  http-proxy Junkbuster webproxy
8080/tcp open  http       Apache httpd 2.0.39 ((Unix) mod_perl/1.99_04-dev [cut])
8081/tcp open  http       Apache httpd 2.0.39 ((Unix) mod_perl/1.99_04-dev [cut])
Device type: general purpose
Running: Linux 2.4.X|2.5.X
OS details: Linux Kernel 2.4.0 - 2.5.20
Uptime 8.653 days (since Fri Aug 29 11:16:40 2003)

Nmap run completed -- 1 IP address (1 host up) scanned in 42.494 seconds

You can see here the way RPC services are treated, with the brute force RPC scanner being used to determine that port 111 is rpcbind version 2. You may also notice that port 515 gives the service as "printer", but that version column is empty. This means that Nmap did determine the service name via its probing, but wasn't able to determine anything else. On the other hand, port 953 gives the service as "rndc?". The question mark tells us that Nmap was not even able to determine the service name through probing. As a fallback, rndc is mentioned because that has port 953 registered in nmap-services. Unfortunately, none of Nmap's probes elicited any sort of response from rndc. If they had, Nmap would have printed a service fingerprint and a submission URL so that it could be recognized in the next version. As it is, a special probe will be needed. One might even be available by the time you read this. See the community contributions section for details on writing your own probes.

It is also worth noting that some services provide much more information than just the version number. Examples above include whether X11 permits connections, the SSH protocol number, and the Apache module versions list. I even had to chop off some of the Apache modules (where it says "[cut]") to avoid widening this page by the verbosity.

A couple early reviewers questioned my sanity for running services like SSH and finger over SSL. I was actually just having fun with stunnel while ensuring that parallel SSL scans actually work :).

Technique

The version scanning implementation is actually pretty straightforward. I tried to keep it as simple as possible while still being powerful enough to scale well, run quickly, and provide accurate results. If you truly want the nitty-gritty details, download the
source code. But here is a synopsis.

Nmap first does a port scan as per your instructions, and then passes all the open TCP and/or UDP ports to the service scanning module. Those ports are then interrogated in parallel, although we can ignore that detail for this discussion and describe how a single port is interrogated:

  1. If the port is TCP, Nmap starts by connecting to it.
  2. Once the TCP connection is made, Nmap just listens for roughly 5 seconds. Many common services, including most ftp, ssh, smtp, telnet, pop3, and imap servers, identify themselves in an initial welcome banner. Nmap refers to this as the "NULL probe", because Nmap just listens for responses without sending any probe data. If any data is received, Nmap compares it to hundreds of signature regular expressions in its nmap-service-probes file. If the service is fully identified, we are done with that port! The regular expression includes substrings that can be used to pick version numbers out of the response. In some cases, Nmap will get a "soft match" on the service type, but no version info. In that case, Nmap will continue but only send probes that are known to recognize the soft-matched service type.
  3. This stage is where Nmap UDP probes start, and TCP connections end up if the NULL probe above fails or soft-matches. Since the reality is that most ports are used by the service they are registered to, every probe has a list of port numbers that are considered "good bets". For example, the probe called GetRequest that recognizes web servers (among other services) lists 80-85, 8000-8010, and 8080-8085 as probable ports. Nmap sequentially executes the probe(s) which match the port number being scanned. Each probe includes a string (can be arbitrary text or \xHH escaped binary) which is sent to the port. Responses that come back are compared to a list of regular expressions as discussed in the NULL probe description above. Again, there can either be a full match (ends processing), a soft match (limits future probes to a certain service), or no match at all.
  4. In most cases, we expect the "NULL probe" or the probable port probe(s) ( there is usually just one) will match the service. Since the NULL "probe" shares its connection with the probable port probe, this allows service detection to be done with only one brief connection in most cases. With UDP only one packet is usually required. But should the NULL probe and probable port probe(s) fail, Nmap will go through all the existing probes sequentially. In that case of TCP, Nmap must make a new connection for each probe to avoid having previous probes corrupt the results. This worst-case scenario can take a bit of time, although we try to limit the pain by having most probes be generic enough to match many services. For example, the GenericLines probe sends two blank lines ("\r\n\r\n") to the service. As of release time, that matches daemons of 10 service types, including ftp, ident, pop3, uucp, postgres, and whois. The GetRequest probe matches even more service types. Other examples include "help\r\n" and generic RPC and MS SMB probes. In addition, any softmatch will reduce the number of tried probes dramatically.
  5. One of the probes tests whether the target port is running SSL. If so (and if OpenSSL is available), Nmap will connect back via SSL and restart the service scan to determine what is listening behind the encryption. A special directive allows for different probable ports for normal and SSL tunneled connections. For example, Nmap should start against port 443 (https) with an SSL probe. But after SSL is detected and enabled, Nmap should try the GetRequest probe against port 443.
  6. If at least one of the probes elicits some sort of response, yet Nmap is unable to recognize the service, the response content will be printed to the user in the form of a "fingerprint" that can be submitted at a provided URL for the next version of Nmap.
  7. A few select services may have a post-processing step to determine more information. For example, any RPC service found will be subjected to the Nmap RPC grinder (-sR) in order to determine the program number and version info. A very limited number of other post-processors may be added later.

If the English description above is not clear enough, you can see for yourself how it works by adding the --version_trace and -d (debugging) options to your Nmap command line. This will show all the connect/send/recv activity of the service scan.

File Format

As with
remote OS detection, Nmap uses a flat file to store the version detection probes and match strings. This discussion of the nmap-service-probes file format assumes you have already ready the technique section above.

As with many UNIX files, nmap-service-probes is line-oriented. Lines starting with a hash (#) are treated as comments and ignored by the parser. Blank lines are ignored as well. Other lines must contain one of the following directives:

Probe <protocol> <probename> <probesendstring>
Examples:
Probe TCP GetRequest q|GET / HTTP/1.0\r\n\r\n|
Probe UDP DNSStatusRequest q|\0\0\x10\0\0\0\0\0\0\0\0\0|
Probe TCP NULL q||

The probe directive tells Nmap what string to send to recognize various services. The arguments are as follows:
protocol:This must be either "TCP" or "UDP". Nmap will only use probes that match the protocol of the service it is trying to scan.
probename:This is a human-readable name for the probe. It is used in service fingerprints to describe which probes elicited responses.
probestring:Tells Nmap what to send. It must start with a q, then a delimiter character which begins and ends the string. Between the delimiter characters is the string that is actually sent. It is formatted similarly to a C or Perl string in that it allows the following standard escape characters: \\ \0 \a \b \f \n \r \t \v \xHH . Exactly one Probe line in nmap-service-probes has an empty probestring, as shown in one of the examples above. This is the TCP NULL probe which just listens for the initial banners that many services send.

match <service> <pattern> [versioninfo]
Examples:
match ftp m/^220.*Welcome to PureFTPd (\d\S+)/ v/PureFTPd/$1//
match ssh m/^SSH-([.\d]+)-OpenSSH_(\S+)/ v/OpenSSH/$2/protocol $1/
match mysql m/^.\0\0\0\n(4\.[-.\w]+)\0...\0/s v/MySQL/$1//
match ssc-agent m|^\0\x1e\0\x06\0\t\0\0$| v/Novell Netware ssc-agent///
match chargen m|@ABCDEFGHIJKLMNOPQRSTUVWXYZ|

The match directive tells Nmap how to recognize services based on responses to the string sent by the previous probe directive. If the pattern matches, an optional version specifier builds the application name, version number, and additional info for Nmap to report. The arguments to this directive follow:
service:This is simply the service name that the pattern matches. Examples would be ssh, smtp, http, or SNMP.
pattern:This pattern is used to determine whether the response received matches the service given in the previous parameter. The format is like Perl, with the syntax being "m/[regex]/[opts]". The "m" tells Nmap that a match string is beginning. The forward slash ('/') is a delimiter, which can be substituted by almost any printable character as long as the second slash is also replaced to match. The regex is a perl-style regular expression, as documented here. This is made possible by the excellent Perl Compatible Regular Expressions (PCRE) library. The only options currently supported are 'i', which makes a match case-insensitive and 's' which includes newlines in the '.' specifier. As you might expect, these two options have the same semantics as in Perl.
versioninfo:This field is of the form v/vendorproductname/version/info/ where the slash can be replace by any delimiter character. Any of the 3 fields can be empty, and the whole argument can be omitted if no further information on the service is available. The vendorproductname includes the vendor and often service name when relevant and is of the form "Sun Solaris rexecd", "ISC Bind named", or "Apache httpd". The version string is the version "number" (may include non-numeric characters, and even multiple words), while "info" is miscellaneous further information that was immediately available and might be useful (like whether an X server is open, or the protocol number of ssh servers).

softmatch <service> <pattern>
Examples:
softmatch ftp m/^220 [-.\w ]+ftp.*\r\n$/i
softmatch smtp m|^220 [-.\w ]+SMTP.*\r\n|
softmatch pop3 m|^\+OK [-\[\]\(\)!,/+:<>@.\w ]+\r\n$|

The softmatch directive is similar in format to the match directive discussed above. The main difference is that scanning continues after a softmatch, it is just limited to probes that are known to match the given service. This allows for a normal ("hard") match to be found later, which may provide useful version information. See the technique section above for more details on how this works. Arguments are not defined here because they are the same as for 'match' above, except that there is never a versioninfo argument.

ports <portlist>
Examples:
ports 21,43,110,113,199,505,540,1248,5432,30444
ports 111,4045,32750-32810,38978

This line tells Nmap what ports the services identified by this Probe are commonly found on. The syntax is a slightly simplified version of that taken by the Nmap -p option. See the examples above. More details on how this works are in the technique section of this paper.

sslports <portlist>
Example:
sslports 443

This is the same as 'ports' directive described above, except that these ports are often used to wrap a service in SSL. For example, the HTTP probe declares 'sslports 443' and SMTP-detecting probes have an 'sslports 465' line. This list portlist format is the same as with 'ports'.

totalwaitms <milliseconds>
Example:
totalwaitms 5000

This rarely-necessary directive specifies the amount of time Nmap should wait before giving up on a particular probe against a particular service. You can skip this directive to use the Nmap default.

Community Contributions

No matter how technically advanced a service detection framework is, it will be nearly useless unless complemented by a comprehensive database of services to match against. This is where Insecure.Org needs your help. While our lab is pretty substantial by geek standards, we can never hope to run more than a tiny percentage of what is out there. Fortunately experience with OS detection fingerprints has shown that Nmap users together run all of the common stuff, plus a staggering array of bizarre equipment as well :). The Nmap OS Fingerprint Database contains all sorts of switches, WAPs, VoIP phones, game consoles, UNIX boxes, Windows hosts, printers, routers, PDAs, firewalls, etc. We have worked hard to release with a decent nmap-service-probes database that recognizes hundreds of daemons, but that leaves many thousands left to go! There are three primary ways that the Nmap community can help turn this into an exceptional database:

1) Submit service fingerprints - If a service responds to one or more of Nmap's probes and yet Nmap is unable to identify that service, Nmap will print a "service fingerprint" like this one:

SF-Port21-TCP:V=3.40PVT16%D=9/6%Time=3F5A961C%r(NULL,3F,"220\x20stage\x20F
SF:TP\x20server\x20\(Version\x202\.1WU\(1\)\+SCO-2\.6\.1\+-sec\)\x20ready\
SF:.\r\n")%r(GenericLines,81,"220\x20stage\x20FTP\x20server\x20\(Version\x
SF:202\.1WU\(1\)\+SCO-2\.6\.1\+-sec\)\x20ready\.\r\n500\x20'':\x20command\
SF:x20not\x20understood\.\r\n500\x20'':\x20command\x20not\x20understood\.\
SF:r\n");

If you receive such a fingerprint, and are sure you know what daemon version is running on the target host, please submit the fingerprint at the URL Nmap gives you. The whole submission process is anonymous (unless you chose to provide identifying info) and should not take more than a couple minutes. If you are feeling particularly hepful, scan the system again using -d (Nmap sometimes gives longer fingerprints that way) and paste both fingerprints into the fingerprint box on the submission form. Sometimes people read the file format section and submit their own working match lines. This is OK, but please submit the service fingerprint(s) as well - we already have scripts that make integrating and testing them relatively easy.

For those who care, the information in the fingerprint above is port number (21), protocol (TCP), Nmap version (3.40PVT16), date (September 6), UNIX time in hex, and a sequence of probe responses in the form r({probename}, {responselength}, "{responsestring}")

2) Submit corrections - This is another easy way to help improve the database. When integrating a service fingerprint submitted for "chargen on Windows XP" or "FooBar FTP server 3.9.213", I don't really know how general the match will be. Will it also match chargen on Solaris or FooBar FTP 2.7? There is no good way to tell. So I generally use a specific name in the hope that people will notify me if the match needs to be generalized. If you scan a host and the service fingerprint gives an incorrect OS, version number, application name, or even service type, please mail the full Nmap output and correct information to fyodor@insecure.org and I will update nmap-service-probes appropriately.

3) Submit new probes - Suppose Nmap fails to detect a service. If it received a response to any probes at all, you should get a fingerprint that can be submitted as described in #1 above. But what if there is no response and thus a fingerprint isn't available? Create and submit your own probe! These are very welcome. Here are the steps to creating and testing your own probe:

  1. Download the latest version of Nmap and try again - you would feel a bit silly spending time developing a new probe just to find out that it has already been added. Make sure no fingerprint is available - it is better to recognize services using existing probes if possible rather than creating too many new ones. But if the service does not respond to any of the existing probes, there is no other choice.
  2. Decide on a good probe string for recognizing the service. An ideal probe should elicit a response from as many instances of the service as possible, and ideally the responses should be unique enough to differentiate between them. This step is easiest if you understand the protocol very well, so consider reading the relevant RFCs and/or product documentation. One simple approach is to simply start a client for the given service and watch what initial handshaking is done by sniffing the network with Ethereal/tcpdump or connecting to a listening Netcat.
  3. Once you have decided on the proper string, add a the appropriate new Probe line to Nmap (see technique and file format sections). Don't put in any match lines at first, although a 'ports' directive to make this new test go first against the registered ports is OK. Then scan the service again with Nmap a few times. You should get a fingerprint back showing the service's response to your new probe. Send the new probe line and the fingerprints (against different machines if possible, but even a few against the same daemon helps us note differences) to Fyodor at fyodor@insecure.org for possible integration into Nmap. Any details you can provide on the nature of your probe string is helpful as well.
Thanks

While the new service detection feature may have just appeared for most users one day in September 2003, it is the result of months of work and contributions by dozens of dedicated beta tests. I would like to particularly thank the following early beta testers and developers (see the ChangeLog for their specific contributions). I have also listed some of the extraordinary projects these people are working on that might interest the average Nmap user:

Nmap Site Navigation
Introduction Documentation Propaganda
Download OS Detection Portability
In The News Related Projects Thanks To
[ Nmap | Tools | Lists | Reading | News | About/Contact | Advertising | Privacy Policy ]