Introduction

This analysis on HTTP headers is separated into two different blog posts :

• describing what HTTP headers are and analyzing the results of the network requests
• analyzing the content of HTTP headers

The separation in two parts follows the logical sequence of events that I had to do to complete the investigation. First I had to map the network and interpret these results and then dive deeper in the returned HTTP header results.

Note that this post contains graphs and tables supported by Google Chart, you’ll have to enable javascript for the graphs and tables.

Summary of the network requests

The detailed results of the network requests can be found below but these are some of the highlights :

• Most Belgian websites are hosted in Belgium, Holland and France at OVH and Combell
• Some Belgian websites are hosted at ‘unusual’ locations like Israel and Panama
• 25% of the IPs hosting a website reply to DNS queries, most often via ISC Bind
• There are still open resolvers
• DNS administrators are creative with the returned ISC Bind versions
• The majority of the websites is hosted on Apache webservers
• 7% of the IPs hosting a website reply to NTP queries
• 4% of the IPs hosting a website reply to SNMP queries with some interesting systems (Cisco, Lexmark)
• Lots of RDP services support 40-bit and 56-bit RC4

What are HTTP headers?

HTTP headers are the central part of HTTP requests and responses. They are sent and received during a web communication and contain the information of your client browser and the requested resource and they return server information (language, content type) to your web browser.

A basic set of HTTP headers sent by a client looks like this

 
GET / HTTP/1.1
Host: www.vanimpe.eu
User-Agent: TestBrowser/1.0

and a basic response of the server to the client looks like this

 
HTTP/1.x 200 OK
Date: Thu, 21 Aug 2014 06:30:45 GMT
Server: Apache/2.2.22

In this example the client instructs the server to use HTTP version 1.1 and requests the root page (/) from the virtual host www.vanimpe.eu. The client identifies itself with TestBrowser/1.0. The server responds with a success message (200) and returns the date and server identification.

HTTP headers are always in the format Name: Value.

Viewing HTTP headers

If you look into the source code of a web page in a browser you will not see the HTTP headers, you will only see the HTML part that comes after the headers.

There are a number of tools and methods that allow you to view the HTTP headers. For example

• HttpRequester
• Live HTTP Headers
• Firebug
• Or via telnet with telnet host 80 GET / HTTP/1.0

Analyzing HTTP headers

There is no single source of information that provides an overview of most popular HTTP headers. I decided to make my own overview of HTTP headers based on the responses of the most popular Belgian websites.

Because I was doing the mapping anyway I also included verifying for open DNS, NTP, SNMP and RDP servers.

In this post I first focus on the network results and then analyze the HTTP headers.

Top Belgian websites

There is no public available list of “Top Belgian Websites”. As a fallback, I downloaded the Alexa Top 1,000,000 Websites from http://s3.amazonaws.com/alexa-static/top-1m.csv.zip. This is an outdated file but it contained enough data for my test.

I unzipped the file, extracted the Belgian URLs and added a ‘www.’ in front of the domain name so it would become useful as a list of websites.

unzip top-1m.csv.zip
cat top-1m.csv| grep "\.be$" | cut -d "," -f 2 | awk '{print "www." $0}' > top-be.csv

I removed doubles (with uniq) and manually added a number of Belgian government sites (.fgov.be). This resulted in a file containing 2940 .be websites.

Scanning for HTTP Headers

For a previous project (scanning for open DNS and NTP servers) I wrote a Python script, NOSS that uses NMAP to do the scanning and then imports everything into a sqlite database. I extended this script to scan for HTTP headers.

Basically what you have to do is run the scan, initialize the database and import the results.

sudo ./noss.py scan -sH -sn -sr -st -sd -ti top-be.csv 
./noss.py init
./noss.py import

NOSS uses the NMAP script http-headers. I set it to add a custom user agent to prevent blocking by unit-testing Web Application Firewalls. The provided user agent is Internet Explorer 10 (Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)). You can also do the HTTP requests with a mobile user agent with the option “–scan-mobile-user-agent”. You can alter the user agent in the NOSS source if you want to test for results with different browsers

HTTP_USER_AGENT="Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)"
HTTP_USER_AGENT_MOBILE="Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Mobile/10A5376e"

The information returned by the different services on the hosts is open, publicly accessible information. No “hacking” took place.

Open network ports

Out of the 2940 domain names in the target list, 2487 hosts replied on requests for tcp/53, udp/53, tcp/80, udp/80, tcp/123, udp/123, tcp/161, udp/161, tcp/3389 or udp/3389. The other domains were either no longer resolvable or were not available. In total there were 2348 unique IPs. Three domain names did not reply to a web request but had one of the other services (dns, ntp, snmp, rdp) running.

Out of the total 2487 hosts replying

• 25% responds to a DNS udp/53 query (recursion request for www.wikipedia.org)
• 7% responds to a network time udp/123 query (time request and “read variables”)
• 4% responds to an SNMP udp/161 query
• 5% responds to a Remote Desktop tcp/3389 query

Network location

In total there were 2348 unique IPs. Because the mapping contained 2487 hostnames, the difference in numbers means that a number of sites were hosted on the same IP.

One host in the list of domain resolved to 127.0.0.1. The majority of the hosts were (according to the whois service of Team Cymru) located in Belgium and Holland.

Seven Belgian websites are hosted in Israel, one in Macedonia and four websites in Panama.

Note that this data is based on the IP, if multiple domain names resolve to the same IP they are only counted once.

The Team Cymru whois data also provides the name of the network block. A substantial number of websites are hosted at OVH (in France). The majority of sites in Belgium is hosted at Combell. The high number of sites hosted on the Belnet network is due to the government sites included in the probes.

Open DNS Servers (tcp/53 & udp/53)

Open DNS recursive servers

Eight DNS servers replied as a DNS open resolver, three of them were Microsoft DNS servers. Out of these three Microsoft DNS servers, two hosts were also running Microsoft IIS. The other one was running Apache and was also running a public Microsoft Terminal Service server (the possibility that someone is running Apache on Windows is unlikely, this mix probably indicates the use of port forwarding or a proxy).

The check for open resolver is done via the NMAP script that verifies if the server resolves www.wikipedia.org .

It’s hard to understand that after the different awareness campaigns people still run an open recursive DNS service.

DNS server identification

Not every service responding to port 53 returned a usable service product identification. Some DNS servers returned a service fingerprint (servicefp) but not a product identification.

The difference in the number of DNS servers replying on tcp and udp might be explainable due to the fact that some servers act as an authoritative name server allowing zone transfers (tcp). Zone transfer requests were not performed. It might also be the result of insufficient firewall filtering.

Most ISC Bind servers returned their version string but the list also contained some unusual results.

• donuts
• djbdns
• Vroum vroum vroum

Some administrators prevented version leaking and returned something “creative” in the response.

Open Web Servers (tcp/80 & udp/80)

The hosts responding to the requests on udp/80 are Google or Google related web sites (www.google.be, www.youtu.be, *.blogspot.be).

The majority of the web services fingerprinted by NMAP are Apache and Microsoft. Some sites returned an unusual product identification. This is probably because of a load balancer or web proxy in front of the site.

According to the NMAP fingerprint there are 113 Microsoft HTTPAPI httpd servers that support SSDP/UPnP. Further investigation in the raw data (matching the service identification and the HTTP server header) revealed that these are Microsoft-IIS/7.0, 7.5, 8.0 or 8.5 servers. The server header also returned Consultix Webserver, Microsoft-HTTPAPI/2.0 and WebServer for the same NMAP fingerprint.

Open NTP servers (tcp/123 & udp/123)

None of the NTP servers replying on port tcp/123 or udp/123 returned a product identification string.

The NMAP script provides some additional information such as the supported NTP version and if a service is NTP synchronized. Four NTP servers were not synchronized.

Open SNMP Servers (tcp/161 & udp/161)

The 41 SNMP servers replying on tcp/161 did not return a product identification.

Seven SNMP server that replied on udp/161 did not return a product identification. The majority of the SNMP services on udp/161 was provided by net-snmp. There were also three Cisco SNMP and one Lexmark SNMP services publicly available. The three Cisco SNMP hosts also ran a Microsoft web service.

Open RDP Servers (tcp/3389 & udp/3389)

The probes indicate that there are 126 hosts answering on tcp/3389. No host replied on udp/3389.

The supported cyphers can be used to fingerprint a system. Some services were not fingerprinted by NMAP as a Microsoft Terminal Service but the service data field from NMAP did contain a fingerprint that corresponded with a valid RDP response.

The script output had, besides error messages (“Received unhandled packet”, “ERROR: Failed to connect to server” or merely “ERROR”), these different outputs

  Security layer
    CredSSP: SUCCESS

 
  Security layer
    CredSSP: SUCCESS
    Native RDP: SUCCESS
    SSL: SUCCESS
  RDP Encryption level: Client Compatible
    40-bit RC4: SUCCESS
    56-bit RC4: SUCCESS
    128-bit RC4: SUCCESS
    FIPS 140-1: SUCCESS

 
  Security layer
    Native RDP: SUCCESS
  RDP Encryption level: Client Compatible
    40-bit RC4: SUCCESS
    56-bit RC4: SUCCESS
    128-bit RC4: SUCCESS
    FIPS 140-1: SUCCESS

  Security layer
    CredSSP: SUCCESS
    SSL: SUCCESS

  Security layer
    CredSSP: SUCCESS
    Native RDP: SUCCESS
    SSL: SUCCESS
  RDP Encryption level: Unknown
    128-bit RC4: SUCCESS

RDP supports different server authentication and encryption levels.

CredSSP is a security support provider that enables a program to use client-side SSP to delegate user credentials from the client computer to the target server. It can pose a security issue when it’s used for double hopping with Powershell by sending clear credentials when the server connecting to is owned by an attacker. A lot of RDP services also seem to support the weaker 40-bit and 56-bit RC4. If you want to know more about this you should read Security Advisory 2868725: Recommendation to disable RC4 and Attack of the week: RC4 is kind of broken in TLS.

Continue reading

This concludes the first part of this analysis. You can continue on reading an analysis of the content of the HTTP headers in the next part.