Diagnosing Firewall Misbehaviour” with Packet Captures”

Our support team received a call from a customer complaining that their next-generation firewall (NGFW) was intermittently blocking access to their new voting website. As we were in the process of making firewall changes inside their environment and are responsible for the management of their network they turned to us for help.

The website was a vanity domain hosted at GoDaddy that redirected users to a deep link within another website. We accessed the site and confirmed that it was working properly and sending a 301 redirect as expected, it worked properly both inside their and our environment. We reviewed the firewall logs and found no indications that traffic was blocked by any firewall or IPS. The customer insisted that there was an intermittent problem accessing the website that seemed to affect some users at random.

We ran a simple script to access the website 100 times and discovered that a small percentage of the requests experienced errors. There were still no logs for denied connections and  we suspected that the traffic may have been blocked or affected by the firewalls even if there were no log messages for the event. We then tested from a machine outside of both our and our customer’s corporate environments with no firewall present and experienced the same issue.

Increasingly mystified by the issue affecting the website, it was time to use packet captures to understand the problem. It seemed certain that the issue was with GoDaddy’s hosting service but we had to be able to demonstrate and explain this to our customer. A packet capture of the traffic immediately revealed the problem. All of the successful requests had the same form, a traditional TCP handshake (SYN, SYN/ACK, ACK), a get request followed by the 301 redirect response as you see here:

The failed requests, however, were very different. In response to the client’s SYN packet we received an ACK packet without the SYN flag as you can see in the Wireshark output below, the client sends an RST packet in response to this response, then retries to open the connection, in this case successfully.

This response packet violates the standard for how a TCP connection must be established, the behaviour and response of the client to this traffic is dependent on operating system and firewall behaviour. With an NGFW stacked behind a traditional firewall, there were any number of ways that the response could fail to reach the client. Our analysis indicated that the traditional firewall, in this environment an ASA, ignored the packet as unrelated to the previous exchange on this port. 

While the Linux machine without a firewall that we used for the test you see captured here ultimately establishing a successful a connection this didn’t happen consistently in the customer environment. A closer examination of the packet shows that it is unrelated to our request apart from the source port matching ours. After turning off relative sequence numbers in Wireshark (required in some cases when examining unusual traffic) we can see that the sequence / acknowledgement numbers are completely unrelated.

Here is our SYN packet with the sequence number highlighted:

And the ACK response which originates from a completely different sequence:

Additionally, the TTL on this response packet is 55, higher than the TTL of 52 seen in the response from the server in the successful transaction above. This higher TTL suggests that the traffic has come through a shorter path than the correct response; It seems likely that some intermediary device has confused two connections based on the matching source / destination port combination.

We reviewed our findings with our customer and suggested that they either take up the issue with GoDaddy support or consider relocating the website. The packet captures had allowed us to identify and help resolve a mysterious intermittent issue, as well as, address the customer’s legitimate concerns that the firewalls were incorrectly blocking traffic. Whenever first diagnostic measures fail, packet captures are the most effective way to completely understand the cause of a problem. We’d love to hear about your own experiences in the comments.

If you’re interested in support to help you and your team deal with more complex issues or are looking for a more complete managed services solution for your network and security infrastructure please feel free to contact us.