How to Diagnose Web Services Blocked

Problem Background

This case study originates from a friend’s sharing on WeChat. Although similar cases of web services blocked have occurred before, this particular instance offers a richer analysis of packet data. Therefore, I’m documenting the troubleshooting process.

Problem Analysis

The end user reported that the mobile 4G network couldn’t open a website. Troubleshooting involved capturing the relevant access data packets using a laptop as a Wi-Fi hotspot.

Typically, I suggest not using capture filters; instead, one can extract desired packets using display filters on the complete packet file.

IP Session Filtering

Knowing the website’s domain, one can use tools like nslookup to confirm its IP address and then filter all packets exchanged between the source and destination.

ip.addr eq 10.10.10.1 and ip.addr eq 192.168.0.1

After exporting specific groups and saving the filtered packet file, it revealed multiple TCP session streams between the source and destination, as shown below:

The packet file statistics indicate that the number of packets decreased from 387 to 249 after IP session filtering.

$ capinfos -csdl 1226.pcap
File name:           1226.pcap
Packet size limit:   file hdr: 262144 bytes
Number of packets:   387
File size:           85 kB
Data size:           79 kB

$ capinfos -csdl 1226-01.pcapng
File name:           1226-01.pcapng
Packet size limit:   file hdr: (not set)
Number of packets:   249
File size:           54 kB
Data size:           46 kB

TCP Stream Session Filtering

Due to the differing behaviors of the client accessing the server, multiple TCP session streams are generated. By tracing the streams, we confirmed the results.

tcp.stream == <num>

Upon reviewing the streams, we found that the RST anomalies were quite similar. We selected one stream for in-depth analysis, represented as follows:

tcp.stream == 4

Actual Analysis

The TCP session stream clearly displayed RST occurrences. Normally, within the data center (without security devices), RSTs typically originate from one side of the session. However, considering security devices and the fact that this case involves internet access (with an unknown network architecture), we must specifically determine the RST’s source.

1. Analyzing IRTT:
   The IRTT information comes from the TCP three-way handshake and serves as a reference throughout the TCP stream, indicating that the round-trip time between source and destination is approximately 119 ms.
2. RST Return Time:
   To ascertain where the RST originated, we can utilize the RST return time as one of our bases. If the RST comes from the server, it would typically return around 119 ms; however, if it is significantly less than the IRTT value, it may indicate interference from an intermediate device (often a security device). Adding the Delta Time column, the expression is frame.time_delta_displayed. We can see that the RST returned approximately 68 ms after the client’s GET request (packet 51), leading us to suspect it’s not a genuine server response. How can we further validate this? By examining the TTL.
3. TTL:
   TTL can indicate how many hops an IP packet has traversed in the network, so we add the TTL field as a reference. The TCP three-way handshake SYN/ACK packet had a TTL of 112, while the RST packet’s TTL was 61. This indicates they did not originate from the same device. Coupling this with the RST return time, we conclude that the web service access is indeed being blocked by a security device in the internet path.

In-Depth Analysis

Regarding TTL, observant readers may notice the varying TTL values (64, 112, 61, 48) from the same server (10.10.10.1). Why do these discrepancies exist?

Initially, different operating systems and protocols yield different default TTL values, as shown below:

Device/OS	TTL	Notes
Linux	64/255
Windows	128

1. TTL 64:
   Indicates the TTL from the client side, suggesting the packet was captured locally.
2. TTL 61:
   The TTL from the RST packet indicates the security device is roughly three hops away, likely from the nearby ISP’s security policy control.
3. TTL 112 and 48:
   The TTL from the server’s SYN/ACK packet is 112, indicating approximately 16 hops from the client. The TTL of 48 also suggests it originated from the server, likely through a security device acting as a proxy during the TCP handshake, allowing normal data packets to be forwarded to the actual server afterward for protection (e.g., against SYN Flood attacks). Thus, TTL 112 comes from the server-side security device, while TTL 48 is from the actual server.

Problem Summary

By meticulously analyzing each packet field, we can discern the true nature of the fault, facilitating an accurate diagnosis of the issue at hand.