On one system I started hearing people complain that one server cannot connect to an outside service normally. The connection is opened and gets reset after a while.
When looking at tcpdump traffic from both the client and the server side we saw that the culprit had to be the Check Point firewall in between. As the internal server got a tcp-rst with the source address of the external server and the external one got a tcp-rst with the source address of the internal server. Smart Console logs show that connection is being allowed through and that is it.
So to find out what is going on I turned to zdebug. Zdebug output showed the following line:
[DATE TIME];[cpu_10];[fw4_5];fw_log_drop_ex: Packet proto=6 Internal_Address:47891 -> ExternalHost:443 dropped by fwpslglue_chain Reason: PSL Reject: ASPII_MT;
Googling the “dropped by fwpslglue_chain Reason: PSL Reject: ASPII_MT;” message lead me to sk119432 and that pointed me to towards application control blade which had been activated on that GW by someone previously. When looking at the Application Control policy I found that the particular internal host was not included in the app control policy. So I added the source host involved to the app control policy and the traffic started flowing normally and stopped getting tcp-rst sent to both hosts.
To me it is interesting that all Application Control rules had logging and accounting defined to them, even the final drop rule. And yet no application control blade intercepts were logged. Besides that, the connection was allowed to be opened and always was up for a few seconds before getting reset.