Browsing: Check Point

Some connections getting reset on CP R77.30

On one system I started hearing people complain that one server cannot connect to an outside service normally. The connection is opened and gets reset after a while.

When looking at tcpdump traffic from both the client and the server side we saw that the culprit had to be the Check Point firewall in between. As the internal server got a tcp-rst with the source address of the external server and the external one got a tcp-rst with the source address of the internal server. Smart Console logs show that connection is being allowed through and that is it.

So to find out what is going on I turned to zdebug. Zdebug output showed the following line:

[DATE TIME];[cpu_10];[fw4_5];fw_log_drop_ex: Packet proto=6 Internal_Address:47891 -> ExternalHost:443 dropped by fwpslglue_chain Reason: PSL Reject: ASPII_MT;

Googling the “dropped by fwpslglue_chain Reason: PSL Reject: ASPII_MT;” message lead me to sk119432 and that pointed me to towards application control blade which had been activated on that GW by someone previously. When looking at the Application Control policy I found that the particular internal host was not included in the app control policy. So I added the source host involved to the app control policy and the traffic started flowing normally and stopped getting tcp-rst sent to both hosts.

To me it is interesting that all Application Control rules had logging and accounting defined to them, even the final drop rule. And yet no application control blade intercepts were logged. Besides that, the connection was allowed to be opened and always was up for a few seconds before getting reset.

{ Add a Comment }

“According to the policy the packet should not have been decrypted” after managment upgrade to R80

It seems that Check Point has changed the way that policies are compiled/handled. At one installation I saw a VPN connection break right after the policy was installed the first time from the upgraded management server. Right after that “According to the policy the packet should not have been decrypted” started appearing in the logs and traffic started being dropped.

After a bit of looking around on the Check Point support site about the message it pointed to multiple objects having similar VPN domains. After going through the object database, I managed to find a “copy” of the peer GW object. Basically 2 gateway objects had same external IP address and similar VPN domains defined. Fortunately that “duplicate” entry wasn’t in use so I deleted it. After deleting the duplicate entry and pushing the policy again, the traffic started flowing again.

I wonder why the upgrade verification process showed no errors or warnings. To me it seems they should have shown some sort of warnings about it, as it breaks things.

{ Add a Comment }

Nice feature/bug in Check Point NAT

It seems that I have stumbled upon a interesting Check Point firewall NAT behavior. Namely the firewall does something that it is not ordered to do, it translates the source IP to a different one than in the policy.

Had a static inbound NAT rule where the source address of the client was supposed to be faked to another static address. What happened was that yes the policy installed fine. Yes the client IP address was changed and hidden from the service. But it was changed to an incorrect address. The rule stated that the client IP address be changed to for example 172.16.2.10, but what it was translated to was 172.16.2.100. And the new address did not exist anywhere in the policy database.

And after upgrading that particular management server to R80.20 it refused to compile the policy with that NAT rule in place. Fortunately recreating the NAT rule removed the issue.

The error in the FWM debug logs was for that issue:
“Invalid Object in Source of Address Translation Rule 57. The range size of Original and Translated columns must be the same.”

The interesting thing is, that it was fixed simply by deleting the rule and just re-adding it. And yes it was a 1 address to 1 address translation, not a network to 1 address.

{ Add a Comment }

Upgrading a secondary management server to R80.20 GA issue

The Check Point upgrade guide for R80.20 is quite clear in saying that on a secondary server you should perform a clean install. All is fine and dandy with that. I don’t know if I am the first one to actually run in to a problem with it.

When using the CPUSE clean install feature in the WebUI, it seemed to work. That is after I had resized the partitions to give it enough space to perform the upgrade. It was arguing that 30GB of free space is actually 29.83GB. So, after making the current active install partition smaller all seemed well. Until the reboot that is.

After the server rebooted, I could see the new WebUI and felt excited, that it actually kept the original IP address. As CPUSE had warned that all settings will be wiped that was a nice surprise.

Unfortunately, the nice things ended there. I was unable to log in. The default “admin/admin” combo didn’t work or the previous credentials that i had. Found no hint on the Check Point support site either.

After a bit of googling around ended up just downloading the clean install ISO file and re-installing the machine. If I had used the ISO before instead of the CPUSE, I would have saved a lot of time. I guess that CPUSE clean install feature is not that clean yet.

{ Add a Comment }

No logs after upgrade from R77.30 to R80.20?

Are you getting no logs your Check Point log server after an upgrade from R77.30 to R80.20?

Are you sure you did the “install database” step in your Smart Center right after it came back up? Before doing that, no logs will be indexed besides some Smart Center local messages.

{ Add a Comment }

Check Point 1400 series SMB device VPN debug log fast rotation work-around

If you have ever had to debug VPN-s on a Check Point SMB device you might have noticed that they rotate their logs every 1MB, which means that sometimes You might actually miss the information You were looking for.  At least for me it was a problem trying to get debug level information on some VPN issues that occurred randomly. 

So in order to get the required output I added a 32GB SD-card to the firewall to extend its small storage made some symlinks and wrote a few little script to get all the output I required for debugging.

So on to the details. After you have mounted your SD-card you have access to it on the path:

/mnt/sd

Before You enable debugging You should make symbolic links for the ikev2.xmll and ike.elg files so that you wouldn’t run out of space on the built-in flash.  You can do that by using the following commands:

touch /mnt/sd/ikev2.xmll && touch /mnt/sd/ike.elg
ln -s /opt/fw1/log/ike.elg /mnt/sd/ike.elg
ln -s /op/fw1/log/ikev2.xmll /mnt/sd/ikev2.xmll

Now enable debugging like you usually would(cp support site SK):

vpn debug trunc
vpn debug on TDERROR_ALL_ALL=5

And here is the script I used to copy the logs to the SD-card as they were rotated:

!/bin/bash
while true
do
fmtime=$(stat -c %Y /opt/fw1/log/sfwd.elg.0)
curtime=$(date +%s)
diff=$(echo $curtime-$fmtime|bc)
if test $diff -le 1
then
cp /opt/fw1/log/sfwd.elg.0 /mnt/sd/sfwd.elg-$fmtime
fi
sleep 1
done

So basically, it checks if the sfwd.elg.0 file has changed every second and copies the changed file to the SD-card. I actually also experimented using logger to send the log to a central server via syslog. Using logger just didn’t work. It sent the first one fine, but then the other changes afterwards were just dropped and I opted for the copying. 

{ Add a Comment }

Fixing Smart Dashboard crashing after receiving “Disconnected_Objects already created by another user” error

Today I happened upon an error Smart Dashboard after it randomly crashed and refused to start again. After the crash it started always showing me the error “Disconnected_Objects already created by another user” and crashing again. Quick lookup on Check Point’s support site gave me the idea that SmartMap cache might be corrupted.  So here is a quick copy paste of the commands needed to reset the Smart Map cache in R77.30 on Gaia.

mkdir -p /var/tmp/SmartMap_Backup/
cpstop
cd $FWDIR/conf/SMC_Files/vpe/
mv mdl_version.C /var/tmp/SmartMap_Backup/mdl_version.C
mv objects_graph.mdl /var/tmp/SmartMap_Backup/objects_graph.mdl
cd $FWDIR/conf/
mv applications.C /var/tmp/SmartMap_Backup/applications.C
mv CPMILinksMgr.db /var/tmp/SmartMap_Backup/CPMILinksMgr.db
cpstart

After doing that I was able to start Smart Dashboard again and continue working! 🙂

If you are running your management server on Windows are actually are using Multi-Domain-Server you can find the commands needed to do the same on those systems in “sk92142” which is about “SmartDashboard crashes when loading SmartMap data, after upgrading the Security Management Server “

{ Add a Comment }

Check Point unable to delete IKE/IPSEC SA on a SMB device cluster

On a Check Point SMB 1400 series appliance cluster with R77.20.75 installed I happened to run in to an issue where after changing the peer Gateway’s IP address the VPN did not want to come up again and VPN TU showed me a SA’s relating to the old peer IP address. VPN TU delete command did not remove them. Also disabling the VPN community/removing the gateways from it did nothing, still the stubborn SA’s remained, even waiting for the timeouts to occur did nothing.

What in the end actually removed the stuck SA was doing “cp stop” “cp start” on both of the devices with manual fail over in between. After that VPN TU didn’t show the stuck SA any more and the VPN started working again with the peer’s new IP address.

{ Add a Comment }

Check Point R77.30 management interface crypto hardening (WebUI and SSH Cipher change)

By default the management interfaces (WebUI/SSH) of a Check Point firewall are using crypto settings that are not that great (MD5 and SSLv3, etc are enabled), but fortunately it is possible to change them.

SSH daemon is configured like in a normal Linux Distribution by just editing the /etc/ssh/sshd_config, Check Point in its support site also recommends you also modify the ssh client configuration located in /etc/ssh/ssh_config.  Basically in order to change the encryption algorithms available when connecting to the firewall using ssh add the following lines to the aforementioned configuration files using the vi command in Expert mode:

Ciphers aes256-ctr,aes256-cbc,aes128-ctr,aes192-ctr,aes128-cbc,aes192-cbc
MACs hmac-sha1

After modifying the config file restart the SSH server using the following command:

 service sshd restart

If everything is fine then your connection survives and if for some strange reason your ssh connectivity breaks and you can’t log back in you can undo the previous changes by using the terminal access that you can get in the WebUI.

Now that the SSHD settings have been changed, lets start changing the Cipher suites available for HTTPS used for WebUI. Just connect to command line using SSH and do the following in Expert mode.

  1. Backup the current file /web/templates/httpd-ssl.conf.templ:
    [Expert@HostName:0]# cp /web/templates/httpd-ssl.conf.templ /web/templates/httpd-ssl.conf.templ_ORIGINAL
  2. Edit the current /web/templates/httpd-ssl.conf.templ file:
    [Expert@HostName:0]# vi /web/templates/httpd-ssl.conf.templ
  3.  Find the line containing the SSLCipherSuite parameter and change the values behind it for example to ECDHE-RSA-AES256-SHA384:AES256-SHA256:!ADH:!EXP:RSA:+HIGH:+MEDIUM:!MD5:!LOW:!NULL:!SSLv2:!SSLv3:!eNULL:!aNULL:!RC4
  4. Close the editor by using :wq!  , the ‘!’ in the end will override the fact that the file has read only permissions.
  5. Update the current configuration of HTTPD daemon based on the modified configuration template:
    [Expert@HostName:0]# /bin/template_xlate : /web/templates/httpd-ssl.conf.templ /web/conf/extra/httpd-ssl.conf < /config/active
  6. To activate the configuration changes restart the HTTPD daemon by using the “tellpm” command:
    [Expert@HostName:0]# tellpm process:httpd2
    
    [Expert@HostName:0]# tellpm process:httpd2 t

To find out what you actually want to use as the SSLCipherSuite value you can use the cpopenssl to see what algorithms will be available with which value. Example:

[Expert@HostName:0]# cpopenssl ciphers -v 'ECDHE-RSA-AES256-SHA384:AES256-SHA256:!ADH:!EXP:RSA:+HIGH:+MEDIUM:!MD5:!LOW:!NULL:!SSLv2:!eNULL:!aNULL:!RC4' | sort -k1

Expected output:

AES128-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(128) Mac=SHA1
AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1
DES-CBC3-SHA SSLv3 Kx=RSA Au=RSA Enc=3DES(168) Mac=SHA1

{ Add a Comment }

Policy Based Routing resulting in no ARP replies from gateway

One might think that when applying Policy Based Routing it will not affect ARP (Address Resolution Protocol) because they are considered to be things working on different layers. PBR clearly should affect only Layer 3 routing decisions and ARP is running somewhere below layer 3.. There are many nice discussions on the internet whether ARP is a Layer2 or Layer3 protocol and some people tend to say its Layer 2,5.

As it turns out PBR can affect ARP. If you for example wish to re-route every packet originating from the 192.168.1.0/24 network and make a policy route stating that everything from source net of 192.168.1.0/24 be routed to lets say to the GW 172.16.1.1 with out specifying any port or protocol. What will happen is that, ARP requests that use broadcast work, but unicast ARP requests won’t get replies any more – at least from Check Point firewalls. So you would need to either make 2 rules stating that it would affect TCP and UDP only based on your needs or follow Check Point supports guide lines: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk84480

{ Add a Comment }