Category: Linux

Finding missing free disk space in Linux, the power of lsof

There might be times when you find that your Linux machines disk seems to be full and you can’t find the reason for it. You try and find the culprit with the du (disk usage) command, but with no success. The numbers just don’t add up.  In that case actually the problem might be that you have some deleted files that are still open by some program. It can actually happen with faulty logrotate configurations where you don’t tell the program that is writing the log to release the file. Or that you manually deleted a file that some program was writing to.

In such cases the “lsof” command comes to the rescue. Basically, it does what the name says, lists open files – even if it has been deleted and is still in use.  Here is an example of a command that I sometimes use to find if there are deleted files that are still open:

lsof | grep deleted|awk '{$7=$7/1048576 "MB"; print}'

The output of the previous command would list you the open deleted files, the process that is still writing to them and the size of the files. This is some random example output from when I last had to look for missing space:

java 32511 32646 tom 1w REG 12980.00024128MB 19510390447 6341662 /var/log/tomcat/log/catalina.out (deleted)
java 32511 32646 tom 2w REG 0.00024128MB 19510390447 6341662 /var/log/tomcat/log/catalina.out (deleted)

To reclaim the disk space, you just simply need to kill/restart the program that is writing to the deleted file.

User permissions issue on migration from MySQL to MariaDB

Today I decided to migrate the website from my old home server that had MySQL installed to a newer web server with MariaDB running on it.

Did it by doing the regular mysqldump and import procedure, which all went fine up to the point when I actually tried to access the site again. Then I got the following error message “Error establishing a database connection“. To see what’s going on I tried logging in to the database using the websites credentials in commandline and it also failed. After that logged in as root and saw that the user was imported, but it had no permissions.

To check what users exist in the database You can use the following SQL statement:

SELECT User, Host, Password FROM mysql.user;

To see what privileges a user has you can use the following SQL statement:

show grants for 'user_name'@'localhost';

In my case it showed the following out put stating that that the user has no privileges:

ERROR 1141 (42000): There is no such grant defined for user 'user_name' on host '%'

After seeing that I tried just re-applying the users rights by using the regular grant command to re-grant the user it’s privileges on the database using the following command:

MariaDB [(none)]> GRANT ALL PRIVILEGES ON database_name.* to 'user_name'@'localhost' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

Then I looked at the users permissions again and unfortunately I got the same result as before, “no such grant defined for the user..”.  After that I tried just flushing privileges, to force the server to reload them by issuing the following command:

Query OK, 0 rows affected (0.00 sec)

It didn’t get any better. Finally ended up revoking, flushing and resetting the permissions by doing the following:

MariaDB [(none)]> REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'user_name'@'localhost';
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> GRANT ALL ON database_name.* TO 'user_name'@'localhost';
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> show grants for 'user_name'@'localhost';
| Grants for user_name@localhost |
| GRANT USAGE ON *.* TO 'user_name'@'localhost' IDENTIFIED BY PASSWORD '*xxxxxxxxxxxxx' |
| GRANT ALL PRIVILEGES ON `database_name`.* TO 'user_name'@'localhost' |
2 rows in set (0.00 sec)

That finally helped and the site is up on the new server.

SSH key based authentication secure and convenient or is it?

SSH key based authentication secure and convenient or is it? Well that seems really obvious that it is secure and convenient no passwords to be guessed and changed all the time, or that can be guessed logging on to servers much faster. But when done improperly it isn’t that safe and secure as it would seem.

The issue

When logging on to SSH servers using authentication agent forwarding for convenience so you could jump hosts using the same key. See nothing wrong with it?  Still seems all good and  secure? Well not that secure any more, as soon as convenience of the authentication agent forwarding comes to play a little issue arrises that a lot of people do not think about. Namely the key you used to authenticate to the server is now accessible to others on the server, not in the sense that they could copy it, but they can use it to authenticate to other servers where your key would be valid and that are accessible from that server. Although it requires escalated privileges to get access to it, it is still a problem. So where is this key located? It goes to the /tmp/ folder. As the following is an example from my test machine:

huxx@lnx:~# ls -la /tmp/

total 10

drwxrwxrwt 10 root     root     3072 Feb  1 01:00 .

drwxr-xr-x 23 root     root     4096 Jun  2  2015 ..

drwx------  2 huxx     huxx     1024 Feb  1 00:36 ssh-DhNiAzWTEV
huxx@lnx:~# ls -la /tmp/ssh-DhNiAzWTEV

total 4

drwx------  2 huxx huxx 1024 Feb  1 00:36 .

drwxrwxrwt 10 root root 3072 Feb  1 01:01 ..

srwxr-xr-x  1 huxx huxx    0 Feb  1 00:36 agent.18922

Is there a solution for it?

So is there a solution for the afore mentioned issue? Well luckily  Yes there is. There are SSH key agents out there that actually ask for your permission first before allowing access to the private key. For Windows one such solution would be to use the KeeAgent plugin for the password manager called KeePass it allows to set a password/confirmation to be prompted for every time someone/something tries to access the private key. The same combination will also work on macOS with a bit of work by porting the Windows application using mono for Mac and adding ssh-askpass script to the system. The exact solutions will be shown in followup posts to come.

Solution for Windows users described here: