Guide to limits.conf / ulimit /open file descriptors under linux

Why does linux have an open-file-limit?

The open-file-limit exists to prevent users/process to use up all resources on a machine. Every file-descriptor uses a certain amount of RAM and a malicious or malfunctioning program could bring down the whole server.

Systemd ignores /etc/security/limits.conf /limits.d !!!

Because this took me quite some time to debug I name it at the beginning: See this and other posts. Systemd does not respect the system limits. You have to add e.g. „LimitNOFILE“ to the daemon file.

[Unit]
Description=Some Daemon
After=syslog.target network.target

[Service]
Type=notify
LimitNOFILE=49152
ExecStart=/usr/sbin/somedaemon

[Install]
WantedBy=multi-user.target

https://fredrikaverpil.github.io/2016/04/27/systemd-and-resource-limits/

https://serverfault.com/questions/628610/increasing-nproc-for-processes-launched-by-systemd-on-centos-7

https://ro-che.info/articles/2017-03-26-increase-open-files-limit

What is an open file?

The lsof-manpage makes it clear:

An open file may be a regular file, a directory, a block special file,
a character special file, an executing text reference, a library, a
stream or a network file (Internet socket, NFS file or UNIX domain
socket.)

It is important to know that also network sockets are open files, because in a high-performance-web-environment lots of these are opened frequently.

How is the limit enforced?

$> man getrlimit
...
The kernel enforces the open-file limit using the functions setrlimit and getrlimit. 

Newer kernels support the prlimit call to get and set various limits on running processes
...

$> man prlimit
...
The prlimit() system call is available  since  Linux  2.6.36.   Library
support is available since glibc 2.13.
...

What is ulimit?

Mostly when people refer to ulimit they mean the bash builtin ‚ulimit‘ which is used to set various different limits in bash context (not to be confused with the deprecated c-routine ulimit(int cmd, long newlimit) from the system libraries). It can be used to set the open file limit of the current bash.

The difference between soft and hard limits

The initial soft and hard limits for open files are set in /etc/security/limits.conf and enforced at login through the PAM-module pam_limits.so . The user then can modify the soft and hard limit using ulimit or the c-functions. The hard limit can never be raised by a regular user. root is the only user who can raise its hard limit. The soft-limit can be freely varied by the user as long as its less than the hard limit. The value that triggers the „24: too many open files“-error is the soft-limit. It is only soft in the sense, that it can be freely set. The user can lower its hard limit, but beware. He can not raise it again (in this shell).

ulimit Mini-Howto

ulimit -n queries the current SOFT limit
ulimit -n [NUMBER] sets the hard and softlimit to the same value
ulimit -Sn queries the current SOFT limit
ulimit -Sn [NUMBER] sets the current soft limit
ulimit -Hn queries the current hard limit (thats the maximum value you can set the softlimit to (if you are not root))
ulimit -Hn [NUMBER] sets the current hard limit

Are there other limits?

Also a system wide open file limit applies. This is the maximum limit of open files the kernel will open for all processes together.

$> man proc
...
       /proc/sys/fs/file-max
              This file defines a system-wide limit  on  the  number  of  open
              files  for  all processes.  (See also setrlimit(2), which can be
              used by a process to set the per-process  limit,  RLIMIT_NOFILE,
              on  the  number of files it may open.)  If you get lots of error
              messages about running out of file handles, try increasing  this
              value:

              echo 100000 > /proc/sys/fs/file-max

              The  kernel constant NR_OPEN imposes an upper limit on the value
              that may be placed in file-max.

              If you  increase  /proc/sys/fs/file-max,  be  sure  to  increase
              /proc/sys/fs/inode-max   to   3-4   times   the   new  value  of
              /proc/sys/fs/file-max, or you will run out of inodes.
...

Note: /proc/sys/fs/inode-max (only present until Linux 2.2)
This file contains the maximum number of in-memory inodes. This
value should be 3-4 times larger than the value in file-max,
since stdin, stdout and network sockets also need an inode to
handle them. When you regularly run out of inodes, you need to
increase this value.

Starting with Linux 2.4, there is no longer a static limit on
the number of inodes, and this file is removed.

To query the maximum possible limit have a look at (this is only informational. Normally a way lower limit is sufficient):

$> cat /proc/sys/fs/nr_open
1048576

Change the system-wide open files limit

Append or change the following line in /etc/sysctl.conf

fs.file-max = 100000

(replace 100000 with the desired number)

Then apply the changes to the running system with:

$> sysctl -p

What does /proc/sys/fs/file-nr show?

$> man proc
       /proc/sys/fs/file-nr
              This (read-only)  file  gives  the  number  of  files  presently
              opened.  It contains three numbers: the number of allocated file
              handles; the number of free file handles; and the maximum number
              of file handles.  The kernel allocates file handles dynamically,
              but it doesn't free them again.   If  the  number  of  allocated
              files  is  close  to the maximum, you should consider increasing
              the maximum.  When the number of free  file  handles  is  large,
              you've  encountered a peak in your usage of file handles and you
              probably don't need to increase the maximum.

So basically it says /proc/sys/fs/file-nr is not the actual number of open files, but the maximum which were opened. It also shows the number of file descriptors which are free for reuse. So max-number – free number = actual number. This applies not only to physical files, but also sockets.

From a newer manpage:

Before Linux 2.6,
the kernel allocated file handles dynamically, but it didn’t
free them again. Instead the free file handles were kept in a
list for reallocation; the „free file handles“ value indicates
the size of that list. A large number of free file handles
indicates that there was a past peak in the usage of open file
handles. Since Linux 2.6, the kernel does deallocate freed file
handles, and the „free file handles“ value is always zero.

$> cat /proc/sys/fs/file-nr 
512	0	36258
max     free    limit

How is it possible to query the number of currently open file descriptors?

System wide

$> cat /proc/sys/fs/file-nr

Process

lsof lists also lots of content which does not count into the open file limit (e.g. anonymous shared memory areas (= /dev/zero entries)). Querying
the /proc-filesystem seems to be most reliable:

$> cd /proc/12345
$> find . 2>&1 | grep '/fd/' | grep -v 'No such file' | sed 's#task/.*/fd#fd#' | sort | uniq | wc -l

If you want to try lsof use this ( the -n prevents hostname lookups and makes lsof way faster for lots of open connections):

lsof -n -p 12345 | wc -l

You can also insert a number of pids for e.g. php5-fpm into lsof with:

lsof -n -p "$(pidof php5-fpm | tr ' ' ',')" | wc -l

Changing the ulimit for users

Edit the file /etc/security/limits.conf or append:

www-data soft nofile 8192
www-data hard nofile 8192

Set the soft to the hard-limit, so you don’t have to raise it manually, as user.

It is also possible to set a wildcard:

* soft nofile 8192
* hard nofile 8192

For root the wildcard will not work and extra lines have to be added:

root soft nofile 8192
root hard nofile 8192

I set my precious limits, I logout and login but they are not applied

As said before the limits in /etc/security/limits.conf are applied by the PAM-module pam_limits.so.
In the directory /etc/pam.d lie various files, which manage different PAM-settings for different commands.
If you don’t log into your account, but change into it using su or execute a command using sudo, then the special config for this program is loaded. Open the config and make sure the line for loading pam_limits.so is
not commented out:
/etc/pam.d/su

session    required   pam_limits.so

Save and now the limits should be applied.

Program specific special cases

nginx

nginx has some special handling:

This is what applied to my Ubuntu Precise 12.04 Testsystem. The init-script seems to be buggy there.

1. You can set the ulimit which nginx should use in /etc/default/nginx
2. /etc/init.d/nginx restart does NOT apply the ulimit settings. The setting is only applied in the start-section of the init script. So you have to do /etc/init.d/nginx stop; /etc/init.d/nginx start to apply the new limit

There is a better distribution independent way to set the worker openfiles limit. Using the config file!:

Syntax:	worker_rlimit_nofile number;
Default:	—
Context:	main

Changes the limit on the maximum number of open files (RLIMIT_NOFILE) for worker processes. Used to increase the limit without restarting the main process.