Guide to limits.conf / ulimit /open file descriptors under linux
Why does linux have an open-file-limit?
The open-file-limit exists to prevent users/process to use up all resources on a machine. Every file-descriptor uses a certain amount of RAM and a malicious or malfunctioning program could bring down the whole server.
Systemd ignores /etc/security/limits.conf /limits.d !!!
Because this took me quite some time to debug I name it at the beginning: See this and other posts. Systemd does not respect the system limits. You have to add e.g. „LimitNOFILE
“ to the daemon file.
[Unit]
Description=Some Daemon
After=syslog.target network.target
[Service]
Type=notify
LimitNOFILE=49152
ExecStart=/usr/sbin/somedaemon
[Install]
WantedBy=multi-user.target
https://fredrikaverpil.github.io/2016/04/27/systemd-and-resource-limits/
https://ro-che.info/articles/2017-03-26-increase-open-files-limit
What is an open file?
The lsof-manpage makes it clear:
An open file may be a regular file, a directory, a block special file,
a character special file, an executing text reference, a library, a
stream or a network file (Internet socket, NFS file or UNIX domain
socket.)
It is important to know that also network sockets are open files, because in a high-performance-web-environment lots of these are opened frequently.
How is the limit enforced?
$> man getrlimit ... The kernel enforces the open-file limit using the functions setrlimit and getrlimit. Newer kernels support the prlimit call to get and set various limits on running processes ... $> man prlimit ... The prlimit() system call is available since Linux 2.6.36. Library support is available since glibc 2.13. ...
What is ulimit?
Mostly when people refer to ulimit they mean the bash builtin ‚ulimit‘ which is used to set various different limits in bash context (not to be confused with the deprecated c-routine ulimit(int cmd, long newlimit) from the system libraries). It can be used to set the open file limit of the current bash.
The difference between soft and hard limits
The initial soft and hard limits for open files are set in /etc/security/limits.conf and enforced at login through the PAM-module pam_limits.so . The user then can modify the soft and hard limit using ulimit or the c-functions. The hard limit can never be raised by a regular user. root is the only user who can raise its hard limit. The soft-limit can be freely varied by the user as long as its less than the hard limit. The value that triggers the „24: too many open files“-error is the soft-limit. It is only soft in the sense, that it can be freely set. The user can lower its hard limit, but beware. He can not raise it again (in this shell).
ulimit Mini-Howto
ulimit -n queries the current SOFT limit ulimit -n [NUMBER] sets the hard and softlimit to the same value ulimit -Sn queries the current SOFT limit ulimit -Sn [NUMBER] sets the current soft limit ulimit -Hn queries the current hard limit (thats the maximum value you can set the softlimit to (if you are not root)) ulimit -Hn [NUMBER] sets the current hard limit
Are there other limits?
Also a system wide open file limit applies. This is the maximum limit of open files the kernel will open for all processes together.
$> man proc ... /proc/sys/fs/file-max This file defines a system-wide limit on the number of open files for all processes. (See also setrlimit(2), which can be used by a process to set the per-process limit, RLIMIT_NOFILE, on the number of files it may open.) If you get lots of error messages about running out of file handles, try increasing this value: echo 100000 > /proc/sys/fs/file-max The kernel constant NR_OPEN imposes an upper limit on the value that may be placed in file-max. If you increase /proc/sys/fs/file-max, be sure to increase /proc/sys/fs/inode-max to 3-4 times the new value of /proc/sys/fs/file-max, or you will run out of inodes. ...
Note: /proc/sys/fs/inode-max (only present until Linux 2.2)
This file contains the maximum number of in-memory inodes. This
value should be 3-4 times larger than the value in file-max,
since stdin, stdout and network sockets also need an inode to
handle them. When you regularly run out of inodes, you need to
increase this value.
Starting with Linux 2.4, there is no longer a static limit on
the number of inodes, and this file is removed.
To query the maximum possible limit have a look at (this is only informational. Normally a way lower limit is sufficient):
$> cat /proc/sys/fs/nr_open 1048576
Change the system-wide open files limit
Append or change the following line in /etc/sysctl.conf
fs.file-max = 100000
(replace 100000 with the desired number)
Then apply the changes to the running system with:
$> sysctl -p
What does /proc/sys/fs/file-nr show?
$> man proc /proc/sys/fs/file-nr This (read-only) file gives the number of files presently opened. It contains three numbers: the number of allocated file handles; the number of free file handles; and the maximum number of file handles. The kernel allocates file handles dynamically, but it doesn't free them again. If the number of allocated files is close to the maximum, you should consider increasing the maximum. When the number of free file handles is large, you've encountered a peak in your usage of file handles and you probably don't need to increase the maximum.
So basically it says /proc/sys/fs/file-nr is not the actual number of open files, but the maximum which were opened. It also shows the number of file descriptors which are free for reuse. So max-number – free number = actual number. This applies not only to physical files, but also sockets.
From a newer manpage:
Before Linux 2.6,
the kernel allocated file handles dynamically, but it didn’t
free them again. Instead the free file handles were kept in a
list for reallocation; the „free file handles“ value indicates
the size of that list. A large number of free file handles
indicates that there was a past peak in the usage of open file
handles. Since Linux 2.6, the kernel does deallocate freed file
handles, and the „free file handles“ value is always zero.
$> cat /proc/sys/fs/file-nr 512 0 36258 max free limit
How is it possible to query the number of currently open file descriptors?
System wide
$> cat /proc/sys/fs/file-nr
Process
lsof lists also lots of content which does not count into the open file limit (e.g. anonymous shared memory areas (= /dev/zero entries)). Querying
the /proc-filesystem seems to be most reliable:
$> cd /proc/12345 $> find . 2>&1 | grep '/fd/' | grep -v 'No such file' | sed 's#task/.*/fd#fd#' | sort | uniq | wc -l
If you want to try lsof use this ( the -n prevents hostname lookups and makes lsof way faster for lots of open connections):
lsof -n -p 12345 | wc -l
You can also insert a number of pids for e.g. php5-fpm into lsof with:
lsof -n -p "$(pidof php5-fpm | tr ' ' ',')" | wc -l
Changing the ulimit for users
Edit the file /etc/security/limits.conf or append:
www-data soft nofile 8192 www-data hard nofile 8192
Set the soft to the hard-limit, so you don’t have to raise it manually, as user.
It is also possible to set a wildcard:
* soft nofile 8192 * hard nofile 8192
For root the wildcard will not work and extra lines have to be added:
root soft nofile 8192 root hard nofile 8192
I set my precious limits, I logout and login but they are not applied
As said before the limits in /etc/security/limits.conf are applied by the PAM-module pam_limits.so.
In the directory /etc/pam.d lie various files, which manage different PAM-settings for different commands.
If you don’t log into your account, but change into it using su or execute a command using sudo, then the special config for this program is loaded. Open the config and make sure the line for loading pam_limits.so is
not commented out:
/etc/pam.d/su
session required pam_limits.so
Save and now the limits should be applied.
Program specific special cases
nginx
nginx has some special handling:
This is what applied to my Ubuntu Precise 12.04 Testsystem. The init-script seems to be buggy there.
- You can set the ulimit which nginx should use in /etc/default/nginx
- /etc/init.d/nginx restart does NOT apply the ulimit settings. The setting is only applied in the start-section of the init script. So you have to do /etc/init.d/nginx stop; /etc/init.d/nginx start to apply the new limit
There is a better distribution independent way to set the worker openfiles limit. Using the config file!:
Syntax: worker_rlimit_nofile number; Default: — Context: main Changes the limit on the maximum number of open files (RLIMIT_NOFILE) for worker processes. Used to increase the limit without restarting the main process.
Wonderful work!!
Clearest explanation I could found around ulimit and limit.conf.
Many thanks
Alexandre
Good explanation.. Thanks!
Thank you. Very helpful article. I used to encounter a critical error setting root to hard nofile 100000, but after clearing with this article it should not be that high. I ended up setting it to 10000 instead.
Thanks you for the Explanation however what i am dealing with is an issue where even after deleting the open files which are marked deleted using
lsof -u hive |grep deleted |while read line ; do cat /dev/null > $line ;done
however my file descriptor count remains unchanged , it does not get flushed from the memory though the corresponding files are no longer available on the disk .
BTW, I cannot have my hiveserver2 restarted frequently to release the held up memeory .
Sorry for the late answer, I just saw your comment. Answering just in case someone else stumbles over this. You have to distinguish between different things.
– filedescriptors almost don’t use any memory, this should not cause a memory problem
– the filedescriptor limit may be too low, so you might have to increase it, this should fix problems with “24: too many open files” errors and not cause any problems
– the method you described (cating /dev/null to the files) releases the DISKSPACE the files occupy, so if DISKSPACE is your problem this should work
– to really free the file descriptors, the application holding the descriptors open has to release some. Sometimes applications have a graceful restart, which does not influence uptime, but might trigger releasing the filedescriptors (only a wild guess)