Coder Perfect

How do I find out what inotify watches have been registered?

Category: linux

Tag:

Problem

My inotify watch limit is 1024 (I believe the default is 128?). Despite this, yeoman, Guard, and Dropbox all fail and inform me that I need to increase my inotify limit. Before I do, I’d like to know what’s eating up all of my timepieces (I have very few files in my Dropbox).

Is there a way to find out what watches are currently registered in /proc or /sys, or a tool I can use?

Asked by frio

Solution #1

inotify filesystem options

sysctl fs.inotify

opened files

wc -l | lsof | grep inotify

Increase the values in this manner.

Answered by David Canós

Solution #2

I already addressed this in the same Unix Stackexchange discussion that @cincodenada mentioned, but I though I’d submit my ready-made response here since no one seems to have anything that works:

I have a script called inotify-consumers that will identify the worst offenders for you (a recent version will also list the username who owns the process, see below):

$ time inotify-consumers  

   INOTIFY
   WATCHER
    COUNT     PID     CMD
----------------------------------------
    6688    27262  /home/dvlpr/apps/WebStorm-2018.3.4/WebStorm-183.5429.34/bin/fsnotifier64
     411    27581  node /home/dvlpr/dev/kiwi-frontend/node_modules/.bin/webpack --config config/webpack.dev.js
      79     1541  /usr/lib/gnome-settings-daemon/gsd-xsettings
      30     1664  /usr/lib/gvfs/gvfsd-trash --spawner :1.22 /org/gtk/gvfs/exec_spaw/0
      14     1630  /usr/bin/gnome-software --gapplication-service
    ....

    7489  WATCHERS TOTAL COUNT

real    0m0.099s
user    0m0.042s
sys 0m0.062s

When facing a node modules folder with thousands of directories, WebStorm quickly reaches its limit of 8K viewers, demonstrating why the default restriction of 8K observers on a development workstation is insufficient. To ensure that there are no issues, add a webpack watcher…

Simon Matter introduced additional speed upgrades for extremely loaded Big Iron Linux (hundreds of cores) for heavily loaded Big Iron Linux (hundreds of cores) that sped it up considerably, taking it down from ten minutes (!) to 15 seconds on his monster rig.

—help inotify-consumers To install it, simply copy the contents of the script and place it in your $PATH, such as /usr/local/bin. You can even forgo copying it and pipe it into bash over http: if you trust this stranger on the internet.

$ curl -s https://raw.githubusercontent.com/fatso83/dotfiles/master/utils/scripts/inotify-consumers | bash 

       INOTIFY
       WATCHER
        COUNT     PID USER     COMMAND
    --------------------------------------
        3044   3933 myuser node /usr/local/bin/tsserver
        2965   3941 myuser /usr/local/bin/node /home/myuser/.config/coc/extensions/node_modules/coc-tsserver/bin/tsserverForkStart /hom
         979   3954 myuser /usr/local/bin/node /home/myuser/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/li
           1   7473 myuser /usr/local/bin/node --no-warnings /home/myuser/dev/dotfiles/common-setup/vim/dotvim/plugged/coc.nvim/build/i
           1   3899 myuser /usr/local/bin/node --no-warnings /home/myuser/dev/dotfiles/common-setup/vim/dotvim/plugged/coc.nvim/build/i

        6990  WATCHERS TOTAL COUNT

For your convenience, the script’s core content is simply this (inspired by this answer)

find /proc/*/fd \
    -lname anon_inode:inotify \
    -printf '%hinfo/%f\n' 2>/dev/null \
    \
    | xargs grep -c '^inotify'  \
    | sort -n -t: -k2 -r

If you’re wondering how to raise the limitations, here’s how.

$ inotify-consumers --limits 

Current limits
-------------
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 524288


Changing settings permanently
-----------------------------
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
sudo sysctl -p # re-read config

Answered by oligofren

Solution #3

Reference: https://askubuntu.com/questions/154255/how-can-i-tell-if-i-am-out-of-inotify-watches https://unix.stackexchange.com/questions/15509/whos-consuming-my-inotify-resources https://bbs.archlinux.org/viewtopic.php?pid=1340049

Answered by zeekvfu

Solution #4

I think

sudo ls -l /proc/*/fd/* | grep notify

could be useful. You’ll see a list of all the pids that have an inotify fd.

I’m not sure where else to look for further information! HTH

Answered by Guido

Solution #5

Because this issue appears frequently in Google results, I’m copying and pasting a portion of my response from a similar query on the Unix/Linux StackExchange:

I ran into this issue, and none of these solutions provide an answer to the question, “How many watches does each process now use?” The one-liners all tell you how many instances are open, but that’s only part of the picture, and the trace stuff is only good for seeing fresh watches open.

This will generate a file containing a list of open inotify instances, the number of watches they have, as well as the pids and binaries that created them, sorted by watch count in descending order:

sudo lsof | awk '/anon_inode/ { gsub(/[urw]$/,"",$4); print "/proc/"$2"/fdinfo/"$4; }' | while read fdi; do count=$(sudo grep -c inotify $fdi); exe=$(sudo readlink $(dirname $(dirname $fdi))/exe); echo -e $count"\t"$fdi"\t"$exe; done | sort -nr > watches

If you’re curious in what that large ball of muck does and why, I went over to the original response and described it in detail.

Answered by cincodenada

Post is based on https://stackoverflow.com/questions/13758877/how-do-i-find-out-what-inotify-watches-have-been-registered