Having ten thousands of mount bind causes various processes to go into loops

[Julien Petit]

Dear,

Not sure i should report a bug so here is a report first. For more
than 10 years now, we've been using mount binds to create shares rw or
ro. It's been working perfectly under older Debian. A few months ago,
we migrated to Ubuntu Jammy and started having processes running 100%
non stop. While examining the processes in question, we could see the
same thing: it seemed to be reading all the mounts indefinitely.
It started with the phpsessionclean.service. We managed to fix it
editing /lib/systemd/system/phpsessionclean.service and disabling
sandboxing entries. But then it started to happen with other
processes.
Anything related to systemd seems affected in a way. For instance, we
cannot start haproxy if the mounts are mounted.
We tested with the last Debian and it is affected too.

We understand that 14 000 mounts is a lot. So maybe our usage will be
questioned. But this has been working for ages so why not now?

The problem can be very easily reproduced:

1. Launch the latest Debian stable
2. Execute the following script to create mounts:
#!/bin/bash
mkdir /home/test/directories
mkdir /home/test/mounts

for i in {1..14000}
do
echo "Mounting dir $i"
mkdir "/home/test/directories/dir_$i"
mkdir "/home/test/mounts/dir_$i"
mount --bind -o rw "/home/test/directories/dir_$i" "/home/test/mounts/dir_$i"
done

After that, the "top" command will show processes getting stuck using
100% of CPU never ending.

Has anyone a clue if this is fixable? Should i report a bug?
Thanks for your help.

[Max Nikulin]

Post by Julien Petit
for i in {1..14000}
do
echo "Mounting dir $i"
mkdir "/home/test/directories/dir_$i"
mkdir "/home/test/mounts/dir_$i"
mount --bind -o rw "/home/test/directories/dir_$i" "/home/test/mounts/dir_$i"
done
After that, the "top" command will show processes getting stuck using
100% of CPU never ending.

What processes are CPU hungry?

Post by Julien Petit
Has anyone a clue if this is fixable? Should i report a bug?

Perhaps it is not a Debian-specific bug, just more active usage of
sandboxing in systemd. If some applications have troubles parsing
/proc/mounts then bugs should be filed against them.

However do you need shared subtrees? It may cause exponential growth of
number of moutpoints, see
https://docs.kernel.org/filesystems/sharedsubtree.html
https://manpages.debian.org/bookworm/mount/mount.8.en.html#Shared_subtree_operations

[Richard]

Best question probably is: what exactly are you needing 14.000 mounts for?
Even snaps shouldn't be that ridiculous. So what's your use case? Maybe
there's a better solution to what you are doing. If it's just about having
a place that is rw only without execution permissions, just crate a
separate partition, mount it somewhere - e.g. /home/test/mounts and tell
mount/fstab to use the option noexec. No need for for your script. Or if
it's a more advanced file system like btrfs you may be able to simply
create a subvolume with the same capabilities, no need to tinker around
with partitions.

It's true this issue should be looked into, but it doesn't look urgent as
long as there are alternatives.

Richard

Post by Julien Petit
Dear,
Not sure i should report a bug so here is a report first. For more
than 10 years now, we've been using mount binds to create shares rw or
ro. It's been working perfectly under older Debian. A few months ago,
we migrated to Ubuntu Jammy and started having processes running 100%
non stop. While examining the processes in question, we could see the
same thing: it seemed to be reading all the mounts indefinitely.
It started with the phpsessionclean.service. We managed to fix it
editing /lib/systemd/system/phpsessionclean.service and disabling
sandboxing entries. But then it started to happen with other
processes.
Anything related to systemd seems affected in a way. For instance, we
cannot start haproxy if the mounts are mounted.
We tested with the last Debian and it is affected too.
We understand that 14 000 mounts is a lot. So maybe our usage will be
questioned. But this has been working for ages so why not now?
1. Launch the latest Debian stable
#!/bin/bash
mkdir /home/test/directories
mkdir /home/test/mounts
for i in {1..14000}
do
echo "Mounting dir $i"
mkdir "/home/test/directories/dir_$i"
mkdir "/home/test/mounts/dir_$i"
mount --bind -o rw "/home/test/directories/dir_$i"
"/home/test/mounts/dir_$i"
done
After that, the "top" command will show processes getting stuck using
100% of CPU never ending.
Has anyone a clue if this is fixable? Should i report a bug?
Thanks for your help.

[Julien Petit]

Best question probably is: what exactly are you needing 14.000 mounts for? Even snaps shouldn't be that ridiculous. So what's your use case? Maybe there's a better solution to what you are doing. If it's just about having a place that is rw only without execution permissions, just crate a separate partition, mount it somewhere - e.g. /home/test/mounts and tell mount/fstab to use the option noexec. No need for for your script. Or if it's a more advanced file system like btrfs you may be able to simply create a subvolume with the same capabilities, no need to tinker around with partitions.

We use the mounts to share an initial folder with either rw or ro
wrights in a user directory. The user directory is then accessible
through a web interface, sftp, webdav and rsync. There is probably
better ways to do that now but that's a legacy app (2009) that we'd
rather leave alone :)

It's true this issue should be looked into, but it doesn't look urgent as long as there are alternatives.

Yes, not urgent and very specific. I'm going to try to set the mounts
to private as Max suggested and see how it goes.
Thanks for your help.

[Richard]

Post by Julien Petit
We use the mounts to share an initial folder with either rw or ro
wrights in a user directory. The user directory is then accessible
through a web interface, sftp, webdav and rsync. There is probably
better ways to do that now but that's a legacy app (2009) that we'd
rather leave alone :)

If there's a better way should be judged on what exactly that app expects. For the web interface, maybe the http server - or whatever makes the web interface accessible to the users - can limit permissions. For the rest of the use cases it would be interesting which circumstances would need to be fulfilled for a user to be able to change permissions on a file they own. And if they could even change the permissions through sftp, webdav or rsync. Because if not, the simplest fix would be a cron job that peridoically sets the permissions on the directory, so you don't need a dedicated mount. But Maybe you want to create a separate topic where you describe exatcly what the basic requirements are and ask for suggestions what the best solution may be. Maybe something like AppArmor rules or other methods that aren't known by your typical user could be a better solution.

Post by Julien Petit
Yes, not urgent and very specific. I'm going to try to set the mounts
to private as Max suggested and see how it goes.
Thanks for your help.

If you haven't already, remember to create a bug report and include as much detail and logs as you can gather, as people will need to be able to tell what the actual issue is. Maybe it's a limitation of the file system, of the hardware or something else.

Richard

[Julien Petit]

Post by Richard
If there's a better way should be judged on what exactly that app expects. For the web interface, maybe the http server - or whatever makes the web interface accessible to the users - can limit permissions. For the rest of the use cases it would be interesting which circumstances would need to be fulfilled for a user to be able to change permissions on a file they own. And if they could even change the permissions through sftp, webdav or rsync. Because if not, the simplest fix would be a cron job that peridoically sets the permissions on the directory, so you don't need a dedicated mount. But Maybe you want to create a separate topic where you describe exatcly what the basic requirements are and ask for suggestions what the best solution may be. Maybe something like AppArmor rules or other methods that aren't known by your typical user could be a better solution.

Rights are not the challenge here. It's to be able to share a
directory across multiple users.
For instance you would have : /users/bob/accounting shared with Alice
and accessible in her home directory /users/alice/accounting

Post by Richard
If you haven't already, remember to create a bug report and include as much detail and logs as you can gather, as people will need to be able to tell what the actual issue is. Maybe it's a limitation of the file system, of the hardware or something else.

I haven't already. I want to test the using private mounts first.

Thanks again for your input.