NFSv4 without Kerberos and permissions

perhaps value defaults of nfs in ubuntu and debian are different and
perhaps nfs versions are different and any of these paremeters aren't
recognized in the host client.

Post by John Ratliff
I recently set up an NFS v4 server on debian stretch. If I use a debian
client to mount the share, everything works fine. However, if I use a
CentOS or a Ubuntu client, the permissions don't work.
I have synchronized the group ids manually, and that's the only
permissions I'm interested in.
/export
/export/ssl
/export/ssl/wildcard
/export/ssl/wildcard is chmod 2750 with root:ssl-cert as its owner.
root user is uid 0, ssl-cert is gid 555.
/export 10.77.9.0/24(rw,sync,fsid=0,crossmnt,no_subtree_check)
/export/ssl 10.77.9.0/24(ro,sync,no_subtree_check)
/export/ssl/wildcard 10.77.9.0/24(ro,sync,no_subtree_check)
There is no firewall on the server. iptables is wide open. Server is
10.77.9.188 and client is 10.77.9.189. hosts.allow has ALL:ALL on the last
line.
I have no trouble mounting the share on the client. But a member of the
ssl-cert group (again, it IS GID 555 -- I made the group manually to ensure
this) CANNOT enter the directory. They cannot read files in the directory.
They cannot do anything with the share.
I have this trouble with Ubuntu 14.04, Ubuntu 16.04, and CentOS 7. I do
not have any issues with Debian 8 or Debian 9 clients.
/etc/default/nfs-common has
NEED_IDMAPD=yes
/etc/default/nfs-kernel-server is stock; no changes.
I have this problem with NFSv4 servers on Debian 8 and Debian 9.
If I run the NFS server on Ubuntu 16.04, I can access it fine from Ubuntu
clients and CentOS clients, but Debian clients have the same problem.
What am I missing? What can I do to further troubleshoot this? Is there
any way to determine what mapping it THINKS is occurring?
Thanks.

John Ratliff

2017-10-15 02:00:01 UTC

perhaps value defaults of nfs in ubuntu and debian are different and
perhaps nfs versions are different and any of these paremeters aren't
recognized in the host client.
I recently set up an NFS v4 server on debian stretch. If I use a
debian client to mount the share, everything works fine. However, if
I use a CentOS or a Ubuntu client, the permissions don't work.
I have synchronized the group ids manually, and that's the only
permissions I'm interested in.
/export
/export/ssl
/export/ssl/wildcard
/export/ssl/wildcard is chmod 2750 with root:ssl-cert as its owner.
root user is uid 0, ssl-cert is gid 555.
/export 10.77.9.0/24(rw,sync,fsid=0,crossmnt,no_subtree_check)
<http://10.77.9.0/24(rw,sync,fsid=0,crossmnt,no_subtree_check)>
/export/ssl 10.77.9.0/24(ro,sync,no_subtree_check)
<http://10.77.9.0/24(ro,sync,no_subtree_check)>
/export/ssl/wildcard 10.77.9.0/24(ro,sync,no_subtree_check)
<http://10.77.9.0/24(ro,sync,no_subtree_check)>
There is no firewall on the server. iptables is wide open. Server is
10.77.9.188 and client is 10.77.9.189. hosts.allow has ALL:ALL on
the last line.
I have no trouble mounting the share on the client. But a member of
the ssl-cert group (again, it IS GID 555 -- I made the group
manually to ensure this) CANNOT enter the directory. They cannot
read files in the directory. They cannot do anything with the share.
I have this trouble with Ubuntu 14.04, Ubuntu 16.04, and CentOS 7. I
do not have any issues with Debian 8 or Debian 9 clients.
/etc/default/nfs-common has
NEED_IDMAPD=yes
/etc/default/nfs-kernel-server is stock; no changes.
I have this problem with NFSv4 servers on Debian 8 and Debian 9.
If I run the NFS server on Ubuntu 16.04, I can access it fine from
Ubuntu clients and CentOS clients, but Debian clients have the same problem.
What am I missing? What can I do to further troubleshoot this? Is
there any way to determine what mapping it THINKS is occurring?
Thanks.

After much googling, I have found the answer.

The Debian NFS server, by default, uses --manage-gids in the
RPCMOUNTDOPTS in /etc/default/nfs-kernel-server. I guess I never looked
very hard at that option, but what it means is that group membership is
checked on the server, not trusted from the client. This is a good thing
overall; it improves security and overcomes a limitation of the NFS
protocol (16 group count).

In my case, the user on the client I was testing was UID 1003, which on
the server he was UID 1000. So they both had the group, but UID 1003 on
the server did not have the group, because that user did not exist.
Therefore, permission denied.

Although it's not the best solution from a security standpoint, I'm
going to disable the manage-gids option for now and limit access by
hosts.allow and the firewall.

Thanks very much to
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1454112 this post.

Christian Seiler

2017-10-15 07:40:02 UTC

Post by John Ratliff
In my case, the user on the client I was testing was UID 1003, which
on the server he was UID 1000. So they both had the group, but UID
1003 on the server did not have the group, because that user did not
exist. Therefore, permission denied.

Then you are not idmapping correctly.

NFSv4 has two modes of operation when it comes to users:

1) Use raw UIDs/GIDs like NFSv2/3 did. This is available since Linux
3.2 or 3.5 (I don't remember which) and only possible if sec=sys
(i.e. no Kerberos) is used. In that case the user IDs are simply
sent over the wire directly.

This requires the UIDs and GIDs to be identical on client and
server.

2) Use names for users / groups. Each user and group is translated
into a string on the client, and the server translates the string
back to a user id. This is done via the idmapping mechanism
(rpc.idmapd on the server side, and nfsidmap in combination with
request-key on the client side, both configured via
/etc/idmapd.conf). Here the UIDs/GIDs don't need to be identical
on client and server.

In your case, since you are not using Kerberos, current Linux
versions will default to not using the ID mapping mechanism when a
non-Kerberos setup is in place, and will use raw UIDs/GIDs instead.
But you _do_ want idmapping since the the UIDs and GIDs on the client
and/or server don't match up.

To make this work, you can tell the server to never accept raw UIDs
and GIDs:

echo 0 > /sys/module/nfsd/parameters/nfs4_disable_idmapping

The setting is a bit of a misnomer: while it is on by default, the
setting does not actually disable idmapping on the server, it being
on will just also enable raw UIDs and GIDs. If a client that
connects doesn't support raw UIDs/GIDs (such as older Linux versions
e.g. 2.6.x or other operating systems) the server will still happily
do idmapping with the setting on. But disabling this setting will
have the NFS server fall back to the mode where it will only accept
idampped strings and never accept raw UIDs/GIDs. This will cause
some log messages to be shown in recent enough Linux clients (the
message being "v4 server [...] does not accept raw uid/gids.
Reenabling the idmapper", but that's actually the behavior you want.

To make the setting permanent, you may add it to your modprobe.conf:

echo "options nfsd nfs4_disable_idmapping=0" > /etc/modprobe.d/nfsd.conf

There is also a setting for the clients here, found in
/sys/module/nfs/parameters/nfs4_disable_idmapping (note the missing
'd' in the module name) that works just the same, but only on the
client side.

I am not aware of any way to change this setting on a per-mount
basis.

It is up to you whether you'll want to add this setting on the server,
the clients, or both.

Post by John Ratliff
Although it's not the best solution from a security standpoint, I'm
going to disable the manage-gids option for now and limit access by
hosts.allow and the firewall.

NFSv4 without Kerberos does not have any security at all. The server
will implicitly trust the UID the client sends to the server, so a
compromised client may impersonate any (!) user on the server except
root (unless no_root_squash is set in /etc/exports, which I don't
recommend). It may also impersonate any group.

Furthermore, the MANAGED_GIDS setting is only for NFSv2/3 and only
for supplementary groups, not the primary group. It is not a security
setting, it really is just for bypassing the 16 group limit of older
NFS protocols, and is useless with NFSv4.

Basically, if you're running a sec=sys NFS server (or an NFSv2/3
server) you're implicitly trusting all the clients that are allowed
to connect to you and all the network components (such as routers)
in the middle. Anything that is not owned by the root user will be
able to be read or written to by any NFS client if the client wants
to: they can just read out the user ID of the file and send that
to the server together with the read/write request. Only the root
user is a bit more protected due to root_squash, and you can make
the entire export read-only - but that's it when it comes to the
protections NFS without Kerberos gives you.

Regards,
Christian

John Ratliff

2017-10-16 18:00:01 UTC

Then you are not idmapping correctly.
1) Use raw UIDs/GIDs like NFSv2/3 did. This is available since Linux
3.2 or 3.5 (I don't remember which) and only possible if sec=sys
(i.e. no Kerberos) is used. In that case the user IDs are simply
sent over the wire directly.
This requires the UIDs and GIDs to be identical on client and
server.
2) Use names for users / groups. Each user and group is translated
into a string on the client, and the server translates the string
back to a user id. This is done via the idmapping mechanism
(rpc.idmapd on the server side, and nfsidmap in combination with
request-key on the client side, both configured via
/etc/idmapd.conf). Here the UIDs/GIDs don't need to be identical
on client and server.
In your case, since you are not using Kerberos, current Linux
versions will default to not using the ID mapping mechanism when a
non-Kerberos setup is in place, and will use raw UIDs/GIDs instead.
But you _do_ want idmapping since the the UIDs and GIDs on the client
and/or server don't match up.
To make this work, you can tell the server to never accept raw UIDs
echo 0 > /sys/module/nfsd/parameters/nfs4_disable_idmapping
The setting is a bit of a misnomer: while it is on by default, the
setting does not actually disable idmapping on the server, it being
on will just also enable raw UIDs and GIDs. If a client that
connects doesn't support raw UIDs/GIDs (such as older Linux versions
e.g. 2.6.x or other operating systems) the server will still happily
do idmapping with the setting on. But disabling this setting will
have the NFS server fall back to the mode where it will only accept
idampped strings and never accept raw UIDs/GIDs. This will cause
some log messages to be shown in recent enough Linux clients (the
message being "v4 server [...] does not accept raw uid/gids.
Reenabling the idmapper", but that's actually the behavior you want.
echo "options nfsd nfs4_disable_idmapping=0" > /etc/modprobe.d/nfsd.conf
There is also a setting for the clients here, found in
/sys/module/nfs/parameters/nfs4_disable_idmapping (note the missing
'd' in the module name) that works just the same, but only on the
client side.
I am not aware of any way to change this setting on a per-mount
basis.
It is up to you whether you'll want to add this setting on the server,
the clients, or both.

Thanks. This all worked great. Once I disabled the setting on both the
server and the client, ID mapping started working.

Post by John Ratliff
Although it's not the best solution from a security standpoint, I'm
going to disable the manage-gids option for now and limit access by
hosts.allow and the firewall.

I'm not sure this is entirely the case. Turning off managed GIDs and
using an NFSv4 mount (i.e. on client: mount -t nfs4 blahblahblah), this
let me use secondary groups on the server. Before, I could not.

Post by Christian Seiler
Basically, if you're running a sec=sys NFS server (or an NFSv2/3
server) you're implicitly trusting all the clients that are allowed
to connect to you and all the network components (such as routers)
in the middle. Anything that is not owned by the root user will be
able to be read or written to by any NFS client if the client wants
to: they can just read out the user ID of the file and send that
to the server together with the read/write request. Only the root
user is a bit more protected due to root_squash, and you can make
the entire export read-only - but that's it when it comes to the
protections NFS without Kerberos gives you.

Thanks. This has been quite helpful. I will adjust the configurations on
my actual servers to make use of ID mapping there as well.

--John

Christian Seiler

2017-10-16 19:40:02 UTC

Post by Christian Seiler
Furthermore, the MANAGED_GIDS setting is only for NFSv2/3 and only
for supplementary groups, not the primary group. It is not a security
setting, it really is just for bypassing the 16 group limit of older
NFS protocols, and is useless with NFSv4.

I'm not sure this is entirely the case. Turning off managed GIDs and
using an NFSv4 mount (i.e. on client: mount -t nfs4 blahblahblah),
this let me use secondary groups on the server. Before, I could not.

Sorry, you're right. rpc.mountd is typically only NFSv2/3, but with
NFSv4 it does provide some corner case functionality (I rechecked
the code just now) and MANAGED_GIDS is one of those.

Thanks. This has been quite helpful. I will adjust the configurations
on my actual servers to make use of ID mapping there as well.

Note that idmapping is not the same as Kerberos. idmapping just
means that you map between the user id and a string before any
ID is sent of the network. (Client translates user id to string,
sends string over network, server receives string, translates
it back to a user ID and uses that - and vice versa.)

NFSv4 without Kerberos _always_ has the same security properties,
irrespective of whether you enable idmapping or not: none.

If you want authentication you need sec=krb5. If you also want
data integrity and tamper-resistance, you need sec=krb5i. And
if you also want encryption you need sec=krb5p. In all of these
cases you need a Kerberos setup, which is not trivial.

Regards,
Christian

John Ratliff

2017-10-20 00:40:01 UTC

I'm not sure this is entirely the case. Turning off managed GIDs and
using an NFSv4 mount (i.e. on client: mount -t nfs4 blahblahblah),
this let me use secondary groups on the server. Before, I could not.

Sorry, you're right. rpc.mountd is typically only NFSv2/3, but with
NFSv4 it does provide some corner case functionality (I rechecked
the code just now) and MANAGED_GIDS is one of those.

Okay, I am unable to make this work. Perhaps I am still confused as to
how id mapping works.

Here is my test setup.

1) Debian 9 NFS4 Server

/etc/hosts.deny is empty
/etc/hosts.allow
ALL: ALL

iptables is wide open.

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Server is on 10.9.111.1/16

/etc/exports is

/srv/nfs4 10.9.0.0/16(rw,sync,fsid=0,crossmnt,no_subtree_check)
/srv/nfs4/ssl 10.9.0.0/16(rw,sync,no_subtree_check)
/srv/nfs4/ssl/wildcard 10.9.0.0/16(ro,sync,no_subtree_check)

I have set nfs_disable_idmapping to 0

$ cat /sys/module/nfsd/parameters/nfs4_disable_idmapping
N

This is my /srv/nfs4 file structure:

$ ls -lR /srv/nfs4
/srv/nfs4:
total 4
drwxr-sr-x 3 root ssl-cert 4096 Oct 19 20:01 ssl

/srv/nfs4/ssl:
total 12
-rw-r----- 1 root ssl-cert 24 Oct 19 18:43 private.txt
-rw-r--r-- 1 root ssl-cert 22 Oct 19 18:43 public.txt
drwxr-s--- 2 root ssl-cert 4096 Oct 19 18:43 wildcard

/srv/nfs4/ssl/wildcard:
total 8
-rw-r----- 1 root ssl-cert 24 Oct 19 18:43 private.txt
-rw-r--r-- 1 root ssl-cert 22 Oct 19 18:43 public.txt

There are five users on the server. These are their permissions:

$ id jratliff
uid=1000(jratliff) gid=1000(jratliff)
groups=1000(jratliff),27(sudo),100(users),112(ssl-cert)

$ id jbail
uid=1001(jbail) gid=1001(jbail) groups=1001(jbail),27(sudo),100(users)

$ id john
uid=1003(john) gid=1003(john) groups=1003(john),112(ssl-cert)

$ id matt
uid=2049(matt) gid=2049(matt) groups=2049(matt),112(ssl-cert)

$ id austin
uid=2050(austin) gid=2050(austin) groups=2050(austin)

Based on this configuration, I would expect the following:

Users ***@domain, ***@domain, and ***@domain should be able to
read the private.txt file and enter the wildcard directory.

Users ***@domain and ***@domain should not be able to do those things.
------------------------------------

2) Ubuntu 14.04 LTS Client

Very similar settings. Open tcp wrappers, open firewalls, same subnet
(10.9.111.2/16).

I removed nfs4_disable_idmapping in both the nfsd and nfs modules.

$ cat /sys/module/nfsd/parameters/nfs4_disable_idmapping
N

$ cat /sys/module/nfs/parameters/nfs4_disable_idmapping
N

Same five users, but the UIDs have been changed a bit.

jratliff is uid 1000
jbail is uid 1001
matt is uid 1003
austin is uid 1002
john is uid 1004

As expected, jratliff and matt can read the private files. jbail and
austin cannot. However, neither can john.

So why do jratliff and matt work while john does not? My hypothesis is
that the UIDs work out that way.

jratliff has the same UID, so the group membership maps properly.
jratliff is a member of ssl-cert on the server, so everything is fine.
Matt is also a member of ssl-cert, but this isn't why he can read the
files; he can read it because his UID is 1003, which maps to john on the
server. User john on the server is a member of ssl-cert, so Matt is
become john. In this case, john does not map to anyone on the server,
and so he has no permissions. Therefore, john cannot read the private files.

Second hypothesis: Ubuntu 14 is broken.

---------------------------------------

3) Debian 9 Client

Again, very similar settings. 10.9.111.3/16 is the IP.
Same five users, but I made one change from the server to the client in
UIDs.

jratliff is uid 1000
jbail is uid 1001
matt is uid 3333
austin is uid 2050
john is uid 1003

Again, jratliff can read the private files. jbail and austin cannot.
This is expected. I thought matt would be able to read the files, but
matt cannot. This seems to be because matt is UID 3333 which means
nothing on the server, permissions-wise anyways.

This time, user john can read the private files, which makes sense
because he is UID 1003, the same as on the server, and is a member of
ssl-cert on the server.

So, Ubuntu 14 seems to work the same as Debian 9, which is expected
behavior. It probably isn't broken.

----------------------------------

So, is my understand of ID Mapping still flawed? Here's what I see
happening:

Client sends UID/GID/GIDs to server in a RPC. Client does not send
username or group name strings. I have used wire shark and I cannot find
any trace of any user or group name within the transferred between
client and server.

Server maps UID sent to local UID. Server ignores secondary GIDs because
--manage-gids is set on the server. Server checks permissions against
UID, GID, and GIDs on the server attached to the sent UID. If the
permissions match, everything goes ahead. IF they do not, permission is
denied.

So, I really don't know what id mapping is supposed to do, but it does
not appear to be doing what I thought you were telling me it would do.
All evidence I had previously gathered to the contrary seems to be the
result of coincidence and convergence of UIDs based on my test
environment and lack of complete and thorough testing of variables until
now.

Thanks for any further insights.

--John

John Ratliff

2017-10-24 17:30:01 UTC