Kerberized NFSv4 in a mixed OS environment

Intro

I have a mixed home environment where Windows, Linux, and FreeBSD are well represented. Unfortunately, documentation is scarce on NFSv4 and Kerberos when it comes to non-enterprise environments and so here are a few notes that come in handy when setting up a mixed environment Kerberized NFSv4 setup.

I'm a big fan of manpages, so I'll refer to relevant manpages. They are your best friends.

Environment

In the rest of these notes, I have used FreeBSD 12.0-RELEASE that acts as the KDC (Kerberos Key Distribution Center), Kerberos server, and NFSv4 server along with Windows 10 Pro with Anniversary Update, Ubuntu 18.10, and 18.04 LTS as clients.

FreeBSD server

KDC

FreeBSD comes with Heimdal Kerberos in base which I have opted to use. The setup is quite straightforward.

First, read the Kerberos section of the FreeBSD Handbook and then kadmin(8), kinit(1), and krb5.conf(5).

The following sections are notes on things that I found different that the guides mentioned.

/etc/krb5.conf

In order to have Kerberos work, you have to use EITHER a full krb5.conf OR a slim krb5.conf + DNS. The Kerberos spec takes a full krb5.conf as preference and ignores DNS records if it finds a full krb5.conf.

A sample full /etc/krb5.conf that I found to work is as follows:

[libdefaults]
default_realm = HOME.LAN

[domain_realm]
.home.lan = HOME.LAN

[realms]
HOME.LAN = {
kdc = fbsd.home.lan
admin_server = fbsd.home.lan
}

I recommend enabling logging just in case anything goes wrong:

[logging]
kdc = FILE:/var/heimdal/kdc.log
kdc = SYSLOG:INFO
default = SYSLOG:INFO

I found a running tail -f /var/heimdal/kdc.log to be very helpful in tracking which servers can talk to the KDC.

DNS records

EITHER DNS OR FULL krb5.conf. I have unbound setup as my DNS resolver for my local lan on pfsense.

First, Kerberos needs to find the realm. To find the realm, kerberos takes the FQDN of the host and successively queries down the domain chain until it finds a TXT record. For example, if you have a hostname of my.long.host.fqdn.com, the following queries in this order.

_kerberos.my.long.host.fqdn.com
_kerberos.long.host.fqdn.com
_kerberos.host.fqdn.com
_kerberos.fqdn.com

As soon as it finds a TXT record, it stops and uses the result as the realm, so I have the following record in unbound so it applies to all my LAN:

_kerberos.home.lan IN TXT HOME.LAN

in pfSense, you can add this under DNS Resolver -> Custom options as follows:

server:
local-data: "_kerberos.home.lan IN TXT HOME.LAN"

Next, kerberos needs to find what to do with this realm, i.e. to answer where the KDC, passwd, etc are located. For this, it queries the following SRV records. While DNS records are case-insensitive, Kerberos is case-SENSITIVE. so, make sure to keep the case proper in your DNS records. I have added the following in my unbound.conf.

_kerberos._udp.HOME.LAN     IN SRV 01 00 88  fbsd.home.lan.
_kerberos._tcp.HOME.LAN     IN SRV 01 00 88  fbsd.home.lan.
_kpasswd._udp.HOME.LAN      IN SRV 01 00 464 fbsd.home.lan.
_kerberos-adm._tcp.HOME.LAN IN SRV 01 00 749 fbsd.home.lan.

Lookup SRV record format to find out what the numbers mean, but as far as Kerberos is concerned, it's looking for "_kerberos._{udp,tcp}.MYREALM" where MYREALM is in capital. Feel free to play around with the records to see which ones are needed for your setup (i.e. tcp vs udp etc.).

When using DNS, consider the security trade-offs vs the use of krb5.conf. Also, make sure hostname --fqdn (on your hosts) returns the full fqdn of your host including the domain part (in this case home.lan).

You can test DNS records using host:

host -t SRV _kerberos._tcp.HOME.LAN

Final DNS records on pfSense/unbound

server:
local-data: "_kerberos._udp.HOME.LAN     IN  SRV 01 00 88 topoli.home.lan."
local-data: "_kerberos._tcp.HOME.LAN     IN  SRV 01 00 88 topoli.home.lan."
local-data: "_kpasswd._udp.HOME.LAN      IN  SRV 01 00 464 topoli.home.lan."
local-data: "_kerberos-adm._tcp.HOME.LAN IN  SRV 01 00 749 topoli.home.lan."
local-data: "_kerberos.home.lan          IN  TXT HOME.LAN"

Kerberos server

Now that KDC is working well, We should setup our FreeBSD server as a server principal. You can do this with kadmin(8). Essentially all you need to do with kerberos from now on is run kadmin -l and add whatever principal you want. add --random-key generates a principal with a random password which is good for host/nfs/etc type of principals which use exported keytabs. For users that need to login, just add <name> will work just fine.

$ kadmin -l
> add myprincipal
> ext_keytab --keytab=/path/to/keytab myprincipal

There are other guides that describe generating principals in more detail. I will add my own notes here.

Keys in kerberos have arbitrary names; however, some agents expect certain names by convention. For example, NFSv4 server expects principals in the format nfs/fqdn.domain. I have read that NFSv4 clients use host/fqdn.domain, but in my own attempts, I have noticed that nfs/fqdn.domain works. You can look at the output of client's gssd with -vvv verbose output to see which principals it uses (run gssd -vvv in foreground and watch its output).

In order to have NFSv4 server work, it needs to have a principal in the above format exported as a keytab placed in /etc/krb5.keytab. Use ext_keytab command above (see kadmin(8)) and copy the resulting keytab to /etc/krb5.keytab.

NFSv4 server

First, create a principal for the server named nfs/fqdn.domain (replace fqdn.domain with the fqdn of your server). In my case, it would be:

$ kadmin -l
> add --random-key nfs/fbsd.home.lan
> ext_keytab nfs/fbsd.home.lan

ext_keytab without --keytab exports to /etc/krb5.keytab for convenience (see man kadmin(8)).

See exports(5) and =nfsv4(4)=and the NFS section on FreeBSD Handbook.

Other relevant manpages: nfsd(8), nfsuserd(8), mountd(8), gssd(8).

# /etc/rc.conf
# NFSv4

# these enable nfs server, see their respective rc scripts
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 4 -h <ip address of nfsv4 server>"
nfsv4_server_enable="YES"

# for manage-gids: see nfsuserd(8)
nfsuserd_enable="YES"
nfs_server_managegids="YES"

mountd_enable="YES"
mountd_flags="-r -h <ip address of nfsv4 server>"

rpcbind_enable="YES"

# gssd is the bridge that makes kerberization possible, see gssd(8)
# you can run this separately in the foregound with -vvv to help you
# debug issues and make sure kerberos side of NFS is working
gssd_enable="YES"
gssd_flags="-h"

# these are optional, some programs might misbehave if there is no file locking
# so I've enabled them
rpc_lockd_enable="YES"
rpc_statd_enable="YES"

Don't forget the V4: line to enable NFSv4. There is no reason to leave sys here. Also, make sure to restart or reload mountd whenever you change /etc/exports.

V4: /         -sec=krb5p:krb5i:krb5:sys
/my/documents -sec=krb5p:krb5i:krb5:sys

Use showmount(8) to make sure everything's exported.

You can also use sysctl to explore a few options you can modify. Look into vfs.nfsd.* sysctl, in particular vfs.nfsd.server_min_nfsvers if you need to restrict NFS to a minimum version.

Linux client (Ubuntu 18.10/18.04)

Ubuntu does a number of convenience things, at the cost of clarity. So, while I found out how to do things in Ubuntu, it wasn't clear how that can translate to other distros.

Kerberos

(here, feel free to use either MIT or heimdal kerberos. I opted for heimdal as I've become familiar with it).

install heimdal-clients and copy your krb5.conf from FreeBSD server to /etc/krb5.conf (or if you decided on DNS, see DNS section above). Now, all you need to do is kinit <principal> to test if everything works. I have created a user principal (just a bare username) for every user I expect to have access to NFSv4 server.

See kinit(1), kdestroy(1), and klist(1). You can use them like so:

$ kinit <principal>
$ klist
$ kdestroy

one quirk to note is that if you're testing multiple principals and keep =kinit=ing with different principals, make sure to umount ALL NFS mounts, kinit, then remount in order for permissions to take effect.

If you want to login to kerberos while you login, install libpam-heimdall and run pam-auth-update to enable kerberos login with pam.

See /etc/pam.d/common-* files to see how pam-auth-update has changed your config to enable kerberos (essentially added a bunch of pam_krb5.so lines).

NFSv4

Install nfs-common. Make sure gssd is run by editing /etc/default/nfs-common like so:

NEED_GSSD=yes
NEED_IDMAPD=yes

Then systemctl start nfs-client.target. Make sure rpc-gssd.service and nfs-idmapd.service are running. I'm not sure if IDMAPD and nfs-idmapd.service are necessary.

See mount.nfs(8), nfs(5).

/etc/fstab will look like this:

client.home.lan:/path/to/nfs/mount  /path/to/local/directory   nfs4 timeo=50,ac,fg,user 0  0

One quirk that I found was that if you set sec=krb5 on the server's exports and try to connect to it with ubuntu, it will fail because it still tries to negotiate krb5i. That's why I have set sec=krb5p:krb5i:krb5:sys to cover these corner cases. nfsstat -m to check the result of negotiation.

you can use nfsstat -m (see nfsstat(8)) to see what options have been negatiated between server and client. I see NFSv4.1 with my setup.

Now you should be able to browse your files with appropriate permissions. If you get a "Stale file handle" error, make sure you have logged into kerberos with kinit.

Windows client (Windows 10 Pro with Anniversary update)

Kerberos

First, create a host/windowshostname.fqdn.domain principal on your KDC with a custom password (no --random-key). This password will be used later.

Windows 10 Pro (I don't know about other editions) comes with Kerberos client. The way to use it is through ksetup.exe. See ksetup.exe /? for help. The following is my setup to get it working. ksetup.exe needs Administrator PowerShell (I wasn't able to run ksetup.exe through cmd.exe).

By this time, I have a local user on the windows machine with the same name as a principal on KDC. We'll call it myuser here.

ksetup /setrealm HOME.LAN
ksetup /addkdc HOME.LAN fbsd.home.lan
ksetup /setmachpassword <password you used for host/windowshostname.home.lan principal>
ksetup /setrealmflags HOME.LAN sendaddress delegate
ksetup /mapuser myuser@HOME.LAN myuser

you can test the connection with the following command:

runas /netonly /user:myuser@HOME.LAN cmd.exe

and you should see a cmd.exe pop up under your kerberos user.

Now logout and log back in as user myuser@HOME.LAN (make sure you put the username in with the domain). Now, if you open cmd.exe and run klist, you should get an output like so:

> klist

Current LogonId is 0:0xSOMEHEXVALUE

Cached Tickets: (A NUMBER MORE THAN 0)

#0>   Client: myuser @ HOME.LAN
    Server: krbtgt/HOME.LAN @ HOME.LAN
    KerbTicket
    Start Time
    End Time
    Renew Time
    Session Key
    cache Flags
    Kdc Called

Now you you've successfully logged into kerberos

You can use ksetup.exe without arguments to see current set of options.

I think I used both /setmachpassword and /setcomputerpassword. I don't know which one did the trick.

If you make a mistake with /mapuser, go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos to edit it.

NFSv4

Anniversary Update for Windows 10 Pro enables Services for NFS. Enable NFS support by going to control panel -> programs and options -> add features (or something along those lines) -> Client for NFS.

Now you should be able to mount the NFS drive from cmd.exe with the following command:

mount -o sec=krb5p -o nolock fbsd.home.lan:/my/exported/dir Z:

This will mount you NFSv4 share on Z:\. Make sure you do this in a non-privileged cmd.exe prompt for it to show up on your Explorer.exe. You can find out more about EnableLinkedConnections registry key (thanks dkeav on #freebsd@freenode.net) if you need to mount as administrator and for it to show up for users.

Optional Components

You can use LDAP to sync your UID/GIDs, but I haven't found it necessary to make NFSv4 work well.

Things left to do

I couldn't find a way to have "Map Network Drive" in Explorer.exe mount with custom options (-o nolock seems to be needed for it to work regardless of server-side lockd). So, I have now sufficed to a batch script which runs the above mount command for multiple directories.

Happy NFSv4'ing! Remember this is not a definitive or comprehensive guide on NFSv4+Kerberos. Other configurations might work, but I arrived at this configuration for my own home network. Let me know if you find any errors, or omissions.

Useful links and Credit

Some of the above info have been taken verbatim from below (for convenience).

Kerberos and LDAP how-to

Authenticating Windows 7 against MIT Kerberos 5

Thanks to all on #freebsd@freenode.net for their help! You can find me there as fengshaun.

Update 2021-09-13: everything and everyone has moved from freenode to libera.chat, including me.