JoshuaWise.com

Kerberized NFSv4 HOWTO


Kerberized NFSv4 HOWTO

This page is a running documentation page for setting up Kerberized NFSv4. As I set up my network (and I guess in some cases after I set up my network ...), I'll document what I'm doing so that other people don't have to go through this mess again later. This page is a work in progress.

Why Kerberized NFSv4?

Words to the effect of:

  • Initially wanted setup between nyus and home network
  • nyus lives a fair ways off on the network (over untrusted terrain, anyway)
  • NFSv3 not working out so well; Kerberos supported or not supported depending on who you ask
  • NFSv3 requires RO export hacks (see link 1), and did not want to export RO to the world

Steps:

  • JOSHUAWISE.COM does not have a Kerberos realm yet
  • Set up a Kerberos realm
  • Test it by Kerberizing SSH
  • Try setting up NFS
  • Kerberize client?

Setting up a client

This assumes that you've already set up a Kerberos realm and that you've already also set up your NFS server. As usual, I'll be using the Kerberos realm JOSHUAWISE.COM for testing, and I'll be using the server nyus.joshuawise.com. I'll be setting up the client on shebang.

  • Verify that kernel has NFSv4 support.
    joshua@shebang:/usr/src/linux-2.6.17.7$ grep nfs4 /proc/filesystems
    nodev   nfs4
  • Install krb5 stuff. You may do different things depending on your distro.
    joshua@shebang:~$ sudo apt-get install krb5-clients krb5-user krb5-doc krb5-config
    words

    Prompt shows up: Enter the hostnames of Kerberos servers in the JOSHUAWISE.COM Kerberos realm separated by spaces. How the heck did it know that this is the JOSHUAWISE.COM realm?? I put in kerberos.joshuawise.com, which is a CNAME for joshuawise.com, which is my KDC. I put in kerberos.joshuawise.com for the administrative server, too. Dumped me back to a prompt.

  • Verify that the realm is set up properly in /etc/krb5.conf:
    [libdefaults]
            default_realm = JOSHUAWISE.COM

    Add in the a setting for my realm in the [domain_realm] section:

            joshuawise.com = JOSHUAWISE.COM
            .joshuawise.com = JOSHUAWISE.COM
  • Try kinit'ing:
    joshua@shebang:~$ kinit
    Password for joshua@JOSHUAWISE.COM:
    joshua@shebang:~$
  • Make a directory for nyus:
    joshua@shebang:~$ sudo mkdir /nfs/nyus
  • Try mounting it to see what I did wrong: (this seemed too damn easy!)
    joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus
    Warning: rpc.idmapd appears not to be running.
             All uids will be mapped to the nobody uid.
    mount: wrong fs type, bad option, bad superblock on nyus.joshuawise.com:/,
           missing codepage or other error
           In some cases useful info is found in syslog - try
           dmesg | tail  or so
  • Turn on idmapd and gssd. Edit /etc/default/nfs-common and change the variable set lines that are currently empty so that they look like this:
    # Do you want to start the idmapd daemon? It is only needed for NFSv4.
    NEED_IDMAPD=yes
    
    # Do you want to start the gssd daemon? It is required for Kerberos mounts.
    NEED_GSSD=yes
  • Make those settings take effect:
    joshua@shebang:~$ sudo /etc/init.d/nfs-common restart
  • Try mounting again:
    joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus
    mount: block device nyus.joshuawise.com:/ is write-protected, mounting read-only
    mount: cannot mount block device nyus.joshuawise.com:/ read-only

    WARNING! WARNING! WARNING! mount IS LYING! IGNORE ITS LIES! THEY SERVE ONLY TO CONFUSE YOU! “write-protected”, especially when followed by “cannot mount read-only”, really means “the mount failed when I tried to mount it read-write for an unspecified reason”, not that it is actually write-protected. Take a look in syslog to see what the issue was:

    Jan 21 22:16:17 shebang rpc.gssd[6676]: ERROR: No such file or directory while beginning keytab scan for keytab '/etc/krb5.keytab'
    Jan 21 22:16:17 shebang rpc.gssd[6676]: ERROR: No usable keytab entries found in keytab '/etc/krb5.keytab'
    Jan 21 22:16:17 shebang rpc.gssd[6676]: Do you have a valid keytab entry for nfs/<your.host>@<YOUR.REALM> in keytab file /etc/krb5.keytab ?
    Jan 21 22:16:17 shebang rpc.gssd[6676]: Continuing without (machine) credentials - nfs4 mounts with Kerberos will fail
    Jan 21 22:17:34 shebang rpc.gssd[6676]: WARNING: Failed to obtain machine credentials for connection to server NYUS.RES.cmu.edu
    Jan 21 22:17:34 shebang rpc.gssd[6676]: WARNING: Failed to obtain machine credentials for connection to server NYUS.RES.cmu.edu
  • Two issues here. Issue one – no keytab. Resolve this by kadminning up and getting the NFS key for nyus.
    joshua@shebang:~$ sudo kadmin
    Authenticating as principal root/admin@JOSHUAWISE.COM with password.
    Password for root/admin@JOSHUAWISE.COM: <type my password here>
    kadmin:  ktadd nfs/nyus.joshuawise.com
    Entry for principal nfs/nyus.joshuawise.com with kvno 4, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab.
    Entry for principal nfs/nyus.joshuawise.com with kvno 4, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab.
    kadmin:  ^Djoshua@shebang:~$

    If you get issues here to the effect of:

    kadmin: Operation requires ``change-password'' privilege while changing nfs/nyus.joshuawise.com@JOSHUAWISE.COM's key

then you have not restarted the Kerberos admin server after adding the */admin * ACL. This can be frustratingly difficult to discover, especially when you thought you restarted it earlier after setting that the first time around, and you thought that anyway it should've discovered it by now, and what is it doing changing a key anyway – shouldn't it be just adding it to the keytab?, and you're still not certain that that's what are supposed to do there, and ... Well, you get the picture.

  • Restart gssd. /etc/init.d/nfs-common restart
  • Issue two – NYUS.RES.cmu.edu != nyus.joshuawise.com. Add to /etc/hosts:
    128.2.162.192 nyus.joshuawise.com
  • Retry the mount:
    joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus
    mount: Connection timed out

    Now what's wrong?!... shebang is behind NAT, but who cares? dmesg reports:

    [17341332.176000] RPC: AUTH_GSS upcall timed out.
    [17341332.176000] Please check user daemon is running!

    OK, so I restarted rpc.gssd using rpc.gssd -vvvf, and got this:

    joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus
    handling krb5 upcall
    Using keytab file '/etc/krb5.keytab'
    INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM' are good until 1169473613
    using FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM as credentials cache for machine creds
    using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM
    creating context using fsuid 0 (save_uid 0)
    creating tcp client for server nyus.joshuawise.com
    creating context with server nfs@nyus.joshuawise.com

    long pause...

    WARNING: Failed to create krb5 context for user with uid 0 for server nyus.joshuawise.com
    WARNING: Failed to create krb5 context for user with uid 0 with credentials cache FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM for server nyus.joshuawise.com
    WARNING: Failed to create krb5 context for user with uid 0 with any credentials cache for server nyus.joshuawise.com
    doing error downcall
    destroying client clnt12
    mount: block device nyus.joshuawise.com:/ is write-protected, mounting read-only

    Note: It appears that rpc.gssd died because I control-c'ed a mount. After I restarted rpc.gssd again, I got some new messages in addition to the old ones:

    handling krb5 upcall
    WARNING: failed reading uid from krb5 upcall pipe: No such file or directory
    WARNING: can't create tcp rpc_clnt for server nyus.joshuawise.com for user with uid 0: RPC: Remote system error - Connection timed out

    OK, well, I should've checked nyus's syslog:

    Jan 21 23:37:04 localhost rpc.svcgssd[8000]: WARNING: gss_accept_sec_context failed
    Jan 21 23:37:04 localhost rpc.svcgssd[8000]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): Miscellaneous failure - Key version number for principal in key table is incorrect
    Jan 21 23:37:04 localhost rpc.svcgssd[8000]: WARNING: failed to write message
  • Verify and fix key versions. A key version, evidently, is also known as a “kvno”. When I did the “ktadd” on shebang, it incremented the kvno, which meant that shebang had the only up to date principal! When I did the ktadd again on nyus to update it, of course, I just broke shebang and dashnine. To prove that this is how it broke, I can do this:
    joshua@shebang:~$ sudo klist -e -k -t /etc/krb5.keytab
    Keytab name: FILE:/etc/krb5.keytab
    KVNO Timestamp         Principal
    ---- ----------------- --------------------------------------------------------
       4 01/21/07 22:24:33 nfs/nyus.joshuawise.com@JOSHUAWISE.COM (Triple DES cbc mode with HMAC/sha1)
       4 01/21/07 22:24:33 nfs/nyus.joshuawise.com@JOSHUAWISE.COM (DES cbc mode with CRC-32)
    joshua@shebang:~$ sudo kadmin
    Authenticating as principal root/admin@JOSHUAWISE.COM with password.
    Password for root/admin@JOSHUAWISE.COM:
    kadmin:  getprinc nfs/nyus.joshuawise.com@JOSHUAWISE.COM
    Principal: nfs/nyus.joshuawise.com@JOSHUAWISE.COM
    <blah blah>
    Number of keys: 2
    Key: vno 7, Triple DES cbc mode with HMAC/sha1, no salt
    Key: vno 7, DES cbc mode with CRC-32, no salt

    I just ended up ktremove'ing the nyus keys from shebang, and ktadding to get the latest on nyus.

  • Make a keytab on shebang. I tturns out that I didn't need nyus's key on shebang, I just needed some key. So, I did this:
    joshua@shebang:~$ sudo kadmin
    Authenticating as principal root/admin@JOSHUAWISE.COM with password.
    Password for root/admin@JOSHUAWISE.COM:
    kadmin:  addprinc -randkey host/shebang.joshuawise.com
    WARNING: no policy specified for host/shebang.joshuawise.com@JOSHUAWISE.COM; defaulting to no policy
    Principal "host/shebang.joshuawise.com@JOSHUAWISE.COM" created.
    kadmin:  addprinc -randkey nfs/shebang.joshuawise.com
    WARNING: no policy specified for nfs/shebang.joshuawise.com@JOSHUAWISE.COM; defaulting to no policy
    Principal "nfs/shebang.joshuawise.com@JOSHUAWISE.COM" created.
    kadmin:  ktadd host/shebang.joshuawise.com
    Entry for principal host/shebang.joshuawise.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab.
    Entry for principal host/shebang.joshuawise.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab.
    kadmin:  ktadd nfs/shebang.joshuawise.com
    Entry for principal nfs/shebang.joshuawise.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab.
    Entry for principal nfs/shebang.joshuawise.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab.
    kadmin:  ^Djoshua@shebang:~$
  • Restart gssd to use the new keytab.
    joshua@shebang:~$ sudo killall rpc.gssd
    joshua@shebang:~$ sudo killall rpc.gssd
    rpc.gssd: no process killed
    joshua@shebang:~$ sudo rpc.gssd
    joshua@shebang:~$
  • Mount nyus.
    joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus
    joshua@shebang:~$
  • Test the security of Kerberized NFS.
    joshua@shebang:~$ kinit
    Password for joshua@JOSHUAWISE.COM:
    joshua@shebang:~$ ls /nfs/nyus
    bin  boot  dev  etc  home  initrd  initrd.img  lib  lost+found  media  mnt  nfs  opt  proc  root  sbin  srv  sys  tmp  usr  var  vmlinuz
    joshua@shebang:~$ touch /nfs/nyus/foo
    touch: cannot touch `/nfs/nyus/foo': Permission denied
    joshua@shebang:~$ touch /nfs/nyus/home/joshua/foo
    joshua@shebang:~$ sudo rm /nfs/nyus/home/joshua/foo # because root has not authenticated as joshua
    rm: remove write-protected regular empty file `/nfs/nyus/home/joshua/foo'? y
    rm: cannot remove `/nfs/nyus/home/joshua/foo': Permission denied
    joshua@shebang:~$ rm /nfs/nyus/home/joshua/foo
    joshua@shebang:~$
  • Add nyus to /etc/fstab:
    nyus.joshuawise.com:/ /nfs/nyus nfs4	user,exec,rsize=8192,wsize=8192,intr,rw,sec=krb5i       0	0

    ...and give it a shot...

    joshua@shebang:~$ kdestroy
    joshua@shebang:~$ sudo umount /nfs/nyus
    joshua@shebang:~$ mount /nfs/nyus
    joshua@shebang:~$ ls /nfs/nyus
    ls: /nfs/nyus: Permission denied
    joshua@shebang:~$ kinit
    Password for joshua@JOSHUAWISE.COM:
    joshua@shebang:~$ ls /nfs/nyus
    bin  boot  dev  etc  home  initrd  initrd.img  lib  lost+found  media  mnt  nfs  opt  proc  root  sbin  srv  sys  tmp  usr  var  vmlinuz
    joshua@shebang:~$ kdestroy
    joshua@shebang:~$ ls /nfs/nyus
    bin  boot  dev  etc  home  initrd  initrd.img  lib  lost+found  media  mnt  nfs  opt  proc  root  sbin  srv  sys  tmp  usr  var  vmlinuz
  • Presto! A mostly working Kerberized NFSv4 setup, in only 19 mind-bogglingly overcomplicated steps! There, now that wasn't so bad, was it?

Setting up PAM

Mode notes:

  • Install pam_krb5: joshua@nyus:~$ sudo apt-get install libpam-krb5
  • Add pam_krb5 to /etc/pam.d/common-auth as first line: auth sufficient pam_krb5.so ignore_root
  • Add pam_krb5 to /etc/pam.d/common-session as first line: session optional pam_krb5.so ignore_root
  • Add pam_krb5 to /etc/pam.d/common-account as first line: account required pam_krb5.so ignore_root
  • Add pam_krb5 to /etc/pam.d/common-password as first line: password optional pam_krb5.so ignore_root
  • Move away allowed keys to force a password login: joshua@nyus:~/.ssh$ mv authorized_keys2 authorized_keys2-old
  • Change password to something that is not my UNIX password: joshua@shebang:~$ kpasswd
  • Try connecting: joshua@shebang:~$ ssh nyus
  • Observe that it uses the new Kerberos password.
  • Change the password using PAM passwd: joshua@nyus:~$ passwd
  • Connect again and observe that the password has been changed.
  • Observe that 'klist' shows that you have a TGT after logging in over SSH using the PAM module.
  • You now have working pam_krb5. passwd is a little messy:
    joshua@nyus:~/.ssh$ passwd
    Password:
    Changing password for joshua
    (current) UNIX password:
    Enter new password:
    Enter it again:
    Enter new UNIX password:
    Retype new UNIX password:
    passwd: password updated successfully

Here's some stuff I've referenced while setting this system up.