Tuesday, October 20, 2020

Install and Configure Kerberos Authentication on Red Hat Enterprise Linux 8

The demonstration includes three installation/configuration parts,

  Part 1. Install and configure Kerberos Key Distribution Center (KDC) Server
  
     Server name: xdc01.lab.dbaplus.ca
     OS version:  Red Hat Enterprise Linux (RHEL) 8

  Part 2. Install and configure Kerberos client on application (SSH) server
  
     Server 1: host01.lab.dbaplus.ca
     Server 2: host02.lab.dbaplus.ca
     OS version:  Red Hat Enterprise Linux (RHEL) 8
  
  Part 3. Install and configure Kerberos client on workstation to test single-sign-on (SSO) for SSH authentication with Kerberos
  
     Workstation name: wkstn01.lab.dbaplus.ca
     OS version: Ubuntu 16.04

     I use machine running Ubuntu because I do not have fourth machine running RHEL 8. Technically, the OS type should not be the issue if the correct version of Kerberos software is installed.

     The test will be done as following,

     A. Run ssh to log into wkstn01 with Kerberos user name (principle) and password
     B. From wkstn01, run ssh to log into server host01 or host02 without user name and password to justify SSO.
     C. From wkstn01, run ssh to log into server host01 (or host02), then run ssh to log into host02 (or host01) to justify authentication delegation.

1. Install and configure Kerberos KDC Server on RHEL 8

1.1 Install the required packages for KDC

Following three packages are required for the KDC on RHEL 8 host,

  krb5-libs - contains the shared libraries needed by Kerberos 5. If you are using Kerberos, you need to install this package.
  krb5-server - contains the programs that must be installed on a Kerberos 5 key distribution center (KDC). 
  krb5-workstation - contains the basic Kerberos programs (kinit, klist, kdestroy, kpasswd).
  
Install these packages with yum,  
[root@xdc01]# yum install krb5-server krb5-libs krb5-workstation
Updating Subscription Management repositories.
Last metadata expiration check: 0:00:54 ago on Sun 13 Sep 2020 03:44:33 PM EDT.
Package krb5-libs-1.17-18.el8.x86_64 is already installed.
Dependencies resolved.
========================================================================================
 Package                    Architecture      Version           Repository         Size
========================================================================================
Installing:
 krb5-server                x86_64            1.17-18.el8       InstallMedia      1.1 M
 krb5-workstation           x86_64            1.17-18.el8       InstallMedia      940 k
Installing dependencies:
 libkadm5                   x86_64            1.17-18.el8       InstallMedia      185 k
 libverto-libevent          x86_64            0.3.0-5.el8       InstallMedia       16 k

Transaction Summary
========================================================================================
Install  4 Packages

Total size: 2.2 M
Installed size: 4.9 M
Is this ok [y/N]: y
Downloading Packages:
warning: /mnt/BaseOS/Packages/krb5-server-1.17-18.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
DVD for Red Hat Enterprise Linux 8.2 Server                                                        1.1 MB/s | 5.0 kB     00:00
Importing GPG key 0xFD431D51:
 Userid     : "Red Hat, Inc. (release key 2) <security@redhat.com>"
 Fingerprint: 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Key imported successfully
Importing GPG key 0xD4082792:
 Userid     : "Red Hat, Inc. (auxiliary key) <security@redhat.com>"
 Fingerprint: 6A6A A7C9 7C88 90AE C6AE BFE2 F76F 66C3 D408 2792
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                           1/1
  Installing       : libkadm5-1.17-18.el8.x86_64                                                                               1/4
  Installing       : libverto-libevent-0.3.0-5.el8.x86_64                                                                      2/4
  Installing       : krb5-server-1.17-18.el8.x86_64                                                                            3/4
  Running scriptlet: krb5-server-1.17-18.el8.x86_64                                                                            3/4
  Installing       : krb5-workstation-1.17-18.el8.x86_64                                                                       4/4
  Running scriptlet: krb5-workstation-1.17-18.el8.x86_64                                                                       4/4
  Verifying        : krb5-server-1.17-18.el8.x86_64                                                                            1/4
  Verifying        : krb5-workstation-1.17-18.el8.x86_64                                                                       2/4
  Verifying        : libkadm5-1.17-18.el8.x86_64                                                                               3/4
  Verifying        : libverto-libevent-0.3.0-5.el8.x86_64                                                                      4/4
Installed products updated.

Installed:
  krb5-server-1.17-18.el8.x86_64                krb5-workstation-1.17-18.el8.x86_64          libkadm5-1.17-18.el8.x86_64
  libverto-libevent-0.3.0-5.el8.x86_64

Complete!

Run rpm to confirm the packages are installed
[root@xdc01]# rpm -qa | grep krb5
krb5-workstation-1.17-18.el8.x86_64
krb5-server-1.17-18.el8.x86_64
krb5-libs-1.17-18.el8.x86_64

1.2 Configure KDC

Edit file /var/kerberos/krb5kdc/kdc.conf and have the following lines included in the file
[kdcdefaults]
    kdc_ports = 88
    kdc_tcp_ports = 88
    spake_preauth_kdc_challenge = edwards25519

[realms]
LAB.DBAPLUS.CA = {
     #master_key_type = aes256-cts
     acl_file = /var/kerberos/krb5kdc/kadm5.acl
     dict_file = /usr/share/dict/words
     admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
     supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal
}

Here, LAB.DBAPLUS.CA is the realm name and always uppercase. You can define your own realm name, make sure your realm name in uppercase.

Edit file /etc/krb5.conf to reflect the realm name and domain-to-realm mappings as following
[logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log

[libdefaults]
    dns_lookup_realm = false
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    rdns = false
    pkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crt
    spake_preauth_groups = edwards25519
    default_realm = LAB.DBAPLUS.CA
    default_ccache_name = KEYRING:persistent:%{uid}

[realms]
 LAB.DBAPLUS.CA = {
     kdc = xdc01.lab.dbaplus.ca
     admin_server = xdc01.lab.dbaplus.ca
 }

[domain_realm]
 .lab.dbaplus.ca = LAB.DBAPLUS.CA
 lab.dbaplus.ca = LAB.DBAPLUS.CA

Under section [realms], xdc01.lab.dbaplus.ca is host name of this server, it is the KDC and administration server of realm LAB.DBAPLUS.CA. Make sure realm LAB.DBAPLUS.CA is the realm configured in file /var/kerberos/krb5kdc/kdc.conf.

Under section [domain_realm], it defined domain-to-realm mappings. I choose LAB.DBAPLUS.CA as the realm for domain lab.dbaplus.ca. It is a good choice to use domain name as realm name (but in uppercase), but it is not MUST. No matter what you choose as realm name, the domain name has to be a real domain which you are going to map to the realm.

1.3 Create the database using the kdb5_util utility

Run kdb5_util to create the database that stores keys for the Kerberos realm as following
[root@xdc01]# kdb5_util create -s
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'LAB.DBAPLUS.CA',
master key name 'K/M@LAB.DBAPLUS.CA'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:

The -s argument creates a stash file in which the master server key is stored. If no stash file is present from which to read the key, the Kerberos server (krb5kdc) prompts the user for the master server password (which can be used to regenerate the key) every time it starts.

1.4 Configure the Kerberos database ACL

Edit file /var/kerberos/krb5kdc/kadm5.acl to replace following line 
*/admin@EXAMPLE.COM     *

with line
*/admin@LAB.DBAPLUS.CA     *

Note, use your own realm instead of LAB.DBAPLUS.CA when configuring the file on your KDC.

The kadm5.acl file is used by Kerberos admin service (kadmin) to determine which principals have administrative access to the Kerberos database and their level of access. In previous configuration, any principal in the LAB.DBAPLUS.CA realm with an admin instance has all administrative privileges. For example, principal joe/admin@LAB.DBAPLUS.CA are able to exert full administrative control over the realm's Kerberos database.

After kadmin has been started on the server, any user (principal) can access its services by running kadmin on any of the clients or servers in the realm. However, only users (principals) listed in the kadm5.acl file can modify the database in any way, except for changing their own passwords. Anyone can change his own principal password.

1.5 Create the first principal using kadmin.local

The command kadmin.local is designed to be used on the same host as the KDC and does not use Kerberos for authentication. Therefore, it can be used to create the first principal
[root@xdc01]# kadmin.local
Authenticating as principal root/admin@LAB.DBAPLUS.CA with password.
kadmin.local:  addprinc dbaplus/admin
WARNING: no policy specified for dbaplus/admin@LAB.DBAPLUS.CA; defaulting to no policy
Enter password for principal "dbaplus/admin@LAB.DBAPLUS.CA":
Re-enter password for principal "dbaplus/admin@LAB.DBAPLUS.CA":
Principal "dbaplus/admin@LAB.DBAPLUS.CA" created.
kadmin.local:  exit

The principal dbaplus is created with instance admin in realm LAB.DBAPLUS.CA (dbaplus/admin@LAB.DBAPLUS.CA). According to the configuration in kadm5.acl file, this first principal has full administrative control over the realm's Kerveros database.

Of course, you can use whatever name you like instead of dbaplus, it is totally up to you. Many people like to use admin (admin/admin@LAB.DBAPLUS.CA) as first principal, but it is not best practice for security policy.

Now, you can be authenticated by KDC as principal dbaplus when you access servers in realm LAB.DBAPLUS.CA.

1.6 Start Kerberos services

Start krb5kdc and kadmin services using command systemctl as following
[root@xdc01]# systemctl start krb5kdc.service
[root@xdc01]# systemctl start kadmin.service

Confirm krb5kdc and kadmin are running using command netstat
[root@xdc01]# netstat -plntu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:464             0.0.0.0:*               LISTEN      25200/kadmind
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      880/sshd
tcp        0      0 0.0.0.0:88              0.0.0.0:*               LISTEN      25142/krb5kdc
tcp        0      0 0.0.0.0:749             0.0.0.0:*               LISTEN      25200/kadmind
tcp6       0      0 :::464                  :::*                    LISTEN      25200/kadmind
tcp6       0      0 :::22                   :::*                    LISTEN      880/sshd
tcp6       0      0 :::88                   :::*                    LISTEN      25142/krb5kdc
tcp6       0      0 :::749                  :::*                    LISTEN      25200/kadmind
udp        0      0 127.0.0.1:323           0.0.0.0:*                           832/chronyd
udp        0      0 0.0.0.0:464             0.0.0.0:*                           25200/kadmind
udp        0      0 0.0.0.0:88              0.0.0.0:*                           25142/krb5kdc
udp6       0      0 ::1:323                 :::*                                832/chronyd
udp6       0      0 :::464                  :::*                                25200/kadmind
udp6       0      0 :::88                   :::*                                25142/krb5kdc

It shows that process krb5kdc (pid 25142, daemon process of krb5kdc service) is listening on port 88 for tcp/udp and process kadmind (pid 25200, daemon process of kadmin service) is listening on port 464 for tcp/udp and port 749 for tcp. it means both services are running. The ports used by krb5kdc and kadmind are commonly defined as network service as following

   kerberos  tcp/88 & udp/88    (Kerberos network authentication protocol server - KDC)
   kadmin    tcp/749            (Kerberos Administration Protocol)
   kpasswd   tcp/464 & udp/464  (Kerberos password server)

Note: kerberos, kadmin and kpasswd are network services and different from krb5kdc and kadmind which are system services, though they are all called service. Network service is helpful when you configure firewall.
   
The status of krb5kdc and kadmin can also be checked using command systemctl
[root@xdc01]# systemctl status krb5kdc.service
krb5kdc.service - Kerberos 5 KDC
   Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-09-13 20:27:52 EDT; 19min ago
  Process: 25141 ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid $KRB5KDC_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 25142 (krb5kdc)
    Tasks: 1 (limit: 5016)
   Memory: 3.1M
   CGroup: /system.slice/krb5kdc.service
           └─25142 /usr/sbin/krb5kdc -P /var/run/krb5kdc.pid
[root@xdc01]#
[root@xdc01]# systemctl status kadmin.service
kadmin.service - Kerberos 5 Password-changing and Administration
   Loaded: loaded (/usr/lib/systemd/system/kadmin.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-09-13 20:56:11 EDT; 4min 56s ago
  Process: 25199 ExecStart=/usr/sbin/kadmind -P /var/run/kadmind.pid $KADMIND_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 25200 (kadmind)
    Tasks: 1 (limit: 5016)
   Memory: 11.7M
   CGroup: /system.slice/kadmin.service
           └─25200 /usr/sbin/kadmind -P /var/run/kadmind.pid

1.7 Verify that the KDC is working

Run kinit to obtain a ticket from KDC
[root@xdc01]# kinit dbaplus/admin
Password for dbaplus/admin@LAB.DBAPLUS.CA:

The command kinit may get following error,
[root@xdc01]# kinit dbaplus/admin
kinit: Cannot contact any KDC for realm 'LAB.DBAPLUS.CA' while getting initial credentials

If the error happened, check following items

   *  Section [realms] of file /etc/krb5.conf, make sure value of "kdc" entry is same as KDC server name

   *  File /etc/hosts, make sure the KDC server name is in the file unless the server name is resolved correctly by DNS 
   
   *  Service krb5kdc status, make krb5kdc is running
   
Run klist to confirm the ticket is cached by kinit
[root@xdc01]# klist
Ticket cache: KCM:0
Default principal: dbaplus/admin@LAB.DBAPLUS.CA

Valid starting       Expires              Service principal
09/15/2020 14:43:49  09/16/2020 14:43:37  krbtgt/LAB.DBAPLUS.CA@LAB.DBAPLUS.CA
        renew until 09/15/2020 14:43:49

If kinit and klist can be executed successfully, it proves KDC is working well.

1.8 Have firewall work with Kerberos authentication if the firewall is enabled

Check if firewall is enabled and started using command systemctl status firewalld.service.

The firewall is enabled and started if the command shows
[root@xdc01]# systemctl status firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2020-09-14 18:45:35 EDT; 1h 39min ago
     Docs: man:firewalld(1)
 Main PID: 25960 (firewalld)
    Tasks: 2 (limit: 5016)
   Memory: 23.7M
   CGroup: /system.slice/firewalld.service
           └─25960 /usr/libexec/platform-python -s /usr/sbin/firewalld --nofork --nopid

The firewall is disabled and stopped if the command shows
[root@xdc01]# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

The firewall is enabled and stopped if the command shows
[root@xdc01]# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

If the firewall is enabled, we have to configure it to work with Kerberos. Before configuring filewall, start the firewall using command systemctl start firewalld.service if the firewall is not running
[root@xdc01]# systemctl start firewalld.service

[root@xdc01]# systemctl status firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2020-09-14 20:43:27 EDT; 9s ago
     Docs: man:firewalld(1)
 Main PID: 26219 (firewalld)
    Tasks: 2 (limit: 5016)
   Memory: 22.6M
   CGroup: /system.slice/firewalld.service
           └─26219 /usr/libexec/platform-python -s /usr/sbin/firewalld --nofork --nopid  

Note: If firewall is disabled or not installed, there is no need to configure firewall. If the firewall is enabled, the firewalld service will be automatically started after the server is restarted even if it is currently stopped. Therefore, make sure the firewall is configured with Kerberos no matter the firewalld service is running or not.

List firewall rules of active zone
[root@xdc01]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s3 enp0s8
  sources:
  services: cockpit dhcpv6-client ssh
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

The active zone is public. Kerberos related network services are not in the services list. Add the Kerberos network services to the active zone
[root@xdc01]# firewall-cmd --add-service=kerberos --permanent
success
[root@xdc01]# firewall-cmd --add-service=kadmin --permanent
success
[root@xdc01]# firewall-cmd --add-service=kpasswd --permanent
success

The Kerveros network services (kerberos, kadmin and kpasswd) are added to firewall configuration permanently, but they are not activated in current firewall session. Restarting filewalld service to make new rule active using command systemctl restart firewalld.service
[root@xdc01]# firewall-cmd --list-services
cockpit dhcpv6-client ssh
[root@xdc01]#
[root@xdc01]# systemctl restart firewalld.service
[root@xdc01]#
[root@xdc01]# firewall-cmd --list-services
cockpit dhcpv6-client kadmin kerberos kpasswd ssh
[root@xdc01]#

2. Install and configure Kerberos client on application (SSH) server

In order to have a whole picture of using Kerberos in single sing on (SSO) and authentication delegation, I am going to configure Kerberos client on two SSH servers. 

2.1 Test connectivity between application (SSH) servers and KDC

Before starting Kerberos client installation on application servers, make sure KDC Kerberos services are accessible from application servers.

Test that KDC host is reachable using ping on application servers host01 and host02
[root@host01]# ping -c 1 xdc01.lab.dbaplus.ca
PING xdc01.lab.dbaplus.ca (10.10.25.254) 56(84) bytes of data.
64 bytes from xdc01.lab.dbaplus.ca (10.10.25.254): icmp_seq=1 ttl=64 time=1.12 ms

--- xdc01.lab.dbaplus.ca ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.115/1.115/1.115/0.000 ms

[root@host02]# ping -c 1 xdc01.lab.dbaplus.ca
PING xdc01.lab.dbaplus.ca (10.10.25.254) 56(84) bytes of data.
64 bytes from xdc01.lab.dbaplus.ca (10.10.25.254): icmp_seq=1 ttl=64 time=2.49 ms

--- xdc01.lab.dbaplus.ca ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.490/2.490/2.490/0.000 ms

Test whether Kerberos services is accessible using telnet on both servers
[root@host01]# telnet xdc01.lab.dbaplus.ca 88
Trying 10.10.25.254...
Connected to xdc01.lab.dbaplus.ca.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

The telnet session can be closed by pressing Ctrl+] and entering command quit. By default, RHEL 8 does not install telnet command, you can use nc instead
[root@host01]# nc -vz xdc01.lab.dbaplus.ca 88
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 10.10.25.254:88.
Ncat: 0 bytes sent, 0 bytes received in 0.03 seconds.

If Kerberos service krb5kdc is not running or the firewall on KDC is not configured properly, telnet/nc will return as following,
[root@host01]# nc -vz xdc01.lab.dbaplus.ca 88
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: No route to host.

[root@host01]# telnet xdc01.lab.dbaplus.ca 88
Trying 10.10.25.254...
telnet: connect to address 10.10.25.254: No route to host

2.2 Install and configure Kerberos client on RHEL 8 application (SSH) servers

Install the krb5-libs and krb5-workstation packages on both servers
[root@host01]# yum install krb5-workstation krb5-libs
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
DVD for Red Hat Enterprise Linux 8.2 Server             44 MB/s | 2.3 MB     00:00
Last metadata expiration check: 0:00:01 ago on Tue 15 Sep 2020 07:23:58 PM EDT.
Package krb5-libs-1.17-18.el8.x86_64 is already installed.
Dependencies resolved.
=================================================================================
 Package                    Architecture     Version        Repository      Size
=================================================================================
Installing:
 krb5-workstation           x86_64           1.17-18.el8    InstallMedia   940 k
Installing dependencies:
 libkadm5                   x86_64           1.17-18.el8    InstallMedia   185 k

Transaction Summary
=================================================================================
Install  2 Packages

Total size: 1.1 M
Installed size: 3.5 M
Is this ok [y/N]: y
Downloading Packages:
warning: /mnt/BaseOS/Packages/krb5-workstation-1.17-18.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
DVD for Red Hat Enterprise Linux 8.2 Server           1.0 MB/s | 5.0 kB     00:00
Importing GPG key 0xFD431D51:
 Userid     : "Red Hat, Inc. (release key 2) <security@redhat.com>"
 Fingerprint: 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Key imported successfully
Importing GPG key 0xD4082792:
 Userid     : "Red Hat, Inc. (auxiliary key) <security@redhat.com>"
 Fingerprint: 6A6A A7C9 7C88 90AE C6AE BFE2 F76F 66C3 D408 2792
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                          1/1
  Installing       : libkadm5-1.17-18.el8.x86_64                              1/2
  Installing       : krb5-workstation-1.17-18.el8.x86_64                      2/2
  Running scriptlet: krb5-workstation-1.17-18.el8.x86_64                      2/2
  Verifying        : krb5-workstation-1.17-18.el8.x86_64                      1/2
  Verifying        : libkadm5-1.17-18.el8.x86_64                              2/2
Installed products updated.

Installed:
  krb5-workstation-1.17-18.el8.x86_64                 libkadm5-1.17-18.el8.x86_64

Complete!

Copy /etc/krb5.conf from KDC to both servers
[root@host01]# scp xdc01:/etc/krb5.conf /etc/krb5.conf
root@xdc01's password:
krb5.conf                                           100%  823   311.0KB/s   00:00

2.3 Add host principal for each server on the KDC

Before a server can use Kerberos to authenticate users who request to access the service (e.g. SSH) running on the server, it must have its own host principal in the Kerberos database.

On KDC, run kadmin.local to add host principals for both servers as following
[root@xdc01]# kadmin.local -q "addprinc -randkey host/host01.lab.dbaplus.ca"
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
WARNING: no policy specified for host/host01.lab.dbaplus.ca@LAB.DBAPLUS.CA; defaulting to no policy
Principal "host/host01.lab.dbaplus.ca@LAB.DBAPLUS.CA" created.

[root@xdc01]# kadmin.local -q "addprinc -randkey host/host02.lab.dbaplus.ca"
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
WARNING: no policy specified for host/host02.lab.dbaplus.ca@LAB.DBAPLUS.CA; defaulting to no policy
Principal "host/host02.lab.dbaplus.ca@LAB.DBAPLUS.CA" created.

2.4 Download host principal to its respective server

Run kinit on both servers to obtain initial ticket from KDC
[root@host01]# kinit dbaplus/admin
Password for dbaplus/admin@LAB.DBAPLUS.CA:

[root@host02]# kinit dbaplus/admin
Password for dbaplus/admin@LAB.DBAPLUS.CA:

After successfully getting initial ticket, when you run Kerberos client commands (e.g. kadmin, kpasswd) as root on the servers, the commands will, by default, use principal dbaplus/admin as authentication user to access Kerberos database. Otherwise, the commands will user root/admin which does not exist in Kerberos database. You may be wondering if we can create principal root/admin. Technically, we can create principal root/admin, but it is not recommended.

Run kadmin command ktadd on server host01 to download principle "host/host01.lab.dbaplus.ca" which is created at step 2.3
[root@host01]# kadmin -q "ktadd host/host01.lab.dbaplus.ca"
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
Password for dbaplus/admin@LAB.DBAPLUS.CA:
Entry for principal host/host01.lab.dbaplus.ca with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host01.lab.dbaplus.ca with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host01.lab.dbaplus.ca with kvno 2, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host01.lab.dbaplus.ca with kvno 2, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host01.lab.dbaplus.ca with kvno 2, encryption type camellia256-cts-cmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host01.lab.dbaplus.ca with kvno 2, encryption type camellia128-cts-cmac added to keytab FILE:/etc/krb5.keytab.

Run kadmin command ktadd on server host02 to download principle "host/host02.lab.dbaplus.ca" which is created at step 2.3
[root@host02]# kadmin -q "ktadd host/host02.lab.dbaplus.ca"
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
Password for dbaplus/admin@LAB.DBAPLUS.CA:
Entry for principal host/host02.lab.dbaplus.ca with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host02.lab.dbaplus.ca with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host02.lab.dbaplus.ca with kvno 3, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host02.lab.dbaplus.ca with kvno 3, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host02.lab.dbaplus.ca with kvno 3, encryption type camellia256-cts-cmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host02.lab.dbaplus.ca with kvno 3, encryption type camellia128-cts-cmac added to keytab FILE:/etc/krb5.keytab.

You may like to run kamin in interactive mode instead of 'kadmin -q' as following,
[root@host02]# kadmin
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
Password for dbaplus/admin@LAB.DBAPLUS.CA:
kadmin:  ktadd host/host02.lab.dbaplus.ca

You may get following errors when run kadmin command ktadd
[root@host01]# kadmin
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
Password for dbaplus/admin@LAB.DBAPLUS.CA:
kadmin:  ktadd -k /etc/krb5.keytab host/host01.lab.dbaplus.ca
kadmin: Operation requires ``change-password'' privilege while changing host/host01.lab.dbaplus.ca's key

kadmin:  list_principals
get_principals: Operation requires ``list'' privilege while retrieving list.

The cause is the configuration error in /var/kerberos/krb5kdc/kadm5.acl, make sure the file is configured as in step 1.4.

So far, Kerberos KDC and client are configured successfully. But what can we do with the configuration? The answer is that many applications/serivces are able to adopt Kerberos as user authentication solution, such as SSH, Telnet, Samba, NFS, SQL Server and Oracle database, etc. Today, I am going to configure SSH with Kerberos authentication to enjoy Kerberos in real world. 

3. Test single-sign-on (SSO) for SSH authentication with Kerberos

3.1 Configure SSH to use Kerberos authentication on both servers (host01 & host02)

Edit file /etc/ssh/sshd_config and add following lines,
    KerberosAuthentication yes
    KerberosTicketCleanup yes
    KerberosOrLocalPasswd yes

Here, 

  * KerberosAuthentication  Specifies whether the password provided by the user for PasswordAuthentication will be validated through the Kerberos KDC.  To use this option, the server needs a Kerberos keytab which allows the verification of the KDC's identity. 
  
  * KerberosTicketCleanup   Specifies whether to automatically destroy the user's ticket cache file on logout. 
  
  * KerberosOrLocalPasswd   If password authentication through Kerberos fails then the password will be validated via any additional local mechanism such as  /etc/passwd.

Restart sshd
[root@host01]# systemctl restart sshd.service

[root@host02]# systemctl restart sshd.service

3.2 Create users for testing on both SSH servers (host01 & host02)

Run useradd on both servers to create testing user user01, user02
[root@host01]# useradd user01
[root@host01]# useradd user02
[root@host01]# passwd --status user01
user01 LK 2020-09-15 0 99999 7 -1 (Password locked.)
[root@host01]# passwd --status user02
user02 LK 2020-09-16 0 99999 7 -1 (Password locked.)

[root@host02]# useradd user01
[root@host02]# useradd user02
[root@host02]# passwd --status user01
user01 LK 2020-09-15 0 99999 7 -1 (Password locked.)
[root@host02]# passwd --status user02
user02 LK 2020-09-16 0 99999 7 -1 (Password locked.)
 
Users user01 & user02 are created on both servers (host01 & host02), and the users are locked by default. That means user01 & user02 cannot log into host01 & host02 with local password authentication.

3.3 Create principals user01 & user02 which will be 'mapped' to OS users with same names respectively

Create principals user01 & user02 on KDC
[root@xdc01]# kadmin.local -q "addprinc user01"
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.1
WARNING: no policy specified for user01@LAB.DBAPLUS.CA; defaulting to no policy
Enter password for principal "user01@LAB.DBAPLUS.CA":
Re-enter password for principal "user01@LAB.DBAPLUS.CA":
Principal "user01@LAB.DBAPLUS.CA" created.

[root@xdc01]# kadmin.local -q "addprinc user02"
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
WARNING: no policy specified for user02@LAB.DBAPLUS.CA; defaulting to no policy
Enter password for principal "user02@LAB.DBAPLUS.CA":
Re-enter password for principal "user02@LAB.DBAPLUS.CA":
Principal "user02@LAB.DBAPLUS.CA" created.

Enter a password when you are being asked. I entered the password same as principal (user name) to make test simple.

Actually, creating principal can be done with kadmin on any hosts where Kerberos client is installed. The difference of remotely running kadmin from locally running kadmin.local is that kadmin will ask you for password of principal which you use to be authenticated by Kerberos. 

Now, I can log into both host01 and host02 with user01 or user02 using their password in Kerberos database. The login can only be authenticated by Kerberos because the accounts are locked locally on servers. If you enjoy the testing, you can unlock user01 on host01 with a password which are different from the password of principal user01 in Kerberos database. Then you can log into host01 as user01 with either local password or Kerberos password, both passwords work because the sshd service option KerberosOrLocalPasswd is enabled. If the option is disabled, only Kerberos password works.

3.4 Install and configure Kerberos client on Ubuntu 16.04 workstation

I did not use a RHEL 8 workstation, because I have a ready-to-go Ubuntu 16.04 machine in hand and do not want waste time to install another RHEL 8 machine. You can install and configure a RHEL 8 workstation following steps in part 2. Actually, it takes exactly same steps to configure Kerberos client on machines which same OS release are running on, no matter the machine is server or workstation.

Install Kerberos client on workstation wkstn01
[admin@wkstn01]$ sudo apt install krb5-user
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  krb5-config
The following NEW packages will be installed:
  krb5-config krb5-user
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/122 kB of archives.
After this operation, 471 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 cdrom://Ubuntu-Server 16.04.6 LTS _Xenial Xerus_ - Release amd64 (20190226) xenial/main amd64 krb5-user 1.13.2+dfsg-5ubuntu2.1 [98.7 kB]
Get:2 cdrom://Ubuntu-Server 16.04.6 LTS _Xenial Xerus_ - Release amd64 (20190226) xenial/main amd64 krb5-config all 2.3 [23.4 kB]
Preconfiguring packages ...
Selecting previously unselected package krb5-config.
(Reading database ... 59771 files and directories currently installed.)
Preparing to unpack .../krb5-config_2.3_all.deb ...
Unpacking krb5-config (2.3) ...
Selecting previously unselected package krb5-user.
Preparing to unpack .../krb5-user_1.13.2+dfsg-5ubuntu2.1_amd64.deb ...
Unpacking krb5-user (1.13.2+dfsg-5ubuntu2.1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up krb5-config (2.3) ...
Use of uninitialized value $admin in string eq at /var/lib/dpkg/info/krb5-config.config line 171, <STDIN> line 19.
Setting up krb5-user (1.13.2+dfsg-5ubuntu2.1) ...

During the installation, answer the question as following,

 Default Kerberos version 5 realm:  LAB.DBAPLUS.CA

 Kerberos servers for your realm:   xdc01.lab.dbaplus.ca

 Administrative server for your Kerberos realm: xdc01.lab.dbaplus.ca

Create principal for workstation
[admin@wkstn01]$ sudo kadmin -p dbaplus/admin
Authenticating as principal dbaplus/admin with password.
Password for dbaplus/admin@LAB.DBAPLUS.CA:
kadmin:
kadmin:  addprinc -randkey host/wkstn01.lab.dbaplus.ca
WARNING: no policy specified for host/wkstn01.lab.dbaplus.ca@LAB.DBAPLUS.CA; defaulting to no policy
Principal "host/wkstn01.lab.dbaplus.ca@LAB.DBAPLUS.CA" created.
kadmin:
kadmin:  ktadd host/wkstn01.lab.dbaplus.ca
Entry for principal host/wkstn01.lab.dbaplus.ca with kvno 4, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/wkstn01.lab.dbaplus.ca with kvno 4, encryption type aes128-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/wkstn01.lab.dbaplus.ca with kvno 4, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/wkstn01.lab.dbaplus.ca with kvno 4, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/wkstn01.lab.dbaplus.ca with kvno 4, encryption type camellia256-cts-cmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/wkstn01.lab.dbaplus.ca with kvno 4, encryption type camellia128-cts-cmac added to keytab FILE:/etc/krb5.keytab.
kadmin:  q
 
Create user01 & user02 on wkstn01
[admin@wkstn01]$ sudo useradd user01
[admin@wkstn01]$ sudo useradd user02
 
3.5 Test SSH authentication with Kerberos

Log onto workstation wkstn01 as user01 with password of principle user01 (user01@LAB.DBAPLUS.CA)
C:\>ssh user01@wkstn01.lab.dbaplus.ca
user01@wkstn01.lab.dbaplus.ca's password:
Welcome to Ubuntu 16.04.6 LTS (GNU/Linux 4.4.0-142-generic x86_64)

Last login: Mon Oct 19 20:12:12 2020 from 10.10.25.2
[user01@wkstn01]$

Log onto server host01 as usesr01 from wkstn01 without password
[user01@wkstn01]$ ssh host01

Last login: Mon Oct 19 20:18:43 2020 from 10.10.25.201
[user01@host01]$

Log onto server host02 as usesr01 from wkstn01 without password
[user01@wkstn01]$ ssh host02

Last login: Mon Oct 19 20:27:07 2020 from 10.10.25.201
[user01@host02]$

When connecting to wkstn01, the user (user01) and password is authenticated by SSH with Kerberos and KDC created a ticket for the user, the ticket is sent to and cached on wkstn01. When logging onto host01 or host02 as user01 from wkstn01, the cached ticket will be used for authentication. Therefore it does not ask you for the password anymore. That looks like single-sign-on (SSO).

3.6 Kerberos authentication delegation

After logging into host01 from wkstn01 without password, does it need password to log into host02 from host01?

[user01@wkstn01]$ ssh host01

Last login: Mon Oct 19 20:52:20 2020 from 10.10.25.201

[user01@host01]$ ssh host02
user01@host02's password:

Last login: Mon Oct 19 20:52:26 2020 from 10.10.25.101
[user01@host02]$

It asked for password again because user01 does not have a ticket on host01. To delegate the ticket, the -K option has to be used with ssh command
[user01@wkstn01]$ klist
Ticket cache: FILE:/tmp/krb5cc_1001_bruTHWVJDG
Default principal: user01@LAB.DBAPLUS.CA

Valid starting       Expires              Service principal
2020-10-19 21:04:23  2020-10-20 21:04:22  krbtgt/LAB.DBAPLUS.CA@LAB.DBAPLUS.CA           <<<<<<<<< Ticket is cached on wkstn01
[user01@wkstn01]$
[user01@wkstn01]$ ssh host01          <<<<<<<<< log onto host01 without delegation

Last login: Mon Oct 19 21:02:56 2020 from 10.10.25.201
[user01@host01]$
[user01@host01]$ klist
klist: Credentials cache 'KCM:1000' not found          <<<<<<<<< No ticket is cached on host01
[user01@host01]$
[user01@host01]$ ssh host02
user01@host02's password:                 <<<<<<<<< Asking for password while logging onto host02 from host 01,  press Ctrl + C to go back to command prompt

[user01@host01]$ exit                   <<<<<<<<< Exit from host01 and back to wkstn01
logout
Connection to host01 closed.
[user01@wkstn01]$
[user01@wkstn01]$ ssh -K host01           <<<<<<<<< Logging onto host01 from wkstn01 with -K option

Last login: Mon Oct 19 21:05:12 2020 from 10.10.25.201
[user01@host01]$
[user01@host01]$ klist
Ticket cache: KCM:1000:20502
Default principal: user01@LAB.DBAPLUS.CA

Valid starting       Expires              Service principal
2020-10-19 21:07:28  2020-10-20 21:04:22  krbtgt/LAB.DBAPLUS.CA@LAB.DBAPLUS.CA    <<<<<<<<< Ticket is delegated from wkstn01 to host01
[user01@host01]$
[user01@host01]$ ssh host02               <<<<<<<<< Not asking for password anymore

Last login: Mon Oct 19 21:01:28 2020 from 10.10.25.101
[user01@host02 ~]$

In order to delegate ticket from wkstn01 to host01 & host02 for all users without -K option, the file /etc/ssh/ssh_config has to be add following lines
Host host01 host02
   GSSAPIDelegateCredentials yes

If you only want delegate for current user, the file ~/.ssh/config is needed instead of file /etc/ssh/ssh_config.

3.7 Troubleshooting

You may get following error if you run Kerberos client on different OS workstation 
[root@host01]# kadmin dbaplus/admin
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
kadmin: KDC has no support for encryption type while initializing kadmin interface

The reason is that the Kerberos client version does not match KDC version, usually the client version is too old. For example, the client is running RHEL4 with build-in Kerberos client.

No comments: