Tuesday, October 20, 2020

Install and Configure Kerberos Authentication on Red Hat Enterprise Linux 8

The demonstration includes three installation/configuration parts,

  Part 1. Install and configure Kerberos Key Distribution Center (KDC) Server
  
     Server name: xdc01.lab.dbaplus.ca
     OS version:  Red Hat Enterprise Linux (RHEL) 8

  Part 2. Install and configure Kerberos client on application (SSH) server
  
     Server 1: host01.lab.dbaplus.ca
     Server 2: host02.lab.dbaplus.ca
     OS version:  Red Hat Enterprise Linux (RHEL) 8
  
  Part 3. Install and configure Kerberos client on workstation to test single-sign-on (SSO) for SSH authentication with Kerberos
  
     Workstation name: wkstn01.lab.dbaplus.ca
     OS version: Ubuntu 16.04

     I use machine running Ubuntu because I do not have fourth machine running RHEL 8. Technically, the OS type should not be the issue if the correct version of Kerberos software is installed.

     The test will be done as following,

     A. Run ssh to log into wkstn01 with Kerberos user name (principle) and password
     B. From wkstn01, run ssh to log into server host01 or host02 without user name and password to justify SSO.
     C. From wkstn01, run ssh to log into server host01 (or host02), then run ssh to log into host02 (or host01) to justify authentication delegation.

1. Install and configure Kerberos KDC Server on RHEL 8

1.1 Install the required packages for KDC

Following three packages are required for the KDC on RHEL 8 host,

  krb5-libs - contains the shared libraries needed by Kerberos 5. If you are using Kerberos, you need to install this package.
  krb5-server - contains the programs that must be installed on a Kerberos 5 key distribution center (KDC). 
  krb5-workstation - contains the basic Kerberos programs (kinit, klist, kdestroy, kpasswd).
  
Install these packages with yum,  
[root@xdc01]# yum install krb5-server krb5-libs krb5-workstation
Updating Subscription Management repositories.
Last metadata expiration check: 0:00:54 ago on Sun 13 Sep 2020 03:44:33 PM EDT.
Package krb5-libs-1.17-18.el8.x86_64 is already installed.
Dependencies resolved.
========================================================================================
 Package                    Architecture      Version           Repository         Size
========================================================================================
Installing:
 krb5-server                x86_64            1.17-18.el8       InstallMedia      1.1 M
 krb5-workstation           x86_64            1.17-18.el8       InstallMedia      940 k
Installing dependencies:
 libkadm5                   x86_64            1.17-18.el8       InstallMedia      185 k
 libverto-libevent          x86_64            0.3.0-5.el8       InstallMedia       16 k

Transaction Summary
========================================================================================
Install  4 Packages

Total size: 2.2 M
Installed size: 4.9 M
Is this ok [y/N]: y
Downloading Packages:
warning: /mnt/BaseOS/Packages/krb5-server-1.17-18.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
DVD for Red Hat Enterprise Linux 8.2 Server                                                        1.1 MB/s | 5.0 kB     00:00
Importing GPG key 0xFD431D51:
 Userid     : "Red Hat, Inc. (release key 2) <security@redhat.com>"
 Fingerprint: 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Key imported successfully
Importing GPG key 0xD4082792:
 Userid     : "Red Hat, Inc. (auxiliary key) <security@redhat.com>"
 Fingerprint: 6A6A A7C9 7C88 90AE C6AE BFE2 F76F 66C3 D408 2792
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                           1/1
  Installing       : libkadm5-1.17-18.el8.x86_64                                                                               1/4
  Installing       : libverto-libevent-0.3.0-5.el8.x86_64                                                                      2/4
  Installing       : krb5-server-1.17-18.el8.x86_64                                                                            3/4
  Running scriptlet: krb5-server-1.17-18.el8.x86_64                                                                            3/4
  Installing       : krb5-workstation-1.17-18.el8.x86_64                                                                       4/4
  Running scriptlet: krb5-workstation-1.17-18.el8.x86_64                                                                       4/4
  Verifying        : krb5-server-1.17-18.el8.x86_64                                                                            1/4
  Verifying        : krb5-workstation-1.17-18.el8.x86_64                                                                       2/4
  Verifying        : libkadm5-1.17-18.el8.x86_64                                                                               3/4
  Verifying        : libverto-libevent-0.3.0-5.el8.x86_64                                                                      4/4
Installed products updated.

Installed:
  krb5-server-1.17-18.el8.x86_64                krb5-workstation-1.17-18.el8.x86_64          libkadm5-1.17-18.el8.x86_64
  libverto-libevent-0.3.0-5.el8.x86_64

Complete!

Run rpm to confirm the packages are installed
[root@xdc01]# rpm -qa | grep krb5
krb5-workstation-1.17-18.el8.x86_64
krb5-server-1.17-18.el8.x86_64
krb5-libs-1.17-18.el8.x86_64

1.2 Configure KDC

Edit file /var/kerberos/krb5kdc/kdc.conf and have the following lines included in the file
[kdcdefaults]
    kdc_ports = 88
    kdc_tcp_ports = 88
    spake_preauth_kdc_challenge = edwards25519

[realms]
LAB.DBAPLUS.CA = {
     #master_key_type = aes256-cts
     acl_file = /var/kerberos/krb5kdc/kadm5.acl
     dict_file = /usr/share/dict/words
     admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
     supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal
}

Here, LAB.DBAPLUS.CA is the realm name and always uppercase. You can define your own realm name, make sure your realm name in uppercase.

Edit file /etc/krb5.conf to reflect the realm name and domain-to-realm mappings as following
[logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log

[libdefaults]
    dns_lookup_realm = false
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    rdns = false
    pkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crt
    spake_preauth_groups = edwards25519
    default_realm = LAB.DBAPLUS.CA
    default_ccache_name = KEYRING:persistent:%{uid}

[realms]
 LAB.DBAPLUS.CA = {
     kdc = xdc01.lab.dbaplus.ca
     admin_server = xdc01.lab.dbaplus.ca
 }

[domain_realm]
 .lab.dbaplus.ca = LAB.DBAPLUS.CA
 lab.dbaplus.ca = LAB.DBAPLUS.CA

Under section [realms], xdc01.lab.dbaplus.ca is host name of this server, it is the KDC and administration server of realm LAB.DBAPLUS.CA. Make sure realm LAB.DBAPLUS.CA is the realm configured in file /var/kerberos/krb5kdc/kdc.conf.

Under section [domain_realm], it defined domain-to-realm mappings. I choose LAB.DBAPLUS.CA as the realm for domain lab.dbaplus.ca. It is a good choice to use domain name as realm name (but in uppercase), but it is not MUST. No matter what you choose as realm name, the domain name has to be a real domain which you are going to map to the realm.

1.3 Create the database using the kdb5_util utility

Run kdb5_util to create the database that stores keys for the Kerberos realm as following
[root@xdc01]# kdb5_util create -s
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'LAB.DBAPLUS.CA',
master key name 'K/M@LAB.DBAPLUS.CA'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:

The -s argument creates a stash file in which the master server key is stored. If no stash file is present from which to read the key, the Kerberos server (krb5kdc) prompts the user for the master server password (which can be used to regenerate the key) every time it starts.

1.4 Configure the Kerberos database ACL

Edit file /var/kerberos/krb5kdc/kadm5.acl to replace following line 
*/admin@EXAMPLE.COM     *

with line
*/admin@LAB.DBAPLUS.CA     *

Note, use your own realm instead of LAB.DBAPLUS.CA when configuring the file on your KDC.

The kadm5.acl file is used by Kerberos admin service (kadmin) to determine which principals have administrative access to the Kerberos database and their level of access. In previous configuration, any principal in the LAB.DBAPLUS.CA realm with an admin instance has all administrative privileges. For example, principal joe/admin@LAB.DBAPLUS.CA are able to exert full administrative control over the realm's Kerberos database.

After kadmin has been started on the server, any user (principal) can access its services by running kadmin on any of the clients or servers in the realm. However, only users (principals) listed in the kadm5.acl file can modify the database in any way, except for changing their own passwords. Anyone can change his own principal password.

1.5 Create the first principal using kadmin.local

The command kadmin.local is designed to be used on the same host as the KDC and does not use Kerberos for authentication. Therefore, it can be used to create the first principal
[root@xdc01]# kadmin.local
Authenticating as principal root/admin@LAB.DBAPLUS.CA with password.
kadmin.local:  addprinc dbaplus/admin
WARNING: no policy specified for dbaplus/admin@LAB.DBAPLUS.CA; defaulting to no policy
Enter password for principal "dbaplus/admin@LAB.DBAPLUS.CA":
Re-enter password for principal "dbaplus/admin@LAB.DBAPLUS.CA":
Principal "dbaplus/admin@LAB.DBAPLUS.CA" created.
kadmin.local:  exit

The principal dbaplus is created with instance admin in realm LAB.DBAPLUS.CA (dbaplus/admin@LAB.DBAPLUS.CA). According to the configuration in kadm5.acl file, this first principal has full administrative control over the realm's Kerveros database.

Of course, you can use whatever name you like instead of dbaplus, it is totally up to you. Many people like to use admin (admin/admin@LAB.DBAPLUS.CA) as first principal, but it is not best practice for security policy.

Now, you can be authenticated by KDC as principal dbaplus when you access servers in realm LAB.DBAPLUS.CA.

1.6 Start Kerberos services

Start krb5kdc and kadmin services using command systemctl as following
[root@xdc01]# systemctl start krb5kdc.service
[root@xdc01]# systemctl start kadmin.service

Confirm krb5kdc and kadmin are running using command netstat
[root@xdc01]# netstat -plntu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:464             0.0.0.0:*               LISTEN      25200/kadmind
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      880/sshd
tcp        0      0 0.0.0.0:88              0.0.0.0:*               LISTEN      25142/krb5kdc
tcp        0      0 0.0.0.0:749             0.0.0.0:*               LISTEN      25200/kadmind
tcp6       0      0 :::464                  :::*                    LISTEN      25200/kadmind
tcp6       0      0 :::22                   :::*                    LISTEN      880/sshd
tcp6       0      0 :::88                   :::*                    LISTEN      25142/krb5kdc
tcp6       0      0 :::749                  :::*                    LISTEN      25200/kadmind
udp        0      0 127.0.0.1:323           0.0.0.0:*                           832/chronyd
udp        0      0 0.0.0.0:464             0.0.0.0:*                           25200/kadmind
udp        0      0 0.0.0.0:88              0.0.0.0:*                           25142/krb5kdc
udp6       0      0 ::1:323                 :::*                                832/chronyd
udp6       0      0 :::464                  :::*                                25200/kadmind
udp6       0      0 :::88                   :::*                                25142/krb5kdc

It shows that process krb5kdc (pid 25142, daemon process of krb5kdc service) is listening on port 88 for tcp/udp and process kadmind (pid 25200, daemon process of kadmin service) is listening on port 464 for tcp/udp and port 749 for tcp. it means both services are running. The ports used by krb5kdc and kadmind are commonly defined as network service as following

   kerberos  tcp/88 & udp/88    (Kerberos network authentication protocol server - KDC)
   kadmin    tcp/749            (Kerberos Administration Protocol)
   kpasswd   tcp/464 & udp/464  (Kerberos password server)

Note: kerberos, kadmin and kpasswd are network services and different from krb5kdc and kadmind which are system services, though they are all called service. Network service is helpful when you configure firewall.
   
The status of krb5kdc and kadmin can also be checked using command systemctl
[root@xdc01]# systemctl status krb5kdc.service
krb5kdc.service - Kerberos 5 KDC
   Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-09-13 20:27:52 EDT; 19min ago
  Process: 25141 ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid $KRB5KDC_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 25142 (krb5kdc)
    Tasks: 1 (limit: 5016)
   Memory: 3.1M
   CGroup: /system.slice/krb5kdc.service
           └─25142 /usr/sbin/krb5kdc -P /var/run/krb5kdc.pid
[root@xdc01]#
[root@xdc01]# systemctl status kadmin.service
kadmin.service - Kerberos 5 Password-changing and Administration
   Loaded: loaded (/usr/lib/systemd/system/kadmin.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-09-13 20:56:11 EDT; 4min 56s ago
  Process: 25199 ExecStart=/usr/sbin/kadmind -P /var/run/kadmind.pid $KADMIND_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 25200 (kadmind)
    Tasks: 1 (limit: 5016)
   Memory: 11.7M
   CGroup: /system.slice/kadmin.service
           └─25200 /usr/sbin/kadmind -P /var/run/kadmind.pid

1.7 Verify that the KDC is working

Run kinit to obtain a ticket from KDC
[root@xdc01]# kinit dbaplus/admin
Password for dbaplus/admin@LAB.DBAPLUS.CA:

The command kinit may get following error,
[root@xdc01]# kinit dbaplus/admin
kinit: Cannot contact any KDC for realm 'LAB.DBAPLUS.CA' while getting initial credentials

If the error happened, check following items

   *  Section [realms] of file /etc/krb5.conf, make sure value of "kdc" entry is same as KDC server name

   *  File /etc/hosts, make sure the KDC server name is in the file unless the server name is resolved correctly by DNS 
   
   *  Service krb5kdc status, make krb5kdc is running
   
Run klist to confirm the ticket is cached by kinit
[root@xdc01]# klist
Ticket cache: KCM:0
Default principal: dbaplus/admin@LAB.DBAPLUS.CA

Valid starting       Expires              Service principal
09/15/2020 14:43:49  09/16/2020 14:43:37  krbtgt/LAB.DBAPLUS.CA@LAB.DBAPLUS.CA
        renew until 09/15/2020 14:43:49

If kinit and klist can be executed successfully, it proves KDC is working well.

1.8 Have firewall work with Kerberos authentication if the firewall is enabled

Check if firewall is enabled and started using command systemctl status firewalld.service.

The firewall is enabled and started if the command shows
[root@xdc01]# systemctl status firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2020-09-14 18:45:35 EDT; 1h 39min ago
     Docs: man:firewalld(1)
 Main PID: 25960 (firewalld)
    Tasks: 2 (limit: 5016)
   Memory: 23.7M
   CGroup: /system.slice/firewalld.service
           └─25960 /usr/libexec/platform-python -s /usr/sbin/firewalld --nofork --nopid

The firewall is disabled and stopped if the command shows
[root@xdc01]# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

The firewall is enabled and stopped if the command shows
[root@xdc01]# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

If the firewall is enabled, we have to configure it to work with Kerberos. Before configuring filewall, start the firewall using command systemctl start firewalld.service if the firewall is not running
[root@xdc01]# systemctl start firewalld.service

[root@xdc01]# systemctl status firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2020-09-14 20:43:27 EDT; 9s ago
     Docs: man:firewalld(1)
 Main PID: 26219 (firewalld)
    Tasks: 2 (limit: 5016)
   Memory: 22.6M
   CGroup: /system.slice/firewalld.service
           └─26219 /usr/libexec/platform-python -s /usr/sbin/firewalld --nofork --nopid  

Note: If firewall is disabled or not installed, there is no need to configure firewall. If the firewall is enabled, the firewalld service will be automatically started after the server is restarted even if it is currently stopped. Therefore, make sure the firewall is configured with Kerberos no matter the firewalld service is running or not.

List firewall rules of active zone
[root@xdc01]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s3 enp0s8
  sources:
  services: cockpit dhcpv6-client ssh
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

The active zone is public. Kerberos related network services are not in the services list. Add the Kerberos network services to the active zone
[root@xdc01]# firewall-cmd --add-service=kerberos --permanent
success
[root@xdc01]# firewall-cmd --add-service=kadmin --permanent
success
[root@xdc01]# firewall-cmd --add-service=kpasswd --permanent
success

The Kerveros network services (kerberos, kadmin and kpasswd) are added to firewall configuration permanently, but they are not activated in current firewall session. Restarting filewalld service to make new rule active using command systemctl restart firewalld.service
[root@xdc01]# firewall-cmd --list-services
cockpit dhcpv6-client ssh
[root@xdc01]#
[root@xdc01]# systemctl restart firewalld.service
[root@xdc01]#
[root@xdc01]# firewall-cmd --list-services
cockpit dhcpv6-client kadmin kerberos kpasswd ssh
[root@xdc01]#

2. Install and configure Kerberos client on application (SSH) server

In order to have a whole picture of using Kerberos in single sing on (SSO) and authentication delegation, I am going to configure Kerberos client on two SSH servers. 

2.1 Test connectivity between application (SSH) servers and KDC

Before starting Kerberos client installation on application servers, make sure KDC Kerberos services are accessible from application servers.

Test that KDC host is reachable using ping on application servers host01 and host02
[root@host01]# ping -c 1 xdc01.lab.dbaplus.ca
PING xdc01.lab.dbaplus.ca (10.10.25.254) 56(84) bytes of data.
64 bytes from xdc01.lab.dbaplus.ca (10.10.25.254): icmp_seq=1 ttl=64 time=1.12 ms

--- xdc01.lab.dbaplus.ca ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.115/1.115/1.115/0.000 ms

[root@host02]# ping -c 1 xdc01.lab.dbaplus.ca
PING xdc01.lab.dbaplus.ca (10.10.25.254) 56(84) bytes of data.
64 bytes from xdc01.lab.dbaplus.ca (10.10.25.254): icmp_seq=1 ttl=64 time=2.49 ms

--- xdc01.lab.dbaplus.ca ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.490/2.490/2.490/0.000 ms

Test whether Kerberos services is accessible using telnet on both servers
[root@host01]# telnet xdc01.lab.dbaplus.ca 88
Trying 10.10.25.254...
Connected to xdc01.lab.dbaplus.ca.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

The telnet session can be closed by pressing Ctrl+] and entering command quit. By default, RHEL 8 does not install telnet command, you can use nc instead
[root@host01]# nc -vz xdc01.lab.dbaplus.ca 88
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 10.10.25.254:88.
Ncat: 0 bytes sent, 0 bytes received in 0.03 seconds.

If Kerberos service krb5kdc is not running or the firewall on KDC is not configured properly, telnet/nc will return as following,
[root@host01]# nc -vz xdc01.lab.dbaplus.ca 88
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: No route to host.

[root@host01]# telnet xdc01.lab.dbaplus.ca 88
Trying 10.10.25.254...
telnet: connect to address 10.10.25.254: No route to host

2.2 Install and configure Kerberos client on RHEL 8 application (SSH) servers

Install the krb5-libs and krb5-workstation packages on both servers
[root@host01]# yum install krb5-workstation krb5-libs
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
DVD for Red Hat Enterprise Linux 8.2 Server             44 MB/s | 2.3 MB     00:00
Last metadata expiration check: 0:00:01 ago on Tue 15 Sep 2020 07:23:58 PM EDT.
Package krb5-libs-1.17-18.el8.x86_64 is already installed.
Dependencies resolved.
=================================================================================
 Package                    Architecture     Version        Repository      Size
=================================================================================
Installing:
 krb5-workstation           x86_64           1.17-18.el8    InstallMedia   940 k
Installing dependencies:
 libkadm5                   x86_64           1.17-18.el8    InstallMedia   185 k

Transaction Summary
=================================================================================
Install  2 Packages

Total size: 1.1 M
Installed size: 3.5 M
Is this ok [y/N]: y
Downloading Packages:
warning: /mnt/BaseOS/Packages/krb5-workstation-1.17-18.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
DVD for Red Hat Enterprise Linux 8.2 Server           1.0 MB/s | 5.0 kB     00:00
Importing GPG key 0xFD431D51:
 Userid     : "Red Hat, Inc. (release key 2) <security@redhat.com>"
 Fingerprint: 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Key imported successfully
Importing GPG key 0xD4082792:
 Userid     : "Red Hat, Inc. (auxiliary key) <security@redhat.com>"
 Fingerprint: 6A6A A7C9 7C88 90AE C6AE BFE2 F76F 66C3 D408 2792
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                          1/1
  Installing       : libkadm5-1.17-18.el8.x86_64                              1/2
  Installing       : krb5-workstation-1.17-18.el8.x86_64                      2/2
  Running scriptlet: krb5-workstation-1.17-18.el8.x86_64                      2/2
  Verifying        : krb5-workstation-1.17-18.el8.x86_64                      1/2
  Verifying        : libkadm5-1.17-18.el8.x86_64                              2/2
Installed products updated.

Installed:
  krb5-workstation-1.17-18.el8.x86_64                 libkadm5-1.17-18.el8.x86_64

Complete!

Copy /etc/krb5.conf from KDC to both servers
[root@host01]# scp xdc01:/etc/krb5.conf /etc/krb5.conf
root@xdc01's password:
krb5.conf                                           100%  823   311.0KB/s   00:00

2.3 Add host principal for each server on the KDC

Before a server can use Kerberos to authenticate users who request to access the service (e.g. SSH) running on the server, it must have its own host principal in the Kerberos database.

On KDC, run kadmin.local to add host principals for both servers as following
[root@xdc01]# kadmin.local -q "addprinc -randkey host/host01.lab.dbaplus.ca"
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
WARNING: no policy specified for host/host01.lab.dbaplus.ca@LAB.DBAPLUS.CA; defaulting to no policy
Principal "host/host01.lab.dbaplus.ca@LAB.DBAPLUS.CA" created.

[root@xdc01]# kadmin.local -q "addprinc -randkey host/host02.lab.dbaplus.ca"
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
WARNING: no policy specified for host/host02.lab.dbaplus.ca@LAB.DBAPLUS.CA; defaulting to no policy
Principal "host/host02.lab.dbaplus.ca@LAB.DBAPLUS.CA" created.

2.4 Download host principal to its respective server

Run kinit on both servers to obtain initial ticket from KDC
[root@host01]# kinit dbaplus/admin
Password for dbaplus/admin@LAB.DBAPLUS.CA:

[root@host02]# kinit dbaplus/admin
Password for dbaplus/admin@LAB.DBAPLUS.CA:

After successfully getting initial ticket, when you run Kerberos client commands (e.g. kadmin, kpasswd) as root on the servers, the commands will, by default, use principal dbaplus/admin as authentication user to access Kerberos database. Otherwise, the commands will user root/admin which does not exist in Kerberos database. You may be wondering if we can create principal root/admin. Technically, we can create principal root/admin, but it is not recommended.

Run kadmin command ktadd on server host01 to download principle "host/host01.lab.dbaplus.ca" which is created at step 2.3
[root@host01]# kadmin -q "ktadd host/host01.lab.dbaplus.ca"
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
Password for dbaplus/admin@LAB.DBAPLUS.CA:
Entry for principal host/host01.lab.dbaplus.ca with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host01.lab.dbaplus.ca with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host01.lab.dbaplus.ca with kvno 2, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host01.lab.dbaplus.ca with kvno 2, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host01.lab.dbaplus.ca with kvno 2, encryption type camellia256-cts-cmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host01.lab.dbaplus.ca with kvno 2, encryption type camellia128-cts-cmac added to keytab FILE:/etc/krb5.keytab.

Run kadmin command ktadd on server host02 to download principle "host/host02.lab.dbaplus.ca" which is created at step 2.3
[root@host02]# kadmin -q "ktadd host/host02.lab.dbaplus.ca"
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
Password for dbaplus/admin@LAB.DBAPLUS.CA:
Entry for principal host/host02.lab.dbaplus.ca with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host02.lab.dbaplus.ca with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host02.lab.dbaplus.ca with kvno 3, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host02.lab.dbaplus.ca with kvno 3, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host02.lab.dbaplus.ca with kvno 3, encryption type camellia256-cts-cmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/host02.lab.dbaplus.ca with kvno 3, encryption type camellia128-cts-cmac added to keytab FILE:/etc/krb5.keytab.

You may like to run kamin in interactive mode instead of 'kadmin -q' as following,
[root@host02]# kadmin
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
Password for dbaplus/admin@LAB.DBAPLUS.CA:
kadmin:  ktadd host/host02.lab.dbaplus.ca

You may get following errors when run kadmin command ktadd
[root@host01]# kadmin
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
Password for dbaplus/admin@LAB.DBAPLUS.CA:
kadmin:  ktadd -k /etc/krb5.keytab host/host01.lab.dbaplus.ca
kadmin: Operation requires ``change-password'' privilege while changing host/host01.lab.dbaplus.ca's key

kadmin:  list_principals
get_principals: Operation requires ``list'' privilege while retrieving list.

The cause is the configuration error in /var/kerberos/krb5kdc/kadm5.acl, make sure the file is configured as in step 1.4.

So far, Kerberos KDC and client are configured successfully. But what can we do with the configuration? The answer is that many applications/serivces are able to adopt Kerberos as user authentication solution, such as SSH, Telnet, Samba, NFS, SQL Server and Oracle database, etc. Today, I am going to configure SSH with Kerberos authentication to enjoy Kerberos in real world. 

3. Test single-sign-on (SSO) for SSH authentication with Kerberos

3.1 Configure SSH to use Kerberos authentication on both servers (host01 & host02)

Edit file /etc/ssh/sshd_config and add following lines,
    KerberosAuthentication yes
    KerberosTicketCleanup yes
    KerberosOrLocalPasswd yes

Here, 

  * KerberosAuthentication  Specifies whether the password provided by the user for PasswordAuthentication will be validated through the Kerberos KDC.  To use this option, the server needs a Kerberos keytab which allows the verification of the KDC's identity. 
  
  * KerberosTicketCleanup   Specifies whether to automatically destroy the user's ticket cache file on logout. 
  
  * KerberosOrLocalPasswd   If password authentication through Kerberos fails then the password will be validated via any additional local mechanism such as  /etc/passwd.

Restart sshd
[root@host01]# systemctl restart sshd.service

[root@host02]# systemctl restart sshd.service

3.2 Create users for testing on both SSH servers (host01 & host02)

Run useradd on both servers to create testing user user01, user02
[root@host01]# useradd user01
[root@host01]# useradd user02
[root@host01]# passwd --status user01
user01 LK 2020-09-15 0 99999 7 -1 (Password locked.)
[root@host01]# passwd --status user02
user02 LK 2020-09-16 0 99999 7 -1 (Password locked.)

[root@host02]# useradd user01
[root@host02]# useradd user02
[root@host02]# passwd --status user01
user01 LK 2020-09-15 0 99999 7 -1 (Password locked.)
[root@host02]# passwd --status user02
user02 LK 2020-09-16 0 99999 7 -1 (Password locked.)
 
Users user01 & user02 are created on both servers (host01 & host02), and the users are locked by default. That means user01 & user02 cannot log into host01 & host02 with local password authentication.

3.3 Create principals user01 & user02 which will be 'mapped' to OS users with same names respectively

Create principals user01 & user02 on KDC
[root@xdc01]# kadmin.local -q "addprinc user01"
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.1
WARNING: no policy specified for user01@LAB.DBAPLUS.CA; defaulting to no policy
Enter password for principal "user01@LAB.DBAPLUS.CA":
Re-enter password for principal "user01@LAB.DBAPLUS.CA":
Principal "user01@LAB.DBAPLUS.CA" created.

[root@xdc01]# kadmin.local -q "addprinc user02"
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
WARNING: no policy specified for user02@LAB.DBAPLUS.CA; defaulting to no policy
Enter password for principal "user02@LAB.DBAPLUS.CA":
Re-enter password for principal "user02@LAB.DBAPLUS.CA":
Principal "user02@LAB.DBAPLUS.CA" created.

Enter a password when you are being asked. I entered the password same as principal (user name) to make test simple.

Actually, creating principal can be done with kadmin on any hosts where Kerberos client is installed. The difference of remotely running kadmin from locally running kadmin.local is that kadmin will ask you for password of principal which you use to be authenticated by Kerberos. 

Now, I can log into both host01 and host02 with user01 or user02 using their password in Kerberos database. The login can only be authenticated by Kerberos because the accounts are locked locally on servers. If you enjoy the testing, you can unlock user01 on host01 with a password which are different from the password of principal user01 in Kerberos database. Then you can log into host01 as user01 with either local password or Kerberos password, both passwords work because the sshd service option KerberosOrLocalPasswd is enabled. If the option is disabled, only Kerberos password works.

3.4 Install and configure Kerberos client on Ubuntu 16.04 workstation

I did not use a RHEL 8 workstation, because I have a ready-to-go Ubuntu 16.04 machine in hand and do not want waste time to install another RHEL 8 machine. You can install and configure a RHEL 8 workstation following steps in part 2. Actually, it takes exactly same steps to configure Kerberos client on machines which same OS release are running on, no matter the machine is server or workstation.

Install Kerberos client on workstation wkstn01
[admin@wkstn01]$ sudo apt install krb5-user
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  krb5-config
The following NEW packages will be installed:
  krb5-config krb5-user
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/122 kB of archives.
After this operation, 471 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 cdrom://Ubuntu-Server 16.04.6 LTS _Xenial Xerus_ - Release amd64 (20190226) xenial/main amd64 krb5-user 1.13.2+dfsg-5ubuntu2.1 [98.7 kB]
Get:2 cdrom://Ubuntu-Server 16.04.6 LTS _Xenial Xerus_ - Release amd64 (20190226) xenial/main amd64 krb5-config all 2.3 [23.4 kB]
Preconfiguring packages ...
Selecting previously unselected package krb5-config.
(Reading database ... 59771 files and directories currently installed.)
Preparing to unpack .../krb5-config_2.3_all.deb ...
Unpacking krb5-config (2.3) ...
Selecting previously unselected package krb5-user.
Preparing to unpack .../krb5-user_1.13.2+dfsg-5ubuntu2.1_amd64.deb ...
Unpacking krb5-user (1.13.2+dfsg-5ubuntu2.1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up krb5-config (2.3) ...
Use of uninitialized value $admin in string eq at /var/lib/dpkg/info/krb5-config.config line 171, <STDIN> line 19.
Setting up krb5-user (1.13.2+dfsg-5ubuntu2.1) ...

During the installation, answer the question as following,

 Default Kerberos version 5 realm:  LAB.DBAPLUS.CA

 Kerberos servers for your realm:   xdc01.lab.dbaplus.ca

 Administrative server for your Kerberos realm: xdc01.lab.dbaplus.ca

Create principal for workstation
[admin@wkstn01]$ sudo kadmin -p dbaplus/admin
Authenticating as principal dbaplus/admin with password.
Password for dbaplus/admin@LAB.DBAPLUS.CA:
kadmin:
kadmin:  addprinc -randkey host/wkstn01.lab.dbaplus.ca
WARNING: no policy specified for host/wkstn01.lab.dbaplus.ca@LAB.DBAPLUS.CA; defaulting to no policy
Principal "host/wkstn01.lab.dbaplus.ca@LAB.DBAPLUS.CA" created.
kadmin:
kadmin:  ktadd host/wkstn01.lab.dbaplus.ca
Entry for principal host/wkstn01.lab.dbaplus.ca with kvno 4, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/wkstn01.lab.dbaplus.ca with kvno 4, encryption type aes128-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/wkstn01.lab.dbaplus.ca with kvno 4, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/wkstn01.lab.dbaplus.ca with kvno 4, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/wkstn01.lab.dbaplus.ca with kvno 4, encryption type camellia256-cts-cmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/wkstn01.lab.dbaplus.ca with kvno 4, encryption type camellia128-cts-cmac added to keytab FILE:/etc/krb5.keytab.
kadmin:  q
 
Create user01 & user02 on wkstn01
[admin@wkstn01]$ sudo useradd user01
[admin@wkstn01]$ sudo useradd user02
 
3.5 Test SSH authentication with Kerberos

Log onto workstation wkstn01 as user01 with password of principle user01 (user01@LAB.DBAPLUS.CA)
C:\>ssh user01@wkstn01.lab.dbaplus.ca
user01@wkstn01.lab.dbaplus.ca's password:
Welcome to Ubuntu 16.04.6 LTS (GNU/Linux 4.4.0-142-generic x86_64)

Last login: Mon Oct 19 20:12:12 2020 from 10.10.25.2
[user01@wkstn01]$

Log onto server host01 as usesr01 from wkstn01 without password
[user01@wkstn01]$ ssh host01

Last login: Mon Oct 19 20:18:43 2020 from 10.10.25.201
[user01@host01]$

Log onto server host02 as usesr01 from wkstn01 without password
[user01@wkstn01]$ ssh host02

Last login: Mon Oct 19 20:27:07 2020 from 10.10.25.201
[user01@host02]$

When connecting to wkstn01, the user (user01) and password is authenticated by SSH with Kerberos and KDC created a ticket for the user, the ticket is sent to and cached on wkstn01. When logging onto host01 or host02 as user01 from wkstn01, the cached ticket will be used for authentication. Therefore it does not ask you for the password anymore. That looks like single-sign-on (SSO).

3.6 Kerberos authentication delegation

After logging into host01 from wkstn01 without password, does it need password to log into host02 from host01?

[user01@wkstn01]$ ssh host01

Last login: Mon Oct 19 20:52:20 2020 from 10.10.25.201

[user01@host01]$ ssh host02
user01@host02's password:

Last login: Mon Oct 19 20:52:26 2020 from 10.10.25.101
[user01@host02]$

It asked for password again because user01 does not have a ticket on host01. To delegate the ticket, the -K option has to be used with ssh command
[user01@wkstn01]$ klist
Ticket cache: FILE:/tmp/krb5cc_1001_bruTHWVJDG
Default principal: user01@LAB.DBAPLUS.CA

Valid starting       Expires              Service principal
2020-10-19 21:04:23  2020-10-20 21:04:22  krbtgt/LAB.DBAPLUS.CA@LAB.DBAPLUS.CA           <<<<<<<<< Ticket is cached on wkstn01
[user01@wkstn01]$
[user01@wkstn01]$ ssh host01          <<<<<<<<< log onto host01 without delegation

Last login: Mon Oct 19 21:02:56 2020 from 10.10.25.201
[user01@host01]$
[user01@host01]$ klist
klist: Credentials cache 'KCM:1000' not found          <<<<<<<<< No ticket is cached on host01
[user01@host01]$
[user01@host01]$ ssh host02
user01@host02's password:                 <<<<<<<<< Asking for password while logging onto host02 from host 01,  press Ctrl + C to go back to command prompt

[user01@host01]$ exit                   <<<<<<<<< Exit from host01 and back to wkstn01
logout
Connection to host01 closed.
[user01@wkstn01]$
[user01@wkstn01]$ ssh -K host01           <<<<<<<<< Logging onto host01 from wkstn01 with -K option

Last login: Mon Oct 19 21:05:12 2020 from 10.10.25.201
[user01@host01]$
[user01@host01]$ klist
Ticket cache: KCM:1000:20502
Default principal: user01@LAB.DBAPLUS.CA

Valid starting       Expires              Service principal
2020-10-19 21:07:28  2020-10-20 21:04:22  krbtgt/LAB.DBAPLUS.CA@LAB.DBAPLUS.CA    <<<<<<<<< Ticket is delegated from wkstn01 to host01
[user01@host01]$
[user01@host01]$ ssh host02               <<<<<<<<< Not asking for password anymore

Last login: Mon Oct 19 21:01:28 2020 from 10.10.25.101
[user01@host02 ~]$

In order to delegate ticket from wkstn01 to host01 & host02 for all users without -K option, the file /etc/ssh/ssh_config has to be add following lines
Host host01 host02
   GSSAPIDelegateCredentials yes

If you only want delegate for current user, the file ~/.ssh/config is needed instead of file /etc/ssh/ssh_config.

3.7 Troubleshooting

You may get following error if you run Kerberos client on different OS workstation 
[root@host01]# kadmin dbaplus/admin
Authenticating as principal dbaplus/admin@LAB.DBAPLUS.CA with password.
kadmin: KDC has no support for encryption type while initializing kadmin interface

The reason is that the Kerberos client version does not match KDC version, usually the client version is too old. For example, the client is running RHEL4 with build-in Kerberos client.

Monday, September 28, 2020

GI 19c opatchauto out-of-place patching failed with file permission issues

Environment,

   Grid Infrastructure Home version : 19.7.0.0.200414
   OPatch version: 12.2.0.1.21
   Patch version: P31305339 - GI Release Update 19.8.0.0.200714

On fist node of the cluster, when runng opatchauto with option -outofplace or -prepare-clone, got errors
[root@rac01]# /u01/app/19.7.0/grid _1/OPatch/opatchauto apply -phBaseDir /u01/stage/grid -outofplace -logLevel FINEST
     ... ...
Creating clone for oracle home /u01/app/19.7.0/grid_1.
Failed to perform clone operation for oracle home /u01/app/19.7.0/grid_1. 

Execution of [CloneOracleHomeAction] patch action failed, check log for more details. Failures:
Patch Target : rac01->/u01/app/19.7.0/grid_1 Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.7.0/grid_1, host: rac01.
Command failed:  /u01/app/19.8.0/grid_1/perl/bin/perl /u01/app/19.8.0/grid_1/clone/bin/clone.pl ORACLE_BASE=/u01/app/grid ORACLE_HOME=/u01/app/19.8.0/grid_1 INVENTORY_LOCATION=/u01/app/oraInventory -defaultHomeName 'CLUSTER_NODES={rac01,rac02}' "LOCAL_NODE=rac01" CRS=false SHOW_ROOTSH_CONFIRMATION=false oracle.install.skipRootPre=true -silent
Command failure output: 
[INFO] [INS-32183] Use of clone.pl is deprecated in this release. Clone operation is equivalent to performing a Software Only installation from the image. 
You must use /u01/app/19.8.0/grid_1/gridSetup.sh script available to perform the Software Only install. For more details on image based installation, refer to help documentation.

Starting Oracle Universal Installer...

[FATAL] [INS-32012] Unable to create directory: /u01/app/19.8.0/grid_1, on this server.
   CAUSE: Either proper permissions were not granted to create the directory or there was no space left in the volume.
   ACTION: Check your permission on the selected directory or choose another directory.
[WARNING] [INS-32029] The Installer has detected that the Oracle Base location is not empty.
   ACTION: Oracle recommends that the Oracle Base location is empty. 

After fixing the cause of failure Run opatchauto resume

]

The process failed with error "Unable to create directory: /u01/app/19.8.0/grid_1", the directory is cloned (new) home and the process is trying to apply RU 19.8.0.0.200714 onto the home.

Check the directory permission
[root@rac01]# ls -ld /u01/app/19.8.0/grid_1       
drwxr-xr-x   94 root     oinstall       4096 Sep 20 18:27 /u01/app/19.8.0/grid_1

Chang the directory owner from root to Grid Home owner grid
[root@rac01]$ chown grid /u01/app/19.8.0/grid_1       

Resume the patch process
[root@rac01]# /u01/app/19.7.0/grid_1/OPatch/opatchauto resume

Different errors occur
Patch Target : rac01->/u01/app/19.7.0/grid_1 Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.7.0/grid_1, host: rac01.
Command failed:  /u01/app/19.8.0/grid_1/perl/bin/perl /u01/app/19.8.0/grid_1/clone/bin/clone.pl ORACLE_BASE=/u01/app/grid ORACLE_HOME=/u01/app/19.8.0/grid_1 INVENTORY_LOCATION=/u01/app/oraInventory -defaultHomeName 'CLUSTER_NODES={rac01,rac02}' "LOCAL_NODE=rac01" CRS=false SHOW_ROOTSH_CONFIRMATION=false oracle.install.skipRootPre=true -silent
Command failure output: 
[INFO] [INS-32183] Use of clone.pl is deprecated in this release. Clone operation is equivalent to performing a Software Only installation from the image. 
You must use /u01/app/19.8.0/grid_1/gridSetup.sh script available to perform the Software Only install. For more details on image based installation, refer to help documentation.

Starting Oracle Universal Installer...

[WARNING] [INS-32029] The Installer has detected that the Oracle Base location is not empty.
   ACTION: Oracle recommends that the Oracle Base location is empty.
You can find the log of this install session at:
 /u01/app/oraInventory/logs/cloneActions2020-09-20_06-38-13PM.log
..................................................   5% Done.
..................................................   10% Done.
..................................................   15% Done.
..................................................   20% Done.
..................................................   25% Done.
..................................................   30% Done.
..................................................   35% Done.
..................................................   40% Done.
..................................................   45% Done.
..................................................   50% Done.
..................................................   55% Done.
..................................................   60% Done.
..................................................   65% Done.
..................................................   70% Done.
..................................................   75% Done.
..................................................   80% Done.
..................................................   85% Done.
..........[FATAL] Failed to restore the saved templates to the Oracle home being cloned. Aborting the clone operation

After fixing the cause of failure Run opatchauto resume

]

The root cause is that the opatchauto did not successfully unlock the new cloned home before applying patches on it. Lots of files/directories under new home are still owned by root
[root@rac01]# find /u01/app/19.8.0/grid_1 -user root
/u01/app/19.8.0/grid_1/acfs
/u01/app/19.8.0/grid_1/acfs/tunables
/u01/app/19.8.0/grid_1/auth
/u01/app/19.8.0/grid_1/auth/crs
/u01/app/19.8.0/grid_1/auth/crs/r10-sohei
   ... ...
/u01/app/19.8.0/grid_1/bin/cloudcli
/u01/app/19.8.0/grid_1/bin/clscfg
/u01/app/19.8.0/grid_1/bin/clscfg.bin
/u01/app/19.8.0/grid_1/bin/clsdiaglsnr
   ... ...
/u01/app/19.8.0/grid_1/suptools/tfa/release/tfa_home/tfa.jks
/u01/app/19.8.0/grid_1/suptools/tfa/release/tfa_home/tfa_directories.txt
/u01/app/19.8.0/grid_1/suptools/tfa/release/tfa_home/tfa_setup.txt
/u01/app/19.8.0/grid_1/xag/mesg/xagus.le

Change the owner of these file/directories to grid
[root@rac01]# find /u01/app/19.8.0/grid_1 -user root -exec chown grid {} \;

Continue opatchauto with resume option,
[root@rac01]# /u01/app/19.7.0/grid _1/OPatch/opatchauto resume

The home will be out-of-place patched, and new home is /u01/app/19.8.0/grid_1.

Sunday, August 2, 2020

HTTP 404 when enabling Friendly URLs or referencing Static Application Files using substitution string #APP_IMAGES#

As it is known, Oracle is trying to deprecate two web listeners Oracle HTTP Server (mod_plsql) and Embedded PL/SQL Gateway, and Oracle REST Data Services (ORDS) will be the only supported web listener for APEX.

Saturday, July 18, 2020

Oracle database 19c "srvctl status service" fails with Java.lang.NullPointerException

The following errors occur while running "srvctl status service -db <db_name>" command from 19.7 database home on AIX and Linux x86_64,
$ srvctl status service -db orcl
Exception in thread "main" java.lang.NullPointerException
        at oracle.ops.opsctl.StatusAction.internalExecuteService(StatusAction.java:2077)
        at oracle.ops.opsctl.StatusAction.executeSIHAService(StatusAction.java:1692)
        at oracle.ops.opsctl.Action.executeSIHA(Action.java:445)
        at oracle.ops.opsctl.OPSCTLDriver.execute(OPSCTLDriver.java:537)
        at oracle.ops.opsctl.OPSCTLDriver.main(OPSCTLDriver.java:252)

Thursday, July 16, 2020

Configure SSL on WebLogic 12c with CA Issued Certificate

Weblogic 12c has demo identity keystore shipped with installation, it makes it very easy to implement SSL communication for WebLogic managed servers in non-production environment. However,  if you are deploying your application in production, especially public accessible, a certificate issued by Certificate Authority (CA) is absolutely necessary. To request and install CA issued certificate on WebLogic 12c, follow these steps,

Sunday, July 12, 2020

Install APEX with deploying ORDS in WebLogic as Web listener

The installation is completed on Oracle Linux 7.5, and following components/ applications are installed,

    JDK 1.8.0_241
    Weblogic 12.2.1.4.0
    Oracle Application Express (APEX) 19.2
    Oracle REST Data Services (ORDS) 19.4

Saturday, June 20, 2020

Agent status shows 'Diagnose for Agent Blocked (Bounce Counter Mismatch)' in OEM 13c

Agent 13.2 home was restored from a backup after failed OS patching maintenance. On the Enterprise Manager 13.2 console, the agent status shows 'Diagnose for Agent Blocked (Bounce Counter Mismatch)', and targets monitored by the agent show 'Diagnose for Status Pending (Post Blackout)'.

Wednesday, May 27, 2020

Database 12.2 crashed with ORA-04031 ("shared pool","unknown object","init_heap_kfsg","ASM extent pointer array")

Database 12.2 instance crashed with following messages in alert log,
2020-05-20T22:28:52.724801-04:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_asmb_15204374.trc  (incident=320299):
ORA-04031: unable to allocate 4120 bytes of shared memory ("shared pool","unknown object","init_heap_kfsg","ASM extent pointer array")
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_320299/orcl_asmb_15204374_i320299.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2020-05-20T22:29:30.066864-04:00
NOTE: ASMB0 terminating
2020-05-20T22:29:30.067840-04:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_asmb_15204374.trc:
ORA-04031: unable to allocate 4120 bytes of shared memory ("shared pool","unknown object","init_heap_kfsg","ASM extent pointer array")
2020-05-20T22:29:30.070882-04:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_asmb_15204374.trc:
ORA-04031: unable to allocate 4120 bytes of shared memory ("shared pool","unknown object","init_heap_kfsg","ASM extent pointer array")
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_asmb_15204374.trc  (incident=320300):
ORA-4031 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_320300/orcl_asmb_15204374_i320300.trc
2020-05-20T22:29:31.376055-04:00
USER (ospid: 15204374): terminating the instance due to error 4031

From the alert log, we can find that asmb process (ospid: 15204374) wrote following errors into trace file orcl_asmb_15204374.trc

ORA-04031: unable to allocate 4120 bytes of shared memory ("shared pool","unknown object","init_heap_kfsg","ASM extent pointer array")

Then, the asmb process terminated the instance.

This issue happened after the database home was applied Oracle database Release Update April 2020 (12.2.0.1.200414).

It is a bug introduce by Release Update 12.2.0.1.200414, and can be fixed by apply patch 31341859.

Sunday, May 17, 2020

12.2.0.1 init.ohasd does not start clusterware stack after April 2020 RU applied

The init.ohasd process does not start clusterware stack after the 12.2.0.1 Grid Infrastructure home is applied April 2020 Release Update. The scenario is not a typical Grid Infrastructure startup issue. It is a human error and happened when DBA, by accident, ran roothas.sh instead of rootcrs.sh while patching GI cluster home. Before knowing it is human error, the diagnose process is done as following.

Wednesday, May 13, 2020

impdp failed at dbms_logrep_imp.instantiate_schema with ORA-01031: insufficient privileges

When execute impdp to import 12.2 database, ORA-01031 occured as following,
[oracle@host01]$ impdp system@pdb01 dumpfile=appdata.dmp schemas=appdata

Import: Release 12.2.0.1.0 - Production on Tue May 12 17:58:56 2020

Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.
Password:

Connected to: Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
Starting "SYSTEM"."SYS_IMPORT_SCHEMA_03":  system/********@pdb01 dumpfile=appdata.dmp schemas=appdata
Processing object type SCHEMA_EXPORT/USER
Processing object type SCHEMA_EXPORT/SYSTEM_GRANT
Processing object type SCHEMA_EXPORT/DEFAULT_ROLE
Processing object type SCHEMA_EXPORT/TABLESPACE_QUOTA
Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA
ORA-39083: Object type PROCACT_SCHEMA failed to create with error:
ORA-31625: Schema APPDATA is needed to import this object, but is unaccessible
ORA-01031: insufficient privileges

Failing sql is:
BEGIN
sys.dbms_logrep_imp.instantiate_schema(schema_name=>SYS_CONTEXT('USERENV','CURRENT_SCHEMA'), export_db_name=>'DB02', inst_scn=>'386040157882');COMMIT; END;
Processing object type SCHEMA_EXPORT/SEQUENCE/SEQUENCE
Processing object type SCHEMA_EXPORT/SEQUENCE/GRANT/OWNER_GRANT/OBJECT_GRANT
ORA-39083: Object type OBJECT_GRANT failed to create with error:
ORA-31625: Schema APPDATA is needed to import this object, but is unaccessible
ORA-01031: insufficient privileges

Failing sql is:
GRANT SELECT ON "APPDATA"."WORKLOAD_SEQ" TO "USER01"

ORA-39083: Object type OBJECT_GRANT failed to create with error:
ORA-31625: Schema APPDATA is needed to import this object, but is unaccessible
ORA-01031: insufficient privileges

Failing sql is:
GRANT SELECT ON "APPDATA"."LOG_SEQ" TO "USER02"

The user SYSTEM, which is used by impdp to connect to database, does not have BECOME USER privilege, 
system@PDB01> select * from dba_sys_privs where privilege='BECOME USER';

GRANTEE    PRIVILEGE       ADMIN_OPTION COMMON INHERITED
---------- --------------- ------------ ------ ---
SYS        BECOME USER     NO           YES    YES

The healthy database privilege grant looks like,
system@PDB01> select * from dba_sys_privs where privilege='BECOME USER';

GRANTEE              PRIVILEGE    ADMIN_OPTION COMMON INHERITED
-------------------- ------------ ------------ ------ ---
DBA                  BECOME USER  NO           YES    YES
IMP_FULL_DATABASE    BECOME USER  NO           YES    YES
SYS                  BECOME USER  NO           YES    YES

It can be fixed by running grand command
grant BECOME USER to DBA,IMP_FULL_DATABASE;

This privilege is normally granted to roles DBA and IMP_FULL_DATABASE. For some reason, it could be revoked. One of the scenario is that DBUA run script dvu121.sql to revoke BECOME USER privileges from DBA and IMP_FULL_DATABASE roles when upgrading database to 12.2 and Database Vault component is VALID.

Friday, April 24, 2020

opatchauto out-of-place patching Oracle Restart and Database home

Out of place (OOP) patching is a new feature introduced by Oracle 12.2 with the latest version of opatchauto. It is a patching method where patching is performed by following operations,

  * Creating a copy of current oracle home
  * Applying patches to the copied home
  * Switching services to the copied home

This patching method helps on moving Oracle home and/or reducing the downtime of patching Oracle home and database. OPatchauto supports OOP patching for the following configurations,

  * Grid Infrastructure (GI) and Real Application Cluster (RAC)
  * Oracle Restart (Standalone Grid Infrastructure) (SIHA) and single instance database (SIDB)

Tuesday, December 24, 2019

DBA_REGISTRY_SQLPATCH shows END status after datapatch ran against database

During postinstallation of a database patching, datapatch utility shows following message while loading modified SQL files into the database,
[oracle@host01]$ $ORACLE_HOME/OPatch/datapatch -verbose
SQL Patching tool version 12.2.0.1.0 Production on Sun Dec 22 14:32:47 2019
Copyright (c) 2012, 2019, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_18481316_2019_12_22_14_32_47/sqlpatch_invocation.log

Connecting to database...OK
Bootstrapping registry and package to current versions...done
Determining current state...done

Current state of SQL patches:
Bundle series DBRU:
  ID 191015 in the binary registry and ID 190416 in the SQL registry

Adding patches to installation queue and performing prereq checks...
Installation queue:
  Nothing to roll back
  The following patches will be applied:
    30138470 (DATABASE OCT 2019 RELEASE UPDATE 12.2.0.1.191015)

Installing patches...
Patch installation complete.  Total patches installed: 3

Validating logfiles...
Patch 30138470 apply: SUCCESS
  logfile:  (no errors)
SQL Patching tool complete on Sun Dec 22 14:33:32 2019


It shows 'Patch 30138470 apply: SUCCESS', but the log file name is missing and patch status shows END in view DBA_REGISTRY_SQLPATCH,
SQL> select patch_id,version,action,status,description
  2   from dba_registry_sqlpatch
  3  order by action_time;

  PATCH_ID VERSION     ACTION   STATUS     DESCRIPTION
---------- ----------- -------- ---------- -------------------------------------------------
  29314339 12.2.0.1    APPLY    SUCCESS    DATABASE APR 2019 RELEASE UPDATE 12.2.0.1.190416
  30138470 12.2.0.1    APPLY    END

SQL>


Check the sql execution log,
[oracle@host01]$ cd /u01/app/oracle/cfgtoollogs/sqlpatch/
[oracle@host01]$ ls -l
total 104
drwxr-xr-x    3 oracle   oracle          256 Jul 21 16:28 29314339
drwxr-xr-x    3 oracle   oracle          256 Dec 22 14:33 30138470
drwxr-xr-x    2 oracle   oracle         4096 Jul 21 16:24 sqlpatch_10879194_2019_07_21_16_24_15
drwxr-xr-x    2 oracle   oracle         4096 Dec 22 14:33 sqlpatch_18481316_2019_12_22_14_32_47
-rw-r--r--    1 oracle   oracle         1373 Dec 22 14:31 sqlpatch_history.txt

[oracle@host01]$ cd 30138470
[oracle@host01]$ ls -l
total 0
drwxr-xr-x    2 oracle   oracle          256 Dec 22 15:10 23136421
[oracle@host01]$ cd 23136421
[oracle@host01]$ ls -l
total 440
-rw-r--r--    1 oracle   oracle       184086 Dec 22 14:33 30138470_apply_DB01_2019Dec22_14_33_31.log
[oracle@host01]$ pwd
/u01/app/oracle/cfgtoollogs/sqlpatch/30138470/23136421
[oracle@host01]$
[oracle@host01]$ grep -n 'ORA-' 30138470_apply_DB01_2019Dec22_14_33_31.log
32:ORA-01693: max # extents (505) reached in lob segment
34:ORA-06512: at "SYS.DBMS_SQLPATCH", line 639
35:ORA-06512: at "SYS.DBMS_SQLPATCH", line 605
36:ORA-06512: at line 2
65:ORA-20004: Uninitialized state in install_file
66:ORA-06512: at "SYS.DBMS_SQLPATCH", line 738
159:ORA-20004: Uninitialized state in install_file
       ...
7305:SQL> PROMPT IGNORABLE ERRORS: ORA-04043
7306:IGNORABLE ERRORS: ORA-04043
7313:ORA-20004: Uninitialized state in install_file
7314:ORA-06512: at "SYS.DBMS_SQLPATCH", line 738
7407:ORA-20004: Uninitialized state in install_file
7408:ORA-06512: at "SYS.DBMS_SQLPATCH", line 738


ORA-01693 is found in line 32 of the log file. List the details of the errors,
[oracle@host01]$ head -37 30138470_apply _DB01_2019Dec22_14_33_31.log
SQL>
SQL> SET PAGESIZE 0
SQL> SELECT 'Starting apply for patch 30138470/23136421 on ' ||
  2         SYSTIMESTAMP FROM dual;
Starting apply for patch 30138470/23136421 on 22-DEC-19 02.33.31.103477 PM -05:0
0


SQL> SET PAGESIZE 10
SQL>
SQL> BEGIN
  2      dbms_sqlpatch.patch_initialize(p_patch_id      => 30138470,
  3                                     p_patch_uid     => 23136421,
  4                                     p_flags         => '&flags',
  5                                     p_description   => '&description',
  6                                     p_action        => 'APPLY',
  7                                     p_logfile       => '&full_logfile',
  8                                     p_bundle_series => '&bundle_series');
  9  END;
 10  /
old   4:                                    p_flags         => '&flags',
new   4:                                    p_flags         => 'NB',
old   5:                                    p_description   => '&description',
new   5:                                    p_description   => 'DATABASE OCT 2019 RELEASE UPDATE 12.2.0.1.191015',
old   7:                                    p_logfile       => '&full_logfile',
new   7:                                    p_logfile       => '/u01/app/oracle/cfgtoollogs/sqlpatch/30138470/23136421/30138470_apply_DB01_2019Dec22_14_33_31.log',
old   8:                                    p_bundle_series => '&bundle_series');
new   8:                                    p_bundle_series => 'DBRU');
BEGIN
*
ERROR at line 1:
ORA-01693: max # extents (505) reached in lob segment
SYS.SYS_LOB0000190607C00013$$
ORA-06512: at "SYS.DBMS_SQLPATCH", line 639
ORA-06512: at "SYS.DBMS_SQLPATCH", line 605
ORA-06512: at line 2

[oracle@host01]$


The segment 'SYS.SYS_LOB0000190607C00013$$' reaches the extent maximum limitation (505), and it caused the SQL failure, find out what the segment is
SQL> select owner,object_name,object_type
  2    from dba_objects
  3   where object_name='SYS_LOB0000190607C00013$$';

OWNER                OBJECT_NAME                    OBJECT_TYPE
-------------------- ------------------------------ -----------------------
SYS                  SYS_LOB0000190607C00013$$      LOB

SQL> select owner,table_name,column_name,segment_name
  2    from dba_lobs
  3   where segment_name='SYS_LOB0000190607C00013$$';

OWNER      TABLE_NAME                     COLUMN_NAME          SEGMENT_NAME
---------- ------------------------------ -------------------- -------------------------
SYS        REGISTRY$SQLPATCH              PATCH_DIRECTORY      SYS_LOB0000190607C00013$$

SQL>
SQL> desc sys.registry$sqlpatch
 Name                          Null?    Type
 ----------------------------- -------- -------------------------
 PATCH_ID                      NOT NULL NUMBER                  
 ACTION                        NOT NULL VARCHAR2(15)            
 ACTION_TIME                   NOT NULL TIMESTAMP(6)            
 DESCRIPTION                            VARCHAR2(100)           
 LOGFILE                                VARCHAR2(500)           
 STATUS                                 VARCHAR2(25)            
 VERSION                       NOT NULL VARCHAR2(20)            
 PATCH_UID                     NOT NULL NUMBER                  
 FLAGS                                  VARCHAR2(10)            
 BUNDLE_SERIES                          VARCHAR2(30)            
 BUNDLE_ID                              NUMBER                  
 INSTALL_ID                             NUMBER                  
 PATCH_DIRECTORY                        BLOB                    
 BUNDLE_DATA                            XMLTYPE                 
 PATCH_DESCRIPTOR                       XMLTYPE                 
 POST_LOGFILE                           VARCHAR2(500)           
SQL>
SQL> select owner,segment_name,max_extents
  2    from dba_segments
  3   where segment_name in ('REGISTRY$SQLPATCH','SYS_LOB0000190607C00013$$');

OWNER      SEGMENT_NAME                   MAX_EXTENTS
---------- ------------------------------ -----------
SYS        REGISTRY$SQLPATCH                      505
SYS        SYS_LOB0000190607C00013$$              505


The segment is BLOB column PATCH_DIRECTORY of table SYS.REGISTRY$SQLPATCH, both the BLOB segment and its table have extent maximum limitation 505. To remove the limitation,
SQL> alter table sys.registry$sqlpatch storage (maxextents unlimited);

Table altered.

SQL> alter table sys.registry$sqlpatch
  2  modify lob (patch_directory) (storage (maxextents unlimited));

Table altered.

SQL> select owner,segment_name,max_extents
  2    from dba_segments
  3   where segment_name in ('REGISTRY$SQLPATCH','SYS_LOB0000190607C00013$$');

OWNER      SEGMENT_NAME                   MAX_EXTENTS
---------- ------------------------------ -----------
SYS        REGISTRY$SQLPATCH               2147483645
SYS        SYS_LOB0000190607C00013$$       2147483645



Then re-run the datapatch utility,
[oracle@host01]$ $ORACLE _HOME/OPatch/datapatch -verbose
SQL Patching tool version 12.2.0.1.0 Production on Sun Dec 22 16:43:43 2019
Copyright (c) 2012, 2019, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_35782878_2019_12_22_16_43_43/sqlpatch_invocation.log

Connecting to database...OK
Bootstrapping registry and package to current versions...done
Determining current state...done

Current state of SQL patches:
Bundle series DBRU:
  ID 191015 in the binary registry and ID 190416 in the SQL registry

Adding patches to installation queue and performing prereq checks...
Installation queue:
  Nothing to roll back
  The following patches will be applied:
    30138470 (DATABASE OCT 2019 RELEASE UPDATE 12.2.0.1.191015)

Installing patches...
Patch installation complete.  Total patches installed: 1

Validating logfiles...
Patch 30138470 apply: SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30138470/23136421/30138470_apply_DB01_2019Dec22_16_44_33.log (no errors)
SQL Patching tool complete on Sun Dec 22 16:45:22 2019


Now, it gives the log file name. Validate the status,
SQL> select patch_id,version,action,status,description
  2    from dba_registry_sqlpatch order by action_time;

  PATCH_ID VERSION     ACTION   STATUS     DESCRIPTION
---------- ----------- -------- ---------- -------------------------------------------------
  29314339 12.2.0.1    APPLY    SUCCESS    DATABASE APR 2019 RELEASE UPDATE 12.2.0.1.190416
  30138470 12.2.0.1    APPLY    END
  30138470 12.2.0.1    APPLY    SUCCESS    DATABASE OCT 2019 RELEASE UPDATE 12.2.0.1.191015


The patch is installed successfully.