Kerberos setup in Cloudera Hadoop

Reference:

http://blog.cloudera.com/blog/2015/03/how-to-quickly-configure-kerberos-for-your-apache-hadoop-cluster/

Cloudera Security manual .pdf – CDH 5.15 on Cloudera Documentation website

http://www.ghostar.org/2015/06/google-chrome-spnego-and-webhdfs-on-hadoop/

 

Environment:

Cloudera CDH 5.15 on Centos 7

MIT KDC Kerberos


 

Setting up Kerberos in Cloudera CDH is somewhat tricky. The above blog is a good step by step way to setup. Also refer to the official Cloudera Security .pdf document on Cloudera Documentation website.

  1. Start with the section below on the Cloudera Security .pdf manual:

     Pg-39: Authentication

Standard practice is to use your organization’s domain name as the Kerberos realm name (in all uppercase characters) to easily distinguish it as part of a Kerberos principal, as shown in this user principal pattern:

username@REALM.EXAMPLE.COM

service-name/hostname.fqdn.example.com@REALM.EXAMPLE.COM

Pg-50: Enabling Kerberos Authentication Using the Wizard

Step 2 (optional): Installing JCE Policy File for AES-256 Encryption

Note: This step is not required when using JDK 1.8.0_161 or greater. JDK 1.8.0_161 enables unlimited strength encryption by default.

[root@]# java -version

java version “1.8.0_172”

Java(TM) SE Runtime Environment (build 1.8.0_172-b11)

Since we have higher java version no need to install JCE Policy File for AES-256.

Check NTP status:

$ ntpstat

$ service ntpd status

$ service ntpd restart

If not installed install NTP on RHEL on all the hosts:

$ yum install ntp

$ service ntpd restart

OS Packages Required:

RHEL/CentOS 7

We are going to install the MIT KDC on the Cloudera Manager host.

  • openldap-clients, krb5-server on the Cloudera Manager Server hosts.
  • krb5-workstation, krb5-libs on ALL hosts

 

# yum install openldap-clients

# yum install krb5-workstation

# yum install krb5-libs

# yum install krb5-server

 

Next, we need to make some changes to the /etc/krb5.conf like uncommenting and changing hostname from EXAMPLE.COM to your own hostname as below. The cent1.example.com is where the MIT KDC server was installed which was same as the Cloudera Manager host:

————————————–

# Configuration snippets may be placed in this directory as well

includedir /etc/krb5.conf.d/

 

[logging]

default = FILE:/var/log/krb5libs.log

kdc = FILE:/var/log/krb5kdc.log

admin_server = FILE:/var/log/kadmind.log

 

[libdefaults]

dns_lookup_realm = false

ticket_lifetime = 24h

renew_lifetime = 7d

forwardable = true

rdns = false

default_realm = EXAMPLE.COM

default_ccache_name = KEYRING:persistent:%{uid}

 

[realms]

EXAMPLE.COM = {

kdc = cent1.example.com

admin_server = cent1.example.com

}

 

[domain_realm]

 .example.com = EXAMPLE.COM

 example.com = EXAMPLE.COM

 


 

NEXT create the kerberos database. You can choose any password here we used cloudera as password.

[root@cent1 ~]# kdb5_util create -s

Loading random data

Initializing database ‘/var/kerberos/krb5kdc/principal’ for realm ‘EXAMPLE.COM’,

master key name ‘K/M@EXAMPLE.COM’

You will be prompted for the database Master Password.

It is important that you NOT FORGET this password.

Enter KDC database master key: cloudera

 

NEXT update the kdc.conf file /var/kerberos/krb5kdc/kdc.conf

For MIT KDC, make sure you have the following lines in the kdc.conf:

max_life = 1d

max_renewable_life = 7d

kdc_tcp_ports = 88

 

[kdcdefaults]

kdc_ports = 88

kdc_tcp_ports = 88

[realms]

EXAMPLE.COM = {

#master_key_type = aes256-cts

acl_file = /var/kerberos/krb5kdc/kadm5.acl

dict_file = /usr/share/dict/words

admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab

supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal

max_life = 1d

max_renewable_life = 7d

}

 

—————————–

 

The acl file may need to be updated so the */admin is enabled with admin privileges in /var/kerberos/krb5kdc/kadm5.acl

*/admin@EXAMPLE.COM     *

 

Restart the KDC server and enable autostart on reboot otherwise hadoop wont start after reboot:

[root@cent1 ~]# service krb5kdc start

Redirecting to /bin/systemctl start krb5kdc.service

[root@cent1 ~]# systemctl restart kadmin

Redirecting to /bin/systemctl start kadmin.service

# systemctl enable krb5kdc

# systemctl enable kadmin

 

Create the Cloudera Manager Server administrator principal as shown below, using the admin instance name and your realm name.

Pg-54: For MIT Kerberos KDC on the local host:

[root@cent1 ~]# kadmin.local

Authenticating as principal root/admin@EXAMPLE.COM with password.

kadmin.local:  addprinc -pw cloudera cloudera-scm/admin@EXAMPLE.COM

WARNING: no policy specified for cloudera-scm/admin@EXAMPLE.COM; defaulting to no policy

Principal “cloudera-scm/admin@EXAMPLE.COM” created.

kadmin.local:  modprinc -maxrenewlife 1week cloudera-scm/admin@EXAMPLE.COM

Principal “cloudera-scm/admin@EXAMPLE.COM” modified.

 

Test the server by authenticating as the CM admin user:

 

[root@cent1 ~]# kinit cloudera-scm/admin@EXAMPLE.COM

Password for cloudera-scm/admin@EXAMPLE.COM: cloudera

 

[root@cent1 ~]# klist -e

Ticket cache: KEYRING:persistent:0:0

Default principal: cloudera-scm/admin@EXAMPLE.COM

 

Valid starting       Expires              Service principal

07/16/2018 14:28:07  07/17/2018 14:28:07  krbtgt/EXAMPLE.COM@EXAMPLE.COM

Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

 

 

NEXT Pg-54: Step 4: Enabling Kerberos Using the Wizard

 

Important:

If YARN Resource Manager HA has been enabled in a non-secure cluster, before enabling Kerberos

you must clear the StateStore znode in ZooKeeper, as follows:

  1. Go to the Cloudera Manager Admin Console home page, click to the right of the YARN service

and select Stop.

  1. When you see a Finished status, the service has stopped.
  2. Go to the YARN service and select Actions > Format State Store.
  3. When the command completes, click Close.

 

To start the Kerberos wizard:

  1. Go to the Cloudera Manager Admin Console and click to the right of the cluster for which you want to enable

Kerberos authentication.

  1. Select Enable Kerberos.

 

For MIT KDC, make sure you have the following lines in the /var/kerberos/krb5kdc/kdc.conf

max_life = 1d

max_renewable_life = 7d

kdc_tcp_ports = 88

 

Click the 3 check boxes and continue.

 

[root@cent1 ~]# klist -e

Ticket cache: KEYRING:persistent:0:0

Default principal: cloudera-scm/admin@EXAMPLE.COM

 

Valid starting       Expires              Service principal

07/16/2018 14:28:07  07/17/2018 14:28:07  krbtgt/EXAMPLE.COM@EXAMPLE.COM

Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

 

Check the Blog at the top to see what needs to be entered in each field.

Select MIT KDC

Select the above Kerberos Encryption Type in dropdown.

Kerberos Security Realm: EXAMPLE.COM or whatever is domain

KDC Server Host: Where the MIT KDC is installed.

KDC Admin Server Host: Where the MIT KDC is installed.

Next screen select:

Manage krb5.conf through Cloudera manager check it.

In the KDC Account enter:

cloudera-scm/admin@EXAMPLE.COM

password:  cloudera

Keep the other screens as default.

It will take about 30min to run the setup. Finally it should say:

Successfully enabled Kerberos.

 

TESTING KERBEROS:

NEXT if you try to run a hdfs command you will get error below which :

# hadoop fs -ls /

18/07/17 12:47:08 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

 

NEXT Step 6: Get or Create a Kerberos Principal for Each User Account

# kadmin.local

Authenticating as principal root/admin@EXAMPLE.COM with password.

kadmin.local:  addprinc joe@EXAMPLE.COM

WARNING: no policy specified for joe@EXAMPLE.COM; defaulting to no policy

Enter password for principal “joe@EXAMPLE.COM”: joe123

Re-enter password for principal “joe@EXAMPLE.COM”:

Principal “joe@EXAMPLE.COM” created.

 

Step 7: Prepare the Cluster for Each User

 

  1. Make sure all hosts in the cluster have a Linux user account with the same name as the first component of that

user’s principal name. For example, the Linux account joe should exist on every box if the user’s principal name is joe@YOUR-REALM.COM. You can use LDAP for this step if it is available in your organization.

 

[root@cent1 ~]# groupadd joe

[root@cent1 ~]# useradd -m -g joe  -c “joe” joe

 

  1. Need to add the user joe in HUE with a home directory otherwise write to inode fails.

org.apache.hadoop.security.AccessControlException: Permission denied: user=joe, access=WRITE, inode=”/user”:hdfs:supergroup:drwxr-xr-x

 

Step 8: Verify that Kerberos Security is Working

  1. Acquire Kerberos credentials for your user account.

[root@cent1 ~]# kinit joe@EXAMPLE.COM

Password for joe@EXAMPLE.COM: joe123

# kadmin.local

Authenticating as principal joe/admin@EXAMPLE.COM with password.

kadmin.local:  list_principals

$ klist -e

Ticket cache: FILE:/tmp/krb5cc_0

Default principal: joe@EXAMPLE.COM

 

Valid starting       Expires              Service principal

07/17/2018 14:20:12  07/18/2018 14:20:12  krbtgt/EXAMPLE.COM@EXAMPLE.COM

Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

 

[root@cent1]# hadoop fs -ls /

Found 4 items

drwx——   – hbase hbase               0 2018-07-17 12:32 /hbase

drwxrwxr-x   – solr  solr                0 2018-07-03 18:06 /solr

drwxrwxrwt   – hdfs  supergroup          0 2018-07-03 19:23 /tmp

drwxr-xr-x   – hdfs  supergroup          0 2018-07-17 14:27 /user

Next, invalidate the Kerberos token so as not to break anything:

[root@cent1 ~]# kdestroy

[root@cent1 ~]#  klist -e

klist: No credentials cache found (filename: /tmp/krb5cc_0)

[root@cent1 ~]# hadoop fs -mkdir /user/joe/temp1

18/07/17 15:50:21 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

 

Troubleshooting the Kerberos Ticket Renewer:

If the Hue Kerberos Ticket Renewer does not start, check the configuration of your Kerberos Key Distribution Center (KDC). Look at the ticket renewal property, maxrenewlife, to ensure that the principals, hue/<hostname> and krbtgt, are renewable. If these principals are not renewable, run the following commands on the KDC to enable them:

kadmin.local:  modprinc -maxrenewlife 90day krbtgt/EXAMPLE.COM

Principal “krbtgt/EXAMPLE.COM@EXAMPLE.COM” modified.

kadmin.local:  modprinc -maxrenewlife 90day +allow_renewable hue/cent1.example.com@EXAMPLE.COM

Principal “hue/cent1.example.com@EXAMPLE.COM” modified.

After this restart the Kerberos Ticket Renewer and HUE role error is resolved.

 

Next check out the blog on this site on SPNEGO and webhdfs access using Chrome browser.

 

 

 

 

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s