Hosting X509 Certificate Authenticator with SSL Termination in a WSO2 IS Multi Node Cluster

11 min readJan 18, 2024

What is X509 Certificate Authentication?

The main goal of authentication is to confirm that you are who you claim to be. Think of certificate-based authentication as your sophisticated ID card, where a client certificate is used in place of a username and password. This ensures that you and the server are legitimate during the digital handshake in addition to establishing an extra layer of security.

Now, let’s talk about X509 — it’s the cool standard that defines how these certificates should look. You’ll find these certificates hanging out in protocols like TLS/SSL.

Think of your keystore as your secret stash — it’s where you keep your private keys, certificates, and other goodies. On the flip side, the truststore is like your digital vault for certificates from trusted Certificate Authorities (CA) — the big shots who vouch for the realness of the certificates.

In the SSL world, both the client and server have their own keystore and truststore. They exchange certificates like friendship bracelets. First, the client says “Hello” and asks the server for its certificate. Then they do a little certificate show-and-tell. The client checks the server’s certificate against its truststore, and the server does the same with its client’s certificate [1]. Once they’ve confirmed each other’s legitimacy, boom — Mutual SSL connection is established, and the digital party begins!

What is SSL Termination?

SSL termination is like having a superhero intercept encrypted traffic before it reaches your servers. This traffic is then decoded and inspected by a friendly neighbourhood Application Delivery Controller (ADC) or a specialized SSL termination device, taking the load off your app server. The ADC is a real multitasker, excelling at handling the complex job of decrypting numerous SSL connections [2]. This leaves your server free to focus on what it does best — processing applications without breaking a sweat!.

Setting Up WSO2 IS Cluster

1. Download the Latest WSO2 Identity Server

Begin your journey by acquiring the latest WSO2 Identity Server from here. After downloading, you can extract the zip. Let’s mark the extracted folder as the “IS Master Node”. Make a copy of the extracted folder (and mark it as the “IS Worker Node”) in order to create a two node cluster.

From here onwards, $CARBON_HOME is defined as the symbolic root for both IS nodes.

2. Configure Hostname

Now, let’s configure the hostname of our two nodes using the deployment.toml file located at $CARBON_HOME/repository/conf directory (Update deployment.toml files in both nodes) as follows.

[server]
hostname = "wso2.is.sample.domain.com"
internal_hostname = "wso2.is.sample.domain.com"

In order to register this domain mapping in our local device, go to /etc/hosts file and add the following line.

127.0.0.1   wso2.is.sample.domain.com

After that, add an offset to the port of the work node, since it needs to be deployed in another port (9444) compared to the master node.

[server]
offset = "1"

Now, let’s proxy both nodes by adding the following configurations into each of the deployment.toml files.

[transport.http.properties]
proxyPort = 80
[transport.https.properties]
proxyPort = 443

To facilitate Hazelcast clustering [3] log functionality, add the following properties into the deployment.toml, so that you can observe the deployment of the two nodes.

[hazelcast]
"hazelcast.shutdownhook.enabled" = "false"
"hazelcast.logging.type"= "log4j2"

3. Enable Clustering in Both Nodes

You can enable clustering in both nodes by adding the following configuration into the deployment.toml. Here, Well-Known Address (WKA) membership scheme is used for clustering. In WKA scheme, a subset of members are predefined for all the members and therefore, at least one member is well-known to the whole cluster.

[clustering]
membership_scheme = "wka"
local_member_host = "<IP_address_of_node>"
local_member_port = "<port_number>"
members = ["<IP_address_of_well_known_node_1>:<port_number>", "<IP_address_of_well_known_node_2>:<port_number>"]

Refer the following sample configurations for the master and worker node for further clarification.

Master Node


[clustering]
membership_scheme = "wka"
local_member_host = "127.0.0.1"
local_member_port = "4000"
members = ["127.0.0.1:4000", "127.0.0.1:4001"]

Worker Node

[clustering]
membership_scheme = "wka"
local_member_host = "127.0.0.1"
local_member_port = "4001"
members = ["127.0.0.1:4000", "127.0.0.1:4001"]

4. Configure Database

Now, it’s time to configure a new Database rather than using the predefined default H2 database in WSO2 IS. For the purpose of this article, let’s configure a MySQL database.

First of all, you need to start a MySQL server and create the following two tables using the relevant database scripts. (Both the nodes will share the same databases since they are operating in a single cluster)

Identity Database

$CARBON_HOME/dbscripts/identity/mysql.sql
$CARBON_HOME/dbscripts/identity/uma/mysql.sql (only if exists)
$CARBON_HOME/dbscripts/consent/mysql.sql

Shared Database

$CARBON_HOME/dbscripts/mysql.sql

In order to register these databases in our IS cluster add the following configurations into each deployment.toml.

[database.identity_db]
type = "mysql"
hostname = "<mysql-hostname>"
name = "<identity_db_name>?verifyServerCertificate=false&amp;useSSL=false&amp;requireSSL=false"
username = "<db_user_name>"
password = "<db_user_password>"
port = "<mysql_server_port>"

[database.shared_db]
type = "mysql"
hostname = "<mysql-hostname>"
name = "<shared_db_name>?verifyServerCertificate=false&amp;useSSL=false&amp;requireSSL=false"
username = "<db_user_name>"
password = "<db_user_password>"
port = "<mysql_server_port>"

Consider the following sample for further clarification.

[database.identity_db]
type = "mysql"
hostname = "localhost"
name = "identitydb?verifyServerCertificate=false&amp;useSSL=false&amp;requireSSL=false"
username = "mysqlUser"
password = "mypassword"
port = "3306"

[database.shared_db]
type = "mysql"
hostname = "localhost"
name = "shareddb?verifyServerCertificate=false&amp;useSSL=false&amp;requireSSL=false"
username = "mysqlUser"
password = "mypassword"
port = "3306"

As the final step of db setup, add the MySQL JDBC driver into $CARBON_HOME/repository/components/lib directory.

5. Configure NGINX Load Balancer

It’s time to setup a load balancer in order to front our IS cluster. For this article, I’ll be using Nginx load balancer.

For macOS, you can download the Nginx server through Homebrew as follows.

brew install nginx

After installing Nginx, add the following configurations into /opt/homebrew/etc/nginx/nginx.conf file (or you can create a new .conf file with the configuration and add it to Nginx).

upstream ssl.wso2.is.sample.domain.com {
        server 127.0.0.1:9443;  # IS Master Node <IP>:<Port>
        server 127.0.0.1:9444;  # IS Worker Node <IP>:<Port>
        ip_hash;
}

 server {
       listen       443 ssl;
       server_name  wso2.is.sample.domain.com;

       ssl_certificate      path/to/cert;
       ssl_certificate_key  path/to/cert-key;

       location / {
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header X-Forwarded-Server $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $http_host;
            proxy_read_timeout 5m;
            proxy_send_timeout 5m;
            proxy_pass https://ssl.wso2.is.sample.domain.com;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_set_header X-SSL-CERT $ssl_client_cert;
       }
    }

In this configuration, HTTPS requests made to “wso2.is.sample.domain.com” will be directed to one of the IS nodes as mentioned in the upstream block. Therefore, our IS cluster is proxied through 443 port. With “ip_hash” in the upstream block, Nginx will use a hash-function to determine which node should selected based on the client’s IP address.

Furthermore, for our domain, we need to create a self singed certificate which is referenced through “ssl_certificate” and “ssl_certificate_key” attributes in the server block. In order to generate this certificate refer this link and a sample command is mentioned below.

openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 \
  -nodes -keyout example.com.key -out example.com.crt -subj "/CN=example.com" \
  -addext "subjectAltName=DNS:example.com,DNS:*.example.com,IP:127.0.0.1"

After adding the above configurations, you can start the Nginx server by using the following command.

brew services start nginx

6. Start the IS Cluster

After Configuring the load balancer, you can start the master node by running the following command at $CARBON_HOME/bin.

sh wso2server.sh

Note the following logs in the terminal which indicates that this master node was selected as the coordinator node of the cluster.

[2024-01-05 15:21:37,837] []  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Local member: [9db3934c-0949-4609-a9e0-76174ad6cdaf] - Host:127.0.0.1, Remote Host:null, Port: 4000, HTTP:9763, HTTPS:8443, Domain: wso2.carbon.domain, Sub-domain:worker, Active:true
[2024-01-05 15:21:37,850] []  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Elected this member [9db3934c-0949-4609-a9e0-76174ad6cdaf] as the Coordinator node

Subsequently, start the worker node by running the same command in worker node’s bin directory. Now, you can observe the following logs in the master node terminal, where it indicates that a new node joined the cluster.

[2024-01-05 15:25:11,106] []  INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - Member joined [fa45f7da-0a67-4d44-8d39-f0b7ddd66545]: /127.0.0.1:4001

Now that we’ve got the IS cluster up and running, if you observe the behaviour mentioned above, that means you’ve nailed the cluster setup!

Configure X509 Authenticator

1. Generate a Self-signed Certificate for Client

As the first step in configuring X509 authenticator, you should generate a self signed certificate with its private key for your client app. You can follow the below instructions in order to create a sample certificate and its private key.

openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out client.pem

This will prompt a set of questions where you can answer them as follows.

Country Name (2 letter code) [AU]: SL
State or Province Name (full name) [Some-State]: Western
Locality Name (eg, city) [ ]: Colombo
Organization Name (eg, company) [Internet Widgits Pty Ltd]: WSO2
Organizational Unit Name (eg, section) [ ]: QA
Common Name (e.g. serverFQDN or YOUR name) [ ]: johndoe
Email Address [ ]: johndoe@gmail.com

This Common Name (CN) will be used as the username when authenticating into the client app.

Created certificate can be viewed using the following command.

openssl x509 -text -noout -in client.pem

Now combine your certificate and the private key into a PKCS#12 (P12) bundle.

openssl pkcs12 -inkey key.pem -in client.pem -export -out client.p12

Your p12 file can be validated as follows.

openssl pkcs12 -in client.p12 -noout -info

Finally, export the created certificate into trust stores in both nodes.

keytool -importcert -alias localcrt -file client.pem -keystore $CARBON_HOME/repository/resources/security/client-truststore.jks -storepass wso2carbon -noprompt

2. Export the Certificate into the Browser

In order to obtain a flawless X509 authentication flow, you need to include your generated certificate in your browser and when you log in to your client app, this certificate will be used for authentication. For the purpose of this article, I’ll be using the Mozilla Firefox browser. Follow the steps as below, in order to include your certificate in the web browser.

2. Search for certificates in the search bar and click on “View Certificates”.

5. Verify that the certificate was added.

3. Configure X509 Certificate Authenticator Attributes

Add the following configurations into both deployment.toml files in order to register our X509 Certificate authenticator in both nodes.

[authentication.authenticator.x509_certificate.parameters]
name ="x509CertificateAuthenticator"
enable=true
AuthenticationEndpoint="https://x509.is.sample.domain.com/x509-certificate-servlet"
username= "CN"

Since we are fronting the X509 authenticator through a load balancer and proxying it through 443 port, there are some extra attributes that need to be set in the ngnix.conf file in order to exchange X509 certificates between the server and the client with SSL termination. For this purpose, we are using a different domain (“x509.is.sample.domain.com”) for the X509 authentication endpoint and all these extra attributes are specifically set with this separate domain.

upstream app.x509 {
        server 127.0.0.1:9443;  # IS Master Node <IP>:<Port>
        server 127.0.0.1:9444;  # IS Worker Node <IP>:<Port>
        ip_hash;
}

server {
        listen               443 ssl;
        server_name          x509.dev.wso2.com;
        ssl_certificate      path/to/cert;
        ssl_certificate_key  path/to/cert-key;
        
        ssl_verify_client optional_no_ca;
        
        access_log path/to/x509/access.log;
        error_log  path/to/x509/error.log;

        location / {
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Server $host;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_set_header X-HTTPS-Protocol $ssl_protocol;
                proxy_set_header X-SSL-CERT $ssl_client_cert;
                proxy_read_timeout 5m;
                proxy_send_timeout 5m;
                proxy_pass https://app.x509;

                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
        }
}

Note: In the above Nginx configuration, note that the following configuration is used within the server block.

ssl_verify_client optional_no_ca;

The value for ssl_verify_client is “off” by default. In that case, client certificate verification is disabled, leading to the occurrence of the following error.

Unable to find X509 Certificate in browser.

Therefore, it is recommended to activate this attribute by selecting one of the following values based on your specific requirements.

“on”: Must require a certificate signed by a trusted authority.
“optional”: Require a certificate signed by a trusted authority only if available.
“optional_no_ca”: Require a certificate only if available and it does not need to be signed by trusted authority.

Now restart the Nginx server using the following command.

brew services restart nginx

In order to register this domain mapping in our local device, go to /etc/hosts file and add the following line.

127.0.0.1   x509.is.sample.domain.com

4. Add X509 Authentication Valve into the Server

Since NGINX decrypts the request and does not pass the X509 Certificate to the server as a request attribute, X509 Authentication is useless in conventional SSL Termination. As a result, X509 authentication fails since the server cannot authenticate the client using its certificate.

To address this issue, a separate valve called “X509 Authentication Valve” was introduced that can manage requests from NGINX and transmit the X509 Certificate as a request attribute to the server.

First, go to $CARBON_HOME/repository/components/dropins and search whether “org.wso2.carbon.extension.identity.x509Certificate.valve.<version>.jar” is available. If it is available, then ignore the rest of the instructions mention under this subtopic (4. Add X509 Authentication Valve into the server).

If it is not available, then got to https://github.com/wso2-extensions/identity-x509-commons, clone the repo and the run the following command at the root of the project.

mvn clean install

After successfully building the project, place the
identity-x509-commons/components/valve/target/org.wso2.carbon.extension.identity.x509Certificate.valve-<version>.jar in $CARBON_HOME/repository/components/dropins directory.

After that in order to register the valve at the server startup, go to $CARBON_HOME/repository/resources/conf/templates/repository/conf/tomcat/catalina-server.xml.j2 and the following highlighted line.

<Valve className="org.wso2.carbon.extension.identity.x509Certificate.valve.X509CertificateAuthenticationValve"/>

Now restart both Servers.

5. Disable Revocation Certificate Validation

Follow the steps in here, in order to disable OCSP and CRL validators.

6. Add a New User to IS

Add a new user with the Common Name (CN) used to create the x509 certificate as the username (ex: johndoe) by following the instructions in here.

7. Setup Sample App

In this article, we will be deploying the “Travelocity” sample app [4]. First, you can download the app using this link and then, add the downloaded war file into the webapps folder of your tomcat server directory.

Next, open the file, $TOMCAT_HOME/webapps/travelocity.com/WEB-INF/classes/travelocity.properties, and change the following value with the configured domain name of your IS.

SAML2.IdPURL=https://wso2.is.sample.domain.com/samlsso

Now, let’s start our tomcat server. After that, follow the steps in Register a service provider and CORS configuration, in order to register our sample app in our IS cluster.

After the basic setup is done, add X509 Certificate authentication as the Authentication mechanism for the sample app.

Log into Carbon Console as an admin, and navigate to Service Providers > List > travelocity.com > Local & Outbound Authentication Configuration. In there, select “Local Authentication” with “X509Certificate” in the drop down menu as follows.

Finally, click on “Update” button.

8. Try out the Login Flow

Now it is time to try out our app with X509 Certificate authentication. First, go to your app login page and click on “Login”. Then the browser will prompt for the client certificate as follows.

At last, you should be able to log in successfully. (Note: the requesting domain name would be the domain name that you gave in step 3).

To wrap it up, great job on getting your app all set up with X509 authenticator in WSO2 IS! I know it was a bit log journey in setting up this cool authentication feature and so you deserve a high-five. Cheers to a more secure and efficient system!