Auth0 ODIC OpenID with Domino & Some other interesting findings  

By Daniel Nashed | 2/22/24 1:21 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

We are working on a ODIC setup with a German business partner for a larger German customer. Auth0 is one of the major providers. We got it working but only with some tricks for now. It turned out the Auth0 OIDC endpoint has a cache expiration for 15 seconds. This looks like a setting that can't be changed. The Domino OIDC cache uses the expiration header to invalidate the cache. So our cache on the Domino side was constantly reloading and invalid in some cases. You really have to have an expiration that is at least a couple of minutes. Better at least 1 hour. Faking the cache expiration This has been reported to HCL and the team is working on an enhancement. Meanwhile I came up with a work-around setting up a Fake provider on a NGINX server to forward the requests.

Domino Backup/Restore with multiple configurations and targets  

By Daniel Nashed | 2/22/24 1:14 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino Back/Restore is a flexible framework for native Domino backup. The dominobackup.nsf plays an important role for backup and restore operation. It contains the following type of content. Backup/restore/prune configuration Inventory documents for restore operations Restore requests Backup logs You could run backup with different excludes defined on command-line. Or just backup selected databases or incremental backups. But there cannot be different active configurations nor different backup retention in one dominobackup.nsf

Domino autoupdate.nsf for fast internal software downloads  

By Daniel Nashed | 2/22/24 1:13 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino Autoupdate has been introduced in Domino 14.0. It offers automatic downloads from My HCLSoftware download, which has been on early access in parallel and has been released at the same time. My Engage session will go into detail about the functionality with tips and tricks and additional information round both features and the new Domino Download script (https://nashcom.github.io/domino-startscript/domdownload/). But I want already provide some details about options available today with simple integrations.

Easy to use container image providing ICAP support for ClamAV for Domino CScan  

By Daniel Nashed | 2/22/24 1:11 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

My friend and fellow Ambassador Roberto Boccadoro submitted an Engage session about Domino CScan with ICAP. Sadly his session did not make it into the agenda. But he is part of two OpenNTF sessions. This session idea lead to a new OpenSource project I initiated to help with ICAP support. Thanks Roberto for pushing me to get this implemented! :-) The new project provides a simple to build container image, which natively offers ICAP services over TLS with a ClamAV container in the back-end. The container is ready to be consumed with Domino CScan/ICAP (https://help.hcltechsw.com/domino/14.0.0/admin/conf_scanningattachmentsforviruses.html). It comes with a docker-compose file which glues the official ClamAV container with this new image.

Running Domino in LXC containers on Proxmox requires a trick  

By Daniel Nashed | 2/22/24 1:10 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Now that VMware might not be everyone's darling any more because of it's new mother ship, I took another look at Proxmox. I know them for quite a while and they are doing a great job. I rebuilt a Intel NUC with 2 TB NVMe disk with the current version of Proxmox. Proxmox supports full VMs and also LXC -- which is an interesting option for testing in lab environments. You can setup a new Linux test machine in minutes from a template. And there are ready to use templates for all major Linux distributions. I had a post long time ago about Proxmox automation on command-line.

Certificate ASN.1 Decoding online  

By Daniel Nashed | 2/15/24 2:35 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Now that I posted the TLS 1.2 interactive information side today, some of you might also want to get details out of certificates. Certificates are usually public information. So it should be OK to paste them into the website https://asn1js.eu/. But there is a GitHub project referenced and you could run it also locally. The inner guts of certificates are presented in ASN.1. When you ever has looked at OpenSSL C code, you will recognize the structures.

The Illustrated TLS 1.2 Connection -- Every byte explained  

By Daniel Nashed | 2/15/24 2:32 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

While debugging a TLS connection issue, I ran into this website -->https://tls12.xargs.org/ It provides more details then most admins ever want to know. But it is a great resource understanding a TLS connection.

Domino Backup customized and centralized logging  

By Daniel Nashed | 2/12/24 2:25 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

This question came up last week in a business partner workshop. The partner wanted to centralize the logging of all Domino backup instances. dominobackup.nsf intended per server. You could configure a global configuration database and local instances for the backup inventory. In theory it could be one database for multiple could be also replicated in smaller environments. This would not be recommended. But there is an easier way for a centralized overview of all your Domino backups.

Domino adding Trusted Roots for Java applications  

By Daniel Nashed | 2/12/24 2:24 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino has different places to store trusted roots depending on the part of the application. Beginning with Domino 12.0.2 HCL started to consolidate root certificates into the new domain wide certstore.nsf. But it will take some time to have all parts of Domino to use the new trusted roots back-end. New callers like OIDC or CScan/ICAP and the certificate URL heath check already use the new back-end including UI integration. JVM trusted roots cacerts overwritten by Domino update Java still uses it's own cacerts file, which is part of the JVM directory. The file is only admin/root writable. Domino release installers replace the cacerts file with the latest cacerts available. But this overwrites custom certificates imported into cacarts. This is a common problem I ran into twice in the last two weeks.

Enable DKIM for Domino  

By Daniel Nashed | 2/12/24 2:22 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Enable DKIM for Domino - DKIM inbound is supported starting with Domino 12.0.1 - DKIM outbound is supported starting with Domino 12.0.2 Now the first providers raise the bar for sending mails. This might not only be relevant for mass mail. Here is a short write up enabling DKIM for RSA and Ed25519 keys.

New default Let’s Encrypt certificate chain with ISRG Root X1 root  

By Daniel Nashed | 2/12/24 2:18 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Let's Encrypt finally changed their default root certificate from DST Root CA X3 to ISRG Root X1. The old root expired already 2 1/2 years ago, but was cross signed with the new chain. Now finally Let's Encrypt uses the new root by default, which results in a shorter chain. They have been using the older, longer chain to specially support older Android devices, which didn't have the X1 root in their trust store. When you are using Let's Encrypt ACME and did not specify an alternate chain, there is nothing to change. The new shorter certificate chain will be automatically used the next time the certificate is renewed. But in case you set specific settings, you might now have to remove those settings, because they flipped the certificate chains. The alternate chain is now the older longer certificate chain. For Domino CertMgr the custom setting is "ACME Alternate Chain Suffix".

The Art of Troubleshooting   

By Daniel Nashed | 1/15/24 3:38 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In all the years I am involved in troubleshooting, I still see the same patterns. I am planning to start a new initiative this year. To start with, I wrote a short abstract end of the year when looking into this. See this as a beginning of a change from my side to better help on community level and also provide better services as a HCL business partner. It will also include troubleshooting steps for different kind of problem types like crashes, hangs, memory leaks, performance problems. Not all of it can be described in howto material. But raising the awareness on all parts of the support process, can significantly help to solve problems faster. I am in the troubleshooting business for over 25 years, read NSD before breakfast and wrote my own troubleshooting tools.

Important to know about Domino 14 Inbound Mail Disclaimer  

By Daniel Nashed | 1/12/24 2:40 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

A partner ask in OpenNTF channel about an issue with the new inbound mail disclaimer in Domino 14. It turned out I was able to reproduce the problem on one of my lab servers and this is critical to know if you enable disclaimers written into the body. There are two different ways to setup the feature. 1. Tag the subject line 2. Write HTML to the body of the message with fall back to subject for signed or encrypted messages. My personal recommendation is always to tag the subject, because modification of the body can lead to all kind of side effects with add-on software and also issues that could happen with incorrect formatted messages.

Running Traveler 14 on Windows 11 for testing  

By Daniel Nashed | 1/8/24 3:57 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The Windows desktop versions are completely untested and unsupported for all Domino related products. It is not recommended to run any Domino software in production on Windows desktop! There are still a couple of scenarios where Windows desktop can be relevant. For example a local notebook installation, a training environment or the Windows Sandbox -- which is also a Windows desktop environment. Domino 14.0 changed the installer to prompt to confirm the unsupported Windows version. But the installer continues to work with Windows 11 after you confirmed the message. As a German partner discovered last week, Traveler 14.0 doesn't install on Windows 11. InstallAnywhere runs into an error: Windows DLL failed to load during Installation. Windows 10 works unchanged. The issue starts with Windows 11.

Automatically Updating Ubuntu Linux including mail notification  

By Daniel Nashed | 1/8/24 3:51 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

This was on my list for a while. I looked into it, but the mail part wasn't what I wanted. Most admins will just install BSD mailx. But this had way too much dependencies for me. I just wrote a simple tool to replace mailx -> https://github.com/nashcom/nsh-tools/tree/main/nshmailx. But you could just continue to use the standard package. It wasn't just an option for me, because I try to not install other mail server components on a Domino server (like sendmail). The setup for automatic updates are pretty easy and base on scripting.

How to send mails native on Linux and MacOS from command-line  

By Daniel Nashed | 1/8/24 3:49 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

End of last year SMTP Smuggling was a hot topic. It turned out Domino is behaving well as you can read in Thomas Hampel's blog post who is also linking an interesting presentation about this topic. (See details here --> https://blog.thomashampel.com/blog/tomcat2000.nsf/dx/is-hcl-notes-domino-affected-by-smtp-smuggling.htm). Because I didn't find any good tool to craft emails required to check, I wrote a simple command-line tool in C. I used the OpenSSL versions of the network communication to abstract the calls from the standard socket operations. It's a nice show case how SMTP works under the covers -- which didn't change for centuries and works the original standards (RFC 821 and 822) are still what is behind all SMTP traffic. Once that worked, I thought adding STARTTLS would be a good addition. After I was done with my smuggling tests and I could sleep well again, I started thinking about to make it a more useful tool. So I added attachment support creating a MIME message with a Base64 encoded attachment. Here is a link to the proejct in my tools repository --> https://github.com/nashcom/nsh-tools/tree/main/nshmailx

Let’s Encrypt new default chain February 8, 2024  

By Daniel Nashed | 1/8/24 3:47 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Let's Encrypt new default chain February 8, 2024 The change has been announced mid last year and there is probably no action for you to take. It was about time for this move and it is well planned ahead of time. If you didn't change anything on Domino CertMgr side, there is very likely no action to take.

Leveraging Domino Autoupdate for company internal downloads  

By Daniel Nashed | 12/18/23 3:33 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Now that we have software automatically downloaded into autoupdate.nsf in Domino 14.0, the next step would be to distribute software inside your organization. The attachment data is stored in autoupdate.nsf. An agent could be used to find the software and redirect to the right URL. I wrote a simple search and redirect agent and a redirect rule. To download a file it can be just referenced by it's file name. The agent takes care of finding the document and generating the right redirect.

Domino 14.0 on Windows important to read before updating!  

By Daniel Nashed | 12/14/23 7:16 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino 14.0 shipped last week and I wanted to keep technical details for my DNUG Deep Dive presentation later this week. However the first customers contacted me with problems installing Domino 14.0 GA on Windows. There is one important change you should be aware of. The change is documented here --> https://help.hcltechsw.com/domino/14.0.0/admin/enabling_domino_nonadmin_user.html But not everyone might read the latest documentation before installing. If you keep the defaults, you will need to adjust your file-system permissions for translog and other external directories not located below the data directory. Else your server will not start!

Domino Download Bash Script leveraging My HCL Software Portal  

By Daniel Nashed | 11/27/23 12:58 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The My HCL Software Portal is still an early access offering in parallel to Domino 14 early access. It is planned to replace the Flexnet download soon and way easier and much faster to navigate. The website just works and has awesome performance. Domino 14 AutoUpdate leverages a new software download API to automatically download software into autoupdate.nsf. The download just needs a download token, which can be requested if you are log into https://my.hcltechsw.com/. I have been looking for a way to automatically download software for a couple of years. Now with the new portal and this new API it is possible to write a Bash script for full command-line operations including a simple to use menu. There are two different modes. By default the script uses My HCL Software navigation. But alternatively it can also leverage Domino 14 Auto Update software.jwt, which has more granular information and allows a more structured download package browsing experience.

Get your Linux environment ready for Domino V14  

By Daniel Nashed | 10/20/23 3:43 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino V14 is planned to ship end of this year. For Windows the system requirements don't really change, because of the universal run-time. But for Linux a newer compiler brings new OS dependencies. Specially the glibc version, which brings the base run-time support for C and also the C++ standard libs are important. An application build with a newer compiler on a newer Linux version does not run on older versions with lower glibc versions. glibc is the The GNU C Library - https://www.gnu.org/software/libc/ The new version required was released in August 2021 and is part of most current long term release Linux distributions.

Running Domino with SELinux on current REHL/CentOS Stream 9 & Co  

By Daniel Nashed | 10/9/23 2:56 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino 12.0.2 added support for SELinux in enforced mode, which is enabled by default by newer installations. SELinux is a lower level security feature, which can even limit processes running with root permissions. But the application needs to have a SELinux profile. I ran into this week on my own on a RHEL 9.2 machine and I got the same problem from a partner yesterday. It turns out that systemd can't read from /tmp any more. But the Domino service from my Nash!Com start script writes the domino process id into the /tmp folder. With SELinux enabled you get the following error message when looking into your service status (domino statusd). The start and stop operations of your server will also hang, because systemctl will hang. systemd[1]: domino.service: Can't convert PID files /tmp/domino.pid O_PATH file descriptor to proper file descriptor: Permission denied

Required Notes and Domino anti-virus file exclusions   

By Daniel Nashed | 10/9/23 2:54 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

This discussion came up in an OpenNTF Discord channel. The question was if this might be a good idea to keep OS level anti-virus enabled for Notes/Domino files. There is a clear statement from HCL about exclusions. But the technote doesn't explain why those exclusions are important. The exclusion might be different for each anti-virus production in detail. It also depends on customer IT policies how to exclude data. This can be either by path, extension or process. There are also recommendations from some anti-virus vendors stating the same exclusions for their specific product: Guidelines for excluding Notes and Domino directory and files when running an operating system Antivirus https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0093046 ible Ans

Domino CertMgr GitHub Repository with additional material  

By Daniel Nashed | 9/27/23 1:21 AM | Infrastructure - Notes / Domino | Added by Oliver Busse

Documentation is always a challenge. This is specially true when it comes to complex topics like SSL/TLS certificates. Many admins still use their old cook books to get certificates created. When HCL introduced CertMgr in Domino 12.0 the team asked for feedback in the early code drops. And the team is keeping asking in public and private forums since then. We really need your help to get it right. We need detailed feedback and questions. My new plan is to turn questions into FAQs and Howto documents in this GitHub repository.

Who moved my Domino keyfile.kyr files?  

By Daniel Nashed | 9/25/23 2:00 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino security in 2023 Domino 12.0 introduced a new, standards based and open way to work with web server certificates. Instead of using command-line tools like OpenSSL and the Domino kyrtool you can now manage all web server certificates in a domain wide certstore.nsf. The new functionality based on the well known text based PEM standard for certificates provides simplified flows and automation options for all type of certificates. Domino 12 also introduces the more modern ECDSA (sometimes referred as ECC) keys/certificates which are based on elliptic crypto which has dramatically less overhead. Moving from keyfile.kyr to certstore.nsf The legacy kyr files can be automatically imported into certstore.nsf with a single command-line operation (load certmgr -importkyr all).

Domino V14 backup for notes.ini  

By Daniel Nashed | 6/1/23 2:03 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino backup is around since 12.0 and it got improvements in every release. There are not many current AHA ideas for Domino Backup & Restore. One smaller feature you can see in EAP1 is the backup of the notes.ini.

How to use Domino OTS on Kubernetes to import an existing TLS Certificate  

By Daniel Nashed | 5/30/23 12:05 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino One Touch Setup has been designed with flexibility in mind, with special focus on getting a server up in a secure way. On Docker you can just mount PEM files into the container. On Kubernetes TLS Certificates and Keys are stored in secrets. Personally I am not a big fan of storing PEM files on disk. But you could at least set a password on the PEM file you import. Here is a basic example how to create a secret on K8s and reference it in OTS. Even the simple environment variable setup supports the security settings for CertMgr. Of course the same functionality is also available with the more flexible JSON based configuration.

Importing trusted MicroCA Roots for a Nomad Lab environment  

By Daniel Nashed | 5/29/23 12:39 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Yesterday I worked on a lab configuration based on Windows Sandbox, Domino and Nomad Web. The biggest challenge is to have a trusted certificate for Nomad Web. Nomad Server running with the Micro CA A Nomad Server can use Domino CertMgr Micro CA Certs. But the root is not trusted in your browser. I took a closer look and came up with a simple solution. which makes the import dramatically easier. No more searching for the right trust store and handling PEM files manually.

Get prepared for Notes/Domino V14 Early Access Code Drop 1  

By Daniel Nashed | 5/26/23 9:32 AM | Business - Events / People | Added by Oliver Busse

You can get hands on experience with Notes/Domino V14 end of this month. Here are some tips to get prepared. All of the software is only intended for non-production use! So you should prepare a VM to get started. But you should really take a look and have a try. Specially for business partners this is a call for action to test their applications with the updated back-end components. As announced earlier Notes/Domino moved to up to date compilers and a newer Java version. Also the client is 64bit only. I hope to see many of you in the EAP forum or at DNUG conference face to face.

Tuning Domino Servers for TLS sessions  

By Daniel Nashed | 5/24/23 2:13 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

My previous post was mainly about HTTP traffic and I mentioned TLS/SSL don't use the maximum number of connections settings, because they have a SSL/TLS session. Establishing a new TLS session has significant overhead! And you have to make sure in any application, that those sessions are cached and resumed. I revisited a blog post from 2012 where I explained a fix, which went into 8.5.3. And was enabled in 8.5.4 by default (which turned into the 9.0 release when shipped as far I recall). There was an issue with the session cache and a new cache had been implemented in 8.5.3. Today the new cache is the default and SSL_USE_ADDSESSION2=1 does not exist any more.