CUSTOMER PROFILE

This HCL defect bug affected countless organizations worldwide, including several Origina customers in the insurance, banking, utility, and education fields. 

  • INDUSTRY:

    Multiple Industries

  • COMPANY SIZE:

    Various

  • LOCATION:

    Various

PRODUCTS
  • All Versions of IBM/HCL Domino

“This solution provided a mail rules agent to fix the router issue related to mail rules along with a mailbox checker agent to resolve the mail routing issue caused by the HCL Domino product defect.” 

THE PROBLEM

HCL Domino bug lingering for 30 years leaves countless customers at a loss 

It was a truly unlucky Friday the 13th in December 2024 for a number of companies as HCL issued a critical alert regarding mail not routing after Domino restarts in their internal email systems. It seems that something sinister had been lingering beneath the code of their Domino applications and databases for some time.  

For context, HCL Domino, and more specifically its client application HCL Notes, provides business collaboration functions, such as email, calendars, to-do lists, contact management, discussion forums, file sharing, websites, instant messaging, blogs, document libraries, user directories, and custom applications.  

HCL issued a statement that read as follows:  

“The HCL Domino Development team has identified an issue which will affect *ALL Domino server versions* as of December 13, 2024. 

If you restart your server, a router error will occur which results in delivery failures due to a routing loop. Also mail rules will also start failing. 

In addition, Domino will use only a single mail.box regardless of what the configuration document calls for. 

This is a date/time issue in our code, the area where the defect was found is used to determine if the configuration needs to be updated. Since 13 December this check fails and so the router will not load your configuration.” 

This resulted in an avoidable error that the users would have to pay for. 

THE SOLUTION

INDEPENDENT SUPPORT STEPS UP TO SOLVE THE PROBLEM

HCL’s official stance on the Domino bug was that any businesses affected would have to download their fix pack to address the issue and even went so far as to say that  ‘”No TPSS [third-party software support] can fix the issue or mitigate with a work-around.”  

We here at Origina took this claim as a challenge and quickly got to work on a fix for all our Domino customers who were affected. 

We compiled a list of Origina’s current Domino customers and their contacts who have access to our self-service portal (SSP), and a communication was sent advising them not to restart servers or router tasks and, if impacted, to open a support ticket with Origina. Our Domino global independent experts (GIEs) immediately began work on understanding the issue and investigating any potential work-arounds in advance of tickets being logged.  

The issue appeared to only impact Notes Remote Procedure Calls (NRPC) mail routing.  Additionally, it seemed to only affect servers that were routing to another server via an intermediary server.  

If the customer environment was impacted, initial options considered were to either create direct connections between all mail servers or to update the configuration so that all mail routing would be updated to Simple Message Transport Protocol (SMTP). 

Since those two configuration changes would be too significant of a change to a customer’s existing environment, Origina product experts opted for a work-around that implemented changes to hold undeliverable messages within the mail.box.  

This meant a new agent within the mail checker database could monitor the contents of the mail.box and update the hop count of held messages then correctly route them. This solution provided a mail rules agent to fix the router issue related to mail rules along with a mailbox checker agent to resolve the mail routing issue caused by the HCL Domino product defect. 

 

Origina illustrates the power of independent support 

Origina product experts assisted customers with several different variations of the HCL Domino issue. Here are a few examples. 

  • Issue: Message routing in the mail.box is not processed by the mail rules established on a server level.  
  • Solution: Origina provided a script in an agent trigger before email arrives, which processes the message and applies the message. 
  • Issue: Message routing in the mail files is not processed by the mail rules established on a mail file level (user level). 
  • Solution: Origina created a script in an agent trigger before email arrives, which processes the message and applies the message. 
  • Issue: Message routing in the mail in database is not processed by the mail rules established on a mail file level with a third-party mail rule database configuration. 
  • Solution: Origina developed a script in an agent trigger before email arrives, which processes the message and applies the message. 

Applying these solutions, the issue was fixed by Origina via a work-around, and emails were again being successfully sent and received by the businesses. 

By HCL’s own admission, this problem was a date/time issue in the IBM source code dating back 30 years. This led to HCL’s Global Director of Strategy Uffe Sorensen to comment that “no provider of third-party software support (TPSS) would be able to fix the issue or mitigate it with a work-around.”  

Origina’s GIEs, utilizing their deep knowledge, successfully mitigated the issue without access to the source code and proved HCL wrong.  

This was a huge win for independent software maintenance and support providers and proof that Origina supports our customers above all else, no matter the issue. 

Origina’s GIEs, utilizing their deep knowledge, successfully mitigated the issue without access to the source code and proved HCL wrong.” 

THE RESULT

Origina delivered: 

  • An independent solution designed to address the HCL Domino bug without altering the source code that worked with all releases of Domino 
  • A fix that did not require a time-consuming Domino update or mandate upgrading to costly OEM support 
  • Issue mitigation for customers who were on end-of-support (EOS) versions of the software or who had let OEM support lapse