Root Certificate Authorities Distrusted

This message is intended for U-M IT staff who are responsible for managing certificates for university websites or systems.

Summary

A root certificate provided by the InCommon Certificate Service, which is used by the university for website security, was distrusted by multiple vendors and package maintainers on Oct. 19 and 20. The result is that some university websites and applications are, or will become, unavailable to visitors.  

For websites and systems that are not behind Cloudflare and currently have the distrusted certificate with an ECC key, administrators need to update to a new certificate that uses a different root, as detailed in the Action Items below.

Problem

The root certificate "emSign Root TLS CA - G3" has been distrusted by major vendors due to weak private keys used by intermediate certificates. Some sites are affected now, and some will be affected over the coming days/weeks as browsers and other software get updated to newer versions.  

Affected Systems

Websites with certificates signed by the following intermediate Certificate Authorities will need to obtain new certificates.  These are certificates obtained from CERTInext that use ECC keys.

  • InCommon Intermediate CA - OVG3C
  • InCommon Intermediate CA - DVG3C
  • InCommon Intermediate CA - EVG3C (not in use at U-M)

InCommon certificates issued through Sectigo are not affected.

InCommon certificates issued through CERTInext that use RSA keys are not currently affected.  There is a chance that RSA certificates from CERTInext may be affected in the future.  ITS is working with InCommon to get more information.

Code Signing certificates are not affected.

Detection

Look at the issuer of the leaf certificate (the one installed on your server) and determine if it is affected: 

  • InCommon Intermediate CA - OVG3C
  • InCommon Intermediate CA - DVG3C

IMPORTANT: You should check at each system between the client and the server, including at any load balancers, proxies, and security appliances.  The certificate seen by the client may be OK, but the system could still break if an intermediary or back-end device uses an affected certificate.

Action Items

  • If your website is using a CERTInext certificate with an ECC key and is not behind Cloudflare, do one of the following to avoid visitors getting errors:
    • Option 1: Obtain a new certificate through Let's Encrypt (using certbot on Linux or simple-acme on Windows).
    • Option 2: Generate a new key using RSA, and obtain a new CERTInext certificate. Note: There is a chance this solution may not be permanent (see below).
    • Option 3: Put your website behind Cloudflare as a temporary workaround. You can reduce the work required to move your website behind Cloudflare by disabling Cloudflare caching.
  • Be prepared to possibly replace CERTInext certificates that use RSA keys.  There is a chance that the root certificate used to sign CERTInext certificates that use RSA keys could become distrusted, too.
    • ITS is working with InCommon to determine whether this will happen and, if so, the timeline and alternative Certificate Authorities to use.  Right now, the main alternative is Let's Encrypt.
  • Advise users not to click through a warning if they receive one.

Technical Details

Tuesday October 20

  • 12:20 p.m. Cloudflare distrusts at least the eMudhra/emSign G3 root certificate, causing visitors to websites to get an HTTP 526 error page.
  • End users with updated Chrome or Firefox browsers start to see certificate validation warnings
  • 1:20 p.m. The university puts a workaround into place to restore access to sites behind Cloudflare (temporarily change the SSL mode for each zone from "Full (Strict)" to "Full").

Information for Users

Users should be advised not to click through a warning if they receive one, and to report the error if they encounter it.

Questions, Concerns, Reports

Please contact ITS Information Assurance through the ITS Service Center.