UPDATE! The community will be go into read-only on April 19, 8am Pacific in preparation for migration on April 21. Read more.
UPDATE! The community will be go into read-only on April 19, 8am Pacific in preparation for migration on April 21.Read more.
Micro Focus Expert
Micro Focus Expert
947 views

Windows may lose ZCM CA Trust after WIndows 1909/2004 Upgrade (Potential MS Defect Alert)

This post is a lot of conjecture, but I have received a report that there is a new Windows 10 1909 (or 2004) OS upgrade issue that may impact ZCM.

In short, it appears that if a Windows 10 PC with the October 2020 Rollup Patches (or other very recent patches) Upgrades to 1909 or 2004 via an IPU (In-Place Upgrade), then all of the previous certificate trusts are lost.  Patching the Upgrade Media prevents the issue.   Again, this is not a ZCM specific issue but simply impacts ZCM since it relies upon Certificate trusts.

While the resolution appears to be updating the upgrade media, I did provide some alternate solutions as well to attempt to mitigate the issue while root cause was confirmed.

  • Deploy the ZCM Internal CA via Domain GPOs.  The advantage to this solution is that any currently broken devices, even unknowingly broken devices, would most likely self-heal once the updated GPO is applied.
  • Create a Force Run Bundle that has a SYSTEM Requirement for HKLM\Software\Microsoft\SystemCertificates\ROOT\Certificates\{RANDOM ID#} that matches your ZCM CA to be missing.  (See Attached Picture)
    • If this value does not exist, it will apply the REG BLOB to restore the lost cert.
    • This bundle should self-heal any device where Cert Trust is lost at a point in time AFTER the bundle is deployed.  The bundle will still operate after trust is lost because the device will be operating in Disconnected mode which can still run previously assigned bundles.
    • This will NOT remediate any currently broken devices, since they cannot get the assignment.
  • Update the Windows 1909 Upgrade Files to restore missing cert trusts during the post-upgrade tasks.
    • Export HKLM\Software\Microsoft\SystemCertificates\ROOT\Certificates\ locally prior to the upgrade and Import local export of HKLM\Software\Microsoft\SystemCertificates\ROOT\Certificates\ post upgrade.
      • This will help ensure ALL ROOT certs are retained since Non-ZCM certs can be impacted.
    • Create a REG Export of HKLM\Software\Microsoft\ROOT\Certificates\{RANDOM ID#} and import this as part of the post upgrade tasks.
    • Run ‘certutil.exe -addstore Root “\\patch\ca.der” ‘ post upgrade.  This would require a copy of the DER to be local or UNC accessible to restore the trust.

At this point, this post is based on significant supposition but the potential impact of the reported certificate issues around upgrading Windows 10 PCs with the October rollup were significant enough I wanted to alert folks quickly.

--
Please give a hearty thumbs up to any post you find helpful!
To find articles by Craig Wilson simply follow the link: Craig Wilson's Tips!
9 Replies
Micro Focus Expert
Micro Focus Expert

Note: Here are some steps for slipstreaming Windows updates into your Windows 10 upgrade media.

https://www.winhelponline.com/blog/slipstream-windows-10-integrate-updates-setup-media-iso/

--
Please give a hearty thumbs up to any post you find helpful!
To find articles by Craig Wilson simply follow the link: Craig Wilson's Tips!
Micro Focus Expert
Micro Focus Expert

0 Likes
Cadet 2nd Class Cadet 2nd Class
Cadet 2nd Class

Craig, thank you so much for continuing to post helpful and useful information like this that impacts us customers. We ran into this issue today during our Feature Upgrade 1909>2004 testing and were at a complete loss initially in determining where to start with troubleshooting and what was failing. I'd love to see this in a KB somewhere because the Forums ended up being my last stop after checking all other resources, fortunately it was worthwhile. 🙂

Thanks again,
Luke

0 Likes
Vice Admiral
Vice Admiral

Hi Craig,

 

I have not been able to find this registry key (HKLM\Software\Microsoft\ROOT\Certificates\)

It does not exists on our 1809 Windows and also on our 2004 Windows.

 

Is there something new here to prevent Zenworks from breaking down after the update from 1809 to 2004 ?

We are still stuck with out older versions because of this problem.

 

Let me know! 

 

I can open a SR if preferable.

0 Likes
Micro Focus Expert
Micro Focus Expert

Sorry about that....I was missing the full path...It should have read "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\ROOT".

I've updated the original post.  This issue of course impacts are more than ZCM, it impacts any and all trusted root certificates.  Quite a major Windows bug.

That being said, when upgrading, you can re-add all required certs as part of the post-upgrade tasks defined in your WIn10 upgrade. 

You can even add the ZCM Certificate to your GPOs.  Either Domain or ZCM Delivered.  This way if lost, they get re-added.  ZCM delivered GPOs would continue to work due to their ability to function in cached mode.

 

--
Please give a hearty thumbs up to any post you find helpful!
To find articles by Craig Wilson simply follow the link: Craig Wilson's Tips!
0 Likes
Vice Admiral
Vice Admiral

You think it might work even if the zenworks agent is broken because of his missing certificate ?

 

(The GPO that add the certificates within Zenworks)

If not sure, we will go ahead with the reg export/imports tasks 

0 Likes
Micro Focus Expert
Micro Focus Expert

So long as the GPO is applied prior.....

Simple test...Apply GPO that delivers certificates via ZCM Device GPO.  Verify they are delivered.  Disconnect from the network.  Reboot and Logon.  See if the certificates are back.

The ZCM agent remains fully functional even when off the network and will continue to do as it was told prior to disconnection.  The disconnected mode works so well, I've had customers call and complain that some new changes are not taking but everything old still works.  They were totally unaware their server was non-functional because everything was working cached so well.

--
Please give a hearty thumbs up to any post you find helpful!
To find articles by Craig Wilson simply follow the link: Craig Wilson's Tips!
0 Likes
Micro Focus Expert
Micro Focus Expert

Bundles work as well after the fact in disconnected mode, and can restore the cert if missing.  System Requirement for it to be missing can always add it back.

--
Please give a hearty thumbs up to any post you find helpful!
To find articles by Craig Wilson simply follow the link: Craig Wilson's Tips!
0 Likes
Micro Focus Expert
Micro Focus Expert

And of course, the SIMPLEST answer is to make sure you use a version of Windows 10 Media from this year to do the upgrade.  Any version either on or after last November should have the fix, but just go with the latest to be safe.  MS should have updated versions of most flavors on the download site.

 

--
Please give a hearty thumbs up to any post you find helpful!
To find articles by Craig Wilson simply follow the link: Craig Wilson's Tips!
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.