How to troubleshoot patches not installing

I have a patch policy set to include critical patches.  I can view a PC > patches page and see patches listed.  They have been assigned and they have been downloaded.  Patch policy is set to run on device shutdown.  If I restart the PC, I get the message about applying the policy as it shuts down.  But after it restarts, the patches are still listed.  How do I figure out what is going on?

  • Some General Questions...

    • Are the Patches NOW installed regardless of what the ZCC Shows?
    • Did the Patches Try and Install by FAIL?
    • Did the Patches NOT TRY to install?

     

    Logs can always help...

    https://support.microfocus.com/kb/doc.php?id=3418069

    See Component Name: Patch Management:

    Most importantly, Detection results logs and deployment results log files are written to %ZENWORKS_HOME%\ zpm as well as Windows Event Viewer.

    Also don't forget you need to run another "Patch Scan" for the results to get uploaded.

    Keep in mind that "Shutdown" related events may always be tricky.  So while troubleshooting I would also try and isolate "Patch not working" from "Patch not working During Shutdown".  There are quite a few GPO hoops that ZCM tries to set to allow this to work.  If it works outside of Shutdown, but not during shutdown, then it's may become GPO troubleshooting.

  • Reminder: "zac ps"  will manually run a "Patch Scan" on a device to scan for needed patches.  "zac pap" will manually apply the patch policy to apply and patches the scan discovered were needed that are included in the policy.  (PAP = Patch Apply Policy)

  • I'm getting a mixed bag of results...

    One PC just installed patches at shutdown, restarted, and the console said it was 100% compliant.  First PC where this actually worked.

    Another PC refused to update.  I manually told it to install the 1909 feature update (it was on 1903).  After it restarted, I ran "zac ps" and it was still out of date.  I ran "zac ps" as administrator and the console now says it is 100% compliant.

    A third PC in Settings > Windows Update states that it is up to date (but that may just be based on an out of date windows scan).  ZCC console says it is missing 2020-11 cumulative and 2020-11 service stack updates.  "zac ps" run as administrator has no affect on this.  And according to the patch install history on the PC, those two have not been installed.  So far not finding any errors as to why those two patches won't install.  I ran "zac pap" and it claimed one or more patches were installed and it ran a patch scan, but nothing changed in the console.  I ran "zac ps" again and still no change in the console.  %ZENWORKS_HOME%\zpm contained two ".plp_results.txt" files that matched up with the missing patches.  All they contain is the word "SUCCESS".  But the console still does not acknowledge that the PC is patched.

  • Keep in mind "zac ps" will not apply/install any patches to a device.  It simply tells a device to scan for needed patches.  "zac pap" is what applies the needed patches discovered by "zac ps".

    You may need to look deeper into the logs for those specific patches in the ZPM folder.  For example, the logs for those patches may contain an error or note that a pre-req is required before they can install.  That pre-req may not be included in your list of patches to apply.  Also sometimes it takes a number of cycles to install all patches.  PatchC may require PatchB which may require PatchA.   And it's possible that PatchA and PatchB require a reboot before the next is allowed to apply.

    The "Majority" if ZPM troubleshooting it looking at the actual log files from the patches themselves.  

    When KB1234.exe tries to install, it will generate logs to be placed in the ZPM folder.  There are native logs from the native patch that will give a high level of insight into why it failed to install.  Any purely ZCM log would likely contain minimal info such as "error 1603"....the most useless error code an admin will ever see.  But the KB1234.log will give details on why a 1603 took place.

  • Going to do some more testing....but I just had one PC where I did the following:

    run as admin "zac ps"
    run as admin "zac pap"
    reboot
    run as admin "zac ps"

    And then it was compliant.

    The install at shutdown is much preferred here because it should not interfere with the users.  But I may need to change to a midnight schedule instead and apply the policy and then reboot.

  • Personally, I would much prefer an overnight job.  WOL and let them run.  While some folks like Shutdown....Drives me nuts when I go to do a quick reboot.....then an hour later I'm up again

    That being said if Shutdown is giving you an issue, dont be shy about Opening SRs.  Support around that process is relatively new so its possible dev could make it more reliable.

  • I scheduled patches to apply at 1 AM last night.  I think 1 or 2 of the 60 computers actually applied patches.  Not sure if this was a Wake-On-Lan issue or if they hadn't refreshed and picked up the policy.  I added the next few days to the schedule, so hopefully over the weekend they will start to patch and get compliant.  If it is a waking up issue, I don't see how to tell the PC to wake up as part of the patch process.  Am I missing something?

  • There are not WOL events built into ZPM itself.  You can publish a WOL bundle that runs at perhaps 15-30 minutes before your ZPM is schedule to apply the patches. 

    By Default on NEWer Zones ZPM updates at 2AM, so you may want to schedule applying patches as 4AM to let stuff settle down.  Older Zones Defaulted to MIDNIGHT, but there are alot of other things going on then so they moved it to 2AM.

    For Customers who have SATS and WANS, they may need more time to replicate or have replication concerns to consider.  I do not believe that applies in your case.

  •   I'm probably missing the obvious...  but how do you create a WOL bundle?  I'm not seeing that option.

    EDIT:  Ok, I created an empty bundle, chose distribution schedule, set it to daily at the needed times, and checked the WOL option.  Assuming that is all that is needed.  I will see how it looks after tonight.

  • Here is a WOL Troubleshooting Guide...

    https://support.microfocus.com/kb/doc.php?id=7004716

    In short, you will want to look in LOADER to see if the server SENT the WOL packet.  If it DID and the PC did not wake up, then there is either a PC or SWITCH configuration issue.  ZCM will sent the WOL packet to "x.x.x.255" which is the PCs specific subnet vs to 255.255.255.255 which is to ever network port in your enterprise...albeit yours is small.  Switches need to be configured to allow for the directed subnet broadcast separately from the global broadcast of 255.255.255.0.   (This is why some folks are confused why a simple WOL tool works, but ZCM fails because many tools just spam the enterprise instead of targeting and the switch allows one but not the other.)

    I often send a WOL job to a turned on PC running wireshark where WOL is not working.

    • If Wireshark SEES the WOL packet, then I know the it's a PC WOL config issue
    • If Wireshark DOES NOT see the WOL packet, I check loader-messages to see if it was SENT.
      • If the packet is shown as sent correct in Loader, then it's a switch issue
      • If the packet does not show as sent or has wrong details, then it is a ZCM issue. (This is RARE.)