ZMF Tracing - Security Issues
Amongst the most common type of issues reported to Serena Support’s ZMF team are problems related to ChangeMan ZMF security profiles. It is reasonably common for us to discuss “Dave being unable to see or select the admin option” or “Heidi not being allowed to create a package in application TOGA”.
So, how do we go about researching such a problem and can customers perform any self-diagnosis of such issues? Obviously the answer to the latter is ‘yes’ or this would be a very short blog entry. So, let’s use an example and work through the diagnosis process.
Our sample situation is that user CMNSUP1 on our ZMF 8.1 subsystem id 4 cannot access the admin option. When they logon to the ZMF ISPF client they see the unavailable ‘A Admin’ option but when they attempt to access it they receive the standard error messages:
However, our Security Administrator is also telling us that they have granted access to the relevant security profiles. Again, for the sake of demonstration let’s say CMN4GBAD and CMN4LCADM. So, where do we go from here?
These are the steps required to commence investigation into this issue.
1. Disconnect from the ZMF system
The first thing we need to do is to have CMNSUP1 exit the ZMF ISPF interface. Many of the security profiles are checked at initial connection time. That can be when the user connects to the ZMF instance in ISPF, when the job executing CMNBATCH or an XML Service logs on or executes the first transaction or request, etc.
2. Enable started task tracing
Once you are as close to reproducing the problem as possible (i.e. a press of the 'enter' key or the submission of a batch job will recreate the problem) the next thing is to enable all started task trace mechanisms using the following MVS modify commands (obviously substitute SERSUPI4 with your started task's name and CMNSUP1 with your problem userid):
Strictly speaking, not all of these trace commands are actually required in order to produce the diagnostics required to investigate this particular example. However, rather than having to repeat this process several times should there be something particularly complex occurring, it could save time and effort in the long run to simply enable them all at this point.
3. Connect to the ZMF system
At this point simply ask user CMNSUP1 to reconnect to the ZMF ISPF client using your standard connection CLIST or rexx. The information that we require will have been generated in the started task’s SERPRINT output file as soon as connection is complete and CMNSUP1 can see the ZMF Primary Options Menu.
4. Disable started task tracing
Many of us are familiar with started task trace data but if you are not be warned: it can generate a significant amount of output! So, to avoid irate Operations staff contacting you asking why they are experiencing spool shortages related to your ZMF subsystem, it is best to disable the traces before doing anything else. This can be achieved by replacing ‘ON’ with ‘OFF’ in all of the previous trace commands:
5. Assess the trace data
Now that the trace data has been generated we can take a look at what it is telling us. Let’s start by going to the started task's SERPRINT output file and issuing a “find authrsrc” command:
This will bring us to the following trace data:
So, what exactly is this information telling us? Let’s take a few of the trace messages and see.
a) Unauthorized access (RC=8)
2015/05/15 06:03:38.83 SER9100T CMNSUP1 AuthRsrc: RC=8 Entity=CMN4GBAD Attr=UPDATE Class=£CMNSUP
This message is telling us that ZMF asked the System Authorization Facility (SAF) whether userid CMNSUP1 possesses UPDATE access to the profile named CMN4GBAD in security class £CMNSUP. The RC=8 indicates that SAF explicitly answered ‘no’. CMNSUP1 does NOT possess UPDATE access to £CMNSUP/CMN4GBAD so any processing associated with that security profile should be made unavailable to this userid.
b) Authorized access (RC=0)
2015/05/15 06:03:38.85 SER9100T CMNSUP1 AuthRsrc: RC=0 Entity=CMNREVRT Attr=UPDATE Class=£CMNSUP
At the opposite end of the scale, this RC=0 message tells us that SAF knows that CMNSUP1 does have access to profile CMNREVRT within security class £CMNSUP and that any processing associated with that profile should be made available to user CMNSUP1.
c) Non-existent or missing profile (RC=4)
2015/05/15 06:03:38.83 SER9100T CMNSUP1 AuthRsrc: RC=4 Entity=CMN4MON Attr=UPDATE Class=£CMNSUP
A RC=4 will always be interpreted as a missing or non-existent profile. So this tells us that no profile named CMN4MON exists within the £CMNSUP security class.
PLEASE NOTE, however, that there can be complications linked to non-existent security profiles:
i) in non-RACF environments (e.g. CA-ACF2 and CA-TSS) the default behaviour can be set to automatically deny access to a profile if it does not exist. This will result in a RACROUTE REQUEST=AUTH call issuing a return code 8.
ii) even in a RACF environment, when defining the CMN ZMF RACF Class Descriptor Table (CDT) entry, the DEFAULTRC value in CDTINFO can be explicitly set or overridden to return a value other than 4 for a security profile not found situation.
So, in both of these situations, and possibly others that we have yet to encounter, a non-existent profile may be interpreted as an unauthorized access situation.
d) Standard security profile processing
2015/05/15 06:03:38.83 SER9100T CMNSUP1 AuthRsrc: RC=8 Entity=CMN4LCAD Attr=UPDATE Class=£CMNSUP
2015/05/15 06:03:38.84 SER9100T CMNSUP1 AuthRsrc: RC=4 Entity=CMN4BKOU Attr=UPDATE Class=£CMNSUP
2015/05/15 06:03:38.84 SER9100T CMNSUP1 AuthRsrc: RC=0 Entity=CMNBKOUT Attr=UPDATE Class=£CMNSUP
As documented in the Installation Guide, some ‘special’ profiles can be overridden for specific subsystem IDs. The subsystem ID can be added in position 4 and this profile will be checked first. If not found, processing continues to check the ‘generic’, catch-all profile access.
So, in these messages we can see that a CMN4LCAD profile does appear to exist in class £CMNSUP and that CMNSUP1 does not have UPDATE access to it. Therefore CMNSUP1 cannot access application administration options.
However, a CMN4BKOU profile does not exist in the same class so we fall through to check access to the catch-all CMNBKOUT profile to decide if this user can execute backout processing on this subsystem. CMNSUP1 does have UPDATE access to CMNBKOUT so they will be able to select these options.
Again, please note that the exceptions noted in c) can be very relevant here, too.
With this trace data in hand, we can return to the Security Administrator and show them that CMNSUP1 is being explicitly denied access to the CMN4GBAD and CMN4LCAD profiles in class £CMNSUP (ERO users should also bear in mind the impact of the CMNxRL* and CMNRL* profiles for this specific example). This is why CMNSUP1 cannot access the Admin option from the ZMF Primary Options Menu. If the Security Administrator wishes to see the exact format of the RACROUTE security calls they can do so in the distributed SERCOMC ASMSRC library, member SERLCSEC. We distribute the source for this component to allow minor changes to processing and to provide absolute transparency in this area.
All of this evidence should confirm that there has been a problem granting access.
Although this document concentrates on connection profiles, exactly the same process can be applied to any security-related problem and to any version of ZMF. Basically, you need to get as close to recreating the problem as possible, enable the documented traces, find all occurrences of ‘authrsrc’ (they may not always be grouped together) and interpret the information in the manner documented here.
And if you need any help setting up the test or interpreting the results, go ahead and open a case with Support who will be happy to assist. Just one small request here: if sending us trace data please ensure that any downloaded output files have a record length of at least 161 to avoid truncating the responses. Thanks.