Information researched, written and provided by Dan Christensen - Senior Software Engineer
Acronyms
SDD: Software Distribution Directory
C/S: Client/Server protocol used by GroupWise client to talk to POA usually over port 1677.
Motivation
Software distribution directories are remnants of times past when the primary method of network access was via shared volumes on a file server. In this world it made sense to have a single repository (or central repositories per geographic location) for software to be installed and referenced by users, admins and components of the GroupWise system.
Today, the world has faster network access and more local hard drive space. In this world it is much more common to download files directly to a local drive from a web server rather than access files from a shared network drive.
The following are the reasons engineering would like to eliminate the use of software distribution directories from GroupWise.
Unnecessary complexity when installing a new system. Before installing a new GroupWise system it requires the administrator to create a new software area and copy all of the install files to that area. The administrator also needs to ensure that the area that they installed the files to is shared on the network for certain admin and client features to work properly. Why is the administrator required to copy the setup files somewhere else when the files were already downloaded locally to run the install in the first place?
Unnecessary complexity when updating an existing system. After installing a new version of the GroupWise agent and client software, an administrator may try to login with the new client and be greeted with an error message that the views are not installed properly for this version of the client. In order for a new client to access a given post office, the administrator must update the files in the software distribution directory and then update the view files on the post office as well. If certain files are required in order for the software to run properly why weren't they installed when the software was updated?
Difficult to manage with a web based admin console. Updating a network software distribution directory with files downloaded on the admins local hard drive via a web browser is unfriendly and difficult. There are hundreds of megabytes that are included in a GroupWise installation and uploading these via a browser is not feasible. To even support backend file copies, the admin would have to setup special permissions on the network share to allow writes from the computer running the admin service while granting only read-only access to everyone else. Of course, this assumes that the software distribution directory is network accessible at all. On Linux, additional configuration is required to make a SDD consumable both by the agents and as a network share.
No longer used as a central installation repository. Engineering assumes and believes that the SDD are used as a place for users or administrators to go to install software. Users will generally download software from a web browser rather than a network share. Admins will download from a web browser and/or use utilities like SCP to copy software around to servers where they need to be installed. In Linux environments, accessing software from a UNC path is possible but generally more trouble than it is worth.
No longer an integral part of the GroupWise architecture. There are not very many GroupWise components that actually consume the software distribution directory. In fact, the primary remaining consumer is the windows client... and it only uses it in a limited fashion. All of the other files stored in the software distribution directory are largely wasted space.
Current Usage
The software distribution directory is consumed by three components/features in GroupWise:
Admin - Admin references the *.dc (and other supporting) files when creating a new system, domain, post office or gateway.
Client Views - Post offices update their local cache of view files by copying them from the software distribution directory.
Client Update - The client auto-update process currently references the software distribution directory to retrieve new installation components. This is by far the most complicated usage of the software distribution directory.
Admin
For Admin, engineering will make the following change. Install the *.dc files (and all other supporting files) alongside the admin components. This allows an admin install to be self-sufficient and include all of the files that it needs to perform its job. This choice has schedule/effort benefits.
Client Views
For Client views, engineering will install the client view files as part of the agent install. This will create an "ofviews" directory relative to the installation path of the agents (i.e. /opt/novell/groupwise/agents/ofviews). The engine code will need to be modified to retrieve view files from the relative path instead of the ofviews directory that is relative to the post office. Or, alternatively the engine could be modified to composite the two directory locations (i.e. try to retrieve from the post office location and fall back to the relative location). This would account for scenarios where the views had been customized by the administrator on the post office. Question: Are you using custom views?
Client Update
This is by far the most complicated use of the software distribution directory and as such requires a little bit of background on how the process currently works.
Key Files Involved
SDD\client\update32.dll - Performs update of client by UNC access to the SDD
SDD\client\gwipupdt.dll - Performs update of client by Client/Server access to SDD in conjunction with setupip.exe
SDD\client\setup.cfg - INI file containing a section with auto-update settings
SDD\client\software.inf - INF file containing the current version of the software
SDD\client\win32\* - Contains all of the client installation files required to launch an installation over a UNC path.
SDD\client\setupip.exe - SetupIP installation stub launches installation and downloads components from pre-configured website URL.
How it works now
The client update process begins with the administrator updating the software distribution directory with the new client files. Most importantly this includes the client\software.inf file which contains the build number of the software installed to the SDD. The client will use this build number to determine if the software in the SDD is newer than what it has installed.
The administrator must increment the "bump" number on the software area record in the domain database. This is done from ConsoleOne in the System\Software Directory Management by clicking on the "Update" button.
Every time that the client logs in, the POA gets the GUID of the software area associated with the post office that the user belongs to. The GUID is used to lookup the software area record from which the UNC path and bump number are retrieved. Both of these fields are returned by the server to the client.
The client compares the bump number to a version in the windows registry. This registry contains the last number that the client processed to trigger an auto-upate check. If the bump number is newer than the number in the registry then the client will begin its auto-update check. Note that this only triggers a check for newer software and will only trigger an actual install if the software in the SDD is newer than what is installed locally.
The client will take the UNC path returned from the server and attempt to load "update32.dll" from that path. This requires the client to have direct network access to the server hosting the SDD shared directory. If the client is able to access and load this DLL then it calls into the DLL to perform and update check and possibly and update:
The routine in update32.dll will attempt to load the "setup.cfg" and "software.inf" directly from the UNC path. Note that on versions of windows newer than Windows XP that Windows prohibits loading INI/INF files from a network drive so the files must first be copied to a local temp directory before they are accessed.
The build number of the software in the SDD is read from the software.inf and compared to the build number that was written to the registry when the client was installed. If the SDD software is newer, the update check proceeds, otherwise it returns without taking any further action.
The auto-update settings are read from the setup.cfg to determine if the system is configured for auto-update, whether it is forced, and whether or not the user is allowed grace logins. Depending on the auto-update parameters the user may be prompted to update or an update may be forced.
Finally, setup.exe is launched via UNC from the SDD\client\win32 directory. Note that on most versions of windows this will at best trigger a security warning, at worst will fail to launch due to settings regarding launching executable from a network drive.
If the client fails to load and process "update32.dll", then it moves on to a second attempt to update over TCP/IP. The client will use the C/S protocol to retrieve the file "gwipupdt.dll" from the SDD into a temporary folder. Once downloaded the client loads the gwipupdt.dll and calls a function in the DLL to continue the update process.
The update function uses a callback into the hosting client process to download additional files from the software distribution directory. These files are software.inf and setup.cfg.
The same processing occurs as described in the UNC update path to process the software.inf and setup.cfg to determine if an update needs to occur.
If an update needs to occur, then gwipupdt.dll uses the C/S protocol to download the setupip.exe stub from the SDD. Note that the admin must have generated the setupip.exe and placed it into the SDD in a separate step for this to work.
Once launched setupip.exe will attach to one of five web servers specified by the administrator to download the client installation packages and start the install.
SetupIP troubles
Novell does not actually ship the file "setupip.exe". This is generated by an administrator at the customer site by running the WriteIP.exe utility. The write ip utility allows the administrator to define up to five URL's where the client installation can be downloaded. It takes these addresses and writes modifies a template executable to generate the actual setupip.exe. Not only is this an additional step for the administrator but the process itself is problematic. CopyIP.exe and SetupIp.exe are frequently flagged as viruses by many anti-virus products. This is because they are:
Writing to an executable file.
Running code that connects to internet, downloads executable files and then runs them.
Incapable of being added to an exclusion list because the dynamic nature of an auto-generated executable provides a constantly change hash or signature.
Incapable of being signed by Novell since the executable is modified after it ships.
Engineering will make all of the following changes.
Change #1: Remove UNC Auto-Update
In order to eliminate the dependence on the software distribution directory, engineering must eliminate the portion of the auto-update process that attempts to access the SDD directly. This means getting rid of "update32.dll". Since the auto-update process automatically falls back to the gwipupdt.dll, the update32.dll portion of the auto-update is entirely redundant. Also, since the update32 portion of the update counts on direct access to a shared network drive and the ability to run software from the network share which is the less reliable of the two processes.
Change #2: Install Auto-Update Files Relative to Agent
Move the necessary auto-update files out of the software distribution directory and into a path relative to the agent (i.e. /opt/novell/groupwise/agents/client-update). One example, the administrator would only need to lay down these files if they were interested in supporting the auto-update functionality of the client. That would imply that these would be a separate *.rpm or *.msi file.
Minimally, this file set would include:
gwipupdt.dll (and supporting language DLL's)
software.inf
setup.cfg (more about this later)
setupip.exe and/or setupip.* (more about this later)
Note that this file set no longer includes the uncompressed client/win32/* set of files that comprise the normal client install image. This means that much less total space is required to support auto-update.
In order to support loading files out of this directory the engine code would have to be modified to look in this directory for software files instead of the software distribution directory.
Change #3: Move setup.cfg Auto-Update settings into Admin Console
In order to prevent the administrator from having to edit the auto-update settings in the setup.cfg that is installed with each agent (as per Proposal #2), The setup.cfg file will be eliminated altogether and the auto-update settings will move into the domain database where they can be modified through the admin console.
The auto-update settings include the following 4 settings:
Enabled: Yes|No
ForceUpdate: Yes|No
GraceLoginCount: 0-n
PromptUntilUpdated: Yes|No
These settings will be added into the domain database and applied to System, Domain and Post Office records. In order to maintain backwards compatibility, these settings will be serialized into a setup.cfg file on demand when the setup.cfg file is requested from the server by the client.
Change #4: Support Update over IP without SetupIp.exe stub
The primary purpose of the setupip.exe stub is to allow administrators to define a location from which the GroupWise client installation can be downloaded. In the case of auto-update, this is not necessary. The connection to a valid repository for the client update files (i.e. the post office) has already been established. Therefore, engineering will modify gwipupdt.dll to directly download the compressed setup files and launch the install rather than downloading setupip.exe. This eliminates the need for the admin to generate a setupip.exe stub and specify an additional web server to host the setup files on.
In order to download the setup files from the POA, engineering will either use the C/S protocol to download the files from the auto-update directory (defined in Proposal #1) or download the files from the POA over HTTP. Note that downloading the files over HTTP is likely to be more efficient but would require HTTP access to the POA and changes to the POA to return the setup files when requested.
Change #5: Add admin setting for "Auto-Update URL"
Thus far, the assumption has been that every post office will host the auto-update files (or not support auto-update). Some administrators may prefer to host a single auto-update repository rather than installing the auto-update files at each post office. Engineering will add a new "Auto-Update URL" admin setting that allows the admin to define a location from which the setup files can be downloaded. This setting would be available on the system, domain and post office objects.
Per change #3, this new admin setting would get generated into the virtual setup.cfg that is returned to gwipupdt.dll. gwipupdt.dll would then look for this setting and if present, attempt to download the software from the specified URL. The update process would first attempt to download the software.inf to check the version of the software installed at the given URL and then would download the individual files.
The gwipupdt.dll update process would then look something like this:
Download gwipupdt.dll from server using C/S protocol.
Download (virtual) setup.cfg from server using C/S protocol.
Check for AutoUpdateURL
If present, download software.inf from HTTP server
If not present, download software.inf form server using C/S protocol.
Check if update necessary and/or prompt user
Download setup via HTTP or C/S depending on whether or not the AutoUpdateURL was set.
Run setup.
Deprecate Software Directories in Admin
Once the dependency on software distribution directories is eliminated, engineering will remove support for certain operations in the admin console. Here are the following changes:
Continue to show existing software areas, but don't allow the admin to create any new software areas.
Eliminate the requirement to create a software area before installing a new system.
Eliminate the requirement to specify a software area when creating a new post office.
Remove the functionality to "update" the files in a software distribution directory (this can still be done manually).
Remove the functionality to submit a system maintenance task to update the views on a post office.
Move the functionality of "bumping" the software directory from the software area to the post office.
These are all of the changes that will be made to facilitate this overall functionality change in the product. These changes will be part of Windermere.
I have long advocated at least an option for the server side model for Archiving. Currently, archiving is a client-side operation only - the server side has no knowledge of the archive's existence. Adding the ability for server-side archiving would not necessitate removing archiving functionality from the client for those sites wanting to continue in that model.
However, what a server-side option would allow is the ability to for a user to keep as much email allowed by local policy that will be available to the powerful GroupWise search features. One of the greatest benefits to administrators is the ability to reduce the nightly and weekend upkeep processes that current archiving provides, therefore, any server-side archiving must also reduce those upkeep processes.
I contend that server-side disk savings can be achieved as well. We are currently Reload customers for backup and restore purposes. To facilitate ease of restoration, we keep 1 to 2 weeks available in the Reload areas. That results in 5 to 6 times the size of the post office in Reload disk space. The thought here is that a smaller active post office would also result in a smaller Reload system.
The key is that the Post Office Agents are aware of an archive store (that could be on tier 2 storage) for searching, etc, but not a part of the scheduled upkeep processes. However, a separately scheduled upkeep process would be necessary most likely.
From a user's perspective, all the items are in a cohesive environment. From an administrator's perspective, we could finally get users to actually archive which would result in leaner and meaner post offices.
Server storage reduction is not the goal of our archiving movement. Because of our retention/archiving requirements, we can't in good conscience recommend a user archive their email to a local HD based archive for several reasons: email is not backed up (an organizational requirement), multiple archives getting created if a user uses multiple machines (now which machine did I archive that email on?), no access to archive email from WebAccess and/or other machines. As such we use shared server based archives.
Our goal with archiving is to reduce GroupWise's time in indexing, structural checks, and contents checks on email that is seldom or ever modified. In other words, we view archiving as a rudimentary HSM solution. An agent/server based archive solution with it's own indexing/checking schedule would greatly reduce the load on our servers and allow the users to still access their email from any client.
Allowing both methodologies would satisfy most organizational requirements and would be satisfactory for us. My only concern would be the engineering effort to keep such diverse and complex methodologies viable.
This is a regularly requested change. This is not something that will likely happen in the Windermere schedule, but we certainly recognize its importance. However, we also have many organizations that do not want their archives on the server....no savings of server disk space. So our approach will probably be to allow both methodologies and allow the administrator to choose which option is deployed within their organization.
In the process of discussing the changes outlined above, there was also a proposal to remove or get rid of setupip.exe. After several discussions and some great input from Tommy Mikkelsen, we have not only decided to keep setupip.exe, but we have revamped it in a number of requested ways.
Thanks Tommy for your insight and suggestions. We think the new experience will be more inline with how administrators want to utilize this process.
Yes, we don't use SDD but some custom views are used.
Whilst you are making these changes... please tidy up the messy client managed "Archives" specially with respect to users forced to store Archives on local drives. {I know with NSS uses can store Archives on the Server}
I think its about time Archive is moved to a SERVER based ARCHIVE AGENT - working much like the POA.
We had a couple of other questions get submitted via email and I wanted to include the questions and the answers in this forum so others could comment or be aware.
Question: Can the client view file versions be correlated to client version enforcement in the admin utility (Similar to restricting mgmt snapin versions in C1) ? This only applies if updated views cannot be used by older clients. To clarify...In large distributed environments it is not unusual for client versioning to have different authoritative standards. For example, when we were running GW 8, the clinical technical management would not let their clients update from their v6.5 clients. This would have likely carried through to the 2012 version as well.
In the past the view files have been backward compatible (check me here). I simply wanted to ensure this legacy binary trail was removed or addressed (the only reason version matching/enforcement was suggested).
AnswerThe client will be enhanced to either a) always prefer local copies or even better b) remove the utilization of view files altogether.
Question If HTTP is used by the POA to update client software should a new discreet port be used? In large environments I do have some concern about normal POA CS activity and client update contention on the same port. If WAN links stand between the update target and the POA, native internet latency could further aggravate contention issues. Additionally if the admin was to decide to prohibit client updates over WAN links this could be accomplished at the firewall/security layer, if a different port was used, and allow extra-net clients to fall back to the configured URL for client updates. All while allowing normal POA CS traffic to cross those links normally with no concern for contention issues (even if just perceived).
I think if ports can be added for pop, imap and soap one can be added for auto-update if it results in a more flexible environment. I think these are some of the simple and operationally valuable things administrators are looking for.
Answer:Regarding HTTP... That part of the update process is still in flux. We have considered several mechanisms some of which may alleviate the concerns for running on a discreet port:
1). Use the existing client/server mechanism to transfer files. This would not run over HTTP at all but may place additional strain on the POA. 2). Expose the files via the POA's HTTP interface... we already do this for a number of files. In fact WebAccess downloads file attachments through this sort of a mechanism. It runs over the HTTP port already specified in admin. 3). Use the jetty instance to host the setup files... Paul has modified the install to automatically configure the admin jetty service to host the setup files. In this case the default URL would actually not run through the post office at all. 4). We've always planned for the ability to set a separate download URL for the client setup files in the Admin Console. This would allow the administrator to host the files anywhere (i.e. on a separate instance of Apache running somewhere) and point all of the update traffic to it. In this case the POA's HTTP would be bypassed altogether. We were reluctant to include this as the only option because we wanted something that would work "out of the box"
Support
Documentation
Training
Partner Portal