By
Guruprasad S (guruprasad.s@microfocus.com) – Director of Software Engineering

With Inputs From
Anirudh Banger (anirudh.banger@microfocus.com) – Engineering Manager
Vamsi Krishna (vamsi.krishna@microfocus.com) – Lead ZENworks Architect

 

As a ZENworks administrator or user, you have probably realized that ZENworks is a big product. It has a lot of capabilities and provides a ton of value. You probably use many of the features it provides, and there are probably a lot you don’t even know exist.  We’ve been asked by several of you, what does it take to build such a complex enterprise product? What processes are followed to ensure quality? How do we test and validate? Etc.  As the director of the ZENworks engineering team, I would like to share with you some of our practices to help you understand what we do and how we do it. So without further ado here’s is an inside look at what we do in the Engineering group to deliver the scalability, quality and capability you’ve come to expect from the ZENworks product.

What We Used To Do…And Why We Changed

Until 2007, Novell used Waterfall as our development methodology. In Waterfall you typically have large, monolithic releases that tend to take a fairly large amount of time to release. For instance, ZENworks 10 took almost 3 years from start to finish, and unfortunately for those of you that adopted the 10.0 version, when you spend 3 years writing new code you are more likely to have more issues.

After the release of ZENworks 10, we wanted to develop in a more iterative fashion that would allow us to release the product faster to the market, with better quality, while responding more quickly to changing market needs.  With these goals in mind, we embraced Scrum, one of the leading agile development processes, as our development methodology.  In the last eight years, we have learned the Scrum process in depth, made some complementary changes, and also introduced some new practices which better support what we want to accomplish.  To a great extent, we have been able to respond to changing market needs more quickly, and we are getting better at delivering more frequent releases.  We still have work to do and improvements to make, but we are definitely moving in the right direction.

What We Do Now

As I previously mentioned, the ZENworks team uses Scrum today, which means we are following all the tenets outlined in the Scrum methodology.  As I explain what we do, I am not going to talk about the standard Scrum methodology; instead, I will focus on the additional practices we have developed that complement Scrum and help us deliver quality ZENworks releases.  I will use many of the Scrum-related terminologies as well as associated activities and explain how we have tailored or modified them to suit our needs.  I am going to outline just a subset of engineering practices we follow, which should give you an idea of what we do as part of the product development process.

1. 1. Planning - In Scrum, planning is a very important activity. During the planning process, the team understands the work, breaks the work into different tasks, and then estimates how much time it takes to complete the tasks. Typically, most of the tasks within a sprint are around development and testing of the User Stories assigned to the sprint. There are different ways in which this development can happen. We have chosen to use Acceptance Test Driven Development (ATDD).
      
      
      1. 1. ATDD – In a nut shell, ATDD brings together the Developer, Tester and the Customer (in our case, the product owner). Each one studies and understands the User Story (feature), and then they collaboratively discuss the software design and test strategy. Once this is done, the test cases are derived from the test strategy and are classified as Unit, Component, and End to End test cases.  Typically, there are a large number of Unit test cases, fewer Component test cases, and even fewer End to End test cases.  The relative numbers can be visualized as a pyramid, with Unit test cases as the base layer, Component test cases as the middle layer, and End to End test cases as the top layer.  The team decides which test cases to automate and which ones to run manually.  Most of the Unit test cases are automated, while Component and End to End test cases are automated based on the effort, availability of required functions, complexity, ROI, and so forth.  The team develops test cases first and then continues to implement the corresponding code.  Initially, many of the test cases fail, but as the code is developed the test cases start passing.  This activity continues until the user story is fully coded and all the test cases pass.  There is also a dashboard through Jenkins to aggregate the test results.
            What do we get from this?
            
            
            • • Better sprint planning because ATDD gives more clarity to the User Story being developed.
            
            
            • • By involving the customer (product owner) in the building process, we are in a better position to ensure the outcome meets the customer’s needs.
            
            
            • • The dashboard helps give the overall status of User Story development, enabling problems to be caught early on.
            
            
            • • The automated test cases can be run sprint over sprint, ensuring known quality of the User Story throughout the release.
            
            
            In all, ATDD helps us build the right feature from the User Story (lowest unit of the product) in the right way and also provides ways to ensure the feature is working correctly in the product throughout the development cycle.
      
      
      1. 1. Rolling Look Ahead Planning – This is a standard practice for coordinating the efforts of multiple Scrum teams. Each Scrum team indicates what the team is going to do in the next few sprints and explains what the team needs from other Scrum teams. We look ahead for three sprints to identify any potential dependencies across teams and repeat the process each sprint until the end of the release.
            Advantage of this practice:
            
            
            • • Ensures that dependencies are called out early, allowing the dependencies to be correctly prioritized and ordered so that no team will have to wait on other teams to finish dependent work.  This clears the dependency issues across teams.
            
            
            
      
      
      

1. 1. Development & Testing - There are many things we do as part of the product development, but here I will outline a couple of practices.
      
      
      1. 1. Feature Branch-based Development – A feature, or a set of related features, is developed in a specific feature branch in the configuration management system. Once a feature is fully implemented, it is merged into the main branch.
            Advantages of this practice:
            
            
            • • Feature branches help ensure the sanctity of the main branch. Any issues encountered during the development in any branch are limited to the Scrum teams using the branch, so other teams can continue to do their activity without interruption.
            
            
            • • By following some of the best practices of Build systems, it is easy to merge changes into main branch.
            
            
            • • Allows features developed in a branch to be easily added to or removed from a release, maintaining the sanctity of the release timeline.
            
            
            
      
      
      1. 1. Performance & Scale Testing – ZENworks supports managing of 100,000 devices. The product needs to be tested continuously at this scale throughout the development process to ensure changes done during the course of development don’t break anything. A separate Performance & Scale Test (PST) team having access to a Lab, which we refer to as the Super Lab, that is capable of simulating close to 100,000 devices runs in parallel to other scrum teams.  The Super Lab team executes a set of defined stories at regular intervals by taking build drops from the main branch.  All performance & scale related issues are debugged and addressed by a focused group of experts.  The fixes are passed back to the main branch.
            How does this help?
            
            
            • • Each Scrum team doesn’t have access to the number of resources available in the Super Lab and can’t focus on Scale and Performance factors throughout the development cycle. This process of centralized testing helps the Scrum teams to focus on functional aspects and not worry about testing scale and performance.
            
            
            
      
      
      

1. 1. Sprint Reviews - In Scrum, Sprint Review meeting focuses mainly on showcasing the User Stories completed in the sprint.  The audience for this includes the Scrum team, Scrum Master, other Scrum teams, Architects, Technical leads, Management, Product Owner, and Documentation team.  We have the following practices as part of Sprint Review.
      
      
      1. 1. Review – The Scrum team presents what the team had planned for the sprint, what they accomplished, impediments they encountered, and where they stand with regards to the release of the feature/product.  This formal review helps the sprint team, management, product owner and everyone else know how the release is progressing and if we need to take any corrective actions in case of any deviations from the plan.
      
      
      1. 1. Demo – A separate meeting where the completed stories are demonstrated to the same audience as mentioned above.
      
      
      1. 1. Demo of unfinished stories – A separate time slot after the Demo to show-case unfinished work for early feedback from stakeholders.
            How does this help?
            
            
            • • ZENworks Engineering has around 12 Scrum teams. The formal review helps present the complete picture of each team’s progress as well as the overall progress for the release. If necessary, corrective actions can be taken to ensure release progress.
            
            
            • • By spending some time on unfinished stories, we are able to get early feedback and make required corrections as we continue to build the story in the subsequent sprint, hopefully resulting in less (or no) re-work.
            
            
            
      
      
      

1. 1. Automation - Automation is a standard practice for ZENworks.  Everyone understands the importance and the advantages of having a good automation suite.  Here we will see different types of automation that we put ZENworks through as part of the release cycle.
      
      
      1. 1. Smoke – This is no-touch/lights-off automation. At regular intervals the Smoke Automation setup polls for newer builds. When new builds are created, Smoke Automation picks up the build, installs it, and runs a set of test cases.  These are basic level-0 test cases to determine if the build is acceptable for the next level of testing.  In the “Feature Branch-based Development” section, we explained how we build features in different branches before we integrate the features into the main branch.  The builds from different feature branches automatically go through Smoke Automation before they are picked up for further testing.
      
      
      1. 1. Sprint – In the “ATDD” section, we explained how the Scrum team evaluates each User Story and comes up with test cases, and then identifies the test cases that can be automated. Sprint Automation includes all automated test cases from each Scrum team that are executed on a continuous basis sprint over sprint by the respective teams.  Some of the Sprint test cases are integrated to form End to End test cases that can be run at the Product Testing level.  This adds to the overall automation coverage.
      
      
      1. 1. End to End – This is also referred to as the “Full Test Pass” automation and forms the heart of ZENworks Automation because it covers most of the commonly used features. There is a dedicated Automation team that continuously picks up features of ZENworks, both existing and new ones, for creating automation test cases and subsequently integrating them into the full suite.
      
      
      1. 1. Large Scale Testing – This needs a different kind of automation. Unlike regular setups where functionality has to be verified, here the need is to verify how the functionality scales up.  The biggest challenge with Scale & Performance automation is to get the setup in place to be able to test the functionality.  So, in this case, the automation is focused more on tasks such as 1) building the required setup with the required set of parameters and 2) having scripts that can collect and analyze logs from a large set of devices.  As mentioned in the “Performance & Scale Testing” section above, there is a need to qualify ZENworks to work at the scale of 100,000 devices. Therefore, this automation is designed to get the setup ready in a short time, have the build deployed, and, after the tests are run, collect the results and analyze the same.  Automation is also used to time various operations to see if they are within acceptable levels.  These automation tests help run the product continuously at the required scale and identify and address any issues that come as part of the testing.
            What do we get from this?
            
            
            • • ZENworks has a complex platform matrix that needs to be tested and supported. Automation helps qualify various combinations more easily and quickly.  This helps us add newer platforms to the list of supported platforms.
            
            
            
      
      
      

1. 1. Roles Outside of Scrum - Scrum has some standard roles – Scrum Master (SM) and Product Owner (PO)– with defined sets of responsibilities.  As product complexity grows and the number of Scrum teams increase, we experienced a need for more roles to support the standard Scrum roles and help in the overall release of the product.
      
      
      1. 1. Product Architecture & Quality Team (PAQT) – A virtual team comprised of Subject Matter Experts (SMEs), Architects, Engineering Managers, Project Managers, and Product Owners/Product Managers.  Each Scrum team has an SME who is the technical leader for the team who supports the Scrum Master and the team in the technical activities of the team.  An Architect and an Engineering Manager oversee the work of multiple Scrum teams, helping them to resolve technical issues and other impediments.  The Lead Architect, the Release Owner, the Project Manager, and Product Line Manager oversee the release of the product, ensuring quality and timeliness of the release.  Here is an illustration depicting the different roles:
            
            How does this structure help?
            
            
            • • This structure ensures there is a clear visibility of what is done within the Scrum team by the SME or SM or PO to next higher levels which is the Architect/Lead Architect or Engineering Manager/Release Owner or the Product Manager respectively. These roles work within their teams as well as with the next level roles in supporting and resolving issues to meet the objectives of the respective teams as well as the overall release goals.
            
            
            
      
      
      

1. 1. Scrum Adherence & Improvisations
      
      
      1. 1. Scrum and Process Evangelists – We have a Scrum Evangelist team comprised of Certified Scrum Masters. This team oversees how the Scrum teams are doing with regards to process implementation, suggests areas of improvement, and also comes up with improvisations in process/practices for the entire group.
            What do we get from this?
            
            
            • • Processes are meant to make the engineering activity efficient and to reap the benefits we need to do it right. This team ensures that we are following the right practices and fixes any short comings for a better delivery of releases.
            
            
            • • This team also comes up with areas of improvement and helps us learn from past mistakes so we can be more effective in what we are doing going forward.
            
            
            
      
      
      1. 1. Scrum Masters Forum – This is a forum for Scrum Masters to meet and interact. This forum facilitates knowledge sharing sessions on Scrum practices, Scrum related trainings for newly inducted Scrum Masters, and soft-skill building on attributes like empowerment, ownership, and accountability.
            What do we get from this?
            
            
            • • The forum helps in overall development of Scrum Masters and enables them to excel in their role.
            
            
            • • This forum of Scrum Masters acts as a place to share best practices, discuss challenges, and learn from each other.
            
            
            
      
      
      

Wrapping up….

ZENworks is an Enterprise product, with flexibility of platforms, huge scalability requirements, and a large amount of value and capability.  The above information, though not exhaustive, should give you an idea of our development practices and the steps we are taking to ensure you get the best ZENworks product ever.  We have been working with this framework for a couple of years now and have seen good improvements in the quality of releases. Several of you have told us that you have seen major improvements in the quality of the product, which ultimately is what really tells us how well we are doing.

We are now embarking on addressing the challenge of delivering more value to you and your users in shorter periods of time, while continuing to maintain and even improve quality. We hope you come with us on that journey as we enter the ZENworks 2016 era.