Agile RUP for Product Development

Best Practices of Requirements Collection

Accurate Requirements form the most essential part of the formula for successful software product development. This article explains why and describes the best practices of writing accurate requirements.

A Fortune 100 company embarked on a project to design and build a sophisticated software package that it would ultimately deploy to its offices throughout the world. Two years and about $10 million later, the field offices refused to use the software because it didn't do what it was intended to do. Instead of helping to streamline an important business process, the software actually hindered it.

Sounds a similar story? Based on several surveys by various groups, there are some stark findings:

The Standish group reports that over 80% of projects are unsuccessful either because they are over budget, late, missing function, or a combination. (http://www.standishgroup.com/sample_research/chaos_1994_1.php)
53 percent of projects cost 189 percent of their original estimate.
Average schedule overruns may be as high as 100%
Over 40% to 60% of defects in software are caused by poor requirements definition.
About One-Quarter of all projects are cancelled.
Customers do not use 20% of their product features.

These are some of the many findings that do not speak highly of successes of software development projects. Based on our experience of over fifteen products, and learning from the several surveys and research, we have identified that the most critical factor of successful software product development is the right requirements documentation. Poor requirements are the biggest problem that end up increasing the budgeted engineering cost and not matching with the expected quality.

If we look little deeper, and see why poor requirements cause all the havoc:

Impact on Design and Architecture: Incorrect or incomplete understanding of the current and future requirements will not enable the architects to design the right architecture and design that can take care of current and immediate future needs.

Increased Dev Iterations: If the development team does not have the complete information, there will be far too many iterations and cycles around requirements understanding.

Impact on Quality: If the development team makes too many incorrect assumptions because of lack of clarity, it may arise in a lot of iterations in the later half of development, when the product managers realize the gap due to incorrect assumptions.

Reduced Testablity: The test engineers do not have the complete picture to write correct test plans and test cases if the requirements are not properly communicated to the test team in the right manner.

As these issues point out, accurate and complete documentation of requirements is very important. In addition, presenting them in a form that will communicate the complete and accurate picture to the development team is equally important. If the development team cannot visualize the end product completely, there will always be a gap between the actual vision and the software developed and the development will go into several iterations trying to bridge that gap.

We have learnt a lot of lessons on requirements process from our vast software product experience. In addition, we have done extensive research on the best practices followed at the top software product companies in the world. Our experience and research has helped us to define the types of requirements needed for capturing all aspects of software. They are:

a. Business Vision and Business Requirements - "Why" requirements. These capture the high level objectives of the organization or customer requesting the system or the product. They can be considered as "why we are doing this" requirements and should capture the fundamental reason for product existence.

b. Functional Requirements - Functional requirements can be considered as the "What" requirements. These capture the functionality that must be built into the system to satisfy business requirements. Use Cases are great way to document functional requirements. A use case defines a goal-oriented set of interactions between external actors and the system under consideration.

c. Non-Functional Requirements: They are the "How" or "How Well" requirements. They capture the technology specific and -ility requirements of the system like compatibility, usability, usability, performance, reliability etc.

d. UI Prototypes - UI prototyping helps in reducing risk, decreased system size and complexity, and reduced requirements change. The UI prototypes must be validated by the (beta) customers before they can be frozen. We have found them to be the most effective way of communicating requirements to the development team. UI prototypes/screenshots provide a great visualization tool for both development and test engineers to understand the exact software flow and they are much more effective in designing classes and test cases.

We recommend "Requirements Scrubbing" after they are documented. It is a process of carefully examining the requirements for unnecessary or overly complex requirements, which are then removed. The idea is to eliminate all non-essential requirements, making each requirement as simple as possible, and substituting with cheaper options whenever possible.

Finally, how would you ensure that you have covered all aspects of requirements through functional and non-functional requirements? We have prepared exhaustive checklists that will help you to make sure that your requirements are ready to be published to the development team.

Functional Requirements Checklist:

a. Are the requirements complete ?

b. Are all the requirements uniquely identifiable?

c. Are the requirements clearly and appropriately prioritized?

d. Are the requirements consistent? ( no internal contradictions)

e. Does the set of requirements adequately address all appropriate exception conditions?

f. Does the set of requirements adequately addresses boundary conditions?

g. Are the requirements feasible? (solution to the set of requirements exists)

h. Can the requirements be implemented within known constraints?

i. Are the requirements sufficient?

j. Are inverse requirements explicitly stated?

k. Are these the simplest set of requirements that meets the stake-holder's needs?

l. Are all the cross-references to other requirements correct?

m. Is each requirement testable/verifiable?

n. Ensure that each requirement needs to be a solution instead of being stake-holder's statement.

o. The requirements need to be in customer's language using customer's terminology.

Non Functional Requirements Checklist:

a. Performance - performance requirements of the application -- Response time or latency requirements; Throughput requirements; data volume requirements ( input, stored, output), Peak or short-term load requirements.

b. Platform support - which Operating systems, prerequisite software and hardware requirements.

c. Licensing - licensing scheme of the product and other OEM/external licensing requirements.

d. Internationalization - what locales/languages is this software going to be built for now and future.

e. Integration requirements - How do you want third party applications to integrate with your application to enable more usage scenarios and sales partnership opportunities? How do you want to integrate with third party applications for similar benefits?

f. Setup - How should the product be deployed? You must consider install, uninstall, repair and upgrade scenarios.

g. Security - What is the security and safety model for your application? Consider authentication, authorization and secure data transfer. You must consider both application and platform (windows) security.

h. Configurations - Are the Supported Configurations specified? Are the Compatibility requirements specified ( backwards, other applications, etc.)

i. Usability - Are the usability requirements specified? Are the look and feel requirements specified? ( Color schemes, standards etc.)

j. Operational - Are any operational constraints or requirements specified? ( the user must be able to operate the system using ski gloves etc)

k. Reliability - Are the reliability requirements specified? Are the availability requirements specified? Are the serviceability requirements specified? Are the robustness requirements specified?

Design Principles for Effective Software Product Design Engineering

Although comprehensive requirement collection or inception is the most critical activity for software product engineering, it will be naïve to think that the product would be successful if fundamentals are not strong. The right architecture and design is most critical artifact after requirement documents.

Product design is the art of applying technology, economics, aesthetics and common sense (!) in order to create software products which make business profitable, sustainable and scalable.

Software product design should be a holistic activity, encompassing and addressing every aspect of software – the end users, the business vision, the release and version plan, the budget, expected revenues, time to markets, expected lifetime of software and much more. It should not be confined to the popular narrow belief that design is all about applying object orientation, patterns, application blocks, frameworks etc., because they are the ‘in’ thing.

In this article, we identify and analyze the factors which should be considered while undertaking the architecture and design activity.

Begin with the end in mind: Product roadmap consists of software feature and release plan for finite number of years. If you are developing product as one time activity then there is probably no point in designing – just skip this step. But that’s hardly the case. The design should consider all foreseeable versions of software, their features, and create enough flexibility to incorporate these features without too much of a fuss. The level of flexibility and extensibility in product design is dictated by product roadmap.

Design to realize the Business Vision: If the business vision dictates aggressive time to market the first version of software and your architect starts with a highly generic design, then the release is bound to get delayed. The design choices should be aligned with your organization’s business model. Specifically, the architect and design team should always be in synchronicity with the following criterion:

Time to market – Don’t over-design if you time is not on your side. For example, there is no need to create Interfaces and corresponding abstract classes for everything if it doesn’t add any value to your organization and end users.
Competition – The design should address the product differentiation requirements from completion. The design should also consider competition from new entrants in the market. Somebody shouldn’t just be able to throw you out of market by replicating your windows product on unix!
Revenue Model: It is important to understand the revenues models which your organization has envisaged around the product. The design for a hosted ASP application would be drastically different from the model where you sell the CDs of your software product.

Design for End Users: How can an end user tell me how to design!! That may be the question in the mind of an egoist software architect. No, end user is not going to tell you. Instead, he will forget your product after downloading the evaluation copy if you did not consider him while designing. An example: I never use a product if it doesn’t support the DD/MM/YYYY date format. The requirement is simple but critical.

Analyze following things about end users before you get to the UML diagrams:

Home or corporate users – Corporate users are more particular about professional look & feel, usability, security and performance. Your design should not compromise these aspects. Home users think in terms of ease of use, simplicity of features and compatibility on multiple platforms. Design what best suits to the end users.

Internationalization – Consider how you will handle when your application is run different cultures. Handling multiple languages, date formats, address formats etc as part of software design.

Reason and context of using the software – Put yourself into users’ shoes and you will surely get inputs for design. A music playing software should support skins while word processing software will not. A frequent traveler will would like the software to be offline capable – he should not be left chewing his nails if there is no connectivity.

End users’ age group, educational background, IT skills, etc also play important role in making the right design choices.

Think Integration: For the end user, your software product is one piece in the IT eco-system of his life. He would like it be interoperable with other members of eco-system. If you develop walled software, you are bound to repent sometime. The software product should be open enough to be integrated with other relevant applications. And to make it open, the level and granularity of openness are important considerations for software design. Ask these questions when you think integration:

Will other ISVs integrate your software with theirs?
Will you allow partners to integrate your software with others?
Do you yourself want to provide professional integration services?

Think Deployment: For some reason, deployment is considered as the last thing in software design (May be because, it is last phase of product life cycle!). Thinking about the deployment scenarios early can prevent products from failing. Software products can be multi-tiered, having a mix of servers, services, clients and databases, SDKs, supporting multiple operating systems. It should be possible to deploy them in a variety of topologies. The designs should make sure that the consistency, security, performance and reliability if not affected by the topology.

It’s also important to understand the typical production environment. Sometimes, the constraints of production environment force design changes. For example, if the connectivity between different components is intermittent then message queues may be ideal compared to synchronous communication mechanism and the client should be offline capable.

Build vs. Buy (vs. Free vs. Open Source): It is very important to make economic sense for product development activity. As the old cliché goes ‘Don’t reinvent the wheel’. If it takes 3 man months in developing a component of your product, you would be better off buying it for 1000 dollars with royalty free distribution or using open source component and spending 1 man month for customization. If you are lucky you may get it free. For example, for .NET based product development it is a good idea to use Application Blocks instead of writing custom code for infrastructure components like Data Access, Logging, Exception Management and Caching etc. It is a better idea to buy UI components from Infragistics than to develop your own. The design and architecture team should make very conscious decisions when it comes to reinventing the wheel.

Choose the right platform(s): Should you use J2EE or .NET or Cold-Fusion or LAMP (Linux, Apache, MySQL, PHP)? This decision largely depends on the kind of user segment you are targeting. For example, if you are developing a fresh product for small and medium size customers having 5-500 user, then .NET would be the best platform, provided that majority of end users use windows. But, individual strengths and weaknesses of each platform also play a role in arriving at the correct choice. If your product uses Web Services or Message Queuing, then you may choose .NET since it provides mature implementation of these technologies. Whatever you platform choice be, it’s important to make sure that it provides enough technology support and infrastructure services to satisfy current and future product requirements.

Security: There are four aspects to security –

Any part of your product should not be usable by unauthenticated and unauthorized parties
The product should not compromise the stability and security of end users’ IT environment
The product should adhere to law
No body should be able to exploit your IP (Intellectual Property). Obfuscate and protect your code.

It is vital to incorporate stringent security into the product design. There is nothing more damaging than bad press about security holes in your product!

Performance & Scalability: Even though the memory is becoming cheaper and processors are getting smarter, there is no escape from factoring performance and scalability issues in your design. Bad design leads to performance degradation when the volume of concurrent operations increases. Bad design leads to redoing the applications when it fails to scale. The better approach is to estimate the volume progression for all foreseeable product releases and consider it as design input for maintaining consistent level of performance and ability to scale the product to new levels.

Proprietary protocols or Open Standards: This is similar to the dilemma of build vs. buy. And the solution is also similar. If proprietary protocol or standard is not going to be the unique selling point for your product, then go for open ones. They are more likely to be stable, mature and acceptable than proprietary ones.

Design to enable Customization: Any enterprise class of product would require some amount of customization before being useful for end users. They may be scenarios where the product has to be customized for an industry vertical. If such is a case for your product, design for points of customization. It may be as simple as providing the facility to change the text of UI object.

Support all Licensing Models: There can be numerous types of software licensing models - One Time, Subscription, Named Users, Server license and so on. Ideally, the design should be flexible enough to incorporate all possible licensing models which the organization may want to offer. The decision on licensing model design should not be left to the end. If it is ambiguous, then the best way is to decouple it from product and design a separate generic licensing manager.

Exploit Tools: Software tools can be effectively used to model the components of product. They can save lot of effort and the model produced by them can be an effective artifact to communicate the design without ambiguities. Another usage of software tools can be to generate code. It can be effective when the model (database model, UML etc) does not change a lot as the product development goes into iterations.

Simplify: Newton was a genius, but not because of the superior computational power of his brain. Newton's genius was, on the contrary, his ability to simplify, idealize, and streamline the world so that it became, in some measure, tractable to the brains of perfectly ordinary men - Gerald M. Weinberg. Remember the KISS principle when you design software – Keep It Simple Stupid.

Requirements is all ‘doing the right things’, while software design is ‘doing the things right.’ Now you know why it is important to be more watchful about design!

Bottomline of any software engineering process...

Design and programming are human activities; forget that and all is lost.

- Bjarne Stroustrup, 1991

About this Blog...

The software product development industry has evolved as one of the most important industries of our time. There is intense competitive environment, and it is primarily the ability to create and deliver software products that better address their real customer needs, that will determine the winners. Almost everybody in this business understands this, then why dont they achieve it ?

The answer lies in the software development process itself. Over the last few decades, several visionaries, teams and companies have worked hard to define processes that will make software development more predictable in terms of time, cost and quality. Beginning from the Waterfall, Spiral models to the RUP, MSF, and now the Agile methodologies like XP, Scrum etc, there are several models that talk about the right software engineering processes. RUP is one of the most well defined, comprehensive process frameworks, and the Agile processes represent the lightweight, people driven methodologies.

Being involved in software product development for the last six years as a developer, architect and manager, this subject has fascinated me a lot. Needless to say, the element that makes this whole subject fascinating is the "people" factor. You think of an improvement in manufacturing process, design it and make necessary changes in the assembly line - more often than not, you will see the process implemented permanently. There are hardly many people would be able to say something similar for software development.

In this blog, I would like to share our experiments at Proteans with various processes and present the "Agile RUP" which we have successfully applied to software product development, and still improving.