Thoughts

Integration and Product Design

Product Design and Integration

A few days, a colleague posted some insights on Design Principles here (http://www.proteans.com/CS/Web/blogs/agilerup/archive/2005/07/06/18.aspx). There were very interesting topics, discussed briefly, and one of them was about the emphasis one should have, when designing, on integration. I have had some thoughts on the same too, and here they are, following a little background information.

What is integration? And where does it come into picture?

Suppose you create a great accounting package, for SMEs. Its very good, got good performance, great UI, the clerks love it, the ladies go nuts over it, if looks could kill and all that, you know. Now we understand that this is not a killer app, because there are many accounting packages out there in the market, many from big and dependable corporations like MicroSoft, many well established like Quickbooks, and many much simpler than yours. But you have a great S&M (Sales and Marketing) team that really toils hard, and sells a thousand copies. You will now, definitely face one of the following scenarios, if not all of them -

1. You S&M girl comes back and tells you that most of the people that she talks to about our product, say that they want an accounting package which can take in inputs from their web-site too. Ours doesn’t support that, so they don't even talk about it any further. Then there are people (Prospects) who have some partners out there in business, providers of raw-materials etc., and the partners have, let’s say, QuickBooks installed on their site, and the prospects want to get rid of paperwork, and want the accounting package they buy, to talk directly to their partners' package. So these guys don't talk to the poor (well yes, there are such rare situations, where you can use the word poor as an adjective for Sales people) lady, and she is now getting red at you, for not thinking of such situations when designing the package.

2. You identify a liquor store as a major prospect, you visit them personally, try to impress them with your tech-talk and all. But they counter everything with some of their own. You tell them about your package, and they start asking you about how your package would work with their existing infrastructure. They start talking about ESBs, BizTalk and all, and you take your sad face home, thinking what went wrong, where are good old stupid liquor store managers going these days.

3. One of your clients grows really fast, and is generally happy about the software that you gave him. But now, he needs an Inventory Management solution too. So he buys one, but later finds out that instead of improving his inventory management, installing this new software has increased the paper-work, and along with it, chaos in the office and his staff productivity has decreased to half. He digs deep to find out that this is because of the technical incompatibility between the accounting package you wrote, and the inventory management software he bought. He is in an indecision mode, and during this period, the company that sold him the inventory management software calls him up, and tells him about the new accounting package they are writing, which seamlessly(the regular sales talk, you know) integrates with their other softwares. The client calls to tell you his decision, for him the decision is great, but for you? The story doesn't end here, this guy goes and posts a recommendation on a web-site, and now the number of distracted clients is really growing, and your business is going down. Would a new version really help in this scenario?

In any case, with almost all enterprise products, and even some standalone ones, you would encounter such cases. This is why software’s ability to provide an integration API, or its ability to seamlessly become a part of an integrated infrastructure of products is important, it’s the need of the hour, and it’s this integration-ready software which would sell. This is the main reason for major players chiming SOA mantras in every major event, on every technical website.

How do you go about the design decision?

Following are three good ways to find out about how you should actually go ahead.

1. If you are in the process of designing an enterprise level solution or for that matter any other solution, research the market to find out the prospect users. Talk to them, before designing the product, and not after the development is done. Ask them about what issue they have faced with paperwork between different departments. Ask them if they are looking for software that does not require manual intervention for communication of data between the software their other departments use. Next, ask your prospect customers what kind of "other" software would they like your software to work with. In other words, research on the environment in which your software would be deployed.

2. If you already have a partner in the market, who complements your line of products with its own, ask them if they want their products to synergize with yours. Ask them what their software would need to "talk" to yours. Find out what would your software need to "talk" to theirs.

3. Do you want to improve your future business, by providing APIs/Platforms for integration today? When you grow with an accounting package (for example), do you want to write and sell an inventory management system that talks to the latter? What happens when you write a shipping management system? In other words, does your vision extend to a scenario where this product you are writing is just the first in the line of a complete enterprise solution in future.

If you have bought the idea of integration by now, you must start making decisions on implementation strategies for integration. I have borrowed some ideas, which you can easily extrapolate on.

Implementation

Now integration is definitely not anything new, and almost all enterprise software vendors have faced customer demands around it in the past. Some have solved it concentrating on the state storage system, some using messaging, and there are other ways too.

Integration based on state storage systems

This is a very fragile architecture, wherein when some data is changed in a software's domain, this software indicates other "integrated" software by either writing the data to a file or invoking a database trigger.

Messaging

Software sends messages to a sink, which other software read to update their data, according to the message. This is a very good and agile mechanism. Unfortunately, traditional messaging techniques are more or less platform or OS specific.

XML based messaging and SOA

This is an extension of the previous, but because of the use of XML, this makes it possible to integrate with a wide range of products. Morover, a good service oriented architecture enables other software systems to indicate data/state change to yours, and thus provides a way to complete integration. My team is freshly through the integration phase, where we integrated an EMR system, running on a Unix machine and exposing a Java web-service, with one of our client's .Net based medical transcription products. There are many ISVs that specialize on integration, some use Enterprise Service Bus(ESB) products (mostly Java), while Microsoft is taking integration forward with Biztalk.

I would therefore strongly suggest SOA, if and when you decide on your integration strategy. And yes, don't forget to put that paragraph on why or why not you are considering integration, in your design document. Follow this up with a good strategy (if yes), based on strong market research. There are some good books and web-sites in the market for Enterprise Application Integration (EAI), EAI Patterns etc. which would definitely help you find out the correct integration strategy. Just open your eyes.

Metrics and XP

The measurement of change, the expected and the implemented, are best expressed as numbers. The numbers might represent different aspects in different situations; in different phases, sprints, pushes, periods or whatever you prefer to call them. The number themselves are more important to the team, «For philosophical reasons, I refuse to break the team into sub-atomic particles» than external elements, and in more cases than not become the pivot around which the development revolves and evolves. Much to our delight, these numbers seem to be very important to managers and clients for obvious reasons, and both ends of the river meet. Its such a nice place to live, this world ain’t it? Be forewarned.

There are some numbers that I would decide upon, if I just started developing a robust «lets hope» software product.

1. Features completed/ Feature backlog
Very basic metric. Divide each iteration into a number of features. While developing, find out whats the % age of work completed in terms of features. The one flavor that really helps is the weighted %. Assuming all features are not equivalent, assign a weightage to each feature. And then calculate the work done % age. This becomes complex with the 2nd iteration when bugs start coming in. You need to add bug fixes as features too. Some people call them «tasks», whatever.

2. Test cases failed %
Another important metric. In subsequent iterations (to the first one), one might want to look at this metric to know the quality of the development, the stress areas, and most importantly, which module requires pair programming. One very important push monitoring this metric should give is the urge to follow TDD to the core. And ofcourse a better design and good distribution of features iteration-wise.

3. Velocity of development
This is a metric followed by some people to track the expertise of the team, their proficiency to code. To add another parameter here, I would like to add the proficiency to understand. This makes it number of features ( x weightage preferably) / «time frame» per developer. Definitely very helpful one the team chart.

4. Team meeting time and its effect
An XP team should meet. Meetings are called with a goal in mind. Without discussing the types of meetings, it must be noted that one has to find out the effect of such meetings, on the overall or iteration-wide goal. The time spent on each meeting must be recorded, and its effect «if there is one expected» should be measured using one of those numbers above.

5. Number of change requests
Change requests are one of the bitter truths of programming. To measure these is again an important decision. Considering them as features is another. But thats not the point. This metric is essential to note down the requirement gathering design issues involved. Another product of this metric is to keep the team updated with their mistakes «if any» .

6. Time spent on change requests
             The time spent on change requests is another important metric, which highlights the design issues, extensibility factor. Noormally a good design should allow a speedy recovery path, for all such requests.

   There are other mainstream metrics, that accompany the Agile RUP process which might be useful. One must not forget that keeping a chart, and updating it with the current metrics is the most important part of a development process. Ultimately, passers by should feel the enthusiasm and excellence of a team just by looking at the chart. It should be something the team must be proud of.

The importance of being Extreme with Development Testing

When developing a product, one of the most important problems that a development team has to tackle is the fear of the unknown. While both services and products concentrate on an act of automation for the end user, in contrast with a service that serves a single company, a product focuses on enhancing productivity of a totally unknown set of customers.
A product development team has to understand that each of their small actions will have a tremendous effect, with (hopefully) thousands of users reacting. If going in the positive direction, the team can really help out the users to high productivity and quality through automation of an activity. On the other hand, a simple flaw in the product can make a lot many customers feel miserable about the whole idea of automating a process.

Importance of Developer Testing and TDD

The foremost feature of any software product is its quality. The development team must ensure perfection of their code. They must never be complacent about the quality of their product. Traditionally, it has been the responsibility of qualified and experienced Quality Engineers to ensure the perfection levels of a software product. There are basic flaws in this being the only quality ensuring process which is followed, and these are because

•   A Quality Analyst directs his/her attention to only the functional aspects of the end-result.

•   A Quality Analyst cannot make sure of extensibility in design. Though quality and extensibility sound unrelated, they have a very deep cause-effect relationship. An inextensible product can be totally flawless, and pass all quality measures, failing some later on enhancement requests, demanding a change in design. Thus a design perspective is wanted.

•   A Quality Analyst has no idea what the code looks like. With the highly dynamic nature of personnel developing a software product, it has become very important for a developer to ensure universal understanding of the code they write. What can be achieved with proper comments and refactoring never (and should not) bothers a Quality Analyst.

•   A Quality Analyst can only test the product through its GUI. When the team wants to expose the central API of the product to 3rd parties to develop enhancements, or decides to customize the GUI for an important customer, a completely new set of use cases come into picture. A Quality Analyst can only test all these if provided a GUI, which not only is time consuming, but unneeded in the wake of new development tools provided for White Box Testing.

•   An automated Regression Testing marathon, does not dig any deeper, as a QA totally depends on the UI of a product.

The most productive option to tackle these flaws is to encourage developers to test their code, feature by feature, following TDD. It is of utmost importance that all features are broken down to the lowest possible granule and then each granule tested. This is roughly equivalent to checking each brick while building a house, which is what the owner, would love the builder to do.

Breaking complex features into small units enables a developer to understand the expectations perfectly. When a test passes on a unit, it gives confidence to the developer that he/she can build over this brick. And it is imperative that the development team will identify design flaws that might result into redesign much earlier, or at the least encourage an extensible design. Automated regression of persisted unit tests ensures much higher quality than the regular GUI testing tools, and what better if both of them are used together!