Requirement Engineering in complex context

Abstract

Requirement Engineering (RE) as profession is taking centre stage in software development and its preparedness to deal with complexities need to be revisited. For long, development methodologies or processes  have been point of discussion to tackle the practical issues - which are evident in complex scenarios - arises between RE work (problem space) and design / implementation (Solution space). A common approach observed across some prime development models is of co-evolution of problem and solution space. This approach is indeed a central aspect of design work. It has been argued that functional origins of complexity demands continuous matching of solution to the problem in hand and a problem centric approach without necessarily committing to a development methodology may give results in complex scenarios.

Introduction

In RE, much of the emphasis has been laid in compartmentalizing the requirements work into analysis of elements like - goals, stakeholders, context and scenarios with the support of rationale, assumptions, constraints and verifiability. These activities have been adopted as principles to get the requirements right. But still conflicts in requirement specification and implementation are common.

Motivated by the fact that prime task of requirement work is to bridge the information gap between stakeholders of varied discipline to bring a conceptual integrity between all and ever increasing complexities in technologies, domain, context of use and available resources for solutions; a convergent , synthesis, approach is also required to be adopted so that the individual requirement element analysis leads to a seamless view of the real requirements and reduce the conflicts.

To explore such an approach, first the functional origins of complexities need to be looked, so that, nature of approach can be envisioned. Second it has to see what present approaches have been found in current methodologies to tackle complexities. Eventually, based on the first two, characteristics of such an approach can be derived.

Functional origins of complexity and Co-evolution in RE

Practice of Requirement Engineering is not new. It has been as old as of the times when humans thought and manufactured the very first tool or utility. Within the safe estimation it can be viewed as some million years ago! Certainly Requirement Engineering as profession has not been that old. The prime reason that give such a stark difference is that it has been evolved from requirement work for self to requirement work as services to facilitate the change. In other words, Requirement work has been evolved from the context of individual space to space as big as society or even country (-ies) ! These works can be identified in To-Do list to achieve a self goal and  government policies to meet public aspirations respectively. This multitudes of beneficiaries or stakeholders introduces complexities in the RE.

Not only this but the domain of practice and rapid technological advancement add to the very complexity too. Like - Technological advancements that are main contributors of solution space have been rapidly becoming so vast that to  keep RE professionals up-to-date with the knowledge of that space as its user and operational knowers is becoming out of reach. Instead it has lead to the class of “Specialists”. This brings “Contextual Knowledge Deficit” .

Others are easy to predict like economic and contractual forces. Such prefixed arrangements (contracts) and continuous urge to lead economic changes have inverse impact on the “time space” given for exploration in RE work. Combining both leads to the premature commitment to the statement of work (Specifications) that results into incomplete and/or arbitrary set of requirements.

Sometimes it has been argued that by deploying experienced requirement engineers in time critical situation can tackle the problem. But experience is more relative to the context and not with respect to the number of years of work experience. This becomes more evident in situations where RE works deals with the problems of conceptualization of new context rather than the problems where context boundaries are more or less going to be same - may with some new order. In this case “characterisation of (requirement) problems and collaboration” becomes more important than process or experience. Once characterisation of problem, lets say - Innovative, Architectural change, Change as per current architectural, has been done then it can be identified that which process, expertise and level of experience suits for speedy work.

Of course, eradicating origins is not an option ! But considering the origins of these forces which are dynamic, incremental in nature and going to prevail for longer time, a fitting response is needed to balance the complexity which is characterized as - Contextual Knowledge Deficit.  Hence, more democratic and participatory approach is needed, otherwise it keep on depriving stakeholders to form a consensus and to maintain conceptual integrity while implementing the change. Co-evolution of requirements with gradual bridging of contextual knowledge deficit can be helpful.

In co-evolution information model for defining problem (requirements) and fitting solution (actual software) develop and refine together with constant iterations of analysis, synthesis and evaluation processes. Figure 1 represents this approach where contextual deficit decreases with each iteration by interchanging information between the two spaces. After each iteration constraints/functions that defines the problems becomes clearer (P0 -> P1 -> P2...) and get refines. So with solution (S0 -> S1 -> S2...). It is expected that within finite iterations and time both spaces should become narrower enough where set of good solution meeting  problem definition can be chosen.

Before moving further it is important to get convinced that addressing Contextual Knowledge Deficit by co-evolution is taking us in right direction. A way to check this is - whether important (Software) development processes can give us hints of balancing these forces or not ?

Co-evolution in models

Waterfall Methodology (Royce and Boehm):

Royce's waterfall method advocates spending time on requirements in early phases of development where the actual conceptualization, requirements defining and Designing happens. This modular model was first presented as linear model with unilateral flow of information from top to bottom. May be it had been visualized that sufficient time is available to overcome the contextual Knowledge Deficit as part of RE work.

But soon it had been realized, through practise, that unidirectional flow is not suitable for all practical purposes - As all assumptions, preferences and intentions behind the goal can not be documented by requirement work in isolation to solution but need to be explored together. As discussion on solution features gives idea about what is not part of the problem function. Hence, one of the variation - Boehm waterfall methodology was proposed that has feedback or reverse information flow from lower layer to the layer one step above.

Arguably Bohem model is able to address the knowledge deficit between the adjacent layers (business context and  RE; ) directly and pseudo connectivity between layers separated by more than 1 step through feedback mechanism. It confirms to the theory discussed in complexity and gives some room for coevolution of requirements with design upto the level of awareness and discussion but not the actual output. Because this model still require frozen requirements before the design work.

Twin Peak Model:

Twin Peak model propose early architectural design formulation along with (system) requirement for rapid development combined with the improved analysis and planning.

This model has further strengthened the bond between the waterfall steps - Requirement and Design (Architectural) - and tries to reduces gaps in architectural understanding of requirement engineer and early identification of sources of set of good solution through co-evolution.

It adds to the notion that co-evolution of RE and work, that can give insight to the actual space of implementation (Design) is necessary for reducing time and the contextual knowledge deficit in technical designs.

Agile Methodology:

Agile methodology proposes incremental model for software development. Software is developed in incremental, rapid cycles with emphasis on embracing the change. Incremental and rapid cycles gives opportunities to test the feasibility of solution meeting the goals or not. Based on the findings, requirements can be changed (evolved) to address the misfits that were made explicit in earlier solution. This confirms the co-evolution argument. This is worth mentioning that though it seems to be a perfect case of co-evolution but on count of following points I argue its limitations -

• Lack of emphasis on necessary designing.
• Too much emphasizing the requirements as “to be” functionality stated by stakeholder. In cases where problems deals with software as service and multitudinous of stakeholders  Agile alone may not be suitable, sighting that, it may shape up as many cycles with no final agreed solution !

Spiral Model:

Arguably spiral model based processes are best suited to the client’s needs and at the same time it is very close to the co-evolutionary model in the sense that each new cycle can reflect upon the prototypes and understanding gained in previous cycle(s) thus reducing the contextual knowledge deficit in each cycle. RE do acknowledge the importance of prototyping and is used in eliciting requirements. By that way it too confirms the complexity argument.

Point here is not be critics of the methodology but to emphasize that with complexity RE take precedence over methodology and let RE to decide the course of co-evolution based on problem in hand. Mapping of models with level of co-evolution that they offer is shown in Figure 2.

Co-evolution as strategy

Convinced by the fact that contextual complexity are real that are undoubtedly going to be prevailing for long and considering the very high expectations and importance pinned on RE work - just to enumerate, not only design and implementation but high level cost estimation, verification method and test planning, deployment and fallback strategy are too dependent on this work - It could not go wrong! Such high expectation is not unnatural considering that RE work is the central work that provides information environment about problem and characteristics of acceptable solution. Indeed all these aspects are now have been part of change specification as well. This needs close alignment in understanding of problem and solution space.

Alignment is possible when solution lines are conceived and discussed during RE work. Context, Information flow patterns are designed and agreed upon at the same time. In other words based on the problem RE work should follow the methodology suitable for developing and refining solution aspects. With the refinement in requirements and coevolution of solution aspect eventually it should able give a fitting solution. In this way, RE involved right from the problem conception till aspects of good fitting solution are get finalized. In this process RE goes through multiple methodologies with different level of requirement specification (Business, function, system).

Aligning RE to support Co-evolution

As observed that development models gives opportunity to evolve requirements with varied degree of co-evolution. But these models are more effective if the resources and team structure functioning are themselves aligned to the concept of Co-evolution in consultation (expertise/ knowledge) and in practise. This raises a question of what are the characteristics that aligns RE structure to supports Co-evolution ?

(Here RE structure means the group of resources and processes responsible for RE work.)

One characteristic that fits in here is continuous focus on co-evolution based self-organized emergent behaviour that should continue to acquire - develop - manage knowledge to position themselves at the lower end of contextual knowledge deficit.

Without a doubt this work require to bring in lot of domain expertise and technical knowledge of current systems and related allied areas like - legal compliance, to find and align the misfits.

These alignments results into -

• Early problem formulation
• Less contextual shifts between problem and solution spaces.
• Discarding bad solutions in early phase.
• Identification of characteristics of good solutions.
• Estimation of investment required (IT and other areas).

Second characteristic is to have flexibility in adopting right process - right people to tackle the problem as per its nature. This shifts the attention of RE from process in which it has been practised to adopt the process that is good for the speedy problem-solution formation.

Aligning and packaging problems as per the expertise level of  requirement engineers and resources at disposal. This takes care of crunch in timespace.

For eg: If many requirement engineers above beginners level are available and problem level is of say P1 (in Figure 2). Division of problems into subproblems (functionalities) at conceptual level will enable parallel work. A twin peak model approach will ensure that conceptual integrity is maintained because the final object targeted here is a solution architecture (design) and it act as a binding force between the divided work.

Third characteristics is derived from the first two. Ability to weigh the complexity of problem. Of Course dividing a work is also a work! This is a matter of professional judgement. If not taken explicitly then the problem itself force it to realize. But at the cost time and efforts. As mentioned in the example above it requires foresightedness to realize the possible separate architectural components.  Solution Architects can lead the splitting of work.

A simplistic view of characterizing the RE problem -

Familiar and Relatively easy : These type of problems deals with minor modifications in the existing business rules. Like  to change the scope of rule or adding extra validation . Such problems does not add to the Information system architectural feature but can be justifiably accommodated within the current information system architecture.

Such problems can be deal by the requirement engineers with less experience in techno-function set-up.

New with medium-high complexity: These type of problems deals with major modifications in information flow and characterised by like creating a new application integrated with current set of applications. For eg. Transport booking application for replenishment based on the sales in stores. In this example Transport booking system is a new application that should integrate well with the existing replenishment and sales application. Such problems can be deal by the requirement engineers which are experienced in that techno-functional setup and can explore the some part of the technical solution on their own without relying on information from developers or architects.

High Complexity:  These type of problems deals with innovative ideas and context of operations are not fully known. For eg. Allowing customers to self scanning the articles, pay and go i.e. a  paradigm shift from cashier based sales to self scanning.  Such problems appears to be out of the present context, hence, framing of system requirements are not possible for such problems.Conceptualization of problem needs co-evolution of characteristics of new context; like - New way of working, new business processes, aligning present processes; and system outline to create a conceptual fit. Generally these changes are identified as new capabilities and values to the users. This does not necessary create a new class of users !

Fourth characteristic Deliberate (self) organization of knowledge : RE work itself is an act of knowledge creation or making things explicit. Like use case specification/realization document. But then why deliberate organization of knowledge ?

RE knowledge as profession lies in its practise oriented approach and judgements are based on preferences, domain constraints, previous decision which had actually resulted into current implementation.  Also considering complexity RE is no more a work of individual but team(s). This becomes a problem of sharing, organizing,measuring and improving  knowledge Index within RE team. The ultimate aim would be to position RE body to handle problems at the high spectrum of contextual knowledge deficit. This demands extra hours in reverse engineer the knowledge and reflection on previous designs. Another impetus is to embrace with the new techno-functional advancements in the domain.

This has two advantages -

• Time reduction in subsequent works
• Stability in the information referencing. As faces in team can change but knowledge remains.

Below activities can be useful -

• Self evaluation : Self rating in functional areas. Lets say out of 5. 1 new and learning, 2 Beginner. 3 Experienced 4. Can work independently and 5 can be consulted. This helps in getting feel of the knowledge deficit. These ratings can be supported by the measuring the number and quality  of knowledge contribution - Knowledge document creation, Document review, KT feedback, certifications (functional/technical).
• Regular knowledge transfer : Sharing knowledge within and from outside p can help in reducing the deficit.

Conclusion and future work

Context knowledge deficit is the outcome of multitudinous of stakeholders and rapid technological advancement . It has been argued that handling complexity demands co-evolution and some important development models confirms the fact.

A theoretical ground has been given to adopt co-evolution as strategy in RE to deal with the complexity. Essentially it emphasise on let RE lead the change through co-evolution and adopting methodology suitable to tackle the problem in hand. This is opposite of general development methodologies where RE seen as one of the important component with varied degree of co-evolution opportunity.

Contractual and deliverable aspects need to be explored when RE work passes from one methodological aspect to another. Requirement traceability could be challenge though a lot of work has been done on this with respect to agile.

Copyright 2015 Prashant Sharma (prashantjnvu@gmail.com)