Communication and Coordination for release management in software ecosystems

Recently, I had an opportunity to work with German, Leif, and Daniel on a very interesting study of the Gnome FOSS Ecosystem. We were interested to understand how release managers coordinate a shared release of a set of interrelated projects within the Gnome Ecosystem. In particular, we investigated:

  • Communication channels used…
  • How developers communicate and coordinate…
  • The key actors, the key tasks, and the most pressing challenges…

With respect to release management in the Gnome FOSS Ecosystem.

What I personally found most interesting are implications for coordinating activities across an ecosystem. In our case, release managers did not exhibit concrete power over developers. They could not just make people do things. This reminds me of some systems and systems as well as ecosystems that I see emerging in industry, where again it is not always clear who has the main responsibility.

In such a situation, our lessons learnt relate to soft factors more than technology. Specifically, we derive the following recommendations from our case study:

  1. Ensure that the release team follows the main communication channels used by developers. The release team needs to be in touch with the developers. IRC channels and face-to-face meetings in developer conferences were highlighted in our case study.
  2. Provide a common place for coordination for an ecosystem. In a globally distributed and multi-cultural environment, we found an asynchronous, global mailing list ideal, when complemented with individual, direct communication (face-to-face or IRC) with developers.
  3. Consider including both good technical and social skills in a release team. In short, technical skills are necessary to earn the respect of developers and to communicate with them, social skills are crucial to convince them to perform necessary actions.
  4. Aim for a diverse release team. Successful release managers can be seen as representatives of the different parts of the ecosystem. A good release team should mirror the diversity of its ecosystem.
  5. Based on lack of power, lobbying and consensus based management must be followed. A release team is well advised to keep close to key influencers in the ecosystem.

 

The Journal of Internet Services and Applications has accepted this work and published it open access.

author Germán Poo-Caamaño and Eric Knauss and Leif Singer and Daniel M. German
title Herding cats in a FOSS ecosystem: a tale of communication and coordination for release management
journal Internet Services and Application, 2017, 8(12), 1-24
abstract Release management in large-scale software development projects requires significant communication and coordination. It is particularly challenging in Free and Open Source Software (FOSS) ecosystems, in which hundreds of loosely connected developers and their projects are coordinated to release software to a schedule. To better understand this process and its challenges, we analyzed over two and half years of communication in the GNOME ecosystem and studied developers’ interactions. Through a case study, we cataloged communication channels, determined the main channel from which we categorized high level communication and coordination activities spanning five releases, and triangulated our results by interviewing ten key developers. We found that a release schedule, influence (instead of direct control), and diversity are the main factors that positively impact the release process in the GNOME ecosystem. We report a set of lessons learned that encapsulates our understanding of how the Release Management process function in a FOSS ecosystem, we learned that: (1) ensure that the release team follows the main communication channels used by developers, (2) provide a common place for coordination for an ecosystem, (3) consider including both good technical and social skills in a release team, (4) aim for a diverse release team, (5) based on lack of power, lobbying and consensus based management must be followed, (6) help the release team in the coordination process with a well defined schedule, and (7) release team work is different from regular software work. Our results can help organizations build better large-scale teams and show that research focused on individual projects might miss important parts of the picture.
doi 10.1186/s13174-017-0063-2
keywords release management, software ecosystems, empirical study
Advertisements

RE for Large-Scale Agile System Development

Featured

I am very proud that our project on requirements engineering for large-scale agile system development at the software center is now going into its second year. The software center focuses on speed based on the following themes: Continous DeliveryContinuous Architecture, Metrics, and Customer Data- and Ecosystem-Driven Development. The software center has seen requirements at the end.

Yet, we were able to form a consortium of six software center companies to investigate how current challenges with managing requirements related knowledge limit the speed and potential of continuous, agile system development. The current way of managing requirements in system development might be indeed at its end. Our goal is to uncover better ways that address these challenges while still allowing to fully leverage advantages of agile teams and continuous delivery.

themes

Fig. 1: Themes from our multiple case study relating to the scope of agile development (RQ1), the role of RE in large-scale agile system development (RQ2), and the challenges of RE for large-scale agile system development (RQ3).

We have now published an initial report on these challenges based on an exploratory multiple case study with four companies in Fall 2016 (see an overview of the challenges in Fig. 1 and details in the pre-print below). In this paper, we come to the following conclusions:

  1. Challenges of RE for large-scale agile system development relate to themes-origcommunication and knowledge management. While related work implies that communication challenges are mitigated by agile approaches and less prominent in agile RE, all our challenges relate to communication and knowledge management. Both aspects are at the core of Agile and RE, indicating a need for fundamental research in these areas specifically for system development.
  2. Challenges relate to two areas of requirements knowledge: User Value and System Understanding. While pre-agile RE approaches differ between user and system requirements specifications, we are not aware of related work that makes this distinction for RE in the scope of agile development. Surprisingly, we found that companies were not very interested in agile RE practices themselves. In contrast, they found it more important to understand how RE can support agile methods in large-scale system development and how agile development can be integrated with existing processes. Our findings indicate that such support cannot be offered sufficiently by traditional, upfront RE. This suggests that continuous and agile development methods on a large scale require new concepts.
  3. Challenges relate to the interplay of stake- holders from three domains: customer, development, and integration & testing. The development domain is generally embracing agility and characterized by a dislike for traditional requirements and bulk updates. The require better synchronization between teams and wish for establishing an agile tool-chain. In contrast, the customer domain is concerned with breaking down customer-visible features in order to communicate customer-value to team. They require better support for writing meaningful user stories and for bridging the gap between plan-driven and agile development. The integration and testing domain is struggling to create and maintain traces and with the fact that user stories and tests are not sufficient to build and maintain sufficient system understanding.
  4. In order to yield their full benefits, agile practices and a holistic system requirements model must be better aligned. Key challenges occur when there is an interaction, or a lack thereof, between the three domains above.

Since the writing of this paper, two more companies have joined our project and we have more closely investigated key challenges in the context of existing system engineering processes and requirements-related artifacts. While we are still working on extending our report on challenges of RE in Large-Scale Agile System Development, we are now, in the second year of this project, turning towards exploring the solution space.

Find more details in the following paper:

Reference: Kasauli, R.; Liebel, G.; Knauss, E.; Gopakumar, S. & Kanagwa, B.: Requirements Engineering Challenges in Large-Scale Agile System Development. In: Proc. of 25th Int. Requirements Engineering Conf. (RE ’17), Lisbon, Portugal, 2017

Pre-Print of the paper: Kasauli2017a

Abstract: Motivated by their success in software development, companies implement agile methods and their practices increasingly for software-intense, large products, such as cars, telecommunication infrastructure, and embedded systems. Such systems are usually subject to safety and regulative concerns as well as different development cycles of hardware and software. Consequently, requirements engineering involves upfront and detailed analysis, which can be at odds with agile (software) development. In this paper, we present results from a multiple case study with two car manufacturers, a telecommunications company, and a technology company that are on the journey to introduce organization wide continuous integration and continuous delivery to customers. Based on 20 qualitative interviews, 5 focus groups, and 2 cross-company workshops, we discuss possible scopes of agile methods within system development, the consequences this has on the role of requirements, and the challenges that arise from the interplay of requirements engineering and agile methods in large-scale system development. These relate in particular to communicating and managing knowledge about a) customer value and b) the system under development. We conclude that better alignment of a holistic requirements model with agile development practices promises rich gains in development speed, flexibility, and overall quality of software and systems.

On the Role of Fitness Dimensions in API Design Assessment

Next week, Imed Hammouda will present our joint work with Amir Zghidi and Brahim Hnich on API Design Assessment at the 1st International workshop on API Usage and Evolution, co-located with ICSE2017 in Buens Aires, Argentina. A pre-print of our paper is linked to this blog post.

This is clearly work in progress, based on our previous works on ecosystemability assessment and our realization that creating and maintaining suitable APIs is crucial for an organization’s ability to maintain a competitive edge in any software ecosystem. In particular, we distinguish technical and cognitive fitness dimensions, i.e. non-functional requirements that API Stakeholders may have. We anticipate that these fitness dimension will play an important role in the evolution of APIs, which we hope to support based on a continuous assessment method.

Title: On the Role of Fitness Dimensions in API Design Assessment – An Empirical Investigation

Abstract: In this paper we present a case study of applying fitness dimensions in API design assessment. We argue that API assessment is company specific and should take into consideration various stakeholders in the API ecosystem. We identified new fitness dimensions and introduced the notion of design considerations for fitness dimensions such as priorities, tradeoffs, and technical versus cognitive classification.

Reference: Zghidi, A.; Hammouda, I.; Hnich, B. & Knauss, E.: On the Role of Fitness Dimensions in API Design Assessment: An Empirical Investigation. In: Proceedings of 1st International Workshop on API Usage and Evolution (WAPI ’17), co-located with the 39th Int. Conf. on Software Engineering (ICSE 2017), 2017

Pre-print: Zghidi2017

Continuous clarification and emergent requirements flows in software ecosystems

I am very happy that Springer’s Requirements Engineering Journal has accepted our paper on continuous clarification and emergent requirements flows in open-commercial software ecosystems (open access to paper). This paper was joint work with Aminah Yussuf, Kelly Blincoe, Daniela Damian, and Alessia Knauss.

For me, the highlight of this paper is the visualization of emergent requirements flows throughout the ecosystem (see Fig. 1) as well as an analysis of the impact these emergent flows have with respect to our previous works on continuous clarification and RE in software ecosystems.

ecosystem

Fig. 1: Emergent communication across product teams and ecosystem actors. Size of nodes depicts number of emergent contributions by this stakeholder, color of links depicts comment type (yellow = requirements negotiation, green = coordination, brown = information).

Title: Continuous clarification and emergent requirements flows in open-commercial software ecosystems (open access to paper)

Abstract: Software engineering practice has shifted from the development of products in closed environments toward more open and collaborative efforts. Software development has become significantly interdependent with other systems (e.g. services, apps) and typically takes place within large ecosystems of networked communities of stakeholder organizations. Such software ecosystems promise increased innovation power and support for consumer-oriented software services at scale and are characterized by a certain openness of their information flows. While such openness supports project and reputation management, it also brings requirements engineering-related challenges within the ecosystem, such as managing dynamic, emergent contributions from the ecosystem stakeholders, as well as collecting their input while protecting their IP. In this paper, we report from a study of requirements communication and management practices within IBM®’s Collaborative Lifecycle Management® product development ecosystem. Our research used multiple methods for data collection, including interviews within several ecosystem actors, on-site participatory observation, and analysis of online project repositories. We chart and describe the flow of product requirements information through the ecosystem, how the open communication paradigm in software ecosystems provides opportunities for “just-in-time” RE—and which relies on emergent contributions from the ecosystem stakeholders—, as well as some of the challenges faced when traditional requirements engineering approaches are applied within such an ecosystem. More importantly, we discuss two tradeoffs brought about by the openness in software ecosystems: (1) allowing open, transparent communication while keeping intellectual property confidential within the ecosystem and (2) having the ability to act globally on a long-term strategy while empowering product teams to act locally to answer end users’ context-specific needs in a timely manner. A sufficient level of openness facilitates contributions of emergent stakeholders. The ability to include important emergent contributors early in requirements elicitation appears to be a crucial asset in software ecosystems.

Keywords: Requirements engineering; Software ecosystem; Mixed method

Reference: Knauss, E., Yussuf, A., Blincoe, K., Damian, D. and Knauss, A.: Continuous clarification and emergent requirements flows in open-commercial software ecosystems. In: Requirements Eng (2016). doi:10.1007/s00766-016-0259-1

Non-functional test oracle problem for continuous delivery

Our paper “Verdict Machinery: On the Need to Automatically Make Sense of Test Results” has been accepted at ISSTA 2016. It is based on Mikael’s and Emre’s Master thesis as well as on a great collaboration with Ericsson AB (see author list below).

Personally, I found this a particular challenging, yet rewarding piece of work, since it brings together aspects of non-functional system testing with process topics such as continuous delivery and quality requirements engineering. Thus, clearly articulating the problem has already been a challenge. But let me give an example:

issta-example

Example of performance goal and performance over a number of deliveries

The Figure shows a fictive example of a performance goal (e.g. number of simultaneous users supported – y-axis). The actual performance (red) changes with each of the 12 deliveries in the example (x-axis). Traditionally, a performance test would fail on delivery number 10, since performance now is below the performance goal (blue). However, this might be unfair and economically wrong.

  • It is unfair, since delivery 5 was actually far more problematic and if delivery 10 was delivered a bit earlier, it would have been accepted.
  • It might be economically wrong, since this delivery probably included good value for the customer, and anticipating performance improving change (delivery 12), a customer might have accepted a temporal breach of the performance goal in exchange for it.

A good verdict machinery in this fictive case should have rejected delivery 5 because of its unusually high performance degradation. Regardless of the concrete mechanism, without an automatic verdict, a large agile organization would need to agree on a case-by-case basis whether a delivery can be accepted or not. This would significantly lengthen time-to-market of all deliveries.

Title: Verdict Machinery: On the Need to Automatically Make Sense of Test Results

Abstract: Along with technological developments and increasing com- petition there is a major incentive for companies to produce and market high quality products before their competitors. In order to conquer a bigger portion of the market share, companies have to ensure the quality of the product in a shorter time frame. To accomplish this task companies try to automate their test processes as much as possible. It is critical to investigate and understand the problems that oc- cur during different stages of test automation processes. In this paper we report on a case study on automatic analy- sis of non-functional test results. We discuss challenges in the face of continuous integration and deployment and pro- vide improvement suggestions based on interviews at a large company in Sweden. The key contributions of this work are filling the knowledge gap in research about performance regression test analysis automation and providing warning signs and a road map for the industry.

Keywords: Non-Functional Testing Oracle; Verdict System; Performance regression test analysis; Automation

Reference: Fagerström, M.; Ismail, E. E.; Liebel, G.; Guliani, R.; Larsson, F.; Nordling, K.; Knauss, E. & Pelliccione, P.: Verdict Machinery: On the need to automatically make sense of test results. In: Proceedings of International Symposium on Software Testing and Analysis (ISSTA ’16), Saarbrücken, Germany, 2016

Pre-print: FIL+2016-ISSTA-Verdict_amchine-camera-ready.pdf

Continue reading

Herding Cats: Release Management in an Open Collaboration Ecosystem

Next week we will present our work on Release Engineering in the GNOME ecosystem at OSS conference in Gothenburg. This is joint work driven by Germán Poo-Caamaño, in collaboration with Leif Singer and Daniel M. German about release engineering, a growing field of interest within software engineering. We found the GNOME ecosystem particularly interesting to investigate, since such an open source ecosystem has different power structures than a software product within a company. Yet, we hope that our findings are relevant for software ecosystems in general, where the end-product is a combination of software products and services from different ecosystem actors. Especially ecosystems that are not dominated by one particular coordinator or platform owner could benefit from our findings on how GNOME release engineers interact with developers.

Title: Herding Cats – A Case Study of Release Management in an Open Collaboration Ecosystem

Abstract: Release management in large-scale software development projects requires significant communication and coordination. It is particularly challenging
in Free and Open Source Software (FOSS) ecosystems, in which hundreds of loosely connected developers and their projects need to be coordinated to release software to a schedule. To better understand this process and its challenges, we analyzed over two and half years of communication in the GNOME ecosystem and studied developers’ interactions. We cataloged communication channels, categorized high level communication and coordination activities in one of them, and triangulated our results by interviewing developers. We found that a release schedule, influence instead of direct control, and diversity are factors that impact positively the release process in the GNOME ecosystem. Our results can help organizations build better large-scale teams and show that research focused on individual projects might miss important parts of the picture.

Reference: Poo-Caamaño, G.; Singer, L.; Knauss, E. & German, D. M.: Herding Cats: A Case Study of Release Management in an Open Collaboration Ecosystem. In: Proceedings of 12th International Conference on Open Source Systems (OSS 2016), 2016

Pre-print: PSKG2016.pdf

Continue reading

Scaling up the Planning Game

Our paper on Scaling up the Planning Game: Collaboration Challenges in Large-Scale Agile Product Development has been accepted at XP conference 2016 in Edinburgh. In this joint work of Felix Evbota, Eric Knauss, Anna Sandberg we discuss how a large-scale agile organization can align views of developers and customers in order to incorporate agile values in their planning. For me, this is a particularly interesting topic because it helps (=is a first step) to understand how customer needs and requirements can and should be communicated in agile organizations.

Title: Scaling up the Planning Game: Collaboration Challenges in Large-Scale Agile Product Development

Abstract: One of the benefits of agile is close collaboration of customer and developer. This ensures good commitment and excellent knowledge flows of information about priorities and efforts. However, it is unclear if this benefit can be leveraged at scale. Clearly, it is infeasible to use practices such as planning game with several agile teams in the room. In this paper, we investigate how a large-scale agile organization manages, what challenges exist, and which opportunities can be leveraged. We found challenges in three areas: (i) the ability to estimate, prioritize, and plan; (ii) the context of planning with respect to working environment, team build-up, and team spirit; and (iii) the ceremonial agreement which promises to allow leveraging abilities in a given context.

Keywords: large-scale agile, planning, collaboration, communication

Pre-Print: EKS2016

Continue reading