Our paper “Verdict Machinery: On the Need to Automatically Make Sense of Test Results” has been accepted at ISSTA 2016. It is based on Mikael’s and Emre’s Master thesis as well as on a great collaboration with Ericsson AB (see author list below).
Personally, I found this a particular challenging, yet rewarding piece of work, since it brings together aspects of non-functional system testing with process topics such as continuous delivery and quality requirements engineering. Thus, clearly articulating the problem has already been a challenge. But let me give an example:
Example of performance goal and performance over a number of deliveries
The Figure shows a fictive example of a performance goal (e.g. number of simultaneous users supported – y-axis). The actual performance (red) changes with each of the 12 deliveries in the example (x-axis). Traditionally, a performance test would fail on delivery number 10, since performance now is below the performance goal (blue). However, this might be unfair and economically wrong.
- It is unfair, since delivery 5 was actually far more problematic and if delivery 10 was delivered a bit earlier, it would have been accepted.
- It might be economically wrong, since this delivery probably included good value for the customer, and anticipating performance improving change (delivery 12), a customer might have accepted a temporal breach of the performance goal in exchange for it.
A good verdict machinery in this fictive case should have rejected delivery 5 because of its unusually high performance degradation. Regardless of the concrete mechanism, without an automatic verdict, a large agile organization would need to agree on a case-by-case basis whether a delivery can be accepted or not. This would significantly lengthen time-to-market of all deliveries.
Title: Verdict Machinery: On the Need to Automatically Make Sense of Test Results
Abstract: Along with technological developments and increasing com- petition there is a major incentive for companies to produce and market high quality products before their competitors. In order to conquer a bigger portion of the market share, companies have to ensure the quality of the product in a shorter time frame. To accomplish this task companies try to automate their test processes as much as possible. It is critical to investigate and understand the problems that oc- cur during different stages of test automation processes. In this paper we report on a case study on automatic analy- sis of non-functional test results. We discuss challenges in the face of continuous integration and deployment and pro- vide improvement suggestions based on interviews at a large company in Sweden. The key contributions of this work are filling the knowledge gap in research about performance regression test analysis automation and providing warning signs and a road map for the industry.
Keywords: Non-Functional Testing Oracle; Verdict System; Performance regression test analysis; Automation
Reference: Fagerström, M.; Ismail, E. E.; Liebel, G.; Guliani, R.; Larsson, F.; Nordling, K.; Knauss, E. & Pelliccione, P.: Verdict Machinery: On the need to automatically make sense of test results. In: Proceedings of International Symposium on Software Testing and Analysis (ISSTA ’16), Saarbrücken, Germany, 2016
Next week we will present our work on Release Engineering in the GNOME ecosystem at OSS conference in Gothenburg. This is joint work driven by Germán Poo-Caamaño, in collaboration with Leif Singer and Daniel M. German about release engineering, a growing field of interest within software engineering. We found the GNOME ecosystem particularly interesting to investigate, since such an open source ecosystem has different power structures than a software product within a company. Yet, we hope that our findings are relevant for software ecosystems in general, where the end-product is a combination of software products and services from different ecosystem actors. Especially ecosystems that are not dominated by one particular coordinator or platform owner could benefit from our findings on how GNOME release engineers interact with developers.
Title: Herding Cats – A Case Study of Release Management in an Open Collaboration Ecosystem
Abstract: Release management in large-scale software development projects requires significant communication and coordination. It is particularly challenging
in Free and Open Source Software (FOSS) ecosystems, in which hundreds of loosely connected developers and their projects need to be coordinated to release software to a schedule. To better understand this process and its challenges, we analyzed over two and half years of communication in the GNOME ecosystem and studied developers’ interactions. We cataloged communication channels, categorized high level communication and coordination activities in one of them, and triangulated our results by interviewing developers. We found that a release schedule, influence instead of direct control, and diversity are factors that impact positively the release process in the GNOME ecosystem. Our results can help organizations build better large-scale teams and show that research focused on individual projects might miss important parts of the picture.
Reference: Poo-Caamaño, G.; Singer, L.; Knauss, E. & German, D. M.: Herding Cats: A Case Study of Release Management in an Open Collaboration Ecosystem. In: Proceedings of 12th International Conference on Open Source Systems (OSS 2016), 2016
Our paper on Scaling up the Planning Game: Collaboration Challenges in Large-Scale Agile Product Development has been accepted at XP conference 2016 in Edinburgh. In this joint work of Felix Evbota, Eric Knauss, Anna Sandberg we discuss how a large-scale agile organization can align views of developers and customers in order to incorporate agile values in their planning. For me, this is a particularly interesting topic because it helps (=is a first step) to understand how customer needs and requirements can and should be communicated in agile organizations.
Title: Scaling up the Planning Game: Collaboration Challenges in Large-Scale Agile Product Development
Abstract: One of the benefits of agile is close collaboration of customer and developer. This ensures good commitment and excellent knowledge flows of information about priorities and efforts. However, it is unclear if this benefit can be leveraged at scale. Clearly, it is infeasible to use practices such as planning game with several agile teams in the room. In this paper, we investigate how a large-scale agile organization manages, what challenges exist, and which opportunities can be leveraged. We found challenges in three areas: (i) the ability to estimate, prioritize, and plan; (ii) the context of planning with respect to working environment, team build-up, and team spirit; and (iii) the ceremonial agreement which promises to allow leveraging abilities in a given context.
Last year, we have started an initiative to better align teaching of agile methodologies at Chalmers | University of Gothenburg in various courses on the undergraduate and graduate level. During this work, we found that we are facing very similar challenges. Our discussions have lead to two publications, first at XP 2015 Conference where we received great feedback and encouragement and then at ICSE 2016 conference, where the paper attached to this post was very well received by the program board. We hope that you can join our discussion and our striving for excellence in agile methods education, either at ICSE next month in Austin, Texas or any time by contacting us directly.
This is joint work by Jan-Philipp Steghöfer, Eric Knauss, Emil Alégroth, Imed Hammouda, Håkan Burden, and Morgan Ericsson.
Title: Teaching Agile – Addressing the Conflict Between Project Delivery and Application of Agile Methods
Abstract: This paper analyses the changes we have made in teaching agile methodologies, practices, and principles in four courses in order to address a specific dilemma: students need to apply agile methods in order to learn them, but when complementing our courses with applied content, we face the problem that students perceive the learning and application of agile methods as less important than delivering a finished product at the end of the course. This causes students to not apply theoretical process knowledge and therefore to not develop necessary skills associated with working with defined processes in the industry. Concretely, we report on our experience with teaching Scrum with Lego, removing formal grading requirements on the delivered product, emphasising process application in post-mortem reports, and organisational changes to support the process during supervision. These changes are analysed in the context of student satisfaction, teacher observations, and achievements of learning outcomes. We also provide an overview of the lessons learnt to help guide the design of courses on agile methodologies.
Pre-print: SKA+2016 Continue reading
Early Bird Deadline approaching: Feb-26!
Call for Participation
REFSQ’16 will take place from March 14 -17, 2016 in Gothenburg, Sweden!
The REFSQ’16 special theme is:
Understanding an ever changing world through the right requirements.
- Magne Jørgenssen, Simula Norway
- Roel Wieringa, University of Twente, the Netherlands
21 papers will be intensively discussed in the REFSQ typical discussion sessions.
3 workshops featuring the topics of:
Posters & Demos Session
Keynote: Sarah Gregory, Intel, USA
7 Industrial Talks and a Discussion Panel
Research Methodology Track
This track features lively discussions giving you the opportunity to learn about research methods used in RE research.
Join us for join us for an unforgettable social Event on March 16, including a guided tour in Universeum, Gothenburg’s famous science park, and a delicious conference dinner in its aquarium.
The detailed program is available at: https://refsq.org/2016/conference-program/
The online registration service is available at: https://refsq.org/2016/event-information/registration/
I am proud to announce that we will host a mini-workshop, pre-ceeding the REFSQ 2016 PC Meeting in Gothenburg on Dec-3rd. Participation is free. More information and registration link.
AUTOSAR aims at facilitating reuse of standardized software components in automotive software development. For this, it defines three layers: the application components (with standardized interfaces between components), the (standardized) runtime environment, and the basic software (which abstracts the hardware and contains the driver modules). Despite this standardization, the combination of application components, runtime and basic software should still offer differentiating functionality for cars. In an exploratory study, we found this situation challenging for Requirements Engineering practice, where standard requirements implied by the AUTOSAR standard should be treated differently from OEM specific requirements targeted towards differentiation or innovation features.
We will discuss our preliminary findings at the 23rd IEEE International Requirements Conference 2015 in Ottawa, Canada during the poster and tool demo session as well as our method in the 5th IEEE International Workshop on Empirical Requirements Engineering.