Yesterday, I participated in the workshop on empirical requirements engineering. My notes on the talks will follow shortly, but I want to highlight the excellent panel that concluded the workshop. The panelists where Nan Niu, Jörg Dörr and Neil Maiden. Note, that the panelists were required to be provocative in their statements. Also note that I will add more links and references to this article later.
Panel: Confronting the challenges in empirical research in RE: What solutions worked well, what did not?
Nan Niu was the first panelist to defend his position on the question: What to avoid when doing empirical research on traceability? Accordingly, studies of [the role of] Human Analysts in Automated Requirements Tracing is difficult but needed.
They take the computed traceability matrix and mess it up!
Dealing with human actors introduces a magnitude of more challenges into the empirical research, but without understanding how and why humans need to adjust results from automatic requirements tracing, more technical research on their precision and recall is hard to interprete.
Next, Jörg Dörr defended his position about Experiment vs. Case Study and Repetition of empirical studies:
- Experiment is good for Research, but does not help to convince industry.
- Good quality of empirical research is not measured in p-value. It is measured by new insights.
- Threats do not reduce quality. But design flaws do and researchers should not mix them up with threats to validity.
- Jörg’s four reasons for replication are: “Almost got it”, “Changing Context”, “Eliminate Threat”, “You are stubborn”.
- He concluded with encouraging researchers to make remote participation possible by sharing their research agenda.
Finally, Neil Maiden defended his position in a way I can best capture by a series of quotes: “I am not sure we should do experiments the way I understand this term”. “Our role should be the one of designers”. “I went there and tried to change the way they work”.
Repeatability – why? We’re designing!
“There is only one Heathrow”. “We have underlying theories”.
Nan pointed out that there are different types of empirical researchers and showed a slide with four quadrants: What you accept as knowledge? (Reference in google-doc). He suggested to use it for telling people what your research is about. Nancy Mead suggested to use it also to inform yourself what to do. The discussion then focussed on how to locate the panelists position on the quadrants. Accordingly, researchers change their position over time or even based on research challenges.
There was a great discussion on the following question:
We have limited resources. Should we design new solutions? Should we focus on replications to offer help in choosing the right existing method?
Neil suggested that we need to maximise our impact as researchers by changing the state of practice. In contrast, Jörg reported from his consulting experience that there is a multitude of methodologies, practices and techniques out there and it is hard to suggest one. Accordingly, there is less value in adding the n-th modelling technique then in evaluating the difference between existing ones.
The next question, introduced by the audience, was:
What empirical evidence is there that we still need the Requirements Engineering Conference?
At first, the panelists struggled a little bit with answering the question. On the one hand, there is demand in industry, as repeatedly shown in documents like the chaos report. Also, we know that Software Managers and Practitioners regularly ask for Requirements Engineering in experience circles. On the other hand, this could also indicate that requirements engineering research does not answer the questions practitioners have. A nice argument pro RE conference is that it is a great community and that it is fun to contribute. Then, of course, we should ask who would finance participation at the conference with private money. Neil Ernst asked if requirements engineering is important at all or if you just need a great team. Jörg replied with a quote from the TwinPeaks (parallel workshop) Keynote:
You can solve each small project with smart people.
One key success factor is “Domain Knowledge” (Neil Maiden referred to a paper from 1987). The value of requirements engineering is then to give the tools to understand new domains and capture this domain knowledge.