Reliability of LLMs in Safety-Critical Requirements Engineering

Safety-critical systems stand or fall on their requirements: precise, unambiguous specifications that all downstream engineering depends on. As large language models entered these workflows, a common reflex followed - give the team a chatbot and tell them to use it.

The question nobody was asking: what does that most basic setup, a general-purpose assistant connected to nothing, actually do in safety-critical work? Most published evaluations stop at benchmark scores. They measure the model's output, not what the human does once the model is in the loop - and that is where the real risk sits. As NASA's report on the subject puts it, automation does not remove human fallibility; it relocates it to oversight roles. Answering that needed a controlled human study, not another leaderboard.

For my Master's thesis at TUM, in collaboration with Fraunhofer IKS, I designed and ran the whole study myself: experimental design, the ethics application, the software, data collection and statistical analysis.

1. I built the experiment platform from scratch. A Python and Streamlit web application carrying participants from consent form through a demographic questionnaire, two tasks and NASA-TLX workload measurement. It logged every step and every chat message with timestamps, so the measurement rested on records rather than self-reports.
2. The setup under test was deliberately the most basic one. The AI group got a chat panel embedded beside the task page - GPT-4o through a corporate interface. No AI system was engineered here: the model was not connected to the reference document the task was built on, could not read files, and had no access to any tool or data source. Participants were told this explicitly. That is what was measured: handing a team an off-the-shelf chatbot.
3. A two-phase, between-subjects design. In Phase 1 participants wrote seven safety requirements for a lane-keeping assistance system - one group with the chat panel, one without. In Phase 2 nobody had AI: everyone reviewed ten pre-written requirements, identified the flawed ones and corrected them. That second phase produced the central finding.
4. Quality was scored against a rubric, not by impression. Two domain experts independently rated each authored requirement across seven weighted criteria: structure, clarity, correctness, testability, scope alignment.
5. Ethics and data protection. Approved by the non-medical subgroup of TUM's Ethics Committee (2025-108-NM-BA). No personal data was stored; participants were represented by randomly generated identifiers.

Findings. The unconnected chat panel produced no measurable improvement in requirement quality (p = 0.96) and no time saving (p = 0.52). Instead, three things happened:

A cognitive shift. Participants using the bot reported lower mental demand (50.8 vs 61.3) and lower effort (36.7 vs 49.5) - but higher frustration (43.8 vs 30.7). The workload didn't drop, it moved: the blank-page effort of drafting disappeared and was replaced by the effort of verifying the output. The lowest-scoring item in the whole acceptance survey confirms it: participants did not find the tool flexible (2.83 out of 5).

Low trust. "I felt confident depending on the AI assistant" scored 2.00 out of 5, while the self-reported rate of double-checking was high. In a safety-critical setting that is good news: blind trust is not an automatic outcome. Users trust a tool that cannot show its sources exactly as much as it deserves.

The learning deficit - the central finding. In Phase 2, with the bot removed from everyone, the group that had used it averaged 54.7 out of 190; the group that never had it averaged 75.5. The participant who ranked first with the bot dropped to eleventh once it was taken away. The unassisted group had been forced to read the reference document themselves and had learned the material; the other group had handed the work to the tool, and when the tool went, so did the understanding.

How to read this result. The study does not show that AI is useless in safety-critical work. It shows something narrower and more useful: putting a chatbot next to a workflow is not, by itself, an AI solution. If you don't give the model the document the work is based on, don't make it cite its sources, and don't require it to stop when it doesn't know, what you get is not a measurable gain - it's a relocated workload and expertise that quietly erodes. Grounding the model in real sources (RAG) is precisely the untested variable the thesis names in its future-work section.

That conclusion shaped the work I've done since: building systems that read the source at query time and answer with the article behind it, rather than leaving the model to its own memory.

A stated limit: the power analysis called for 72 participants; institutional and time constraints capped it at 12. That reduces statistical power, so the results are presented as exploratory findings rather than proof - as they are in the thesis itself.

Stack: Python, Streamlit, GPT-4o (via Fraunhofer FhGenie), NASA-TLX and Technology Acceptance Model instruments, independent-samples t-tests.