Reliability of LLMs in Safety-Critical Requirements Engineering

Safety-critical systems live or die by their requirements: precise, unambiguous specifications that all downstream engineering depends on. As Large Language Models started entering engineering workflows, Fraunhofer IKS needed an evidence-based answer to a high-stakes question - do LLMs genuinely help engineers write better requirements, or do they introduce new failure modes such as over-reliance, subtle errors and misplaced trust?

Most published LLM evaluations stop at benchmark scores. They say little about how human engineers actually behave when a model sits in the loop, or how that behaviour shifts with the engineer's own experience. Answering that called for a controlled human study, not another leaderboard.

For my Master's thesis at Fraunhofer IKS I designed and ran an end-to-end study on the reliability of LLMs in safety-critical requirements engineering, carrying it from study design and ethics approval through to final data analysis.

I built and deployed a custom experimental platform in Streamlit that put participants through requirements-engineering tasks under controlled conditions, comparing LLM-assisted workflows against human-only ones. The platform captured each interaction so I could measure the LLM's impact on correctness, efficiency, user trust and learning rather than relying on self-reports.

Analysing the results showed a roughly 15% efficiency gain for experienced engineers working with LLM assistance - but the same assistance produced a clear over-reliance risk that grew as user expertise fell, with less-experienced participants more likely to accept flawed model output uncritically. The finding: the benefit of LLMs in safety-critical work is real, but conditional on user expertise.