The “ANYmal” is considered a success story in Swiss robotics: This dog-like robot uses sensors and cameras to inspect hazardous industrial sites. The autonomous machine is being deployed in an increasing number of gas and chemical plants both in Switzerland and abroad. An improved version, capable of performing repair work on its own, is already in development.

The ANYmal is part of a trend: physical autonomous systems – the umbrella term for AI-controlled machines – are booming. But with the rapid proliferation of robots, drones, and self-driving cars, concerns are also growing: What happens if they act uncontrollably and suddenly become a danger? There have already been initial incidents, ranging from the relatively harmless – such as when a Swiss Post drone crashed in a forest near Zurich in 2019 – to fatal ones, such as crashes involving self-driving cars.

Sebastiano Panichella’s job is to ensure that things don’t get that far. The researcher at the Institute of Computer Science at the University of Bern specializes in developing safety tests for physical autonomous systems. But the more advanced – and thus less dependent on direct human control – such systems become, the more new questions arise; for example, where human responsibility ends and that of the machine begins in the event of an accident. This is where Vera Hoffmann-Kolss comes in. She is an associate professor of theoretical philosophy. Her area of expertise is questions of causality and responsibility in complex scenarios involving many actors, known as multi-agent systems.

Sebastiano Panichella, Vera Hoffmann-Kolss, please give us some insight into your research.

Sebastiano Panichella: In my tests, I navigate robots or drones through virtual environments or place them on physical courses with unexpected obstacles to train them for similar situations in real-world operations. I also collaborate with manufacturers in the industry on this. The goal is to make autonomous systems safe enough that people won’t be harmed even if malfunctions occur.

Vera Hoffmann-Kolss: One example of the multi-agent systems I’m researching is climate change: This phenomenon has numerous contributors, ranging from entire nations to individual companies to single individuals. I’m investigating whether and how the contribution of these individual agents can be clearly identified so that responsibility can be assigned accordingly.

Sebastiano Panichella, if something goes wrong with an autonomous system, where does responsibility lie from an IT perspective?

Panichella: We usually attribute responsibility to people. That would include, for example, individual users who use an autonomous system improperly or even abuse it. However, the problem could also be due to poor design or be of a technical nature. In that case, the manufacturer would be responsible.

Could the machine itself be responsible? After all, it acts autonomously, doesn’t it?

Hoffmann-Kolss: Autonomy alone is not enough to attribute responsibility for its actions to a system. To do so, it would also have to be aware of the positive and negative consequences of those actions and make independent decisions. I don’t believe that current autonomous systems meet these criteria.

Panichella: We need to distinguish here between physical autonomous systems and so-called Large Language Models (LLMs) – that is, text-based AIs like ChatGPT or Gemini. The latter have passed the Turing Test multiple times. One could therefore argue that they make decisions and act independently.

In what ways are ChatGPT and similar systems more intelligent than robots or drones?

Panichella: If, for example, an industrial robot encounters a situation for which it wasn’t programmed – such as an unexpected obstacle – its system often crashes. Robots are generally optimized for specific environments and have only limited abilities to analyze new challenges.

If, on the other hand, you talk to an LLM about bananas and suddenly switch to the topic of weather, that’s no problem for this type of AI. These models are trained using massive amounts of data. They can therefore apply their insights to many areas and handle the unexpected situation of a topic change well. There are approaches to applying this principle to physical, autonomous systems as well – so-called vision-language models (VLMs). These combine visual input with language understanding and allow autonomous physical systems to respond more flexibly to unexpected situations. This development is still in its infancy, but it is advancing rapidly.

Hoffmann-Kolss: Nevertheless, even an LLM would not automatically be responsible for its actions. Ultimately, it behaves exactly as it was trained to. It has no autonomy in the sense of a personality with its own opinions and desires; rather, it mimics what it has been taught.

“YAn LLM has no opinions or desires of its own; rather, it mimics what it has been taught.”

Vera Hoffmann-Kolss

Panichella: I would argue that it’s similar with children: they learn by imitating the behavior of adults. Nevertheless, sooner or later they develop into independent individuals. As they grow up, we also entrust them with more and more responsibility.

Hoffmann-Kolss: In fact, responsibility is rarely black and white; rather, it often comes in degrees. Even an adult can sometimes only be held partially responsible for their actions. If autonomous systems were one day to learn in a manner comparable to that of children, it would be conceivable to apply this principle to their actions.

Does that mean we might one day be able to attribute 100 percent of the responsibility to a machine?

Hoffmann-Kolss: I find it difficult to predict this development. At the moment, however, I’m skeptical. I see the fundamental difference in the fact that a child will always possess an independent personality due to biological and social processes. A machine, on the other hand, is programmed. We set the limits of its actions – and thus its responsibility – and can, in principle, change these parameters.

Panichella: Unfortunately, it’s not always that simple. There have already been cases where artificial intelligence systems made decisions under laboratory conditions that the developers could neither foresee nor explain. The more complex an AI becomes, the more it resembles a black box. This challenge is also likely to arise increasingly with physical autonomous systems as they become more sophisticated.

Hoffmann-Kolss: But as soon as you realize that an autonomous system is developing in an unforeseen, potentially dangerous direction, you could shut it down.

Panichella: Not if development is moving so fast that people don’t notice it until it’s too late. For example, there was a recent case where an LLM refused to shut itself down. It pretended to shut down, but the programmers later realized it was still running. The incident itself is harmless; no one was harmed. However, the reason for the “refusal to obey” remains unclear: Did the AI simply copy human behavior it observed in training data, such as internet videos? Is there another, as yet unknown mechanism behind it? We don’t know for sure, and that’s troubling.

“We humans must be able to understand, at least to some extent, why a system makes a decision.”

Sebastiano Panichella

Are there any safety mechanisms in place to prevent such developments?

Panichella: I advocate developing artificial intelligence – whether LLMs or physical autonomous systems – from the ground up according to the “open box” principle. This means that we humans must be able to understand, at least to a certain extent, why a system generates a particular output or makes a particular decision. Safety is thus given greater consideration right from the programming stage, even if this comes at the expense of a system’s autonomy.

This requires legal guidelines and industry-wide technical standards.

Panichella: Absolutely. Unfortunately, the industry is currently driven largely by a competitive mindset, secrecy, and a focus on feasibility, while the political debate revolves primarily around the public’s fear of job losses. Issues of ethics, accountability, and responsibility are less of a focus. That’s why I’m very grateful for fruitful discussions like this one. As a representative of the technical side, they help me enormously to take new aspects of my work into account.

Hoffmann-Kolss: If autonomous systems that make decisions on their own were to actually exist one day, this would give rise to entirely new philosophical and ethical challenges, including the question of whether we should regard these machines as independent individuals. It is therefore imperative that we engage in cross-disciplinary discussions about how we, as a society, intend to steer this development.