M2 research ckpt #00: project presentation

An integral part of the IP Paris Master in Cybersecurity is a final research project. And here we are! During the following months, I’ll be working on the project “Learning to Exploit a Backdoored RL Controller: Coordinated MARL Attacks in Dynamic 6G Cyber-Physical Systems”, supervised by PhD candidate Alex Pierron and Professor Joaquin Garcia-Alfaro. The idea is to use these posts to organize my ideas and keep track of the progress. In this very first checkpoint, I go into more detail about what exactly this project is about and our roadmap.

The project

A Reconfigurable Intelligent Surface (RIS) is a planar array of programmable metasurfaces capable of reflecting electromagnetic waves. This capability is especially interesting in the context of Beyond-5G/6G signals, where researchers are proposing to use this technology to redirect beams over obstacles and thus enhance network coverage.

As this technology relies on programmable entities, one major attack surface is the software running on it. If it contains a backdoor, for example, an attacker can take advantage of it to eavesdrop on or manipulate signals intended for legitimate users sharing this same infrastructure.

Simple RIS operation illustration, with a legitimate User Equipment (UE) and an eavesdropper. Image credits to Google Gemini.

In this project, we consider the simulation of a RIS containing a backdoor. For the role of the attackers, we consider a set of autonomous agents to be trained using reinforcement learning (Multi-Agent Reinforcement Learning, MARL). Our goal is to determine which training choices lead to effective coordination, with these agents succeeding in exploiting the backdoor. As an important additional step, we also want to work on the scenario where agents are not static, i.e., how their movement in space can affect the exploit. The idea is to use this knowledge to further enhance current proposals of RIS algorithms by establishing a simulation environment with capable attackers.

Therefore, in synthesis, the questions we propose to work on are:

• Which training parameters have the strongest effect on attack effectiveness, convergence, and stability?
• Are the best configurations robust across random seeds and initial conditions?
• What changes when legitimate and/or adversarial nodes move during an episode?

Roadmap

The project started in October 2026, and the final project defense is expected to happen on February 25th 2027. During these five months, our roadmap is the following:

Oct	Review core literature and work on the fundamentals of RL/MARL.
Nov	Reproduce the existing static scenario and establish baselines.
Nov-Dec	Work on the static scenario, characterizing attack coordination and which training configuration makes it possible.
Jan-Fev	Introduce mobility to the agents.
Fev	Final results and reports.

So far

During these first few weeks, I have been working on the fundamentals of Reinforcement Learning (RL), particularly through the book “Multi-Agent Reinforcement Learning: Foundations and Modern Approaches”¹ and through the visualizations offered in the “Reinforcement Learning By the Book” series of videos published by Mutual Information on YouTube.

In the book, I covered the chapters:

• Chapter 2 (“Reinforcement Learning”): Provides the initial basis for RL with a single agent, defining concepts such as the Markov Decision Process, what a policy is, the Markov property, state-value and action-value functions, and Bellman equations. It also introduces us to the first algorithms to compute optimal policies in MDPs, such as Dynamic Programming and the ones from the Temporal-Difference learning family, like Sarsa and Q-learning.

• Chapter 7 (“Deep Learning”): Motivates the use of neural networks as function approximators in RL, addressing limitations of previously mentioned tabular algorithms, such as the restricted state space size and limited generalization. Also provides a crash course on deep learning, including a glance at Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).

• Chapter 8 (“Deep Reinforcement Learning”): Effectively introduces the use of NNs in RL, first by approximating the action-value function (Deep Q-Networks, DQN) and inferring the policy from it, and after proposing the Policy Gradient Algorithms, that estimates directly the policy function relying on the policy gradient theorem to guarantee performance. Among these, they cite REINFORCE and Actor-Critic algorithms, including Advantage Actor-Critic (A2C), Trust Region Policy Optimization (TRPO), and Proximal Policy Optimization (PPO)², with the latter being the starting point for our project.

I also studied a Python implementation of PPO to get a little bit of practice among all this theory, and played with the Gymnasium lib, which provides a collection of environments for RL agents. It’s also worth citing the OpenAI Spinning Up documentation, which provides great material on Deep RL.

Next steps

With the established basis in Single-Agent Reinforcement Learning, the next step is to study Multi-Agent RL, and in particular Multi-Agent Deep RL. The target is to reach the Heterogeneous-Agent Proximal Policy Optimisation (HAPPO)³ algorithm, whose paper will be the subject of a presentation I’ll give to some classmates as a way to consolidate ideas and close this first stage. Excited for the upcoming work!

This post was made as a record of the progress of the research project “Learning to Exploit a Backdoored RL Controller: Coordinated MARL Attacks in Dynamic 6G Cyber-Physical Systems”, supervised by PhD candidate Alex Pierron and Professor Joaquin Garcia-Alfaro, in the IP Paris Master in Cybersecurity. Student supported by the France Excellence Eiffel scholarship, funded by the French Ministry for Europe and Foreign Affairs.

1. Stefano V. Albrecht, Filippos Christianos, and Lukas Schäfer. “Multi-Agent Reinforcement Learning: Foundations and Modern Approaches.” MIT Press, 2024. ↩︎

2. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. “Proximal policy optimization algorithms.” arXiv, 2017, doi:10.48550/arXiv.1707.06347. ↩︎

3. Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. “Trust region policy optimisation in multi-agent reinforcement learning.” In International Conference on Learning Representations (ICLR), 2022, doi:10.48550/arXiv.2109.11251. ↩︎