Integrating the RAPTOR foundation policy

[jonas-eschmann]

As requested by @mrpollo here I moved the Raptor implementation from an out-of-tree module into a proper in-tree module. It uses the external mode registration.

Raptor is an end-to-end foundation policy that takes position, orientation, linear velocity and angular velocity as inputs and outputs motor commands (actuator_motors). The policy is adaptive and can control virtually any quadrotor, zero-shot. We tested it on 10 different real quadrotors (from 30g to 2.4kg). More info can be found here: https://raptor.rl.tools/.

The goal of this integration is to provide a baseline for reinforcement learning-based quadrotor control research. This module is a robust baseline as well as a starting point for researchers and practitioners to train and deploy their own end-to-end control policies.

A highlight feature of this new implementation is that the neural network weights can be loaded from the SD card and (unlike the nn_mc_control module) do not need to be compiled into the firmware. This drastically reduces the flash usage and allows testing new neural networks without re-flashing the flight controller and remotely by transferring the checkpoint via Mavlink FTP.

The policy is trained for position control (controlling the position error to zero) but as we show in the paper also performs well for trajectory tracking (by offsetting position and linear velocity with the desired values from the trajectory). Hence, this module supports receiving trajectory_setpoint. The problem is that the forwarding of the trajectory setpoint from Mavlink is hardcoded to require the offboard mode in mavlink_receiver.cpp. Because of this we propose to add a request_offboard_setpoints into the mode registration request. In combination with minimal adjustments in the commander and mode management, this allows to forward the setpoint in a configurable way.

The neural networks are trained and run using RLtools, a pure C++, dependency-free, header-only library. It was created with a major focus on stability and is maintained by me. Compared to e.g. Tensorflow Lite I expect it to cause less issues going forward due to its simplicity.

We tested the previous implementation (OOT module) extensively on three different real platforms. This implementation shares major parts of the code with that but has not been tested on a real platform, yet (I will hopefully be able to test it next week). But I tested it in the SITL and SIH. These are the steps I use to run it:

SITL

Standalone Usage (Without External Trajectory Setpoint)

Build PX4 SITL with Raptor, disable QGC requirement, and adjust the IMU_GYRO_RATEMAX to match the simulation IMU rate

make px4_sitl_raptor gz_x500
param set NAV_DLL_ACT 0
param set COM_DISARM_LAND -1 # When taking off in offboard the landing detector can cause mid-air disarms
param set IMU_GYRO_RATEMAX 250 # Just for SITL. Tested with IMU_GYRO_RATEMAX=400 on real FCUs
param set MC_RAPTOR_ENABLE 1
param set MC_RAPTOR_OFFB 0

Upload the RAPTOR checkpoint to the "SD card": Separate terminal

mavproxy.py --master udp:127.0.0.1:14540
ftp mkdir /raptor # for the real FMU use: /fs/microsd/raptor
ftp put src/modules/mc_raptor/blob/policy.tar /raptor/policy.tar

restart (ctrl+c)

make px4_sitl_raptor gz_x500
commander takeoff
commander status

Note the external mode ID of RAPTOR in the status report

commander mode ext{RAPTOR_MODE_ID}

Usage with External Trajectory Setpoint

Send Lissajous setpoints via Mavlink:

pip install px4
px4 udp:localhost:14540 track lissajous --A 2.0 --B 1.0 --duration 10 --ramp-duration 5 --takeoff 10.0 --iterations 2

SIH

make px4_fmu-v6c_raptor upload

In QGroundControl:

• Airframes => SIH Quadrotor X
• Settings => Comm Links => Disable Pixhawk (disable automatic USB serial connection)

mavproxy.py --master /dev/serial/by-id/usb-Auterion_PX4_FMU_v6C.x_0-if00 --out udp:localhost:14550 --out udp:localhost:13337 --out udp:localhost:13338

New terminal (optional):

./Tools/mavlink_shell.py udp:localhost:13338

param set SIH_IXX 0.005
param set SIH_IYY 0.005
param set SIH_IZZ 0.010
param set IMU_GYRO_RATEMAX 400
param save
reboot

New terminal:

./Tools/simulation/jmavsim/jmavsim_run.sh -u -p 13337 -o

[hamishwillee]

So your raptor is implemented an external mode like offboard and you still want to publish trajectory setpoints? As long as this still works for offboard this is OK.

I don't understand what Raptor is - foundation policy sounds like some kind of corporate or academic doublespeak. Can you please explain it in little words (I'm a tech writer).

What does it do, who would want to use it, how can they enable it.

This also needs docs enough to answer that.

[jonas-eschmann]

Yes exactly, it uses the mode registration to register a new mode. I think this is the cleanest way to separate it from the other parts of the control stack and it was also requested by @AlexKlimaj here .

We chose the term "foundation policy" because the way it is trained and the way it works resembles foundation models. "Policy" is the standard term in the reinforcement learning community, it is equivalent to "controller" in the robotics community.

What does it do:
The idea is to have an adaptive policy, that can quickly adapt to any quadrotor by observing its response and adapting its control behavior. The goal is to get rid of the requirement for PID tuning and to be more robust wrt. disturbances or other changes in the dynamics. It is an end-to-end policy, meaning it takes state information (position, orientation, linear velocity, angular velocity) as input and directly outputs motor commands (actuator motors). Please refer to this video: https://www.youtube.com/watch?v=hVzdWRFTX3k and for more details to the paper: https://arxiv.org/abs/2509.11481

Who would want to use it:
Mainly academic use at this point (which is only the beginning), e.g. as a baseline for alternative control methods, but possibly also some commercial use-cases, where, e.g. the hardware (or payload) is changed often and the cost for tuning the controllers is prohibitive.

How can they enable it:
Please refer to the instructions: #26082 (comment). But I also plan to make a short video that shows how to use it step-by-step on a Holybro x500 v2 for the documentation.

I hope I find time to work on the docs in the next days. I'm conducting real-world flight tests for this PR right now

[hamishwillee]

Thanks very much. So this sounds a bit like https://docs.px4.io/main/en/advanced/neural_networks to me. That's just an observation, but it does mean we need to think about positioning this, and if they have overlap, grouping them.

[jonas-eschmann]

Sorry for the late response but I wanted to do some more thorough real-world testing. I have added documentation in docs/en/advanced/raptor.md, including an interactive playground to test the RAPTOR policy in simulation as well as a trajectory parameter generator app. Please let me know what you think

[hamishwillee]

FWIW this is the sort of stuff you'd have in the docs, then link to the docs from here if you need a readme.

[jonas-eschmann]

Added docs in docs/en/advanced/raptor.md

[Jaeyoung-Lim]

@SindreMHegre Could you have a look?

[SindreMHegre]

@SindreMHegre Could you have a look?

@Jaeyoung-Lim I think I have time for this tomorrow, will take a look then. But this sounds really promising, adding SD card storage of the network and sending setpoints over Mavlink are huge upgrades over my work! I tried a bit on the setpoints but weren't able to within the time limit of my work.

[SindreMHegre]

I did not have time to look into the details of the actual mc_raptor.cpp and hpp files. But it looks a lot like my module. I think it's really nice to have both options so I think its best to merge the documentation somewhat in the top script and then divide into to sub files for module specific information. If you have any specific questions @jonas-eschmann don't hesitate to ask, and it looks like you've already looked at the mc_nn_module, but if you need the full history you can find it here. It would be really cool to get the possibility to send mavlink setpoints to the mc_nn_module as well, but I can try to ping the lab at NTNU if it's not a quick fix possible to integrate into this PR. The same goes for storing the networks on the SD card, super usefull upgrade that would be great for the mc_nn_control. I still thinks mc_nn_control still has a place in PX4, as it supports using different frameworks and could still be used for other things than control! Really nice to see the possibilities for NNs in PX4 expand and improve!!!!

[SindreMHegre]

Alphabetically sorted?

[jonas-eschmann]

Good point! I wasn't aware they should be sorted. I sorted ./boards/sitl/raptor.px4board and boards/px4/fmu-v6c/raptor.px4board

[SindreMHegre]

alphabetically sorted?

[jonas-eschmann]

Sorted in e4ad0a

[SindreMHegre]

shouldn't message version be bumped here as well?

[jonas-eschmann]

Thanks a lot for the hint! I bumped the version and archived the old one

[SindreMHegre]

Also, if this can be used on Ubuntu versions prior to 24, then it should be clear in the docs, as the mc_nn_control module does not, so for those on older versions it would be great with a plug and play neural module

[Jaeyoung-Lim]

@SindreMHegre Wouldn't it be better to make the Aerial gym training work with the dynamic loading of neural networks?

For me I think it might be a good opportunity to get rid of the tfmicro dependency, and the convoluted CMake hacks we had to go through if we can make them share the neural network infrastructure.

[github-actions]

No flaws found

[jonas-eschmann]

@SindreMHegre Thanks a lot for the review! Your implementation, particularly using the external mode registration, was very useful as a guide! We have had an out-of-tree module solution (one-line modification to PX4 to use actuator_motors from our module's topic + out-of-tree modules that handle trajectory generation, inference etc.) for two years now but for upstreaming using the external mode registration is much cleaner!

Also, if this can be used on Ubuntu versions prior to 24, then it should be clear in the docs, as the mc_nn_control module does not, so for those on older versions it would be great with a plug and play neural module

This should work on any OS/version, as long as the C++ compiler supports C++17 (GCC/Clang/MSVC releases from 2017/2018 onwards).

In the meantime, I have also conducted extensive outdoor flying tests using mc_raptor for control (up to 17 m/s) and it is very reliable. I experienced zero hiccups and no crashes and think it is really stable and easy to use at this point.

Videos (links to YouTube):

I have also added relatively extensive documentation in docs/en/advanced/raptor.md

@Jaeyoung-Lim Please let me know how to proceed

[hamishwillee]

FYI, there is no convention for bool metadata - i.e. you can infer this from the field so it can't have a unit. That said, aguably this is more clear. Let's leave it in and I'll fix globally.

[hamishwillee]

@SindreMHegre We need to put these both under a parent topic, and the logical name for that topic is IMO "Neural Networks". That means we'll need to push your doc one deeper and give it a distinct name.

Would you be happy to rename your module top level "Multicopter Neural Network Module (NN)"? I can then move it as needed.

So it would look like

• Neural Networks - a parent topic linking to both these solutions
  • Multicopter Neural Network (NN) - your current top level topic
  • RAPTOR (or whatever name we use)

[hamishwillee]

By whatever name we use I mean something like "RAPTOR Adaptive RL Neural Module" or similar. Something that is accessible and findable by someone just learning about neural nets.

[jonas-eschmann]

I agree that it makes sense to structure it like you propose!
Maybe: "RAPTOR: Adaptive Neural Controller"? Or if you want to include RL: "RAPTOR: RL-Based Adaptive Neural Controller"?

[hamishwillee]

As discussed, the previous name "RAPTOR: A Foundation Policy for Quadrotor Control" is not suitable for this context.

It might make sense from an academic point of view where all your readers are machine learning experts, but it immediately makes what is otherwise an interesting technology far less likely to be discovered.

To "fix" this I have renamed to something that someone ignorant (i.e. me) might find useful for discovery). I have also added a two paragraph summary so that people can differentiate it from the other topic without having to read both in detail. I have tried to capture that this is a module that you can use with many frames without having to retune.

You are free to suggest an alternative - because I might be completely off base with what this is - but we should try and target non-experts as far as possible.

[jonas-eschmann]

Thanks a lot for your re-writing and editing, I think it is great! I would just sneak "Adaptive" into the title because the capability to adapt to a wide range of platforms (as you describe well) is the main point of distinction to other approaches and especially the existing NN-based control module.

[hamishwillee]

What is an internal reference? What is Position, What is Lissajous?

Do these matter? Or to put it another way, can the point of this section be dumbed down to be useful to a graduate engineer level - someone who knows about UAVs but little about neural nets.

[jonas-eschmann]

Good point! I have added clarifications on why the internal reference trajectory generator is needed and a brief description on what the Lissajous parameterization does. Essentially, it is used to e.g. generate smooth figure-eight trajectories of varying agility to test the control behavior and tracking error. I have also added a link to the Lissajous article on Wikipedia and added a reference to the embedded tool for adjusting the Lissajous parameters to see the resulting trajectories

[hamishwillee]

Again, I still really don't understand what a foundation policy is, and maybe 0.001% of people here will. You should either use other words or define it. Unless you don't want people to use this?

[jonas-eschmann]

Fair enough! We refer to it as a "foundation policy" because of the parallels to foundation models for language modelling (LLMs). In particular, the aspects of 1. Pre-training on a diverse dataset and 2. In-context adaptation. But I think adding these extensive explanations would bloat this documentation, which should probably focus on what it does and how it can be used. People who are interested in more details can then refer to the video and the paper for more details. So tldr: I agree and removed the "foundation policy" framing

[SindreMHegre]

@SindreMHegre Wouldn't it be better to make the Aerial gym training work with the dynamic loading of neural networks?

For me I think it might be a good opportunity to get rid of the tfmicro dependency, and the convoluted CMake hacks we had to go through if we can make them share the neural network infrastructure.

@Jaeyoung-Lim I agree that dynamic loading of neural networks is much better, I don't know if we should get rid of the old module just yet. My main goal was to make it easier for people who already has a reinforcment learning setup to deploy their policies to cheap COTS hardware to perform real world tests and easily modify anything they might want to. Please correct me if I'm wrong @jonas-eschmann but this module seems to be more tailored to the specific policy and rl-tools, I'm unsure of how easy it is to covert any pre-existing policy to the rl-tools format and then deploy it through this module. I did look into rl-tools when starting to make my module and it looked great (and probably even better now), but I wanted people to be able to use their existing frameworks and not have to switch. If this is possible with this module then there is no big reason to keep my original module, and I think this raptor module is better if you are starting from scratch.

This should work on any OS/version, as long as the C++ compiler supports C++17 (GCC/Clang/MSVC releases from 2017/2018 onwards).

That is the same as TFLM, so then it can probably only build out of the box on Ubuntu 24 and onwards. You can manually install a newer GCC version on older Ubuntu versions though.

@SindreMHegre We need to put these both under a parent topic, and the logical name for that topic is IMO "Neural Networks". >That means we'll need to push your doc one deeper and give it a distinct name.
Would you be happy to rename your module top level "Multicopter Neural Network Module (NN)"? I can then move it as needed.

@hamishwillee that sounds great, completely agree. And as you discussed above, the first landing page directly under the "Neural Networks" heading should explain the differences. I think the list under features for the rapor policy is nice to copy over to the landing page. To my understanding the raptor module is better if you are starting from scratch or want to play around with a pre-trained network (it looks like it performs a lot better on the x500 than my network did) you should go with the raptor module. If you have a pre-existing setup you want to use or a strong wish to use other frameworks, like PyTorch or TF then maybe the mc_nn_module is a better fit. Could possibly also add a link to my thesis in the mc_nn_control docs now that it's publically available, as this describes my process in great detail to anyone wanting to do something similar. https://nva.sikt.no/registration/019b26689144-efeebae8-84d6-4413-ad7f-9aceb4ff7374

Does the raptor module accept manual setpoints, as in using the sticks on a RC controller? That could potentially also be a difference as my module does accept it (although not recomended)