Navigating the Invisible Hazard: How Deep Recurrent Reinforcement Learning and POMDPs Automate Orbital Collision Avoidance Under Sensor Uncertainty

Author(s): Aarush Mahajan Originally published on Towards AI. Navigating the Invisible Hazard: How Deep Recurrent Reinforcement Learning and POMDPs Automate Orbital Collision Avoidance Under Sensor Uncertainty An autonomous satellite executing a real-time thruster burn in Low Earth Orbit to navigate

Towards AIMontag 15:42 · 02.11.18 Min. Lesezeit

An autonomous satellite executing a real-time thruster burn in Low Earth Orbit to navigate space debris Author: Aarush Mahajan Primary Source: Spacecraft Autonomous Decision-Planning for Collision Avoidance: a Reinforcement Learning Approach (Bourriez, Loizeau, & Abdin, 2023) Thesis Question: Where do traditional Markov Decision Processes (MDPs) fail in space domain awareness, and how does a Partially Observable Markov Decision Process (POMDP) paired with Deep Recurrent Q-Networks (DRQN) enable autonomous satellite collision avoidance under severe tracking noise?,

Every second, the orbital infrastructure that powers modern civilization — GPS navigation, financial transaction synchronization, climate monitoring, and emergency disaster response — hangs in a delicate balance 2,000 kilometers above Earth.

Low Earth Orbit (LEO) is currently experiencing an unprecedented population explosion. Ground-based tracking networks actively monitor over 35,000 resident space objects larger than 10 centimeters. Alongside these trackable objects float an estimated 900,000 lethal fragments between 1 cm and 10 cm — objects far too small for radar networks to track consistently, yet carrying enough kinetic energy at hyper-velocities (7.8 km/s or ~17,500 mph) to obliterate an active satellite upon impact.

To prevent catastrophic conjunctions, global bodies like the U.S. Space Surveillance Network (SSN) continuously screen orbits and issue Conjunction Data Messages (CDMs). A single satellite operator in LEO receives hundreds of CDMs per week, detailing close approaches with uncertain miss distances and collision probabilities,.

Currently, collision risk mitigation relies on human-in-the-loop workflows,:

[Ground Radar Tracking] --- > [CDM Generation] --- > [Human Analyst Review (Hours/Days)] --- > [Uplink Maneuver Command] | Operational Bottleneck! (Cannot scale to mega-constellations) With commercial mega-constellations deploying tens of thousands of new satellites, manual analysis is reaching an unsustainable tipping point. Mega-constellation operators like SpaceX’s Starlink execute thousands of Collision Avoidance Maneuvers (CAMs) annually. If every satellite requires manual human oversight for every close approach, satellite operations will stall, response latency will spike, and human error under stress will lead to orbital impacts.

To protect humanity’s shared orbital environment, we must transition from manual ground-based control to on-board autonomous decision-making.

Takeaway — Section 1: Over 35,000 tracked space objects and 900,000 untracked lethal fragments threaten LEO infrastructure. Traditional human-in-the-loop collision avoidance cannot scale to the era of satellite mega-constellations, demanding on-board AI systems capable of autonomous decision-making,.

When Reinforcement Learning (RL) is applied to terrestrial robotics or games, researchers typically model the problem as a Markov Decision Process (MDP). In a standard MDP, the system assumes full state observability: at any time step t, the autonomous agent knows its exact state in state space S such as its precise 3D position, velocity, and orientation.

Under a fully observable MDP, an RL policy π(s) maps the exact environment state directly to an optimal control action in action space A. If space traffic management were a true MDP, a satellite’s flight computer could observe the exact positions and velocity vectors of all surrounding space debris and execute a single, deterministic thruster burn to dodge the collision.

+ -----------------------------------------------------------------------------------+ | FULL OBSERVABILITY ILLUSION (MDP) | + -----------------------------------------------------------------------------------+ | True State (s_t) ---> Agent Directly Observes (s_t) ---> Optimal Action (a_t) | | (Exact Debris Pos) (Zero Tracking Error) (Single Evasive Burn) | + -----------------------------------------------------------------------------------+ In Low Earth Orbit (LEO), however, true state observability is impossible,.

Space Situational Awareness (SSA) data is corrupted by two distinct, compounded layers of uncertainty,:

+ -----------------------------------------------------------------------------------+ | REAL - WORLD ORBITAL UNCERTAINTY | + -----------------------------------------------------------------------------------+ | True State (s_t) + [Aleatory Noise (Radar Jitter)] | | + [Epistemic Noise (SGP4 Drag / J2 Errors)] | | | | | v | | Noisy Observation (o_t) RECEIVED | + -----------------------------------------------------------------------------------+ Both aleatory and epistemic uncertainties are characterized statistically using Gaussian distribution models,. Because of this noise, a satellite never knows the true state; it only receives a noisy, corrupted observation in observation space O.

To understand why standard RL algorithms fail in space, consider a plain-English analogy:

The Highway Fog Analogy: Imagine driving a high-performance sports car down a highway at 17,500 miles per hour in pitch-black, dense fog. You cannot see through the windshield. Instead, your dashboard displays a flickering GPS screen that updates once every few miles with a random error margin of ±100 meters.

A standard MDP agent operates under the naive assumption that whatever appears on the GPS screen is 100% real. If the GPS glitches for a fraction of a second and shows a concrete wall directly in front of you, the MDP agent will slam on the brakes or jerk the steering wheel violently. In orbit, this leads to catastrophic thruster over-corrections that deplete the satellite’s limited fuel supply and steer the spacecraft directly into the path of another unmonitored piece of space debris.

To navigate safely through tracking noise without wasting thruster propellant, the AI cannot react to single, isolated radar observations. It must maintain a continuous temporal memory of past observations to construct a probabilistic “belief state” of where hazards are located over time,.

This requires transitioning from a Markov Decision Process (MDP) to a Partially Observable Markov Decision Process (POMDP),.

Takeaway — Section 2: Standard MDP reinforcement learning models fail in space because they assume full state observability. Real satellite telemetry is corrupted by aleatory sensor noise and epistemic SGP4 propagation errors,. To avoid violent thruster over-corrections, satellites must use POMDPs to filter noisy observations into temporal belief states.

To overcome the full observability trap of standard Reinforcement Learning, Bourriez et al. (2023) reformulate autonomous spacecraft collision avoidance as a Partially Observable Markov Decision Process (POMDP) and solve it using a Deep Recurrent Q-Network (DRQN).

A POMDP generalizes a standard Markov Decision Process to sequential decision-making problems where the agent cannot directly observe the true environment state. Formally, the onboard satellite collision avoidance agent is defined by the seven-tuple (S, A, T, R, O, Z, γ):

+-----------------------------------------------------------------------------------------------------+ | THE POMDP 7 -TUPLE FOR SPACE NAVIGATION | +-----------------------------------------------------------------------------------------------------+ | Element | Definition | Description in Bourriez et al. ( 2023 ) | +---------+--------------------------+----------------------------------------------------------------+ | S | Continuous State Space | True 3 D relative positions & velocities: (x, y, z, dx, dy, dz) | | A | Discrete Action Space | 625 impulsive thruster burn combinations ( 5 ⁴ action size) | | T | State Transition Function | Model-free state transitions governed by perturbed Keplerian | | | | orbit dynamics (Pykep engine) | | R | Multi-Objective Reward | Composite reward penalizing collision risk, fuel, & orbit drift| | O | Observation Space | Noisy continuous relative state vectors from sensors/radar | | Z | Observation Model | Probabilistic distribution Z(o|s ',a) combining Gaussian noise | | | | from sensor jitter (aleatory) and SGP4 drag errors (epistemic) | | γ | Discount Factor | Set to 0.99 for long -horizon trade- off evaluations | +-----------------------------------------------------------------------------------------------------+ 1. State Space (S) and Observation Space (O) The true state s_t in S represents the exact Cartesian position and velocity vectors of both the primary satellite and the secondary space debris in three-dimensional space:

s_t = [x_sat, y_sat, z_sat, dx_sat, dy_sat, dz_sat, x_deb, y_deb, z_deb, dx_deb, dy_deb, dz_deb]^T ∈ R¹² However, the agent never receives s_t. Instead, the observation model Z(o_t | s_t, a_{t-1}) generates a noisy observation vector o_t in O corrupted by zero-mean Gaussian noise N(0, σ²) simulating both SGP4 propagation uncertainty and tracking sensor noise:

o_t = s_t + e_t, where e_t ~ N ( 0, Σ_sensor + Σ_SGP4) 2. Discretized Action Space (A) To execute evasive maneuvers, the spacecraft modifies its velocity by applying short, impulsive thruster burns (Δv),. The action space is discretized into 5 possible thrust magnitude choices per dimension (0.0, 0.01, 0.05, 0.1, and -0.05 m/s) across 4 operational dimensions, yielding a discrete action space of size:

|A| = 5 ⁴ = 625 discrete actions 3.2 The Multi-Objective Reward Function (R) Designing a self-navigating satellite requires balancing competing operational goals: dodging debris, preserving fuel, and maintaining the satellite’s primary mission orbit. Bourriez et al. construct a multi-objective composite reward function governed by strict operational threshold values:

R ( s_t, a_t ) = R_collision + R_fuel + R_orbit_drift +---------------------------------------------------------------------------------------------------+ | MULTI-OBJECTIVE REWARD COMPONENTS & THRESHOLDS | +---------------------------------------------------------------------------------------------------+ | Component | Operational Threshold | Mathematical Penalty Applied | +--------------------+-----------------------+------------------------------------------------------+ | Collision Risk | Pc > 10⁻⁴ | Severe negative penalty if collision probability | | | | exceeds 10⁻⁴ (NASA operational safety threshold) | | Fuel Level | > 10 units | Negative penalty for every non-zero thruster burn; | | | | additional penalty if total fuel exceeds 10 units | | Trajectory Drift | Δa > 100 m | Proportional negative penalty for drifting from | | (Keplerian Elements| Δe > 0.01 | nominal osculating elements (a, e, i, Ω, ω) | | a, e, i, Ω, ω) | Δi, ΔΩ, Δω > 0.01 rad | | +---------------------------------------------------------------------------------------------------+ By enforcing penalties when the semi-major axis Δa, eccentricity Δe, inclination Δi, longitude of ascending node ΔΩ, or argument of periapsis Δω deviate beyond threshold bounds, the reward function prevents the AI from executing wild evasive burns that would render the satellite’s primary mission useless.

In a classical POMDP, an agent calculates a formal probability distribution over all possible true states, known as a belief state b_t(s) = P(s_t = s | h_t) using Bayes’ rule. However, updating exact Bayesian belief distributions over continuous 12-dimensional orbital space is computationally intractable for resource-constrained onboard flight computers.

Bourriez et al. bypass explicit belief calculations by integrating a Long Short-Term Memory (LSTM) layer directly into a Deep Q-Network, forming a Deep Recurrent Q-Network (DRQN).

+ ---------------------------------------------------------------------------------------------------+ | DRQN SOLVER ARCHITECTURE FLOW | + ---------------------------------------------------------------------------------------------------+ | | | Noisy Radar Observation (o_t) | | | | | v | | + ------------------+ Hidden Cell State (h_{t-1}, c_{t-1}) | | | LSTM Recurrent | & lt; ---------------------+ | | | Memory Layer | | | | + ------------------+ | (Implicit Belief State Update) | | | | | | | Output Hidden Vector (h_t) ------+ | | v | | + ------------------+ | | | Fully - Connected | | | | Linear Layers | | | + ------------------+ | | | | | v | | Q - Values for 625 Actions: [ Q(b_t, a_1), Q(b_t, a_2),..., Q(b_t, a_625) ] | | | | | v | | Action Selection: a_t = argmax_a Q(b_t, a) (ε - greedy during training) | + ---------------------------------------------------------------------------------------------------+ How LSTM Serves as an Implicit Belief Updater The hidden state h_t of the LSTM acts as an implicit belief state:

h_t = LSTM ( o_t, h_{t -1 }) Instead of throwing away past measurements, the LSTM cell retains a temporal history of previous observations h_t = (o_1, a_1, o_2, a_2, …, o_t). If a single radar measurement glitches due to sensor noise, the recurrent memory weights the new observation against past trajectory trends, maintaining a stable trajectory estimate and preventing thruster panic.

Training Deep RL in space environments introduces rare, extreme events — such as catastrophic collision penalties or sudden thruster firings. Standard Mean Squared Error (MSE) loss squares these error terms, producing massive gradient spikes that cause neural network training to diverge.

To stabilize learning, the DRQN utilizes the Huber Loss function (L_δ), which smoothly transitions from quadratic loss for small errors to linear loss for large outliers:

L_δ(a) = 0.5 * a² for |a| ≤ δ L_δ(a) = δ * (|a| - 0.5 * δ) for |a| > δ where a = Q_predicted — Y_target, and δ sets the threshold between quadratic and linear behavior.

HUBER LOSS vs MSE BEHAVIOR Loss ^ | / (MSE: Quadratic growth -> Gradient Explosion) | / | / | +----- (Huber Loss: Linear growth for outliers |a| > δ) | / | / | / +---+------------------------& gt; Prediction Error |a| & lt;-- δ --& gt; Quadratic Additionally, the architecture incorporates two essential training stabilizers:

Takeaway — Section 3: The POMDP framework models orbital uncertainty using a 7-tuple (S, A, T, R, O, Z, γ) with a 625-action space and a multi-objective reward function. By embedding an LSTM layer inside a DRQN, the satellite maintains an implicit belief state across past observations, while Huber loss prevents extreme collision penalties from destabilizing neural network convergence.

A primary challenge in training Deep Reinforcement Learning models for space traffic management is the lack of real-world collision datasets. Real collisions in Low Earth Orbit are rare, catastrophic events that destroy the assets involved, leaving no onboard telemetry for training. Public Two-Line Element (TLE) sets and historical Conjunction Data Messages (CDMs) provide close-approach warnings, but do not capture the post-conjunction flight dynamics required to train a decision agent,.

To solve this data scarcity problem, Bourriez et al. (2023) constructed a custom space flight simulator using the Pykep astrodynamics library. Pykep provides high-precision orbital mechanics computations under perturbed Keplerian dynamics.

+ ----------------------------------------------------------------------------------------------------+ | RETROGRADE COLLISION RECONSTRUCTION WORKFLOW | + ----------------------------------------------------------------------------------------------------+ | | | 1. Initialize Protected Spacecraft ---> Set Keplerian elements (a, e, i, Ω, ω, M) & radius | | | | 2. Sample Collision Epoch (t_c) ---> Draw t_c ~ Uniform(start_time, end_time) | | | | 3.

SGP4 Trajectory Projection ---> Propagate protected spacecraft to t_c using SGP4 | | | | 4. Instantiate Debris Position ---> Sample position ~ N(r_projected, σ_pos) | | | | 5. Retrograde Velocity Adjustment ---> Calculate collision trajectory velocity: | | v_new = v * N( 1, σ_vr) with rotated angle θ | | | | 6.

Backtrack Simulation ---> Propagate backwards to start_time to initialize episode | + ----------------------------------------------------------------------------------------------------+ The Retrograde Backtracking Algorithm To ensure every simulated environment contains a physically plausible conjunction event, the simulator uses a retrograde reconstruction method:

v_new = v * N( 1, σ_vr) Backtracking: The simulator backtracks both objects from t_collision to t_start, creating a fully deterministic conjunction episode where a collision will occur unless an evasive burn is applied. 4.2 Hyperparameter Tuning via Grid Search Deep Recurrent Q-Networks are sensitive to hyperparameter selection; small variations in replay buffer capacity or soft-update factors (τ) can cause neural network divergence. To establish a stable training baseline, the authors conducted exhaustive grid search sweeps across neural network hyperparameter spaces.

The agent’s performance was evaluated by tracking the Huber Loss trajectory and average cumulative reward profile across two experimental configurations:

LOSS CONVERGENCE TRAJECTORY Loss Value 0.35 + --* (Initial Random Actions: High Error) | \ 0.30 | \ | \ ---\ 0.20 | \ --------------+ (Single Env Convergence: ~0.26) | | 0.15 + -------------------------+========* (200 Envs Multi-Setting Loss: ~0.16) + -----------------------------------> Training Steps 0 10 k 20 k 30 k 40 k Key Experimental Findings: Loss Reduction: As training progressed over 40,000 simulation steps, the DRQN loss curve decreased monotonically from above 0.35 down to ~0.16 in the multi-environment setup. Generalization Advantage: Counterintuitively, the agent trained across 200 stochastic environments achieved lower overall loss (~0.16) than the agent trained on a single fixed environment (~0.26). Exposure to diverse approach vectors prevented the LSTM from memorizing single-trajectory overfits, encouraging the network to learn robust, generalizable belief state representations.

Operational Efficiency: The trained agent successfully executed collision avoidance maneuvers that reduced the collision probability below the 10⁻⁴ safety threshold while minimizing thruster fuel consumption and preserving the satellite’s primary Keplerian orbital elements. Takeaway — Section 4: By combining a Pykep-based retrograde collision simulator with SGP4 propagators, the authors generated synthetic conjunction scenarios to train the DRQN agent. Hyperparameter grid searches established that multi-environment training (across 200 distinct collision geometries) improved policy convergence, reducing Huber loss to ~0.16 while ensuring compliance with NASA safety thresholds.

While Bourriez et al. (2023) present a proof-of-concept for onboard autonomous decision-making, deploying Deep Reinforcement Learning directly into flight heritage software introduces critical open questions and engineering trade-offs:

+ ---------------------------------------------------------------------------------------------------+ | SIMULATED ASSUMPTIONS vs. OPERATIONAL REALITIES | + ---------------------------------------------------------------------------------------------------+ | Parameter / Dimension | Bourriez et al. ( 2023 ) Assumption | Operational LEO Reality | + ---------------------------+--------------------------------------+------------------------------------+ | Orbital Dynamics | Two - body Keplerian + Pykep | Non - Keplerian perturbations (J2, | | | | drag spikes, SRP, 3 rd body) | | Action Space | Discrete 625 impulsive burns (Δv) | Continuous electric propulsion | | | | low - thrust profiles | | Space Environment | Single & amp; static multi - debris scenarios | Dense, multi - conjunction clusters | | | | and repeating encounters | | Onboard Hardware | Unconstrained desktop GPU execution | Radiation - hardened limits | | | | (e.g., RAD750, ~ 200 MHz CPUs) | + ---------------------------------------------------------------------------------------------------+ 1. The Physics Gap: Idealized Two-Body vs. Unmodeled Perturbations The simulator relies on two-body Keplerian orbital mechanics combined with SGP4 propagation. In actual Low Earth Orbit, unmodeled environmental perturbations — such as solar flare-induced atmospheric density spikes that dramatically alter drag coefficients, solar radiation pressure (SRP), and higher-order geopotential gravitational harmonics — introduce physical distribution shifts.

A policy trained purely on idealized SGP4 trajectories risks failing when faced with severe, unmodeled drag surges.

The paper discretizes thruster actions into 625 impulsive velocity increments (Δv). However, modern satellite constellations are increasingly shifting toward low-thrust electric propulsion (EP) systems (such as Hall effect or gridded ion thrusters) due to their high fuel efficiency (I_sp). Electric propulsion cannot execute instant impulsive burns; it requires continuous, low-thrust burns spread over multiple orbital revolutions. Discretized Q-learning cannot easily model continuous, long-duration low-thrust trajectories, highlighting the need for continuous action space architectures (such as Soft Actor-Critic or PPO).

In space traffic management, satellite operators demand deterministic safety guarantees. Deep neural networks are notoriously uninterpretable; a pure Deep RL policy can output unexpected control actions when encountering out-of-distribution tracking inputs. Integrating deterministic safety shields — such as Control Barrier Functions (CBFs) or Reachability Analysis — around Deep RL policy loops remains an essential prerequisite before flight space agencies will trust an AI agent with multi-million-dollar orbital assets.

Takeaway— Section 5: Major gaps remain between simulated two-body Keplerian environments and operational LEO realities,. Overcoming the physics gap, adapting discrete actions to continuous electric propulsion low-thrust profiles, and embedding deterministic safety bounds around neural network policies represent the key open research frontiers in space AI.

+ -----------------------------------------------------------------------------------------------------+ | THE EVOLUTION OF SPACE SAFETY AI | + -----------------------------------------------------------------------------------------------------+ | Manual Ground Control ---> Sequence Forecasting (TCN) ---> Onboard POMDP / DRQN Agent| | (Hours/Days latency) (Early CDM Risk Warnings) (Autonomous Onboard Evasion) | | [Bourriez / GMV] [Tok et al. 2026 / HMMs] [Bourriez 2023 / Patnala 2024 ] | + -----------------------------------------------------------------------------------------------------+ Main Takeaways The LEO Crisis Demand: Rapid satellite constellation deployment makes manual human-in-the-loop collision avoidance unsustainable, mandating automated on-board decision-making,. Partial Observability is Essential: Space telemetry is corrupted by aleatory sensor noise and epistemic SGP4 propagation errors.

Formulating spacecraft navigation as a POMDP prevents thruster over-correction by reasoning under state uncertainty,. LSTM as Implicit Belief Updater: Integrating an LSTM layer into a DRQN allows spacecraft to maintain temporal memory over past noisy observations, constructing an implicit belief state without requiring intractable exact Bayesian updates. Generalization Matters: Training across 200 diverse, stochastic collision environments produces lower Huber loss (~0.16) and superior policy stability compared to training on single fixed environments.

The Road Ahead: What’s Next in Space Domain Protection The research landscape is advancing along three complementary pillars:

Takeaway — Section 6: The future of space domain awareness lies in combining onboard POMDP decision-making, sequence-based CDM forecasting, and formal safety verification,,. Transitioning these AI architectures from simulation to flight software ensures that Earth’s orbital infrastructure remains safe, sustainable, and resilient.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

15 engineers. 100,000+ students. Towards AI Academy teaches what actually survives production.

→ AI Engineering For Production — our new book, coming before the end of 2026. Join the waitlist.

→ Agents Architecture Cheatsheet — 3 years of architecture decisions in 6 pages

→ AI Engineering Certification — 90+ lessons from project selection to deployed product. The most comprehensive practical LLM course out there.

→ Agent Engineering Course — Hands on with production agent architectures, memory, routing, and eval frameworks — built from real enterprise engagements.

→ AI for Work — Understand, evaluate, and apply AI for complex work tasks.

Note: Article content contains the views of the contributing authors and not Towards AI.

Quelle: Towards AI (Towards AI). Original öffnen