OPEN-SOURCE SCRIPT
AetherEdge - PPO Policy Optimizer
🖊️ Overview
A self-evolving policy-gradient agent built on Proximal Policy Optimization (PPO) that optimizes its trading policy itself. A NeuraLib policy network outputs a softmax probability distribution over LONG / SHORT / FLAT, improved through a PPO-style clipped objective — the probability ratio between the new and old policy is clipped to [1−ε, 1+ε], so a single update can never shove the policy too far, the key to PPO's stability. Updates are driven by a virtual advantage estimate (GAE-style) computed against a learned value baseline. Learning from experience on your chart, it visualizes its evolving conviction with a policy-probability line (LONG %) and a trend line that recolors with the policy's tilt.
🔶 Key Features
PPO clipping — the probability ratio is clipped to [1−ε, 1+ε], structurally preventing oversized policy updates (the heart of PPO stability)
Actor-Critic architecture — a shared trunk forks into an Actor head (softmax policy) and a Critic head (value baseline)
Virtual advantage (GAE-style) — forward-return-minus-baseline smoothed over time for a low-bias, low-variance advantage signal
Entropy bonus — prevents premature collapse to a single action, encouraging exploration
Multi-epoch optimization — each collected minibatch is reused several times per bar (PPO's data efficiency)
Policy LONG % line — the policy's LONG probability plotted 0–100 (best on its own scale)
Color-shifting trend line — nudged up/down by policy tilt, recolored by dominant action and confidence
Intelligence panel — per-action probabilities, dominant action, value baseline, policy entropy, and rollout state at a glance
🧠 Technical Architecture
The agent perceives the market as a four-dimensional state vector: momentum spread (ATR-normalized fast/slow EMA gap), RSI deviation, a volatility-regime ratio, and position-within-range — all z-normalized. This state passes through a NeuraLib-style Actor-Critic network: a shared trunk (tanh hidden) forks into an Actor head producing three logits → softmax policy π(a|s), and a Critic head producing a scalar value V(s).
The virtual advantage follows GAE (Generalized Advantage Estimation). Iterating backward through the rollout, the TD residual δ = r + γV(s′) − V(s) is smoothed by γλ into  = δ + γλ·Â_next, then normalized to zero mean and unit variance (standard PPO practice).
The heart of the PPO update is the clipped objective. With the ratio r(θ) = π_new(a|s)/π_old(a|s), the objective is min(r·Â, clip(r, 1−ε, 1+ε)·Â). Outside the trust region — advantage positive with r above 1+ε, or negative with r below 1−ε — the gradient is zeroed, structurally bounding each update step. This combines with the policy gradient ∂logπ(a)/∂logit = 𝟙[a] − p, an entropy bonus, and the Critic's squared-error value regression (weighted by vfCoef), all backpropagated manually into the shared trunk. The collected rollout is re-optimized over several epochs and minibatches.
⚙️ Recommended Settings & Tuning Guide
BTC (1H–4H): Training Horizon 800–1500, Clip ε 0.2, GAE λ 0.95, γ 0.94, Epochs 3, Hidden 8. Standard PPO settings fit well
ETH (1H–4H): As BTC, with Reward Lookahead 5–8 to capture slightly longer advantage
SOL (15m–1H): High volatility favors Clip ε 0.15–0.2 (more conservative updates), Entropy ≈ 0.02 to strengthen exploration, ATR-normalization always ON
XRP (1H–4H): Spike-prone; GAE λ ≈ 0.9 to curb variance, Value Loss Weight 0.5–0.7 for baseline accuracy
Clip ε: smaller (0.1–0.15) is more conservative and stable; larger (0.25–0.3) learns faster but less stably — match to market stability
Learning Rate α: 0.02–0.04 is the stable zone; lower it if diverging, raise Epochs if convergence is slow
Entropy Bonus: raise to 0.02–0.05 if the policy biases too early to one side
💡 How to Use in Practice
Reacting to the LONG % line: 50% is neutral. A cross above 60% is a clearly bullish policy; below 40% is bearish. Drag it to its own scale to watch threshold breaks
Using the color-shifting trend line: when the line turns the LONG color and sits above price, it acts as a bullish-bias support line; the SHORT color makes it a bearish-bias resistance line
Policy tilt (flips): the moment the dominant action flips LONG↔SHORT, the trend line's color change is your directional-shift signal
Reading entropy: low panel entropy = the policy is confident; high = undecided. Low entropy plus a strong probability line marks the highest-confidence conditions
Multi-timeframe usage: confirm the big-picture policy bias on the higher timeframe (4H), then refine timing on aligned lower-timeframe (15m–1H) LONG %
Combinations: use divergence between price and the LONG % line as an early reversal warning
⚠️ Important Notes
Initial learning period: right after launch the rollout is nearly empty and both policy and value are unstable; treat signals as low-confidence until it fills (several hundred bars)
Learning resets: changing parameters, switching symbol/timeframe, or recompiling reinitializes the network weights and rollout, restarting learning from zero
On-policy nature: PPO is on-policy, so actions are sampled stochastically from the policy; selection can vary even in similar conditions — this is normal exploration
On forward-looking reward: rewards and advantages use closed-bar forward returns (a standard RL training construct); current-bar policy evaluation is on confirmed values, but as with any adaptive system, historical and live behavior can differ — always forward-test
Constraints: this is a lightweight implementation operating within Pine's compute budget
🚨 Disclaimer
This indicator is an analytical and educational visualization tool. The Proximal Policy Optimization, policy network, virtual advantage computation, reward shaping, and probability outputs are quantitative heuristics computed on-chart from price data — they are not financial advice, buy/sell signals, or any guarantee of future performance. Reinforcement-learning agents can and do make wrong calls. Always combine any tool with your own analysis and disciplined risk management.
A self-evolving policy-gradient agent built on Proximal Policy Optimization (PPO) that optimizes its trading policy itself. A NeuraLib policy network outputs a softmax probability distribution over LONG / SHORT / FLAT, improved through a PPO-style clipped objective — the probability ratio between the new and old policy is clipped to [1−ε, 1+ε], so a single update can never shove the policy too far, the key to PPO's stability. Updates are driven by a virtual advantage estimate (GAE-style) computed against a learned value baseline. Learning from experience on your chart, it visualizes its evolving conviction with a policy-probability line (LONG %) and a trend line that recolors with the policy's tilt.
🔶 Key Features
PPO clipping — the probability ratio is clipped to [1−ε, 1+ε], structurally preventing oversized policy updates (the heart of PPO stability)
Actor-Critic architecture — a shared trunk forks into an Actor head (softmax policy) and a Critic head (value baseline)
Virtual advantage (GAE-style) — forward-return-minus-baseline smoothed over time for a low-bias, low-variance advantage signal
Entropy bonus — prevents premature collapse to a single action, encouraging exploration
Multi-epoch optimization — each collected minibatch is reused several times per bar (PPO's data efficiency)
Policy LONG % line — the policy's LONG probability plotted 0–100 (best on its own scale)
Color-shifting trend line — nudged up/down by policy tilt, recolored by dominant action and confidence
Intelligence panel — per-action probabilities, dominant action, value baseline, policy entropy, and rollout state at a glance
🧠 Technical Architecture
The agent perceives the market as a four-dimensional state vector: momentum spread (ATR-normalized fast/slow EMA gap), RSI deviation, a volatility-regime ratio, and position-within-range — all z-normalized. This state passes through a NeuraLib-style Actor-Critic network: a shared trunk (tanh hidden) forks into an Actor head producing three logits → softmax policy π(a|s), and a Critic head producing a scalar value V(s).
The virtual advantage follows GAE (Generalized Advantage Estimation). Iterating backward through the rollout, the TD residual δ = r + γV(s′) − V(s) is smoothed by γλ into  = δ + γλ·Â_next, then normalized to zero mean and unit variance (standard PPO practice).
The heart of the PPO update is the clipped objective. With the ratio r(θ) = π_new(a|s)/π_old(a|s), the objective is min(r·Â, clip(r, 1−ε, 1+ε)·Â). Outside the trust region — advantage positive with r above 1+ε, or negative with r below 1−ε — the gradient is zeroed, structurally bounding each update step. This combines with the policy gradient ∂logπ(a)/∂logit = 𝟙[a] − p, an entropy bonus, and the Critic's squared-error value regression (weighted by vfCoef), all backpropagated manually into the shared trunk. The collected rollout is re-optimized over several epochs and minibatches.
⚙️ Recommended Settings & Tuning Guide
BTC (1H–4H): Training Horizon 800–1500, Clip ε 0.2, GAE λ 0.95, γ 0.94, Epochs 3, Hidden 8. Standard PPO settings fit well
ETH (1H–4H): As BTC, with Reward Lookahead 5–8 to capture slightly longer advantage
SOL (15m–1H): High volatility favors Clip ε 0.15–0.2 (more conservative updates), Entropy ≈ 0.02 to strengthen exploration, ATR-normalization always ON
XRP (1H–4H): Spike-prone; GAE λ ≈ 0.9 to curb variance, Value Loss Weight 0.5–0.7 for baseline accuracy
Clip ε: smaller (0.1–0.15) is more conservative and stable; larger (0.25–0.3) learns faster but less stably — match to market stability
Learning Rate α: 0.02–0.04 is the stable zone; lower it if diverging, raise Epochs if convergence is slow
Entropy Bonus: raise to 0.02–0.05 if the policy biases too early to one side
💡 How to Use in Practice
Reacting to the LONG % line: 50% is neutral. A cross above 60% is a clearly bullish policy; below 40% is bearish. Drag it to its own scale to watch threshold breaks
Using the color-shifting trend line: when the line turns the LONG color and sits above price, it acts as a bullish-bias support line; the SHORT color makes it a bearish-bias resistance line
Policy tilt (flips): the moment the dominant action flips LONG↔SHORT, the trend line's color change is your directional-shift signal
Reading entropy: low panel entropy = the policy is confident; high = undecided. Low entropy plus a strong probability line marks the highest-confidence conditions
Multi-timeframe usage: confirm the big-picture policy bias on the higher timeframe (4H), then refine timing on aligned lower-timeframe (15m–1H) LONG %
Combinations: use divergence between price and the LONG % line as an early reversal warning
⚠️ Important Notes
Initial learning period: right after launch the rollout is nearly empty and both policy and value are unstable; treat signals as low-confidence until it fills (several hundred bars)
Learning resets: changing parameters, switching symbol/timeframe, or recompiling reinitializes the network weights and rollout, restarting learning from zero
On-policy nature: PPO is on-policy, so actions are sampled stochastically from the policy; selection can vary even in similar conditions — this is normal exploration
On forward-looking reward: rewards and advantages use closed-bar forward returns (a standard RL training construct); current-bar policy evaluation is on confirmed values, but as with any adaptive system, historical and live behavior can differ — always forward-test
Constraints: this is a lightweight implementation operating within Pine's compute budget
🚨 Disclaimer
This indicator is an analytical and educational visualization tool. The Proximal Policy Optimization, policy network, virtual advantage computation, reward shaping, and probability outputs are quantitative heuristics computed on-chart from price data — they are not financial advice, buy/sell signals, or any guarantee of future performance. Reinforcement-learning agents can and do make wrong calls. Always combine any tool with your own analysis and disciplined risk management.
סקריפט קוד פתוח
ברוח האמיתית של TradingView, יוצר הסקריפט הזה הפך אותו לקוד פתוח, כך שסוחרים יוכלו לעיין בו ולאמת את פעולתו. כל הכבוד למחבר! אמנם ניתן להשתמש בו בחינם, אך זכור כי פרסום חוזר של הקוד כפוף ל־כללי הבית שלנו.
כתב ויתור
המידע והפרסומים אינם מיועדים להיות, ואינם מהווים, ייעוץ או המלצה פיננסית, השקעתית, מסחרית או מכל סוג אחר המסופקת או מאושרת על ידי TradingView. קרא עוד ב־תנאי השימוש.
סקריפט קוד פתוח
ברוח האמיתית של TradingView, יוצר הסקריפט הזה הפך אותו לקוד פתוח, כך שסוחרים יוכלו לעיין בו ולאמת את פעולתו. כל הכבוד למחבר! אמנם ניתן להשתמש בו בחינם, אך זכור כי פרסום חוזר של הקוד כפוף ל־כללי הבית שלנו.
כתב ויתור
המידע והפרסומים אינם מיועדים להיות, ואינם מהווים, ייעוץ או המלצה פיננסית, השקעתית, מסחרית או מכל סוג אחר המסופקת או מאושרת על ידי TradingView. קרא עוד ב־תנאי השימוש.