OPEN-SOURCE SCRIPT
AetherEdge - PPO Policy Optimizer
🖊️ Overview
A self-evolving policy-gradient agent built on Proximal Policy Optimization (PPO) that optimizes its trading policy itself. A NeuraLib policy network outputs a softmax probability distribution over LONG / SHORT / FLAT, improved through a PPO-style clipped objective — the probability ratio between the new and old policy is clipped to [1−ε, 1+ε], so a single update can never shove the policy too far, the key to PPO's stability. Updates are driven by a virtual advantage estimate (GAE-style) computed against a learned value baseline. Learning from experience on your chart, it visualizes its evolving conviction with a policy-probability line (LONG %) and a trend line that recolors with the policy's tilt.
🔶 Key Features
PPO clipping — the probability ratio is clipped to [1−ε, 1+ε], structurally preventing oversized policy updates (the heart of PPO stability)
Actor-Critic architecture — a shared trunk forks into an Actor head (softmax policy) and a Critic head (value baseline)
Virtual advantage (GAE-style) — forward-return-minus-baseline smoothed over time for a low-bias, low-variance advantage signal
Entropy bonus — prevents premature collapse to a single action, encouraging exploration
Multi-epoch optimization — each collected minibatch is reused several times per bar (PPO's data efficiency)
Policy LONG % line — the policy's LONG probability plotted 0–100 (best on its own scale)
Color-shifting trend line — nudged up/down by policy tilt, recolored by dominant action and confidence
Intelligence panel — per-action probabilities, dominant action, value baseline, policy entropy, and rollout state at a glance
🧠 Technical Architecture
The agent perceives the market as a four-dimensional state vector: momentum spread (ATR-normalized fast/slow EMA gap), RSI deviation, a volatility-regime ratio, and position-within-range — all z-normalized. This state passes through a NeuraLib-style Actor-Critic network: a shared trunk (tanh hidden) forks into an Actor head producing three logits → softmax policy π(a|s), and a Critic head producing a scalar value V(s).
The virtual advantage follows GAE (Generalized Advantage Estimation). Iterating backward through the rollout, the TD residual δ = r + γV(s′) − V(s) is smoothed by γλ into  = δ + γλ·Â_next, then normalized to zero mean and unit variance (standard PPO practice).
The heart of the PPO update is the clipped objective. With the ratio r(θ) = π_new(a|s)/π_old(a|s), the objective is min(r·Â, clip(r, 1−ε, 1+ε)·Â). Outside the trust region — advantage positive with r above 1+ε, or negative with r below 1−ε — the gradient is zeroed, structurally bounding each update step. This combines with the policy gradient ∂logπ(a)/∂logit = 𝟙[a] − p, an entropy bonus, and the Critic's squared-error value regression (weighted by vfCoef), all backpropagated manually into the shared trunk. The collected rollout is re-optimized over several epochs and minibatches.
⚙️ Recommended Settings & Tuning Guide
BTC (1H–4H): Training Horizon 800–1500, Clip ε 0.2, GAE λ 0.95, γ 0.94, Epochs 3, Hidden 8. Standard PPO settings fit well
ETH (1H–4H): As BTC, with Reward Lookahead 5–8 to capture slightly longer advantage
SOL (15m–1H): High volatility favors Clip ε 0.15–0.2 (more conservative updates), Entropy ≈ 0.02 to strengthen exploration, ATR-normalization always ON
XRP (1H–4H): Spike-prone; GAE λ ≈ 0.9 to curb variance, Value Loss Weight 0.5–0.7 for baseline accuracy
Clip ε: smaller (0.1–0.15) is more conservative and stable; larger (0.25–0.3) learns faster but less stably — match to market stability
Learning Rate α: 0.02–0.04 is the stable zone; lower it if diverging, raise Epochs if convergence is slow
Entropy Bonus: raise to 0.02–0.05 if the policy biases too early to one side
💡 How to Use in Practice
Reacting to the LONG % line: 50% is neutral. A cross above 60% is a clearly bullish policy; below 40% is bearish. Drag it to its own scale to watch threshold breaks
Using the color-shifting trend line: when the line turns the LONG color and sits above price, it acts as a bullish-bias support line; the SHORT color makes it a bearish-bias resistance line
Policy tilt (flips): the moment the dominant action flips LONG↔SHORT, the trend line's color change is your directional-shift signal
Reading entropy: low panel entropy = the policy is confident; high = undecided. Low entropy plus a strong probability line marks the highest-confidence conditions
Multi-timeframe usage: confirm the big-picture policy bias on the higher timeframe (4H), then refine timing on aligned lower-timeframe (15m–1H) LONG %
Combinations: use divergence between price and the LONG % line as an early reversal warning
⚠️ Important Notes
Initial learning period: right after launch the rollout is nearly empty and both policy and value are unstable; treat signals as low-confidence until it fills (several hundred bars)
Learning resets: changing parameters, switching symbol/timeframe, or recompiling reinitializes the network weights and rollout, restarting learning from zero
On-policy nature: PPO is on-policy, so actions are sampled stochastically from the policy; selection can vary even in similar conditions — this is normal exploration
On forward-looking reward: rewards and advantages use closed-bar forward returns (a standard RL training construct); current-bar policy evaluation is on confirmed values, but as with any adaptive system, historical and live behavior can differ — always forward-test
Constraints: this is a lightweight implementation operating within Pine's compute budget
🚨 Disclaimer
This indicator is an analytical and educational visualization tool. The Proximal Policy Optimization, policy network, virtual advantage computation, reward shaping, and probability outputs are quantitative heuristics computed on-chart from price data — they are not financial advice, buy/sell signals, or any guarantee of future performance. Reinforcement-learning agents can and do make wrong calls. Always combine any tool with your own analysis and disciplined risk management.
A self-evolving policy-gradient agent built on Proximal Policy Optimization (PPO) that optimizes its trading policy itself. A NeuraLib policy network outputs a softmax probability distribution over LONG / SHORT / FLAT, improved through a PPO-style clipped objective — the probability ratio between the new and old policy is clipped to [1−ε, 1+ε], so a single update can never shove the policy too far, the key to PPO's stability. Updates are driven by a virtual advantage estimate (GAE-style) computed against a learned value baseline. Learning from experience on your chart, it visualizes its evolving conviction with a policy-probability line (LONG %) and a trend line that recolors with the policy's tilt.
🔶 Key Features
PPO clipping — the probability ratio is clipped to [1−ε, 1+ε], structurally preventing oversized policy updates (the heart of PPO stability)
Actor-Critic architecture — a shared trunk forks into an Actor head (softmax policy) and a Critic head (value baseline)
Virtual advantage (GAE-style) — forward-return-minus-baseline smoothed over time for a low-bias, low-variance advantage signal
Entropy bonus — prevents premature collapse to a single action, encouraging exploration
Multi-epoch optimization — each collected minibatch is reused several times per bar (PPO's data efficiency)
Policy LONG % line — the policy's LONG probability plotted 0–100 (best on its own scale)
Color-shifting trend line — nudged up/down by policy tilt, recolored by dominant action and confidence
Intelligence panel — per-action probabilities, dominant action, value baseline, policy entropy, and rollout state at a glance
🧠 Technical Architecture
The agent perceives the market as a four-dimensional state vector: momentum spread (ATR-normalized fast/slow EMA gap), RSI deviation, a volatility-regime ratio, and position-within-range — all z-normalized. This state passes through a NeuraLib-style Actor-Critic network: a shared trunk (tanh hidden) forks into an Actor head producing three logits → softmax policy π(a|s), and a Critic head producing a scalar value V(s).
The virtual advantage follows GAE (Generalized Advantage Estimation). Iterating backward through the rollout, the TD residual δ = r + γV(s′) − V(s) is smoothed by γλ into  = δ + γλ·Â_next, then normalized to zero mean and unit variance (standard PPO practice).
The heart of the PPO update is the clipped objective. With the ratio r(θ) = π_new(a|s)/π_old(a|s), the objective is min(r·Â, clip(r, 1−ε, 1+ε)·Â). Outside the trust region — advantage positive with r above 1+ε, or negative with r below 1−ε — the gradient is zeroed, structurally bounding each update step. This combines with the policy gradient ∂logπ(a)/∂logit = 𝟙[a] − p, an entropy bonus, and the Critic's squared-error value regression (weighted by vfCoef), all backpropagated manually into the shared trunk. The collected rollout is re-optimized over several epochs and minibatches.
⚙️ Recommended Settings & Tuning Guide
BTC (1H–4H): Training Horizon 800–1500, Clip ε 0.2, GAE λ 0.95, γ 0.94, Epochs 3, Hidden 8. Standard PPO settings fit well
ETH (1H–4H): As BTC, with Reward Lookahead 5–8 to capture slightly longer advantage
SOL (15m–1H): High volatility favors Clip ε 0.15–0.2 (more conservative updates), Entropy ≈ 0.02 to strengthen exploration, ATR-normalization always ON
XRP (1H–4H): Spike-prone; GAE λ ≈ 0.9 to curb variance, Value Loss Weight 0.5–0.7 for baseline accuracy
Clip ε: smaller (0.1–0.15) is more conservative and stable; larger (0.25–0.3) learns faster but less stably — match to market stability
Learning Rate α: 0.02–0.04 is the stable zone; lower it if diverging, raise Epochs if convergence is slow
Entropy Bonus: raise to 0.02–0.05 if the policy biases too early to one side
💡 How to Use in Practice
Reacting to the LONG % line: 50% is neutral. A cross above 60% is a clearly bullish policy; below 40% is bearish. Drag it to its own scale to watch threshold breaks
Using the color-shifting trend line: when the line turns the LONG color and sits above price, it acts as a bullish-bias support line; the SHORT color makes it a bearish-bias resistance line
Policy tilt (flips): the moment the dominant action flips LONG↔SHORT, the trend line's color change is your directional-shift signal
Reading entropy: low panel entropy = the policy is confident; high = undecided. Low entropy plus a strong probability line marks the highest-confidence conditions
Multi-timeframe usage: confirm the big-picture policy bias on the higher timeframe (4H), then refine timing on aligned lower-timeframe (15m–1H) LONG %
Combinations: use divergence between price and the LONG % line as an early reversal warning
⚠️ Important Notes
Initial learning period: right after launch the rollout is nearly empty and both policy and value are unstable; treat signals as low-confidence until it fills (several hundred bars)
Learning resets: changing parameters, switching symbol/timeframe, or recompiling reinitializes the network weights and rollout, restarting learning from zero
On-policy nature: PPO is on-policy, so actions are sampled stochastically from the policy; selection can vary even in similar conditions — this is normal exploration
On forward-looking reward: rewards and advantages use closed-bar forward returns (a standard RL training construct); current-bar policy evaluation is on confirmed values, but as with any adaptive system, historical and live behavior can differ — always forward-test
Constraints: this is a lightweight implementation operating within Pine's compute budget
🚨 Disclaimer
This indicator is an analytical and educational visualization tool. The Proximal Policy Optimization, policy network, virtual advantage computation, reward shaping, and probability outputs are quantitative heuristics computed on-chart from price data — they are not financial advice, buy/sell signals, or any guarantee of future performance. Reinforcement-learning agents can and do make wrong calls. Always combine any tool with your own analysis and disciplined risk management.
開源腳本
秉持TradingView一貫精神，這個腳本的創作者將其設為開源，以便交易者檢視並驗證其功能。向作者致敬！您可以免費使用此腳本，但請注意，重新發佈代碼需遵守我們的社群規範。
免責聲明
這些資訊和出版物並非旨在提供，也不構成TradingView提供或認可的任何形式的財務、投資、交易或其他類型的建議或推薦。請閱讀使用條款以了解更多資訊。
開源腳本
秉持TradingView一貫精神，這個腳本的創作者將其設為開源，以便交易者檢視並驗證其功能。向作者致敬！您可以免費使用此腳本，但請注意，重新發佈代碼需遵守我們的社群規範。
免責聲明
這些資訊和出版物並非旨在提供，也不構成TradingView提供或認可的任何形式的財務、投資、交易或其他類型的建議或推薦。請閱讀使用條款以了解更多資訊。