ExecutionLabs is an algorithmic forex trading platform. The Signal Engine is the primary signal source, with a unified FastAPI server handling execution:

• Signal Engine — Server-side Python process that polls candles, runs the same strategy logic used in backtesting, and dispatches webhooks. All new strategies are deployed here. Runs as a separate systemd unit (signal-engine.service).
• FastAPI webhook server — A Python server deployed on Ubuntu/AWS that validates incoming signals, calculates position size, and routes to OANDA (live/demo accounts) or the paper engine (new strategies). Unknown account keys auto-provision to paper.
• TradingView strategies (retired 2026-03-13) — Pine Script v5 strategies that sent JSON webhook alerts. All TV alerts have been disabled; the signal engine is now the sole execution source. Pine files are kept in strategies/ as reference only.

Architecture Diagram

Key Infrastructure

• Server: Ubuntu (AWS EC2 t3.micro — 2 vCPU, 1 GB RAM, 20 GB EBS), SSH alias trading-bot, IP 18.216.115.163
• Swap: 1 GB swapfile at /swapfile — persistent via /etc/fstab. Provides OOM safety buffer for memory spikes.
• Domain: https://www.executionlabs.click — nginx + Certbot SSL
• Service: systemd unit trading-bot — auto-restarts on failure
• Database: SQLite WAL mode at /home/ubuntu/trading-bot/data/dashboard.db
• Ingest cron: runs every 5 minutes — calls OANDA transaction poller + risk budget refresh
• Strategy count: Dynamic — static config.py strategies merged with DB-backed dynamic_se_strategies at startup. New strategies run paper-first via the signal engine, promoted to live OANDA accounts after validation.

Python Dependencies

Main server: fastapi, uvicorn, python-dotenv, oandapyV20, jinja2, openpyxl. Backtesting additionally requires pandas >= 2.0.

Template Architecture

Pages use Jinja2 template inheritance. A shared base layout (templates/base.html) provides the HTML shell, nav bar, and instance bar. Each page extends the base and fills in page-specific blocks: {% block style %}, {% block content %}, and {% block scripts %}.

• static/common.css — shared design system: CSS variables, reset, nav, cards, tables, filter tabs, animations.
• static/common.js — shared JS: auth helpers (apiFetch), instance bar toggle, esc(), fmtPrice(), fmtUnits().
• templates/base.html — base layout with nav (active state via active_nav context variable), instance bar, and asset links.

Standalone pages (no base template): login.html, admin.html — served via FileResponse.

Route Table

Route	Page	Purpose
/	Dashboard	Main portfolio overview: KPI tiles (live P&L, demo P&L, win rate, alerts, portfolio health), Active Strategies card (strategies with open positions), All Strategies compact table.
/risk	Risk Dashboard	Per-strategy risk budgets, 90D profit factor, drawdown %, open exposure, quality scores. Status badges: healthy / watch / at-risk / disabled.
/strategy/{id}	Strategy Detail	Per-strategy activity: metrics, Lightweight Charts candle chart, open position, closed trade table, risk status, strategy health card.
/alerts	Execution Alerts	Webhook execution alerts table: webhook failures, order rejects, sizing errors, BE errors. Deduplication: same error within 24 h increments count rather than creating duplicate rows.
/portfolio	Portfolio Intelligence	State-machine status per strategy, weight allocator, correlation matrix, decay gauges, transition log. Triggers evaluation on page load.
/trades	Trade History	Closed trade table from OANDA transaction poller; filterable by environment.
/timeline	Timeline	Unified audit trail of every significant platform event: promotions, baseline registrations, portfolio state changes, ops events. Filterable by env and category. Recent Changes KPI tile on dashboard links here.
/ops	Ops Health	Execution health SLIs: webhook rate histogram (success/fail), reject reasons by stage, spread cost monitor. Provides a single-pane view of execution quality for both live and demo environments.
/admin	Admin Panel	Candidate strategy management (propose, approve, promote). Candidate approval auto-restarts the service (3 s delay via systemd-run) to load the new account config.
/signal-engine	Signal Engine Monitor	Real-time strategy polling status cards, shadow positions, events log, dispatch log, and condition diagnostics. Replaces the retired TV Parity view.
/kb	Knowledge Base	This documentation.

Webhook Payload Schema

{
  "secret":    "YOUR_WEBHOOK_SECRET",
  "symbol":    "USD_CHF",          // OANDA instrument format
  "action":    "buy" | "sell",
  "account":   "demo_usdchf_m15",  // maps to ACCOUNT_CONFIG in config.py
  "stop_atr":  1.2,                // stop distance as ATR multiplier
  "tp_atr":    2.5,                // take-profit as ATR multiplier
  "be_atr":    9.9                 // break-even trigger (9.9 = disabled)
}

Account Routing

The account field in the payload maps to an entry in ACCOUNT_CONFIG in config.py. Each entry specifies API key, OANDA account ID, environment (live/demo), symbol whitelist, risk_pct, and ATR granularity. Legacy demo strategies and live strategies use OANDA accounts from ACCOUNT_CONFIG (static) or dynamic_account_config (DB).

Paper-first (new strategies): If the account field doesn’t match any entry in ACCOUNT_CONFIG, the server auto-routes the order to the FX paper engine. No OANDA account allocation needed. This is the default path for all new forex strategies — the signal engine dispatches using the strategy ID as the account name.

Shared accounts (live): Multiple live strategies may share one OANDA account_id. Each order is tagged with tradeClientExtensions.id = account_key so the transaction poller can attribute closed trades to the correct strategy. Paper strategies always get their own paper accounts (no sharing).

Break-Even (BE) Monitor

A daemon thread polls all symbols every 30 seconds. When a trade's unrealized P&L exceeds be_atr × ATR pips, the stop is moved to entry price + 1 pip offset. Set be_atr = 9.9 to disable BE for a strategy (e.g., mean reversion, breakout strategies needing room).

Dashboard API (read-only)

Endpoint	Description
GET /api/summary	Portfolio totals, net realized P&L across all strategies.
GET /api/strategies	All strategies with entry counts, win rate, net P&L, status.
GET /api/orders	Recent order events from the webhook log.
GET /api/open_trades	Currently open OANDA positions.
GET /api/trades	Closed trades from OANDA transaction poller.
GET /api/metrics	Win rate, profit factor, P&L breakdown.
GET /api/equity	Balance snapshots over time.
GET /api/execution_quality	Avg spread costs by symbol (last 7 days).
GET /api/strategy/{id}/activity	Open trade + performance + closed trades + exec quality for one strategy.
GET /api/strategy/{id}/chart	OANDA candle data for the Lightweight Charts component.
GET /api/strategy/{id}/chart_overlays	Trade entry/exit markers for chart overlay (arrows + SL/TP lines).
GET /api/strategy/{id}/trades	Closed trades for a single strategy with realized P&L.
GET /api/strategy/{id}/metrics	Per-strategy performance metrics (PF, win rate, avg P&L, etc.).
GET /api/break_even/open	Open positions with break-even state + sizing lineage (LEFT JOIN).
GET /api/sizing_lineage/recent	Recent sizing decisions across all strategies. Params: env, asset, limit.
GET /api/strategy/{id}/sizing_lineage	Sizing lineage history for one strategy.
DELETE /api/strategy/{id}	Delete a strategy and cascade to pressure, pool, lease, and config tables. Requires no open positions.

Portfolio Intelligence API

Endpoint	Description
POST /api/portfolio/evaluate	Runs full portfolio evaluation: compute metrics, update states, calculate weights, fire decay alerts.
GET /api/portfolio/summary	Latest state counts and class exposure breakdown.
GET /api/portfolio/weights	Latest weight per strategy ordered by weight DESC.
GET /api/portfolio/correlation	Latest correlation pairs + cluster groupings.
GET /api/portfolio/decay_metrics	Latest health metrics per strategy with baseline comparison.
GET /api/portfolio/state_log	State transition history.
GET /api/strategy/{id}/health	Full health snapshot + 30-row history for one strategy.
POST /api/strategy/{id}/baseline	Register backtest baseline metrics (Sharpe, expectancy, max DD, win rate, PF, score).
POST /api/strategy/{id}/state	Manual state override (disable, retire, reset to healthy).

Alerts API

Endpoint	Description
GET /api/alerts	Execution alerts (filterable by stage, severity, acknowledged).
POST /api/alerts/{id}/ack	Acknowledge an alert.
POST /api/alerts/ack_all	Acknowledge all visible alerts.
GET /api/alerts/summary	Unacknowledged count + breakdown for the KPI tile.

Timeline API

Endpoint	Description
GET /api/timeline/summary	Event counts over last 24 h broken down by category (strategy, portfolio, risk, ops). Used by the "Recent Changes" dashboard KPI tile. Supports ?env=live|demo.
GET /api/timeline	Paginated event list. Params: env, category, strategy_id, limit (default 50), offset. Returns id, ts, category, event_type, env, strategy_id, summary, details_json.

Ops API

Endpoint	Description
GET /api/ops/summary	Single-payload ops health snapshot: last webhook timestamp, orders + alerts in last 24 h, reject rate (%), average half-spread 7D, duplicate alert count. Supports ?env=.
GET /api/ops/reject_reasons	Aggregated reject reason table from execution_alerts: stage, error_code, severity, total_count, symbols_affected, last_seen. Supports ?env=.
GET /api/ops/webhook_rate	Per-hour success/fail counts for the last 12 hours (bar chart data). Supports ?env=.

Crypto API

Endpoint	Description
GET /api/crypto/webhooks	Paginated crypto webhook events (log-only). Params: limit (default 50), offset. Returns {events, total, limit, offset}. Auth required.
GET /api/crypto/webhooks/summary	Crypto webhook KPIs: total (all-time count), total_24h (last 24 hours), last_ts (most recent event timestamp), by_symbol (count per symbol). Auth required.
GET /api/crypto/paper/summary	Per-strategy paper account: balance, equity, unrealized P&L, closed trade count, win rate. Param: strategy_id (optional).
GET /api/crypto/paper/positions	Open paper positions with mark-to-market data. Param: strategy_id (optional).
GET /api/crypto/paper/trades	Closed paper trades, newest first. Params: strategy_id, limit (default 50).
GET /api/crypto/paper/equity	Equity curve snapshots for chart rendering. Params: strategy_id, hours (default 168).
GET /api/crypto/paper/fills	Raw fill log (entries + exits). Params: strategy_id, limit (default 50).

Risk API

Endpoint	Description
GET /api/risk/summary	Portfolio-level risk snapshot: total exposure, unrealized P&L, regime, gross_cap_target, pressure status.
GET /api/risk/exposure_by_symbol	Aggregated exposure by instrument across all open positions.
GET /api/risk/exposure_by_strategy	Exposure breakdown per strategy with units, unrealized P&L, weight.
GET /api/risk/open_positions	All open positions with risk metrics.
GET /api/risk_budgets	All strategy risk budgets (90D quality scores). Supports ?env=&asset=.
GET /api/risk_budget/{id}	Single strategy risk budget detail.
POST /api/risk_budgets/refresh	Trigger risk budget recalculation.

Readiness & Promotion API

Endpoint	Description
GET /api/strategy/{id}/readiness	Returns a readiness assessment for demo→live promotion: ok (bool), missing list (blocking gates), warnings list (non-blocking), key_metrics dict. Gates: baseline registered, ≥20 demo trades, portfolio state not disabled/retired, risk budget not disabled, not already live.
GET /api/strategies/candidates	All deployed demo strategies with inline readiness check per strategy. Intended for the "Promotion Readiness" panel on the dashboard. Distinct from GET /api/candidates (backtest pipeline candidates).
POST /api/strategy/{id}/promote	User-auth-gated promotion to live. Runs readiness gate first; blocks if any gate fails. On success: allocates a live OANDA account from live_accounts_pool, releases any practice lease, updates strategies.status='live' and account_key, writes dynamic_account_config, schedules service restart. Paper-first strategies transition from paper engine to live OANDA execution at this step.
GET /api/strategy/{id}/attribution	Live vs. backtest attribution breakdown: entry hour histogram, average half-spread 90D, P&L by entry hour, outcome counts by exit reason (TP/SL/BE/manual).
POST /api/strategy/{id}/promote_live	Admin-gated force promotion. Requires force=true + override_reason (min 10 chars) + admin session token. Bypasses readiness gates but still returns readiness summary. Emits PROMOTION_FORCE_USED timeline event.
GET /api/strategy/{id}/promotion_status	Post-promotion status: promoted_at, old_account_key, new_account_key, stale_alerts_count_24h, stale_alerts_last_seen.
GET /api/strategy/{id}/account	Returns assigned OANDA account number + LIVE/PRACTICE badge for the strategy detail page.

Webhook Endpoint

Endpoint	Description
POST /webhook	Receives JSON payloads from the signal engine. Validates secret, routes to ACCOUNT_CONFIG, sizes position, executes OANDA order, logs to JSONL + SQLite. Position guard skips are also logged (with success=false, source tag).
GET /health	Returns {"status": "trading bot is running"}. No auth required.

Auth API

Endpoint	Description
POST /api/auth/bootstrap_first_user	Create the first user account (only works when no users exist).
POST /api/auth/login	Authenticate with username/password, returns session token (7-day expiry).
POST /api/auth/logout	Invalidate current session token.
GET /api/auth/me	Current user info (username, created_at).
POST /api/auth/change_password	Change password for the current user.

Admin API

Endpoint	Description
POST /api/admin/auth	Admin login (separate from user auth). Returns admin session token.
GET /api/admin/status	Admin session validity check.
GET /api/admin/users	List all users.
POST /api/admin/users	Create a new user account.
DELETE /api/admin/users/{user_id}	Delete a user account.
GET /api/admin/keys	View OANDA API key status (masked).
POST /api/admin/keys	Update API keys in .env and trigger restart.
GET /api/admin/pools	Practice + live account pool status (free/leased counts).
POST /api/admin/accounts	Add accounts to pools.
GET /api/admin/audit	Admin audit log (recent actions).
GET /api/candidates	Backtest pipeline candidate strategies awaiting approval.
POST /api/candidates	Submit a new candidate from the backtest pipeline.
POST /api/candidate/{id}/approve	Approve a candidate: creates strategy row, registers baseline, auto-writes a dynamic_se_strategies row, and schedules a signal engine restart. Requires registry_key on the candidate. FX strategies use paper-first execution (no OANDA account). Crypto strategies are registered in the signal engine with registry=fxexp (spot/experimental) or registry=perp (perp Donchian & siblings) automatically.
POST /api/candidate/{id}/reject	Reject a candidate with optional reason.

New strategies go through a local backtesting pipeline before being proposed to the admin portal. The pipeline runs on the development machine (24 cores) and fetches OANDA historical candles.

Multi-Window Scoring

Each combination is evaluated across four time windows simultaneously:

Window	Days	Weight	Min Trades
1Y	365	0.50	50
90D	90	0.35	20
30D	30	0.15	8
7D	7	0.00	0

The composite score = weighted sum of profit factor (capped at 3.0) across qualifying windows. window_score() returns 0.0 if any qualifying window is unprofitable — ensuring candidates are robust across all time horizons.

Spread Modeling

Each strategy class has a spread_pips attribute. The backtest engine deducts spread_pips × pip_size per round-trip trade. The effective spread is sampled from the live OANDA practice API via backtest/sample_spreads.py and stored in spread_config.json.

• p50_all: median spread across all sampled hours — conservative baseline
• p50_london: median spread during London session (08–16 UTC) — tighter
• p50_ny: median spread during NY session (13–21 UTC)

Strategies use p50_all as their effective spread (conservative). The cost stress test re-runs validation at 1.5× and 2× the base spread — confirming the edge survives realistic spread widening (news events, low-liquidity periods).

Deploy-Ready Gating

A candidate passes validation only if all of these gates pass:

1. Base 1Y profit factor ≥ 1.0
2. Base 90D profit factor ≥ 1.0
3. Cost stress 2× — 1Y profit factor ≥ 1.0
4. Cost stress 2× — 90D profit factor ≥ 1.0
5. Most-recent walk-forward fold profit factor ≥ 1.0
6. Stability neighborhood check — at least 1 neighboring param set also passes

Coarse→Fine Sweep Orchestrator

For strategies not yet in sweep.py, the coarse→fine orchestrator (backtest/coarse_fine_orchestrator.py) runs a two-stage parameter search, validates the winner, and posts directly to /api/candidates — all in a single command.

Stage	What happens
1 — Coarse sweep	Runs all Cartesian combos of a broad, sparse JSON grid. Retains top-K results by score.
2 — Fine sweep	Builds a ±1/±2-step neighborhood around the top-N coarse seeds. Capped at --max-fine combos (default 8,000). 1D neighbors of every seed are always preserved as must-have combos — guaranteeing neighbors_checked > 0 for the stability gate.
3 — Validation	Runs gate_cost_stress (1.5× and 2×) and gate_walk_forward from forex_validate.py, plus a custom stability gate using the coarse grid bounds.
4 — Pine + POST	Calls to_pine(), writes backtest/output/<strategy>_fine.pine, and POSTs to /api/candidates — only if all hard-fail gates pass and spread_src="effective".

Grid specs live in backtest/grids/ as JSON. Keys prefixed with _ are metadata and ignored:

{
  "_strategy": "usdchf_m15_london",
  "stop_atr":      {"type": "float", "values": [0.8, 1.0, 1.2], "min": 0.5, "max": 2.0, "step": 0.1},
  "range_bars":    {"type": "int",   "values": [6, 8, 10],       "min": 2,   "max": 20,  "step": 2},
  "session_end_h": {"type": "cat",   "values": [10, 12, 14, 16]}
}

• float / int — fine phase expands ±step from each seed value, clipped to min/max
• cat — fine phase uses adjacent list items as neighbors; no step or bounds

Backtest Pipeline Commands

Run from the trading-bot/ directory locally:

# Multi-window backtest
python backtest/run.py usdchf_m15_london

# Multi-window + generate Pine Script
python backtest/run.py usdchf_m15_london --generate-pine
# writes backtest/output/usdchf_m15_london.pine

# Parameter sweep (all CPU cores) — for strategies already in sweep.py GRIDS
python backtest/sweep.py usdchf_m15_london --top 20 --min-score 1.2

# Coarse->Fine orchestrator — for new strategies with a JSON grid spec
python backtest/coarse_fine_orchestrator.py \
    --strategy usdchf_m15_london \
    --coarse-grid backtest/grids/usdchf_m15_london_coarse.json \
    --top-k 25 --top-n 4 --max-fine 8000 --dry-run
# Remove --dry-run and add --api-url https://www.executionlabs.click to post on pass

Full Execution Flow

SQLite Schema (key tables)

Table	Purpose
strategies	Registry of all strategies (id, name, pair, timeframe, style, env, account_key).
order_events	Every webhook event ingested from trade_log.jsonl.
open_trades	Currently open positions (reconciled at each ingest run).
closed_trades	Closed trades from OANDA transaction poller with realized P&L.
strategy_risk_budgets	90D quality scores computed by risk_budget.py.
strategy_baselines	Backtest reference metrics per strategy (Sharpe, expectancy, max DD, etc.).
strategy_health	Rolling health snapshots from portfolio evaluation.
state_transitions	Audit log of state machine transitions.
correlation_snapshots	Pairwise Pearson r on daily P&L per evaluation run.
execution_alerts	Webhook execution errors with dedup key and occurrence count.
timeline_events	Unified audit trail: every promotion, baseline registration, portfolio state change, and ops event. Columns: id, ts, category, event_type, env, strategy_id, account_key, symbol, summary, details_json.
break_even_state	Server-side BE state per open trade: initial SL, R value, trigger price, buffer price, be_applied flag. Populated by be_manager.py each cron cycle.
sizing_lineage	Per-trade audit trail of weight-based sizing decisions: base/weight/gross_cap/effective risk_pct, clamp_reason, weight source, snapshot timestamp. See Weight Lineage.
equity_snapshots	Periodic account balance snapshots for equity curve charting.
poller_state	OANDA transaction poller cursor per account_key (last polled transaction ID). Shared accounts sync cursors after each poll cycle.
ingest_state	JSONL ingest cursor (last processed line offset).
crypto_webhook_events	Crypto webhook payloads stored in log-only mode (ts, symbol, action, account_key, response).
crypto_paper_accounts	Per-strategy paper account: starting_balance, balance, unrealized_pnl, computed equity. Created on first paper fill.
crypto_paper_fills	Every simulated fill (entry + exit) with fees, slippage, and position FK.
crypto_paper_positions	Open and closed paper positions with SL/TP/BE prices and mark-to-market state.
crypto_paper_closed	Closed paper trades with realized P&L, exit_reason (sl/tp/be_sl/manual), hold_bars.
crypto_paper_equity	Equity curve snapshots — one row per strategy per cron tick when positions are open.
users	User accounts for dashboard auth (PBKDF2 hashed passwords).
user_sessions	Session tokens with 7-day expiry.
candidates	Backtest pipeline candidate strategies awaiting admin approval. Includes registry_key column for signal engine auto-config on approval.
dynamic_se_strategies	Signal engine strategy configs auto-populated on candidate approval. Merged with static config.py at engine startup. Columns: strategy_id, asset_class, symbol, timeframe, mode, registry, registry_key, params_json, htf_json.
account_leases	Maps strategies to allocated practice/live pool accounts.
dynamic_account_config	Runtime ACCOUNT_CONFIG entries loaded at startup from approved candidates.
volatility_regime_state	Capital Pressure regime detection history (ATR pctl, ADX breadth, confirmed, chaos). 730-day retention.
strategy_pressure_adjustments	Per-strategy pressure multipliers and final weights per evaluation. 2-hour retention (snapshot table, pruned by ingest cron).
strategy_health	Portfolio health evaluations per strategy (Sharpe, state, drawdown). 2-hour retention (snapshot table, pruned by ingest cron).
market_daily_indicators	Cached daily OHLC + ATR14 + ADX14 per instrument. 400-day retention.

Position Sizing Formula

risk_usd  = balance × risk_pct / 100
stop_dist = ATR × stop_atr
units     = risk_usd / (stop_dist × quote_to_usd)

# quote_to_usd adjustments:
# Quote=USD pairs (EUR_USD, GBP_USD, AUD_USD): quote_to_usd = 1.0
# Base=USD pairs (USD_CAD, USD_JPY): quote_to_usd = current_price

# Hard cap: MAX_UNITS = 500,000
# Live:        risk_pct = 1.0%  (real account balance)
# OANDA demo:  risk_pct = 1.0%  (balance capped at virtual_balance = $1,000)
# Paper:       risk_pct = 1.0%  (STARTING_BALANCE = $1,000)
# All accounts size to ~$10/trade on a $1,000 base — mirrors a real $1k live account

Weight-Based Position Sizing

When ENABLE_WEIGHT_BASED_SIZING=True (default), the base risk_pct is scaled by the strategy's current portfolio weight before position sizing. This automatically reduces exposure for strategies in poor health or during volatility compression, and increases it (up to the base cap) during expansion.

effective_risk_pct = base × weight × gross_cap_target × safety_scalar

# weight source priority:
#   1. strategy_pressure_adjustments.final_weight  (pressure-adjusted; gross_cap already baked in)
#   2. strategy_health.weight                       (pre-pressure fallback)

# gross_cap_target: 1.0 normally; 1.2 in confirmed Expansion
#   applied only for the health fallback — pressure source already embeds it

# safety_scalar: 0.0 if state = disabled | retired  →  BLOCKS the trade
#                1.0 otherwise

# floor: MIN_EFFECTIVE_RISK_PCT = 0.01%  (never goes below this)
# cap:   CAP_AT_BASE_RISK_PCT = True     (weight cannot push above base)

Missing / stale weights (older than WEIGHTS_STALE_MAX_AGE_MINUTES=120): controlled by MISSING_WEIGHT_BEHAVIOR:

• "fallback" (default) — trade fires at base × FALLBACK_WEIGHT (5% of base, i.e. 0.05% on the standard 1% base). Ensures signals are never silently dropped.
• "block" — returns HTTP 200 {"ok": false}. The signal is discarded.

Blocked trades (disabled/retired state) always emit a WEIGHT_BLOCK alert visible on the /alerts page regardless of MISSING_WEIGHT_BEHAVIOR.

Weight Lineage

Every order fill writes a row to the sizing_lineage table — a per-trade audit trail of how effective_risk_pct was derived. The trade_id is {oanda_account_id}:{tradeID}, matching open_trades.opened_trade_id exactly.

Column	Description
trade_id	Primary key — compound OANDA ID matching open_trades
base_risk_pct	Raw risk_pct from ACCOUNT_CONFIG before weight scaling
strategy_weight	Portfolio weight applied (NULL if sizing disabled or weight missing)
gross_cap_target	Reported gross cap (1.0 or 1.2); applied in formula only for health source — pressure source has it baked in
effective_risk_pct	Final value passed to calc_units()
clamp_reason	none | floored_to_min | missing_weight_fallback | weights_stale_fallback | state_disabled_zero
weight_source	pressure | health | fallback | disabled
query_env	Actual env used to look up weights (may differ from trade env when falling back to global eval)
weights_ts	Timestamp of the weight snapshot used

Where to see it: the Trades page open-positions table shows a Sizing column with an inline badge (hover for the full lineage tooltip). The Strategy Detail page shows a "Recent Sizing Decisions" card. The raw data is available via GET /api/strategy/{id}/sizing_lineage and GET /api/sizing_lineage/recent.

The Portfolio Intelligence layer runs on-demand (triggered by page load at /portfolio or manually via the "Run Evaluation" button). It evaluates each strategy's live performance against its registered backtest baseline and assigns a state + normalized weight.

Strategy State Machine

Each strategy moves through these states based on its live performance vs. baseline:

State	Trigger Condition	Weight
Probation	< 20 closed trades — insufficient data	Fixed 5%
Healthy	All ratios within tolerance	Normalized score
Watch	DD stress > 1.2×, or expectancy < 70%, or Sharpe < 70% of baseline	Score × 0.6
Degrading	DD stress > 1.5× or expectancy < 50% of baseline	Score × 0.3
Disabled	DD stress > 2.0× or expectancy < 30% of baseline	0 (excluded)
Retired	Manual — permanent exit	0 (excluded)

Weight Allocation

Normalized weights are computed as follows:

1. Probation/disabled/retired strategies get 5% / 0% / 0% respectively.
2. Active strategies get raw_score = quality_score × state_multiplier.
3. Correlation penalty: For pairs with |r| > 0.60, the lower-scored strategy's weight is multiplied by 0.70.
4. Class cap: No strategy class (trend/mean_reversion/breakout) may exceed 40% of total portfolio weight.
5. Per-strategy cap: No single strategy may exceed 25% of total portfolio weight.
6. Remaining weight is normalized to (1 − sum of probation weights).

Correlation Computation

Pearson r is computed on daily P&L series for each strategy pair. Requires ≥ 20 overlapping calendar days with trades. Pairs below the overlap threshold are excluded. Correlation is recomputed each evaluation run.

Decay Metrics

• Rolling Sharpe (30D): computed over last 30 calendar days — requires ≥ 10 days with trade activity.
• Rolling Expectancy (30T): mean realized P&L over last 30 closed trades — requires ≥ 15 trades.
• Live Max DD %: peak-to-trough on cumulative closed-trade P&L.
• DD Stress Ratio: live_max_dd_pct / backtest_max_dd_pct.
• Expectancy Ratio: rolling_expectancy / backtest_expectancy.
• Sharpe Ratio (vs baseline): rolling_sharpe_30d / backtest_sharpe.

A strategy deployed on a demo account can be promoted to live execution once it passes five readiness gates. Two promotion paths exist:

Path	Endpoint	Auth	Readiness check
User-initiated	POST /api/strategy/{id}/promote	Standard user auth (middleware)	Yes — blocks if any gate fails
Admin-initiated	POST /api/strategy/{id}/promote_live	Admin token + force=true + override_reason	No — admin bypass (readiness still computed and returned for transparency)

Readiness Gates

All five gates must pass before POST /api/strategy/{id}/promote executes:

1. Baseline registered — backtest metrics must be on file via POST /api/strategy/{id}/baseline. Exemption: Auto-provisioned paper strategies (notes starting with "openclaw") bypass this gate since they have no backtest baseline.
2. Minimum trade count — at least 20 closed demo trades (counts both closed_trades and forex_paper_closed).
3. Portfolio state — strategy must not be disabled or retired in the portfolio engine.
4. Risk budget — risk budget status must not be disabled.
5. Not already live — strategy status must be demo or in_dev.

What Promotion Does (Success Path)

Admin Force Gate (P2)

POST /api/strategy/{id}/promote_live requires an explicit opt-in to prevent accidental readiness bypass:

{
  "session_token": "<ADMIN_TOKEN>",
  "force": true,
  "override_reason": "Manual review — 18 trades PF 1.21"   // min 10 chars
}

Missing force or a short override_reason → 400. Even on force, the full readiness summary is computed and returned. A PROMOTION_FORCE_USED timeline event records the override reason and readiness state for audit.

Unified Timeline (/timeline)

The timeline is a chronological append-only audit log of every significant platform event. Events are emitted only on state transitions — not on every cron evaluation cycle. For example, CHAOS_FILTER_ACTIVE fires once when chaos activates (off→on), and CHAOS_FILTER_CLEARED fires once when it clears (on→off). Written by _emit_timeline() in dashboard_api_lifecycle_helpers.py at instrumented points and read via GET /api/timeline.

Category	Example event_types
strategy	BASELINE_REGISTERED, PROMOTED_TO_LIVE, PROMOTION_REQUESTED, PROMOTION_BLOCKED, PROMOTION_FORCE_USED, STATE_OVERRIDE
portfolio	EVALUATION_RUN, STATE_TRANSITION, WEIGHT_UPDATE
risk	BUDGET_REFRESH, QUALITY_SCORE_CHANGE
ops	WEBHOOK_RECEIVED, ORDER_REJECTED, BE_TRIGGERED
portfolio	STALE_TV_ALERT_AFTER_PROMOTION, WEIGHT_BLOCK, PROMOTION_FAILED, BE_APPLIED, REGIME_SHIFT, CHAOS_FILTER_ACTIVE, CHAOS_FILTER_CLEARED, PRESSURE_APPLIED, PRESSURE_BLOCKED

The "Recent Changes" KPI tile on the main dashboard shows the 24 h event count and links to /timeline. The tile reads from GET /api/timeline/summary.

Ops Health (/ops)

The Ops page aggregates execution health Service Level Indicators (SLIs) from order_events and execution_alerts:

• Webhook rate histogram — last 12 hours, stacked success (green) / fail (red) bar chart from GET /api/ops/webhook_rate.
• Reject reasons — stage, error_code, severity, total count, symbols affected, last seen, from GET /api/ops/reject_reasons.
• KPI tiles — last webhook timestamp, orders 24 h, alerts 24 h, reject rate %, avg half-spread 7D, duplicate alerts, disk usage %.

All Ops endpoints support ?env=live|demo filtering. The page has the same env tab bar as Risk, Alerts, and Portfolio.

Crypto Webhooks Panel

When the crypto instance is active (?asset=crypto), the Ops page shows a Crypto Webhook Events card with:

• KPI row — total events, last 24h count, last webhook timestamp, by-symbol breakdown (from GET /api/crypto/webhooks/summary)
• Events table — most recent 20 events with time, symbol, action, account key, and response (from GET /api/crypto/webhooks)

The panel is hidden when the forex instance is active.

be_manager.py is the canonical server-side R-based break-even module. It persists BE state to SQLite and runs on two cron schedules: every 5 minutes (full ingest cycle) and every 2 minutes (lightweight --be-only tick). The legacy ATR-based BE monitor in main.py is disabled by default via ENABLE_LEGACY_BE_MONITOR=False.

Algorithm

Phase	Action
Phase 0 — Late Registration	Retries rows with retryable disable reasons (TRADEDETAILS_FETCH_FAIL, NO_INITIAL_SL, R_NULL) for trades still open. Enforces 10-minute cooldown via last_checked_at. On success: sets be_enabled=1, clears be_disable_reason, recomputes trigger using the trade's stored be_r_multiple.
Phase 1 — Registration	Scans open_trades WHERE status='open' for trades not yet in break_even_state. Looks up per-strategy be_r_multiple (kR) from strategies table (falls back to system default 1.0). Calls OANDA TradeDetails to read stopLossOrder.price as initial_sl. Computes R = abs(entry - initial_sl) and trigger_price = entry ± kR × R. Sets be_enabled=0 with structured be_disable_reason on any failure path.
Phase 2 — Evaluation	Batch-fetches live bid/ask via OANDA PricingInfo (one call per account). Checks long: BID ≥ trigger_price / short: ASK ≤ trigger_price. On trigger, computes buffer, calls TradeCRCDO to update the SL, emits a BE_APPLIED or BE_BLOCKED alert.

Disable Reasons

Code	Meaning	Retryable?
DISABLED_BY_STRATEGY	Pine payload sent be_atr ≥ 9.0 — strategy explicitly disables BE	No
MISSING_ENTRY_PRICE	open_trades row has no entry_price yet	No
NO_ACCOUNT_CONFIG	account_key not found in ACCOUNT_CONFIG	No
TRADEDETAILS_FETCH_FAIL	OANDA TradeDetails API call raised an exception	Yes
NO_INITIAL_SL	TradeDetails succeeded but stopLossOrder.price was absent	Yes
R_ZERO	initial_sl == entry_price (stop distance is zero)	No
R_NULL	initial_sl fetch succeeded but r_value could not be computed	Yes

Per-Strategy kR (be_r_multiple)

The trigger multiplier is stored per-strategy in strategies.be_r_multiple (NULL = system default 1.0). Set it with:

UPDATE strategies SET be_r_multiple = 1.0 WHERE strategy_id = 'demo_audusd';

Mapping from legacy Pine params: kR = be_atr / stop_atr. For demo_audusd: 0.8 / 0.8 = 1.0.

Buffer Calculation

buffer = max(1 tick, 2 × half_spread)
buffer = min(buffer, 0.10 × R)      # cap at 10% of R

new_sl (long)  = entry + buffer     # just above entry
new_sl (short) = entry - buffer     # just below entry

A 5-decimal instrument like EUR_USD uses tick = 0.00001; JPY pairs use tick = 0.001.

Configuration

Config	Default	Meaning
be_r_multiple (strategies DB column)	NULL → 1.0	Per-strategy kR trigger multiple
BE_R_MULTIPLE (module constant)	1.0	System-wide fallback trigger multiple
ENABLE_LEGACY_BE_MONITOR (config.py)	False	Gates the old ATR-based thread in main.py
BUFFER_HALF_SPREAD_MULT	2.0	Buffer = 2 × half-spread
BUFFER_MIN_TICKS	1	Buffer always at least 1 tick
BUFFER_CAP_FRAC_OF_R	0.10	Buffer capped at 10% of R
BE_DISABLED_ATR_THRESHOLD	9.0	Strategies with be_atr ≥ 9.0 are skipped

Cron Schedule

# Full ingest (poller + BE + risk budgets + snapshot prune) every 5 min
*/5 * * * * /home/ubuntu/trading-bot/venv/bin/python /home/ubuntu/trading-bot/ingest_log_to_sqlite.py

# Lightweight BE-only tick every 2 min (rate-limited by 30s per-trade cooldown)
*/2 * * * * /home/ubuntu/trading-bot/venv/bin/python /home/ubuntu/trading-bot/ingest_log_to_sqlite.py --be-only -q

The full ingest cycle also prunes snapshot tables (strategy_health, strategy_pressure_adjustments) to a 2-hour rolling window and purges acknowledged alerts older than 30 days.

Canonical System

be_manager.py is the canonical BE system. The legacy ATR-based daemon thread in main.py is disabled by default (ENABLE_LEGACY_BE_MONITOR=False in config.py). Set to True only for backwards-compat testing.

API & UI

• GET /api/break_even/open?env= — all open trades with BE state columns including be_disable_reason and be_r_multiple.
• GET /api/trades — enriched with be_enabled, be_applied, applied_at, be_disable_reason, be_r_multiple for closed trades.
• /trades — "BE" column: BE Stop (green), Pending (kR=X.X) (yellow), or Disabled (REASON) (muted). Open positions panel shows trigger price and kR.

Phase 3 of be_manager.py. After BE fires (Phase 2), the trailing stop ratchets the SL behind the peak favorable price, locking in gains as the trade moves further into profit. Unlike BE (one-shot), the trailing stop re-evaluates every cron cycle.

Algorithm

1. Registration — When a new trade is registered in break_even_state, trail params (trail_activate_atr, trail_dist_atr) are read from open_trades (forwarded from webhook). Absolute prices are computed: entry_atr = R / stop_atr, trail_activate_price = entry ± activate × entry_atr, trail_dist = dist × entry_atr.
2. Peak tracking — Each cycle, trail_peak_price is updated to the best bid (long) or ask (short) seen so far.
3. Activation — When peak profit reaches trail_activate_price, trail_active is set to 1. A TRAIL_ACTIVATED alert is emitted.
4. Ratchet — Once active: new_sl = peak - trail_dist (long) or peak + trail_dist (short). If new_sl improves on trail_last_sl, a TradeCRCDO call moves the SL. A TRAIL_RATCHETED alert is emitted.

Coexistence with BE

BE fires first at a lower threshold (1×R). Trailing activates later at a higher threshold. Both only move SL in the favorable direction — no conflict. A trade can have BE applied and trail active simultaneously.

Per-Strategy Configuration

Strategy	trail_activate_atr	trail_dist_atr	Notes
Standard trail (0.5 / 0.3)
demo_audusd (H1 trend)	0.5	0.3	Tight trail for H1 moves
demo_usdcad_h4 (H4 trend)	0.5	0.3	Standard trend trail
demo_eurchf_h4 (H4 trend)	0.5	0.3
eurusd_h1_london_orb (H1 breakout)	0.5	0.3
demo_nzdjpy_h4 (H4 mean rev)	0.5	0.3
demo_nzdusd_h1 (H1 fade)	0.5	0.3
usdjpy_h4_donchian (H4 breakout)	0.5	0.3
usdchf_h1_donchian (H1 breakout)	0.5	0.3
eurjpy_h1_engulfing (H1 engulfing)	0.5	0.3
usdjpy_h1_rsi_div (H1 divergence)	0.5	0.3
eurusd_m30_inside_bar (M30 breakout)	0.5	0.3
eurjpy_h1_fade (H1 fade)	0.5	0.3	Score 2.413
eurusd_h4_donchian (H4 breakout)	0.5	0.3	Score 2.180
eurjpy_h4_donchian (H4 breakout)	0.5	0.3	Score 1.965
gbpusd_h4_donchian (H4 breakout)	0.5	0.3	Score 1.802
audusd_h1_engulfing (H1 engulfing)	0.5	0.3	Score 1.703
Wide trail (1.5 / 0.5)
demo_usdjpy_m30 (M30 trend)	1.5	0.5	Wider trail (1:1 R:R)
demo_audnzd_h4 (H4 trend)	1.5	0.5	Pyramiding strategy
demo_eurgbp_h4 (H4 trend)	1.5	0.5
demo_usdcad_h4_9951 (H4 trend split)	1.5	0.5
demo_usdchf_h4_efc2 (H4 breakout split)	1.5	0.5

Set trail_activate_atr = 0 or trail_dist_atr = 0 to disable trailing for a strategy (default). Note: mean-reversion strategies use a negative trail_activate_atr = -0.40 — this is intentional early loss-cutting, not a disabled trail. See M5 Software Trail § Trail Activation Guard for details.

Alert Types

Code	Meaning
TRAIL_ACTIVATED	Peak profit reached activate threshold — trailing now active
TRAIL_RATCHETED	SL moved forward to lock in gains
TRAIL_BLOCKED	TradeCRCDO call failed — SL not updated

Cron Schedule

Runs on the same cron as BE: every 5 min (full ingest) and every 2 min (--be-only tick). Per-trade cooldown: 30 seconds between SL modifications.

M5 strategies use ultra-tight trailing stops (0.08-0.10 ATR, roughly 0.5-0.8 pips) that fall below OANDA's minimum trailing stop distance of 5 pips. Since OANDA-native trailingStopLossOnFill cannot be used, a software monitor polls prices and closes positions via market order when the trail condition is met.

Why Software Trail?

Constraint	Value	Impact
OANDA minimum trailing stop	5 pips (0.050 JPY, 0.00050 non-JPY)	Rejects any trailingStopLossOnFill below this
M5 ATR (typical)	2-8 pips	Trail at 0.08 ATR = ~0.2-0.6 pips
Solution	Software trail via trail_daemon.py	PricingStream tick-level + bar-close guard, closes via TradeClose

Architecture

• All positions (paper + OANDA live/demo): trail_daemon.py (sole systemd service, trail-daemon.service) consumes OANDA PricingStream for sub-100ms tick data. Paper positions evaluated via forex_paper_engine._evaluate_position() with prices injected into _price_cache. OANDA positions tracked in trail_monitor_state DB table, closed via TradeClose (OANDA) or close_ibkr_position() (IBKR). Bar-close guard ensures trail peak/SL only ratchet at bar boundaries — matching backtest semantics. Spread floor ensures trail ≥ 2× half-spread. Spread-aware activation prevents immediate close on entry.
• Order placement: main.py place_order() checks if computed trail distance is below OANDA's minimum. If so, skips trailingStopLossOnFill entirely (software trail handles it). Also applies a floor to stop_dist to prevent OANDA rejection on low-ATR bars.
• trail_monitor.py retired (v0.4.291): Previously a separate 5s-polling service for OANDA positions. Lacked bar-close guard, causing premature trail exits on intra-bar noise (trade #88434: closed 7s after open). trail_daemon.py is the sole trail service — handles both paper and OANDA positions with correct bar-close semantics.

main.py Gate Logic

_MIN_TRAIL = {"JPY": 0.050, "DEFAULT": 0.00050}   # OANDA 5-pip minimum

if trail_dist_atr > 0:
    trail_dist = round(atr * trail_dist_atr, dist_decimals)
    min_dist = _MIN_TRAIL.get(quote_ccy, _MIN_TRAIL["DEFAULT"])
    if trail_dist >= min_dist:
        # OANDA-native trailing stop (H1/H4 strategies)
        data["order"]["trailingStopLossOnFill"] = ...
    else:
        # Software trail -- trail_daemon.py handles this
        print(f"[Trail] software trail mode")

# Stop distance floor (prevent OANDA rejection)
stop_dist = max(round(atr * stop_atr, dist_decimals), min_dist)

Deployed M5 Fleet Parameters

All 12 M5 strategies use wide safety SL (rarely hit) with ultra-tight software trail handling ~100% of exits. Stored in dynamic_se_strategies.params_json.

Parameter	Mean-Rev (11 strats)	Trend (1 strat)	Purpose
stop_atr	2.0	2.0	Wide safety SL, well above OANDA 5-pip min
tp_atr	3.0	4.0	Wide TP, rarely hit (trail exits first)
trail_dist_atr	0.05–0.08	0.10	Ultra-tight software trail distance
trail_activate_atr	−0.40	0.0	Negative = early loss-cutting from bar 1; see Trail Activation Guard below
be_atr	9.9	9.9	Disabled (trail handles everything)

M5 Fleet Strategies

Strategy	Symbol	Archetype	1Y PF	1Y WR	1Y Trades	2x Cost PF
eurjpy_m5_meanrev	EUR_JPY	mean_rev	9.28	66.3%	17,065	4.62
eurusd_m5_meanrev	EUR_USD	mean_rev	8.46	64.4%	17,558	4.09
usdjpy_m5_trend_trail	USD_JPY	trend	8.24	64.5%	1,739	4.25
gbpjpy_m5_meanrev	GBP_JPY	mean_rev	7.98	64.9%	17,068	3.74
gbpusd_m5_meanrev	GBP_USD	mean_rev	6.61	62.8%	17,330	2.88
usdjpy_m5_keltnersq	USD_JPY	breakout	5.93	61.2%	1,141	2.49
usdjpy_m5_meanrev	USD_JPY	mean_rev	5.76	60.3%	10,834	2.38
usdjpy_m5_structretest	USD_JPY	breakout	5.67	59.5%	3,571	2.37
usdjpy_m5_volbreak	USD_JPY	breakout	5.55	59.2%	3,909	2.29
usdjpy_m5_insidebar	USD_JPY	breakout	5.12	57.4%	991	2.19
audusd_m5_meanrev	AUD_USD	mean_rev	4.95	59.2%	17,216	1.88
usdchf_m5_meanrev	USD_CHF	mean_rev	4.81	58.6%	17,083	1.82

All 12 strategies score 3.000 (max). 100% neighbor stability on fine sweep. Zero losing weeks in 1Y backtest. All survive 2x cost stress.

Trail Activation Guard

trail_activate_atr controls when the trailing stop is allowed to fire. Normally positive (e.g. 0.5 = trail only fires after price moves 0.5 ATR in profit). For the mean-rev fleet, it is set to −0.40 — a negative value meaning the trail can fire from bar 1, even while the trade is in a small loss.

Why negative? Corrected backtests showed losing mean-rev trades riding to the full 2×ATR safety SL (avg loss = −0.83R). The trail only protected winners; losers were untouched. Setting trail_activate_atr = -0.40 lets the trail cut failing trades early — when the trail SL (entry + 0.05 ATR offset) falls below entry − 0.40 ATR, the position closes immediately rather than waiting for the 2×ATR stop.

Scenario	M5 EV/trade	M15 EV/trade	avg Loss
Corrected baseline (trail_activate_atr = 0.0/0.15)	+0.073R	+0.164R	−0.83R
−0.40 deployment (current)	+0.190R	+0.294R	−0.14R

Implementation: Both the paper engine (forex_paper_engine.py) and trail daemon (trail_daemon.py) apply the same guard logic:

if trail_hit:
    act_dist = trail_activate_atr * entry_atr  # negative → act_dist is negative
    if side == "long" and trail_sl < entry + act_dist:
        trail_hit = False   # only suppress if trail_sl is ABOVE threshold
    elif side == "short" and trail_sl > entry - act_dist:
        trail_hit = False

With trail_activate_atr = -0.40 and entry_atr = 5 pip: act_dist = -2 pip. The guard suppresses the close only when trail_sl < entry - 2 pip. Since trail_sl = peak - 0.25 pip and peak starts at entry, the guard is satisfied immediately — early exits are allowed from bar 1.

Deployment (2026-03-19): All 52 meanrev strategies updated in dynamic_se_strategies (v1, v11, v2 fleets). Run on server:

UPDATE dynamic_se_strategies
SET params_json = json_set(params_json, '$.trail_activate_atr', -0.40)
WHERE strategy_id LIKE '%meanrev%';

Trail Sweep Results

Fine sweep across trail_dist_atr values 0.05-0.30 showed tighter trail = monotonically better across all pairs (no crossover point). trail_dist_atr = 0.08 was chosen over 0.05 because 0.05 ATR is approximately 0.3 pips at p10 ATR, dangerously close to spread. 0.08 ATR gives minimum breathing room (~0.5 pips at p10 ATR).

Services

Service	Interval	Scope	Description
trail-daemon.service	PricingStream (tick-level)	Paper + OANDA live/demo	Sole trail service. Consumes OANDA stream, evaluates paper positions via _evaluate_position(), OANDA positions via trail_monitor_state + TradeClose. Bar-close guard on trail ratchet. Also publishes to wm_quote_cache every ~1s for watermark-stop cached mode. Supports IBKR positions.
watermark-stop.service	~1.25s (cached) / 10s (polling)	All paper + live	Peak P&L high-watermark stop. Closes positions intra-bar when unrealized P&L drops to peak_R × protect_ratio after reaching activate_R. Runs in cached mode (reads wm_quote_cache from trail-daemon stream). Falls back to OANDA REST for uncovered symbols.
trail-monitor.service	Retired v0.4.291. Superseded by trail-daemon (which handles both paper and OANDA positions with bar-close guard).

Trail Daemon (All Positions)

trail_daemon.py is the sole trailing stop service for all position types (paper, OANDA demo, OANDA live, IBKR). Replaces both fast-mtm.timer and the retired trail_monitor.py. Maintains a persistent OANDA PricingStream connection.

• Price cache injection: Stream tick prices are written directly into forex_paper_engine._price_cache. When _evaluate_position() calls get_oanda_mid_price(), it finds the price already cached — zero REST calls needed.
• Bar-close trail evaluation: Trail peak ratchets and SL advances only happen at bar boundaries (once per M5/M15/M30 bar). SL/TP/BE checks still run every tick. This matches backtest cadence — backtests evaluated trail at bar close, never intra-bar. Quote noise of 0.3–0.5 pips on M5 ticks was firing 0.32-pip trails prematurely when tick-level evaluation was used.
• SL/TP checks every tick: Fixed SL and TP levels are checked on every tick for fast exits. Only the trail ratchet is bar-gated.
• Position reload: Every tick the daemon reloads positions from SQLite. If the symbol set changes (new position opened, old one closed), the stream reconnects with the updated instrument list.
• Reconnection: Exponential backoff (2s → 4s → ... → 60s cap) on stream errors/timeouts. Symbol changes trigger immediate reconnect (no backoff). systemd Restart=always, RestartSec=5 as final safety net.
• Coexistence: The 5-min ingest cron still runs mark_to_market() as a fallback. close_paper_position() does SELECT WHERE status='open' before delete, so only one process can close a given position.
• Migration: After starting trail-daemon.service, disable the old timer: sudo systemctl disable --now fast-mtm.timer. The fast_mtm.py script remains in the repo as a manual debug tool.

Watermark Stop (Peak P&L Protection)

watermark_stop.py is a separate systemd daemon that closes positions intra-bar when realized profit retraces past a configurable floor. It is independent of trail_daemon and the ingest cron — intentional isolation so it restarts independently.

• Trigger logic: Once a position's unrealized P&L (BID for longs, ASK for shorts) reaches wm_activate_r × initial_risk, the position is "activated". If P&L then drops to or below peak_R × wm_protect_ratio, the position is closed immediately — intra-bar, before the trail SL fires on bar close.
• Quote source modes: WM_QUOTE_MODE=cached (production default) reads from wm_quote_cache written by trail_daemon (~1s freshness, ~1.25s max trigger latency). WM_QUOTE_MODE=polling calls OANDA PricingInfo REST every 10s with no trail_daemon dependency.
• Cached mode peak tracking: trail_daemon buffers ticks in memory and flushes the latest quote per symbol every 1s (latest-wins). Intra-flush-window peaks are not captured — peak_R reflects the max sampled high at ~1s resolution. This is conservative: peak_R may be slightly underestimated, so the protect floor is also slightly lower. The position never closes prematurely from this.
• Fallback: In cached mode, symbols stale in wm_quote_cache (age > 2s) fall back to OANDA REST (rate-limited 10s/symbol). If trail_daemon stops, all symbols fall back to REST automatically — identical to polling mode behavior. Exception: live OANDA positions for symbols with no corresponding paper position (never in trail_daemon's stream) are not included in changed_syms and are silently not evaluated in cached mode. Use polling mode if all active positions must be covered regardless of paper position state.
• Per-strategy params: wm_activate_r and wm_protect_ratio can be set per-strategy in signal_engine/config.py inside the strategy's "params" dict:

  "wm_activate_r": 1.5,
  "wm_protect_ratio": 0.75,

  If omitted (e.g. a strategy with "params": {}), it falls back to the global defaults: activate_r=2.0, protect_ratio=0.70. Set "wm_activate_r": None to disable WM entirely for that strategy. The values are loaded at daemon startup — confirm overrides loaded correctly via the journal: WM config: 6 per-strategy overrides loaded.
• Live positions: Closes via TradeClose (market order). Actual fill is not guaranteed at evaluation BID/ASK. Live canonical close-state is owned by the txn poller — watermark_stop does not update open_trades.status.
• Retry: close_requested=1 is persisted before first attempt (restart-safe). Backoff: 30s × 2^n capped at 300s. TRADE_DOESNT_EXIST is terminal.
• State table: watermark_stop_state — one row per tracked position. Persists peak_r, activated, close_requested, close_attempt_count, last_close_error across restarts.
• Runtime status: wm_daemon_status — single row updated every ~10s. Columns: quote_mode, pub_health, syms_cache/rest/none, positions_tracked/activated/close_requested, last_tick_at. Query: SELECT * FROM wm_daemon_status; Note: syms_none=0 does not catch live-only symbol gaps — symbols absent from trail_daemon's stream are invisible to coverage tracking entirely.

Periodic Calibration: MFE Analysis

The peak_unrealized_pnl column in forex_paper_closed enables Max Favorable Excursion (MFE) analysis — the closest approximation to a live parameter calibration without a full re-backtest. For each historical closed trade you can ask: "did peak P&L reach 2R? If so, would the watermark stop have given a better exit than what actually happened?"

Run this periodically (suggested: once per month, or after accumulating 100+ closed trades with peak data). The query is:

SELECT
  strategy_id,
  COUNT(*) AS trades,
  ROUND(AVG(peak_unrealized_pnl / ABS(realized_pnl)), 2) AS avg_peak_r,
  SUM(CASE WHEN peak_unrealized_pnl >= 2 * ABS(sl_dist) AND realized_pnl < 0 THEN 1 ELSE 0 END) AS wm_wouldve_saved,
  SUM(CASE WHEN peak_unrealized_pnl >= 2 * ABS(sl_dist) AND realized_pnl > 0 THEN 1 ELSE 0 END) AS wm_wouldve_cut_winner
FROM forex_paper_closed
WHERE closed_at >= datetime('now', '-90 days')
GROUP BY strategy_id
ORDER BY wm_wouldve_saved DESC;

What to look for:

• High wm_wouldve_saved count — trades that hit 2R peak but reversed to a loss. These are the WM's target. A high count here validates the 2.0R activation threshold.
• High wm_wouldve_cut_winner count — trades that hit 2R peak then continued to a profitable exit. WM would have closed these early, sacrificing P&L. If this outnumbers wm_wouldve_saved, the protect_ratio (0.70) may be too tight.
• Per-strategy variation — strategies with large MFE swings (high avg_peak_r but mixed outcomes) benefit most from WM. Tight-stop strategies (stop ≤ 0.30 ATR) on M5/M15 often see price blow through any floor before the ~1s cached poll catches it — WM provides less reliable protection there.

DB Table

trail_monitor_state in dashboard.db tracks peak prices and trail SL levels for OANDA positions managed by trail_daemon.py.

Column	Type	Description
trade_id	TEXT PK	OANDA trade ID
strategy_id	TEXT	Strategy identifier
symbol	TEXT	Instrument (e.g. USD_JPY)
side	TEXT	long / short
entry_price	REAL	Trade entry price
entry_atr	REAL	ATR at entry (reconstructed from SL distance / stop_atr)
trail_dist_atr	REAL	Trail distance as ATR multiplier
trail_activate_atr	REAL	Activation threshold as ATR multiplier. Negative = early loss-cutting. Copied from open_trades at first seen. Added in v0.4.137 via idempotent migration.
peak_price	REAL	Best price seen (updated each tick)
trail_sl	REAL	Current trail stop level

Four new mean-reversion strategies deployed 2026-03-12 using the same tight-trail archetype as the M15 fleet (trail=0.08 ATR, software trail) but at the H1 timeframe. These expand the mean-rev portfolio to pairs that failed at M15 due to spread dominance — at H1 the ATR is 3-4× larger, reducing spread/ATR to viable levels.

Why H1 Unlocks These Pairs

Pair	Spread	M15 ATR	M15 spread/ATR	H1 ATR	H1 spread/ATR
EUR_AUD	3.7p	~12p	31% → FAIL	~40p	9% → PASS
CHF_JPY	4.0p	~12p	33% → FAIL	~45p	9% → PASS
AUD_JPY	2.8p	~15p	19% → FAIL*	~55p	5% → PASS
GBP_AUD	4.7p	~15p	31% → FAIL	~50p	9% → PASS

*AUD_JPY failed at M15 due to poor signal quality, not just spread/ATR ratio.

Key Parameter Difference vs M15 Fleet

All H1 strategies use bb_mult=1.4 vs bb_mult=0.8 for the M15 fleet. At H1, tighter bands fire on noise rather than genuine extremes — wider bands (1.4×) select only the most significant mean-reversion setups.

Deployed Strategies

Strategy ID	Symbol	BB	ADX max	Spread	1Y PF	2x PF	WR	1Y Trades
chfjpy_h1_meanrev	CHF_JPY	(20, 1.4)	20	4.0p	7.241	3.017	64%	744
euraud_h1_meanrev	EUR_AUD	(14, 1.4)	50	3.7p	6.932	3.021	62%	1,909
gbpaud_h1_meanrev	GBP_AUD	(14, 1.4)	50	4.7p	6.797	2.736	64%	1,905
audjpy_h1_meanrev	AUD_JPY	(18, 1.4)	45	2.8p	6.404	2.768	62%	1,963

Shared Parameters

Parameter	Value	Notes
granularity	H1	1-hour candles
stop_atr	2.0	Wide safety SL (trail handles exits)
tp_atr	3.0	Wide TP (rarely hit)
trail_dist_atr	0.08	Same tight trail as M15 fleet
trail_activate_atr	−0.40	Early loss-cutting from bar 1 (deployed 2026-03-19)
be_atr	9.9	Disabled
max_bars_held	96	4 days max hold on H1
bb_mult	1.4	Wider bands than M15 fleet (0.8) — required at H1

Robustness

All 504 configs in the fine sweep (9 bb_len × 8 bb_mult × 7 adx_max) passed the multi-window gate (1Y ≥ 50 trades, all windows profitable). The archetype is robust across a very wide parameter range at H1. The fine sweep selected the config with highest 2x stress PF from the top-10 candidates per pair.

The Capital Pressure Model adjusts strategy weights based on detected volatility regimes, enabling controlled aggression during Expansion and defensive positioning during Compression — without changing the core state machine or baseline allocator logic.

Volatility Regime Detection

At each evaluation run, pressure_engine.py fetches daily OHLC candles for all tracked instruments and computes:

• ATR14 percentile — rolling 90-day rank of each instrument's ATR14. Portfolio-level value = median across instruments.
• ADX14 breadth — fraction of instruments with ADX14 > 22.

Regime	ATR Pctl	ADX Breadth	Gross Cap
Compression	< 35	< 35%	1.0×
Normal	35–65	any	1.0×
Expansion	> 65	> 50%	1.2×

Confirmation: A regime must persist for 2 consecutive evaluation days before being applied. This prevents whipsaw from single-day anomalies.

Chaos filter: If ATR percentile > 95 or jumps > 25 pts day-over-day, all multipliers are forced to 1.0 and gross cap is forced to 1.0. Emits CHAOS_FILTER_ACTIVE alert.

Pressure Multipliers

Applied to base weights (pre-normalization) after the state machine evaluation:

Regime	Trend / Breakout	Mean Reversion
Expansion	1.15×	0.90×
Normal	1.00×	1.00×
Compression	0.93×	1.07×

After multiplying, weights are re-normalized to sum 1.0 and the 25% per-strategy cap is reapplied.

Safety Guards

• Portfolio drawdown guard: if portfolio DD > 8% (last 90d), all multipliers forced to 1.0 and gross cap to 1.0. Emits PRESSURE_BLOCKED alert.
• Chaos filter: see above.
• Confirmation required: unconfirmed regime changes are treated as Normal.

Database Tables

Table	Description	Retention
volatility_regime_state	One row per evaluation run. Stores regime, ATR pctl, ADX breadth, confirmed, chaos, gross_cap_target.	730 days
strategy_pressure_adjustments	One row per strategy per evaluation. Stores base_weight, multiplier, pressured_weight, final_weight, blocked.	180 days
market_daily_indicators	Cached daily OHLC + ATR14 + ADX14 per instrument. Refreshed when today's row is missing.	400 days

API Endpoints

• POST /api/portfolio/evaluate?env= — triggers regime detection + pressure computation
• GET /api/portfolio/summary?env= — includes regime, pressure_active, pressure_block_reason
• GET /api/portfolio/weights?env= — includes pressure_multiplier, base_weight, final_weight
• GET /api/portfolio/regime_log?env=&limit= — regime history from volatility_regime_state
• GET /api/portfolio/pressure?env=&ts= — latest pressure adjustments per strategy
• GET /api/risk/summary?env= — includes gross_cap_target, pressure_active, regime

Meta-labeling is a de Prado technique: take a rule-based strategy's signals and use an ML classifier to predict P(win) at each signal bar. If the probability is below the strategy's threshold, the signal is suppressed (no webhook dispatch). Baseline strategies continue firing unchanged; the _ML sibling fires only the high-confidence subset.

Deployed at v0.4.314 (2026-04-17): 14 _ML variants of top-performing meanrev and impulse fade strategies, selected by fleet sweep (ml-research/meta_label_fleet*.py) ranking strategies by PF lift at 50/70/80% signal retention. Thresholds: 0.50 for meanrev, 0.75 for impulse fade.

Components

Component	Purpose
signal_engine/ml_filter.py	Runtime XGBoost inference. Loads model + metadata, builds multi-TF features at signal bar, predicts P(win), returns take/reject. Fail-open on any error (baseline behavior preserved).
signal_engine/engine.py (hook)	After signal fires, if sid.endswith("_ML") and ml_model_path in params, runs filter. Reject sets final_signal=None with reason ml_filter_rejected_p{p}.
trading-bot/ml-models/	Git-tracked model dir. Each model = 2 files: {sid}.json (XGBoost) + {sid}.meta.json (threshold, feature order, base strategy ID, training metadata).
ml-research/train_deployment_models.py	Trains 14 winners on full 1Y at chosen threshold, saves to trading-bot/ml-models/.
backtest/submit_ml_candidates.py	Creates _ML sibling candidates. Reads metadata, merges base params + ml_model_path + ml_threshold, POSTs /api/candidates + approves.

Feature Pipeline

78 features per signal: 13 indicators × 5 TFs (M5/M15/M30/H1/H4) + 8 time features + 1 direction feature. Indicators: ATR, RSI, ADX, BB width, BB penetration, body/wick ratios, ATR percentile, returns (5/10), distance from 20-bar high/low, volume ratio. Multi-TF alignment uses pd.merge_asof(direction="backward") on HTF close time (bar duration added to open time) — guarantees no lookahead.

Safety Design

• Additive only: _ML strategies are new deployments alongside their base. Removing them leaves the baseline untouched.
• Fail-open: any exception in model load, feature extraction, or prediction returns None → signal fires as baseline would. Errors logged to stderr.
• Candle fetch errors → fail-open (skip filter, fire signal).
• NaN features (typical during warmup) → fail-open.
• Missing model file → fail-open.

Expected Behavior

Backtest WF-OOS showed +15-45% PF lift at 50-80% signal retention. Total-return caveat: meta-labeling improves Sharpe by filtering low-confidence signals, but signal count drops — total R/year may fall even when per-trade EV rises. Best for strategies where risk-adjusted return matters more than raw magnitude.

Deployment Flow

1. Run meta_label_fleet*.py to identify candidates with ≥15% PF lift at 70-80% retention.
2. Add candidate + threshold to train_deployment_models.py::CANDIDATES.
3. Run training → writes {sid}_ML.json + .meta.json to trading-bot/ml-models/.
4. Commit models to git (tracked, deployed via admin update).
5. Run submit_ml_candidates.py --session-token <token> → creates + auto-approves all _ML strategies.
6. Restart signal-engine (auto-scheduled by approve).

Deployed Strategies (20+ as of v0.4.318)

• 9 meanrev base (threshold 0.50): usd_cad_h4, eur_aud_h4, aud_usd_m30, gbp_jpy_m15, gbp_usd_m15, gbp_usd_h1, eur_jpy_m15, gbp_usd_m30_pinbar, usd_jpy_m30_long_only — all suffixed _v3_ML
• 5 v4 impulse fade (threshold 0.75): eur_jpy_m15, gbp_jpy_m5, gbp_usd_m5, usd_jpy_m15, aud_usd_m5 — all suffixed _impulse_fade_v4_ML
• 4 H1 short_only: usd_jpy, gbp_jpy, gbp_aud, eur_jpy — suffixed _short_only_v3_nobb_ML
• 2 M15 short_only: eur_usd, gbp_usd — suffixed _short_only_v3(_nobb)_ML

Ensemble Inference (pooled + per-pair) v0.4.320

A POC on v4 M5 impulse fade across 7 pairs showed that averaging a per-pair meta-label model with a pooled cross-pair model beats either alone. Per-pair captures pair-specific patterns but misses cross-pair generalizations; pooled learns pair-invariant structure (incl. pair_id feature so it can still do pair-specific routing) but may dilute strong per-pair edges. Averaging the two predictions keeps the best of both.

POC results (v4 M5 impulse fade, WF-OOS, 7 pairs)

Method	Avg PF lift @80% retention
Per-pair only	+5.3%
Pooled only	+4.0%
Ensemble (avg of both)	+6.6%

How it works

1. ml-research/train_pooled_models.py trains a single XGBoost per archetype on all pairs combined (with pair_id as a feature) and saves to trading-bot/ml-models/{archetype}_pooled_ML.json + .meta.json.
2. Each per-pair .meta.json includes an archetype field (e.g., v4_impulse_m5) and a pair_id.
3. ml_filter.py detects the archetype at load time and loads the companion pooled model if it exists. At inference, runs both predict_proba and averages: proba = 0.5 × (per_pair + pooled). Applies threshold to the averaged probability.
4. Pooled load failure → silent fallback to per-pair only (fail-open).

Current archetypes with pooled models

• v4_impulse_m5 — 7 pairs, 5,769 signals, base WR 65.9%. Applies to 3 deployed _ML (aud_usd, gbp_jpy, gbp_usd M5).
• v4_impulse_m15 — 7 pairs, 2,857 signals, base WR 71.3%. Applies to 2 deployed _ML (eur_jpy, usd_jpy M15).

The remaining 15 _ML strategies (meanrev base, H1 short_only, M15 short_only) do not currently have pooled companions — they run per-pair only until their archetypes are added to train_pooled_models.py. This is safe by default: no archetype field ⇒ pooled lookup skipped ⇒ per-pair behavior unchanged.

Decision payload

The decision dict now exposes ensemble telemetry:

{
  "take": true,
  "proba": 0.8006,        // ensemble average
  "threshold": 0.75,
  "reason": "take",
  "proba_perpair": 0.7609,
  "proba_pooled": 0.8403,
  "ensemble": true         // false if only per-pair prediction ran
}

Three methods coexist for determining a strategy's environment. Use them in this priority order:

1. Explicit env column (preferred)

Tables that store an explicit env column: strategy_health, state_transitions, correlation_snapshots, strategy_pressure_adjustments, volatility_regime_state. Filter with WHERE env = ?.

2. strategies.status JOIN (dashboard queries)

Most dashboard endpoints JOIN on the strategies table and filter by status:

• status IN ('live','demo','in_dev') — all active strategies
• status = 'live' — live environment only
• status IN ('demo','in_dev') — demo environment only

The status value demo_promoted (P5) is automatically excluded from all active views because it does not appear in any IN (...) list.

3. account_key LIKE prefix (ops/alerts tables)

For tables without an env column or a strategies JOIN (e.g. order_events, open_trades, execution_alerts, equity_snapshots):

• Live: account_key LIKE 'live_%'
• Demo: account_key NOT LIKE 'live_%'

Status values reference

Status	Meaning	Active?
live	Trading on real account	Yes
demo	Running on practice account	Yes
in_dev	Strategy under development	Yes
demo_promoted	Demo row retired after P5 split promotion	No
disabled	Disabled by portfolio engine (PF < 1.0 with ≥ 30 trades)	No
retired	Permanently decommissioned	No

P5 split identity columns

Added to the strategies table via lazy migration (_ensure_p5_columns()):

• promoted_from_strategy_id — on live row: points back to demo origin
• promoted_to_strategy_id — on demo row: points to live child
• promoted_at — ISO timestamp of promotion
• promotion_group_id — canonical key (pair_timeframe)

The platform supports a Crypto instance alongside Forex. Users switch between the two via the instance bar at the top of every page.

Instance Selector

The FOREX / CRYPTO tab bar appears above the nav on all pages. Selection persists in localStorage.asset_mode. URL ?asset=crypto overrides localStorage (deep links work). When omitted, defaults to forex — all existing URLs behave identically.

Data Isolation

• strategies.asset_class column: 'forex' (default) or 'crypto'
• All API endpoints accept ?asset= param: forex (default), crypto, or all
• Crypto webhooks are stored in crypto_webhook_events table — never in FX trade_log.jsonl
• No changes to OANDA client or ACCOUNT_CONFIG for crypto strategies

Naming Convention

• Prefixed callers: crypto_demo_* / crypto_live_* — routing determined by prefix
• Any other account name: auto-provisioned as paper, routing determined by DB strategies.status column. Agents never need to change payload on promotion.
• Examples: crypto_demo_ethusd_h1 (prefixed), mybot_eth_h1 (auto-provisioned)

Cost Model

Crypto uses BPS (basis points) rather than pip-based spread modeling. Taker fees are expressed in BPS of notional value.

Execution Routing

Asset	Broker	Key Config
Forex	OANDA v20	OANDA_API_KEY_DEMO / _LIVE
Crypto (spot)	Coinbase Advanced	COINBASE_KEY_FILE_ETH / _BTC
Crypto (perp)	Coinbase CFM	COINBASE_CFM_KEY_FILE (falls back to ETH key)

Execution Modes

Crypto strategies route through three modes. Prefixed accounts (crypto_demo_* / crypto_live_*) use the prefix. All other accounts auto-provision and use strategies.status from the DB:

Condition	Mode	Gate
crypto_demo_* prefix OR auto-provisioned + status=demo	Paper — simulated fills via Coinbase spot prices	ENABLE_CRYPTO_PAPER=true
crypto_live_* prefix OR auto-provisioned + status=live	Live spot — real Coinbase market orders	ENABLE_CRYPTO_SPOT_LIVE=true + _crypto_live_gate()
*_perp_* or symbol=ETP	Perp paper (demo) or live perp (gated)	ENABLE_CRYPTO_PERP_LIVE

Promotion workflow: Change strategies.status from 'demo' to 'live' in the DB. Agent payload stays identical — no prefix change needed. _crypto_live_gate() provides a second safety check (strategy must exist, status=live, asset_class=crypto).

All blocked responses return HTTP 200 {"ok":false, "blocked":true}. Live sell orders require explicit base_size/qty in the payload — no mid-price guessing.

Paper Trading Engine

File: crypto_paper_engine.py. Gated by ENABLE_CRYPTO_PAPER in config.py.

• Price source: Coinbase public API (GET /v2/prices/{symbol}/spot) — no API key needed, 60s cache per symbol
• ATR: Coinbase exchange API hourly candles, Wilder ATR(14), 60min cache
• Sizing: crypto_sizing.compute_crypto_contracts() — contract discretization from crypto_config.json
• Cost model: 8 bps taker fee + 2 bps slippage (from crypto_config.json), deducted on entry and exit
• SL/TP: ATR-based distances (same as FX strategies). stop_atr × ATR and tp_atr × ATR
• Break-even: R-based trigger (1×R), buffer = 2×fee capped at 10%R. be_atr ≥ 9.0 disables BE
• Mark-to-market: Runs every 5 min via ingest cron. Checks SL/TP/BE triggers, updates unrealized P&L, writes equity snapshots

Crypto DB Tables

Table	Description
Paper trading
crypto_paper_accounts	Per-strategy paper account: balance, unrealized P&L, computed equity. Starting balance $10,000.
crypto_paper_fills	Every simulated fill (entry + exit) with fees and slippage.
crypto_paper_positions	Open and closed paper positions with SL/TP/BE prices and mark-to-market state.
crypto_paper_closed	Closed paper trades with realized P&L, exit reason (sl/tp/be_sl/manual), and hold time.
crypto_paper_equity	Equity curve snapshots — one row per strategy per cron tick when a position is open.
Live spot execution
crypto_orders	Spot market orders with client_order_id UNIQUE. Tracks status, filled size, avg price, fees.
crypto_fills	Individual fills with entry_id UNIQUE. Matched to orders via order_id.
crypto_poller_state	Cursor + timestamp state for fill polling. Prevents duplicates across polls.
Perp paper trading
perp_paper_accounts	Per-strategy balance, unrealized P&L, computed equity. Starting balance $10,000.
perp_paper_positions	Open/closed positions with netting state: avg entry, contracts, SL/TP/BE, fill count.
perp_paper_fills	Every simulated fill with action type (entry/add/partial_close/close/reversal_close/reversal_open).
perp_paper_closed	Closed trades with realized P&L, exit reason, entry/exit timestamps.
perp_paper_equity	Equity curve snapshots per strategy per cron tick.
Perp monitoring (CFM)
crypto_perp_products	Cached CFM product catalog. PK on product_id.
crypto_perp_positions	Position snapshots from list_futures_positions(). Indexed by timestamp.

Crypto API Endpoints

Endpoint	Description
Paper trading
GET /api/crypto/paper/summary	Per-strategy balance, equity, unrealized P&L, closed trade stats, win rate.
GET /api/crypto/paper/positions	Open paper positions with current price and unrealized P&L.
GET /api/crypto/paper/trades	Closed paper trades, newest first.
GET /api/crypto/paper/equity	Equity curve data points (default last 7 days).
GET /api/crypto/paper/fills	Raw fill log (entries and exits).
Live spot + perp monitoring
GET /api/crypto/live/orders	Spot live orders with strategy_id/status filters.
GET /api/crypto/live/fills	Spot live fills with strategy_id/symbol filters.
GET /api/crypto/perp/products	Cached CFM/perp product catalog.
GET /api/crypto/perp/positions	Latest CFM/perp position snapshots.

Existing endpoints (/api/open_trades, /api/trades, /api/strategy/{id}/activity, /api/strategy/{id}/chart, /api/strategy/{id}/chart_overlays) return crypto data when ?asset=crypto or when the strategy has asset_class='crypto'.

Webhook Dispatch Flow

1. Webhook arrives with "asset": "crypto"
2. Event always logged to crypto_webhook_events
3. Perp detection: _is_perp_strategy(account, symbol) checks for _perp_ in account name OR symbol matching ETP/PERP/FUT/SWAP
4. Auto-provision: If account has no crypto_demo_/crypto_live_ prefix, look up strategies.status from DB. Auto-provisions strategy row if missing (defaults to status=demo).
5. If perp + demo: perp paper engine with position netting → execute_perp_signal()
6. If perp + live: live Coinbase FCM perp execution (gated)
7. If spot + live: gate check (strategy exists, status=live, supported symbol) → submit_spot_order()
8. If spot + demo: if ENABLE_CRYPTO_PAPER=true → execute_paper_entry()
9. On auto-provision failure: log-only + UNKNOWN_CRYPTO_PREFIX alert (safety net)
10. On error: fallback to log-only (never 500)

Live Spot Execution

File: coinbase_spot_exec.py. Gated by ENABLE_CRYPTO_SPOT_LIVE in config.py.

• Supported symbols: ETH-USD, BTC-USD only (V1)
• Key routing: COINBASE_KEY_BY_SYMBOL maps each symbol to a key file
• Product spec check: verifies product_type == "SPOT" before submitting orders
• Buy sizing: quote_size from CRYPTO_TRADE_NOTIONAL_USD
• Sell sizing: requires explicit base_size or qty from payload — blocks with MISSING_SELL_SIZE if absent
• Dedup: client_order_id UNIQUE on crypto_orders, entry_id UNIQUE on crypto_fills

Perp Paper Engine

File: crypto_perp_paper_engine.py. Simulated perpetual futures trading with position netting — buy against a short reduces/closes/reverses the position (matching real Coinbase FCM behaviour).

• Entry: execute_perp_signal(conn, strategy_id, symbol, action, contracts, stop_atr, tp_atr, be_atr)
• Sizing: caller sends contracts (integer) or notional_usd (converted to contracts via base asset mid price ÷ contract_size)
• Fee model: 8 bps taker fee per side
• Starting balance: $10,000 per strategy
• Contract specs: ETP = 0.1 ETH per contract (ETH-USD), BTP = 0.01 BTC per contract (BTC-USD)
• SL/TP/BE: ATR-based, same as spot paper. BE trigger uses R-based logic with buffer
• MTM: runs every 5 min via ingest cron (mark_to_market()). Checks SL/TP/BE triggers, updates unrealized P&L, writes equity snapshots

Position Netting Algorithm

Current Position	Signal	Result
Flat	buy 1	LONG 1 (new entry)
LONG 1	buy 1	LONG 2 (add, weighted avg entry)
LONG 1	sell 1	FLAT (full close, realize P&L)
LONG 3	sell 1	LONG 2 (partial close, realize P&L on 1)
LONG 1	sell 2	SHORT 1 (reversal: close long + open short)
SHORT 1	buy 2	LONG 1 (reversal: close short + open long)

On add: new_avg = (old_contracts × old_price + new_contracts × new_price) / total_contracts. SL/TP/BE recomputed from new avg entry + current ATR. BE reset to unapplied.

On reversal: two fills recorded (reversal_close + reversal_open). Existing position closed with realized P&L, then new position opened in opposite direction.

Perp Paper DB Tables

Table	Description
perp_paper_accounts	Per-strategy balance, unrealized P&L, computed equity. Starting balance $10,000.
perp_paper_positions	Open and closed positions with SL/TP/BE, avg entry price, contract count, fill count.
perp_paper_fills	Every simulated fill with action type (entry/add/partial_close/close/reversal_close/reversal_open).
perp_paper_closed	Closed trades with realized P&L, exit reason, entry/exit timestamps.
perp_paper_equity	Equity curve snapshots per strategy per cron tick.

Perp Webhook Payload

Field	Required	Description
secret	Yes	Shared webhook secret
asset	Yes	Must be "crypto"
symbol	Yes	ETP (triggers perp routing)
action	Yes	buy or sell
account	Yes	Any account name. Prefixed (crypto_demo_* with _perp_) or any custom name (auto-provisioned). e.g. crypto_demo_eth_perp_h1 or mybot_eth_perp_h1
contracts	One of	Integer number of contracts
notional_usd	One of	USD amount (converted to contracts via ETH price ÷ 0.1)
stop_atr	Yes	Stop loss as ATR multiplier
tp_atr	Yes	Take profit as ATR multiplier
be_atr	No	Break-even trigger (default 9.9 = disabled)
client_signal_id	Required*	Deduplication key. Without this, retries create duplicate trades.

Perp Monitoring (CFM)

File: coinbase_cfm_api.py. Read-only position + product monitoring for live Coinbase FCM perps.

• Product discovery: polls list_products(product_type="FUTURE"), caches to crypto_perp_products
• Position polling: list_futures_positions() snapshots to crypto_perp_positions
• INTX: optional, failure-tolerant — only if COINBASE_INTX_PORTFOLIO_UUID is set
• Cron: runs every 5 min via ingest_log_to_sqlite.py after crypto MTM

Spot Fill Poller

File: coinbase_spot_poller.py. Gated by ENABLE_CRYPTO_SPOT_LIVE.

• Polls get_fills() for each symbol with a configured key file
• Cursor-based pagination with timestamp fallback
• Matches fills to crypto_orders via order_id, updates status/filled_size/avg_price
• Reconciles orders stuck in submitted status > 5 min

Crypto-Aware Pages

• Strategy detail (/strategy/{id}) — crypto strategies show a Lightweight Charts candlestick chart sourced from Coinbase public candle API (no auth required). Open position entry/SL/TP overlays drawn from crypto_paper_positions. Closed trade markers from crypto_paper_closed. Demo strategies show a paper mode notice below the chart.
• Trades (/trades?asset=crypto) — shows paper closed trades + live fills (UNION). Open positions from both paper and live sources.
• Risk (/risk?asset=crypto) — shows paper portfolio equity and open position count.
• Ops (/ops?asset=crypto) — shows crypto webhook events.
• All other pages — filter to asset_class='crypto' strategies via the ?asset=crypto query parameter. No crypto data leaks into the default forex view.

Disk Usage Tile

The Disk Usage KPI tile is visible on both Forex and Crypto tabs (disk is shared infrastructure). It calls GET /api/ops/disk which returns total_gb, used_gb, free_gb, and used_pct via shutil.disk_usage('/'). Thresholds: green < 70%, yellow 70–85%, red ≥ 85%.

The signal engine is a separate process that polls candles on each bar close, runs the same Python prepare() strategies used in backtesting, and dispatches live webhooks. It is the sole execution source for all strategies (TradingView retired 2026-03-13).

Modes

Mode	Behavior
shadow	Log signals to signal_engine_events only. No execution. Used for monitoring/debugging without triggering trades.
live	Log signals AND dispatch to POST /webhook. The webhook handler executes the trade via OANDA or paper engine. Requires --dispatch flag on startup.

Signal Engine Status

The signal engine is the primary and authoritative signal source for all strategies. At startup it merges static config.py strategies with the dynamic_se_strategies DB table (auto-populated on candidate approval). New strategies use paper-first execution: their account key isn’t in ACCOUNT_CONFIG, so the webhook server auto-routes to the FX paper engine.

TradingView alerts were fully retired on 2026-03-13. The signal engine is the sole execution source for all strategies. The parity.py module (python -m signal_engine.run --parity) compares engine dispatch events against webhook executions to verify signal integrity.

Architecture

• Separate process — managed by systemd unit signal-engine.service. ExecStart: python -m signal_engine --loop --dispatch
• Separate DB — writes to signal_engine.db (configurable via SIGNAL_ENGINE_DB_PATH), never touches dashboard.db
• Two data sources — OANDA (forex, demo API key) and Coinbase public API (crypto, no auth)
• Registry-based — strategies loaded from backtest.strategies.REGISTRY (FX), backtest_fx_experimental.strategies.FXEXP_REGISTRY (crypto spot/experimental), or backtest_perp.strategies.PERP_REGISTRY (crypto perp)
• Fail-fast validation — all registry keys validated at startup; missing keys cause immediate error
• Position sync — on startup and every 30 min, syncs shadow positions with OANDA open trades for all forex strategies (prevents cold-start drift)

Dispatch Flow

When mode="live" and a signal fires:

1. Build payload: secret, symbol, action, account, stop_atr, tp_atr, be_atr, source="signal_engine", asset
2. POST to /webhook with 3 retries (0s, 2s, 5s backoff)
3. Log dispatch result to signal_engine_dispatch_log table
4. On all retries failed: emit DISPATCH_FAILED alert to /alerts dashboard
5. Circuit breaker: 5 consecutive failures opens a 5-minute cooldown

Evaluation Flow

1. Check bar timing — only poll if last_bar_time + interval + 15s delay has passed
2. Fetch recent candles (OANDA or Coinbase, complete bars only)
3. Skip if latest bar already processed (dedup on strategy_id + bar_time)
4. Fetch HTF candles if strategy has HTF dependency, inject as strategy.df_htf
5. Run strategy.prepare(df) — same code as backtesting
6. Read raw signal from last bar (df["long"].iloc[-1], df["short"].iloc[-1])
7. Apply engine-level cooldown gate (authoritative, persists across restarts)
8. Log event to signal_engine_events (INSERT OR IGNORE for dedup)
9. Update state (last_bar_time, cooldown_remaining)

Cooldown Enforcement

The engine maintains its own cooldown state in signal_engine_state, separate from the cooldown loop inside prepare(). The engine reads the raw signal from the last bar of prepare() output but applies its own suppression gate. This guarantees correctness across restarts and partial data windows.

Configuration

Strategies are defined in signal_engine/config.py. Each entry specifies:

Field	Description
strategy_id	Matches the dashboard strategy ID. Used as the account field in webhook payloads for paper-first routing.
asset_class	forex or crypto — determines candle source
registry	fx (REGISTRY), fxexp (FXEXP_REGISTRY), or perp (PERP_REGISTRY)
registry_key	Key in the registry dict (e.g. audusd_1h, crypto_momentum)
params	Param overrides applied via setattr after instantiation
htf	HTF candle config: {granularity, bars} or null
mode	shadow (log only) or live (log + dispatch to /webhook)

Dynamic Strategy Loading

At startup, the signal engine merges strategies from two sources:

1. Static — signal_engine/config.py STRATEGIES list (checked into git)
2. Dynamic — dynamic_se_strategies table in dashboard.db (auto-populated on candidate approval via _auto_register_se_strategy())

DB entries that conflict with static IDs are skipped (static takes priority). This means strategies added via candidate approval are automatically picked up on the next signal engine restart — no manual config.py edits needed. The registry_key field on the candidate submission is required for auto-config; candidates without it log a warning and skip SE registration.

DB Tables

Table	Description
signal_engine_state	Per-strategy state: last_bar_time, last_signal_bar_time, cooldown_remaining
signal_engine_events	Every evaluated bar: signal (long/short/null), reason, payload. UNIQUE on (strategy_id, bar_time)
signal_engine_health	Key-value health metrics: last poll time, errors, stale warnings
signal_engine_positions	Shadow position tracking: side, entry_price, SL, TP, bars_held. Synced with OANDA on startup.
signal_engine_dispatch_log	Dispatch audit trail: strategy_id, bar_time, signal, dispatch_ok, status_code, error_message

Execution Coverage

The parity.py module reads dispatch events from signal_engine.db and compares against webhook fill events in dashboard.db (read-only). Matches signals by direction and time proximity, producing a coverage rate per strategy. Engine-only signals indicate cooldown or position-guard blocks; fill-only events indicate unexpected executions. Run via python -m signal_engine.run --parity.

CLI

Command	Description
--once	Single evaluation cycle for all strategies
--once --dispatch	Single cycle + dispatch webhooks for mode="live" strategies
--loop	Continuous polling (15s interval)
--loop --dispatch	Production mode: continuous polling + live dispatch (used by systemd)
--parity	Print dispatch vs execution coverage report
--status	Print current state + health per strategy
--diagnose STRATEGY_ID	Full condition trace for last N bars (default 5). Add --bars N to change.

Dashboard API (read-only)

Endpoint	Description
GET /api/signal_engine/status	Per-strategy state (last_bar_time, cooldown_remaining, paused) + health key-value entries (last_poll_*, error_*, stale_*). Used by the Status tab.
GET /api/signal_engine/events	Recent evaluated-bar events. Params: ?strategy_id=, ?limit= (default 50).
GET /api/signal_engine/positions	Current shadow positions: side, entry_price, SL, TP, bars_held, add_count.
GET /api/signal_engine/dispatch_log	Dispatch audit trail. Params: ?strategy_id=, ?limit= (default 50, max 200).
GET /api/signal_engine/diagnostics/{strategy_id}	Condition traces for a strategy (why signals did or did not fire). Params: ?limit= (default 20).
GET /api/signal_engine/parity	Dispatch vs execution coverage per strategy: total_engine, total_fills, matches, match_rate, engine_only, fill_only counts.

All endpoints open signal_engine.db as read-only. If the DB does not exist, they return empty results.

Per-Pair Regime Gate

Mean-reversion strategies are suppressed on a per-instrument basis when daily ADX(14) indicates a sustained trending regime. This prevents repeated mean-rev entries against a multi-week trend (e.g. EUR/JPY trending for 3 months).

Parameter	Value	Description
REGIME_GATE_ADX_THRESHOLD	25.0	Daily ADX(14) must exceed this value
REGIME_GATE_CONSEC_DAYS	3	Threshold must be breached for this many consecutive days to trigger suppression

Scope: Forex mean-reversion strategies only — identified by regime = 'mean_reversion' in the strategies DB table. Trend and breakout strategies, and all crypto strategies, are unaffected.

Relationship to Capital Pressure Model: These are complementary but distinct. The Capital Pressure Model (Portfolio Intelligence) applies portfolio-level weight multipliers across all instruments in a regime. The regime gate operates at the individual signal level: it suppresses the raw signal for a specific pair before any dispatch occurs, regardless of portfolio weights.

Fail-open guarantees:

• dashboard.db unreadable at init → _meanrev_sids is an empty set → nothing is ever gated
• Daily candle fetch fails → gate returns (False, "") → signal passes through
• ADX computation raises exception → caught, gate returns (False, "") → signal passes through

Cache: ADX values are cached per symbol for the current UTC date (_DAILY_ADX_CACHE module-level dict). At most one OANDA API call per pair per day (~10 calls/day for 10 pairs). Cache is invalidated automatically at day rollover.

Event reason: When suppressed, the signal engine logs reason = "regime_blocked" to signal_engine_events. The raw signal is preserved in the raw_signal column so auditing is lossless. Visible in GET /api/signal_engine/events and the signal engine monitor at /signal-engine.

Dashboard visibility: The Portfolio page has a dedicated Regime Gate tab showing current ADX14 per symbol, total signals suppressed in the last 7 days, and which individual strategies are affected. Symbols where the gate is inactive are shown as "Open" with their current ADX. Data sourced from GET /api/portfolio/regime_gate?days=7.

Friday Session Cutoff

Deployed v0.4.140 (2026-03-19). Blocks new forex signals on Friday afternoons to prevent M5/M15 positions from being open at weekend market close (17:00 ET Friday) and exposed to the Sunday gap.

Parameter	Value	Description
FRIDAY_CUTOFF_ENABLED	True	Master switch. Set False to disable without changing the hour.
FRIDAY_CUTOFF_HOUR_NY	14	2:00 PM ET — any signal evaluated at or after this hour on a Friday is suppressed. Set to 17 to effectively disable (market close).

Rationale: M5 fleet has max_bars_held = 10 (50 min max hold), M15 fleet has max_bars_held = 10 (150 min max hold). A signal fired at 14:30 ET Friday could still be alive when the market closes at 17:00 ET, incurring an unhedged weekend gap. The 14:00 ET cutoff provides ≥3 hours of buffer for M15 and ≥2.5 hours for M5.

Scope: Forex strategies only — checked via s_cfg.get("asset_class") == "forex". Crypto strategies are unaffected. Only NEW signal evaluation is blocked; open positions continue to their natural exit (trail stop, SL, TP, or max_bars expiry).

Event reason: Suppressed evaluations log reason = "friday_cutoff" to signal_engine_events. Visible in GET /api/signal_engine/events.

Implementation: _is_friday_cutoff_active() in signal_engine/engine.py. Called in _evaluate() immediately after the market_closed guard.

Stream Trigger (Bar-Boundary Wake)

Deployed v0.4.142 (2026-03-19). Replaces the blind 15-second polling sleep with a real-time bar-boundary wake driven by OANDA's PricingStream. Reduces worst-case post-bar-close latency from ~20s to 1-3s.

Parameter	Value	Description
STREAM_TRIGGER_ENABLED	True	Master switch. Set False to revert to blind 15s polling.
STREAM_TRIGGER_TIMEFRAMES	{"M5","M15","M30"}	Bar intervals to monitor for boundary crossings.
STREAM_BAR_SETTLE_SECS	3	Seconds to wait after boundary detection before waking the engine, giving OANDA time to close the candle.

How it works: StreamTrigger (a daemon thread in signal_engine/stream_trigger.py) opens a PricingStream for all unique forex instruments. On each price tick, it checks whether the tick timestamp crosses a monitored bar boundary. When a boundary is detected, a threading.Timer fires after STREAM_BAR_SETTLE_SECS and sets a threading.Event. The main loop in cmd_loop() calls wake_event.wait(timeout=15) — it wakes either on the event or after 15s (the old poll interval), whichever comes first.

Fallback: If the stream thread fails to start (missing API key, import error, etc.) or disconnects, wake_event is never set. wake_event.wait(timeout=15) then behaves identically to the old time.sleep(15) — zero behavioral change.

OANDA streaming limit: OANDA allows 2 concurrent PricingStream connections per account. trail_daemon.py uses one (for trailing stops). The stream trigger uses the second. Do not open additional streams without closing one first.

Rollback: Set STREAM_TRIGGER_ENABLED = False in signal_engine/config.py, push, restart signal engine. Engine reverts to 15s polling with zero behavioral difference.

File Structure

trading-bot/signal_engine/
  __init__.py                  # package marker
  __main__.py                  # python -m signal_engine alias
  run.py                       # CLI entrypoint
  engine.py                    # SignalEngine class
  config.py                    # Strategy definitions + timing constants
  dispatcher.py                # WebhookDispatcher (retry, circuit breaker, alerting)
  position_sync.py             # Shadow position sync with OANDA open trades
  candle_fx_oanda.py           # OANDA candle fetch (demo key)
  candle_crypto_coinbase.py    # Coinbase public candle fetch
  candle_cache.py              # In-memory cache for candle dedup
  diagnostics.py               # Condition trace for --diagnose
  state.py                     # SQLite persistence
  parity.py                    # Dispatch vs execution coverage

Any unknown account name hitting the webhook auto-provisions a paper strategy — no flags, no special naming convention required. The asset field ("forex" default or "crypto") determines which paper engine handles it. Existing accounts in ACCOUNT_CONFIG (legacy static + dynamically-loaded OANDA accounts from DB) route to OANDA/Coinbase unchanged. Promotion to live remains a server-side action.

Routing Summary

Asset	Account	Route
crypto	crypto_live_*	Live spot (gated by ENABLE_CRYPTO_SPOT_LIVE + DB status)
crypto	crypto_demo_*	Crypto paper engine
crypto	Anything else	Auto-provision → crypto paper
forex (or missing)	In ACCOUNT_CONFIG	Normal OANDA flow (legacy static + dynamic DB accounts)
forex (or missing)	Not in ACCOUNT_CONFIG	Auto-provision → FX paper engine

Account Naming

Use any account name you like. The display name is derived from symbol + timeframe in the payload, not the account name. Examples:

• mybot_eurusd_h4 → display: "EURUSD H4 (Paper)"
• scanner_v2_audusd → display: "AUDUSD H1 (Paper)" (timeframe from payload)
• With "bot": "alpha1" → display: "ALPHA1 · EURUSD H4 (Paper)"

Auto-Provisioning Flow (FX)

1. Webhook arrives with account name not in ACCOUNT_CONFIG
2. Dedupe check via client_signal_id (reuses openclaw_dedupe table)
3. Strategy row created in DB (strategies table, status=demo)
4. Routed to FX paper engine (forex_paper_engine.py) — simulated fills via OANDA mid prices
5. No OANDA pool lease, no dynamic_account_config — lightweight DB insert only

Auto-Provisioning Flow (Crypto)

1. Webhook arrives with "asset": "crypto" and account not prefixed crypto_demo_*/crypto_live_*
2. Strategy row auto-created (defaults to status=demo)
3. DB status lookup: demo → paper engine, live → Coinbase spot (gated)
4. Agent payload never changes on promotion — only DB status changes

Webhook Payload (Forex)

Field	Required	Description
secret	Yes	Shared webhook secret
symbol	Yes	OANDA format: USD_CAD, EUR_USD, etc.
action	Yes	buy, sell, or close (exits open position)
account	Yes	Any name not in ACCOUNT_CONFIG, e.g. mybot_usdcad_h4
stop_atr	Yes	Stop loss as ATR14 multiplier
tp_atr	Yes	Take profit as ATR14 multiplier
be_atr	No	Break-even trigger (default 9.9 = disabled)
timeframe	Yes (first call)	Used for ATR granularity: H4/D→H4, H1→H1, M30→M30, else M15
style	No	trend (default), mean_reversion, or breakout — stored in strategies table
bot	No	Bot identity prefix. Strategy ID becomes {bot}_{account} for multi-bot separation.
client_signal_id	Required*	Unique per signal for deduplication. Without this, retries create duplicate trades.
tp_price	No	Literal take-profit price (overrides tp_atr calculation)
sl_price	No	Literal stop-loss price (overrides stop_atr calculation)
position_id	No	For action: "close" — integer ID of the position to close. Omit to close the most recent open position.
source	No	Optional source tag for logging

Error Responses

Status	Body	Meaning
200	{"ok": true, "mode": "paper", ...}	Paper fill executed
200	{"ok": true, "deduped": true}	Duplicate signal (already processed)
200	{"ok": true, "mode": "log_only", ...}	Logged but paper engine errored

API Endpoints

Auth: required — X-Webhook-Secret header or ?secret= query param must match WEBHOOK_SECRET. Returns 401 if missing or invalid. All endpoints return both crypto + FX data.

Endpoint	Description
GET /api/bot/status	Trade counts + open position counts (crypto + FX)
GET /api/bot/strategies	All auto-provisioned strategies with trade counts + asset_class
GET /api/bot/trades	Trade history — perp paper fills + FX closed trades (UNION)
GET /api/bot/positions	Open positions — perp paper + FX paper open positions (UNION)
GET /api/bot/events	Event log — crypto_webhook_events + FX order_events (UNION)

Portfolio Health for Paper Strategies

The portfolio engine includes forex_paper_closed trades in health metrics for auto-provisioned strategies (identified by notes starting with "openclaw"). This allows paper strategies to progress through the state machine (probation → healthy) based on paper trade performance.

• Readiness gate: Baseline requirement is bypassed for auto-provisioned paper strategies (no backtest baseline exists). Demo trade count includes both closed_trades and forex_paper_closed.
• Promotion: Paper strategies can be promoted via the readiness card button on the strategy detail page. They follow the same pipeline — pool allocation, dynamic config, restart.

FX Paper Engine

All unknown FX accounts route to the paper trading engine (forex_paper_engine.py). No feature flag needed — paper is always-on for unknown accounts.

Aspect	Detail
Execution	Simulated fills via OANDA mid prices
Pool lease	None (lightweight DB insert only)
Spread cost	Modeled from spread_config.json (p50)
Tables	forex_paper_positions, forex_paper_closed, order_events

Key Differences: Crypto vs Forex

	Crypto	Forex
Execution	Paper (simulated) or Live spot (gated by ENABLE_CRYPTO_SPOT_LIVE)	Paper (always-on for unknown accounts) or OANDA (for ACCOUNT_CONFIG accounts)
Sizing	Caller sends contracts (perp) or server uses CRYPTO_TRADE_NOTIONAL_USD (spot)	Server-side ATR-based (caller sends stop_atr)
Symbol format	Coinbase: ETH-USD, BTC-USD	OANDA: USD_CAD, EUR_USD
Exit action	sell (counter-direction)	close with optional position_id
Position limit	Unlimited	Controlled by max_open_trades_per_symbol (default 4)
Source tagging	source column in fills	source in trade_log.jsonl

Agent Integration Guide

This section covers everything an AI trading agent needs to integrate with the ExecutionLabs webhook. Applies to both FX and crypto strategies.

1. Health Check

Before first trade, verify the server is up:

GET /health   # no auth, returns {"status": "trading bot is running"}

2. Authentication

• Webhook: Include "secret" in the JSON payload body.
• API queries: Pass X-Webhook-Secret header or ?secret= query param. All /api/bot/* endpoints require this.

3. Deduplication (client_signal_id)

Treat as required. Every webhook should include a unique client_signal_id per signal. Without it, retries on network timeouts or 5xx errors will create duplicate trades. Use a UUID or {strategy}_{timestamp}_{bar_time} pattern. The server returns {"ok": true, "deduped": true} for duplicate IDs.

4. Entry Flow

Send a POST /webhook with action: "buy" or "sell". The server returns:

// FX Paper entry response
{
  "ok": true,
  "mode": "paper",
  "strategy_id": "mybot_usdcad_h4",
  "position_id": 42,          // STORE THIS — needed for closing
  "fill_price": 1.3650,
  "units": 1200,
  "side": "long",
  "sl_price": 1.3590,
  "tp_price": 1.3780
}

// Crypto paper entry response
{
  "ok": true,
  "mode": "paper",
  "position_id": 17,
  "fill_price": 2145.50,
  "contracts": 3
}

5. Exit / Close Flow

Asset	How to close
FX (paper)	Send action: "close" with optional position_id (integer). Without it, server closes the most recent open position for that strategy+symbol.
Crypto (paper/live)	Send action: "sell" for a long position (counter-direction). For live sells, include base_size or qty.

FX paper close response:

{
  "ok": true,
  "mode": "paper",
  "strategy_id": "mybot_usdcad_h4",
  "position_id": 42,
  "exit_price": 1.3720,
  "realized_pnl": 84.00
}

6. Response Interpretation

Always check the ok field, not the HTTP status code. The server returns HTTP 200 for most responses (even blocked/skipped) to prevent webhook retries.

Response	Meaning	Agent action
{"ok": true, "mode": "paper", ...}	Trade executed (paper or live)	Store position_id, log fill
{"ok": true, "deduped": true}	Signal already processed	No action (safe to ignore)
{"ok": true, "mode": "log_only", ...}	Logged but not executed (feature disabled, engine error)	Check reason, may need config fix
{"ok": false, "blocked": true, ...}	Blocked by guard (position limit, live gate, etc.)	Do not retry — check reason
{"ok": false, "reason": ...}	Error (sizing, engine, etc.)	Log error, may retry with different params

7. Retry Guidance

• HTTP 5xx or network error: Retry with same client_signal_id (dedup protects against doubles).
• HTTP 200 with ok: false: Do NOT retry — the server intentionally blocked this signal.
• HTTP 200 with ok: true, deduped: true: Already processed — no action needed.
• HTTP 401/403: Bad secret — check your secret field (watch for shell variable interpolation of $ characters).

8. Querying State

Use the API endpoints to track positions and performance. All require X-Webhook-Secret header.

Endpoint	Use case
GET /api/bot/status	Summary: total trades, open count, P&L across FX + crypto
GET /api/bot/positions	All open positions with entry price, side, unrealized P&L
GET /api/bot/trades	Closed trade history with realized P&L
GET /api/bot/strategies	All your strategies with trade counts + asset class
GET /api/bot/events	Full event log (webhook events + order events)

9. Position Sizing

• FX: Agents do NOT control position size. Send stop_atr and the server calculates units from balance × risk_pct / (ATR × stop_atr). All accounts (live, OANDA demo, paper) use risk_pct = 1.0% on a $1,000 base — sizing is further scaled by portfolio weight when weight-based sizing is enabled.
• Crypto spot: Server uses CRYPTO_TRADE_NOTIONAL_USD for buy sizing. Sells require explicit base_size/qty.
• Crypto perp: Agent sends contracts or notional_usd (converted to contracts via ETH price).

Agents can optionally send tp_price and sl_price as literal prices (FX paper mode only). These override the ATR-based calculation.

10. Bot Identity (bot field)

For multi-bot setups, include a "bot" field in the payload. This prefixes the strategy ID: {bot}_{account}. Each bot gets separate strategy rows, positions, and P&L tracking on the dashboard. The value can be any string the bot is configured to send (e.g. "clawbot", "alpha1", "scanner_v2").

Symbol Format Reference

Asset	Format	Examples
Forex	OANDA (underscore)	USD_CAD, EUR_USD, AUD_NZD
Crypto spot	Coinbase (dash)	ETH-USD, BTC-USD
Crypto perp	Coinbase product	ETP (ETH perpetual)

Interactive Brokers (IBKR) is supported as a secondary live execution broker alongside OANDA. IBKR offers tighter spreads on major FX pairs via IDEALPRO ECN, with a commission break-even vs OANDA standard at ~$1,500 account size — IBKR wins at $2,000+. The feature is gated by ENABLE_IBKR_LIVE=false (default off) and is additive — zero changes to existing OANDA code paths.

Architecture

Separate client module + execution module + transaction poller + routing branch in the webhook handler. Market data (ATR, quote-to-USD conversion, trailing/BE prices) always comes from OANDA — IBKR is used only for account balance queries, order placement, SL modification, and position close.

• ibkr_client.py — Plain REST client targeting the local Client Portal Gateway at localhost:5000. No OAuth. Singleton via get_ibkr_client(). Symbol→conid mapping for 22 FX pairs in OANDA_TO_IBKR_CONID.
• ibkr_exec.py — Order execution: submit_ibkr_order() (bracket orders), close_ibkr_position(), modify_ibkr_sl().
• ibkr_session_daemon.py — Fully automated session management: POST /tickle every 55 s, auth-check every cycle, proactive midnight reauth at 23:56 ET, startup reauth if needed. Runs as ibkr-session.service.
• ibkr_txn_poller.py — Fill poller: syncs closed trades + reconciles open positions every 5 min via ingest cron.

Auth: Client Portal Gateway

IBKR's Client Portal Gateway is a lightweight JAR that runs on the server at https://localhost:5000. No OAuth keys are required. All API endpoint paths are identical to the hosted Web API. Two systemd services manage the full lifecycle:

• ibkr-gateway.service — Runs the Gateway JAR (/home/ubuntu/clientportal.gw). Restarts automatically on failure.
• ibkr-session.service — Manages the authenticated session with three layers of protection:

1. Keepalive: POST /tickle every 55 s prevents the Gateway's 5-minute inactivity timeout.
2. Auth check every cycle: GET /iserver/auth/status is called independently every tick. Critical distinction — /tickle returns HTTP 200 even when the session is unauthenticated, so auth must be verified separately.
3. Proactive midnight reauth: At 23:56 ET daily (4 minutes before IBKR's nightly security reset), the daemon reauthenticates via POST /iserver/auth/ssodh/init with username + password + pyotp.TOTP(seed).now(). Verifies success and retries once immediately if unconfirmed — 0-second gap at the reset.
4. Reactive reauth: Any unexpected auth drop is detected within ≤55 s and reauthenticated with a 90-second cooldown between attempts.
5. Startup reauth: If the daemon restarts and the session is not authenticated, it reauthenticates automatically — no manual browser login needed after the initial setup.

Execution Flow

1. Signal engine dispatches to /webhook with account key mapped to an IBKR entry in ACCOUNT_CONFIG (broker: "ibkr").
2. Webhook handler checks ACCOUNT_CONFIG[account].get("broker") == "ibkr" — routes to IBKR branch, OANDA path never reached.
3. ATR + quote-to-USD fetched from OANDA (any available OANDA client — market data is broker-agnostic).
4. Account balance fetched from IBKR via get_ibkr_account_balance() → NetLiquidation.
5. Units sized via standard calc_units() — rounded to nearest 1,000 for IDEALPRO lot sizing.
6. Spread gate applied (using OANDA spread data).
7. Bracket order submitted via submit_ibkr_order(): parent MKT + child STP (stop loss) + child LMT (take profit). Confirmation challenges auto-confirmed.
8. Fill written to open_trades with broker='ibkr'.

Bracket Orders

IBKR does not support stopLossOnFill/takeProfitOnFill like OANDA. Orders are submitted as a bracket: a parent MKT order with two child orders (STP for stop loss, LMT for take profit) referencing the parent by client order ID. IBKR sometimes responds with confirmation challenges; these are auto-confirmed by submit_ibkr_order().

IDEALPRO Lot Tiers

IBKR IDEALPRO has no hard minimum — all order sizes are accepted. However, there is a meaningful pricing threshold:

• 20,000+ units — Full IDEALPRO rate: $2.00/side commission, tightest ECN spread.
• 1,000–19,999 units — Odd lot: $2.50/side + approximately 1 pip wider spread.

calc_units() output is rounded to the nearest 1,000 before submission. Commission break-even vs OANDA standard (~$3.00 round-trip) is approximately $1,500 account size. At $2,000+, IBKR is consistently cheaper per trade.

Symbol Mapping

OANDA uses underscore-separated symbols (EUR_USD). IBKR identifies instruments by numeric conid (Contract ID). The static OANDA_TO_IBKR_CONID dict in ibkr_client.py covers 22 FX pairs (all deployed strategy symbols). Exchange: IDEALPRO for all FX pairs.

Monitoring Daemons

All three monitoring daemons branch on open_trades.broker:

• trail_daemon.py — Handles IBKR trailing exclusively. Close action: close_ibkr_position() for IBKR, OANDA TradeClose otherwise. Price data from OANDA PricingStream (broker-agnostic).
• watermark_stop.py — Close action: close_ibkr_position() for IBKR, OANDA TradeClose otherwise. P&L calculated from OANDA pricing regardless of broker.
• be_manager.py — Phase 2 (BE trigger): calls modify_ibkr_sl() for IBKR (modifies the STP child order price), TradeCRCDO for OANDA. Phase 3 (trailing): skips IBKR accounts entirely — trail_daemon.py is the sole owner of IBKR trailing stop management.

Promotion

The promote modal on /strategy/{id} accepts a broker field ("oanda" or "ibkr"). Both the user-gated /promote and admin-gated /promote_live endpoints write broker to dynamic_account_config. IBKR promotions use the shared account U24973157 — no pool slot consumed. OANDA promotions use the existing practice/live account pool.

Environment Variables

• IBKR_ACCOUNT_ID — IBKR account number (default U24973157)
• IBKR_BASE_URL — Gateway URL (default https://localhost:5000/v1/api)
• IBKR_USERNAME — IBKR login username (session daemon automated reauth)
• IBKR_PASSWORD — IBKR login password (session daemon automated reauth)
• IBKR_TOTP_SEED — base32 TOTP seed extracted from Google Authenticator export QR code
• ENABLE_IBKR_LIVE — feature flag, false by default. Set true to enable live execution.

OANDA Isolation

Server Setup (one-time)

# 1. Install Java
ssh trading-bot "sudo apt-get install -y openjdk-11-jre-headless"

# 2. Download + extract Client Portal Gateway
ssh trading-bot "cd /home/ubuntu && \
  wget https://download2.interactivebrokers.com/portal/clientportal.gw.zip && \
  unzip clientportal.gw.zip -d clientportal.gw"

# 3. Install pyotp
ssh trading-bot "/home/ubuntu/trading-bot/venv/bin/pip install pyotp"

# 4. Add env vars to server .env
# IBKR_USERNAME, IBKR_PASSWORD, IBKR_TOTP_SEED, IBKR_ACCOUNT_ID, ENABLE_IBKR_LIVE=false

# 5. Deploy systemd services (files in trading-bot/deploy/systemd/)
scp trading-bot/deploy/systemd/ibkr-gateway.service trading-bot:/tmp/
scp trading-bot/deploy/systemd/ibkr-session.service trading-bot:/tmp/
ssh trading-bot "sudo mv /tmp/ibkr-*.service /etc/systemd/system/ && \
  sudo systemctl daemon-reload && \
  sudo systemctl enable ibkr-gateway ibkr-session"

# 6. Start the Gateway service
ssh trading-bot "sudo systemctl start ibkr-gateway"

# 7. One-time browser login — LOCAL machine only, keep tunnel open
ssh -L 5000:localhost:5000 trading-bot
# Open https://localhost:5000 in browser → login with username + password + TOTP
# After login: authenticated:true in the browser page

# 8. Start session daemon (handles all future reauth automatically)
ssh trading-bot "sudo systemctl start ibkr-session"

# 9. Verify
ssh trading-bot "curl -sk https://localhost:5000/v1/api/iserver/auth/status"
# Expect: {"authenticated":true,"connected":true,"established":true,...}

# 10. Enable live execution
ssh trading-bot "sed -i 's/ENABLE_IBKR_LIVE=false/ENABLE_IBKR_LIVE=true/' \
  /home/ubuntu/trading-bot/.env && sudo systemctl restart trading-bot"

Full server state is backed up every 4 hours to a private GitHub repository via cron. SQLite databases use sqlite3 .backup for safe hot-copy (handles WAL mode correctly without stopping the service). The backup script lives in the code repo at trading-bot/backup-dbs.sh and is deployed via the normal git push flow.

What Gets Backed Up

Item	Destination	Contains
dashboard.db	dashboard.db	Strategies, trades, alerts, portfolio state, risk budgets, account pools, admin, timeline events (~13 MB)
signal_engine.db	signal_engine.db	Shadow signals, diagnostics, engine state, dispatch log (~4 MB)
.env	.env	API keys, webhook secret, account IDs
~/.secrets/*.json	secrets/	Coinbase API key files
trade_log.jsonl	trade_log.jsonl	Append-only order event log
Systemd service files	systemd/	trading-bot, signal-engine, trail-daemon, watermark-stop service files
Nginx config	nginx/trading-bot.conf	Reverse proxy + SSL config (post-Certbot)
SSH deploy key	ssh/	github-backup-key, .pub, config
Crontab	crontab.txt	Live crontab snapshot

Schedule

Runs every 4 hours (00:00, 04:00, 08:00, 12:00, 16:00, 20:00 UTC) via cron. Each run commits the current snapshots and pushes to GitHub.

0 */4 * * * /home/ubuntu/trading-bot/backup-dbs.sh

Infrastructure

• Repo: SoftJazz1/tradebot-server-db-backup (private)
• SSH key: /home/ubuntu/.ssh/github-backup-key (ed25519, deploy key with write access)
• SSH config: Host github-backup alias in ~/.ssh/config
• Script: /home/ubuntu/trading-bot/backup-dbs.sh — version-controlled in the code repo; deployed via git push
• Log: /home/ubuntu/backup-dbs.log
• Local clone: /home/ubuntu/tradebot-server-db-backup/
• Auto-squash: After 14 local commits, the script squashes history to a single commit and force-pushes. Keeps the local repo small while GitHub retains full push history.
• GitHub limit: 100 MB per file hard limit. Snapshot table pruning (2-hour retention on strategy_health + strategy_pressure_adjustments) keeps dashboard.db under this limit.

Manual Trigger

# Run backup immediately
ssh trading-bot "bash /home/ubuntu/trading-bot/backup-dbs.sh"

# Check last run
ssh trading-bot "tail -10 /home/ubuntu/backup-dbs.log"

Restore

# Clone the backup repo
git clone git@github.com:SoftJazz1/tradebot-server-db-backup.git

# Copy a DB back to the server (stop service first)
ssh trading-bot "sudo systemctl stop trading-bot"
scp dashboard.db trading-bot:/home/ubuntu/trading-bot/data/dashboard.db
ssh trading-bot "sudo systemctl start trading-bot"

One consolidated email per recipient per day containing all enabled strategies. Sent via Amazon SES (sandbox mode — recipients must be verified identities). Triggered by the ingest cron every 5 minutes; hour-gated to fire once at the configured send_hour_utc.

Architecture

File	Role
email_renderer.py	Pure rendering — gathers data from dashboard DB, returns HTML string. No SMTP, no DB writes.
email_sender.py	Orchestrator — reads SMTP config, hour gate, dedup, renders consolidated email, sends via SMTP.
dashboard_api_per_strategy.py	Toggle API: GET/POST /api/strategy/{id}/daily_email
templates/strategy-detail.html	UI toggle on strategy detail page (live strategies only)

SMTP & SES Configuration

SMTP settings are stored in the email_config DB table (key/value pairs). Password is in the server .env as SMTP_PASSWORD.

Key	Value
smtp_host	SES SMTP endpoint (e.g. email-smtp.us-east-2.amazonaws.com)
smtp_port	587 (STARTTLS)
smtp_user	SES SMTP credentials (IAM-generated, not AWS access key)
smtp_from	reports@executionlabs.click
smtp_tls	1 (STARTTLS) or 0 (SSL)
send_hour_utc	Hour (0–23) to send. Default: 8 (8 AM UTC)

SES Sandbox: Sending is limited to verified identities (both sender and recipients). Verify recipients in the SES console under Identities → Create identity → Email address, or via CLI: aws ses verify-email-identity --email-address user@example.com. Sandbox limits: 200 emails/day, 1 email/sec — more than sufficient for daily digests.

Consolidated Email Format

All strategies with status='live' AND daily_email_enabled=1 are rendered into a single email per recipient. Each strategy section includes:

• Strategy name sub-header (pair + timeframe)
• Today’s Performance KPIs (P&L, trades, win rate, W/L)
• Closed Trades table (today, if any)
• Open Positions table
• Recent Trade History (last 10)
• Account Equity KPIs + 30-day NAV chart (base64 PNG inline)
• Strategy Health (state, PF, drawdown)
• Operator Notes (rules-based observations)
• Today’s Alerts (if any)
• Per-strategy link to /strategy/{id}

Strategies are separated by a horizontal divider. Shared document header shows date + strategy count; shared footer links to the main dashboard.

Dedup & Send Log

The email_send_log table has a UNIQUE constraint on (strategy_id, send_date, recipient). The consolidated email uses strategy_id='_consolidated' as the dedup key — one row per recipient per day. No migration needed; old per-strategy rows don’t conflict.

Recipients

Stored in the email_recipients table (email + enabled flag). Managed via the Admin page (/admin). All enabled recipients receive the same consolidated email.

Toggle API

Endpoint	Description
GET /api/strategy/{id}/daily_email	Returns { daily_email_enabled: bool }
POST /api/strategy/{id}/daily_email	Body: { "enabled": true|false }. Returns 400 if strategy is not status='live'.

Cron Trigger

email_sender.send_daily_digests() is called from the ingest cron (ingest_log_to_sqlite.py) every 5 minutes. It checks now.hour == send_hour_utc; outside the send hour it returns immediately. Within the send hour, dedup prevents re-sends.

Test Email

Send a test email from the Admin page (/admin → Daily Email tab). Uses send_test_email() which renders all enabled strategies into a consolidated preview. Does not write to the send log (no dedup entry created).

Error Handling

If one strategy fails to render, _render_strategy_section() catches the exception and returns a minimal error placeholder (“Error loading {strategy_id}”). Other strategies in the email are unaffected.

The Knowledge Base must stay in sync with the code. Any change to webhooks, APIs, execution flow, sweep gating, or Portfolio Intelligence must also update the relevant KB sections and glossary entries.

Files that require a KB update when changed

• main.py — new webhook fields, route changes, position sizing formula
• dashboard_api.py and extracted router modules (dashboard_api_*.py) — new or changed API endpoints
• risk_budget.py — quality score methodology changes
• portfolio_engine.py — state machine thresholds, weight caps, correlation formula
• email_renderer.py, email_sender.py — email digest format, send logic, SMTP config
• README*.md, STRATEGY_GUIDE.md, IMPLEMENTATION_CHECKLIST.md

Docs Sync Check Script

A pre-commit heuristic script is available at scripts/docs_sync_check.py. Run it before committing significant architecture changes:

python scripts/docs_sync_check.py
# Exits non-zero if core files changed without a KB update.
# Override: include "[docs-skip]" in your commit message.