How to use it
- 1. Press playThe agent lives one period per step: see inventory, order 0/5/10/15, meet Poisson demand, pay the costs.
- 2. Watch three viewsThe Q-table heat-map, the cost-per-episode curve vs two benchmarks, and the implied policy chart.
- 3. Steer the learningε controls exploration, α the learning rate, and the speed buttons run 1×/10×/100×.
- 4. Quiz the agentPause anytime and predict what it will do in a random state before revealing its answer.
The learning room
Run controls
The world
Changing the world resets the agent - it must learn the new economics from scratch. Benchmarks retune automatically.
Q-table: rows are inventory states 0–20, columns the four order sizes. Darker navy = better action value; the red ring is the agent's current state. Watch columns crystallise as it learns.
Cost per period, episode by episode (red). Dashed teal = tuned (s,S) policy; dashed grey = naive fixed order. Learning shows as the red curve settling between them - then hugging the teal line.
The implied policy: for each inventory level, the order size the agent currently thinks best. A learned (s,S)-like shape is a downward staircase - big orders when empty, nothing when full.
Try these experiments
- Set ε to 0 early (first ~50 episodes) and watch the agent get stuck on a mediocre policy it never explores its way out of.
- Raise the fixed order cost K and watch the learned policy widen its gap - order rarely, but big, exactly like (s,S) says.
- Find ε/α settings that get within 10% of the benchmark inside 300 episodes - then argue why the same tension (explore vs exploit) appears in every supply chain pilot.
About this tool
The concept
Reinforcement learning improves a policy from experienced rewards alone - no demand forecast, no formula, just trial, error and feedback.
How it works
Tabular Q-learning: Q(s,a) ← Q + α(r + γ·maxQ(s′) − Q). ε-greedy exploration; states are inventory levels, actions are order sizes.
The benchmark
Inventory theory says an (s,S) policy is optimal here. The agent never hears that - yet its learned policy converges toward the same shape.