Poker RL · PPO, DQN & MCTS
Train, compare and understand poker agents. A Texas Hold’em environment bringing together PPO, DQN and Monte Carlo tree search.

Texas Hold’em · Multi-agent environment
Play to learn.Measure to understand.
In poker, a good decision does not guarantee a winning hand. This project explores that distinction: learning a strategy in an imperfect-information game, then observing what it actually does.
- players at the table
- 2–5
- discrete actions
- 8
- approaches compared
- 3
Deciding without the full picture.
Opponents’ cards stay hidden, bets change throughout a hand, and randomness clouds the outcome of an individual decision. Texas Hold’em provides a concrete setting for exploring risk, exploration and policy learning.
The project connects three elements: an engine that applies the rules, agents that choose actions, and tools for comparing their behaviour. The aim is to make experiments observable and reproducible, beyond the percentage of hands won.
From observation to action.
A shared interface lets agents change without changing the game engine. At each decision, an agent receives only information available to its seat and a mask of legal actions.
Observe
Private cards, board, position, stacks, bets and amount to call.
Decide
The agent selects one of eight actions, respecting the legal-action mask.
Play
The engine applies the action, advances the hand and settles rewards.
Learn
During training, transitions feed the model’s updates.
Fold, check, call, raise to 3 BB, half-pot, pot, two pots or go all-in: actions remain discrete. Poker features enrich observations with ranks, known combinations and the relative cost of calling. They never use hidden opponent cards. The Monte Carlo equity displayed in the interface does not feed the PPO policy.
Three ways to choose a move.
Agents share an environment but use different decision mechanisms. Random players, heuristics and a human mode complement the comparisons.
Learn a policy
Stable-Baselines3 Contrib’s MaskablePPO trains a PyTorch actor–critic network. The current model is a memoryless MLP. Illegal actions are masked, and the policy supports sampled or deterministic decisions.
Estimate action values
A network predicts a Q-value for each action. A replay buffer reuses past transitions during training, while exploration follows an epsilon-greedy strategy.
Explore possible futures
Information-set search samples hidden cards compatible with the current situation. Simulations and PUCT selection guide exploration, using a rollout policy based on hand strength and pot odds.
Separate learning, selection and testing.
A training curve describes a changing agent. Assessing a model also requires playing without updates on situations reserved for evaluation.
Train
Independent hands, terminal rewards normalised by the initial stack for new PPO models, varied opponents and frozen snapshots. Campaigns can mix 20, 50 and 100 BB stacks.
Select
Compare checkpoints and decision modes on a separate selection set. Campaigns can evaluate multiple initialisations and learning rates before choosing a setting.
Test
Evaluate frozen copies using new deals, rotated positions and reset stacks on every hand. The arena separately evaluates complete matches with persistent stacks.
Results need their context.
The archived seed42_transfer benchmark compares agents trained at two- and four-player tables, including transfer between formats. Its validation report records 24 models, 216 final matchups and 259,200 simulated hands, with no truncated hands. These counts cover the whole benchmark.
Excerpt · PPO trained with 2 players · curriculum followed by self-play · 2-player table · 50 BB stacks · 1,200 hands per opponent · training seed 42.
| Opponent | Profit BB/100 | 95% interval |
|---|---|---|
| Random | +279.0 | [+123.9 → +434.1] |
| Frozen PPO | -48.9 | [-215.7 → +118.0] |
| Frozen DQN | +87.4 | [-117.0 → +291.8] |
| Frozen MCTS | -175.0 | [-302.4 → -47.7] |
In this excerpt, estimated profit is positive against the random agent and negative against MCTS. The respective intervals stay above and below zero. The intervals against PPO and DQN cross zero, leaving those matchups uncertain at this sample size.
Intervals are pointwise normal approximations at 95%, computed separately for each matchup. This experiment uses one training seed: it does not measure variability across initialisations, exploitability or strength against human players. The interface screenshots below show other sessions. Their curves do not correspond to this excerpt.
See what the agent learns.
The Pyglet application supports the experiment: choose agents, watch a game, follow a campaign, save a checkpoint and inspect validation hands. These screenshots come from the local project.
Behaviour matters as much as the score.
The project exposes the connections between card representation, opponent selection, evaluation design and the resulting strategy. Next steps include comparing more initialisations, broadening opponents and examining weaknesses by situation rather than seeking a single score.
Explore the GitHub repositoryThis presentation draws on the local code and experiment archives. The public repository may differ from this working version. The engine retains Neuron Poker’s original MIT licence, attributed to Nicolas Dickreuter.

